* [RFC PATCH 1/6] bus/cdx: introduce cdx bus
2023-01-24 14:07 [RFC PATCH 0/6] add support for CDX bus Nipun Gupta
@ 2023-01-24 14:07 ` Nipun Gupta
2023-01-24 14:07 ` [RFC PATCH 2/6] bus/cdx: add dma map and unmap support Nipun Gupta
` (12 subsequent siblings)
13 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-01-24 14:07 UTC (permalink / raw)
To: dev, thomas, david.marchand; +Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal
CDX bus supports multiple type of devices, which can be
exposed to user-space via vfio-cdx.
vfio-cdx provides the MMIO IO_MEMORY regions as well as the
DMA interface for the device (IOMMU).
This support aims to enable the DPDK to support the cdx
devices in user-space using VFIO interface.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
MAINTAINERS | 5 +
drivers/bus/cdx/cdx.c | 564 ++++++++++++++++++++++++++++++++++
drivers/bus/cdx/cdx.h | 54 ++++
drivers/bus/cdx/cdx_logs.h | 37 +++
drivers/bus/cdx/cdx_vfio.c | 428 ++++++++++++++++++++++++++
drivers/bus/cdx/meson.build | 13 +
drivers/bus/cdx/rte_bus_cdx.h | 219 +++++++++++++
drivers/bus/cdx/version.map | 12 +
8 files changed, 1332 insertions(+)
create mode 100644 drivers/bus/cdx/cdx.c
create mode 100644 drivers/bus/cdx/cdx.h
create mode 100644 drivers/bus/cdx/cdx_logs.h
create mode 100644 drivers/bus/cdx/cdx_vfio.c
create mode 100644 drivers/bus/cdx/meson.build
create mode 100644 drivers/bus/cdx/rte_bus_cdx.h
create mode 100644 drivers/bus/cdx/version.map
diff --git a/MAINTAINERS b/MAINTAINERS
index 9a0f416d2e..9edce0e739 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -563,6 +563,11 @@ M: Parav Pandit <parav@nvidia.com>
M: Xueming Li <xuemingl@nvidia.com>
F: drivers/bus/auxiliary/
+CDX bus driver
+M: Nipun Gupta <nipun.gupta@amd.com>
+M: Nikhil Agarwal <nikhil.agarwal@amd.com>
+F: drivers/bus/cdx/
+
Intel FPGA bus
M: Rosen Xu <rosen.xu@intel.com>
F: drivers/bus/ifpga/
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
new file mode 100644
index 0000000000..4c85c5dc01
--- /dev/null
+++ b/drivers/bus/cdx/cdx.c
@@ -0,0 +1,564 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+/*
+ * Architecture Overview
+ * =====================
+ * CDX is a Hardware Architecture designed for AMD FPGA devices. It
+ * consists of sophisticated mechanism for interaction between FPGA,
+ * Firmware and the APUs (Application CPUs).
+ *
+ * Firmware resides on RPU (Realtime CPUs) which interacts with
+ * the FPGA program manager and the APUs. The RPU provides memory-mapped
+ * interface (RPU if) which is used to communicate with APUs.
+ *
+ * The diagram below shows an overview of the CDX architecture:
+ *
+ * +--------------------------------------+
+ * | DPDK |
+ * | DPDK CDX drivers |
+ * | | |
+ * | DPDK CDX bus |
+ * | | |
+ * +-----------------------------|--------+
+ * |
+ * +-----------------------------|--------+
+ * | Application CPUs (APU) | |
+ * | | |
+ * | VFIO CDX driver |
+ * | Linux OS | |
+ * | Linux CDX bus |
+ * | | |
+ * +-----------------------------|--------+
+ * |
+ * |
+ * +------------------------| RPU if |----+
+ * | | |
+ * | V |
+ * | Realtime CPUs (RPU) |
+ * | |
+ * +--------------------------------------+
+ * |
+ * +---------------------|----------------+
+ * | FPGA | |
+ * | +-----------------------+ |
+ * | | | | |
+ * | +-------+ +-------+ +-------+ |
+ * | | dev 1 | | dev 2 | | dev 3 | |
+ * | +-------+ +-------+ +-------+ |
+ * +--------------------------------------+
+ *
+ * The RPU firmware extracts the device information from the loaded FPGA
+ * image and implements a mechanism that allows the APU drivers to
+ * enumerate such devices (device personality and resource details) via
+ * a dedicated communication channel.
+ *
+ * VFIO CDX driver provides the CDX device resources like MMIO and interrupts
+ * to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
+ * driver to discover and initialize the CDX devices for user-space
+ * applications.
+ */
+
+#include <string.h>
+#include <dirent.h>
+
+#include <rte_log.h>
+#include <rte_bus.h>
+#include <rte_bus_cdx.h>
+#include <rte_eal_paging.h>
+#include <rte_errno.h>
+#include <rte_devargs.h>
+#include <rte_malloc.h>
+#include <rte_memcpy.h>
+#include <rte_vfio.h>
+
+#include <eal_filesystem.h>
+
+#include "cdx.h"
+#include "cdx_logs.h"
+
+#define SYSFS_CDX_DEVICES "/sys/bus/cdx/devices"
+#define CDX_BUS_NAME cdx
+
+/**
+ * @file
+ * CDX probing using Linux sysfs.
+ */
+
+/* Add a device to CDX bus */
+static void
+rte_cdx_add_device(struct rte_cdx_device *cdx_dev)
+{
+ TAILQ_INSERT_TAIL(&rte_cdx_bus.device_list, cdx_dev, next);
+}
+
+static int
+cdx_get_kernel_driver_by_path(const char *filename, char *driver_name,
+ size_t len)
+{
+ int count;
+ char path[PATH_MAX];
+ char *name;
+
+ if (!filename || !driver_name)
+ return -1;
+
+ count = readlink(filename, path, PATH_MAX);
+ if (count >= PATH_MAX)
+ return -1;
+
+ /* For device does not have a driver */
+ if (count < 0)
+ return 1;
+
+ path[count] = '\0';
+
+ name = strrchr(path, '/');
+ if (name) {
+ strlcpy(driver_name, name + 1, len);
+ return 0;
+ }
+
+ return -1;
+}
+
+int rte_cdx_map_device(struct rte_cdx_device *dev)
+{
+ return cdx_vfio_map_resource(dev);
+}
+
+void rte_cdx_unmap_device(struct rte_cdx_device *dev)
+{
+ cdx_vfio_unmap_resource(dev);
+}
+
+static int
+find_max_end_va(const struct rte_memseg_list *msl, void *arg)
+{
+ size_t sz = msl->len;
+ void *end_va = RTE_PTR_ADD(msl->base_va, sz);
+ void **max_va = arg;
+
+ if (*max_va < end_va)
+ *max_va = end_va;
+ return 0;
+}
+
+void *
+cdx_find_max_end_va(void)
+{
+ void *va = NULL;
+
+ rte_memseg_list_walk(find_max_end_va, &va);
+ return va;
+}
+
+static struct rte_devargs *
+cdx_devargs_lookup(const char *dev_name)
+{
+ struct rte_devargs *devargs;
+
+ RTE_EAL_DEVARGS_FOREACH("cdx", devargs) {
+ if (strcmp(devargs->name, dev_name) == 0)
+ return devargs;
+ }
+ return NULL;
+}
+
+static bool
+cdx_ignore_device(const char *dev_name)
+{
+ struct rte_devargs *devargs = cdx_devargs_lookup(dev_name);
+
+ switch (rte_cdx_bus.bus.conf.scan_mode) {
+ case RTE_BUS_SCAN_ALLOWLIST:
+ if (devargs && devargs->policy == RTE_DEV_ALLOWED)
+ return false;
+ break;
+ case RTE_BUS_SCAN_UNDEFINED:
+ case RTE_BUS_SCAN_BLOCKLIST:
+ if (devargs == NULL || devargs->policy != RTE_DEV_BLOCKED)
+ return false;
+ break;
+ }
+ return true;
+}
+
+/*
+ * Scan one cdx sysfs entry, and fill the devices list from it.
+ * It checks if the CDX device is bound to vfio-cdx driver. In case
+ * the device is vfio bound, it reads the vendor and device id and
+ * stores it for device-driver matching.
+ */
+static int
+cdx_scan_one(const char *dirname, const char *dev_name)
+{
+ char filename[PATH_MAX];
+ struct rte_cdx_device *dev = NULL;
+ char driver[PATH_MAX];
+ unsigned long tmp;
+ char *name = NULL;
+ int ret;
+
+ dev = calloc(1, sizeof(*dev));
+ if (!dev)
+ return -ENOMEM;
+
+ name = calloc(1, RTE_DEV_NAME_MAX_LEN);
+ if (!name) {
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ dev->device.bus = &rte_cdx_bus.bus;
+ memcpy(name, dev_name, RTE_DEV_NAME_MAX_LEN);
+ dev->device.name = name;
+
+ /* parse driver */
+ snprintf(filename, sizeof(filename), "%s/driver", dirname);
+ ret = cdx_get_kernel_driver_by_path(filename, driver, sizeof(driver));
+ if (ret < 0) {
+ CDX_BUS_ERR("Fail to get kernel driver");
+ ret = -1;
+ goto err;
+ }
+
+ /*
+ * Check if device is bound to 'vfio-cdx' driver, so that user-space
+ * can gracefully access the device.
+ */
+ if (ret || strcmp(driver, "vfio-cdx")) {
+ ret = 0;
+ goto err;
+ }
+
+ /* get vendor id */
+ snprintf(filename, sizeof(filename), "%s/vendor", dirname);
+ if (eal_parse_sysfs_value(filename, &tmp) < 0) {
+ ret = -1;
+ goto err;
+ }
+ dev->id.vendor_id = (uint16_t)tmp;
+
+ /* get device id */
+ snprintf(filename, sizeof(filename), "%s/device", dirname);
+ if (eal_parse_sysfs_value(filename, &tmp) < 0) {
+ free(dev);
+ return -1;
+ }
+ dev->id.device_id = (uint16_t)tmp;
+
+ rte_cdx_add_device(dev);
+
+ return 0;
+
+err:
+ if (name)
+ free(name);
+ if (dev)
+ free(dev);
+ return ret;
+}
+
+/*
+ * Scan the content of the CDX bus, and the devices in the devices
+ * list.
+ */
+static int
+rte_cdx_scan(void)
+{
+ struct dirent *e;
+ DIR *dir;
+ char dirname[PATH_MAX];
+
+ dir = opendir(rte_cdx_get_sysfs_path());
+ if (dir == NULL) {
+ CDX_BUS_ERR("%s(): opendir failed: %s", __func__,
+ strerror(errno));
+ return -1;
+ }
+
+ while ((e = readdir(dir)) != NULL) {
+ if (e->d_name[0] == '.')
+ continue;
+
+ if (cdx_ignore_device(e->d_name))
+ continue;
+
+ snprintf(dirname, sizeof(dirname), "%s/%s",
+ rte_cdx_get_sysfs_path(), e->d_name);
+
+ if (cdx_scan_one(dirname, e->d_name) < 0)
+ goto error;
+ }
+ closedir(dir);
+ return 0;
+
+error:
+ closedir(dir);
+ return -1;
+}
+
+const char *
+rte_cdx_get_sysfs_path(void)
+{
+ return SYSFS_CDX_DEVICES;
+}
+
+/* map a particular resource from a file */
+void *
+cdx_map_resource(void *requested_addr, int fd, off_t offset, size_t size,
+ int additional_flags)
+{
+ void *mapaddr;
+
+ /* Map the cdx MMIO memory resource of device */
+ mapaddr = rte_mem_map(requested_addr, size,
+ RTE_PROT_READ | RTE_PROT_WRITE,
+ RTE_MAP_SHARED | additional_flags, fd, offset);
+ if (mapaddr == NULL) {
+ CDX_BUS_ERR("%s(): cannot map resource(%d, %p, 0x%zx, 0x%llx): %s (%p)",
+ __func__, fd, requested_addr, size,
+ (unsigned long long)offset,
+ rte_strerror(rte_errno), mapaddr);
+ }
+ CDX_BUS_DEBUG("CDX MMIO memory mapped at %p", mapaddr);
+
+ return mapaddr;
+}
+
+/* unmap a particular resource */
+void
+cdx_unmap_resource(void *requested_addr, size_t size)
+{
+ if (requested_addr == NULL)
+ return;
+
+ /* Unmap the CDX memory resource of device */
+ if (rte_mem_unmap(requested_addr, size)) {
+ CDX_BUS_ERR("%s(): cannot mem unmap(%p, %#zx): %s", __func__,
+ requested_addr, size, rte_strerror(rte_errno));
+ }
+ CDX_BUS_DEBUG("CDX memory unmapped at %p", requested_addr);
+}
+/*
+ * Match the CDX Driver and Device using device id and vendor id.
+ */
+static int
+rte_cdx_match(const struct rte_cdx_driver *cdx_drv,
+ const struct rte_cdx_device *cdx_dev)
+{
+ const struct rte_cdx_id *id_table;
+
+ for (id_table = cdx_drv->id_table; id_table->vendor_id != 0;
+ id_table++) {
+ /* check if device's identifiers match the driver's ones */
+ if (id_table->vendor_id != cdx_dev->id.vendor_id &&
+ id_table->vendor_id != RTE_CDX_ANY_ID)
+ continue;
+ if (id_table->device_id != cdx_dev->id.device_id &&
+ id_table->device_id != RTE_CDX_ANY_ID)
+ continue;
+
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * If vendor id and device id match, call the probe() function of the
+ * driver.
+ */
+static int
+rte_cdx_probe_one_driver(struct rte_cdx_driver *dr,
+ struct rte_cdx_device *dev)
+{
+ const char *dev_name = dev->device.name;
+ bool already_probed;
+ int ret;
+
+ if ((dr == NULL) || (dev == NULL))
+ return -EINVAL;
+
+ /* The device is not blocked; Check if driver supports it */
+ if (!rte_cdx_match(dr, dev))
+ /* Match of device and driver failed */
+ return 1;
+
+ already_probed = rte_dev_is_probed(&dev->device);
+ if (already_probed) {
+ CDX_BUS_INFO("Device %s is already probed", dev->device.name);
+ return -EEXIST;
+ }
+
+ CDX_BUS_DEBUG(" probe device %s using driver: %s", dev_name,
+ dr->driver.name);
+
+ ret = cdx_vfio_map_resource(dev);
+ if (ret != 0) {
+ CDX_BUS_ERR("CDX map device failed: %d", ret);
+ goto error_map_device;
+ }
+
+ /* call the driver probe() function */
+ ret = dr->probe(dr, dev);
+ if (ret) {
+ CDX_BUS_ERR("Probe CDX driver: %s device: %s failed: %d",
+ dr->driver.name, dev_name, ret);
+ goto error_probe;
+ } else {
+ dev->device.driver = &dr->driver;
+ }
+
+ return ret;
+
+error_probe:
+ cdx_vfio_unmap_resource(dev);
+error_map_device:
+ return ret;
+}
+
+/*
+ * If vendor/device ID match, call the probe() function of all
+ * registered driver for the given device. Return < 0 if initialization
+ * failed, return 1 if no driver is found for this device.
+ */
+static int
+cdx_probe_all_drivers(struct rte_cdx_device *dev)
+{
+ struct rte_cdx_driver *dr = NULL;
+ int rc = 0;
+
+ if (dev == NULL)
+ return -EINVAL;
+
+ FOREACH_DRIVER_ON_CDXBUS(dr) {
+ rc = rte_cdx_probe_one_driver(dr, dev);
+ if (rc < 0)
+ /* negative value is an error */
+ return rc;
+ if (rc > 0)
+ /* positive value means driver doesn't support it */
+ continue;
+ return 0;
+ }
+ return 1;
+}
+
+/*
+ * Scan the content of the CDX bus, and call the probe() function for
+ * all registered drivers that have a matching entry in its id_table
+ * for discovered devices.
+ */
+static int
+cdx_probe(void)
+{
+ struct rte_cdx_device *dev = NULL;
+ size_t probed = 0, failed = 0;
+ int ret = 0;
+
+ FOREACH_DEVICE_ON_CDXBUS(dev) {
+ probed++;
+
+ ret = cdx_probe_all_drivers(dev);
+ if (ret < 0) {
+ CDX_BUS_ERR("Requested device %s cannot be used",
+ dev->device.name);
+ rte_errno = errno;
+ failed++;
+ ret = 0;
+ }
+ }
+
+ return (probed && probed == failed) ? -1 : 0;
+}
+
+/* dump one device */
+static int
+cdx_dump_one_device(FILE *f, struct rte_cdx_device *dev)
+{
+ int i;
+
+ fprintf(f, "cdx device %s\n", dev->device.name);
+
+ for (i = 0; i != sizeof(dev->mem_resource) /
+ sizeof(dev->mem_resource[0]); i++) {
+ if (dev->mem_resource[i].len)
+ fprintf(f, " Resource[%d]: %p %16.16"PRIx64"\n",
+ i, dev->mem_resource[i].addr,
+ dev->mem_resource[i].len);
+ }
+ return 0;
+}
+
+/* dump devices on the bus */
+void
+rte_cdx_dump(FILE *f)
+{
+ struct rte_cdx_device *dev = NULL;
+
+ FOREACH_DEVICE_ON_CDXBUS(dev)
+ cdx_dump_one_device(f, dev);
+}
+
+static int
+cdx_parse(const char *name, void *addr)
+{
+ if (addr)
+ strcpy(addr, name);
+
+ return 0;
+}
+
+/* register a driver */
+void
+rte_cdx_register(struct rte_cdx_driver *driver)
+{
+ TAILQ_INSERT_TAIL(&rte_cdx_bus.driver_list, driver, next);
+ driver->bus = &rte_cdx_bus;
+}
+
+/* unregister a driver */
+void
+rte_cdx_unregister(struct rte_cdx_driver *driver)
+{
+ TAILQ_REMOVE(&rte_cdx_bus.driver_list, driver, next);
+ driver->bus = NULL;
+}
+
+static struct rte_device *
+cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
+ const void *data)
+{
+ const struct rte_cdx_device *cdx_start;
+ struct rte_cdx_device *cdx_dev;
+
+ if (start != NULL) {
+ cdx_start = RTE_DEV_TO_CDX_DEV_CONST(start);
+ cdx_dev = TAILQ_NEXT(cdx_start, next);
+ } else {
+ cdx_dev = TAILQ_FIRST(&rte_cdx_bus.device_list);
+ }
+ while (cdx_dev != NULL) {
+ if (cmp(&cdx_dev->device, data) == 0)
+ return &cdx_dev->device;
+ cdx_dev = TAILQ_NEXT(cdx_dev, next);
+ }
+ return NULL;
+}
+
+struct rte_cdx_bus rte_cdx_bus = {
+ .bus = {
+ .scan = rte_cdx_scan,
+ .probe = cdx_probe,
+ .find_device = cdx_find_device,
+ .parse = cdx_parse,
+ },
+ .device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
+ .driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
+};
+
+RTE_REGISTER_BUS(cdx, rte_cdx_bus.bus);
+RTE_LOG_REGISTER_DEFAULT(cdx_logtype_bus, NOTICE);
diff --git a/drivers/bus/cdx/cdx.h b/drivers/bus/cdx/cdx.h
new file mode 100644
index 0000000000..d428149bb9
--- /dev/null
+++ b/drivers/bus/cdx/cdx.h
@@ -0,0 +1,54 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef _CDX_H_
+#define _CDX_H_
+
+#include <stdbool.h>
+#include <stdio.h>
+
+#include "rte_bus_cdx.h"
+
+extern struct rte_cdx_bus rte_cdx_bus;
+
+/**
+ * Map a particular resource from a file.
+ *
+ * @param requested_addr
+ * The starting address for the new mapping range.
+ * @param fd
+ * The file descriptor.
+ * @param offset
+ * The offset for the mapping range.
+ * @param size
+ * The size for the mapping range.
+ * @param additional_flags
+ * The additional rte_mem_map() flags for the mapping range.
+ * @return
+ * - On success, the function returns a pointer to the mapped area.
+ * - On error, NULL is returned.
+ */
+void *cdx_map_resource(void *requested_addr, int fd, off_t offset,
+ size_t size, int additional_flags);
+
+/**
+ * Unmap a particular resource.
+ *
+ * @param requested_addr
+ * The address for the unmapping range.
+ * @param size
+ * The size for the unmapping range.
+ */
+void cdx_unmap_resource(void *requested_addr, size_t size);
+
+/*
+ * Helper function to map CDX resources right after hugepages in virtual memory
+ */
+void *cdx_find_max_end_va(void);
+
+/* map/unmap VFIO resource */
+int cdx_vfio_map_resource(struct rte_cdx_device *dev);
+int cdx_vfio_unmap_resource(struct rte_cdx_device *dev);
+
+#endif /* _CDX_H_ */
diff --git a/drivers/bus/cdx/cdx_logs.h b/drivers/bus/cdx/cdx_logs.h
new file mode 100644
index 0000000000..7a53633a1b
--- /dev/null
+++ b/drivers/bus/cdx/cdx_logs.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef _CDX_LOGS_H_
+#define _CDX_LOGS_H_
+
+extern int cdx_logtype_bus;
+
+#define CDX_BUS_LOG(level, fmt, args...) \
+ rte_log(RTE_LOG_ ## level, cdx_logtype_bus, "fslmc: " fmt "\n", \
+ ##args)
+
+/* Debug logs are with Function names */
+#define CDX_BUS_DEBUG(fmt, args...) \
+ rte_log(RTE_LOG_DEBUG, cdx_logtype_bus, "fslmc: %s(): " fmt "\n", \
+ __func__, ##args)
+
+#define CDX_BUS_INFO(fmt, args...) \
+ CDX_BUS_LOG(INFO, fmt, ## args)
+#define CDX_BUS_ERR(fmt, args...) \
+ CDX_BUS_LOG(ERR, fmt, ## args)
+#define CDX_BUS_WARN(fmt, args...) \
+ CDX_BUS_LOG(WARNING, fmt, ## args)
+
+/* DP Logs, toggled out at compile time if level lower than current level */
+#define CDX_BUS_DP_LOG(level, fmt, args...) \
+ RTE_LOG_DP(level, PMD, fmt, ## args)
+
+#define CDX_BUS_DP_DEBUG(fmt, args...) \
+ CDX_BUS_DP_LOG(DEBUG, fmt, ## args)
+#define CDX_BUS_DP_INFO(fmt, args...) \
+ CDX_BUS_DP_LOG(INFO, fmt, ## args)
+#define CDX_BUS_DP_WARN(fmt, args...) \
+ CDX_BUS_DP_LOG(WARNING, fmt, ## args)
+
+#endif /* _CDX_LOGS_H_ */
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
new file mode 100644
index 0000000000..f52adc4655
--- /dev/null
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -0,0 +1,428 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+/**
+ * @file
+ * CDX probing using Linux VFIO.
+ *
+ * This code tries to determine if the CDX device is bound to VFIO driver,
+ * and initialize it (map BARs, set up interrupts) if that's the case.
+ *
+ */
+
+#include <string.h>
+#include <fcntl.h>
+#include <sys/eventfd.h>
+#include <sys/socket.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <stdbool.h>
+
+#include <rte_log.h>
+#include <rte_bus_cdx.h>
+#include <rte_eal_paging.h>
+#include <rte_malloc.h>
+#include <rte_vfio.h>
+#include <rte_eal.h>
+#include <rte_bus.h>
+#include <rte_spinlock.h>
+#include <rte_tailq.h>
+
+#include "eal_filesystem.h"
+
+#include "cdx.h"
+#include "cdx_logs.h"
+
+/**
+ * A structure describing a CDX mapping.
+ */
+struct cdx_map {
+ void *addr;
+ char *path;
+ uint64_t offset;
+ uint64_t size;
+};
+
+/**
+ * A structure describing a mapped CDX resource.
+ * For multi-process we need to reproduce all CDX mappings in secondary
+ * processes, so save them in a tailq.
+ */
+struct mapped_cdx_resource {
+ TAILQ_ENTRY(mapped_cdx_resource) next;
+ char name[RTE_DEV_NAME_MAX_LEN]; /**< CDX device name */
+ char path[PATH_MAX];
+ int nb_maps;
+ struct cdx_map maps[CDX_MAX_RESOURCE];
+};
+
+/** mapped cdx device list */
+TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
+
+static struct rte_tailq_elem cdx_vfio_tailq = {
+ .name = "VFIO_CDX_RESOURCE_LIST",
+};
+EAL_REGISTER_TAILQ(cdx_vfio_tailq)
+
+static struct mapped_cdx_resource *
+cdx_vfio_find_and_unmap_resource(struct mapped_cdx_res_list *vfio_res_list,
+ struct rte_cdx_device *dev)
+{
+ struct mapped_cdx_resource *vfio_res = NULL;
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+ int i;
+
+ /* Get vfio_res */
+ TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+ if (strcmp(vfio_res->name, dev_name))
+ continue;
+ break;
+ }
+
+ if (vfio_res == NULL)
+ return vfio_res;
+
+ CDX_BUS_INFO("Releasing CDX mapped resource for %s", dev_name);
+
+ maps = vfio_res->maps;
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ /*
+ * We do not need to be aware of MSI-X table BAR mappings as
+ * when mapping. Just using current maps array is enough
+ */
+ if (maps[i].addr) {
+ CDX_BUS_DEBUG("Calling cdx_unmap_resource for %s at %p",
+ dev_name, maps[i].addr);
+ cdx_unmap_resource(maps[i].addr, maps[i].size);
+ }
+ }
+
+ return vfio_res;
+}
+
+static int
+cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
+{
+ char cdx_addr[PATH_MAX] = {0};
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list;
+
+ vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ vfio_res = cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for cdx device!",
+ cdx_addr);
+ return -1;
+ }
+
+ TAILQ_REMOVE(vfio_res_list, vfio_res, next);
+ rte_free(vfio_res);
+ return 0;
+}
+
+static int
+cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
+{
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list;
+
+ vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ vfio_res = cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for CDX device!",
+ dev->device.name);
+ return -1;
+ }
+
+ return 0;
+}
+
+int
+cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
+{
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return cdx_vfio_unmap_resource_primary(dev);
+ else
+ return cdx_vfio_unmap_resource_secondary(dev);
+}
+
+static int
+cdx_rte_vfio_setup_device(int vfio_dev_fd)
+{
+ /*
+ * Reset the device. If the device is not capable of resetting,
+ * then it updates errno as EINVAL.
+ */
+ if (ioctl(vfio_dev_fd, VFIO_DEVICE_RESET) && errno != EINVAL) {
+ CDX_BUS_ERR("Unable to reset device! Error: %d (%s)", errno,
+ strerror(errno));
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+cdx_vfio_mmap_resource(int vfio_dev_fd, struct mapped_cdx_resource *vfio_res,
+ int index, int additional_flags)
+{
+ struct memreg {
+ uint64_t offset;
+ size_t size;
+ } memreg[2] = { 0 };
+ void *vaddr;
+ struct cdx_map *map = &vfio_res->maps[index];
+
+ if (map->size == 0) {
+ CDX_BUS_DEBUG("map size is 0, skip region %d", index);
+ return 0;
+ }
+
+ memreg[0].offset = map->offset;
+ memreg[0].size = map->size;
+
+ /* reserve the address using an inaccessible mapping */
+ vaddr = mmap(map->addr, map->size, 0, MAP_PRIVATE |
+ MAP_ANONYMOUS | additional_flags, -1, 0);
+ if (vaddr != MAP_FAILED) {
+ void *map_addr = NULL;
+
+ if (memreg[0].size) {
+ /* actual map of first part */
+ map_addr = cdx_map_resource(vaddr, vfio_dev_fd,
+ memreg[0].offset,
+ memreg[0].size,
+ RTE_MAP_FORCE_ADDRESS);
+ }
+
+ if (map_addr == NULL) {
+ munmap(vaddr, map->size);
+ vaddr = MAP_FAILED;
+ CDX_BUS_ERR("Failed to map cdx MMIO region %d", index);
+ return -1;
+ }
+ } else {
+ CDX_BUS_ERR("Failed to create inaccessible mapping for MMIO region %d",
+ index);
+ return -1;
+ }
+
+ map->addr = vaddr;
+ return 0;
+}
+
+/*
+ * region info may contain capability headers, so we need to keep reallocating
+ * the memory until we match allocated memory size with argsz.
+ */
+static int
+cdx_vfio_get_region_info(int vfio_dev_fd, struct vfio_region_info **info,
+ int region)
+{
+ struct vfio_region_info *ri;
+ size_t argsz = sizeof(*ri);
+ int ret;
+
+ ri = malloc(sizeof(*ri));
+ if (ri == NULL) {
+ CDX_BUS_ERR("Cannot allocate memory for VFIO region info");
+ return -1;
+ }
+again:
+ memset(ri, 0, argsz);
+ ri->argsz = argsz;
+ ri->index = region;
+
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_REGION_INFO, ri);
+ if (ret < 0) {
+ free(ri);
+ return ret;
+ }
+ if (ri->argsz != argsz) {
+ struct vfio_region_info *tmp;
+
+ argsz = ri->argsz;
+ tmp = realloc(ri, argsz);
+
+ if (tmp == NULL) {
+ /* realloc failed but the ri is still there */
+ free(ri);
+ CDX_BUS_ERR("Cannot reallocate memory for VFIO region info");
+ return -1;
+ }
+ ri = tmp;
+ goto again;
+ }
+ *info = ri;
+
+ return 0;
+}
+
+static int
+cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ char cdx_addr[PATH_MAX] = {0};
+ static void *cdx_map_addr;
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+ int vfio_dev_fd, i, ret;
+
+ ret = rte_vfio_setup_device(rte_cdx_get_sysfs_path(), dev_name,
+ &vfio_dev_fd, &device_info);
+ if (ret)
+ return ret;
+
+ /* allocate vfio_res and get region info */
+ vfio_res = rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0);
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("Cannot store VFIO mmap details");
+ goto err_vfio_dev_fd;
+ }
+ memcpy(vfio_res->name, dev_name, RTE_DEV_NAME_MAX_LEN);
+
+ /* get number of registers */
+ vfio_res->nb_maps = device_info.num_regions;
+
+ /* map memory regions */
+ maps = vfio_res->maps;
+
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ struct vfio_region_info *reg = NULL;
+ void *vaddr;
+
+ ret = cdx_vfio_get_region_info(vfio_dev_fd, ®, i);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s cannot get device region info error %i (%s)",
+ dev_name, errno, strerror(errno));
+ goto err_vfio_res;
+ }
+
+ /* skip non-mmappable regions */
+ if ((reg->flags & VFIO_REGION_INFO_FLAG_MMAP) == 0) {
+ free(reg);
+ continue;
+ }
+
+ /* try mapping somewhere close to the end of hugepages */
+ if (cdx_map_addr == NULL)
+ cdx_map_addr = cdx_find_max_end_va();
+
+ vaddr = cdx_map_addr;
+ cdx_map_addr = RTE_PTR_ADD(vaddr, (size_t)reg->size);
+
+ cdx_map_addr = RTE_PTR_ALIGN(cdx_map_addr,
+ sysconf(_SC_PAGE_SIZE));
+
+ maps[i].addr = vaddr;
+ maps[i].offset = reg->offset;
+ maps[i].size = reg->size;
+ maps[i].path = NULL; /* vfio doesn't have per-resource paths */
+
+ ret = cdx_vfio_mmap_resource(vfio_dev_fd, vfio_res, i, 0);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s mapping region %i failed: %s",
+ cdx_addr, i, strerror(errno));
+ free(reg);
+ goto err_vfio_res;
+ }
+
+ dev->mem_resource[i].addr = maps[i].addr;
+ dev->mem_resource[i].len = maps[i].size;
+
+ free(reg);
+ }
+
+ if (cdx_rte_vfio_setup_device(vfio_dev_fd) < 0) {
+ CDX_BUS_ERR("%s setup device failed", dev_name);
+ goto err_vfio_res;
+ }
+
+ TAILQ_INSERT_TAIL(vfio_res_list, vfio_res, next);
+
+ return 0;
+err_vfio_res:
+ cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+ rte_free(vfio_res);
+err_vfio_dev_fd:
+ rte_vfio_release_device(rte_cdx_get_sysfs_path(),
+ dev_name, vfio_dev_fd);
+ return -1;
+}
+
+static int
+cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ char cdx_addr[PATH_MAX] = {0};
+ int vfio_dev_fd;
+ int i, ret;
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+
+ /* if we're in a secondary process, just find our tailq entry */
+ TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+ if (strcmp(vfio_res->name, dev_name))
+ continue;
+ break;
+ }
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for cdx device!",
+ dev_name);
+ return -1;
+ }
+
+ ret = rte_vfio_setup_device(rte_cdx_get_sysfs_path(), dev_name,
+ &vfio_dev_fd, &device_info);
+ if (ret)
+ return ret;
+
+ /* map MMIO regions */
+ maps = vfio_res->maps;
+
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ ret = cdx_vfio_mmap_resource(vfio_dev_fd, vfio_res, i, MAP_FIXED);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s mapping MMIO region %i failed: %s",
+ dev_name, i, strerror(errno));
+ goto err_vfio_dev_fd;
+ }
+
+ dev->mem_resource[i].addr = maps[i].addr;
+ dev->mem_resource[i].len = maps[i].size;
+ }
+
+ return 0;
+err_vfio_dev_fd:
+ rte_vfio_release_device(rte_cdx_get_sysfs_path(),
+ cdx_addr, vfio_dev_fd);
+ return -1;
+}
+
+/*
+ * map the CDX resources of a CDX device in virtual memory (VFIO version).
+ * primary and secondary processes follow almost exactly the same path
+ */
+int
+cdx_vfio_map_resource(struct rte_cdx_device *dev)
+{
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return cdx_vfio_map_resource_primary(dev);
+ else
+ return cdx_vfio_map_resource_secondary(dev);
+}
diff --git a/drivers/bus/cdx/meson.build b/drivers/bus/cdx/meson.build
new file mode 100644
index 0000000000..2fb0e1cefd
--- /dev/null
+++ b/drivers/bus/cdx/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+
+if not is_linux
+ build = false
+ reason = 'only supported on Linux'
+endif
+
+headers = files('rte_bus_cdx.h')
+sources = files(
+ 'cdx.c',
+ 'cdx_vfio.c',
+)
diff --git a/drivers/bus/cdx/rte_bus_cdx.h b/drivers/bus/cdx/rte_bus_cdx.h
new file mode 100644
index 0000000000..ef04dfc88b
--- /dev/null
+++ b/drivers/bus/cdx/rte_bus_cdx.h
@@ -0,0 +1,219 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef _RTE_BUS_CDX_H_
+#define _RTE_BUS_CDX_H_
+
+/**
+ * @file
+ *
+ * CDX device & driver interface
+ *
+ * @warning
+ * @b EXPERIMENTAL:
+ * All functions in this file may be changed or removed without prior notice.
+ *
+ * CDX is a Hardware Architecture designed for AMD FPGA and HNIC devices.
+ * These devices are provided as CDX devices for the user. This driver
+ * provides user interface for devices on the CDX bus.
+ *
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <limits.h>
+#include <errno.h>
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <bus_driver.h>
+#include <dev_driver.h>
+#include <rte_debug.h>
+#include <rte_interrupts.h>
+#include <rte_dev.h>
+#include <rte_bus.h>
+
+/* Forward declarations */
+struct rte_cdx_device;
+struct rte_cdx_driver;
+
+#define CDX_MAX_RESOURCE 4
+
+/** List of CDX devices */
+RTE_TAILQ_HEAD(rte_cdx_device_list, rte_cdx_device);
+/** List of CDX drivers */
+RTE_TAILQ_HEAD(rte_cdx_driver_list, rte_cdx_driver);
+
+/* CDX Bus iterators */
+#define FOREACH_DEVICE_ON_CDXBUS(p) \
+ RTE_TAILQ_FOREACH(p, &rte_cdx_bus.device_list, next)
+
+#define FOREACH_DRIVER_ON_CDXBUS(p) \
+ RTE_TAILQ_FOREACH(p, &rte_cdx_bus.driver_list, next)
+
+/** Any CDX device identifier (vendor, device) */
+#define RTE_CDX_ANY_ID (0xffff)
+
+#define RTE_PMD_REGISTER_CDX_TABLE(name, table) \
+static const char DRV_EXP_TAG(name, cdx_tbl_export)[] __rte_used = \
+RTE_STR(table)
+
+/**
+ * A structure describing an ID for a CDX driver. Each driver provides a
+ * table of these IDs for each device that it supports.
+ */
+struct rte_cdx_id {
+ uint16_t vendor_id; /**< Vendor ID. */
+ uint16_t device_id; /**< Device ID. */
+};
+
+/**
+ * A structure describing a CDX device.
+ */
+struct rte_cdx_device {
+ RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
+ struct rte_device device; /**< Inherit core device */
+ struct rte_cdx_id id; /**< CDX ID. */
+ struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
+ /**< CDX Memory Resource */
+};
+
+/**
+ * @internal
+ * Helper macro for drivers that need to convert to struct rte_cdx_device.
+ */
+#define RTE_DEV_TO_CDX_DEV(ptr) \
+ container_of(ptr, struct rte_cdx_device, device)
+
+#define RTE_DEV_TO_CDX_DEV_CONST(ptr) \
+ container_of(ptr, const struct rte_cdx_device, device)
+
+#define RTE_ETH_DEV_TO_CDX_DEV(eth_dev) RTE_DEV_TO_CDX_DEV((eth_dev)->device)
+
+#ifdef __cplusplus
+/** C++ macro used to help building up tables of device IDs */
+#define RTE_CDX_DEVICE(vend, dev) \
+ (vend), \
+ (dev)
+#else
+/** Macro used to help building up tables of device IDs */
+#define RTE_CDX_DEVICE(vend, dev) \
+ .vendor_id = (vend), \
+ .device_id = (dev)
+#endif
+
+/**
+ * Initialisation function for the driver called during CDX probing.
+ */
+typedef int (rte_cdx_probe_t)(struct rte_cdx_driver *, struct rte_cdx_device *);
+
+/**
+ * Uninitialisation function for the driver called during hotplugging.
+ */
+typedef int (rte_cdx_remove_t)(struct rte_cdx_device *);
+
+/**
+ * A structure describing a CDX driver.
+ */
+struct rte_cdx_driver {
+ RTE_TAILQ_ENTRY(rte_cdx_driver) next; /**< Next in list. */
+ struct rte_driver driver; /**< Inherit core driver. */
+ struct rte_cdx_bus *bus; /**< CDX bus reference. */
+ rte_cdx_probe_t *probe; /**< Device probe function. */
+ rte_cdx_remove_t *remove; /**< Device remove function. */
+ const struct rte_cdx_id *id_table; /**< ID table, NULL terminated. */
+ uint32_t drv_flags; /**< Flags RTE_CDX_DRV_*. */
+};
+
+/**
+ * Structure describing the CDX bus
+ */
+struct rte_cdx_bus {
+ struct rte_bus bus; /**< Inherit the generic class */
+ struct rte_cdx_device_list device_list; /**< List of CDX devices */
+ struct rte_cdx_driver_list driver_list; /**< List of CDX drivers */
+};
+
+/**
+ * Get Pathname of CDX devices directory.
+ *
+ * @return
+ * sysfs path for CDX devices.
+ */
+__rte_experimental
+const char *rte_cdx_get_sysfs_path(void);
+
+/**
+ * Map the CDX device resources in user space virtual memory address
+ *
+ * @param dev
+ * A pointer to a rte_cdx_device structure describing the device
+ * to use
+ *
+ * @return
+ * 0 on success, negative on error and positive if no driver
+ * is found for the device.
+ */
+__rte_experimental
+int rte_cdx_map_device(struct rte_cdx_device *dev);
+
+/**
+ * Unmap this device
+ *
+ * @param dev
+ * A pointer to a rte_cdx_device structure describing the device
+ * to use
+ */
+__rte_experimental
+void rte_cdx_unmap_device(struct rte_cdx_device *dev);
+
+/**
+ * Dump the content of the CDX bus.
+ *
+ * @param f
+ * A pointer to a file for output
+ */
+__rte_experimental
+void rte_cdx_dump(FILE *f);
+
+/**
+ * Register a CDX driver.
+ *
+ * @param driver
+ * A pointer to a rte_cdx_driver structure describing the driver
+ * to be registered.
+ */
+__rte_experimental
+void rte_cdx_register(struct rte_cdx_driver *driver);
+
+/**
+ * Helper for CDX device registration from driver (eth, crypto, raw) instance
+ */
+#define RTE_PMD_REGISTER_CDX(nm, cdx_drv) \
+ RTE_INIT(cdxinitfn_ ##nm) \
+ {\
+ (cdx_drv).driver.name = RTE_STR(nm);\
+ rte_cdx_register(&cdx_drv); \
+ } \
+ RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+/**
+ * Unregister a CDX driver.
+ *
+ * @param driver
+ * A pointer to a rte_cdx_driver structure describing the driver
+ * to be unregistered.
+ */
+__rte_experimental
+void rte_cdx_unregister(struct rte_cdx_driver *driver);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_BUS_CDX_H_ */
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
new file mode 100644
index 0000000000..23dff36125
--- /dev/null
+++ b/drivers/bus/cdx/version.map
@@ -0,0 +1,12 @@
+EXPERIMENTAL {
+ global:
+
+ rte_cdx_dump;
+ rte_cdx_get_sysfs_path;
+ rte_cdx_map_device;
+ rte_cdx_register;
+ rte_cdx_unmap_device;
+ rte_cdx_unregister;
+
+ local: *;
+};
--
2.25.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [RFC PATCH 2/6] bus/cdx: add dma map and unmap support
2023-01-24 14:07 [RFC PATCH 0/6] add support for CDX bus Nipun Gupta
2023-01-24 14:07 ` [RFC PATCH 1/6] bus/cdx: introduce cdx bus Nipun Gupta
@ 2023-01-24 14:07 ` Nipun Gupta
2023-01-24 14:07 ` [RFC PATCH 3/6] bus/cdx: add support for MSI Nipun Gupta
` (11 subsequent siblings)
13 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-01-24 14:07 UTC (permalink / raw)
To: dev, thomas, david.marchand; +Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal
CDX bus can use VFIO interface for mapping and unmapping
of DMA addresses in the IOMMU. This change adds the callback
support for map and unmap APIs as well as fetching the IOMMU
class.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
drivers/bus/cdx/cdx.c | 40 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index 4c85c5dc01..cb04639b0d 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -549,12 +549,52 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
return NULL;
}
+static int
+cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+ struct rte_cdx_device *cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+
+ if (!cdx_dev) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ return rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ (uintptr_t)addr, iova, len);
+}
+
+static int
+cdx_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+ struct rte_cdx_device *cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+
+ if (!cdx_dev) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ return rte_vfio_container_dma_unmap(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ (uintptr_t)addr, iova, len);
+}
+
+static enum rte_iova_mode
+rte_cdx_get_iommu_class(void)
+{
+ if (TAILQ_EMPTY(&rte_cdx_bus.device_list))
+ return RTE_IOVA_DC;
+
+ return RTE_IOVA_VA;
+}
+
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
.scan = rte_cdx_scan,
.probe = cdx_probe,
.find_device = cdx_find_device,
.parse = cdx_parse,
+ .dma_map = cdx_dma_map,
+ .dma_unmap = cdx_dma_unmap,
+ .get_iommu_class = rte_cdx_get_iommu_class,
},
.device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
.driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
--
2.25.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [RFC PATCH 3/6] bus/cdx: add support for MSI
2023-01-24 14:07 [RFC PATCH 0/6] add support for CDX bus Nipun Gupta
2023-01-24 14:07 ` [RFC PATCH 1/6] bus/cdx: introduce cdx bus Nipun Gupta
2023-01-24 14:07 ` [RFC PATCH 2/6] bus/cdx: add dma map and unmap support Nipun Gupta
@ 2023-01-24 14:07 ` Nipun Gupta
2023-01-24 14:07 ` [RFC PATCH 4/6] bus/cdx: support plug unplug and dev iterator Nipun Gupta
` (10 subsequent siblings)
13 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-01-24 14:07 UTC (permalink / raw)
To: dev, thomas, david.marchand; +Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal
MSI's are exposed to the devices using VFIO (vfio-cdx). This
patch uses the same to add support for MSI for the devices on
the cdx bus.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
drivers/bus/cdx/cdx.c | 11 ++
drivers/bus/cdx/cdx_vfio.c | 182 ++++++++++++++++++++++++-
drivers/bus/cdx/rte_bus_cdx.h | 25 ++++
drivers/bus/cdx/version.map | 9 ++
lib/eal/common/eal_common_interrupts.c | 21 +++
lib/eal/common/eal_interrupts.h | 1 +
lib/eal/include/rte_interrupts.h | 32 +++++
lib/eal/version.map | 2 +
8 files changed, 281 insertions(+), 2 deletions(-)
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index cb04639b0d..d0adfb74ef 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -224,6 +224,15 @@ cdx_scan_one(const char *dirname, const char *dev_name)
goto err;
}
+ /* Allocate interrupt instance for cdx device */
+ dev->intr_handle =
+ rte_intr_instance_alloc(RTE_INTR_INSTANCE_F_PRIVATE);
+ if (dev->intr_handle == NULL) {
+ CDX_BUS_ERR("Failed to create interrupt instance for %s\n",
+ dev->device.name);
+ return -ENOMEM;
+ }
+
/*
* Check if device is bound to 'vfio-cdx' driver, so that user-space
* can gracefully access the device.
@@ -415,6 +424,8 @@ rte_cdx_probe_one_driver(struct rte_cdx_driver *dr,
return ret;
error_probe:
+ rte_intr_instance_free(dev->intr_handle);
+ dev->intr_handle = NULL;
cdx_vfio_unmap_resource(dev);
error_map_device:
return ret;
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index f52adc4655..fd90da6727 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -60,6 +60,10 @@ struct mapped_cdx_resource {
/** mapped cdx device list */
TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
+/* irq set buffer length for MSI interrupts */
+#define MSI_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \
+ sizeof(int) * (RTE_MAX_RXTX_INTR_VEC_ID + 1))
+
static struct rte_tailq_elem cdx_vfio_tailq = {
.name = "VFIO_CDX_RESOURCE_LIST",
};
@@ -108,6 +112,27 @@ cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
char cdx_addr[PATH_MAX] = {0};
struct mapped_cdx_resource *vfio_res = NULL;
struct mapped_cdx_res_list *vfio_res_list;
+ int ret, vfio_dev_fd;
+
+ if (rte_intr_fd_get(dev->intr_handle) < 0)
+ return -1;
+
+ if (close(rte_intr_fd_get(dev->intr_handle)) < 0) {
+ CDX_BUS_ERR("Error when closing eventfd file descriptor for %s",
+ dev->device.name);
+ return -1;
+ }
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ ret = rte_vfio_release_device(rte_cdx_get_sysfs_path(), dev->device.name,
+ vfio_dev_fd);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot release VFIO device");
+ return ret;
+ }
vfio_res_list =
RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
@@ -130,6 +155,18 @@ cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
{
struct mapped_cdx_resource *vfio_res = NULL;
struct mapped_cdx_res_list *vfio_res_list;
+ int ret, vfio_dev_fd;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ ret = rte_vfio_release_device(rte_cdx_get_sysfs_path(), dev->device.name,
+ vfio_dev_fd);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot release VFIO device");
+ return ret;
+ }
vfio_res_list =
RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
@@ -154,9 +191,80 @@ cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
return cdx_vfio_unmap_resource_secondary(dev);
}
+/* set up interrupt support (but not enable interrupts) */
static int
-cdx_rte_vfio_setup_device(int vfio_dev_fd)
+cdx_vfio_setup_interrupts(struct rte_cdx_device *dev, int vfio_dev_fd,
+ int num_irqs)
{
+ int i, ret;
+
+ if (num_irqs == 0)
+ return 0;
+
+ /* start from MSI interrupt type */
+ for (i = 0; i < num_irqs; i++) {
+ struct vfio_irq_info irq = { .argsz = sizeof(irq) };
+ int fd = -1;
+
+ irq.index = i;
+
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_IRQ_INFO, &irq);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot get VFIO IRQ info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /* if this vector cannot be used with eventfd, fail if we explicitly
+ * specified interrupt type, otherwise continue
+ */
+ if ((irq.flags & VFIO_IRQ_INFO_EVENTFD) == 0)
+ continue;
+
+ if (rte_intr_irq_count_set(dev->intr_handle, irq.count))
+ return -1;
+
+ /* Reallocate the efds and elist fields of intr_handle based
+ * on CDX device MSI size.
+ */
+ if ((uint32_t)rte_intr_nb_intr_get(dev->intr_handle) < irq.count &&
+ rte_intr_event_list_update(dev->intr_handle, irq.count))
+ return -1;
+
+ /* set up an eventfd for interrupts */
+ fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
+ if (fd < 0) {
+ CDX_BUS_ERR("Cannot set up eventfd, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ if (rte_intr_fd_set(dev->intr_handle, fd))
+ return -1;
+
+ /* DPDK CDX bus currently supports only MSI-X */
+ if (rte_intr_type_set(dev->intr_handle, RTE_INTR_HANDLE_VFIO_MSIX))
+ return -1;
+
+ if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
+ return -1;
+
+ return 0;
+ }
+
+ /* if we're here, we haven't found a suitable interrupt vector */
+ return -1;
+}
+
+static int
+cdx_rte_vfio_setup_device(struct rte_cdx_device *dev, int vfio_dev_fd,
+ int num_irqs)
+{
+ if (cdx_vfio_setup_interrupts(dev, vfio_dev_fd, num_irqs) != 0) {
+ CDX_BUS_ERR("Error setting up interrupts!");
+ return -1;
+ }
+
/*
* Reset the device. If the device is not capable of resetting,
* then it updates errno as EINVAL.
@@ -279,6 +387,9 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
struct cdx_map *maps;
int vfio_dev_fd, i, ret;
+ if (rte_intr_fd_set(dev->intr_handle, -1))
+ return -1;
+
ret = rte_vfio_setup_device(rte_cdx_get_sysfs_path(), dev_name,
&vfio_dev_fd, &device_info);
if (ret)
@@ -344,7 +455,7 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
free(reg);
}
- if (cdx_rte_vfio_setup_device(vfio_dev_fd) < 0) {
+ if (cdx_rte_vfio_setup_device(dev, vfio_dev_fd, device_info.num_irqs) < 0) {
CDX_BUS_ERR("%s setup device failed", dev_name);
goto err_vfio_res;
}
@@ -374,6 +485,9 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
const char *dev_name = dev->device.name;
struct cdx_map *maps;
+ if (rte_intr_fd_set(dev->intr_handle, -1))
+ return -1;
+
/* if we're in a secondary process, just find our tailq entry */
TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
if (strcmp(vfio_res->name, dev_name))
@@ -407,6 +521,10 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
dev->mem_resource[i].len = maps[i].size;
}
+ /* we need save vfio_dev_fd, so it can be used during release */
+ if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
+ goto err_vfio_dev_fd;
+
return 0;
err_vfio_dev_fd:
rte_vfio_release_device(rte_cdx_get_sysfs_path(),
@@ -426,3 +544,63 @@ cdx_vfio_map_resource(struct rte_cdx_device *dev)
else
return cdx_vfio_map_resource_secondary(dev);
}
+
+int
+rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle)
+{
+ char irq_set_buf[MSI_IRQ_SET_BUF_LEN];
+ struct vfio_irq_set *irq_set;
+ int *fd_ptr, vfio_dev_fd, i;
+ int ret;
+
+ irq_set = (struct vfio_irq_set *) irq_set_buf;
+ irq_set->count = rte_intr_irq_count_get(intr_handle);
+ irq_set->argsz = sizeof(struct vfio_irq_set) +
+ (sizeof(int) * irq_set->count);
+
+ irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER;
+ irq_set->index = 0;
+ irq_set->start = 0;
+ fd_ptr = (int *) &irq_set->data;
+
+ for (i = 0; i < rte_intr_nb_efd_get(intr_handle); i++)
+ fd_ptr[i] = rte_intr_efds_index_get(intr_handle, i);
+
+ vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
+
+ if (ret) {
+ CDX_BUS_ERR("Error enabling MSI interrupts for fd %d",
+ rte_intr_fd_get(intr_handle));
+ return -1;
+ }
+
+ return 0;
+}
+
+/* disable MSI interrupts */
+int
+rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle)
+{
+ struct vfio_irq_set *irq_set;
+ char irq_set_buf[MSI_IRQ_SET_BUF_LEN];
+ int len, ret, vfio_dev_fd;
+
+ len = sizeof(struct vfio_irq_set);
+
+ irq_set = (struct vfio_irq_set *) irq_set_buf;
+ irq_set->argsz = len;
+ irq_set->count = 0;
+ irq_set->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_TRIGGER;
+ irq_set->index = 0;
+ irq_set->start = 0;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
+
+ if (ret)
+ CDX_BUS_ERR("Error disabling MSI interrupts for fd %d",
+ rte_intr_fd_get(intr_handle));
+
+ return ret;
+}
diff --git a/drivers/bus/cdx/rte_bus_cdx.h b/drivers/bus/cdx/rte_bus_cdx.h
index ef04dfc88b..43d75bc3b4 100644
--- a/drivers/bus/cdx/rte_bus_cdx.h
+++ b/drivers/bus/cdx/rte_bus_cdx.h
@@ -81,6 +81,7 @@ struct rte_cdx_device {
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
/**< CDX Memory Resource */
+ struct rte_intr_handle *intr_handle; /**< Interrupt handle */
};
/**
@@ -181,6 +182,30 @@ void rte_cdx_unmap_device(struct rte_cdx_device *dev);
__rte_experimental
void rte_cdx_dump(FILE *f);
+/**
+ * Enables VFIO Interrupts for CDX bus devices.
+ *
+ * @param intr_handle
+ * Pointer to the interrupt handle.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle);
+
+/**
+ * Disable VFIO Interrupts for CDX bus devices.
+ *
+ * @param intr_handle
+ * Pointer to the interrupt handle.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle);
+
/**
* Register a CDX driver.
*
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
index 23dff36125..8fc51b090b 100644
--- a/drivers/bus/cdx/version.map
+++ b/drivers/bus/cdx/version.map
@@ -10,3 +10,12 @@ EXPERIMENTAL {
local: *;
};
+
+INTERNAL {
+ global:
+
+ rte_cdx_vfio_intr_disable;
+ rte_cdx_vfio_intr_enable;
+
+ local: *;
+};
diff --git a/lib/eal/common/eal_common_interrupts.c b/lib/eal/common/eal_common_interrupts.c
index 97b64fed58..a0167d9ad4 100644
--- a/lib/eal/common/eal_common_interrupts.c
+++ b/lib/eal/common/eal_common_interrupts.c
@@ -398,6 +398,27 @@ int rte_intr_elist_index_set(struct rte_intr_handle *intr_handle,
return -rte_errno;
}
+int rte_intr_irq_count_set(struct rte_intr_handle *intr_handle,
+ int irq_count)
+{
+ CHECK_VALID_INTR_HANDLE(intr_handle);
+
+ intr_handle->irq_count = irq_count;
+
+ return 0;
+fail:
+ return -rte_errno;
+}
+
+int rte_intr_irq_count_get(const struct rte_intr_handle *intr_handle)
+{
+ CHECK_VALID_INTR_HANDLE(intr_handle);
+
+ return intr_handle->irq_count;
+fail:
+ return -rte_errno;
+}
+
int rte_intr_vec_list_alloc(struct rte_intr_handle *intr_handle,
const char *name, int size)
{
diff --git a/lib/eal/common/eal_interrupts.h b/lib/eal/common/eal_interrupts.h
index 482781b862..237f471a76 100644
--- a/lib/eal/common/eal_interrupts.h
+++ b/lib/eal/common/eal_interrupts.h
@@ -16,6 +16,7 @@ struct rte_intr_handle {
};
uint32_t alloc_flags; /**< flags passed at allocation */
enum rte_intr_handle_type type; /**< handle type */
+ uint32_t irq_count; /**< IRQ count provided via VFIO */
uint32_t max_intr; /**< max interrupt requested */
uint32_t nb_efd; /**< number of available efd(event fd) */
uint8_t efd_counter_size; /**< size of efd counter, used for vdev */
diff --git a/lib/eal/include/rte_interrupts.h b/lib/eal/include/rte_interrupts.h
index 487e3c8875..bc477a483f 100644
--- a/lib/eal/include/rte_interrupts.h
+++ b/lib/eal/include/rte_interrupts.h
@@ -506,6 +506,38 @@ __rte_internal
int
rte_intr_max_intr_get(const struct rte_intr_handle *intr_handle);
+/**
+ * @internal
+ * Set the irq count field of interrupt handle with user
+ * provided irq count value.
+ *
+ * @param intr_handle
+ * pointer to the interrupt handle.
+ * @param irq_count
+ * IRQ count
+ *
+ * @return
+ * - On success, zero.
+ * - On failure, a negative value and rte_errno is set.
+ */
+__rte_internal
+int
+rte_intr_irq_count_set(struct rte_intr_handle *intr_handle, int irq_count);
+
+/**
+ * @internal
+ * Returns the irq count field of the given interrupt handle instance.
+ *
+ * @param intr_handle
+ * pointer to the interrupt handle.
+ *
+ * @return
+ * - On success, ir count.
+ * - On failure, a negative value and rte_errno is set.
+ */
+__rte_internal
+int rte_intr_irq_count_get(const struct rte_intr_handle *intr_handle);
+
/**
* @internal
* Set the number of event fd field of interrupt handle
diff --git a/lib/eal/version.map b/lib/eal/version.map
index 7ad12a7dc9..8cda43247d 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -468,6 +468,8 @@ INTERNAL {
rte_intr_instance_dup;
rte_intr_instance_windows_handle_get;
rte_intr_instance_windows_handle_set;
+ rte_intr_irq_count_get;
+ rte_intr_irq_count_set;
rte_intr_max_intr_get;
rte_intr_max_intr_set;
rte_intr_nb_efd_get;
--
2.25.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [RFC PATCH 4/6] bus/cdx: support plug unplug and dev iterator
2023-01-24 14:07 [RFC PATCH 0/6] add support for CDX bus Nipun Gupta
` (2 preceding siblings ...)
2023-01-24 14:07 ` [RFC PATCH 3/6] bus/cdx: add support for MSI Nipun Gupta
@ 2023-01-24 14:07 ` Nipun Gupta
2023-01-24 14:07 ` [RFC PATCH 5/6] bus: enable cdx bus Nipun Gupta
` (9 subsequent siblings)
13 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-01-24 14:07 UTC (permalink / raw)
To: dev, thomas, david.marchand; +Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal
Add support for plugging and unplugging CDX devices on
the CDX bus. Also, CDX dev iterator support has been added
for the CDX bus.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
drivers/bus/cdx/cdx.c | 142 +++++++++++++++++++++++++++++++---
drivers/bus/cdx/rte_bus_cdx.h | 1 +
2 files changed, 133 insertions(+), 10 deletions(-)
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index d0adfb74ef..151d35ffd3 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -69,6 +69,7 @@
#include <rte_eal_paging.h>
#include <rte_errno.h>
#include <rte_devargs.h>
+#include <rte_kvargs.h>
#include <rte_malloc.h>
#include <rte_memcpy.h>
#include <rte_vfio.h>
@@ -81,6 +82,15 @@
#define SYSFS_CDX_DEVICES "/sys/bus/cdx/devices"
#define CDX_BUS_NAME cdx
+enum cdx_params {
+ RTE_CDX_PARAM_NAME,
+};
+
+static const char * const cdx_params_keys[] = {
+ [RTE_CDX_PARAM_NAME] = "name",
+ NULL,
+};
+
/**
* @file
* CDX probing using Linux sysfs.
@@ -88,7 +98,7 @@
/* Add a device to CDX bus */
static void
-rte_cdx_add_device(struct rte_cdx_device *cdx_dev)
+cdx_add_device(struct rte_cdx_device *cdx_dev)
{
TAILQ_INSERT_TAIL(&rte_cdx_bus.device_list, cdx_dev, next);
}
@@ -258,7 +268,7 @@ cdx_scan_one(const char *dirname, const char *dev_name)
}
dev->id.device_id = (uint16_t)tmp;
- rte_cdx_add_device(dev);
+ cdx_add_device(dev);
return 0;
@@ -275,7 +285,7 @@ cdx_scan_one(const char *dirname, const char *dev_name)
* list.
*/
static int
-rte_cdx_scan(void)
+cdx_scan(void)
{
struct dirent *e;
DIR *dir;
@@ -355,7 +365,7 @@ cdx_unmap_resource(void *requested_addr, size_t size)
* Match the CDX Driver and Device using device id and vendor id.
*/
static int
-rte_cdx_match(const struct rte_cdx_driver *cdx_drv,
+cdx_match(const struct rte_cdx_driver *cdx_drv,
const struct rte_cdx_device *cdx_dev)
{
const struct rte_cdx_id *id_table;
@@ -381,7 +391,7 @@ rte_cdx_match(const struct rte_cdx_driver *cdx_drv,
* driver.
*/
static int
-rte_cdx_probe_one_driver(struct rte_cdx_driver *dr,
+cdx_probe_one_driver(struct rte_cdx_driver *dr,
struct rte_cdx_device *dev)
{
const char *dev_name = dev->device.name;
@@ -392,7 +402,7 @@ rte_cdx_probe_one_driver(struct rte_cdx_driver *dr,
return -EINVAL;
/* The device is not blocked; Check if driver supports it */
- if (!rte_cdx_match(dr, dev))
+ if (!cdx_match(dr, dev))
/* Match of device and driver failed */
return 1;
@@ -420,6 +430,7 @@ rte_cdx_probe_one_driver(struct rte_cdx_driver *dr,
} else {
dev->device.driver = &dr->driver;
}
+ dev->driver = dr;
return ret;
@@ -446,7 +457,7 @@ cdx_probe_all_drivers(struct rte_cdx_device *dev)
return -EINVAL;
FOREACH_DRIVER_ON_CDXBUS(dr) {
- rc = rte_cdx_probe_one_driver(dr, dev);
+ rc = cdx_probe_one_driver(dr, dev);
if (rc < 0)
/* negative value is an error */
return rc;
@@ -560,6 +571,71 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
return NULL;
}
+/* Remove a device from CDX bus */
+static void
+cdx_remove_device(struct rte_cdx_device *cdx_dev)
+{
+ TAILQ_REMOVE(&rte_cdx_bus.device_list, cdx_dev, next);
+}
+
+/*
+ * If vendor/device ID match, call the remove() function of the
+ * driver.
+ */
+static int
+cdx_detach_dev(struct rte_cdx_device *dev)
+{
+ struct rte_cdx_driver *dr;
+ int ret = 0;
+
+ if (dev == NULL)
+ return -EINVAL;
+
+ dr = dev->driver;
+
+ CDX_BUS_DEBUG("detach device %s using driver: %s",
+ dev->device.name, dr->driver.name);
+
+ if (dr->remove) {
+ ret = dr->remove(dev);
+ if (ret < 0)
+ return ret;
+ }
+
+ /* clear driver structure */
+ dev->driver = NULL;
+ dev->device.driver = NULL;
+
+ rte_cdx_unmap_device(dev);
+
+ rte_intr_instance_free(dev->intr_handle);
+ dev->intr_handle = NULL;
+
+ return 0;
+}
+
+static int
+cdx_plug(struct rte_device *dev)
+{
+ return cdx_probe_all_drivers(RTE_DEV_TO_CDX_DEV(dev));
+}
+
+static int
+cdx_unplug(struct rte_device *dev)
+{
+ struct rte_cdx_device *cdx_dev;
+ int ret;
+
+ cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+ ret = cdx_detach_dev(cdx_dev);
+ if (ret == 0) {
+ cdx_remove_device(cdx_dev);
+ rte_devargs_remove(dev->devargs);
+ free(cdx_dev);
+ }
+ return ret;
+}
+
static int
cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
{
@@ -589,7 +665,7 @@ cdx_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
}
static enum rte_iova_mode
-rte_cdx_get_iommu_class(void)
+cdx_get_iommu_class(void)
{
if (TAILQ_EMPTY(&rte_cdx_bus.device_list))
return RTE_IOVA_DC;
@@ -597,15 +673,61 @@ rte_cdx_get_iommu_class(void)
return RTE_IOVA_VA;
}
+static int
+cdx_dev_match(const struct rte_device *dev,
+ const void *_kvlist)
+{
+ const struct rte_kvargs *kvlist = _kvlist;
+ const char *key = cdx_params_keys[RTE_CDX_PARAM_NAME];
+ const char *name;
+
+ /* no kvlist arg, all devices match */
+ if (kvlist == NULL)
+ return 0;
+
+ /* if key is present in kvlist and does not match, filter device */
+ name = rte_kvargs_get(kvlist, key);
+ if (name != NULL && strcmp(name, dev->name))
+ return -1;
+
+ return 0;
+}
+
+static void *
+cdx_dev_iterate(const void *start,
+ const char *str,
+ const struct rte_dev_iterator *it __rte_unused)
+{
+ rte_bus_find_device_t find_device;
+ struct rte_kvargs *kvargs = NULL;
+ struct rte_device *dev;
+
+ if (str != NULL) {
+ kvargs = rte_kvargs_parse(str, cdx_params_keys);
+ if (kvargs == NULL) {
+ CDX_BUS_ERR("cannot parse argument list %s", str);
+ rte_errno = EINVAL;
+ return NULL;
+ }
+ }
+ find_device = rte_cdx_bus.bus.find_device;
+ dev = find_device(start, cdx_dev_match, kvargs);
+ rte_kvargs_free(kvargs);
+ return dev;
+}
+
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
- .scan = rte_cdx_scan,
+ .scan = cdx_scan,
.probe = cdx_probe,
.find_device = cdx_find_device,
+ .plug = cdx_plug,
+ .unplug = cdx_unplug,
.parse = cdx_parse,
.dma_map = cdx_dma_map,
.dma_unmap = cdx_dma_unmap,
- .get_iommu_class = rte_cdx_get_iommu_class,
+ .get_iommu_class = cdx_get_iommu_class,
+ .dev_iterate = cdx_dev_iterate,
},
.device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
.driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
diff --git a/drivers/bus/cdx/rte_bus_cdx.h b/drivers/bus/cdx/rte_bus_cdx.h
index 43d75bc3b4..564fa8d047 100644
--- a/drivers/bus/cdx/rte_bus_cdx.h
+++ b/drivers/bus/cdx/rte_bus_cdx.h
@@ -78,6 +78,7 @@ struct rte_cdx_id {
struct rte_cdx_device {
RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
struct rte_device device; /**< Inherit core device */
+ struct rte_cdx_driver *driver; /**< CDX driver used in probing */
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
/**< CDX Memory Resource */
--
2.25.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [RFC PATCH 5/6] bus: enable cdx bus
2023-01-24 14:07 [RFC PATCH 0/6] add support for CDX bus Nipun Gupta
` (3 preceding siblings ...)
2023-01-24 14:07 ` [RFC PATCH 4/6] bus/cdx: support plug unplug and dev iterator Nipun Gupta
@ 2023-01-24 14:07 ` Nipun Gupta
2023-01-24 14:07 ` [RFC PATCH 6/6] config/arm: add AMD CDX Nipun Gupta
` (8 subsequent siblings)
13 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-01-24 14:07 UTC (permalink / raw)
To: dev, thomas, david.marchand; +Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal
enable compilation of cdx bus
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
drivers/bus/meson.build | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
index 45eab5233d..884470234c 100644
--- a/drivers/bus/meson.build
+++ b/drivers/bus/meson.build
@@ -7,6 +7,7 @@ drivers = [
'fslmc',
'ifpga',
'pci',
+ 'cdx',
'vdev',
'vmbus',
]
--
2.25.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [RFC PATCH 6/6] config/arm: add AMD CDX
2023-01-24 14:07 [RFC PATCH 0/6] add support for CDX bus Nipun Gupta
` (4 preceding siblings ...)
2023-01-24 14:07 ` [RFC PATCH 5/6] bus: enable cdx bus Nipun Gupta
@ 2023-01-24 14:07 ` Nipun Gupta
2023-04-07 6:01 ` [PATCH 0/6] add support for CDX bus Nipun Gupta
` (7 subsequent siblings)
13 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-01-24 14:07 UTC (permalink / raw)
To: dev, thomas, david.marchand; +Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal
Adding support for AMD CDX devices
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
config/arm/arm64_cdx_linux_gcc | 17 +++++++++++++++++
config/arm/meson.build | 15 +++++++++++++++
2 files changed, 32 insertions(+)
create mode 100644 config/arm/arm64_cdx_linux_gcc
diff --git a/config/arm/arm64_cdx_linux_gcc b/config/arm/arm64_cdx_linux_gcc
new file mode 100644
index 0000000000..8e6d619dae
--- /dev/null
+++ b/config/arm/arm64_cdx_linux_gcc
@@ -0,0 +1,17 @@
+[binaries]
+c = ['ccache', 'aarch64-linux-gnu-gcc']
+cpp = ['ccache', 'aarch64-linux-gnu-g++']
+ar = 'aarch64-linux-gnu-ar'
+as = 'aarch64-linux-gnu-as'
+strip = 'aarch64-linux-gnu-strip'
+pkgconfig = 'aarch64-linux-gnu-pkg-config'
+pcap-config = ''
+
+[host_machine]
+system = 'linux'
+cpu_family = 'aarch64'
+cpu = 'armv8-a'
+endian = 'little'
+
+[properties]
+platform = 'cdx'
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 6442ec9596..76806b2820 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -63,6 +63,7 @@ part_number_config_arm = {
'0xd09': {'compiler_options': ['-mcpu=cortex-a73']},
'0xd0a': {'compiler_options': ['-mcpu=cortex-a75']},
'0xd0b': {'compiler_options': ['-mcpu=cortex-a76']},
+ '0xd42': {'compiler_options': ['-mcpu=cortex-a78']},
'0xd0c': {
'march': 'armv8.2-a',
'march_features': ['crypto'],
@@ -302,6 +303,18 @@ soc_bluefield = {
'numa': false
}
+soc_cdx = {
+ 'description': 'AMD CDX',
+ 'implementer': '0x41',
+ 'part_number': '0xd42',
+ 'flags': [
+ ['RTE_MACHINE', '"cdx"'],
+ ['RTE_MAX_LCORE', 16],
+ ['RTE_MAX_NUMA_NODES', 1]
+ ],
+ 'numa': false
+}
+
soc_centriq2400 = {
'description': 'Qualcomm Centriq 2400',
'implementer': '0x51',
@@ -448,6 +461,7 @@ generic: Generic un-optimized build for armv8 aarch64 execution mode.
generic_aarch32: Generic un-optimized build for armv8 aarch32 execution mode.
armada: Marvell ARMADA
bluefield: NVIDIA BlueField
+cdx: AMD CDX
centriq2400: Qualcomm Centriq 2400
cn9k: Marvell OCTEON 9
cn10k: Marvell OCTEON 10
@@ -474,6 +488,7 @@ socs = {
'generic_aarch32': soc_generic_aarch32,
'armada': soc_armada,
'bluefield': soc_bluefield,
+ 'cdx': soc_cdx,
'centriq2400': soc_centriq2400,
'cn9k': soc_cn9k,
'cn10k' : soc_cn10k,
--
2.25.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH 0/6] add support for CDX bus
2023-01-24 14:07 [RFC PATCH 0/6] add support for CDX bus Nipun Gupta
` (5 preceding siblings ...)
2023-01-24 14:07 ` [RFC PATCH 6/6] config/arm: add AMD CDX Nipun Gupta
@ 2023-04-07 6:01 ` Nipun Gupta
2023-04-07 6:01 ` [PATCH 1/6] bus/cdx: introduce cdx bus Nipun Gupta
` (6 more replies)
2023-04-13 13:26 ` [PATCH v2 " Nipun Gupta
` (6 subsequent siblings)
13 siblings, 7 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-04-07 6:01 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
Support AMD CDX bus, for FPGA based CDX devices. The CDX
devices are memory mapped on system bus for embedded CPUs.
It uses sysfs interface and the vfio-cdx driver to discover
and initialize the CDX devices.
The patches are intended for DPDK 23.07 release, and have been sent
as an RFC as patches are yet to be merged in Linux.
Linux CDX bus patches has been added into linux next:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/cdx
VFIO patches are also submitted in upstream:
https://www.spinics.net/lists/kvm/msg310623.html
CDX is a Hardware Architecture designed for AMD FPGA devices. It
consists of mechanism for interaction between FPGA, Firmware and
the APUs (Application CPUs).
Firmware resides on RPU (Realtime CPUs) which interacts with
the FPGA program manager and the APUs. The RPU provides memory-mapped
interface (RPU if) which is used to communicate with APUs.
VFIO CDX driver provides the CDX device resources like MMIO and interrupts
to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
driver to discover and initialize the CDX devices for user-space
applications.
RFC patches in DPDK were submitted at:
http://patches.dpdk.org/project/dpdk/patch/20230124140746.594066-2-nipun.gupta@amd.com/
Changes RFC->v1:
- Marked few API's as internal which were not required
to be provided to user.
Nipun Gupta (6):
bus/cdx: introduce cdx bus
bus/cdx: add dma map and unmap support
bus/cdx: add support for MSI
bus/cdx: support plug unplug and dev iterator
bus: enable cdx bus
config/arm: add AMD CDX
MAINTAINERS | 5 +
config/arm/arm64_cdx_linux_gcc | 17 +
config/arm/meson.build | 14 +
drivers/bus/cdx/cdx.c | 743 +++++++++++++++++++++++++
drivers/bus/cdx/cdx.h | 54 ++
drivers/bus/cdx/cdx_logs.h | 37 ++
drivers/bus/cdx/cdx_vfio.c | 598 ++++++++++++++++++++
drivers/bus/cdx/meson.build | 13 +
drivers/bus/cdx/rte_bus_cdx.h | 245 ++++++++
drivers/bus/cdx/version.map | 21 +
drivers/bus/meson.build | 1 +
lib/eal/common/eal_common_interrupts.c | 21 +
lib/eal/common/eal_interrupts.h | 1 +
lib/eal/include/rte_interrupts.h | 32 ++
lib/eal/version.map | 2 +
15 files changed, 1804 insertions(+)
create mode 100644 config/arm/arm64_cdx_linux_gcc
create mode 100644 drivers/bus/cdx/cdx.c
create mode 100644 drivers/bus/cdx/cdx.h
create mode 100644 drivers/bus/cdx/cdx_logs.h
create mode 100644 drivers/bus/cdx/cdx_vfio.c
create mode 100644 drivers/bus/cdx/meson.build
create mode 100644 drivers/bus/cdx/rte_bus_cdx.h
create mode 100644 drivers/bus/cdx/version.map
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH 1/6] bus/cdx: introduce cdx bus
2023-04-07 6:01 ` [PATCH 0/6] add support for CDX bus Nipun Gupta
@ 2023-04-07 6:01 ` Nipun Gupta
2023-04-07 6:01 ` [PATCH 2/6] bus/cdx: add dma map and unmap support Nipun Gupta
` (5 subsequent siblings)
6 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-04-07 6:01 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
CDX bus supports multiple type of devices, which can be
exposed to user-space via vfio-cdx.
vfio-cdx provides the MMIO IO_MEMORY regions as well as the
DMA interface for the device (IOMMU).
This support aims to enable the DPDK to support the cdx
devices in user-space using VFIO interface.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
MAINTAINERS | 5 +
drivers/bus/cdx/cdx.c | 570 ++++++++++++++++++++++++++++++++++
drivers/bus/cdx/cdx.h | 54 ++++
drivers/bus/cdx/cdx_logs.h | 37 +++
drivers/bus/cdx/cdx_vfio.c | 420 +++++++++++++++++++++++++
drivers/bus/cdx/meson.build | 13 +
drivers/bus/cdx/rte_bus_cdx.h | 219 +++++++++++++
drivers/bus/cdx/version.map | 19 ++
8 files changed, 1337 insertions(+)
create mode 100644 drivers/bus/cdx/cdx.c
create mode 100644 drivers/bus/cdx/cdx.h
create mode 100644 drivers/bus/cdx/cdx_logs.h
create mode 100644 drivers/bus/cdx/cdx_vfio.c
create mode 100644 drivers/bus/cdx/meson.build
create mode 100644 drivers/bus/cdx/rte_bus_cdx.h
create mode 100644 drivers/bus/cdx/version.map
diff --git a/MAINTAINERS b/MAINTAINERS
index 8df23e5099..1f9b6af9b9 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -569,6 +569,11 @@ M: Parav Pandit <parav@nvidia.com>
M: Xueming Li <xuemingl@nvidia.com>
F: drivers/bus/auxiliary/
+CDX bus driver
+M: Nipun Gupta <nipun.gupta@amd.com>
+M: Nikhil Agarwal <nikhil.agarwal@amd.com>
+F: drivers/bus/cdx/
+
Intel FPGA bus
M: Rosen Xu <rosen.xu@intel.com>
F: drivers/bus/ifpga/
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
new file mode 100644
index 0000000000..c5f8e37a2f
--- /dev/null
+++ b/drivers/bus/cdx/cdx.c
@@ -0,0 +1,570 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+/*
+ * Architecture Overview
+ * =====================
+ * CDX is a Hardware Architecture designed for AMD FPGA devices. It
+ * consists of sophisticated mechanism for interaction between FPGA,
+ * Firmware and the APUs (Application CPUs).
+ *
+ * Firmware resides on RPU (Realtime CPUs) which interacts with
+ * the FPGA program manager and the APUs. The RPU provides memory-mapped
+ * interface (RPU if) which is used to communicate with APUs.
+ *
+ * The diagram below shows an overview of the CDX architecture:
+ *
+ * +--------------------------------------+
+ * | DPDK |
+ * | DPDK CDX drivers |
+ * | | |
+ * | DPDK CDX bus |
+ * | | |
+ * +-----------------------------|--------+
+ * |
+ * +-----------------------------|--------+
+ * | Application CPUs (APU) | |
+ * | | |
+ * | VFIO CDX driver |
+ * | Linux OS | |
+ * | Linux CDX bus |
+ * | | |
+ * +-----------------------------|--------+
+ * |
+ * |
+ * +------------------------| RPU if |----+
+ * | | |
+ * | V |
+ * | Realtime CPUs (RPU) |
+ * | |
+ * +--------------------------------------+
+ * |
+ * +---------------------|----------------+
+ * | FPGA | |
+ * | +-----------------------+ |
+ * | | | | |
+ * | +-------+ +-------+ +-------+ |
+ * | | dev 1 | | dev 2 | | dev 3 | |
+ * | +-------+ +-------+ +-------+ |
+ * +--------------------------------------+
+ *
+ * The RPU firmware extracts the device information from the loaded FPGA
+ * image and implements a mechanism that allows the APU drivers to
+ * enumerate such devices (device personality and resource details) via
+ * a dedicated communication channel.
+ *
+ * VFIO CDX driver provides the CDX device resources like MMIO and interrupts
+ * to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
+ * driver to discover and initialize the CDX devices for user-space
+ * applications.
+ */
+
+#include <string.h>
+#include <dirent.h>
+
+#include <rte_log.h>
+#include <rte_bus.h>
+#include <rte_bus_cdx.h>
+#include <rte_eal_paging.h>
+#include <rte_errno.h>
+#include <rte_devargs.h>
+#include <rte_malloc.h>
+#include <rte_memcpy.h>
+#include <rte_vfio.h>
+
+#include <eal_filesystem.h>
+
+#include "cdx.h"
+#include "cdx_logs.h"
+
+#define SYSFS_CDX_DEVICES "/sys/bus/cdx/devices"
+#define CDX_BUS_NAME cdx
+#define CDX_DEV_PREFIX "cdx-"
+
+/**
+ * @file
+ * CDX probing using Linux sysfs.
+ */
+
+/* Add a device to CDX bus */
+static void
+rte_cdx_add_device(struct rte_cdx_device *cdx_dev)
+{
+ TAILQ_INSERT_TAIL(&rte_cdx_bus.device_list, cdx_dev, next);
+}
+
+static int
+cdx_get_kernel_driver_by_path(const char *filename, char *driver_name,
+ size_t len)
+{
+ int count;
+ char path[PATH_MAX];
+ char *name;
+
+ if (!filename || !driver_name)
+ return -1;
+
+ count = readlink(filename, path, PATH_MAX);
+ if (count >= PATH_MAX)
+ return -1;
+
+ /* For device does not have a driver */
+ if (count < 0)
+ return 1;
+
+ path[count] = '\0';
+
+ name = strrchr(path, '/');
+ if (name) {
+ strlcpy(driver_name, name + 1, len);
+ return 0;
+ }
+
+ return -1;
+}
+
+int rte_cdx_map_device(struct rte_cdx_device *dev)
+{
+ return cdx_vfio_map_resource(dev);
+}
+
+void rte_cdx_unmap_device(struct rte_cdx_device *dev)
+{
+ cdx_vfio_unmap_resource(dev);
+}
+
+static int
+find_max_end_va(const struct rte_memseg_list *msl, void *arg)
+{
+ size_t sz = msl->len;
+ void *end_va = RTE_PTR_ADD(msl->base_va, sz);
+ void **max_va = arg;
+
+ if (*max_va < end_va)
+ *max_va = end_va;
+ return 0;
+}
+
+void *
+cdx_find_max_end_va(void)
+{
+ void *va = NULL;
+
+ rte_memseg_list_walk(find_max_end_va, &va);
+ return va;
+}
+
+static struct rte_devargs *
+cdx_devargs_lookup(const char *dev_name)
+{
+ struct rte_devargs *devargs;
+
+ RTE_EAL_DEVARGS_FOREACH("cdx", devargs) {
+ if (strcmp(devargs->name, dev_name) == 0)
+ return devargs;
+ }
+ return NULL;
+}
+
+static bool
+cdx_ignore_device(const char *dev_name)
+{
+ struct rte_devargs *devargs = cdx_devargs_lookup(dev_name);
+
+ switch (rte_cdx_bus.bus.conf.scan_mode) {
+ case RTE_BUS_SCAN_ALLOWLIST:
+ if (devargs && devargs->policy == RTE_DEV_ALLOWED)
+ return false;
+ break;
+ case RTE_BUS_SCAN_UNDEFINED:
+ case RTE_BUS_SCAN_BLOCKLIST:
+ if (devargs == NULL || devargs->policy != RTE_DEV_BLOCKED)
+ return false;
+ break;
+ }
+ return true;
+}
+
+/*
+ * Scan one cdx sysfs entry, and fill the devices list from it.
+ * It checks if the CDX device is bound to vfio-cdx driver. In case
+ * the device is vfio bound, it reads the vendor and device id and
+ * stores it for device-driver matching.
+ */
+static int
+cdx_scan_one(const char *dirname, const char *dev_name)
+{
+ char filename[PATH_MAX];
+ struct rte_cdx_device *dev = NULL;
+ char driver[PATH_MAX];
+ unsigned long tmp;
+ char *name = NULL;
+ int ret;
+
+ dev = calloc(1, sizeof(*dev));
+ if (!dev)
+ return -ENOMEM;
+
+ name = calloc(1, RTE_DEV_NAME_MAX_LEN);
+ if (!name) {
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ dev->device.bus = &rte_cdx_bus.bus;
+ memcpy(name, dev_name, RTE_DEV_NAME_MAX_LEN);
+ dev->device.name = name;
+
+ /* parse driver */
+ snprintf(filename, sizeof(filename), "%s/driver", dirname);
+ ret = cdx_get_kernel_driver_by_path(filename, driver, sizeof(driver));
+ if (ret < 0) {
+ CDX_BUS_ERR("Fail to get kernel driver");
+ ret = -1;
+ goto err;
+ }
+
+ /*
+ * Check if device is bound to 'vfio-cdx' driver, so that user-space
+ * can gracefully access the device.
+ */
+ if (ret || strcmp(driver, "vfio-cdx")) {
+ ret = 0;
+ goto err;
+ }
+
+ /* get vendor id */
+ snprintf(filename, sizeof(filename), "%s/vendor", dirname);
+ if (eal_parse_sysfs_value(filename, &tmp) < 0) {
+ ret = -1;
+ goto err;
+ }
+ dev->id.vendor_id = (uint16_t)tmp;
+
+ /* get device id */
+ snprintf(filename, sizeof(filename), "%s/device", dirname);
+ if (eal_parse_sysfs_value(filename, &tmp) < 0) {
+ free(dev);
+ return -1;
+ }
+ dev->id.device_id = (uint16_t)tmp;
+
+ rte_cdx_add_device(dev);
+
+ return 0;
+
+err:
+ if (name)
+ free(name);
+ if (dev)
+ free(dev);
+ return ret;
+}
+
+/*
+ * Scan the content of the CDX bus, and the devices in the devices
+ * list.
+ */
+static int
+rte_cdx_scan(void)
+{
+ struct dirent *e;
+ DIR *dir;
+ char dirname[PATH_MAX];
+
+ dir = opendir(rte_cdx_get_sysfs_path());
+ if (dir == NULL) {
+ CDX_BUS_ERR("%s(): opendir failed: %s", __func__,
+ strerror(errno));
+ return -1;
+ }
+
+ while ((e = readdir(dir)) != NULL) {
+ if (e->d_name[0] == '.')
+ continue;
+
+ if (cdx_ignore_device(e->d_name))
+ continue;
+
+ snprintf(dirname, sizeof(dirname), "%s/%s",
+ rte_cdx_get_sysfs_path(), e->d_name);
+
+ if (cdx_scan_one(dirname, e->d_name) < 0)
+ goto error;
+ }
+ closedir(dir);
+ return 0;
+
+error:
+ closedir(dir);
+ return -1;
+}
+
+const char *
+rte_cdx_get_sysfs_path(void)
+{
+ return SYSFS_CDX_DEVICES;
+}
+
+/* map a particular resource from a file */
+void *
+cdx_map_resource(void *requested_addr, int fd, off_t offset, size_t size,
+ int additional_flags)
+{
+ void *mapaddr;
+
+ /* Map the cdx MMIO memory resource of device */
+ mapaddr = rte_mem_map(requested_addr, size,
+ RTE_PROT_READ | RTE_PROT_WRITE,
+ RTE_MAP_SHARED | additional_flags, fd, offset);
+ if (mapaddr == NULL) {
+ CDX_BUS_ERR("%s(): cannot map resource(%d, %p, 0x%zx, 0x%llx): %s (%p)",
+ __func__, fd, requested_addr, size,
+ (unsigned long long)offset,
+ rte_strerror(rte_errno), mapaddr);
+ }
+ CDX_BUS_DEBUG("CDX MMIO memory mapped at %p", mapaddr);
+
+ return mapaddr;
+}
+
+/* unmap a particular resource */
+void
+cdx_unmap_resource(void *requested_addr, size_t size)
+{
+ if (requested_addr == NULL)
+ return;
+
+ /* Unmap the CDX memory resource of device */
+ if (rte_mem_unmap(requested_addr, size)) {
+ CDX_BUS_ERR("%s(): cannot mem unmap(%p, %#zx): %s", __func__,
+ requested_addr, size, rte_strerror(rte_errno));
+ }
+ CDX_BUS_DEBUG("CDX memory unmapped at %p", requested_addr);
+}
+/*
+ * Match the CDX Driver and Device using device id and vendor id.
+ */
+static int
+rte_cdx_match(const struct rte_cdx_driver *cdx_drv,
+ const struct rte_cdx_device *cdx_dev)
+{
+ const struct rte_cdx_id *id_table;
+
+ for (id_table = cdx_drv->id_table; id_table->vendor_id != 0;
+ id_table++) {
+ /* check if device's identifiers match the driver's ones */
+ if (id_table->vendor_id != cdx_dev->id.vendor_id &&
+ id_table->vendor_id != RTE_CDX_ANY_ID)
+ continue;
+ if (id_table->device_id != cdx_dev->id.device_id &&
+ id_table->device_id != RTE_CDX_ANY_ID)
+ continue;
+
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * If vendor id and device id match, call the probe() function of the
+ * driver.
+ */
+static int
+rte_cdx_probe_one_driver(struct rte_cdx_driver *dr,
+ struct rte_cdx_device *dev)
+{
+ const char *dev_name = dev->device.name;
+ bool already_probed;
+ int ret;
+
+ if ((dr == NULL) || (dev == NULL))
+ return -EINVAL;
+
+ /* The device is not blocked; Check if driver supports it */
+ if (!rte_cdx_match(dr, dev))
+ /* Match of device and driver failed */
+ return 1;
+
+ already_probed = rte_dev_is_probed(&dev->device);
+ if (already_probed) {
+ CDX_BUS_INFO("Device %s is already probed", dev->device.name);
+ return -EEXIST;
+ }
+
+ CDX_BUS_DEBUG(" probe device %s using driver: %s", dev_name,
+ dr->driver.name);
+
+ ret = cdx_vfio_map_resource(dev);
+ if (ret != 0) {
+ CDX_BUS_ERR("CDX map device failed: %d", ret);
+ goto error_map_device;
+ }
+
+ /* call the driver probe() function */
+ ret = dr->probe(dr, dev);
+ if (ret) {
+ CDX_BUS_ERR("Probe CDX driver: %s device: %s failed: %d",
+ dr->driver.name, dev_name, ret);
+ goto error_probe;
+ } else {
+ dev->device.driver = &dr->driver;
+ }
+
+ return ret;
+
+error_probe:
+ cdx_vfio_unmap_resource(dev);
+error_map_device:
+ return ret;
+}
+
+/*
+ * If vendor/device ID match, call the probe() function of all
+ * registered driver for the given device. Return < 0 if initialization
+ * failed, return 1 if no driver is found for this device.
+ */
+static int
+cdx_probe_all_drivers(struct rte_cdx_device *dev)
+{
+ struct rte_cdx_driver *dr = NULL;
+ int rc = 0;
+
+ if (dev == NULL)
+ return -EINVAL;
+
+ FOREACH_DRIVER_ON_CDXBUS(dr) {
+ rc = rte_cdx_probe_one_driver(dr, dev);
+ if (rc < 0)
+ /* negative value is an error */
+ return rc;
+ if (rc > 0)
+ /* positive value means driver doesn't support it */
+ continue;
+ return 0;
+ }
+ return 1;
+}
+
+/*
+ * Scan the content of the CDX bus, and call the probe() function for
+ * all registered drivers that have a matching entry in its id_table
+ * for discovered devices.
+ */
+static int
+cdx_probe(void)
+{
+ struct rte_cdx_device *dev = NULL;
+ size_t probed = 0, failed = 0;
+ int ret = 0;
+
+ FOREACH_DEVICE_ON_CDXBUS(dev) {
+ probed++;
+
+ ret = cdx_probe_all_drivers(dev);
+ if (ret < 0) {
+ CDX_BUS_ERR("Requested device %s cannot be used",
+ dev->device.name);
+ rte_errno = errno;
+ failed++;
+ ret = 0;
+ }
+ }
+
+ return (probed && probed == failed) ? -1 : 0;
+}
+
+/* dump one device */
+static int
+cdx_dump_one_device(FILE *f, struct rte_cdx_device *dev)
+{
+ int i;
+
+ fprintf(f, "cdx device %s\n", dev->device.name);
+
+ for (i = 0; i != sizeof(dev->mem_resource) /
+ sizeof(dev->mem_resource[0]); i++) {
+ if (dev->mem_resource[i].len)
+ fprintf(f, " Resource[%d]: %p %16.16"PRIx64"\n",
+ i, dev->mem_resource[i].addr,
+ dev->mem_resource[i].len);
+ }
+ return 0;
+}
+
+/* dump devices on the bus */
+void
+rte_cdx_dump(FILE *f)
+{
+ struct rte_cdx_device *dev = NULL;
+
+ FOREACH_DEVICE_ON_CDXBUS(dev)
+ cdx_dump_one_device(f, dev);
+}
+
+static int
+cdx_parse(const char *name, void *addr)
+{
+ const char **out = addr;
+ int ret;
+
+ ret = strncmp(name, CDX_DEV_PREFIX, strlen(CDX_DEV_PREFIX));
+
+ if (ret == 0 && addr)
+ *out = name;
+
+ return ret;
+}
+
+/* register a driver */
+void
+rte_cdx_register(struct rte_cdx_driver *driver)
+{
+ TAILQ_INSERT_TAIL(&rte_cdx_bus.driver_list, driver, next);
+ driver->bus = &rte_cdx_bus;
+}
+
+/* unregister a driver */
+void
+rte_cdx_unregister(struct rte_cdx_driver *driver)
+{
+ TAILQ_REMOVE(&rte_cdx_bus.driver_list, driver, next);
+ driver->bus = NULL;
+}
+
+static struct rte_device *
+cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
+ const void *data)
+{
+ const struct rte_cdx_device *cdx_start;
+ struct rte_cdx_device *cdx_dev;
+
+ if (start != NULL) {
+ cdx_start = RTE_DEV_TO_CDX_DEV_CONST(start);
+ cdx_dev = TAILQ_NEXT(cdx_start, next);
+ } else {
+ cdx_dev = TAILQ_FIRST(&rte_cdx_bus.device_list);
+ }
+ while (cdx_dev != NULL) {
+ if (cmp(&cdx_dev->device, data) == 0)
+ return &cdx_dev->device;
+ cdx_dev = TAILQ_NEXT(cdx_dev, next);
+ }
+ return NULL;
+}
+
+struct rte_cdx_bus rte_cdx_bus = {
+ .bus = {
+ .scan = rte_cdx_scan,
+ .probe = cdx_probe,
+ .find_device = cdx_find_device,
+ .parse = cdx_parse,
+ },
+ .device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
+ .driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
+};
+
+RTE_REGISTER_BUS(cdx, rte_cdx_bus.bus);
+RTE_LOG_REGISTER_DEFAULT(cdx_logtype_bus, NOTICE);
diff --git a/drivers/bus/cdx/cdx.h b/drivers/bus/cdx/cdx.h
new file mode 100644
index 0000000000..d428149bb9
--- /dev/null
+++ b/drivers/bus/cdx/cdx.h
@@ -0,0 +1,54 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef _CDX_H_
+#define _CDX_H_
+
+#include <stdbool.h>
+#include <stdio.h>
+
+#include "rte_bus_cdx.h"
+
+extern struct rte_cdx_bus rte_cdx_bus;
+
+/**
+ * Map a particular resource from a file.
+ *
+ * @param requested_addr
+ * The starting address for the new mapping range.
+ * @param fd
+ * The file descriptor.
+ * @param offset
+ * The offset for the mapping range.
+ * @param size
+ * The size for the mapping range.
+ * @param additional_flags
+ * The additional rte_mem_map() flags for the mapping range.
+ * @return
+ * - On success, the function returns a pointer to the mapped area.
+ * - On error, NULL is returned.
+ */
+void *cdx_map_resource(void *requested_addr, int fd, off_t offset,
+ size_t size, int additional_flags);
+
+/**
+ * Unmap a particular resource.
+ *
+ * @param requested_addr
+ * The address for the unmapping range.
+ * @param size
+ * The size for the unmapping range.
+ */
+void cdx_unmap_resource(void *requested_addr, size_t size);
+
+/*
+ * Helper function to map CDX resources right after hugepages in virtual memory
+ */
+void *cdx_find_max_end_va(void);
+
+/* map/unmap VFIO resource */
+int cdx_vfio_map_resource(struct rte_cdx_device *dev);
+int cdx_vfio_unmap_resource(struct rte_cdx_device *dev);
+
+#endif /* _CDX_H_ */
diff --git a/drivers/bus/cdx/cdx_logs.h b/drivers/bus/cdx/cdx_logs.h
new file mode 100644
index 0000000000..7a53633a1b
--- /dev/null
+++ b/drivers/bus/cdx/cdx_logs.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef _CDX_LOGS_H_
+#define _CDX_LOGS_H_
+
+extern int cdx_logtype_bus;
+
+#define CDX_BUS_LOG(level, fmt, args...) \
+ rte_log(RTE_LOG_ ## level, cdx_logtype_bus, "fslmc: " fmt "\n", \
+ ##args)
+
+/* Debug logs are with Function names */
+#define CDX_BUS_DEBUG(fmt, args...) \
+ rte_log(RTE_LOG_DEBUG, cdx_logtype_bus, "fslmc: %s(): " fmt "\n", \
+ __func__, ##args)
+
+#define CDX_BUS_INFO(fmt, args...) \
+ CDX_BUS_LOG(INFO, fmt, ## args)
+#define CDX_BUS_ERR(fmt, args...) \
+ CDX_BUS_LOG(ERR, fmt, ## args)
+#define CDX_BUS_WARN(fmt, args...) \
+ CDX_BUS_LOG(WARNING, fmt, ## args)
+
+/* DP Logs, toggled out at compile time if level lower than current level */
+#define CDX_BUS_DP_LOG(level, fmt, args...) \
+ RTE_LOG_DP(level, PMD, fmt, ## args)
+
+#define CDX_BUS_DP_DEBUG(fmt, args...) \
+ CDX_BUS_DP_LOG(DEBUG, fmt, ## args)
+#define CDX_BUS_DP_INFO(fmt, args...) \
+ CDX_BUS_DP_LOG(INFO, fmt, ## args)
+#define CDX_BUS_DP_WARN(fmt, args...) \
+ CDX_BUS_DP_LOG(WARNING, fmt, ## args)
+
+#endif /* _CDX_LOGS_H_ */
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
new file mode 100644
index 0000000000..a53137c416
--- /dev/null
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -0,0 +1,420 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+/**
+ * @file
+ * CDX probing using Linux VFIO.
+ *
+ * This code tries to determine if the CDX device is bound to VFIO driver,
+ * and initialize it (map BARs, set up interrupts) if that's the case.
+ *
+ */
+
+#include <string.h>
+#include <fcntl.h>
+#include <sys/eventfd.h>
+#include <sys/socket.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <stdbool.h>
+
+#include <rte_log.h>
+#include <rte_bus_cdx.h>
+#include <rte_eal_paging.h>
+#include <rte_malloc.h>
+#include <rte_vfio.h>
+#include <rte_eal.h>
+#include <rte_bus.h>
+#include <rte_spinlock.h>
+#include <rte_tailq.h>
+
+#include "eal_filesystem.h"
+
+#include "cdx.h"
+#include "cdx_logs.h"
+
+/**
+ * A structure describing a CDX mapping.
+ */
+struct cdx_map {
+ void *addr;
+ char *path;
+ uint64_t offset;
+ uint64_t size;
+};
+
+/**
+ * A structure describing a mapped CDX resource.
+ * For multi-process we need to reproduce all CDX mappings in secondary
+ * processes, so save them in a tailq.
+ */
+struct mapped_cdx_resource {
+ TAILQ_ENTRY(mapped_cdx_resource) next;
+ char name[RTE_DEV_NAME_MAX_LEN]; /**< CDX device name */
+ char path[PATH_MAX];
+ int nb_maps;
+ struct cdx_map maps[CDX_MAX_RESOURCE];
+};
+
+/** mapped cdx device list */
+TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
+
+static struct rte_tailq_elem cdx_vfio_tailq = {
+ .name = "VFIO_CDX_RESOURCE_LIST",
+};
+EAL_REGISTER_TAILQ(cdx_vfio_tailq)
+
+static struct mapped_cdx_resource *
+cdx_vfio_find_and_unmap_resource(struct mapped_cdx_res_list *vfio_res_list,
+ struct rte_cdx_device *dev)
+{
+ struct mapped_cdx_resource *vfio_res = NULL;
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+ int i;
+
+ /* Get vfio_res */
+ TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+ if (strcmp(vfio_res->name, dev_name))
+ continue;
+ break;
+ }
+
+ if (vfio_res == NULL)
+ return vfio_res;
+
+ CDX_BUS_INFO("Releasing CDX mapped resource for %s", dev_name);
+
+ maps = vfio_res->maps;
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ /*
+ * We do not need to be aware of MSI-X table BAR mappings as
+ * when mapping. Just using current maps array is enough
+ */
+ if (maps[i].addr) {
+ CDX_BUS_DEBUG("Calling cdx_unmap_resource for %s at %p",
+ dev_name, maps[i].addr);
+ cdx_unmap_resource(maps[i].addr, maps[i].size);
+ }
+ }
+
+ return vfio_res;
+}
+
+static int
+cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
+{
+ char cdx_addr[PATH_MAX] = {0};
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list;
+
+ vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ vfio_res = cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for cdx device!",
+ cdx_addr);
+ return -1;
+ }
+
+ TAILQ_REMOVE(vfio_res_list, vfio_res, next);
+ rte_free(vfio_res);
+ return 0;
+}
+
+static int
+cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
+{
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list;
+
+ vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ vfio_res = cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for CDX device!",
+ dev->device.name);
+ return -1;
+ }
+
+ return 0;
+}
+
+int
+cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
+{
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return cdx_vfio_unmap_resource_primary(dev);
+ else
+ return cdx_vfio_unmap_resource_secondary(dev);
+}
+
+static int
+cdx_rte_vfio_setup_device(int vfio_dev_fd)
+{
+ /*
+ * Reset the device. If the device is not capable of resetting,
+ * then it updates errno as EINVAL.
+ */
+ if (ioctl(vfio_dev_fd, VFIO_DEVICE_RESET) && errno != EINVAL) {
+ CDX_BUS_ERR("Unable to reset device! Error: %d (%s)", errno,
+ strerror(errno));
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+cdx_vfio_mmap_resource(int vfio_dev_fd, struct mapped_cdx_resource *vfio_res,
+ int index, int additional_flags)
+{
+ struct cdx_map *map = &vfio_res->maps[index];
+ void *vaddr;
+
+ if (map->size == 0) {
+ CDX_BUS_DEBUG("map size is 0, skip region %d", index);
+ return 0;
+ }
+
+ /* reserve the address using an inaccessible mapping */
+ vaddr = mmap(map->addr, map->size, 0, MAP_PRIVATE |
+ MAP_ANONYMOUS | additional_flags, -1, 0);
+ if (vaddr != MAP_FAILED) {
+ void *map_addr = NULL;
+
+ if (map->size) {
+ /* actual map of first part */
+ map_addr = cdx_map_resource(vaddr, vfio_dev_fd,
+ map->offset, map->size,
+ RTE_MAP_FORCE_ADDRESS);
+ }
+
+ if (map_addr == NULL) {
+ munmap(vaddr, map->size);
+ vaddr = MAP_FAILED;
+ CDX_BUS_ERR("Failed to map cdx MMIO region %d", index);
+ return -1;
+ }
+ } else {
+ CDX_BUS_ERR("Failed to create inaccessible mapping for MMIO region %d",
+ index);
+ return -1;
+ }
+
+ map->addr = vaddr;
+ return 0;
+}
+
+/*
+ * region info may contain capability headers, so we need to keep reallocating
+ * the memory until we match allocated memory size with argsz.
+ */
+static int
+cdx_vfio_get_region_info(int vfio_dev_fd, struct vfio_region_info **info,
+ int region)
+{
+ struct vfio_region_info *ri;
+ size_t argsz = sizeof(*ri);
+ int ret;
+
+ ri = malloc(sizeof(*ri));
+ if (ri == NULL) {
+ CDX_BUS_ERR("Cannot allocate memory for VFIO region info");
+ return -1;
+ }
+again:
+ memset(ri, 0, argsz);
+ ri->argsz = argsz;
+ ri->index = region;
+
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_REGION_INFO, ri);
+ if (ret < 0) {
+ free(ri);
+ return ret;
+ }
+ if (ri->argsz != argsz) {
+ struct vfio_region_info *tmp;
+
+ argsz = ri->argsz;
+ tmp = realloc(ri, argsz);
+
+ if (tmp == NULL) {
+ /* realloc failed but the ri is still there */
+ free(ri);
+ CDX_BUS_ERR("Cannot reallocate memory for VFIO region info");
+ return -1;
+ }
+ ri = tmp;
+ goto again;
+ }
+ *info = ri;
+
+ return 0;
+}
+
+static int
+cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ char cdx_addr[PATH_MAX] = {0};
+ static void *cdx_map_addr;
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+ int vfio_dev_fd, i, ret;
+
+ ret = rte_vfio_setup_device(rte_cdx_get_sysfs_path(), dev_name,
+ &vfio_dev_fd, &device_info);
+ if (ret)
+ return ret;
+
+ /* allocate vfio_res and get region info */
+ vfio_res = rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0);
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("Cannot store VFIO mmap details");
+ goto err_vfio_dev_fd;
+ }
+ memcpy(vfio_res->name, dev_name, RTE_DEV_NAME_MAX_LEN);
+
+ /* get number of registers */
+ vfio_res->nb_maps = device_info.num_regions;
+
+ /* map memory regions */
+ maps = vfio_res->maps;
+
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ struct vfio_region_info *reg = NULL;
+ void *vaddr;
+
+ ret = cdx_vfio_get_region_info(vfio_dev_fd, ®, i);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s cannot get device region info error %i (%s)",
+ dev_name, errno, strerror(errno));
+ goto err_vfio_res;
+ }
+
+ /* skip non-mmappable regions */
+ if ((reg->flags & VFIO_REGION_INFO_FLAG_MMAP) == 0) {
+ free(reg);
+ continue;
+ }
+
+ /* try mapping somewhere close to the end of hugepages */
+ if (cdx_map_addr == NULL)
+ cdx_map_addr = cdx_find_max_end_va();
+
+ vaddr = cdx_map_addr;
+ cdx_map_addr = RTE_PTR_ADD(vaddr, (size_t)reg->size);
+
+ cdx_map_addr = RTE_PTR_ALIGN(cdx_map_addr,
+ sysconf(_SC_PAGE_SIZE));
+
+ maps[i].addr = vaddr;
+ maps[i].offset = reg->offset;
+ maps[i].size = reg->size;
+ maps[i].path = NULL; /* vfio doesn't have per-resource paths */
+
+ ret = cdx_vfio_mmap_resource(vfio_dev_fd, vfio_res, i, 0);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s mapping region %i failed: %s",
+ cdx_addr, i, strerror(errno));
+ free(reg);
+ goto err_vfio_res;
+ }
+
+ dev->mem_resource[i].addr = maps[i].addr;
+ dev->mem_resource[i].len = maps[i].size;
+
+ free(reg);
+ }
+
+ if (cdx_rte_vfio_setup_device(vfio_dev_fd) < 0) {
+ CDX_BUS_ERR("%s setup device failed", dev_name);
+ goto err_vfio_res;
+ }
+
+ TAILQ_INSERT_TAIL(vfio_res_list, vfio_res, next);
+
+ return 0;
+err_vfio_res:
+ cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+ rte_free(vfio_res);
+err_vfio_dev_fd:
+ rte_vfio_release_device(rte_cdx_get_sysfs_path(),
+ dev_name, vfio_dev_fd);
+ return -1;
+}
+
+static int
+cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ char cdx_addr[PATH_MAX] = {0};
+ int vfio_dev_fd;
+ int i, ret;
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+
+ /* if we're in a secondary process, just find our tailq entry */
+ TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+ if (strcmp(vfio_res->name, dev_name))
+ continue;
+ break;
+ }
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for cdx device!",
+ dev_name);
+ return -1;
+ }
+
+ ret = rte_vfio_setup_device(rte_cdx_get_sysfs_path(), dev_name,
+ &vfio_dev_fd, &device_info);
+ if (ret)
+ return ret;
+
+ /* map MMIO regions */
+ maps = vfio_res->maps;
+
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ ret = cdx_vfio_mmap_resource(vfio_dev_fd, vfio_res, i, MAP_FIXED);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s mapping MMIO region %i failed: %s",
+ dev_name, i, strerror(errno));
+ goto err_vfio_dev_fd;
+ }
+
+ dev->mem_resource[i].addr = maps[i].addr;
+ dev->mem_resource[i].len = maps[i].size;
+ }
+
+ return 0;
+err_vfio_dev_fd:
+ rte_vfio_release_device(rte_cdx_get_sysfs_path(),
+ cdx_addr, vfio_dev_fd);
+ return -1;
+}
+
+/*
+ * map the CDX resources of a CDX device in virtual memory (VFIO version).
+ * primary and secondary processes follow almost exactly the same path
+ */
+int
+cdx_vfio_map_resource(struct rte_cdx_device *dev)
+{
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return cdx_vfio_map_resource_primary(dev);
+ else
+ return cdx_vfio_map_resource_secondary(dev);
+}
diff --git a/drivers/bus/cdx/meson.build b/drivers/bus/cdx/meson.build
new file mode 100644
index 0000000000..2fb0e1cefd
--- /dev/null
+++ b/drivers/bus/cdx/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+
+if not is_linux
+ build = false
+ reason = 'only supported on Linux'
+endif
+
+headers = files('rte_bus_cdx.h')
+sources = files(
+ 'cdx.c',
+ 'cdx_vfio.c',
+)
diff --git a/drivers/bus/cdx/rte_bus_cdx.h b/drivers/bus/cdx/rte_bus_cdx.h
new file mode 100644
index 0000000000..2ebe2a0dcc
--- /dev/null
+++ b/drivers/bus/cdx/rte_bus_cdx.h
@@ -0,0 +1,219 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef _RTE_BUS_CDX_H_
+#define _RTE_BUS_CDX_H_
+
+/**
+ * @file
+ *
+ * CDX device & driver interface
+ *
+ * @warning
+ * @b EXPERIMENTAL:
+ * All functions in this file may be changed or removed without prior notice.
+ *
+ * CDX is a Hardware Architecture designed for AMD FPGA and HNIC devices.
+ * These devices are provided as CDX devices for the user. This driver
+ * provides user interface for devices on the CDX bus.
+ *
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <limits.h>
+#include <errno.h>
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <bus_driver.h>
+#include <dev_driver.h>
+#include <rte_debug.h>
+#include <rte_interrupts.h>
+#include <rte_dev.h>
+#include <rte_bus.h>
+
+/* Forward declarations */
+struct rte_cdx_device;
+struct rte_cdx_driver;
+
+#define CDX_MAX_RESOURCE 4
+
+/** List of CDX devices */
+RTE_TAILQ_HEAD(rte_cdx_device_list, rte_cdx_device);
+/** List of CDX drivers */
+RTE_TAILQ_HEAD(rte_cdx_driver_list, rte_cdx_driver);
+
+/* CDX Bus iterators */
+#define FOREACH_DEVICE_ON_CDXBUS(p) \
+ RTE_TAILQ_FOREACH(p, &rte_cdx_bus.device_list, next)
+
+#define FOREACH_DRIVER_ON_CDXBUS(p) \
+ RTE_TAILQ_FOREACH(p, &rte_cdx_bus.driver_list, next)
+
+/** Any CDX device identifier (vendor, device) */
+#define RTE_CDX_ANY_ID (0xffff)
+
+#define RTE_PMD_REGISTER_CDX_TABLE(name, table) \
+static const char DRV_EXP_TAG(name, cdx_tbl_export)[] __rte_used = \
+RTE_STR(table)
+
+/**
+ * A structure describing an ID for a CDX driver. Each driver provides a
+ * table of these IDs for each device that it supports.
+ */
+struct rte_cdx_id {
+ uint16_t vendor_id; /**< Vendor ID. */
+ uint16_t device_id; /**< Device ID. */
+};
+
+/**
+ * A structure describing a CDX device.
+ */
+struct rte_cdx_device {
+ RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
+ struct rte_device device; /**< Inherit core device */
+ struct rte_cdx_id id; /**< CDX ID. */
+ struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
+ /**< CDX Memory Resource */
+};
+
+/**
+ * @internal
+ * Helper macro for drivers that need to convert to struct rte_cdx_device.
+ */
+#define RTE_DEV_TO_CDX_DEV(ptr) \
+ container_of(ptr, struct rte_cdx_device, device)
+
+#define RTE_DEV_TO_CDX_DEV_CONST(ptr) \
+ container_of(ptr, const struct rte_cdx_device, device)
+
+#define RTE_ETH_DEV_TO_CDX_DEV(eth_dev) RTE_DEV_TO_CDX_DEV((eth_dev)->device)
+
+#ifdef __cplusplus
+/** C++ macro used to help building up tables of device IDs */
+#define RTE_CDX_DEVICE(vend, dev) \
+ (vend), \
+ (dev)
+#else
+/** Macro used to help building up tables of device IDs */
+#define RTE_CDX_DEVICE(vend, dev) \
+ .vendor_id = (vend), \
+ .device_id = (dev)
+#endif
+
+/**
+ * Initialisation function for the driver called during CDX probing.
+ */
+typedef int (rte_cdx_probe_t)(struct rte_cdx_driver *, struct rte_cdx_device *);
+
+/**
+ * Uninitialisation function for the driver called during hotplugging.
+ */
+typedef int (rte_cdx_remove_t)(struct rte_cdx_device *);
+
+/**
+ * A structure describing a CDX driver.
+ */
+struct rte_cdx_driver {
+ RTE_TAILQ_ENTRY(rte_cdx_driver) next; /**< Next in list. */
+ struct rte_driver driver; /**< Inherit core driver. */
+ struct rte_cdx_bus *bus; /**< CDX bus reference. */
+ rte_cdx_probe_t *probe; /**< Device probe function. */
+ rte_cdx_remove_t *remove; /**< Device remove function. */
+ const struct rte_cdx_id *id_table; /**< ID table, NULL terminated. */
+ uint32_t drv_flags; /**< Flags RTE_CDX_DRV_*. */
+};
+
+/**
+ * Structure describing the CDX bus
+ */
+struct rte_cdx_bus {
+ struct rte_bus bus; /**< Inherit the generic class */
+ struct rte_cdx_device_list device_list; /**< List of CDX devices */
+ struct rte_cdx_driver_list driver_list; /**< List of CDX drivers */
+};
+
+/**
+ * Get Pathname of CDX devices directory.
+ *
+ * @return
+ * sysfs path for CDX devices.
+ */
+__rte_internal
+const char *rte_cdx_get_sysfs_path(void);
+
+/**
+ * Map the CDX device resources in user space virtual memory address
+ *
+ * @param dev
+ * A pointer to a rte_cdx_device structure describing the device
+ * to use
+ *
+ * @return
+ * 0 on success, negative on error and positive if no driver
+ * is found for the device.
+ */
+__rte_internal
+int rte_cdx_map_device(struct rte_cdx_device *dev);
+
+/**
+ * Unmap this device
+ *
+ * @param dev
+ * A pointer to a rte_cdx_device structure describing the device
+ * to use
+ */
+__rte_internal
+void rte_cdx_unmap_device(struct rte_cdx_device *dev);
+
+/**
+ * Dump the content of the CDX bus.
+ *
+ * @param f
+ * A pointer to a file for output
+ */
+__rte_experimental
+void rte_cdx_dump(FILE *f);
+
+/**
+ * Register a CDX driver.
+ *
+ * @param driver
+ * A pointer to a rte_cdx_driver structure describing the driver
+ * to be registered.
+ */
+__rte_internal
+void rte_cdx_register(struct rte_cdx_driver *driver);
+
+/**
+ * Helper for CDX device registration from driver (eth, crypto, raw) instance
+ */
+#define RTE_PMD_REGISTER_CDX(nm, cdx_drv) \
+ RTE_INIT(cdxinitfn_ ##nm) \
+ {\
+ (cdx_drv).driver.name = RTE_STR(nm);\
+ rte_cdx_register(&cdx_drv); \
+ } \
+ RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+/**
+ * Unregister a CDX driver.
+ *
+ * @param driver
+ * A pointer to a rte_cdx_driver structure describing the driver
+ * to be unregistered.
+ */
+__rte_internal
+void rte_cdx_unregister(struct rte_cdx_driver *driver);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_BUS_CDX_H_ */
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
new file mode 100644
index 0000000000..4e8dd24f91
--- /dev/null
+++ b/drivers/bus/cdx/version.map
@@ -0,0 +1,19 @@
+EXPERIMENTAL {
+ global:
+
+ rte_cdx_dump;
+
+ local: *;
+};
+
+INTERNAL {
+ global:
+
+ rte_cdx_get_sysfs_path;
+ rte_cdx_map_device;
+ rte_cdx_register;
+ rte_cdx_unmap_device;
+ rte_cdx_unregister;
+
+ local: *;
+};
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH 2/6] bus/cdx: add dma map and unmap support
2023-04-07 6:01 ` [PATCH 0/6] add support for CDX bus Nipun Gupta
2023-04-07 6:01 ` [PATCH 1/6] bus/cdx: introduce cdx bus Nipun Gupta
@ 2023-04-07 6:01 ` Nipun Gupta
2023-04-07 6:01 ` [PATCH 3/6] bus/cdx: add support for MSI Nipun Gupta
` (4 subsequent siblings)
6 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-04-07 6:01 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
CDX bus can use VFIO interface for mapping and unmapping
of DMA addresses in the IOMMU. This change adds the callback
support for map and unmap APIs as well as fetching the IOMMU
class.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
drivers/bus/cdx/cdx.c | 40 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index c5f8e37a2f..fa6bfe3bc2 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -555,12 +555,52 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
return NULL;
}
+static int
+cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+ struct rte_cdx_device *cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+
+ if (!cdx_dev) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ return rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ (uintptr_t)addr, iova, len);
+}
+
+static int
+cdx_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+ struct rte_cdx_device *cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+
+ if (!cdx_dev) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ return rte_vfio_container_dma_unmap(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ (uintptr_t)addr, iova, len);
+}
+
+static enum rte_iova_mode
+rte_cdx_get_iommu_class(void)
+{
+ if (TAILQ_EMPTY(&rte_cdx_bus.device_list))
+ return RTE_IOVA_DC;
+
+ return RTE_IOVA_VA;
+}
+
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
.scan = rte_cdx_scan,
.probe = cdx_probe,
.find_device = cdx_find_device,
.parse = cdx_parse,
+ .dma_map = cdx_dma_map,
+ .dma_unmap = cdx_dma_unmap,
+ .get_iommu_class = rte_cdx_get_iommu_class,
},
.device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
.driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH 3/6] bus/cdx: add support for MSI
2023-04-07 6:01 ` [PATCH 0/6] add support for CDX bus Nipun Gupta
2023-04-07 6:01 ` [PATCH 1/6] bus/cdx: introduce cdx bus Nipun Gupta
2023-04-07 6:01 ` [PATCH 2/6] bus/cdx: add dma map and unmap support Nipun Gupta
@ 2023-04-07 6:01 ` Nipun Gupta
2023-04-07 6:01 ` [PATCH 4/6] bus/cdx: support plug unplug and dev iterator Nipun Gupta
` (3 subsequent siblings)
6 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-04-07 6:01 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
MSI's are exposed to the devices using VFIO (vfio-cdx). This
patch uses the same to add support for MSI for the devices on
the cdx bus.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
drivers/bus/cdx/cdx.c | 11 ++
drivers/bus/cdx/cdx_vfio.c | 182 ++++++++++++++++++++++++-
drivers/bus/cdx/rte_bus_cdx.h | 25 ++++
drivers/bus/cdx/version.map | 2 +
lib/eal/common/eal_common_interrupts.c | 21 +++
lib/eal/common/eal_interrupts.h | 1 +
lib/eal/include/rte_interrupts.h | 32 +++++
lib/eal/version.map | 2 +
8 files changed, 274 insertions(+), 2 deletions(-)
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index fa6bfe3bc2..9732a32eb1 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -225,6 +225,15 @@ cdx_scan_one(const char *dirname, const char *dev_name)
goto err;
}
+ /* Allocate interrupt instance for cdx device */
+ dev->intr_handle =
+ rte_intr_instance_alloc(RTE_INTR_INSTANCE_F_PRIVATE);
+ if (dev->intr_handle == NULL) {
+ CDX_BUS_ERR("Failed to create interrupt instance for %s\n",
+ dev->device.name);
+ return -ENOMEM;
+ }
+
/*
* Check if device is bound to 'vfio-cdx' driver, so that user-space
* can gracefully access the device.
@@ -416,6 +425,8 @@ rte_cdx_probe_one_driver(struct rte_cdx_driver *dr,
return ret;
error_probe:
+ rte_intr_instance_free(dev->intr_handle);
+ dev->intr_handle = NULL;
cdx_vfio_unmap_resource(dev);
error_map_device:
return ret;
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index a53137c416..93c2ad0221 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -60,6 +60,10 @@ struct mapped_cdx_resource {
/** mapped cdx device list */
TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
+/* irq set buffer length for MSI interrupts */
+#define MSI_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \
+ sizeof(int) * (RTE_MAX_RXTX_INTR_VEC_ID + 1))
+
static struct rte_tailq_elem cdx_vfio_tailq = {
.name = "VFIO_CDX_RESOURCE_LIST",
};
@@ -108,6 +112,27 @@ cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
char cdx_addr[PATH_MAX] = {0};
struct mapped_cdx_resource *vfio_res = NULL;
struct mapped_cdx_res_list *vfio_res_list;
+ int ret, vfio_dev_fd;
+
+ if (rte_intr_fd_get(dev->intr_handle) < 0)
+ return -1;
+
+ if (close(rte_intr_fd_get(dev->intr_handle)) < 0) {
+ CDX_BUS_ERR("Error when closing eventfd file descriptor for %s",
+ dev->device.name);
+ return -1;
+ }
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ ret = rte_vfio_release_device(rte_cdx_get_sysfs_path(), dev->device.name,
+ vfio_dev_fd);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot release VFIO device");
+ return ret;
+ }
vfio_res_list =
RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
@@ -130,6 +155,18 @@ cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
{
struct mapped_cdx_resource *vfio_res = NULL;
struct mapped_cdx_res_list *vfio_res_list;
+ int ret, vfio_dev_fd;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ ret = rte_vfio_release_device(rte_cdx_get_sysfs_path(), dev->device.name,
+ vfio_dev_fd);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot release VFIO device");
+ return ret;
+ }
vfio_res_list =
RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
@@ -154,9 +191,80 @@ cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
return cdx_vfio_unmap_resource_secondary(dev);
}
+/* set up interrupt support (but not enable interrupts) */
static int
-cdx_rte_vfio_setup_device(int vfio_dev_fd)
+cdx_vfio_setup_interrupts(struct rte_cdx_device *dev, int vfio_dev_fd,
+ int num_irqs)
{
+ int i, ret;
+
+ if (num_irqs == 0)
+ return 0;
+
+ /* start from MSI interrupt type */
+ for (i = 0; i < num_irqs; i++) {
+ struct vfio_irq_info irq = { .argsz = sizeof(irq) };
+ int fd = -1;
+
+ irq.index = i;
+
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_IRQ_INFO, &irq);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot get VFIO IRQ info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /* if this vector cannot be used with eventfd, fail if we explicitly
+ * specified interrupt type, otherwise continue
+ */
+ if ((irq.flags & VFIO_IRQ_INFO_EVENTFD) == 0)
+ continue;
+
+ if (rte_intr_irq_count_set(dev->intr_handle, irq.count))
+ return -1;
+
+ /* Reallocate the efds and elist fields of intr_handle based
+ * on CDX device MSI size.
+ */
+ if ((uint32_t)rte_intr_nb_intr_get(dev->intr_handle) < irq.count &&
+ rte_intr_event_list_update(dev->intr_handle, irq.count))
+ return -1;
+
+ /* set up an eventfd for interrupts */
+ fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
+ if (fd < 0) {
+ CDX_BUS_ERR("Cannot set up eventfd, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ if (rte_intr_fd_set(dev->intr_handle, fd))
+ return -1;
+
+ /* DPDK CDX bus currently supports only MSI-X */
+ if (rte_intr_type_set(dev->intr_handle, RTE_INTR_HANDLE_VFIO_MSIX))
+ return -1;
+
+ if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
+ return -1;
+
+ return 0;
+ }
+
+ /* if we're here, we haven't found a suitable interrupt vector */
+ return -1;
+}
+
+static int
+cdx_rte_vfio_setup_device(struct rte_cdx_device *dev, int vfio_dev_fd,
+ int num_irqs)
+{
+ if (cdx_vfio_setup_interrupts(dev, vfio_dev_fd, num_irqs) != 0) {
+ CDX_BUS_ERR("Error setting up interrupts!");
+ return -1;
+ }
+
/*
* Reset the device. If the device is not capable of resetting,
* then it updates errno as EINVAL.
@@ -271,6 +379,9 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
struct cdx_map *maps;
int vfio_dev_fd, i, ret;
+ if (rte_intr_fd_set(dev->intr_handle, -1))
+ return -1;
+
ret = rte_vfio_setup_device(rte_cdx_get_sysfs_path(), dev_name,
&vfio_dev_fd, &device_info);
if (ret)
@@ -336,7 +447,7 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
free(reg);
}
- if (cdx_rte_vfio_setup_device(vfio_dev_fd) < 0) {
+ if (cdx_rte_vfio_setup_device(dev, vfio_dev_fd, device_info.num_irqs) < 0) {
CDX_BUS_ERR("%s setup device failed", dev_name);
goto err_vfio_res;
}
@@ -366,6 +477,9 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
const char *dev_name = dev->device.name;
struct cdx_map *maps;
+ if (rte_intr_fd_set(dev->intr_handle, -1))
+ return -1;
+
/* if we're in a secondary process, just find our tailq entry */
TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
if (strcmp(vfio_res->name, dev_name))
@@ -399,6 +513,10 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
dev->mem_resource[i].len = maps[i].size;
}
+ /* we need save vfio_dev_fd, so it can be used during release */
+ if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
+ goto err_vfio_dev_fd;
+
return 0;
err_vfio_dev_fd:
rte_vfio_release_device(rte_cdx_get_sysfs_path(),
@@ -418,3 +536,63 @@ cdx_vfio_map_resource(struct rte_cdx_device *dev)
else
return cdx_vfio_map_resource_secondary(dev);
}
+
+int
+rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle)
+{
+ char irq_set_buf[MSI_IRQ_SET_BUF_LEN];
+ struct vfio_irq_set *irq_set;
+ int *fd_ptr, vfio_dev_fd, i;
+ int ret;
+
+ irq_set = (struct vfio_irq_set *) irq_set_buf;
+ irq_set->count = rte_intr_irq_count_get(intr_handle);
+ irq_set->argsz = sizeof(struct vfio_irq_set) +
+ (sizeof(int) * irq_set->count);
+
+ irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER;
+ irq_set->index = 0;
+ irq_set->start = 0;
+ fd_ptr = (int *) &irq_set->data;
+
+ for (i = 0; i < rte_intr_nb_efd_get(intr_handle); i++)
+ fd_ptr[i] = rte_intr_efds_index_get(intr_handle, i);
+
+ vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
+
+ if (ret) {
+ CDX_BUS_ERR("Error enabling MSI interrupts for fd %d",
+ rte_intr_fd_get(intr_handle));
+ return -1;
+ }
+
+ return 0;
+}
+
+/* disable MSI interrupts */
+int
+rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle)
+{
+ struct vfio_irq_set *irq_set;
+ char irq_set_buf[MSI_IRQ_SET_BUF_LEN];
+ int len, ret, vfio_dev_fd;
+
+ len = sizeof(struct vfio_irq_set);
+
+ irq_set = (struct vfio_irq_set *) irq_set_buf;
+ irq_set->argsz = len;
+ irq_set->count = 0;
+ irq_set->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_TRIGGER;
+ irq_set->index = 0;
+ irq_set->start = 0;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
+
+ if (ret)
+ CDX_BUS_ERR("Error disabling MSI interrupts for fd %d",
+ rte_intr_fd_get(intr_handle));
+
+ return ret;
+}
diff --git a/drivers/bus/cdx/rte_bus_cdx.h b/drivers/bus/cdx/rte_bus_cdx.h
index 2ebe2a0dcc..a6170347dd 100644
--- a/drivers/bus/cdx/rte_bus_cdx.h
+++ b/drivers/bus/cdx/rte_bus_cdx.h
@@ -81,6 +81,7 @@ struct rte_cdx_device {
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
/**< CDX Memory Resource */
+ struct rte_intr_handle *intr_handle; /**< Interrupt handle */
};
/**
@@ -181,6 +182,30 @@ void rte_cdx_unmap_device(struct rte_cdx_device *dev);
__rte_experimental
void rte_cdx_dump(FILE *f);
+/**
+ * Enables VFIO Interrupts for CDX bus devices.
+ *
+ * @param intr_handle
+ * Pointer to the interrupt handle.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle);
+
+/**
+ * Disable VFIO Interrupts for CDX bus devices.
+ *
+ * @param intr_handle
+ * Pointer to the interrupt handle.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle);
+
/**
* Register a CDX driver.
*
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
index 4e8dd24f91..b35bf8481f 100644
--- a/drivers/bus/cdx/version.map
+++ b/drivers/bus/cdx/version.map
@@ -14,6 +14,8 @@ INTERNAL {
rte_cdx_register;
rte_cdx_unmap_device;
rte_cdx_unregister;
+ rte_cdx_vfio_intr_disable;
+ rte_cdx_vfio_intr_enable;
local: *;
};
diff --git a/lib/eal/common/eal_common_interrupts.c b/lib/eal/common/eal_common_interrupts.c
index 97b64fed58..a0167d9ad4 100644
--- a/lib/eal/common/eal_common_interrupts.c
+++ b/lib/eal/common/eal_common_interrupts.c
@@ -398,6 +398,27 @@ int rte_intr_elist_index_set(struct rte_intr_handle *intr_handle,
return -rte_errno;
}
+int rte_intr_irq_count_set(struct rte_intr_handle *intr_handle,
+ int irq_count)
+{
+ CHECK_VALID_INTR_HANDLE(intr_handle);
+
+ intr_handle->irq_count = irq_count;
+
+ return 0;
+fail:
+ return -rte_errno;
+}
+
+int rte_intr_irq_count_get(const struct rte_intr_handle *intr_handle)
+{
+ CHECK_VALID_INTR_HANDLE(intr_handle);
+
+ return intr_handle->irq_count;
+fail:
+ return -rte_errno;
+}
+
int rte_intr_vec_list_alloc(struct rte_intr_handle *intr_handle,
const char *name, int size)
{
diff --git a/lib/eal/common/eal_interrupts.h b/lib/eal/common/eal_interrupts.h
index 482781b862..237f471a76 100644
--- a/lib/eal/common/eal_interrupts.h
+++ b/lib/eal/common/eal_interrupts.h
@@ -16,6 +16,7 @@ struct rte_intr_handle {
};
uint32_t alloc_flags; /**< flags passed at allocation */
enum rte_intr_handle_type type; /**< handle type */
+ uint32_t irq_count; /**< IRQ count provided via VFIO */
uint32_t max_intr; /**< max interrupt requested */
uint32_t nb_efd; /**< number of available efd(event fd) */
uint8_t efd_counter_size; /**< size of efd counter, used for vdev */
diff --git a/lib/eal/include/rte_interrupts.h b/lib/eal/include/rte_interrupts.h
index 487e3c8875..bc477a483f 100644
--- a/lib/eal/include/rte_interrupts.h
+++ b/lib/eal/include/rte_interrupts.h
@@ -506,6 +506,38 @@ __rte_internal
int
rte_intr_max_intr_get(const struct rte_intr_handle *intr_handle);
+/**
+ * @internal
+ * Set the irq count field of interrupt handle with user
+ * provided irq count value.
+ *
+ * @param intr_handle
+ * pointer to the interrupt handle.
+ * @param irq_count
+ * IRQ count
+ *
+ * @return
+ * - On success, zero.
+ * - On failure, a negative value and rte_errno is set.
+ */
+__rte_internal
+int
+rte_intr_irq_count_set(struct rte_intr_handle *intr_handle, int irq_count);
+
+/**
+ * @internal
+ * Returns the irq count field of the given interrupt handle instance.
+ *
+ * @param intr_handle
+ * pointer to the interrupt handle.
+ *
+ * @return
+ * - On success, ir count.
+ * - On failure, a negative value and rte_errno is set.
+ */
+__rte_internal
+int rte_intr_irq_count_get(const struct rte_intr_handle *intr_handle);
+
/**
* @internal
* Set the number of event fd field of interrupt handle
diff --git a/lib/eal/version.map b/lib/eal/version.map
index 6d6978f108..14bf7ade77 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -458,6 +458,8 @@ INTERNAL {
rte_intr_instance_dup;
rte_intr_instance_windows_handle_get;
rte_intr_instance_windows_handle_set;
+ rte_intr_irq_count_get;
+ rte_intr_irq_count_set;
rte_intr_max_intr_get;
rte_intr_max_intr_set;
rte_intr_nb_efd_get;
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH 4/6] bus/cdx: support plug unplug and dev iterator
2023-04-07 6:01 ` [PATCH 0/6] add support for CDX bus Nipun Gupta
` (2 preceding siblings ...)
2023-04-07 6:01 ` [PATCH 3/6] bus/cdx: add support for MSI Nipun Gupta
@ 2023-04-07 6:01 ` Nipun Gupta
2023-04-07 6:01 ` [PATCH 5/6] bus: enable cdx bus Nipun Gupta
` (2 subsequent siblings)
6 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-04-07 6:01 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
This change adds support for plugging and unplugging
CDX devices on the CDX bus. Also, CDX dev iterator support
has been added for the CDX bus.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
drivers/bus/cdx/cdx.c | 142 +++++++++++++++++++++++++++++++---
drivers/bus/cdx/rte_bus_cdx.h | 1 +
2 files changed, 133 insertions(+), 10 deletions(-)
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index 9732a32eb1..8c72de9fe4 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -69,6 +69,7 @@
#include <rte_eal_paging.h>
#include <rte_errno.h>
#include <rte_devargs.h>
+#include <rte_kvargs.h>
#include <rte_malloc.h>
#include <rte_memcpy.h>
#include <rte_vfio.h>
@@ -82,6 +83,15 @@
#define CDX_BUS_NAME cdx
#define CDX_DEV_PREFIX "cdx-"
+enum cdx_params {
+ RTE_CDX_PARAM_NAME,
+};
+
+static const char * const cdx_params_keys[] = {
+ [RTE_CDX_PARAM_NAME] = "name",
+ NULL,
+};
+
/**
* @file
* CDX probing using Linux sysfs.
@@ -89,7 +99,7 @@
/* Add a device to CDX bus */
static void
-rte_cdx_add_device(struct rte_cdx_device *cdx_dev)
+cdx_add_device(struct rte_cdx_device *cdx_dev)
{
TAILQ_INSERT_TAIL(&rte_cdx_bus.device_list, cdx_dev, next);
}
@@ -259,7 +269,7 @@ cdx_scan_one(const char *dirname, const char *dev_name)
}
dev->id.device_id = (uint16_t)tmp;
- rte_cdx_add_device(dev);
+ cdx_add_device(dev);
return 0;
@@ -276,7 +286,7 @@ cdx_scan_one(const char *dirname, const char *dev_name)
* list.
*/
static int
-rte_cdx_scan(void)
+cdx_scan(void)
{
struct dirent *e;
DIR *dir;
@@ -356,7 +366,7 @@ cdx_unmap_resource(void *requested_addr, size_t size)
* Match the CDX Driver and Device using device id and vendor id.
*/
static int
-rte_cdx_match(const struct rte_cdx_driver *cdx_drv,
+cdx_match(const struct rte_cdx_driver *cdx_drv,
const struct rte_cdx_device *cdx_dev)
{
const struct rte_cdx_id *id_table;
@@ -382,7 +392,7 @@ rte_cdx_match(const struct rte_cdx_driver *cdx_drv,
* driver.
*/
static int
-rte_cdx_probe_one_driver(struct rte_cdx_driver *dr,
+cdx_probe_one_driver(struct rte_cdx_driver *dr,
struct rte_cdx_device *dev)
{
const char *dev_name = dev->device.name;
@@ -393,7 +403,7 @@ rte_cdx_probe_one_driver(struct rte_cdx_driver *dr,
return -EINVAL;
/* The device is not blocked; Check if driver supports it */
- if (!rte_cdx_match(dr, dev))
+ if (!cdx_match(dr, dev))
/* Match of device and driver failed */
return 1;
@@ -421,6 +431,7 @@ rte_cdx_probe_one_driver(struct rte_cdx_driver *dr,
} else {
dev->device.driver = &dr->driver;
}
+ dev->driver = dr;
return ret;
@@ -447,7 +458,7 @@ cdx_probe_all_drivers(struct rte_cdx_device *dev)
return -EINVAL;
FOREACH_DRIVER_ON_CDXBUS(dr) {
- rc = rte_cdx_probe_one_driver(dr, dev);
+ rc = cdx_probe_one_driver(dr, dev);
if (rc < 0)
/* negative value is an error */
return rc;
@@ -566,6 +577,71 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
return NULL;
}
+/* Remove a device from CDX bus */
+static void
+cdx_remove_device(struct rte_cdx_device *cdx_dev)
+{
+ TAILQ_REMOVE(&rte_cdx_bus.device_list, cdx_dev, next);
+}
+
+/*
+ * If vendor/device ID match, call the remove() function of the
+ * driver.
+ */
+static int
+cdx_detach_dev(struct rte_cdx_device *dev)
+{
+ struct rte_cdx_driver *dr;
+ int ret = 0;
+
+ if (dev == NULL)
+ return -EINVAL;
+
+ dr = dev->driver;
+
+ CDX_BUS_DEBUG("detach device %s using driver: %s",
+ dev->device.name, dr->driver.name);
+
+ if (dr->remove) {
+ ret = dr->remove(dev);
+ if (ret < 0)
+ return ret;
+ }
+
+ /* clear driver structure */
+ dev->driver = NULL;
+ dev->device.driver = NULL;
+
+ rte_cdx_unmap_device(dev);
+
+ rte_intr_instance_free(dev->intr_handle);
+ dev->intr_handle = NULL;
+
+ return 0;
+}
+
+static int
+cdx_plug(struct rte_device *dev)
+{
+ return cdx_probe_all_drivers(RTE_DEV_TO_CDX_DEV(dev));
+}
+
+static int
+cdx_unplug(struct rte_device *dev)
+{
+ struct rte_cdx_device *cdx_dev;
+ int ret;
+
+ cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+ ret = cdx_detach_dev(cdx_dev);
+ if (ret == 0) {
+ cdx_remove_device(cdx_dev);
+ rte_devargs_remove(dev->devargs);
+ free(cdx_dev);
+ }
+ return ret;
+}
+
static int
cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
{
@@ -595,7 +671,7 @@ cdx_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
}
static enum rte_iova_mode
-rte_cdx_get_iommu_class(void)
+cdx_get_iommu_class(void)
{
if (TAILQ_EMPTY(&rte_cdx_bus.device_list))
return RTE_IOVA_DC;
@@ -603,15 +679,61 @@ rte_cdx_get_iommu_class(void)
return RTE_IOVA_VA;
}
+static int
+cdx_dev_match(const struct rte_device *dev,
+ const void *_kvlist)
+{
+ const struct rte_kvargs *kvlist = _kvlist;
+ const char *key = cdx_params_keys[RTE_CDX_PARAM_NAME];
+ const char *name;
+
+ /* no kvlist arg, all devices match */
+ if (kvlist == NULL)
+ return 0;
+
+ /* if key is present in kvlist and does not match, filter device */
+ name = rte_kvargs_get(kvlist, key);
+ if (name != NULL && strcmp(name, dev->name))
+ return -1;
+
+ return 0;
+}
+
+static void *
+cdx_dev_iterate(const void *start,
+ const char *str,
+ const struct rte_dev_iterator *it __rte_unused)
+{
+ rte_bus_find_device_t find_device;
+ struct rte_kvargs *kvargs = NULL;
+ struct rte_device *dev;
+
+ if (str != NULL) {
+ kvargs = rte_kvargs_parse(str, cdx_params_keys);
+ if (kvargs == NULL) {
+ CDX_BUS_ERR("cannot parse argument list %s", str);
+ rte_errno = EINVAL;
+ return NULL;
+ }
+ }
+ find_device = rte_cdx_bus.bus.find_device;
+ dev = find_device(start, cdx_dev_match, kvargs);
+ rte_kvargs_free(kvargs);
+ return dev;
+}
+
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
- .scan = rte_cdx_scan,
+ .scan = cdx_scan,
.probe = cdx_probe,
.find_device = cdx_find_device,
+ .plug = cdx_plug,
+ .unplug = cdx_unplug,
.parse = cdx_parse,
.dma_map = cdx_dma_map,
.dma_unmap = cdx_dma_unmap,
- .get_iommu_class = rte_cdx_get_iommu_class,
+ .get_iommu_class = cdx_get_iommu_class,
+ .dev_iterate = cdx_dev_iterate,
},
.device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
.driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
diff --git a/drivers/bus/cdx/rte_bus_cdx.h b/drivers/bus/cdx/rte_bus_cdx.h
index a6170347dd..a715dee50b 100644
--- a/drivers/bus/cdx/rte_bus_cdx.h
+++ b/drivers/bus/cdx/rte_bus_cdx.h
@@ -78,6 +78,7 @@ struct rte_cdx_id {
struct rte_cdx_device {
RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
struct rte_device device; /**< Inherit core device */
+ struct rte_cdx_driver *driver; /**< CDX driver used in probing */
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
/**< CDX Memory Resource */
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH 5/6] bus: enable cdx bus
2023-04-07 6:01 ` [PATCH 0/6] add support for CDX bus Nipun Gupta
` (3 preceding siblings ...)
2023-04-07 6:01 ` [PATCH 4/6] bus/cdx: support plug unplug and dev iterator Nipun Gupta
@ 2023-04-07 6:01 ` Nipun Gupta
2023-04-07 6:01 ` [PATCH 6/6] config/arm: add AMD CDX Nipun Gupta
2023-04-07 7:18 ` [PATCH 0/6] add support for CDX bus David Marchand
6 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-04-07 6:01 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
enable the compilation of cdx bus
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
drivers/bus/meson.build | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
index 6d2520c543..a78b4283bf 100644
--- a/drivers/bus/meson.build
+++ b/drivers/bus/meson.build
@@ -3,6 +3,7 @@
drivers = [
'auxiliary',
+ 'cdx',
'dpaa',
'fslmc',
'ifpga',
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH 6/6] config/arm: add AMD CDX
2023-04-07 6:01 ` [PATCH 0/6] add support for CDX bus Nipun Gupta
` (4 preceding siblings ...)
2023-04-07 6:01 ` [PATCH 5/6] bus: enable cdx bus Nipun Gupta
@ 2023-04-07 6:01 ` Nipun Gupta
2023-04-07 7:18 ` [PATCH 0/6] add support for CDX bus David Marchand
6 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-04-07 6:01 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
Adding support for AMD CDX devices
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
config/arm/arm64_cdx_linux_gcc | 17 +++++++++++++++++
config/arm/meson.build | 14 ++++++++++++++
2 files changed, 31 insertions(+)
create mode 100644 config/arm/arm64_cdx_linux_gcc
diff --git a/config/arm/arm64_cdx_linux_gcc b/config/arm/arm64_cdx_linux_gcc
new file mode 100644
index 0000000000..8e6d619dae
--- /dev/null
+++ b/config/arm/arm64_cdx_linux_gcc
@@ -0,0 +1,17 @@
+[binaries]
+c = ['ccache', 'aarch64-linux-gnu-gcc']
+cpp = ['ccache', 'aarch64-linux-gnu-g++']
+ar = 'aarch64-linux-gnu-ar'
+as = 'aarch64-linux-gnu-as'
+strip = 'aarch64-linux-gnu-strip'
+pkgconfig = 'aarch64-linux-gnu-pkg-config'
+pcap-config = ''
+
+[host_machine]
+system = 'linux'
+cpu_family = 'aarch64'
+cpu = 'armv8-a'
+endian = 'little'
+
+[properties]
+platform = 'cdx'
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 5213434ca4..39b8929534 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -305,6 +305,18 @@ soc_bluefield = {
'numa': false
}
+soc_cdx = {
+ 'description': 'AMD CDX',
+ 'implementer': '0x41',
+ 'part_number': '0xd42',
+ 'flags': [
+ ['RTE_MACHINE', '"cdx"'],
+ ['RTE_MAX_LCORE', 16],
+ ['RTE_MAX_NUMA_NODES', 1]
+ ],
+ 'numa': false
+}
+
soc_centriq2400 = {
'description': 'Qualcomm Centriq 2400',
'implementer': '0x51',
@@ -463,6 +475,7 @@ generic_aarch32: Generic un-optimized build for armv8 aarch32 execution mode.
armada: Marvell ARMADA
bluefield: NVIDIA BlueField
bluefield3: NVIDIA BlueField-3
+cdx: AMD CDX
centriq2400: Qualcomm Centriq 2400
cn9k: Marvell OCTEON 9
cn10k: Marvell OCTEON 10
@@ -490,6 +503,7 @@ socs = {
'armada': soc_armada,
'bluefield': soc_bluefield,
'bluefield3': soc_bluefield3,
+ 'cdx': soc_cdx,
'centriq2400': soc_centriq2400,
'cn9k': soc_cn9k,
'cn10k' : soc_cn10k,
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH 0/6] add support for CDX bus
2023-04-07 6:01 ` [PATCH 0/6] add support for CDX bus Nipun Gupta
` (5 preceding siblings ...)
2023-04-07 6:01 ` [PATCH 6/6] config/arm: add AMD CDX Nipun Gupta
@ 2023-04-07 7:18 ` David Marchand
2023-04-07 7:29 ` Nipun Gupta
6 siblings, 1 reply; 102+ messages in thread
From: David Marchand @ 2023-04-07 7:18 UTC (permalink / raw)
To: Nipun Gupta; +Cc: dev, thomas, ferruh.yigit, harpreet.anand, nikhil.agarwal
Hello,
On Fri, Apr 7, 2023 at 8:02 AM Nipun Gupta <nipun.gupta@amd.com> wrote:
>
> Support AMD CDX bus, for FPGA based CDX devices. The CDX
> devices are memory mapped on system bus for embedded CPUs.
>
> It uses sysfs interface and the vfio-cdx driver to discover
> and initialize the CDX devices.
>
> The patches are intended for DPDK 23.07 release, and have been sent
> as an RFC as patches are yet to be merged in Linux.
>
> Linux CDX bus patches has been added into linux next:
> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/cdx
>
> VFIO patches are also submitted in upstream:
> https://www.spinics.net/lists/kvm/msg310623.html
Hard to tell just from this link what the status is.
Has it been reviewed?
When are you expecting this to get merged?
>
> CDX is a Hardware Architecture designed for AMD FPGA devices. It
> consists of mechanism for interaction between FPGA, Firmware and
> the APUs (Application CPUs).
> Firmware resides on RPU (Realtime CPUs) which interacts with
> the FPGA program manager and the APUs. The RPU provides memory-mapped
> interface (RPU if) which is used to communicate with APUs.
>
> VFIO CDX driver provides the CDX device resources like MMIO and interrupts
> to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
> driver to discover and initialize the CDX devices for user-space
> applications.
>
> RFC patches in DPDK were submitted at:
> http://patches.dpdk.org/project/dpdk/patch/20230124140746.594066-2-nipun.gupta@amd.com/
>
> Changes RFC->v1:
> - Marked few API's as internal which were not required
> to be provided to user.
Just a quick pass on this series.
- There is an issue with the exported header rte_bus_cdx.h.
All the bus specific / driver only parts must be moved to a dedicated
bus_cdx_driver.h header (which must be listed in driver_sdk_headers).
- To whom is the rte_cdx_dump symbol intended to? Is this for use by
user applications?
If so, the rte_bus_cdx.h can be kept, to export this symbol.
But otherwise, everything in rte_bus_cdx.h can be moved to bus_cdx_driver.h.
- Please remove the rte_ prefix for static symbols in *.c.
--
David Marchand
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH 0/6] add support for CDX bus
2023-04-07 7:18 ` [PATCH 0/6] add support for CDX bus David Marchand
@ 2023-04-07 7:29 ` Nipun Gupta
2023-04-13 13:25 ` Gupta, Nipun
0 siblings, 1 reply; 102+ messages in thread
From: Nipun Gupta @ 2023-04-07 7:29 UTC (permalink / raw)
To: David Marchand; +Cc: dev, thomas, ferruh.yigit, harpreet.anand, nikhil.agarwal
On 4/7/2023 12:48 PM, David Marchand wrote:
> Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
>
>
> Hello,
>
> On Fri, Apr 7, 2023 at 8:02 AM Nipun Gupta <nipun.gupta@amd.com> wrote:
>>
>> Support AMD CDX bus, for FPGA based CDX devices. The CDX
>> devices are memory mapped on system bus for embedded CPUs.
>>
>> It uses sysfs interface and the vfio-cdx driver to discover
>> and initialize the CDX devices.
>>
>> The patches are intended for DPDK 23.07 release, and have been sent
>> as an RFC as patches are yet to be merged in Linux.
>>
>> Linux CDX bus patches has been added into linux next:
>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/cdx
>>
>> VFIO patches are also submitted in upstream:
>> https://www.spinics.net/lists/kvm/msg310623.html
>
> Hard to tell just from this link what the status is.
> Has it been reviewed?
> When are you expecting this to get merged?
The CDX bus code has been merged and VFIO code is under review. Apart
from this, we will soon have Open source Linux from AMD which has all
the required patches for CDX in a week or so (I will provide the link
once that is available).
>
>
>>
>> CDX is a Hardware Architecture designed for AMD FPGA devices. It
>> consists of mechanism for interaction between FPGA, Firmware and
>> the APUs (Application CPUs).
>> Firmware resides on RPU (Realtime CPUs) which interacts with
>> the FPGA program manager and the APUs. The RPU provides memory-mapped
>> interface (RPU if) which is used to communicate with APUs.
>>
>> VFIO CDX driver provides the CDX device resources like MMIO and interrupts
>> to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
>> driver to discover and initialize the CDX devices for user-space
>> applications.
>>
>> RFC patches in DPDK were submitted at:
>> http://patches.dpdk.org/project/dpdk/patch/20230124140746.594066-2-nipun.gupta@amd.com/
>>
>> Changes RFC->v1:
>> - Marked few API's as internal which were not required
>> to be provided to user.
>
> Just a quick pass on this series.
>
> - There is an issue with the exported header rte_bus_cdx.h.
> All the bus specific / driver only parts must be moved to a dedicated
> bus_cdx_driver.h header (which must be listed in driver_sdk_headers).
>
> - To whom is the rte_cdx_dump symbol intended to? Is this for use by
> user applications?
> If so, the rte_bus_cdx.h can be kept, to export this symbol.
> But otherwise, everything in rte_bus_cdx.h can be moved to bus_cdx_driver.h.
>
> - Please remove the rte_ prefix for static symbols in *.c.
Agree to this.
Regards,
Nipun
>
>
> --
> David Marchand
>
^ permalink raw reply [flat|nested] 102+ messages in thread
* RE: [PATCH 0/6] add support for CDX bus
2023-04-07 7:29 ` Nipun Gupta
@ 2023-04-13 13:25 ` Gupta, Nipun
0 siblings, 0 replies; 102+ messages in thread
From: Gupta, Nipun @ 2023-04-13 13:25 UTC (permalink / raw)
To: David Marchand
Cc: dev, thomas, Yigit, Ferruh, Anand, Harpreet, Agarwal, Nikhil
> -----Original Message-----
> From: Gupta, Nipun
> Sent: Friday, April 7, 2023 1:00 PM
> To: David Marchand <david.marchand@redhat.com>
> Cc: dev@dpdk.org; thomas@monjalon.net; Yigit, Ferruh
> <Ferruh.Yigit@amd.com>; Anand, Harpreet <harpreet.anand@amd.com>;
> Agarwal, Nikhil <nikhil.agarwal@amd.com>
> Subject: Re: [PATCH 0/6] add support for CDX bus
>
>
>
> On 4/7/2023 12:48 PM, David Marchand wrote:
> > Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
> >
> >
> > Hello,
> >
> > On Fri, Apr 7, 2023 at 8:02 AM Nipun Gupta <nipun.gupta@amd.com> wrote:
> >>
> >> Support AMD CDX bus, for FPGA based CDX devices. The CDX
> >> devices are memory mapped on system bus for embedded CPUs.
> >>
> >> It uses sysfs interface and the vfio-cdx driver to discover
> >> and initialize the CDX devices.
> >>
> >> The patches are intended for DPDK 23.07 release, and have been sent
> >> as an RFC as patches are yet to be merged in Linux.
> >>
> >> Linux CDX bus patches has been added into linux next:
> >> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-
> next.git/tree/drivers/cdx
> >>
> >> VFIO patches are also submitted in upstream:
> >> https://www.spinics.net/lists/kvm/msg310623.html
> >
> > Hard to tell just from this link what the status is.
> > Has it been reviewed?
> > When are you expecting this to get merged?
>
> The CDX bus code has been merged and VFIO code is under review. Apart
> from this, we will soon have Open source Linux from AMD which has all
> the required patches for CDX in a week or so (I will provide the link
> once that is available).
The CDX bus and VFIO support is now available at Xilinx open-source tree:
https://github.com/Xilinx/linux-xlnx (drivers/cdx/ and drivers/vfio/cdx)
Regards,
Nipun
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v2 0/6] add support for CDX bus
2023-01-24 14:07 [RFC PATCH 0/6] add support for CDX bus Nipun Gupta
` (6 preceding siblings ...)
2023-04-07 6:01 ` [PATCH 0/6] add support for CDX bus Nipun Gupta
@ 2023-04-13 13:26 ` Nipun Gupta
2023-04-13 13:26 ` [PATCH v2 1/6] bus/cdx: introduce cdx bus Nipun Gupta
` (6 more replies)
2023-04-21 14:54 ` [PATCH v3 0/5] Support AMD CDX bus, for FPGA based CDX devices. The CDX Nipun Gupta
` (5 subsequent siblings)
13 siblings, 7 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-04-13 13:26 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
Support AMD CDX bus, for FPGA based CDX devices. The CDX
devices are memory mapped on system bus for embedded CPUs.
It uses sysfs interface and the vfio-cdx driver to discover
and initialize the CDX devices.
The patches are intended for DPDK 23.07 release, and have been sent
as an RFC as patches are yet to be merged in Linux.
The CDX bus and VFIO support is available at Xilinx open source tree:
https://github.com/Xilinx/linux-xlnx (drivers/cdx/ and drivers/vfio/cdx)
Linux CDX bus patches has been added into linux next:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/cdx
VFIO patches are also submitted in upstream:
https://www.spinics.net/lists/kvm/msg310623.html
CDX is a Hardware Architecture designed for AMD FPGA devices. It
consists of mechanism for interaction between FPGA, Firmware and
the APUs (Application CPUs).
Firmware resides on RPU (Realtime CPUs) which interacts with
the FPGA program manager and the APUs. The RPU provides memory-mapped
interface (RPU if) which is used to communicate with APUs.
VFIO CDX driver provides the CDX device resources like MMIO and interrupts
to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
driver to discover and initialize the CDX devices for user-space
applications.
Changes v1->v2:
- Moved file rte_cdx_bus.h to internal bus_cdx_driver.h
and added this file to deivce_cdx_headers
- Moved cdx.h to private.h
- Removed rte_ prefix from the static symbols in .c files.
Changes RFC->v1:
- Marked few API's as internal which were not required
to be provided to user.
Nipun Gupta (6):
bus/cdx: introduce cdx bus
bus/cdx: add dma map and unmap support
bus/cdx: add support for MSI
bus/cdx: support plug unplug and dev iterator
bus: enable cdx bus
config/arm: add AMD CDX
MAINTAINERS | 5 +
config/arm/arm64_cdx_linux_gcc | 17 +
config/arm/meson.build | 14 +
drivers/bus/cdx/bus_cdx_driver.h | 227 ++++++++
drivers/bus/cdx/cdx.c | 694 +++++++++++++++++++++++++
drivers/bus/cdx/cdx_logs.h | 37 ++
drivers/bus/cdx/cdx_vfio.c | 619 ++++++++++++++++++++++
drivers/bus/cdx/meson.build | 12 +
drivers/bus/cdx/private.h | 49 ++
drivers/bus/cdx/version.map | 13 +
drivers/bus/meson.build | 1 +
lib/eal/common/eal_common_interrupts.c | 21 +
lib/eal/common/eal_interrupts.h | 1 +
lib/eal/include/rte_interrupts.h | 32 ++
lib/eal/version.map | 2 +
15 files changed, 1744 insertions(+)
create mode 100644 config/arm/arm64_cdx_linux_gcc
create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
create mode 100644 drivers/bus/cdx/cdx.c
create mode 100644 drivers/bus/cdx/cdx_logs.h
create mode 100644 drivers/bus/cdx/cdx_vfio.c
create mode 100644 drivers/bus/cdx/meson.build
create mode 100644 drivers/bus/cdx/private.h
create mode 100644 drivers/bus/cdx/version.map
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v2 1/6] bus/cdx: introduce cdx bus
2023-04-13 13:26 ` [PATCH v2 " Nipun Gupta
@ 2023-04-13 13:26 ` Nipun Gupta
2023-04-14 16:45 ` Ferruh Yigit
2023-04-13 13:27 ` [PATCH v2 2/6] bus/cdx: add dma map and unmap support Nipun Gupta
` (5 subsequent siblings)
6 siblings, 1 reply; 102+ messages in thread
From: Nipun Gupta @ 2023-04-13 13:26 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
CDX bus supports multiple type of devices, which can be
exposed to user-space via vfio-cdx.
vfio-cdx provides the MMIO IO_MEMORY regions as well as the
DMA interface for the device (IOMMU).
This support aims to enable the DPDK to support the cdx
devices in user-space using VFIO interface.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
MAINTAINERS | 5 +
drivers/bus/cdx/bus_cdx_driver.h | 201 ++++++++++++
drivers/bus/cdx/cdx.c | 521 +++++++++++++++++++++++++++++++
drivers/bus/cdx/cdx_logs.h | 37 +++
drivers/bus/cdx/cdx_vfio.c | 437 ++++++++++++++++++++++++++
drivers/bus/cdx/meson.build | 13 +
drivers/bus/cdx/private.h | 49 +++
drivers/bus/cdx/version.map | 11 +
8 files changed, 1274 insertions(+)
create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
create mode 100644 drivers/bus/cdx/cdx.c
create mode 100644 drivers/bus/cdx/cdx_logs.h
create mode 100644 drivers/bus/cdx/cdx_vfio.c
create mode 100644 drivers/bus/cdx/meson.build
create mode 100644 drivers/bus/cdx/private.h
create mode 100644 drivers/bus/cdx/version.map
diff --git a/MAINTAINERS b/MAINTAINERS
index 8df23e5099..1f9b6af9b9 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -569,6 +569,11 @@ M: Parav Pandit <parav@nvidia.com>
M: Xueming Li <xuemingl@nvidia.com>
F: drivers/bus/auxiliary/
+CDX bus driver
+M: Nipun Gupta <nipun.gupta@amd.com>
+M: Nikhil Agarwal <nikhil.agarwal@amd.com>
+F: drivers/bus/cdx/
+
Intel FPGA bus
M: Rosen Xu <rosen.xu@intel.com>
F: drivers/bus/ifpga/
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
new file mode 100644
index 0000000000..7edcb019eb
--- /dev/null
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -0,0 +1,201 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef _BUS_CDX_DRIVER_H_
+#define _BUS_CDX_DRIVER_H_
+
+/**
+ * @file
+ *
+ * CDX bus interface
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <limits.h>
+#include <errno.h>
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <bus_driver.h>
+#include <dev_driver.h>
+#include <rte_debug.h>
+#include <rte_interrupts.h>
+#include <rte_dev.h>
+#include <rte_bus.h>
+
+/* Forward declarations */
+struct rte_cdx_device;
+struct rte_cdx_driver;
+
+#define CDX_MAX_RESOURCE 4
+
+/** List of CDX devices */
+RTE_TAILQ_HEAD(rte_cdx_device_list, rte_cdx_device);
+/** List of CDX drivers */
+RTE_TAILQ_HEAD(rte_cdx_driver_list, rte_cdx_driver);
+
+/* CDX Bus iterators */
+#define FOREACH_DEVICE_ON_CDXBUS(p) \
+ RTE_TAILQ_FOREACH(p, &rte_cdx_bus.device_list, next)
+
+#define FOREACH_DRIVER_ON_CDXBUS(p) \
+ RTE_TAILQ_FOREACH(p, &rte_cdx_bus.driver_list, next)
+
+/** Any CDX device identifier (vendor, device) */
+#define RTE_CDX_ANY_ID (0xffff)
+
+#define RTE_PMD_REGISTER_CDX_TABLE(name, table) \
+static const char DRV_EXP_TAG(name, cdx_tbl_export)[] __rte_used = \
+RTE_STR(table)
+
+/**
+ * A structure describing an ID for a CDX driver. Each driver provides a
+ * table of these IDs for each device that it supports.
+ */
+struct rte_cdx_id {
+ uint16_t vendor_id; /**< Vendor ID. */
+ uint16_t device_id; /**< Device ID. */
+};
+
+/**
+ * A structure describing a CDX device.
+ */
+struct rte_cdx_device {
+ RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
+ struct rte_device device; /**< Inherit core device */
+ struct rte_cdx_id id; /**< CDX ID. */
+ struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
+ /**< CDX Memory Resource */
+};
+
+/**
+ * @internal
+ * Helper macro for drivers that need to convert to struct rte_cdx_device.
+ */
+#define RTE_DEV_TO_CDX_DEV(ptr) \
+ container_of(ptr, struct rte_cdx_device, device)
+
+#define RTE_DEV_TO_CDX_DEV_CONST(ptr) \
+ container_of(ptr, const struct rte_cdx_device, device)
+
+#define RTE_ETH_DEV_TO_CDX_DEV(eth_dev) RTE_DEV_TO_CDX_DEV((eth_dev)->device)
+
+#ifdef __cplusplus
+/** C++ macro used to help building up tables of device IDs */
+#define RTE_CDX_DEVICE(vend, dev) \
+ (vend), \
+ (dev)
+#else
+/** Macro used to help building up tables of device IDs */
+#define RTE_CDX_DEVICE(vend, dev) \
+ .vendor_id = (vend), \
+ .device_id = (dev)
+#endif
+
+/**
+ * Initialisation function for the driver called during CDX probing.
+ */
+typedef int (rte_cdx_probe_t)(struct rte_cdx_driver *, struct rte_cdx_device *);
+
+/**
+ * Uninitialisation function for the driver called during hotplugging.
+ */
+typedef int (rte_cdx_remove_t)(struct rte_cdx_device *);
+
+/**
+ * A structure describing a CDX driver.
+ */
+struct rte_cdx_driver {
+ RTE_TAILQ_ENTRY(rte_cdx_driver) next; /**< Next in list. */
+ struct rte_driver driver; /**< Inherit core driver. */
+ struct rte_cdx_bus *bus; /**< CDX bus reference. */
+ rte_cdx_probe_t *probe; /**< Device probe function. */
+ rte_cdx_remove_t *remove; /**< Device remove function. */
+ const struct rte_cdx_id *id_table; /**< ID table, NULL terminated. */
+ uint32_t drv_flags; /**< Flags RTE_CDX_DRV_*. */
+};
+
+/**
+ * Structure describing the CDX bus
+ */
+struct rte_cdx_bus {
+ struct rte_bus bus; /**< Inherit the generic class */
+ struct rte_cdx_device_list device_list; /**< List of CDX devices */
+ struct rte_cdx_driver_list driver_list; /**< List of CDX drivers */
+};
+
+/**
+ * Get Pathname of CDX devices directory.
+ *
+ * @return
+ * sysfs path for CDX devices.
+ */
+__rte_internal
+const char *rte_cdx_get_sysfs_path(void);
+
+/**
+ * Map the CDX device resources in user space virtual memory address
+ *
+ * @param dev
+ * A pointer to a rte_cdx_device structure describing the device
+ * to use
+ *
+ * @return
+ * 0 on success, negative on error and positive if no driver
+ * is found for the device.
+ */
+__rte_internal
+int rte_cdx_map_device(struct rte_cdx_device *dev);
+
+/**
+ * Unmap this device
+ *
+ * @param dev
+ * A pointer to a rte_cdx_device structure describing the device
+ * to use
+ */
+__rte_internal
+void rte_cdx_unmap_device(struct rte_cdx_device *dev);
+
+/**
+ * Register a CDX driver.
+ *
+ * @param driver
+ * A pointer to a rte_cdx_driver structure describing the driver
+ * to be registered.
+ */
+__rte_internal
+void rte_cdx_register(struct rte_cdx_driver *driver);
+
+/**
+ * Helper for CDX device registration from driver (eth, crypto, raw) instance
+ */
+#define RTE_PMD_REGISTER_CDX(nm, cdx_drv) \
+ RTE_INIT(cdxinitfn_ ##nm) \
+ {\
+ (cdx_drv).driver.name = RTE_STR(nm);\
+ rte_cdx_register(&cdx_drv); \
+ } \
+ RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+/**
+ * Unregister a CDX driver.
+ *
+ * @param driver
+ * A pointer to a rte_cdx_driver structure describing the driver
+ * to be unregistered.
+ */
+__rte_internal
+void rte_cdx_unregister(struct rte_cdx_driver *driver);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _BUS_CDX_DRIVER_H_ */
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
new file mode 100644
index 0000000000..bb23b32312
--- /dev/null
+++ b/drivers/bus/cdx/cdx.c
@@ -0,0 +1,521 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+/*
+ * Architecture Overview
+ * =====================
+ * CDX is a Hardware Architecture designed for AMD FPGA devices. It
+ * consists of sophisticated mechanism for interaction between FPGA,
+ * Firmware and the APUs (Application CPUs).
+ *
+ * Firmware resides on RPU (Realtime CPUs) which interacts with
+ * the FPGA program manager and the APUs. The RPU provides memory-mapped
+ * interface (RPU if) which is used to communicate with APUs.
+ *
+ * The diagram below shows an overview of the CDX architecture:
+ *
+ * +--------------------------------------+
+ * | DPDK |
+ * | DPDK CDX drivers |
+ * | | |
+ * | DPDK CDX bus |
+ * | | |
+ * +-----------------------------|--------+
+ * |
+ * +-----------------------------|--------+
+ * | Application CPUs (APU) | |
+ * | | |
+ * | VFIO CDX driver |
+ * | Linux OS | |
+ * | Linux CDX bus |
+ * | | |
+ * +-----------------------------|--------+
+ * |
+ * |
+ * +------------------------| RPU if |----+
+ * | | |
+ * | V |
+ * | Realtime CPUs (RPU) |
+ * | |
+ * +--------------------------------------+
+ * |
+ * +---------------------|----------------+
+ * | FPGA | |
+ * | +-----------------------+ |
+ * | | | | |
+ * | +-------+ +-------+ +-------+ |
+ * | | dev 1 | | dev 2 | | dev 3 | |
+ * | +-------+ +-------+ +-------+ |
+ * +--------------------------------------+
+ *
+ * The RPU firmware extracts the device information from the loaded FPGA
+ * image and implements a mechanism that allows the APU drivers to
+ * enumerate such devices (device personality and resource details) via
+ * a dedicated communication channel.
+ *
+ * VFIO CDX driver provides the CDX device resources like MMIO and interrupts
+ * to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
+ * driver to discover and initialize the CDX devices for user-space
+ * applications.
+ */
+
+#include <string.h>
+#include <dirent.h>
+
+#include <rte_log.h>
+#include <rte_bus.h>
+#include <rte_eal_paging.h>
+#include <rte_errno.h>
+#include <rte_devargs.h>
+#include <rte_malloc.h>
+#include <rte_memcpy.h>
+#include <rte_vfio.h>
+
+#include <eal_filesystem.h>
+
+#include "bus_cdx_driver.h"
+#include "cdx_logs.h"
+#include "private.h"
+
+#define SYSFS_CDX_DEVICES "/sys/bus/cdx/devices"
+#define CDX_BUS_NAME cdx
+#define CDX_DEV_PREFIX "cdx-"
+
+/**
+ * @file
+ * CDX probing using Linux sysfs.
+ */
+
+/* Add a device to CDX bus */
+static void
+cdx_add_device(struct rte_cdx_device *cdx_dev)
+{
+ TAILQ_INSERT_TAIL(&rte_cdx_bus.device_list, cdx_dev, next);
+}
+
+static int
+cdx_get_kernel_driver_by_path(const char *filename, char *driver_name,
+ size_t len)
+{
+ int count;
+ char path[PATH_MAX];
+ char *name;
+
+ if (!filename || !driver_name)
+ return -1;
+
+ count = readlink(filename, path, PATH_MAX);
+ if (count >= PATH_MAX)
+ return -1;
+
+ /* For device does not have a driver */
+ if (count < 0)
+ return 1;
+
+ path[count] = '\0';
+
+ name = strrchr(path, '/');
+ if (name) {
+ strlcpy(driver_name, name + 1, len);
+ return 0;
+ }
+
+ return -1;
+}
+
+int rte_cdx_map_device(struct rte_cdx_device *dev)
+{
+ return cdx_vfio_map_resource(dev);
+}
+
+void rte_cdx_unmap_device(struct rte_cdx_device *dev)
+{
+ cdx_vfio_unmap_resource(dev);
+}
+
+static struct rte_devargs *
+cdx_devargs_lookup(const char *dev_name)
+{
+ struct rte_devargs *devargs;
+
+ RTE_EAL_DEVARGS_FOREACH("cdx", devargs) {
+ if (strcmp(devargs->name, dev_name) == 0)
+ return devargs;
+ }
+ return NULL;
+}
+
+static bool
+cdx_ignore_device(const char *dev_name)
+{
+ struct rte_devargs *devargs = cdx_devargs_lookup(dev_name);
+
+ switch (rte_cdx_bus.bus.conf.scan_mode) {
+ case RTE_BUS_SCAN_ALLOWLIST:
+ if (devargs && devargs->policy == RTE_DEV_ALLOWED)
+ return false;
+ break;
+ case RTE_BUS_SCAN_UNDEFINED:
+ case RTE_BUS_SCAN_BLOCKLIST:
+ if (devargs == NULL || devargs->policy != RTE_DEV_BLOCKED)
+ return false;
+ break;
+ }
+ return true;
+}
+
+/*
+ * Scan one cdx sysfs entry, and fill the devices list from it.
+ * It checks if the CDX device is bound to vfio-cdx driver. In case
+ * the device is vfio bound, it reads the vendor and device id and
+ * stores it for device-driver matching.
+ */
+static int
+cdx_scan_one(const char *dirname, const char *dev_name)
+{
+ char filename[PATH_MAX];
+ struct rte_cdx_device *dev = NULL;
+ char driver[PATH_MAX];
+ unsigned long tmp;
+ char *name = NULL;
+ int ret;
+
+ dev = calloc(1, sizeof(*dev));
+ if (!dev)
+ return -ENOMEM;
+
+ name = calloc(1, RTE_DEV_NAME_MAX_LEN);
+ if (!name) {
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ dev->device.bus = &rte_cdx_bus.bus;
+ memcpy(name, dev_name, RTE_DEV_NAME_MAX_LEN);
+ dev->device.name = name;
+
+ /* parse driver */
+ snprintf(filename, sizeof(filename), "%s/driver", dirname);
+ ret = cdx_get_kernel_driver_by_path(filename, driver, sizeof(driver));
+ if (ret < 0) {
+ CDX_BUS_ERR("Fail to get kernel driver");
+ ret = -1;
+ goto err;
+ }
+
+ /*
+ * Check if device is bound to 'vfio-cdx' driver, so that user-space
+ * can gracefully access the device.
+ */
+ if (ret || strcmp(driver, "vfio-cdx")) {
+ ret = 0;
+ goto err;
+ }
+
+ /* get vendor id */
+ snprintf(filename, sizeof(filename), "%s/vendor", dirname);
+ if (eal_parse_sysfs_value(filename, &tmp) < 0) {
+ ret = -1;
+ goto err;
+ }
+ dev->id.vendor_id = (uint16_t)tmp;
+
+ /* get device id */
+ snprintf(filename, sizeof(filename), "%s/device", dirname);
+ if (eal_parse_sysfs_value(filename, &tmp) < 0) {
+ free(dev);
+ return -1;
+ }
+ dev->id.device_id = (uint16_t)tmp;
+
+ cdx_add_device(dev);
+
+ return 0;
+
+err:
+ if (name)
+ free(name);
+ if (dev)
+ free(dev);
+ return ret;
+}
+
+/*
+ * Scan the content of the CDX bus, and the devices in the devices
+ * list.
+ */
+static int
+cdx_scan(void)
+{
+ struct dirent *e;
+ DIR *dir;
+ char dirname[PATH_MAX];
+
+ dir = opendir(rte_cdx_get_sysfs_path());
+ if (dir == NULL) {
+ CDX_BUS_ERR("%s(): opendir failed: %s", __func__,
+ strerror(errno));
+ return -1;
+ }
+
+ while ((e = readdir(dir)) != NULL) {
+ if (e->d_name[0] == '.')
+ continue;
+
+ if (cdx_ignore_device(e->d_name))
+ continue;
+
+ snprintf(dirname, sizeof(dirname), "%s/%s",
+ rte_cdx_get_sysfs_path(), e->d_name);
+
+ if (cdx_scan_one(dirname, e->d_name) < 0)
+ goto error;
+ }
+ closedir(dir);
+ return 0;
+
+error:
+ closedir(dir);
+ return -1;
+}
+
+const char *
+rte_cdx_get_sysfs_path(void)
+{
+ return SYSFS_CDX_DEVICES;
+}
+
+/* map a particular resource from a file */
+void *
+cdx_map_resource(void *requested_addr, int fd, off_t offset, size_t size,
+ int additional_flags)
+{
+ void *mapaddr;
+
+ /* Map the cdx MMIO memory resource of device */
+ mapaddr = rte_mem_map(requested_addr, size,
+ RTE_PROT_READ | RTE_PROT_WRITE,
+ RTE_MAP_SHARED | additional_flags, fd, offset);
+ if (mapaddr == NULL) {
+ CDX_BUS_ERR("%s(): cannot map resource(%d, %p, 0x%zx, 0x%llx): %s (%p)",
+ __func__, fd, requested_addr, size,
+ (unsigned long long)offset,
+ rte_strerror(rte_errno), mapaddr);
+ }
+ CDX_BUS_DEBUG("CDX MMIO memory mapped at %p", mapaddr);
+
+ return mapaddr;
+}
+
+/* unmap a particular resource */
+void
+cdx_unmap_resource(void *requested_addr, size_t size)
+{
+ if (requested_addr == NULL)
+ return;
+
+ /* Unmap the CDX memory resource of device */
+ if (rte_mem_unmap(requested_addr, size)) {
+ CDX_BUS_ERR("%s(): cannot mem unmap(%p, %#zx): %s", __func__,
+ requested_addr, size, rte_strerror(rte_errno));
+ }
+ CDX_BUS_DEBUG("CDX memory unmapped at %p", requested_addr);
+}
+/*
+ * Match the CDX Driver and Device using device id and vendor id.
+ */
+static int
+cdx_match(const struct rte_cdx_driver *cdx_drv,
+ const struct rte_cdx_device *cdx_dev)
+{
+ const struct rte_cdx_id *id_table;
+
+ for (id_table = cdx_drv->id_table; id_table->vendor_id != 0;
+ id_table++) {
+ /* check if device's identifiers match the driver's ones */
+ if (id_table->vendor_id != cdx_dev->id.vendor_id &&
+ id_table->vendor_id != RTE_CDX_ANY_ID)
+ continue;
+ if (id_table->device_id != cdx_dev->id.device_id &&
+ id_table->device_id != RTE_CDX_ANY_ID)
+ continue;
+
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * If vendor id and device id match, call the probe() function of the
+ * driver.
+ */
+static int
+cdx_probe_one_driver(struct rte_cdx_driver *dr,
+ struct rte_cdx_device *dev)
+{
+ const char *dev_name = dev->device.name;
+ bool already_probed;
+ int ret;
+
+ if ((dr == NULL) || (dev == NULL))
+ return -EINVAL;
+
+ /* The device is not blocked; Check if driver supports it */
+ if (!cdx_match(dr, dev))
+ /* Match of device and driver failed */
+ return 1;
+
+ already_probed = rte_dev_is_probed(&dev->device);
+ if (already_probed) {
+ CDX_BUS_INFO("Device %s is already probed", dev->device.name);
+ return -EEXIST;
+ }
+
+ CDX_BUS_DEBUG(" probe device %s using driver: %s", dev_name,
+ dr->driver.name);
+
+ ret = cdx_vfio_map_resource(dev);
+ if (ret != 0) {
+ CDX_BUS_ERR("CDX map device failed: %d", ret);
+ goto error_map_device;
+ }
+
+ /* call the driver probe() function */
+ ret = dr->probe(dr, dev);
+ if (ret) {
+ CDX_BUS_ERR("Probe CDX driver: %s device: %s failed: %d",
+ dr->driver.name, dev_name, ret);
+ goto error_probe;
+ } else {
+ dev->device.driver = &dr->driver;
+ }
+
+ return ret;
+
+error_probe:
+ cdx_vfio_unmap_resource(dev);
+error_map_device:
+ return ret;
+}
+
+/*
+ * If vendor/device ID match, call the probe() function of all
+ * registered driver for the given device. Return < 0 if initialization
+ * failed, return 1 if no driver is found for this device.
+ */
+static int
+cdx_probe_all_drivers(struct rte_cdx_device *dev)
+{
+ struct rte_cdx_driver *dr = NULL;
+ int rc = 0;
+
+ if (dev == NULL)
+ return -EINVAL;
+
+ FOREACH_DRIVER_ON_CDXBUS(dr) {
+ rc = cdx_probe_one_driver(dr, dev);
+ if (rc < 0)
+ /* negative value is an error */
+ return rc;
+ if (rc > 0)
+ /* positive value means driver doesn't support it */
+ continue;
+ return 0;
+ }
+ return 1;
+}
+
+/*
+ * Scan the content of the CDX bus, and call the probe() function for
+ * all registered drivers that have a matching entry in its id_table
+ * for discovered devices.
+ */
+static int
+cdx_probe(void)
+{
+ struct rte_cdx_device *dev = NULL;
+ size_t probed = 0, failed = 0;
+ int ret = 0;
+
+ FOREACH_DEVICE_ON_CDXBUS(dev) {
+ probed++;
+
+ ret = cdx_probe_all_drivers(dev);
+ if (ret < 0) {
+ CDX_BUS_ERR("Requested device %s cannot be used",
+ dev->device.name);
+ rte_errno = errno;
+ failed++;
+ ret = 0;
+ }
+ }
+
+ return (probed && probed == failed) ? -1 : 0;
+}
+
+static int
+cdx_parse(const char *name, void *addr)
+{
+ const char **out = addr;
+ int ret;
+
+ ret = strncmp(name, CDX_DEV_PREFIX, strlen(CDX_DEV_PREFIX));
+
+ if (ret == 0 && addr)
+ *out = name;
+
+ return ret;
+}
+
+/* register a driver */
+void
+rte_cdx_register(struct rte_cdx_driver *driver)
+{
+ TAILQ_INSERT_TAIL(&rte_cdx_bus.driver_list, driver, next);
+ driver->bus = &rte_cdx_bus;
+}
+
+/* unregister a driver */
+void
+rte_cdx_unregister(struct rte_cdx_driver *driver)
+{
+ TAILQ_REMOVE(&rte_cdx_bus.driver_list, driver, next);
+ driver->bus = NULL;
+}
+
+static struct rte_device *
+cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
+ const void *data)
+{
+ const struct rte_cdx_device *cdx_start;
+ struct rte_cdx_device *cdx_dev;
+
+ if (start != NULL) {
+ cdx_start = RTE_DEV_TO_CDX_DEV_CONST(start);
+ cdx_dev = TAILQ_NEXT(cdx_start, next);
+ } else {
+ cdx_dev = TAILQ_FIRST(&rte_cdx_bus.device_list);
+ }
+ while (cdx_dev != NULL) {
+ if (cmp(&cdx_dev->device, data) == 0)
+ return &cdx_dev->device;
+ cdx_dev = TAILQ_NEXT(cdx_dev, next);
+ }
+ return NULL;
+}
+
+struct rte_cdx_bus rte_cdx_bus = {
+ .bus = {
+ .scan = cdx_scan,
+ .probe = cdx_probe,
+ .find_device = cdx_find_device,
+ .parse = cdx_parse,
+ },
+ .device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
+ .driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
+};
+
+RTE_REGISTER_BUS(cdx, rte_cdx_bus.bus);
+RTE_LOG_REGISTER_DEFAULT(cdx_logtype_bus, NOTICE);
diff --git a/drivers/bus/cdx/cdx_logs.h b/drivers/bus/cdx/cdx_logs.h
new file mode 100644
index 0000000000..e5e23d8b94
--- /dev/null
+++ b/drivers/bus/cdx/cdx_logs.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef _CDX_LOGS_H_
+#define _CDX_LOGS_H_
+
+extern int cdx_logtype_bus;
+
+#define CDX_BUS_LOG(level, fmt, args...) \
+ rte_log(RTE_LOG_ ## level, cdx_logtype_bus, "cdx: " fmt "\n", \
+ ##args)
+
+/* Debug logs are with Function names */
+#define CDX_BUS_DEBUG(fmt, args...) \
+ rte_log(RTE_LOG_DEBUG, cdx_logtype_bus, "cdx: %s(): " fmt "\n", \
+ __func__, ##args)
+
+#define CDX_BUS_INFO(fmt, args...) \
+ CDX_BUS_LOG(INFO, fmt, ## args)
+#define CDX_BUS_ERR(fmt, args...) \
+ CDX_BUS_LOG(ERR, fmt, ## args)
+#define CDX_BUS_WARN(fmt, args...) \
+ CDX_BUS_LOG(WARNING, fmt, ## args)
+
+/* DP Logs, toggled out at compile time if level lower than current level */
+#define CDX_BUS_DP_LOG(level, fmt, args...) \
+ RTE_LOG_DP(level, PMD, fmt, ## args)
+
+#define CDX_BUS_DP_DEBUG(fmt, args...) \
+ CDX_BUS_DP_LOG(DEBUG, fmt, ## args)
+#define CDX_BUS_DP_INFO(fmt, args...) \
+ CDX_BUS_DP_LOG(INFO, fmt, ## args)
+#define CDX_BUS_DP_WARN(fmt, args...) \
+ CDX_BUS_DP_LOG(WARNING, fmt, ## args)
+
+#endif /* _CDX_LOGS_H_ */
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
new file mode 100644
index 0000000000..ae11f589b3
--- /dev/null
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -0,0 +1,437 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+/**
+ * @file
+ * CDX probing using Linux VFIO.
+ *
+ * This code tries to determine if the CDX device is bound to VFIO driver,
+ * and initialize it (map MMIO regions, set up interrupts) if that's the case.
+ *
+ */
+
+#include <string.h>
+#include <fcntl.h>
+#include <sys/eventfd.h>
+#include <sys/socket.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <stdbool.h>
+
+#include <rte_log.h>
+#include <rte_eal_paging.h>
+#include <rte_malloc.h>
+#include <rte_vfio.h>
+#include <rte_eal.h>
+#include <rte_bus.h>
+#include <rte_spinlock.h>
+#include <rte_tailq.h>
+
+#include "eal_filesystem.h"
+
+#include "bus_cdx_driver.h"
+#include "cdx_logs.h"
+#include "private.h"
+
+/**
+ * A structure describing a CDX mapping.
+ */
+struct cdx_map {
+ void *addr;
+ char *path;
+ uint64_t offset;
+ uint64_t size;
+};
+
+/**
+ * A structure describing a mapped CDX resource.
+ * For multi-process we need to reproduce all CDX mappings in secondary
+ * processes, so save them in a tailq.
+ */
+struct mapped_cdx_resource {
+ TAILQ_ENTRY(mapped_cdx_resource) next;
+ char name[RTE_DEV_NAME_MAX_LEN]; /**< CDX device name */
+ char path[PATH_MAX];
+ int nb_maps;
+ struct cdx_map maps[CDX_MAX_RESOURCE];
+};
+
+/** mapped cdx device list */
+TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
+
+static struct rte_tailq_elem cdx_vfio_tailq = {
+ .name = "VFIO_CDX_RESOURCE_LIST",
+};
+EAL_REGISTER_TAILQ(cdx_vfio_tailq)
+
+static struct mapped_cdx_resource *
+cdx_vfio_find_and_unmap_resource(struct mapped_cdx_res_list *vfio_res_list,
+ struct rte_cdx_device *dev)
+{
+ struct mapped_cdx_resource *vfio_res = NULL;
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+ int i;
+
+ /* Get vfio_res */
+ TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+ if (strcmp(vfio_res->name, dev_name))
+ continue;
+ break;
+ }
+
+ if (vfio_res == NULL)
+ return vfio_res;
+
+ CDX_BUS_INFO("Releasing CDX mapped resource for %s", dev_name);
+
+ maps = vfio_res->maps;
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ if (maps[i].addr) {
+ CDX_BUS_DEBUG("Calling cdx_unmap_resource for %s at %p",
+ dev_name, maps[i].addr);
+ cdx_unmap_resource(maps[i].addr, maps[i].size);
+ }
+ }
+
+ return vfio_res;
+}
+
+static int
+cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
+{
+ char cdx_addr[PATH_MAX] = {0};
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list;
+
+ vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ vfio_res = cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for cdx device!",
+ cdx_addr);
+ return -1;
+ }
+
+ TAILQ_REMOVE(vfio_res_list, vfio_res, next);
+ rte_free(vfio_res);
+ return 0;
+}
+
+static int
+cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
+{
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list;
+
+ vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ vfio_res = cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for CDX device!",
+ dev->device.name);
+ return -1;
+ }
+
+ return 0;
+}
+
+int
+cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
+{
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return cdx_vfio_unmap_resource_primary(dev);
+ else
+ return cdx_vfio_unmap_resource_secondary(dev);
+}
+
+static int
+cdx_rte_vfio_setup_device(int vfio_dev_fd)
+{
+ /*
+ * Reset the device. If the device is not capable of resetting,
+ * then it updates errno as EINVAL.
+ */
+ if (ioctl(vfio_dev_fd, VFIO_DEVICE_RESET) && errno != EINVAL) {
+ CDX_BUS_ERR("Unable to reset device! Error: %d (%s)", errno,
+ strerror(errno));
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+cdx_vfio_mmap_resource(int vfio_dev_fd, struct mapped_cdx_resource *vfio_res,
+ int index, int additional_flags)
+{
+ struct cdx_map *map = &vfio_res->maps[index];
+ void *vaddr;
+
+ if (map->size == 0) {
+ CDX_BUS_DEBUG("map size is 0, skip region %d", index);
+ return 0;
+ }
+
+ /* reserve the address using an inaccessible mapping */
+ vaddr = mmap(map->addr, map->size, 0, MAP_PRIVATE |
+ MAP_ANONYMOUS | additional_flags, -1, 0);
+ if (vaddr != MAP_FAILED) {
+ void *map_addr = NULL;
+
+ if (map->size) {
+ /* actual map of first part */
+ map_addr = cdx_map_resource(vaddr, vfio_dev_fd,
+ map->offset, map->size,
+ RTE_MAP_FORCE_ADDRESS);
+ }
+
+ if (map_addr == NULL) {
+ munmap(vaddr, map->size);
+ vaddr = MAP_FAILED;
+ CDX_BUS_ERR("Failed to map cdx MMIO region %d", index);
+ return -1;
+ }
+ } else {
+ CDX_BUS_ERR("Failed to create inaccessible mapping for MMIO region %d",
+ index);
+ return -1;
+ }
+
+ map->addr = vaddr;
+ return 0;
+}
+
+/*
+ * region info may contain capability headers, so we need to keep reallocating
+ * the memory until we match allocated memory size with argsz.
+ */
+static int
+cdx_vfio_get_region_info(int vfio_dev_fd, struct vfio_region_info **info,
+ int region)
+{
+ struct vfio_region_info *ri;
+ size_t argsz = sizeof(*ri);
+ int ret;
+
+ ri = malloc(sizeof(*ri));
+ if (ri == NULL) {
+ CDX_BUS_ERR("Cannot allocate memory for VFIO region info");
+ return -1;
+ }
+again:
+ memset(ri, 0, argsz);
+ ri->argsz = argsz;
+ ri->index = region;
+
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_REGION_INFO, ri);
+ if (ret < 0) {
+ free(ri);
+ return ret;
+ }
+ if (ri->argsz != argsz) {
+ struct vfio_region_info *tmp;
+
+ argsz = ri->argsz;
+ tmp = realloc(ri, argsz);
+
+ if (tmp == NULL) {
+ /* realloc failed but the ri is still there */
+ free(ri);
+ CDX_BUS_ERR("Cannot reallocate memory for VFIO region info");
+ return -1;
+ }
+ ri = tmp;
+ goto again;
+ }
+ *info = ri;
+
+ return 0;
+}
+
+static int
+find_max_end_va(const struct rte_memseg_list *msl, void *arg)
+{
+ size_t sz = msl->len;
+ void *end_va = RTE_PTR_ADD(msl->base_va, sz);
+ void **max_va = arg;
+
+ if (*max_va < end_va)
+ *max_va = end_va;
+ return 0;
+}
+
+static void *
+cdx_find_max_end_va(void)
+{
+ void *va = NULL;
+
+ rte_memseg_list_walk(find_max_end_va, &va);
+ return va;
+}
+
+static int
+cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ char cdx_addr[PATH_MAX] = {0};
+ static void *cdx_map_addr;
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+ int vfio_dev_fd, i, ret;
+
+ ret = rte_vfio_setup_device(rte_cdx_get_sysfs_path(), dev_name,
+ &vfio_dev_fd, &device_info);
+ if (ret)
+ return ret;
+
+ /* allocate vfio_res and get region info */
+ vfio_res = rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0);
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("Cannot store VFIO mmap details");
+ goto err_vfio_dev_fd;
+ }
+ memcpy(vfio_res->name, dev_name, RTE_DEV_NAME_MAX_LEN);
+
+ /* get number of registers */
+ vfio_res->nb_maps = device_info.num_regions;
+
+ /* map memory regions */
+ maps = vfio_res->maps;
+
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ struct vfio_region_info *reg = NULL;
+ void *vaddr;
+
+ ret = cdx_vfio_get_region_info(vfio_dev_fd, ®, i);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s cannot get device region info error %i (%s)",
+ dev_name, errno, strerror(errno));
+ goto err_vfio_res;
+ }
+
+ /* skip non-mmappable regions */
+ if ((reg->flags & VFIO_REGION_INFO_FLAG_MMAP) == 0) {
+ free(reg);
+ continue;
+ }
+
+ /* try mapping somewhere close to the end of hugepages */
+ if (cdx_map_addr == NULL)
+ cdx_map_addr = cdx_find_max_end_va();
+
+ vaddr = cdx_map_addr;
+ cdx_map_addr = RTE_PTR_ADD(vaddr, (size_t)reg->size);
+
+ cdx_map_addr = RTE_PTR_ALIGN(cdx_map_addr,
+ sysconf(_SC_PAGE_SIZE));
+
+ maps[i].addr = vaddr;
+ maps[i].offset = reg->offset;
+ maps[i].size = reg->size;
+ maps[i].path = NULL; /* vfio doesn't have per-resource paths */
+
+ ret = cdx_vfio_mmap_resource(vfio_dev_fd, vfio_res, i, 0);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s mapping region %i failed: %s",
+ cdx_addr, i, strerror(errno));
+ free(reg);
+ goto err_vfio_res;
+ }
+
+ dev->mem_resource[i].addr = maps[i].addr;
+ dev->mem_resource[i].len = maps[i].size;
+
+ free(reg);
+ }
+
+ if (cdx_rte_vfio_setup_device(vfio_dev_fd) < 0) {
+ CDX_BUS_ERR("%s setup device failed", dev_name);
+ goto err_vfio_res;
+ }
+
+ TAILQ_INSERT_TAIL(vfio_res_list, vfio_res, next);
+
+ return 0;
+err_vfio_res:
+ cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+ rte_free(vfio_res);
+err_vfio_dev_fd:
+ rte_vfio_release_device(rte_cdx_get_sysfs_path(),
+ dev_name, vfio_dev_fd);
+ return -1;
+}
+
+static int
+cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ char cdx_addr[PATH_MAX] = {0};
+ int vfio_dev_fd;
+ int i, ret;
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+
+ /* if we're in a secondary process, just find our tailq entry */
+ TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+ if (strcmp(vfio_res->name, dev_name))
+ continue;
+ break;
+ }
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for cdx device!",
+ dev_name);
+ return -1;
+ }
+
+ ret = rte_vfio_setup_device(rte_cdx_get_sysfs_path(), dev_name,
+ &vfio_dev_fd, &device_info);
+ if (ret)
+ return ret;
+
+ /* map MMIO regions */
+ maps = vfio_res->maps;
+
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ ret = cdx_vfio_mmap_resource(vfio_dev_fd, vfio_res, i, MAP_FIXED);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s mapping MMIO region %i failed: %s",
+ dev_name, i, strerror(errno));
+ goto err_vfio_dev_fd;
+ }
+
+ dev->mem_resource[i].addr = maps[i].addr;
+ dev->mem_resource[i].len = maps[i].size;
+ }
+
+ return 0;
+err_vfio_dev_fd:
+ rte_vfio_release_device(rte_cdx_get_sysfs_path(),
+ cdx_addr, vfio_dev_fd);
+ return -1;
+}
+
+/*
+ * map the CDX resources of a CDX device in virtual memory (VFIO version).
+ * primary and secondary processes follow almost exactly the same path
+ */
+int
+cdx_vfio_map_resource(struct rte_cdx_device *dev)
+{
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return cdx_vfio_map_resource_primary(dev);
+ else
+ return cdx_vfio_map_resource_secondary(dev);
+}
diff --git a/drivers/bus/cdx/meson.build b/drivers/bus/cdx/meson.build
new file mode 100644
index 0000000000..f2ca104d34
--- /dev/null
+++ b/drivers/bus/cdx/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+
+if not is_linux
+ build = false
+ reason = 'only supported on Linux'
+endif
+
+driver_sdk_headers = files('bus_cdx_driver.h')
+sources = files(
+ 'cdx.c',
+ 'cdx_vfio.c',
+)
diff --git a/drivers/bus/cdx/private.h b/drivers/bus/cdx/private.h
new file mode 100644
index 0000000000..03a5093473
--- /dev/null
+++ b/drivers/bus/cdx/private.h
@@ -0,0 +1,49 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef _CDX_PRIVATE_H_
+#define _CDX_PRIVATE_H_
+
+#include <stdbool.h>
+#include <stdio.h>
+
+#include "bus_cdx_driver.h"
+
+extern struct rte_cdx_bus rte_cdx_bus;
+
+/**
+ * Map a particular resource from a file.
+ *
+ * @param requested_addr
+ * The starting address for the new mapping range.
+ * @param fd
+ * The file descriptor.
+ * @param offset
+ * The offset for the mapping range.
+ * @param size
+ * The size for the mapping range.
+ * @param additional_flags
+ * The additional rte_mem_map() flags for the mapping range.
+ * @return
+ * - On success, the function returns a pointer to the mapped area.
+ * - On error, NULL is returned.
+ */
+void *cdx_map_resource(void *requested_addr, int fd, off_t offset,
+ size_t size, int additional_flags);
+
+/**
+ * Unmap a particular resource.
+ *
+ * @param requested_addr
+ * The address for the unmapping range.
+ * @param size
+ * The size for the unmapping range.
+ */
+void cdx_unmap_resource(void *requested_addr, size_t size);
+
+/* map/unmap VFIO resource */
+int cdx_vfio_map_resource(struct rte_cdx_device *dev);
+int cdx_vfio_unmap_resource(struct rte_cdx_device *dev);
+
+#endif /* _CDX_PRIVATE_H_ */
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
new file mode 100644
index 0000000000..957fcab978
--- /dev/null
+++ b/drivers/bus/cdx/version.map
@@ -0,0 +1,11 @@
+INTERNAL {
+ global:
+
+ rte_cdx_get_sysfs_path;
+ rte_cdx_map_device;
+ rte_cdx_register;
+ rte_cdx_unmap_device;
+ rte_cdx_unregister;
+
+ local: *;
+};
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v2 1/6] bus/cdx: introduce cdx bus
2023-04-13 13:26 ` [PATCH v2 1/6] bus/cdx: introduce cdx bus Nipun Gupta
@ 2023-04-14 16:45 ` Ferruh Yigit
2023-04-16 9:20 ` Gupta, Nipun
0 siblings, 1 reply; 102+ messages in thread
From: Ferruh Yigit @ 2023-04-14 16:45 UTC (permalink / raw)
To: Nipun Gupta, dev, thomas, david.marchand; +Cc: harpreet.anand, nikhil.agarwal
On 4/13/2023 2:26 PM, Nipun Gupta wrote:
> CDX bus supports multiple type of devices, which can be
> exposed to user-space via vfio-cdx.
>
> vfio-cdx provides the MMIO IO_MEMORY regions as well as the
> DMA interface for the device (IOMMU).
>
> This support aims to enable the DPDK to support the cdx
> devices in user-space using VFIO interface.
>
> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
<...>
> +/* map a particular resource from a file */
> +void *
> +cdx_map_resource(void *requested_addr, int fd, off_t offset, size_t size,
> + int additional_flags)
> +{
> + void *mapaddr;
> +
> + /* Map the cdx MMIO memory resource of device */
> + mapaddr = rte_mem_map(requested_addr, size,
> + RTE_PROT_READ | RTE_PROT_WRITE,
> + RTE_MAP_SHARED | additional_flags, fd, offset);
> + if (mapaddr == NULL) {
> + CDX_BUS_ERR("%s(): cannot map resource(%d, %p, 0x%zx, 0x%llx): %s (%p)",
> + __func__, fd, requested_addr, size,
> + (unsigned long long)offset,
`checkpatches.sh` complains about '%l' usage, and 'offset' seems defined
as 'off_t' type.
As far as I can see 'off_t' is not part of C standard (but posix
standard), and it is signed, also not clear if 32 bits or 64 bits.
Since caller of this function already passes parameter with 'uint64_t'
type, why not change the 'offset' type of this function to 'uint64_t'
for simplification, and use 'PRIu64' format specifier?
^ permalink raw reply [flat|nested] 102+ messages in thread* RE: [PATCH v2 1/6] bus/cdx: introduce cdx bus
2023-04-14 16:45 ` Ferruh Yigit
@ 2023-04-16 9:20 ` Gupta, Nipun
0 siblings, 0 replies; 102+ messages in thread
From: Gupta, Nipun @ 2023-04-16 9:20 UTC (permalink / raw)
To: Yigit, Ferruh, dev, thomas, david.marchand
Cc: Anand, Harpreet, Agarwal, Nikhil
> -----Original Message-----
> From: Yigit, Ferruh <Ferruh.Yigit@amd.com>
> Sent: Friday, April 14, 2023 10:15 PM
> To: Gupta, Nipun <Nipun.Gupta@amd.com>; dev@dpdk.org;
> thomas@monjalon.net; david.marchand@redhat.com
> Cc: Anand, Harpreet <harpreet.anand@amd.com>; Agarwal, Nikhil
> <nikhil.agarwal@amd.com>
> Subject: Re: [PATCH v2 1/6] bus/cdx: introduce cdx bus
>
> On 4/13/2023 2:26 PM, Nipun Gupta wrote:
> > CDX bus supports multiple type of devices, which can be
> > exposed to user-space via vfio-cdx.
> >
> > vfio-cdx provides the MMIO IO_MEMORY regions as well as the
> > DMA interface for the device (IOMMU).
> >
> > This support aims to enable the DPDK to support the cdx
> > devices in user-space using VFIO interface.
> >
> > Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
>
> <...>
>
> > +/* map a particular resource from a file */
> > +void *
> > +cdx_map_resource(void *requested_addr, int fd, off_t offset, size_t size,
> > + int additional_flags)
> > +{
> > + void *mapaddr;
> > +
> > + /* Map the cdx MMIO memory resource of device */
> > + mapaddr = rte_mem_map(requested_addr, size,
> > + RTE_PROT_READ | RTE_PROT_WRITE,
> > + RTE_MAP_SHARED | additional_flags, fd, offset);
> > + if (mapaddr == NULL) {
> > + CDX_BUS_ERR("%s(): cannot map resource(%d, %p, 0x%zx,
> 0x%llx): %s (%p)",
> > + __func__, fd, requested_addr, size,
> > + (unsigned long long)offset,
>
> `checkpatches.sh` complains about '%l' usage, and 'offset' seems defined
> as 'off_t' type.
>
> As far as I can see 'off_t' is not part of C standard (but posix
> standard), and it is signed, also not clear if 32 bits or 64 bits.
>
> Since caller of this function already passes parameter with 'uint64_t'
> type, why not change the 'offset' type of this function to 'uint64_t'
> for simplification, and use 'PRIu64' format specifier?
Makes sense. Will update this in next respin.
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v2 2/6] bus/cdx: add dma map and unmap support
2023-04-13 13:26 ` [PATCH v2 " Nipun Gupta
2023-04-13 13:26 ` [PATCH v2 1/6] bus/cdx: introduce cdx bus Nipun Gupta
@ 2023-04-13 13:27 ` Nipun Gupta
2023-04-13 13:27 ` [PATCH v2 3/6] bus/cdx: add support for MSI Nipun Gupta
` (4 subsequent siblings)
6 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-04-13 13:27 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
CDX bus can use VFIO interface for mapping and unmapping
of DMA addresses in the IOMMU. This change adds the callback
support for map and unmap APIs as well as fetching the IOMMU
class.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
drivers/bus/cdx/cdx.c | 40 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index bb23b32312..b1d8f8382f 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -506,12 +506,52 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
return NULL;
}
+static int
+cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+ struct rte_cdx_device *cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+
+ if (!cdx_dev) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ return rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ (uintptr_t)addr, iova, len);
+}
+
+static int
+cdx_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+ struct rte_cdx_device *cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+
+ if (!cdx_dev) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ return rte_vfio_container_dma_unmap(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ (uintptr_t)addr, iova, len);
+}
+
+static enum rte_iova_mode
+cdx_get_iommu_class(void)
+{
+ if (TAILQ_EMPTY(&rte_cdx_bus.device_list))
+ return RTE_IOVA_DC;
+
+ return RTE_IOVA_VA;
+}
+
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
.scan = cdx_scan,
.probe = cdx_probe,
.find_device = cdx_find_device,
.parse = cdx_parse,
+ .dma_map = cdx_dma_map,
+ .dma_unmap = cdx_dma_unmap,
+ .get_iommu_class = cdx_get_iommu_class,
},
.device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
.driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v2 3/6] bus/cdx: add support for MSI
2023-04-13 13:26 ` [PATCH v2 " Nipun Gupta
2023-04-13 13:26 ` [PATCH v2 1/6] bus/cdx: introduce cdx bus Nipun Gupta
2023-04-13 13:27 ` [PATCH v2 2/6] bus/cdx: add dma map and unmap support Nipun Gupta
@ 2023-04-13 13:27 ` Nipun Gupta
2023-04-14 16:45 ` Ferruh Yigit
2023-04-13 13:27 ` [PATCH v2 4/6] bus/cdx: support plug unplug and dev iterator Nipun Gupta
` (3 subsequent siblings)
6 siblings, 1 reply; 102+ messages in thread
From: Nipun Gupta @ 2023-04-13 13:27 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
MSI's are exposed to the devices using VFIO (vfio-cdx). This
patch uses the same to add support for MSI for the devices on
the cdx bus.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
drivers/bus/cdx/bus_cdx_driver.h | 25 ++++
drivers/bus/cdx/cdx.c | 11 ++
drivers/bus/cdx/cdx_vfio.c | 182 ++++++++++++++++++++++++-
drivers/bus/cdx/version.map | 2 +
lib/eal/common/eal_common_interrupts.c | 21 +++
lib/eal/common/eal_interrupts.h | 1 +
lib/eal/include/rte_interrupts.h | 32 +++++
lib/eal/version.map | 2 +
8 files changed, 274 insertions(+), 2 deletions(-)
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index 7edcb019eb..fdeaf46664 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -72,6 +72,7 @@ struct rte_cdx_device {
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
/**< CDX Memory Resource */
+ struct rte_intr_handle *intr_handle; /**< Interrupt handle */
};
/**
@@ -173,6 +174,30 @@ void rte_cdx_unmap_device(struct rte_cdx_device *dev);
__rte_internal
void rte_cdx_register(struct rte_cdx_driver *driver);
+/**
+ * Enables VFIO Interrupts for CDX bus devices.
+ *
+ * @param intr_handle
+ * Pointer to the interrupt handle.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle);
+
+/**
+ * Disable VFIO Interrupts for CDX bus devices.
+ *
+ * @param intr_handle
+ * Pointer to the interrupt handle.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle);
+
/**
* Helper for CDX device registration from driver (eth, crypto, raw) instance
*/
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index b1d8f8382f..f34736a54a 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -204,6 +204,15 @@ cdx_scan_one(const char *dirname, const char *dev_name)
goto err;
}
+ /* Allocate interrupt instance for cdx device */
+ dev->intr_handle =
+ rte_intr_instance_alloc(RTE_INTR_INSTANCE_F_PRIVATE);
+ if (dev->intr_handle == NULL) {
+ CDX_BUS_ERR("Failed to create interrupt instance for %s\n",
+ dev->device.name);
+ return -ENOMEM;
+ }
+
/*
* Check if device is bound to 'vfio-cdx' driver, so that user-space
* can gracefully access the device.
@@ -395,6 +404,8 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr,
return ret;
error_probe:
+ rte_intr_instance_free(dev->intr_handle);
+ dev->intr_handle = NULL;
cdx_vfio_unmap_resource(dev);
error_map_device:
return ret;
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index ae11f589b3..1422b98503 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -60,6 +60,10 @@ struct mapped_cdx_resource {
/** mapped cdx device list */
TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
+/* irq set buffer length for MSI interrupts */
+#define MSI_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \
+ sizeof(int) * (RTE_MAX_RXTX_INTR_VEC_ID + 1))
+
static struct rte_tailq_elem cdx_vfio_tailq = {
.name = "VFIO_CDX_RESOURCE_LIST",
};
@@ -104,6 +108,27 @@ cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
char cdx_addr[PATH_MAX] = {0};
struct mapped_cdx_resource *vfio_res = NULL;
struct mapped_cdx_res_list *vfio_res_list;
+ int ret, vfio_dev_fd;
+
+ if (rte_intr_fd_get(dev->intr_handle) < 0)
+ return -1;
+
+ if (close(rte_intr_fd_get(dev->intr_handle)) < 0) {
+ CDX_BUS_ERR("Error when closing eventfd file descriptor for %s",
+ dev->device.name);
+ return -1;
+ }
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ ret = rte_vfio_release_device(rte_cdx_get_sysfs_path(), dev->device.name,
+ vfio_dev_fd);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot release VFIO device");
+ return ret;
+ }
vfio_res_list =
RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
@@ -126,6 +151,18 @@ cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
{
struct mapped_cdx_resource *vfio_res = NULL;
struct mapped_cdx_res_list *vfio_res_list;
+ int ret, vfio_dev_fd;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ ret = rte_vfio_release_device(rte_cdx_get_sysfs_path(), dev->device.name,
+ vfio_dev_fd);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot release VFIO device");
+ return ret;
+ }
vfio_res_list =
RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
@@ -150,9 +187,80 @@ cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
return cdx_vfio_unmap_resource_secondary(dev);
}
+/* set up interrupt support (but not enable interrupts) */
static int
-cdx_rte_vfio_setup_device(int vfio_dev_fd)
+cdx_vfio_setup_interrupts(struct rte_cdx_device *dev, int vfio_dev_fd,
+ int num_irqs)
{
+ int i, ret;
+
+ if (num_irqs == 0)
+ return 0;
+
+ /* start from MSI interrupt type */
+ for (i = 0; i < num_irqs; i++) {
+ struct vfio_irq_info irq = { .argsz = sizeof(irq) };
+ int fd = -1;
+
+ irq.index = i;
+
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_IRQ_INFO, &irq);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot get VFIO IRQ info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /* if this vector cannot be used with eventfd, fail if we explicitly
+ * specified interrupt type, otherwise continue
+ */
+ if ((irq.flags & VFIO_IRQ_INFO_EVENTFD) == 0)
+ continue;
+
+ if (rte_intr_irq_count_set(dev->intr_handle, irq.count))
+ return -1;
+
+ /* Reallocate the efds and elist fields of intr_handle based
+ * on CDX device MSI size.
+ */
+ if ((uint32_t)rte_intr_nb_intr_get(dev->intr_handle) < irq.count &&
+ rte_intr_event_list_update(dev->intr_handle, irq.count))
+ return -1;
+
+ /* set up an eventfd for interrupts */
+ fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
+ if (fd < 0) {
+ CDX_BUS_ERR("Cannot set up eventfd, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ if (rte_intr_fd_set(dev->intr_handle, fd))
+ return -1;
+
+ /* DPDK CDX bus currently supports only MSI-X */
+ if (rte_intr_type_set(dev->intr_handle, RTE_INTR_HANDLE_VFIO_MSIX))
+ return -1;
+
+ if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
+ return -1;
+
+ return 0;
+ }
+
+ /* if we're here, we haven't found a suitable interrupt vector */
+ return -1;
+}
+
+static int
+cdx_vfio_setup_device(struct rte_cdx_device *dev, int vfio_dev_fd,
+ int num_irqs)
+{
+ if (cdx_vfio_setup_interrupts(dev, vfio_dev_fd, num_irqs) != 0) {
+ CDX_BUS_ERR("Error setting up interrupts!");
+ return -1;
+ }
+
/*
* Reset the device. If the device is not capable of resetting,
* then it updates errno as EINVAL.
@@ -288,6 +396,9 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
struct cdx_map *maps;
int vfio_dev_fd, i, ret;
+ if (rte_intr_fd_set(dev->intr_handle, -1))
+ return -1;
+
ret = rte_vfio_setup_device(rte_cdx_get_sysfs_path(), dev_name,
&vfio_dev_fd, &device_info);
if (ret)
@@ -353,7 +464,7 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
free(reg);
}
- if (cdx_rte_vfio_setup_device(vfio_dev_fd) < 0) {
+ if (cdx_vfio_setup_device(dev, vfio_dev_fd, device_info.num_irqs) < 0) {
CDX_BUS_ERR("%s setup device failed", dev_name);
goto err_vfio_res;
}
@@ -383,6 +494,9 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
const char *dev_name = dev->device.name;
struct cdx_map *maps;
+ if (rte_intr_fd_set(dev->intr_handle, -1))
+ return -1;
+
/* if we're in a secondary process, just find our tailq entry */
TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
if (strcmp(vfio_res->name, dev_name))
@@ -416,6 +530,10 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
dev->mem_resource[i].len = maps[i].size;
}
+ /* we need save vfio_dev_fd, so it can be used during release */
+ if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
+ goto err_vfio_dev_fd;
+
return 0;
err_vfio_dev_fd:
rte_vfio_release_device(rte_cdx_get_sysfs_path(),
@@ -435,3 +553,63 @@ cdx_vfio_map_resource(struct rte_cdx_device *dev)
else
return cdx_vfio_map_resource_secondary(dev);
}
+
+int
+rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle)
+{
+ char irq_set_buf[MSI_IRQ_SET_BUF_LEN];
+ struct vfio_irq_set *irq_set;
+ int *fd_ptr, vfio_dev_fd, i;
+ int ret;
+
+ irq_set = (struct vfio_irq_set *) irq_set_buf;
+ irq_set->count = rte_intr_irq_count_get(intr_handle);
+ irq_set->argsz = sizeof(struct vfio_irq_set) +
+ (sizeof(int) * irq_set->count);
+
+ irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER;
+ irq_set->index = 0;
+ irq_set->start = 0;
+ fd_ptr = (int *) &irq_set->data;
+
+ for (i = 0; i < rte_intr_nb_efd_get(intr_handle); i++)
+ fd_ptr[i] = rte_intr_efds_index_get(intr_handle, i);
+
+ vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
+
+ if (ret) {
+ CDX_BUS_ERR("Error enabling MSI interrupts for fd %d",
+ rte_intr_fd_get(intr_handle));
+ return -1;
+ }
+
+ return 0;
+}
+
+/* disable MSI interrupts */
+int
+rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle)
+{
+ struct vfio_irq_set *irq_set;
+ char irq_set_buf[MSI_IRQ_SET_BUF_LEN];
+ int len, ret, vfio_dev_fd;
+
+ len = sizeof(struct vfio_irq_set);
+
+ irq_set = (struct vfio_irq_set *) irq_set_buf;
+ irq_set->argsz = len;
+ irq_set->count = 0;
+ irq_set->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_TRIGGER;
+ irq_set->index = 0;
+ irq_set->start = 0;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
+
+ if (ret)
+ CDX_BUS_ERR("Error disabling MSI interrupts for fd %d",
+ rte_intr_fd_get(intr_handle));
+
+ return ret;
+}
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
index 957fcab978..2f3d484ebd 100644
--- a/drivers/bus/cdx/version.map
+++ b/drivers/bus/cdx/version.map
@@ -6,6 +6,8 @@ INTERNAL {
rte_cdx_register;
rte_cdx_unmap_device;
rte_cdx_unregister;
+ rte_cdx_vfio_intr_disable;
+ rte_cdx_vfio_intr_enable;
local: *;
};
diff --git a/lib/eal/common/eal_common_interrupts.c b/lib/eal/common/eal_common_interrupts.c
index 97b64fed58..a0167d9ad4 100644
--- a/lib/eal/common/eal_common_interrupts.c
+++ b/lib/eal/common/eal_common_interrupts.c
@@ -398,6 +398,27 @@ int rte_intr_elist_index_set(struct rte_intr_handle *intr_handle,
return -rte_errno;
}
+int rte_intr_irq_count_set(struct rte_intr_handle *intr_handle,
+ int irq_count)
+{
+ CHECK_VALID_INTR_HANDLE(intr_handle);
+
+ intr_handle->irq_count = irq_count;
+
+ return 0;
+fail:
+ return -rte_errno;
+}
+
+int rte_intr_irq_count_get(const struct rte_intr_handle *intr_handle)
+{
+ CHECK_VALID_INTR_HANDLE(intr_handle);
+
+ return intr_handle->irq_count;
+fail:
+ return -rte_errno;
+}
+
int rte_intr_vec_list_alloc(struct rte_intr_handle *intr_handle,
const char *name, int size)
{
diff --git a/lib/eal/common/eal_interrupts.h b/lib/eal/common/eal_interrupts.h
index 482781b862..237f471a76 100644
--- a/lib/eal/common/eal_interrupts.h
+++ b/lib/eal/common/eal_interrupts.h
@@ -16,6 +16,7 @@ struct rte_intr_handle {
};
uint32_t alloc_flags; /**< flags passed at allocation */
enum rte_intr_handle_type type; /**< handle type */
+ uint32_t irq_count; /**< IRQ count provided via VFIO */
uint32_t max_intr; /**< max interrupt requested */
uint32_t nb_efd; /**< number of available efd(event fd) */
uint8_t efd_counter_size; /**< size of efd counter, used for vdev */
diff --git a/lib/eal/include/rte_interrupts.h b/lib/eal/include/rte_interrupts.h
index 487e3c8875..bc477a483f 100644
--- a/lib/eal/include/rte_interrupts.h
+++ b/lib/eal/include/rte_interrupts.h
@@ -506,6 +506,38 @@ __rte_internal
int
rte_intr_max_intr_get(const struct rte_intr_handle *intr_handle);
+/**
+ * @internal
+ * Set the irq count field of interrupt handle with user
+ * provided irq count value.
+ *
+ * @param intr_handle
+ * pointer to the interrupt handle.
+ * @param irq_count
+ * IRQ count
+ *
+ * @return
+ * - On success, zero.
+ * - On failure, a negative value and rte_errno is set.
+ */
+__rte_internal
+int
+rte_intr_irq_count_set(struct rte_intr_handle *intr_handle, int irq_count);
+
+/**
+ * @internal
+ * Returns the irq count field of the given interrupt handle instance.
+ *
+ * @param intr_handle
+ * pointer to the interrupt handle.
+ *
+ * @return
+ * - On success, ir count.
+ * - On failure, a negative value and rte_errno is set.
+ */
+__rte_internal
+int rte_intr_irq_count_get(const struct rte_intr_handle *intr_handle);
+
/**
* @internal
* Set the number of event fd field of interrupt handle
diff --git a/lib/eal/version.map b/lib/eal/version.map
index 6d6978f108..14bf7ade77 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -458,6 +458,8 @@ INTERNAL {
rte_intr_instance_dup;
rte_intr_instance_windows_handle_get;
rte_intr_instance_windows_handle_set;
+ rte_intr_irq_count_get;
+ rte_intr_irq_count_set;
rte_intr_max_intr_get;
rte_intr_max_intr_set;
rte_intr_nb_efd_get;
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v2 3/6] bus/cdx: add support for MSI
2023-04-13 13:27 ` [PATCH v2 3/6] bus/cdx: add support for MSI Nipun Gupta
@ 2023-04-14 16:45 ` Ferruh Yigit
2023-04-16 9:20 ` Gupta, Nipun
0 siblings, 1 reply; 102+ messages in thread
From: Ferruh Yigit @ 2023-04-14 16:45 UTC (permalink / raw)
To: Nipun Gupta, dev, thomas, david.marchand; +Cc: harpreet.anand, nikhil.agarwal
On 4/13/2023 2:27 PM, Nipun Gupta wrote:
> MSI's are exposed to the devices using VFIO (vfio-cdx). This
> patch uses the same to add support for MSI for the devices on
> the cdx bus.
>
> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
This commit also adds two new internal eal APIs and extends "struct
rte_intr_handle", can you please document this in the commit log?
^ permalink raw reply [flat|nested] 102+ messages in thread
* RE: [PATCH v2 3/6] bus/cdx: add support for MSI
2023-04-14 16:45 ` Ferruh Yigit
@ 2023-04-16 9:20 ` Gupta, Nipun
0 siblings, 0 replies; 102+ messages in thread
From: Gupta, Nipun @ 2023-04-16 9:20 UTC (permalink / raw)
To: Yigit, Ferruh, dev, thomas, david.marchand
Cc: Anand, Harpreet, Agarwal, Nikhil
> -----Original Message-----
> From: Yigit, Ferruh <Ferruh.Yigit@amd.com>
> Sent: Friday, April 14, 2023 10:15 PM
> To: Gupta, Nipun <Nipun.Gupta@amd.com>; dev@dpdk.org;
> thomas@monjalon.net; david.marchand@redhat.com
> Cc: Anand, Harpreet <harpreet.anand@amd.com>; Agarwal, Nikhil
> <nikhil.agarwal@amd.com>
> Subject: Re: [PATCH v2 3/6] bus/cdx: add support for MSI
>
> On 4/13/2023 2:27 PM, Nipun Gupta wrote:
> > MSI's are exposed to the devices using VFIO (vfio-cdx). This
> > patch uses the same to add support for MSI for the devices on
> > the cdx bus.
> >
> > Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
>
> This commit also adds two new internal eal APIs and extends "struct
> rte_intr_handle", can you please document this in the commit log?
Sure. Will update.
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v2 4/6] bus/cdx: support plug unplug and dev iterator
2023-04-13 13:26 ` [PATCH v2 " Nipun Gupta
` (2 preceding siblings ...)
2023-04-13 13:27 ` [PATCH v2 3/6] bus/cdx: add support for MSI Nipun Gupta
@ 2023-04-13 13:27 ` Nipun Gupta
2023-04-13 13:27 ` [PATCH v2 5/6] bus: enable cdx bus Nipun Gupta
` (2 subsequent siblings)
6 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-04-13 13:27 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
This change adds support for plugging and unplugging
CDX devices on the CDX bus. Also, CDX dev iterator support
has been added for the CDX bus.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
drivers/bus/cdx/bus_cdx_driver.h | 1 +
drivers/bus/cdx/cdx.c | 122 +++++++++++++++++++++++++++++++
2 files changed, 123 insertions(+)
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index fdeaf46664..3d89e7c054 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -69,6 +69,7 @@ struct rte_cdx_id {
struct rte_cdx_device {
RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
struct rte_device device; /**< Inherit core device */
+ struct rte_cdx_driver *driver; /**< CDX driver used in probing */
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
/**< CDX Memory Resource */
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index f34736a54a..e69ee503aa 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -68,6 +68,7 @@
#include <rte_eal_paging.h>
#include <rte_errno.h>
#include <rte_devargs.h>
+#include <rte_kvargs.h>
#include <rte_malloc.h>
#include <rte_memcpy.h>
#include <rte_vfio.h>
@@ -82,6 +83,15 @@
#define CDX_BUS_NAME cdx
#define CDX_DEV_PREFIX "cdx-"
+enum cdx_params {
+ RTE_CDX_PARAM_NAME,
+};
+
+static const char * const cdx_params_keys[] = {
+ [RTE_CDX_PARAM_NAME] = "name",
+ NULL,
+};
+
/**
* @file
* CDX probing using Linux sysfs.
@@ -400,6 +410,7 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr,
} else {
dev->device.driver = &dr->driver;
}
+ dev->driver = dr;
return ret;
@@ -517,6 +528,71 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
return NULL;
}
+/* Remove a device from CDX bus */
+static void
+cdx_remove_device(struct rte_cdx_device *cdx_dev)
+{
+ TAILQ_REMOVE(&rte_cdx_bus.device_list, cdx_dev, next);
+}
+
+/*
+ * If vendor/device ID match, call the remove() function of the
+ * driver.
+ */
+static int
+cdx_detach_dev(struct rte_cdx_device *dev)
+{
+ struct rte_cdx_driver *dr;
+ int ret = 0;
+
+ if (dev == NULL)
+ return -EINVAL;
+
+ dr = dev->driver;
+
+ CDX_BUS_DEBUG("detach device %s using driver: %s",
+ dev->device.name, dr->driver.name);
+
+ if (dr->remove) {
+ ret = dr->remove(dev);
+ if (ret < 0)
+ return ret;
+ }
+
+ /* clear driver structure */
+ dev->driver = NULL;
+ dev->device.driver = NULL;
+
+ rte_cdx_unmap_device(dev);
+
+ rte_intr_instance_free(dev->intr_handle);
+ dev->intr_handle = NULL;
+
+ return 0;
+}
+
+static int
+cdx_plug(struct rte_device *dev)
+{
+ return cdx_probe_all_drivers(RTE_DEV_TO_CDX_DEV(dev));
+}
+
+static int
+cdx_unplug(struct rte_device *dev)
+{
+ struct rte_cdx_device *cdx_dev;
+ int ret;
+
+ cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+ ret = cdx_detach_dev(cdx_dev);
+ if (ret == 0) {
+ cdx_remove_device(cdx_dev);
+ rte_devargs_remove(dev->devargs);
+ free(cdx_dev);
+ }
+ return ret;
+}
+
static int
cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
{
@@ -554,15 +630,61 @@ cdx_get_iommu_class(void)
return RTE_IOVA_VA;
}
+static int
+cdx_dev_match(const struct rte_device *dev,
+ const void *_kvlist)
+{
+ const struct rte_kvargs *kvlist = _kvlist;
+ const char *key = cdx_params_keys[RTE_CDX_PARAM_NAME];
+ const char *name;
+
+ /* no kvlist arg, all devices match */
+ if (kvlist == NULL)
+ return 0;
+
+ /* if key is present in kvlist and does not match, filter device */
+ name = rte_kvargs_get(kvlist, key);
+ if (name != NULL && strcmp(name, dev->name))
+ return -1;
+
+ return 0;
+}
+
+static void *
+cdx_dev_iterate(const void *start,
+ const char *str,
+ const struct rte_dev_iterator *it __rte_unused)
+{
+ rte_bus_find_device_t find_device;
+ struct rte_kvargs *kvargs = NULL;
+ struct rte_device *dev;
+
+ if (str != NULL) {
+ kvargs = rte_kvargs_parse(str, cdx_params_keys);
+ if (kvargs == NULL) {
+ CDX_BUS_ERR("cannot parse argument list %s", str);
+ rte_errno = EINVAL;
+ return NULL;
+ }
+ }
+ find_device = rte_cdx_bus.bus.find_device;
+ dev = find_device(start, cdx_dev_match, kvargs);
+ rte_kvargs_free(kvargs);
+ return dev;
+}
+
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
.scan = cdx_scan,
.probe = cdx_probe,
.find_device = cdx_find_device,
+ .plug = cdx_plug,
+ .unplug = cdx_unplug,
.parse = cdx_parse,
.dma_map = cdx_dma_map,
.dma_unmap = cdx_dma_unmap,
.get_iommu_class = cdx_get_iommu_class,
+ .dev_iterate = cdx_dev_iterate,
},
.device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
.driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v2 5/6] bus: enable cdx bus
2023-04-13 13:26 ` [PATCH v2 " Nipun Gupta
` (3 preceding siblings ...)
2023-04-13 13:27 ` [PATCH v2 4/6] bus/cdx: support plug unplug and dev iterator Nipun Gupta
@ 2023-04-13 13:27 ` Nipun Gupta
2023-04-14 16:45 ` Ferruh Yigit
2023-04-13 13:27 ` [PATCH v2 6/6] config/arm: add AMD CDX Nipun Gupta
2023-04-14 16:45 ` [PATCH v2 0/6] add support for CDX bus Ferruh Yigit
6 siblings, 1 reply; 102+ messages in thread
From: Nipun Gupta @ 2023-04-13 13:27 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
enable the compilation of cdx bus
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
drivers/bus/meson.build | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
index 6d2520c543..a78b4283bf 100644
--- a/drivers/bus/meson.build
+++ b/drivers/bus/meson.build
@@ -3,6 +3,7 @@
drivers = [
'auxiliary',
+ 'cdx',
'dpaa',
'fslmc',
'ifpga',
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v2 5/6] bus: enable cdx bus
2023-04-13 13:27 ` [PATCH v2 5/6] bus: enable cdx bus Nipun Gupta
@ 2023-04-14 16:45 ` Ferruh Yigit
2023-04-16 9:21 ` Gupta, Nipun
0 siblings, 1 reply; 102+ messages in thread
From: Ferruh Yigit @ 2023-04-14 16:45 UTC (permalink / raw)
To: Nipun Gupta, dev, thomas, david.marchand; +Cc: harpreet.anand, nikhil.agarwal
On 4/13/2023 2:27 PM, Nipun Gupta wrote:
> enable the compilation of cdx bus
>
> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> ---
> drivers/bus/meson.build | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
> index 6d2520c543..a78b4283bf 100644
> --- a/drivers/bus/meson.build
> +++ b/drivers/bus/meson.build
> @@ -3,6 +3,7 @@
>
> drivers = [
> 'auxiliary',
> + 'cdx',
> 'dpaa',
> 'fslmc',
> 'ifpga',
Why not squash this to the first patch?
I think better to enable bus with the first patch and gradually expand
the feature, this way each patch is checked that it builds fine.
There are cases some interim patches can't be built successfully, which
may required enabling after a few patches, but I think this is not the case.
^ permalink raw reply [flat|nested] 102+ messages in thread
* RE: [PATCH v2 5/6] bus: enable cdx bus
2023-04-14 16:45 ` Ferruh Yigit
@ 2023-04-16 9:21 ` Gupta, Nipun
0 siblings, 0 replies; 102+ messages in thread
From: Gupta, Nipun @ 2023-04-16 9:21 UTC (permalink / raw)
To: Yigit, Ferruh, dev, thomas, david.marchand
Cc: Anand, Harpreet, Agarwal, Nikhil
> -----Original Message-----
> From: Yigit, Ferruh <Ferruh.Yigit@amd.com>
> Sent: Friday, April 14, 2023 10:16 PM
> To: Gupta, Nipun <Nipun.Gupta@amd.com>; dev@dpdk.org;
> thomas@monjalon.net; david.marchand@redhat.com
> Cc: Anand, Harpreet <harpreet.anand@amd.com>; Agarwal, Nikhil
> <nikhil.agarwal@amd.com>
> Subject: Re: [PATCH v2 5/6] bus: enable cdx bus
>
> On 4/13/2023 2:27 PM, Nipun Gupta wrote:
> > enable the compilation of cdx bus
> >
> > Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> > ---
> > drivers/bus/meson.build | 1 +
> > 1 file changed, 1 insertion(+)
> >
> > diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
> > index 6d2520c543..a78b4283bf 100644
> > --- a/drivers/bus/meson.build
> > +++ b/drivers/bus/meson.build
> > @@ -3,6 +3,7 @@
> >
> > drivers = [
> > 'auxiliary',
> > + 'cdx',
> > 'dpaa',
> > 'fslmc',
> > 'ifpga',
>
> Why not squash this to the first patch?
>
> I think better to enable bus with the first patch and gradually expand
> the feature, this way each patch is checked that it builds fine.
>
> There are cases some interim patches can't be built successfully, which
> may required enabling after a few patches, but I think this is not the case.
Sure, I will merge into the first patch. Compilation should be working there too.
Thanks,
Nipun
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v2 6/6] config/arm: add AMD CDX
2023-04-13 13:26 ` [PATCH v2 " Nipun Gupta
` (4 preceding siblings ...)
2023-04-13 13:27 ` [PATCH v2 5/6] bus: enable cdx bus Nipun Gupta
@ 2023-04-13 13:27 ` Nipun Gupta
2023-04-14 16:45 ` [PATCH v2 0/6] add support for CDX bus Ferruh Yigit
6 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-04-13 13:27 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
Adding support for AMD CDX devices
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
config/arm/arm64_cdx_linux_gcc | 17 +++++++++++++++++
config/arm/meson.build | 14 ++++++++++++++
2 files changed, 31 insertions(+)
create mode 100644 config/arm/arm64_cdx_linux_gcc
diff --git a/config/arm/arm64_cdx_linux_gcc b/config/arm/arm64_cdx_linux_gcc
new file mode 100644
index 0000000000..8e6d619dae
--- /dev/null
+++ b/config/arm/arm64_cdx_linux_gcc
@@ -0,0 +1,17 @@
+[binaries]
+c = ['ccache', 'aarch64-linux-gnu-gcc']
+cpp = ['ccache', 'aarch64-linux-gnu-g++']
+ar = 'aarch64-linux-gnu-ar'
+as = 'aarch64-linux-gnu-as'
+strip = 'aarch64-linux-gnu-strip'
+pkgconfig = 'aarch64-linux-gnu-pkg-config'
+pcap-config = ''
+
+[host_machine]
+system = 'linux'
+cpu_family = 'aarch64'
+cpu = 'armv8-a'
+endian = 'little'
+
+[properties]
+platform = 'cdx'
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 5213434ca4..39b8929534 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -305,6 +305,18 @@ soc_bluefield = {
'numa': false
}
+soc_cdx = {
+ 'description': 'AMD CDX',
+ 'implementer': '0x41',
+ 'part_number': '0xd42',
+ 'flags': [
+ ['RTE_MACHINE', '"cdx"'],
+ ['RTE_MAX_LCORE', 16],
+ ['RTE_MAX_NUMA_NODES', 1]
+ ],
+ 'numa': false
+}
+
soc_centriq2400 = {
'description': 'Qualcomm Centriq 2400',
'implementer': '0x51',
@@ -463,6 +475,7 @@ generic_aarch32: Generic un-optimized build for armv8 aarch32 execution mode.
armada: Marvell ARMADA
bluefield: NVIDIA BlueField
bluefield3: NVIDIA BlueField-3
+cdx: AMD CDX
centriq2400: Qualcomm Centriq 2400
cn9k: Marvell OCTEON 9
cn10k: Marvell OCTEON 10
@@ -490,6 +503,7 @@ socs = {
'armada': soc_armada,
'bluefield': soc_bluefield,
'bluefield3': soc_bluefield3,
+ 'cdx': soc_cdx,
'centriq2400': soc_centriq2400,
'cn9k': soc_cn9k,
'cn10k' : soc_cn10k,
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v2 0/6] add support for CDX bus
2023-04-13 13:26 ` [PATCH v2 " Nipun Gupta
` (5 preceding siblings ...)
2023-04-13 13:27 ` [PATCH v2 6/6] config/arm: add AMD CDX Nipun Gupta
@ 2023-04-14 16:45 ` Ferruh Yigit
6 siblings, 0 replies; 102+ messages in thread
From: Ferruh Yigit @ 2023-04-14 16:45 UTC (permalink / raw)
To: Nipun Gupta, dev, thomas, david.marchand; +Cc: harpreet.anand, nikhil.agarwal
On 4/13/2023 2:26 PM, Nipun Gupta wrote:
> Support AMD CDX bus, for FPGA based CDX devices. The CDX
> devices are memory mapped on system bus for embedded CPUs.
>
> It uses sysfs interface and the vfio-cdx driver to discover
> and initialize the CDX devices.
>
> The patches are intended for DPDK 23.07 release, and have been sent
> as an RFC as patches are yet to be merged in Linux.
>
not RFC anymore
> The CDX bus and VFIO support is available at Xilinx open source tree:
> https://github.com/Xilinx/linux-xlnx (drivers/cdx/ and drivers/vfio/cdx)
>
> Linux CDX bus patches has been added into linux next:
> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/cdx
>
> VFIO patches are also submitted in upstream:
> https://www.spinics.net/lists/kvm/msg310623.html
>
> CDX is a Hardware Architecture designed for AMD FPGA devices. It
> consists of mechanism for interaction between FPGA, Firmware and
> the APUs (Application CPUs).
> Firmware resides on RPU (Realtime CPUs) which interacts with
> the FPGA program manager and the APUs. The RPU provides memory-mapped
> interface (RPU if) which is used to communicate with APUs.
>
> VFIO CDX driver provides the CDX device resources like MMIO and interrupts
> to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
> driver to discover and initialize the CDX devices for user-space
> applications.
>
Overall looks good to me,
there are a few warnings by `check-git-log.sh`, can you please check?
Also what do you think to add a release notes update for new bus?
> Changes v1->v2:
> - Moved file rte_cdx_bus.h to internal bus_cdx_driver.h
> and added this file to deivce_cdx_headers
> - Moved cdx.h to private.h
> - Removed rte_ prefix from the static symbols in .c files.
>
> Changes RFC->v1:
> - Marked few API's as internal which were not required
> to be provided to user.
>
> Nipun Gupta (6):
> bus/cdx: introduce cdx bus
> bus/cdx: add dma map and unmap support
> bus/cdx: add support for MSI
> bus/cdx: support plug unplug and dev iterator
> bus: enable cdx bus
> config/arm: add AMD CDX
>
> MAINTAINERS | 5 +
> config/arm/arm64_cdx_linux_gcc | 17 +
> config/arm/meson.build | 14 +
> drivers/bus/cdx/bus_cdx_driver.h | 227 ++++++++
> drivers/bus/cdx/cdx.c | 694 +++++++++++++++++++++++++
> drivers/bus/cdx/cdx_logs.h | 37 ++
> drivers/bus/cdx/cdx_vfio.c | 619 ++++++++++++++++++++++
> drivers/bus/cdx/meson.build | 12 +
> drivers/bus/cdx/private.h | 49 ++
> drivers/bus/cdx/version.map | 13 +
> drivers/bus/meson.build | 1 +
> lib/eal/common/eal_common_interrupts.c | 21 +
> lib/eal/common/eal_interrupts.h | 1 +
> lib/eal/include/rte_interrupts.h | 32 ++
> lib/eal/version.map | 2 +
> 15 files changed, 1744 insertions(+)
> create mode 100644 config/arm/arm64_cdx_linux_gcc
> create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
> create mode 100644 drivers/bus/cdx/cdx.c
> create mode 100644 drivers/bus/cdx/cdx_logs.h
> create mode 100644 drivers/bus/cdx/cdx_vfio.c
> create mode 100644 drivers/bus/cdx/meson.build
> create mode 100644 drivers/bus/cdx/private.h
> create mode 100644 drivers/bus/cdx/version.map
>
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v3 0/5] Support AMD CDX bus, for FPGA based CDX devices. The CDX
2023-01-24 14:07 [RFC PATCH 0/6] add support for CDX bus Nipun Gupta
` (7 preceding siblings ...)
2023-04-13 13:26 ` [PATCH v2 " Nipun Gupta
@ 2023-04-21 14:54 ` Nipun Gupta
2023-04-21 14:54 ` [PATCH v3 1/5] bus/cdx: introduce cdx bus Nipun Gupta
` (4 more replies)
2023-05-08 11:18 ` [PATCH v4 0/4] Support AMD CDX bus, for FPGA based CDX devices. The CDX Nipun Gupta
` (4 subsequent siblings)
13 siblings, 5 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-04-21 14:54 UTC (permalink / raw)
To: dev, thomas, david.marchand, ferruh.yigit
Cc: harpreet.anand, nikhil.agarwal, Nipun Gupta
It uses sysfs interface and the vfio-cdx driver to discover
and initialize the CDX devices.
The CDX bus and VFIO support is available at Xilinx open source tree:
https://github.com/Xilinx/linux-xlnx (drivers/cdx/ and drivers/vfio/cdx)
Linux CDX bus patches has been added into linux next:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/cdx
VFIO patches are also submitted in upstream:
https://www.spinics.net/lists/kvm/msg310623.html
CDX is a Hardware Architecture designed for AMD FPGA devices. It
consists of mechanism for interaction between FPGA, Firmware and
the APUs (Application CPUs).
Firmware resides on RPU (Realtime CPUs) which interacts with
the FPGA program manager and the APUs. The RPU provides memory-mapped
interface (RPU if) which is used to communicate with APUs.
VFIO CDX driver provides the CDX device resources like MMIO and interrupts
to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
driver to discover and initialize the CDX devices for user-space
applications.
Changes v2->v3:
- merged cdx bus compilation enablement in the first patch
- fixed issue reported by check-git-log.sh
- updated release notes
- updated offset to uint64_t instead of off_t in cdx_map_resource
Changes v1->v2:
- Moved file rte_cdx_bus.h to internal bus_cdx_driver.h
and added this file to deivce_cdx_headers
- Moved cdx.h to private.h
- Removed rte_ prefix from the static symbols in .c files.
Changes RFC->v1:
- Marked few API's as internal which were not required
to be provided to user.
Nipun Gupta (5):
bus/cdx: introduce cdx bus
bus/cdx: add DMA map and unmap support
bus/cdx: add support for MSI
bus/cdx: support plug unplug and dev iterator
config/arm: add AMD CDX
MAINTAINERS | 5 +
config/arm/arm64_cdx_linux_gcc | 17 +
config/arm/meson.build | 14 +
doc/guides/rel_notes/release_23_07.rst | 6 +
drivers/bus/cdx/bus_cdx_driver.h | 227 ++++++++
drivers/bus/cdx/cdx.c | 693 +++++++++++++++++++++++++
drivers/bus/cdx/cdx_logs.h | 37 ++
drivers/bus/cdx/cdx_vfio.c | 615 ++++++++++++++++++++++
drivers/bus/cdx/meson.build | 13 +
drivers/bus/cdx/private.h | 49 ++
drivers/bus/cdx/version.map | 13 +
drivers/bus/meson.build | 1 +
lib/eal/common/eal_common_interrupts.c | 21 +
lib/eal/common/eal_interrupts.h | 1 +
lib/eal/include/rte_interrupts.h | 32 ++
lib/eal/version.map | 2 +
16 files changed, 1746 insertions(+)
create mode 100644 config/arm/arm64_cdx_linux_gcc
create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
create mode 100644 drivers/bus/cdx/cdx.c
create mode 100644 drivers/bus/cdx/cdx_logs.h
create mode 100644 drivers/bus/cdx/cdx_vfio.c
create mode 100644 drivers/bus/cdx/meson.build
create mode 100644 drivers/bus/cdx/private.h
create mode 100644 drivers/bus/cdx/version.map
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v3 1/5] bus/cdx: introduce cdx bus
2023-04-21 14:54 ` [PATCH v3 0/5] Support AMD CDX bus, for FPGA based CDX devices. The CDX Nipun Gupta
@ 2023-04-21 14:54 ` Nipun Gupta
2023-04-21 14:54 ` [PATCH v3 2/5] bus/cdx: add DMA map and unmap support Nipun Gupta
` (3 subsequent siblings)
4 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-04-21 14:54 UTC (permalink / raw)
To: dev, thomas, david.marchand, ferruh.yigit
Cc: harpreet.anand, nikhil.agarwal, Nipun Gupta
CDX bus supports multiple type of devices, which can be
exposed to user-space via vfio-cdx.
vfio-cdx provides the MMIO IO_MEMORY regions as well as the
DMA interface for the device (IOMMU).
This support aims to enable the DPDK to support the cdx
devices in user-space using VFIO interface.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
MAINTAINERS | 5 +
doc/guides/rel_notes/release_23_07.rst | 6 +
drivers/bus/cdx/bus_cdx_driver.h | 201 ++++++++++
drivers/bus/cdx/cdx.c | 520 +++++++++++++++++++++++++
drivers/bus/cdx/cdx_logs.h | 37 ++
drivers/bus/cdx/cdx_vfio.c | 437 +++++++++++++++++++++
drivers/bus/cdx/meson.build | 13 +
drivers/bus/cdx/private.h | 49 +++
drivers/bus/cdx/version.map | 11 +
drivers/bus/meson.build | 1 +
10 files changed, 1280 insertions(+)
create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
create mode 100644 drivers/bus/cdx/cdx.c
create mode 100644 drivers/bus/cdx/cdx_logs.h
create mode 100644 drivers/bus/cdx/cdx_vfio.c
create mode 100644 drivers/bus/cdx/meson.build
create mode 100644 drivers/bus/cdx/private.h
create mode 100644 drivers/bus/cdx/version.map
diff --git a/MAINTAINERS b/MAINTAINERS
index 8df23e5099..1f9b6af9b9 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -569,6 +569,11 @@ M: Parav Pandit <parav@nvidia.com>
M: Xueming Li <xuemingl@nvidia.com>
F: drivers/bus/auxiliary/
+CDX bus driver
+M: Nipun Gupta <nipun.gupta@amd.com>
+M: Nikhil Agarwal <nikhil.agarwal@amd.com>
+F: drivers/bus/cdx/
+
Intel FPGA bus
M: Rosen Xu <rosen.xu@intel.com>
F: drivers/bus/ifpga/
diff --git a/doc/guides/rel_notes/release_23_07.rst b/doc/guides/rel_notes/release_23_07.rst
index a9b1293689..a5f581dc3d 100644
--- a/doc/guides/rel_notes/release_23_07.rst
+++ b/doc/guides/rel_notes/release_23_07.rst
@@ -55,6 +55,12 @@ New Features
Also, make sure to start the actual text at the margin.
=======================================================
+* **Added CDX bus support.**
+
+ CDX bus driver has been added to support AMD CDX bus, which operates
+ on FPGA based CDX devices. The CDX devices are memory mapped on system
+ bus for embedded CPUs.
+
Removed Items
-------------
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
new file mode 100644
index 0000000000..7edcb019eb
--- /dev/null
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -0,0 +1,201 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef _BUS_CDX_DRIVER_H_
+#define _BUS_CDX_DRIVER_H_
+
+/**
+ * @file
+ *
+ * CDX bus interface
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <limits.h>
+#include <errno.h>
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <bus_driver.h>
+#include <dev_driver.h>
+#include <rte_debug.h>
+#include <rte_interrupts.h>
+#include <rte_dev.h>
+#include <rte_bus.h>
+
+/* Forward declarations */
+struct rte_cdx_device;
+struct rte_cdx_driver;
+
+#define CDX_MAX_RESOURCE 4
+
+/** List of CDX devices */
+RTE_TAILQ_HEAD(rte_cdx_device_list, rte_cdx_device);
+/** List of CDX drivers */
+RTE_TAILQ_HEAD(rte_cdx_driver_list, rte_cdx_driver);
+
+/* CDX Bus iterators */
+#define FOREACH_DEVICE_ON_CDXBUS(p) \
+ RTE_TAILQ_FOREACH(p, &rte_cdx_bus.device_list, next)
+
+#define FOREACH_DRIVER_ON_CDXBUS(p) \
+ RTE_TAILQ_FOREACH(p, &rte_cdx_bus.driver_list, next)
+
+/** Any CDX device identifier (vendor, device) */
+#define RTE_CDX_ANY_ID (0xffff)
+
+#define RTE_PMD_REGISTER_CDX_TABLE(name, table) \
+static const char DRV_EXP_TAG(name, cdx_tbl_export)[] __rte_used = \
+RTE_STR(table)
+
+/**
+ * A structure describing an ID for a CDX driver. Each driver provides a
+ * table of these IDs for each device that it supports.
+ */
+struct rte_cdx_id {
+ uint16_t vendor_id; /**< Vendor ID. */
+ uint16_t device_id; /**< Device ID. */
+};
+
+/**
+ * A structure describing a CDX device.
+ */
+struct rte_cdx_device {
+ RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
+ struct rte_device device; /**< Inherit core device */
+ struct rte_cdx_id id; /**< CDX ID. */
+ struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
+ /**< CDX Memory Resource */
+};
+
+/**
+ * @internal
+ * Helper macro for drivers that need to convert to struct rte_cdx_device.
+ */
+#define RTE_DEV_TO_CDX_DEV(ptr) \
+ container_of(ptr, struct rte_cdx_device, device)
+
+#define RTE_DEV_TO_CDX_DEV_CONST(ptr) \
+ container_of(ptr, const struct rte_cdx_device, device)
+
+#define RTE_ETH_DEV_TO_CDX_DEV(eth_dev) RTE_DEV_TO_CDX_DEV((eth_dev)->device)
+
+#ifdef __cplusplus
+/** C++ macro used to help building up tables of device IDs */
+#define RTE_CDX_DEVICE(vend, dev) \
+ (vend), \
+ (dev)
+#else
+/** Macro used to help building up tables of device IDs */
+#define RTE_CDX_DEVICE(vend, dev) \
+ .vendor_id = (vend), \
+ .device_id = (dev)
+#endif
+
+/**
+ * Initialisation function for the driver called during CDX probing.
+ */
+typedef int (rte_cdx_probe_t)(struct rte_cdx_driver *, struct rte_cdx_device *);
+
+/**
+ * Uninitialisation function for the driver called during hotplugging.
+ */
+typedef int (rte_cdx_remove_t)(struct rte_cdx_device *);
+
+/**
+ * A structure describing a CDX driver.
+ */
+struct rte_cdx_driver {
+ RTE_TAILQ_ENTRY(rte_cdx_driver) next; /**< Next in list. */
+ struct rte_driver driver; /**< Inherit core driver. */
+ struct rte_cdx_bus *bus; /**< CDX bus reference. */
+ rte_cdx_probe_t *probe; /**< Device probe function. */
+ rte_cdx_remove_t *remove; /**< Device remove function. */
+ const struct rte_cdx_id *id_table; /**< ID table, NULL terminated. */
+ uint32_t drv_flags; /**< Flags RTE_CDX_DRV_*. */
+};
+
+/**
+ * Structure describing the CDX bus
+ */
+struct rte_cdx_bus {
+ struct rte_bus bus; /**< Inherit the generic class */
+ struct rte_cdx_device_list device_list; /**< List of CDX devices */
+ struct rte_cdx_driver_list driver_list; /**< List of CDX drivers */
+};
+
+/**
+ * Get Pathname of CDX devices directory.
+ *
+ * @return
+ * sysfs path for CDX devices.
+ */
+__rte_internal
+const char *rte_cdx_get_sysfs_path(void);
+
+/**
+ * Map the CDX device resources in user space virtual memory address
+ *
+ * @param dev
+ * A pointer to a rte_cdx_device structure describing the device
+ * to use
+ *
+ * @return
+ * 0 on success, negative on error and positive if no driver
+ * is found for the device.
+ */
+__rte_internal
+int rte_cdx_map_device(struct rte_cdx_device *dev);
+
+/**
+ * Unmap this device
+ *
+ * @param dev
+ * A pointer to a rte_cdx_device structure describing the device
+ * to use
+ */
+__rte_internal
+void rte_cdx_unmap_device(struct rte_cdx_device *dev);
+
+/**
+ * Register a CDX driver.
+ *
+ * @param driver
+ * A pointer to a rte_cdx_driver structure describing the driver
+ * to be registered.
+ */
+__rte_internal
+void rte_cdx_register(struct rte_cdx_driver *driver);
+
+/**
+ * Helper for CDX device registration from driver (eth, crypto, raw) instance
+ */
+#define RTE_PMD_REGISTER_CDX(nm, cdx_drv) \
+ RTE_INIT(cdxinitfn_ ##nm) \
+ {\
+ (cdx_drv).driver.name = RTE_STR(nm);\
+ rte_cdx_register(&cdx_drv); \
+ } \
+ RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+/**
+ * Unregister a CDX driver.
+ *
+ * @param driver
+ * A pointer to a rte_cdx_driver structure describing the driver
+ * to be unregistered.
+ */
+__rte_internal
+void rte_cdx_unregister(struct rte_cdx_driver *driver);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _BUS_CDX_DRIVER_H_ */
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
new file mode 100644
index 0000000000..d479daa315
--- /dev/null
+++ b/drivers/bus/cdx/cdx.c
@@ -0,0 +1,520 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+/*
+ * Architecture Overview
+ * =====================
+ * CDX is a Hardware Architecture designed for AMD FPGA devices. It
+ * consists of sophisticated mechanism for interaction between FPGA,
+ * Firmware and the APUs (Application CPUs).
+ *
+ * Firmware resides on RPU (Realtime CPUs) which interacts with
+ * the FPGA program manager and the APUs. The RPU provides memory-mapped
+ * interface (RPU if) which is used to communicate with APUs.
+ *
+ * The diagram below shows an overview of the CDX architecture:
+ *
+ * +--------------------------------------+
+ * | DPDK |
+ * | DPDK CDX drivers |
+ * | | |
+ * | DPDK CDX bus |
+ * | | |
+ * +-----------------------------|--------+
+ * |
+ * +-----------------------------|--------+
+ * | Application CPUs (APU) | |
+ * | | |
+ * | VFIO CDX driver |
+ * | Linux OS | |
+ * | Linux CDX bus |
+ * | | |
+ * +-----------------------------|--------+
+ * |
+ * |
+ * +------------------------| RPU if |----+
+ * | | |
+ * | V |
+ * | Realtime CPUs (RPU) |
+ * | |
+ * +--------------------------------------+
+ * |
+ * +---------------------|----------------+
+ * | FPGA | |
+ * | +-----------------------+ |
+ * | | | | |
+ * | +-------+ +-------+ +-------+ |
+ * | | dev 1 | | dev 2 | | dev 3 | |
+ * | +-------+ +-------+ +-------+ |
+ * +--------------------------------------+
+ *
+ * The RPU firmware extracts the device information from the loaded FPGA
+ * image and implements a mechanism that allows the APU drivers to
+ * enumerate such devices (device personality and resource details) via
+ * a dedicated communication channel.
+ *
+ * VFIO CDX driver provides the CDX device resources like MMIO and interrupts
+ * to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
+ * driver to discover and initialize the CDX devices for user-space
+ * applications.
+ */
+
+#include <string.h>
+#include <dirent.h>
+
+#include <rte_log.h>
+#include <rte_bus.h>
+#include <rte_eal_paging.h>
+#include <rte_errno.h>
+#include <rte_devargs.h>
+#include <rte_malloc.h>
+#include <rte_memcpy.h>
+#include <rte_vfio.h>
+
+#include <eal_filesystem.h>
+
+#include "bus_cdx_driver.h"
+#include "cdx_logs.h"
+#include "private.h"
+
+#define SYSFS_CDX_DEVICES "/sys/bus/cdx/devices"
+#define CDX_BUS_NAME cdx
+#define CDX_DEV_PREFIX "cdx-"
+
+/**
+ * @file
+ * CDX probing using Linux sysfs.
+ */
+
+/* Add a device to CDX bus */
+static void
+cdx_add_device(struct rte_cdx_device *cdx_dev)
+{
+ TAILQ_INSERT_TAIL(&rte_cdx_bus.device_list, cdx_dev, next);
+}
+
+static int
+cdx_get_kernel_driver_by_path(const char *filename, char *driver_name,
+ size_t len)
+{
+ int count;
+ char path[PATH_MAX];
+ char *name;
+
+ if (!filename || !driver_name)
+ return -1;
+
+ count = readlink(filename, path, PATH_MAX);
+ if (count >= PATH_MAX)
+ return -1;
+
+ /* For device does not have a driver */
+ if (count < 0)
+ return 1;
+
+ path[count] = '\0';
+
+ name = strrchr(path, '/');
+ if (name) {
+ strlcpy(driver_name, name + 1, len);
+ return 0;
+ }
+
+ return -1;
+}
+
+int rte_cdx_map_device(struct rte_cdx_device *dev)
+{
+ return cdx_vfio_map_resource(dev);
+}
+
+void rte_cdx_unmap_device(struct rte_cdx_device *dev)
+{
+ cdx_vfio_unmap_resource(dev);
+}
+
+static struct rte_devargs *
+cdx_devargs_lookup(const char *dev_name)
+{
+ struct rte_devargs *devargs;
+
+ RTE_EAL_DEVARGS_FOREACH("cdx", devargs) {
+ if (strcmp(devargs->name, dev_name) == 0)
+ return devargs;
+ }
+ return NULL;
+}
+
+static bool
+cdx_ignore_device(const char *dev_name)
+{
+ struct rte_devargs *devargs = cdx_devargs_lookup(dev_name);
+
+ switch (rte_cdx_bus.bus.conf.scan_mode) {
+ case RTE_BUS_SCAN_ALLOWLIST:
+ if (devargs && devargs->policy == RTE_DEV_ALLOWED)
+ return false;
+ break;
+ case RTE_BUS_SCAN_UNDEFINED:
+ case RTE_BUS_SCAN_BLOCKLIST:
+ if (devargs == NULL || devargs->policy != RTE_DEV_BLOCKED)
+ return false;
+ break;
+ }
+ return true;
+}
+
+/*
+ * Scan one cdx sysfs entry, and fill the devices list from it.
+ * It checks if the CDX device is bound to vfio-cdx driver. In case
+ * the device is vfio bound, it reads the vendor and device id and
+ * stores it for device-driver matching.
+ */
+static int
+cdx_scan_one(const char *dirname, const char *dev_name)
+{
+ char filename[PATH_MAX];
+ struct rte_cdx_device *dev = NULL;
+ char driver[PATH_MAX];
+ unsigned long tmp;
+ char *name = NULL;
+ int ret;
+
+ dev = calloc(1, sizeof(*dev));
+ if (!dev)
+ return -ENOMEM;
+
+ name = calloc(1, RTE_DEV_NAME_MAX_LEN);
+ if (!name) {
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ dev->device.bus = &rte_cdx_bus.bus;
+ memcpy(name, dev_name, RTE_DEV_NAME_MAX_LEN);
+ dev->device.name = name;
+
+ /* parse driver */
+ snprintf(filename, sizeof(filename), "%s/driver", dirname);
+ ret = cdx_get_kernel_driver_by_path(filename, driver, sizeof(driver));
+ if (ret < 0) {
+ CDX_BUS_ERR("Fail to get kernel driver");
+ ret = -1;
+ goto err;
+ }
+
+ /*
+ * Check if device is bound to 'vfio-cdx' driver, so that user-space
+ * can gracefully access the device.
+ */
+ if (ret || strcmp(driver, "vfio-cdx")) {
+ ret = 0;
+ goto err;
+ }
+
+ /* get vendor id */
+ snprintf(filename, sizeof(filename), "%s/vendor", dirname);
+ if (eal_parse_sysfs_value(filename, &tmp) < 0) {
+ ret = -1;
+ goto err;
+ }
+ dev->id.vendor_id = (uint16_t)tmp;
+
+ /* get device id */
+ snprintf(filename, sizeof(filename), "%s/device", dirname);
+ if (eal_parse_sysfs_value(filename, &tmp) < 0) {
+ free(dev);
+ return -1;
+ }
+ dev->id.device_id = (uint16_t)tmp;
+
+ cdx_add_device(dev);
+
+ return 0;
+
+err:
+ if (name)
+ free(name);
+ if (dev)
+ free(dev);
+ return ret;
+}
+
+/*
+ * Scan the content of the CDX bus, and the devices in the devices
+ * list.
+ */
+static int
+cdx_scan(void)
+{
+ struct dirent *e;
+ DIR *dir;
+ char dirname[PATH_MAX];
+
+ dir = opendir(rte_cdx_get_sysfs_path());
+ if (dir == NULL) {
+ CDX_BUS_ERR("%s(): opendir failed: %s", __func__,
+ strerror(errno));
+ return -1;
+ }
+
+ while ((e = readdir(dir)) != NULL) {
+ if (e->d_name[0] == '.')
+ continue;
+
+ if (cdx_ignore_device(e->d_name))
+ continue;
+
+ snprintf(dirname, sizeof(dirname), "%s/%s",
+ rte_cdx_get_sysfs_path(), e->d_name);
+
+ if (cdx_scan_one(dirname, e->d_name) < 0)
+ goto error;
+ }
+ closedir(dir);
+ return 0;
+
+error:
+ closedir(dir);
+ return -1;
+}
+
+const char *
+rte_cdx_get_sysfs_path(void)
+{
+ return SYSFS_CDX_DEVICES;
+}
+
+/* map a particular resource from a file */
+void *
+cdx_map_resource(void *requested_addr, int fd, uint64_t offset, size_t size,
+ int additional_flags)
+{
+ void *mapaddr;
+
+ /* Map the cdx MMIO memory resource of device */
+ mapaddr = rte_mem_map(requested_addr, size,
+ RTE_PROT_READ | RTE_PROT_WRITE,
+ RTE_MAP_SHARED | additional_flags, fd, offset);
+ if (mapaddr == NULL) {
+ CDX_BUS_ERR("%s(): cannot map resource(%d, %p, 0x%zx, 0x%"PRIx64"): %s (%p)",
+ __func__, fd, requested_addr, size, offset,
+ rte_strerror(rte_errno), mapaddr);
+ }
+ CDX_BUS_DEBUG("CDX MMIO memory mapped at %p", mapaddr);
+
+ return mapaddr;
+}
+
+/* unmap a particular resource */
+void
+cdx_unmap_resource(void *requested_addr, size_t size)
+{
+ if (requested_addr == NULL)
+ return;
+
+ /* Unmap the CDX memory resource of device */
+ if (rte_mem_unmap(requested_addr, size)) {
+ CDX_BUS_ERR("%s(): cannot mem unmap(%p, %#zx): %s", __func__,
+ requested_addr, size, rte_strerror(rte_errno));
+ }
+ CDX_BUS_DEBUG("CDX memory unmapped at %p", requested_addr);
+}
+/*
+ * Match the CDX Driver and Device using device id and vendor id.
+ */
+static int
+cdx_match(const struct rte_cdx_driver *cdx_drv,
+ const struct rte_cdx_device *cdx_dev)
+{
+ const struct rte_cdx_id *id_table;
+
+ for (id_table = cdx_drv->id_table; id_table->vendor_id != 0;
+ id_table++) {
+ /* check if device's identifiers match the driver's ones */
+ if (id_table->vendor_id != cdx_dev->id.vendor_id &&
+ id_table->vendor_id != RTE_CDX_ANY_ID)
+ continue;
+ if (id_table->device_id != cdx_dev->id.device_id &&
+ id_table->device_id != RTE_CDX_ANY_ID)
+ continue;
+
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * If vendor id and device id match, call the probe() function of the
+ * driver.
+ */
+static int
+cdx_probe_one_driver(struct rte_cdx_driver *dr,
+ struct rte_cdx_device *dev)
+{
+ const char *dev_name = dev->device.name;
+ bool already_probed;
+ int ret;
+
+ if ((dr == NULL) || (dev == NULL))
+ return -EINVAL;
+
+ /* The device is not blocked; Check if driver supports it */
+ if (!cdx_match(dr, dev))
+ /* Match of device and driver failed */
+ return 1;
+
+ already_probed = rte_dev_is_probed(&dev->device);
+ if (already_probed) {
+ CDX_BUS_INFO("Device %s is already probed", dev->device.name);
+ return -EEXIST;
+ }
+
+ CDX_BUS_DEBUG(" probe device %s using driver: %s", dev_name,
+ dr->driver.name);
+
+ ret = cdx_vfio_map_resource(dev);
+ if (ret != 0) {
+ CDX_BUS_ERR("CDX map device failed: %d", ret);
+ goto error_map_device;
+ }
+
+ /* call the driver probe() function */
+ ret = dr->probe(dr, dev);
+ if (ret) {
+ CDX_BUS_ERR("Probe CDX driver: %s device: %s failed: %d",
+ dr->driver.name, dev_name, ret);
+ goto error_probe;
+ } else {
+ dev->device.driver = &dr->driver;
+ }
+
+ return ret;
+
+error_probe:
+ cdx_vfio_unmap_resource(dev);
+error_map_device:
+ return ret;
+}
+
+/*
+ * If vendor/device ID match, call the probe() function of all
+ * registered driver for the given device. Return < 0 if initialization
+ * failed, return 1 if no driver is found for this device.
+ */
+static int
+cdx_probe_all_drivers(struct rte_cdx_device *dev)
+{
+ struct rte_cdx_driver *dr = NULL;
+ int rc = 0;
+
+ if (dev == NULL)
+ return -EINVAL;
+
+ FOREACH_DRIVER_ON_CDXBUS(dr) {
+ rc = cdx_probe_one_driver(dr, dev);
+ if (rc < 0)
+ /* negative value is an error */
+ return rc;
+ if (rc > 0)
+ /* positive value means driver doesn't support it */
+ continue;
+ return 0;
+ }
+ return 1;
+}
+
+/*
+ * Scan the content of the CDX bus, and call the probe() function for
+ * all registered drivers that have a matching entry in its id_table
+ * for discovered devices.
+ */
+static int
+cdx_probe(void)
+{
+ struct rte_cdx_device *dev = NULL;
+ size_t probed = 0, failed = 0;
+ int ret = 0;
+
+ FOREACH_DEVICE_ON_CDXBUS(dev) {
+ probed++;
+
+ ret = cdx_probe_all_drivers(dev);
+ if (ret < 0) {
+ CDX_BUS_ERR("Requested device %s cannot be used",
+ dev->device.name);
+ rte_errno = errno;
+ failed++;
+ ret = 0;
+ }
+ }
+
+ return (probed && probed == failed) ? -1 : 0;
+}
+
+static int
+cdx_parse(const char *name, void *addr)
+{
+ const char **out = addr;
+ int ret;
+
+ ret = strncmp(name, CDX_DEV_PREFIX, strlen(CDX_DEV_PREFIX));
+
+ if (ret == 0 && addr)
+ *out = name;
+
+ return ret;
+}
+
+/* register a driver */
+void
+rte_cdx_register(struct rte_cdx_driver *driver)
+{
+ TAILQ_INSERT_TAIL(&rte_cdx_bus.driver_list, driver, next);
+ driver->bus = &rte_cdx_bus;
+}
+
+/* unregister a driver */
+void
+rte_cdx_unregister(struct rte_cdx_driver *driver)
+{
+ TAILQ_REMOVE(&rte_cdx_bus.driver_list, driver, next);
+ driver->bus = NULL;
+}
+
+static struct rte_device *
+cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
+ const void *data)
+{
+ const struct rte_cdx_device *cdx_start;
+ struct rte_cdx_device *cdx_dev;
+
+ if (start != NULL) {
+ cdx_start = RTE_DEV_TO_CDX_DEV_CONST(start);
+ cdx_dev = TAILQ_NEXT(cdx_start, next);
+ } else {
+ cdx_dev = TAILQ_FIRST(&rte_cdx_bus.device_list);
+ }
+ while (cdx_dev != NULL) {
+ if (cmp(&cdx_dev->device, data) == 0)
+ return &cdx_dev->device;
+ cdx_dev = TAILQ_NEXT(cdx_dev, next);
+ }
+ return NULL;
+}
+
+struct rte_cdx_bus rte_cdx_bus = {
+ .bus = {
+ .scan = cdx_scan,
+ .probe = cdx_probe,
+ .find_device = cdx_find_device,
+ .parse = cdx_parse,
+ },
+ .device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
+ .driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
+};
+
+RTE_REGISTER_BUS(cdx, rte_cdx_bus.bus);
+RTE_LOG_REGISTER_DEFAULT(cdx_logtype_bus, NOTICE);
diff --git a/drivers/bus/cdx/cdx_logs.h b/drivers/bus/cdx/cdx_logs.h
new file mode 100644
index 0000000000..e5e23d8b94
--- /dev/null
+++ b/drivers/bus/cdx/cdx_logs.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef _CDX_LOGS_H_
+#define _CDX_LOGS_H_
+
+extern int cdx_logtype_bus;
+
+#define CDX_BUS_LOG(level, fmt, args...) \
+ rte_log(RTE_LOG_ ## level, cdx_logtype_bus, "cdx: " fmt "\n", \
+ ##args)
+
+/* Debug logs are with Function names */
+#define CDX_BUS_DEBUG(fmt, args...) \
+ rte_log(RTE_LOG_DEBUG, cdx_logtype_bus, "cdx: %s(): " fmt "\n", \
+ __func__, ##args)
+
+#define CDX_BUS_INFO(fmt, args...) \
+ CDX_BUS_LOG(INFO, fmt, ## args)
+#define CDX_BUS_ERR(fmt, args...) \
+ CDX_BUS_LOG(ERR, fmt, ## args)
+#define CDX_BUS_WARN(fmt, args...) \
+ CDX_BUS_LOG(WARNING, fmt, ## args)
+
+/* DP Logs, toggled out at compile time if level lower than current level */
+#define CDX_BUS_DP_LOG(level, fmt, args...) \
+ RTE_LOG_DP(level, PMD, fmt, ## args)
+
+#define CDX_BUS_DP_DEBUG(fmt, args...) \
+ CDX_BUS_DP_LOG(DEBUG, fmt, ## args)
+#define CDX_BUS_DP_INFO(fmt, args...) \
+ CDX_BUS_DP_LOG(INFO, fmt, ## args)
+#define CDX_BUS_DP_WARN(fmt, args...) \
+ CDX_BUS_DP_LOG(WARNING, fmt, ## args)
+
+#endif /* _CDX_LOGS_H_ */
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
new file mode 100644
index 0000000000..ae11f589b3
--- /dev/null
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -0,0 +1,437 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+/**
+ * @file
+ * CDX probing using Linux VFIO.
+ *
+ * This code tries to determine if the CDX device is bound to VFIO driver,
+ * and initialize it (map MMIO regions, set up interrupts) if that's the case.
+ *
+ */
+
+#include <string.h>
+#include <fcntl.h>
+#include <sys/eventfd.h>
+#include <sys/socket.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <stdbool.h>
+
+#include <rte_log.h>
+#include <rte_eal_paging.h>
+#include <rte_malloc.h>
+#include <rte_vfio.h>
+#include <rte_eal.h>
+#include <rte_bus.h>
+#include <rte_spinlock.h>
+#include <rte_tailq.h>
+
+#include "eal_filesystem.h"
+
+#include "bus_cdx_driver.h"
+#include "cdx_logs.h"
+#include "private.h"
+
+/**
+ * A structure describing a CDX mapping.
+ */
+struct cdx_map {
+ void *addr;
+ char *path;
+ uint64_t offset;
+ uint64_t size;
+};
+
+/**
+ * A structure describing a mapped CDX resource.
+ * For multi-process we need to reproduce all CDX mappings in secondary
+ * processes, so save them in a tailq.
+ */
+struct mapped_cdx_resource {
+ TAILQ_ENTRY(mapped_cdx_resource) next;
+ char name[RTE_DEV_NAME_MAX_LEN]; /**< CDX device name */
+ char path[PATH_MAX];
+ int nb_maps;
+ struct cdx_map maps[CDX_MAX_RESOURCE];
+};
+
+/** mapped cdx device list */
+TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
+
+static struct rte_tailq_elem cdx_vfio_tailq = {
+ .name = "VFIO_CDX_RESOURCE_LIST",
+};
+EAL_REGISTER_TAILQ(cdx_vfio_tailq)
+
+static struct mapped_cdx_resource *
+cdx_vfio_find_and_unmap_resource(struct mapped_cdx_res_list *vfio_res_list,
+ struct rte_cdx_device *dev)
+{
+ struct mapped_cdx_resource *vfio_res = NULL;
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+ int i;
+
+ /* Get vfio_res */
+ TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+ if (strcmp(vfio_res->name, dev_name))
+ continue;
+ break;
+ }
+
+ if (vfio_res == NULL)
+ return vfio_res;
+
+ CDX_BUS_INFO("Releasing CDX mapped resource for %s", dev_name);
+
+ maps = vfio_res->maps;
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ if (maps[i].addr) {
+ CDX_BUS_DEBUG("Calling cdx_unmap_resource for %s at %p",
+ dev_name, maps[i].addr);
+ cdx_unmap_resource(maps[i].addr, maps[i].size);
+ }
+ }
+
+ return vfio_res;
+}
+
+static int
+cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
+{
+ char cdx_addr[PATH_MAX] = {0};
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list;
+
+ vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ vfio_res = cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for cdx device!",
+ cdx_addr);
+ return -1;
+ }
+
+ TAILQ_REMOVE(vfio_res_list, vfio_res, next);
+ rte_free(vfio_res);
+ return 0;
+}
+
+static int
+cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
+{
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list;
+
+ vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ vfio_res = cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for CDX device!",
+ dev->device.name);
+ return -1;
+ }
+
+ return 0;
+}
+
+int
+cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
+{
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return cdx_vfio_unmap_resource_primary(dev);
+ else
+ return cdx_vfio_unmap_resource_secondary(dev);
+}
+
+static int
+cdx_rte_vfio_setup_device(int vfio_dev_fd)
+{
+ /*
+ * Reset the device. If the device is not capable of resetting,
+ * then it updates errno as EINVAL.
+ */
+ if (ioctl(vfio_dev_fd, VFIO_DEVICE_RESET) && errno != EINVAL) {
+ CDX_BUS_ERR("Unable to reset device! Error: %d (%s)", errno,
+ strerror(errno));
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+cdx_vfio_mmap_resource(int vfio_dev_fd, struct mapped_cdx_resource *vfio_res,
+ int index, int additional_flags)
+{
+ struct cdx_map *map = &vfio_res->maps[index];
+ void *vaddr;
+
+ if (map->size == 0) {
+ CDX_BUS_DEBUG("map size is 0, skip region %d", index);
+ return 0;
+ }
+
+ /* reserve the address using an inaccessible mapping */
+ vaddr = mmap(map->addr, map->size, 0, MAP_PRIVATE |
+ MAP_ANONYMOUS | additional_flags, -1, 0);
+ if (vaddr != MAP_FAILED) {
+ void *map_addr = NULL;
+
+ if (map->size) {
+ /* actual map of first part */
+ map_addr = cdx_map_resource(vaddr, vfio_dev_fd,
+ map->offset, map->size,
+ RTE_MAP_FORCE_ADDRESS);
+ }
+
+ if (map_addr == NULL) {
+ munmap(vaddr, map->size);
+ vaddr = MAP_FAILED;
+ CDX_BUS_ERR("Failed to map cdx MMIO region %d", index);
+ return -1;
+ }
+ } else {
+ CDX_BUS_ERR("Failed to create inaccessible mapping for MMIO region %d",
+ index);
+ return -1;
+ }
+
+ map->addr = vaddr;
+ return 0;
+}
+
+/*
+ * region info may contain capability headers, so we need to keep reallocating
+ * the memory until we match allocated memory size with argsz.
+ */
+static int
+cdx_vfio_get_region_info(int vfio_dev_fd, struct vfio_region_info **info,
+ int region)
+{
+ struct vfio_region_info *ri;
+ size_t argsz = sizeof(*ri);
+ int ret;
+
+ ri = malloc(sizeof(*ri));
+ if (ri == NULL) {
+ CDX_BUS_ERR("Cannot allocate memory for VFIO region info");
+ return -1;
+ }
+again:
+ memset(ri, 0, argsz);
+ ri->argsz = argsz;
+ ri->index = region;
+
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_REGION_INFO, ri);
+ if (ret < 0) {
+ free(ri);
+ return ret;
+ }
+ if (ri->argsz != argsz) {
+ struct vfio_region_info *tmp;
+
+ argsz = ri->argsz;
+ tmp = realloc(ri, argsz);
+
+ if (tmp == NULL) {
+ /* realloc failed but the ri is still there */
+ free(ri);
+ CDX_BUS_ERR("Cannot reallocate memory for VFIO region info");
+ return -1;
+ }
+ ri = tmp;
+ goto again;
+ }
+ *info = ri;
+
+ return 0;
+}
+
+static int
+find_max_end_va(const struct rte_memseg_list *msl, void *arg)
+{
+ size_t sz = msl->len;
+ void *end_va = RTE_PTR_ADD(msl->base_va, sz);
+ void **max_va = arg;
+
+ if (*max_va < end_va)
+ *max_va = end_va;
+ return 0;
+}
+
+static void *
+cdx_find_max_end_va(void)
+{
+ void *va = NULL;
+
+ rte_memseg_list_walk(find_max_end_va, &va);
+ return va;
+}
+
+static int
+cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ char cdx_addr[PATH_MAX] = {0};
+ static void *cdx_map_addr;
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+ int vfio_dev_fd, i, ret;
+
+ ret = rte_vfio_setup_device(rte_cdx_get_sysfs_path(), dev_name,
+ &vfio_dev_fd, &device_info);
+ if (ret)
+ return ret;
+
+ /* allocate vfio_res and get region info */
+ vfio_res = rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0);
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("Cannot store VFIO mmap details");
+ goto err_vfio_dev_fd;
+ }
+ memcpy(vfio_res->name, dev_name, RTE_DEV_NAME_MAX_LEN);
+
+ /* get number of registers */
+ vfio_res->nb_maps = device_info.num_regions;
+
+ /* map memory regions */
+ maps = vfio_res->maps;
+
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ struct vfio_region_info *reg = NULL;
+ void *vaddr;
+
+ ret = cdx_vfio_get_region_info(vfio_dev_fd, ®, i);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s cannot get device region info error %i (%s)",
+ dev_name, errno, strerror(errno));
+ goto err_vfio_res;
+ }
+
+ /* skip non-mmappable regions */
+ if ((reg->flags & VFIO_REGION_INFO_FLAG_MMAP) == 0) {
+ free(reg);
+ continue;
+ }
+
+ /* try mapping somewhere close to the end of hugepages */
+ if (cdx_map_addr == NULL)
+ cdx_map_addr = cdx_find_max_end_va();
+
+ vaddr = cdx_map_addr;
+ cdx_map_addr = RTE_PTR_ADD(vaddr, (size_t)reg->size);
+
+ cdx_map_addr = RTE_PTR_ALIGN(cdx_map_addr,
+ sysconf(_SC_PAGE_SIZE));
+
+ maps[i].addr = vaddr;
+ maps[i].offset = reg->offset;
+ maps[i].size = reg->size;
+ maps[i].path = NULL; /* vfio doesn't have per-resource paths */
+
+ ret = cdx_vfio_mmap_resource(vfio_dev_fd, vfio_res, i, 0);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s mapping region %i failed: %s",
+ cdx_addr, i, strerror(errno));
+ free(reg);
+ goto err_vfio_res;
+ }
+
+ dev->mem_resource[i].addr = maps[i].addr;
+ dev->mem_resource[i].len = maps[i].size;
+
+ free(reg);
+ }
+
+ if (cdx_rte_vfio_setup_device(vfio_dev_fd) < 0) {
+ CDX_BUS_ERR("%s setup device failed", dev_name);
+ goto err_vfio_res;
+ }
+
+ TAILQ_INSERT_TAIL(vfio_res_list, vfio_res, next);
+
+ return 0;
+err_vfio_res:
+ cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+ rte_free(vfio_res);
+err_vfio_dev_fd:
+ rte_vfio_release_device(rte_cdx_get_sysfs_path(),
+ dev_name, vfio_dev_fd);
+ return -1;
+}
+
+static int
+cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ char cdx_addr[PATH_MAX] = {0};
+ int vfio_dev_fd;
+ int i, ret;
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+
+ /* if we're in a secondary process, just find our tailq entry */
+ TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+ if (strcmp(vfio_res->name, dev_name))
+ continue;
+ break;
+ }
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for cdx device!",
+ dev_name);
+ return -1;
+ }
+
+ ret = rte_vfio_setup_device(rte_cdx_get_sysfs_path(), dev_name,
+ &vfio_dev_fd, &device_info);
+ if (ret)
+ return ret;
+
+ /* map MMIO regions */
+ maps = vfio_res->maps;
+
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ ret = cdx_vfio_mmap_resource(vfio_dev_fd, vfio_res, i, MAP_FIXED);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s mapping MMIO region %i failed: %s",
+ dev_name, i, strerror(errno));
+ goto err_vfio_dev_fd;
+ }
+
+ dev->mem_resource[i].addr = maps[i].addr;
+ dev->mem_resource[i].len = maps[i].size;
+ }
+
+ return 0;
+err_vfio_dev_fd:
+ rte_vfio_release_device(rte_cdx_get_sysfs_path(),
+ cdx_addr, vfio_dev_fd);
+ return -1;
+}
+
+/*
+ * map the CDX resources of a CDX device in virtual memory (VFIO version).
+ * primary and secondary processes follow almost exactly the same path
+ */
+int
+cdx_vfio_map_resource(struct rte_cdx_device *dev)
+{
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return cdx_vfio_map_resource_primary(dev);
+ else
+ return cdx_vfio_map_resource_secondary(dev);
+}
diff --git a/drivers/bus/cdx/meson.build b/drivers/bus/cdx/meson.build
new file mode 100644
index 0000000000..f2ca104d34
--- /dev/null
+++ b/drivers/bus/cdx/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+
+if not is_linux
+ build = false
+ reason = 'only supported on Linux'
+endif
+
+driver_sdk_headers = files('bus_cdx_driver.h')
+sources = files(
+ 'cdx.c',
+ 'cdx_vfio.c',
+)
diff --git a/drivers/bus/cdx/private.h b/drivers/bus/cdx/private.h
new file mode 100644
index 0000000000..c4ca76dc1b
--- /dev/null
+++ b/drivers/bus/cdx/private.h
@@ -0,0 +1,49 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef _CDX_PRIVATE_H_
+#define _CDX_PRIVATE_H_
+
+#include <stdbool.h>
+#include <stdio.h>
+
+#include "bus_cdx_driver.h"
+
+extern struct rte_cdx_bus rte_cdx_bus;
+
+/**
+ * Map a particular resource from a file.
+ *
+ * @param requested_addr
+ * The starting address for the new mapping range.
+ * @param fd
+ * The file descriptor.
+ * @param offset
+ * The offset for the mapping range.
+ * @param size
+ * The size for the mapping range.
+ * @param additional_flags
+ * The additional rte_mem_map() flags for the mapping range.
+ * @return
+ * - On success, the function returns a pointer to the mapped area.
+ * - On error, NULL is returned.
+ */
+void *cdx_map_resource(void *requested_addr, int fd, uint64_t offset,
+ size_t size, int additional_flags);
+
+/**
+ * Unmap a particular resource.
+ *
+ * @param requested_addr
+ * The address for the unmapping range.
+ * @param size
+ * The size for the unmapping range.
+ */
+void cdx_unmap_resource(void *requested_addr, size_t size);
+
+/* map/unmap VFIO resource */
+int cdx_vfio_map_resource(struct rte_cdx_device *dev);
+int cdx_vfio_unmap_resource(struct rte_cdx_device *dev);
+
+#endif /* _CDX_PRIVATE_H_ */
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
new file mode 100644
index 0000000000..957fcab978
--- /dev/null
+++ b/drivers/bus/cdx/version.map
@@ -0,0 +1,11 @@
+INTERNAL {
+ global:
+
+ rte_cdx_get_sysfs_path;
+ rte_cdx_map_device;
+ rte_cdx_register;
+ rte_cdx_unmap_device;
+ rte_cdx_unregister;
+
+ local: *;
+};
diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
index 6d2520c543..a78b4283bf 100644
--- a/drivers/bus/meson.build
+++ b/drivers/bus/meson.build
@@ -3,6 +3,7 @@
drivers = [
'auxiliary',
+ 'cdx',
'dpaa',
'fslmc',
'ifpga',
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v3 2/5] bus/cdx: add DMA map and unmap support
2023-04-21 14:54 ` [PATCH v3 0/5] Support AMD CDX bus, for FPGA based CDX devices. The CDX Nipun Gupta
2023-04-21 14:54 ` [PATCH v3 1/5] bus/cdx: introduce cdx bus Nipun Gupta
@ 2023-04-21 14:54 ` Nipun Gupta
2023-04-21 14:54 ` [PATCH v3 3/5] bus/cdx: add support for MSI Nipun Gupta
` (2 subsequent siblings)
4 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-04-21 14:54 UTC (permalink / raw)
To: dev, thomas, david.marchand, ferruh.yigit
Cc: harpreet.anand, nikhil.agarwal, Nipun Gupta
CDX bus can use VFIO interface for mapping and unmapping
of DMA addresses in the IOMMU. This change adds the callback
support for map and unmap APIs as well as fetching the IOMMU
class.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
drivers/bus/cdx/cdx.c | 40 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index d479daa315..8cc273336e 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -505,12 +505,52 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
return NULL;
}
+static int
+cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+ struct rte_cdx_device *cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+
+ if (!cdx_dev) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ return rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ (uintptr_t)addr, iova, len);
+}
+
+static int
+cdx_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+ struct rte_cdx_device *cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+
+ if (!cdx_dev) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ return rte_vfio_container_dma_unmap(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ (uintptr_t)addr, iova, len);
+}
+
+static enum rte_iova_mode
+cdx_get_iommu_class(void)
+{
+ if (TAILQ_EMPTY(&rte_cdx_bus.device_list))
+ return RTE_IOVA_DC;
+
+ return RTE_IOVA_VA;
+}
+
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
.scan = cdx_scan,
.probe = cdx_probe,
.find_device = cdx_find_device,
.parse = cdx_parse,
+ .dma_map = cdx_dma_map,
+ .dma_unmap = cdx_dma_unmap,
+ .get_iommu_class = cdx_get_iommu_class,
},
.device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
.driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v3 3/5] bus/cdx: add support for MSI
2023-04-21 14:54 ` [PATCH v3 0/5] Support AMD CDX bus, for FPGA based CDX devices. The CDX Nipun Gupta
2023-04-21 14:54 ` [PATCH v3 1/5] bus/cdx: introduce cdx bus Nipun Gupta
2023-04-21 14:54 ` [PATCH v3 2/5] bus/cdx: add DMA map and unmap support Nipun Gupta
@ 2023-04-21 14:54 ` Nipun Gupta
2023-04-21 14:54 ` [PATCH v3 4/5] bus/cdx: support plug unplug and dev iterator Nipun Gupta
2023-04-21 14:54 ` [PATCH v3 5/5] config/arm: add AMD CDX Nipun Gupta
4 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-04-21 14:54 UTC (permalink / raw)
To: dev, thomas, david.marchand, ferruh.yigit
Cc: harpreet.anand, nikhil.agarwal, Nipun Gupta
MSI's are exposed to the devices using VFIO (vfio-cdx). This
patch uses the same to add support for MSI for the devices on
the cdx bus.
A couple of API's have been introduced in the EAL interrupt
framework:
- rte_intr_irq_count_set: This API is used to set the total
interrupts on the interrupt handle. This would be provided
by VFIO (irq.count) for VFIO enabled devices.
- rte_intr_irq_count_get: This API returns the total number
interrupts which were set.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
drivers/bus/cdx/bus_cdx_driver.h | 25 ++++
drivers/bus/cdx/cdx.c | 11 ++
drivers/bus/cdx/cdx_vfio.c | 182 ++++++++++++++++++++++++-
drivers/bus/cdx/version.map | 2 +
lib/eal/common/eal_common_interrupts.c | 21 +++
lib/eal/common/eal_interrupts.h | 1 +
lib/eal/include/rte_interrupts.h | 32 +++++
lib/eal/version.map | 2 +
8 files changed, 274 insertions(+), 2 deletions(-)
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index 7edcb019eb..fdeaf46664 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -72,6 +72,7 @@ struct rte_cdx_device {
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
/**< CDX Memory Resource */
+ struct rte_intr_handle *intr_handle; /**< Interrupt handle */
};
/**
@@ -173,6 +174,30 @@ void rte_cdx_unmap_device(struct rte_cdx_device *dev);
__rte_internal
void rte_cdx_register(struct rte_cdx_driver *driver);
+/**
+ * Enables VFIO Interrupts for CDX bus devices.
+ *
+ * @param intr_handle
+ * Pointer to the interrupt handle.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle);
+
+/**
+ * Disable VFIO Interrupts for CDX bus devices.
+ *
+ * @param intr_handle
+ * Pointer to the interrupt handle.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle);
+
/**
* Helper for CDX device registration from driver (eth, crypto, raw) instance
*/
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index 8cc273336e..6c9ceaaf7f 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -204,6 +204,15 @@ cdx_scan_one(const char *dirname, const char *dev_name)
goto err;
}
+ /* Allocate interrupt instance for cdx device */
+ dev->intr_handle =
+ rte_intr_instance_alloc(RTE_INTR_INSTANCE_F_PRIVATE);
+ if (dev->intr_handle == NULL) {
+ CDX_BUS_ERR("Failed to create interrupt instance for %s\n",
+ dev->device.name);
+ return -ENOMEM;
+ }
+
/*
* Check if device is bound to 'vfio-cdx' driver, so that user-space
* can gracefully access the device.
@@ -394,6 +403,8 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr,
return ret;
error_probe:
+ rte_intr_instance_free(dev->intr_handle);
+ dev->intr_handle = NULL;
cdx_vfio_unmap_resource(dev);
error_map_device:
return ret;
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index ae11f589b3..1422b98503 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -60,6 +60,10 @@ struct mapped_cdx_resource {
/** mapped cdx device list */
TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
+/* irq set buffer length for MSI interrupts */
+#define MSI_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \
+ sizeof(int) * (RTE_MAX_RXTX_INTR_VEC_ID + 1))
+
static struct rte_tailq_elem cdx_vfio_tailq = {
.name = "VFIO_CDX_RESOURCE_LIST",
};
@@ -104,6 +108,27 @@ cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
char cdx_addr[PATH_MAX] = {0};
struct mapped_cdx_resource *vfio_res = NULL;
struct mapped_cdx_res_list *vfio_res_list;
+ int ret, vfio_dev_fd;
+
+ if (rte_intr_fd_get(dev->intr_handle) < 0)
+ return -1;
+
+ if (close(rte_intr_fd_get(dev->intr_handle)) < 0) {
+ CDX_BUS_ERR("Error when closing eventfd file descriptor for %s",
+ dev->device.name);
+ return -1;
+ }
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ ret = rte_vfio_release_device(rte_cdx_get_sysfs_path(), dev->device.name,
+ vfio_dev_fd);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot release VFIO device");
+ return ret;
+ }
vfio_res_list =
RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
@@ -126,6 +151,18 @@ cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
{
struct mapped_cdx_resource *vfio_res = NULL;
struct mapped_cdx_res_list *vfio_res_list;
+ int ret, vfio_dev_fd;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ ret = rte_vfio_release_device(rte_cdx_get_sysfs_path(), dev->device.name,
+ vfio_dev_fd);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot release VFIO device");
+ return ret;
+ }
vfio_res_list =
RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
@@ -150,9 +187,80 @@ cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
return cdx_vfio_unmap_resource_secondary(dev);
}
+/* set up interrupt support (but not enable interrupts) */
static int
-cdx_rte_vfio_setup_device(int vfio_dev_fd)
+cdx_vfio_setup_interrupts(struct rte_cdx_device *dev, int vfio_dev_fd,
+ int num_irqs)
{
+ int i, ret;
+
+ if (num_irqs == 0)
+ return 0;
+
+ /* start from MSI interrupt type */
+ for (i = 0; i < num_irqs; i++) {
+ struct vfio_irq_info irq = { .argsz = sizeof(irq) };
+ int fd = -1;
+
+ irq.index = i;
+
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_IRQ_INFO, &irq);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot get VFIO IRQ info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /* if this vector cannot be used with eventfd, fail if we explicitly
+ * specified interrupt type, otherwise continue
+ */
+ if ((irq.flags & VFIO_IRQ_INFO_EVENTFD) == 0)
+ continue;
+
+ if (rte_intr_irq_count_set(dev->intr_handle, irq.count))
+ return -1;
+
+ /* Reallocate the efds and elist fields of intr_handle based
+ * on CDX device MSI size.
+ */
+ if ((uint32_t)rte_intr_nb_intr_get(dev->intr_handle) < irq.count &&
+ rte_intr_event_list_update(dev->intr_handle, irq.count))
+ return -1;
+
+ /* set up an eventfd for interrupts */
+ fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
+ if (fd < 0) {
+ CDX_BUS_ERR("Cannot set up eventfd, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ if (rte_intr_fd_set(dev->intr_handle, fd))
+ return -1;
+
+ /* DPDK CDX bus currently supports only MSI-X */
+ if (rte_intr_type_set(dev->intr_handle, RTE_INTR_HANDLE_VFIO_MSIX))
+ return -1;
+
+ if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
+ return -1;
+
+ return 0;
+ }
+
+ /* if we're here, we haven't found a suitable interrupt vector */
+ return -1;
+}
+
+static int
+cdx_vfio_setup_device(struct rte_cdx_device *dev, int vfio_dev_fd,
+ int num_irqs)
+{
+ if (cdx_vfio_setup_interrupts(dev, vfio_dev_fd, num_irqs) != 0) {
+ CDX_BUS_ERR("Error setting up interrupts!");
+ return -1;
+ }
+
/*
* Reset the device. If the device is not capable of resetting,
* then it updates errno as EINVAL.
@@ -288,6 +396,9 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
struct cdx_map *maps;
int vfio_dev_fd, i, ret;
+ if (rte_intr_fd_set(dev->intr_handle, -1))
+ return -1;
+
ret = rte_vfio_setup_device(rte_cdx_get_sysfs_path(), dev_name,
&vfio_dev_fd, &device_info);
if (ret)
@@ -353,7 +464,7 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
free(reg);
}
- if (cdx_rte_vfio_setup_device(vfio_dev_fd) < 0) {
+ if (cdx_vfio_setup_device(dev, vfio_dev_fd, device_info.num_irqs) < 0) {
CDX_BUS_ERR("%s setup device failed", dev_name);
goto err_vfio_res;
}
@@ -383,6 +494,9 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
const char *dev_name = dev->device.name;
struct cdx_map *maps;
+ if (rte_intr_fd_set(dev->intr_handle, -1))
+ return -1;
+
/* if we're in a secondary process, just find our tailq entry */
TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
if (strcmp(vfio_res->name, dev_name))
@@ -416,6 +530,10 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
dev->mem_resource[i].len = maps[i].size;
}
+ /* we need save vfio_dev_fd, so it can be used during release */
+ if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
+ goto err_vfio_dev_fd;
+
return 0;
err_vfio_dev_fd:
rte_vfio_release_device(rte_cdx_get_sysfs_path(),
@@ -435,3 +553,63 @@ cdx_vfio_map_resource(struct rte_cdx_device *dev)
else
return cdx_vfio_map_resource_secondary(dev);
}
+
+int
+rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle)
+{
+ char irq_set_buf[MSI_IRQ_SET_BUF_LEN];
+ struct vfio_irq_set *irq_set;
+ int *fd_ptr, vfio_dev_fd, i;
+ int ret;
+
+ irq_set = (struct vfio_irq_set *) irq_set_buf;
+ irq_set->count = rte_intr_irq_count_get(intr_handle);
+ irq_set->argsz = sizeof(struct vfio_irq_set) +
+ (sizeof(int) * irq_set->count);
+
+ irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER;
+ irq_set->index = 0;
+ irq_set->start = 0;
+ fd_ptr = (int *) &irq_set->data;
+
+ for (i = 0; i < rte_intr_nb_efd_get(intr_handle); i++)
+ fd_ptr[i] = rte_intr_efds_index_get(intr_handle, i);
+
+ vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
+
+ if (ret) {
+ CDX_BUS_ERR("Error enabling MSI interrupts for fd %d",
+ rte_intr_fd_get(intr_handle));
+ return -1;
+ }
+
+ return 0;
+}
+
+/* disable MSI interrupts */
+int
+rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle)
+{
+ struct vfio_irq_set *irq_set;
+ char irq_set_buf[MSI_IRQ_SET_BUF_LEN];
+ int len, ret, vfio_dev_fd;
+
+ len = sizeof(struct vfio_irq_set);
+
+ irq_set = (struct vfio_irq_set *) irq_set_buf;
+ irq_set->argsz = len;
+ irq_set->count = 0;
+ irq_set->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_TRIGGER;
+ irq_set->index = 0;
+ irq_set->start = 0;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
+
+ if (ret)
+ CDX_BUS_ERR("Error disabling MSI interrupts for fd %d",
+ rte_intr_fd_get(intr_handle));
+
+ return ret;
+}
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
index 957fcab978..2f3d484ebd 100644
--- a/drivers/bus/cdx/version.map
+++ b/drivers/bus/cdx/version.map
@@ -6,6 +6,8 @@ INTERNAL {
rte_cdx_register;
rte_cdx_unmap_device;
rte_cdx_unregister;
+ rte_cdx_vfio_intr_disable;
+ rte_cdx_vfio_intr_enable;
local: *;
};
diff --git a/lib/eal/common/eal_common_interrupts.c b/lib/eal/common/eal_common_interrupts.c
index 97b64fed58..a0167d9ad4 100644
--- a/lib/eal/common/eal_common_interrupts.c
+++ b/lib/eal/common/eal_common_interrupts.c
@@ -398,6 +398,27 @@ int rte_intr_elist_index_set(struct rte_intr_handle *intr_handle,
return -rte_errno;
}
+int rte_intr_irq_count_set(struct rte_intr_handle *intr_handle,
+ int irq_count)
+{
+ CHECK_VALID_INTR_HANDLE(intr_handle);
+
+ intr_handle->irq_count = irq_count;
+
+ return 0;
+fail:
+ return -rte_errno;
+}
+
+int rte_intr_irq_count_get(const struct rte_intr_handle *intr_handle)
+{
+ CHECK_VALID_INTR_HANDLE(intr_handle);
+
+ return intr_handle->irq_count;
+fail:
+ return -rte_errno;
+}
+
int rte_intr_vec_list_alloc(struct rte_intr_handle *intr_handle,
const char *name, int size)
{
diff --git a/lib/eal/common/eal_interrupts.h b/lib/eal/common/eal_interrupts.h
index 482781b862..237f471a76 100644
--- a/lib/eal/common/eal_interrupts.h
+++ b/lib/eal/common/eal_interrupts.h
@@ -16,6 +16,7 @@ struct rte_intr_handle {
};
uint32_t alloc_flags; /**< flags passed at allocation */
enum rte_intr_handle_type type; /**< handle type */
+ uint32_t irq_count; /**< IRQ count provided via VFIO */
uint32_t max_intr; /**< max interrupt requested */
uint32_t nb_efd; /**< number of available efd(event fd) */
uint8_t efd_counter_size; /**< size of efd counter, used for vdev */
diff --git a/lib/eal/include/rte_interrupts.h b/lib/eal/include/rte_interrupts.h
index 487e3c8875..bc477a483f 100644
--- a/lib/eal/include/rte_interrupts.h
+++ b/lib/eal/include/rte_interrupts.h
@@ -506,6 +506,38 @@ __rte_internal
int
rte_intr_max_intr_get(const struct rte_intr_handle *intr_handle);
+/**
+ * @internal
+ * Set the irq count field of interrupt handle with user
+ * provided irq count value.
+ *
+ * @param intr_handle
+ * pointer to the interrupt handle.
+ * @param irq_count
+ * IRQ count
+ *
+ * @return
+ * - On success, zero.
+ * - On failure, a negative value and rte_errno is set.
+ */
+__rte_internal
+int
+rte_intr_irq_count_set(struct rte_intr_handle *intr_handle, int irq_count);
+
+/**
+ * @internal
+ * Returns the irq count field of the given interrupt handle instance.
+ *
+ * @param intr_handle
+ * pointer to the interrupt handle.
+ *
+ * @return
+ * - On success, ir count.
+ * - On failure, a negative value and rte_errno is set.
+ */
+__rte_internal
+int rte_intr_irq_count_get(const struct rte_intr_handle *intr_handle);
+
/**
* @internal
* Set the number of event fd field of interrupt handle
diff --git a/lib/eal/version.map b/lib/eal/version.map
index 6d6978f108..14bf7ade77 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -458,6 +458,8 @@ INTERNAL {
rte_intr_instance_dup;
rte_intr_instance_windows_handle_get;
rte_intr_instance_windows_handle_set;
+ rte_intr_irq_count_get;
+ rte_intr_irq_count_set;
rte_intr_max_intr_get;
rte_intr_max_intr_set;
rte_intr_nb_efd_get;
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v3 4/5] bus/cdx: support plug unplug and dev iterator
2023-04-21 14:54 ` [PATCH v3 0/5] Support AMD CDX bus, for FPGA based CDX devices. The CDX Nipun Gupta
` (2 preceding siblings ...)
2023-04-21 14:54 ` [PATCH v3 3/5] bus/cdx: add support for MSI Nipun Gupta
@ 2023-04-21 14:54 ` Nipun Gupta
2023-04-21 14:54 ` [PATCH v3 5/5] config/arm: add AMD CDX Nipun Gupta
4 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-04-21 14:54 UTC (permalink / raw)
To: dev, thomas, david.marchand, ferruh.yigit
Cc: harpreet.anand, nikhil.agarwal, Nipun Gupta
This change adds support for plugging and unplugging
CDX devices on the CDX bus. Also, CDX dev iterator support
has been added for the CDX bus.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
drivers/bus/cdx/bus_cdx_driver.h | 1 +
drivers/bus/cdx/cdx.c | 122 +++++++++++++++++++++++++++++++
2 files changed, 123 insertions(+)
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index fdeaf46664..3d89e7c054 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -69,6 +69,7 @@ struct rte_cdx_id {
struct rte_cdx_device {
RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
struct rte_device device; /**< Inherit core device */
+ struct rte_cdx_driver *driver; /**< CDX driver used in probing */
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
/**< CDX Memory Resource */
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index 6c9ceaaf7f..0a30b8648a 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -68,6 +68,7 @@
#include <rte_eal_paging.h>
#include <rte_errno.h>
#include <rte_devargs.h>
+#include <rte_kvargs.h>
#include <rte_malloc.h>
#include <rte_memcpy.h>
#include <rte_vfio.h>
@@ -82,6 +83,15 @@
#define CDX_BUS_NAME cdx
#define CDX_DEV_PREFIX "cdx-"
+enum cdx_params {
+ RTE_CDX_PARAM_NAME,
+};
+
+static const char * const cdx_params_keys[] = {
+ [RTE_CDX_PARAM_NAME] = "name",
+ NULL,
+};
+
/**
* @file
* CDX probing using Linux sysfs.
@@ -399,6 +409,7 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr,
} else {
dev->device.driver = &dr->driver;
}
+ dev->driver = dr;
return ret;
@@ -516,6 +527,71 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
return NULL;
}
+/* Remove a device from CDX bus */
+static void
+cdx_remove_device(struct rte_cdx_device *cdx_dev)
+{
+ TAILQ_REMOVE(&rte_cdx_bus.device_list, cdx_dev, next);
+}
+
+/*
+ * If vendor/device ID match, call the remove() function of the
+ * driver.
+ */
+static int
+cdx_detach_dev(struct rte_cdx_device *dev)
+{
+ struct rte_cdx_driver *dr;
+ int ret = 0;
+
+ if (dev == NULL)
+ return -EINVAL;
+
+ dr = dev->driver;
+
+ CDX_BUS_DEBUG("detach device %s using driver: %s",
+ dev->device.name, dr->driver.name);
+
+ if (dr->remove) {
+ ret = dr->remove(dev);
+ if (ret < 0)
+ return ret;
+ }
+
+ /* clear driver structure */
+ dev->driver = NULL;
+ dev->device.driver = NULL;
+
+ rte_cdx_unmap_device(dev);
+
+ rte_intr_instance_free(dev->intr_handle);
+ dev->intr_handle = NULL;
+
+ return 0;
+}
+
+static int
+cdx_plug(struct rte_device *dev)
+{
+ return cdx_probe_all_drivers(RTE_DEV_TO_CDX_DEV(dev));
+}
+
+static int
+cdx_unplug(struct rte_device *dev)
+{
+ struct rte_cdx_device *cdx_dev;
+ int ret;
+
+ cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+ ret = cdx_detach_dev(cdx_dev);
+ if (ret == 0) {
+ cdx_remove_device(cdx_dev);
+ rte_devargs_remove(dev->devargs);
+ free(cdx_dev);
+ }
+ return ret;
+}
+
static int
cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
{
@@ -553,15 +629,61 @@ cdx_get_iommu_class(void)
return RTE_IOVA_VA;
}
+static int
+cdx_dev_match(const struct rte_device *dev,
+ const void *_kvlist)
+{
+ const struct rte_kvargs *kvlist = _kvlist;
+ const char *key = cdx_params_keys[RTE_CDX_PARAM_NAME];
+ const char *name;
+
+ /* no kvlist arg, all devices match */
+ if (kvlist == NULL)
+ return 0;
+
+ /* if key is present in kvlist and does not match, filter device */
+ name = rte_kvargs_get(kvlist, key);
+ if (name != NULL && strcmp(name, dev->name))
+ return -1;
+
+ return 0;
+}
+
+static void *
+cdx_dev_iterate(const void *start,
+ const char *str,
+ const struct rte_dev_iterator *it __rte_unused)
+{
+ rte_bus_find_device_t find_device;
+ struct rte_kvargs *kvargs = NULL;
+ struct rte_device *dev;
+
+ if (str != NULL) {
+ kvargs = rte_kvargs_parse(str, cdx_params_keys);
+ if (kvargs == NULL) {
+ CDX_BUS_ERR("cannot parse argument list %s", str);
+ rte_errno = EINVAL;
+ return NULL;
+ }
+ }
+ find_device = rte_cdx_bus.bus.find_device;
+ dev = find_device(start, cdx_dev_match, kvargs);
+ rte_kvargs_free(kvargs);
+ return dev;
+}
+
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
.scan = cdx_scan,
.probe = cdx_probe,
.find_device = cdx_find_device,
+ .plug = cdx_plug,
+ .unplug = cdx_unplug,
.parse = cdx_parse,
.dma_map = cdx_dma_map,
.dma_unmap = cdx_dma_unmap,
.get_iommu_class = cdx_get_iommu_class,
+ .dev_iterate = cdx_dev_iterate,
},
.device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
.driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v3 5/5] config/arm: add AMD CDX
2023-04-21 14:54 ` [PATCH v3 0/5] Support AMD CDX bus, for FPGA based CDX devices. The CDX Nipun Gupta
` (3 preceding siblings ...)
2023-04-21 14:54 ` [PATCH v3 4/5] bus/cdx: support plug unplug and dev iterator Nipun Gupta
@ 2023-04-21 14:54 ` Nipun Gupta
2023-05-04 15:28 ` Ferruh Yigit
4 siblings, 1 reply; 102+ messages in thread
From: Nipun Gupta @ 2023-04-21 14:54 UTC (permalink / raw)
To: dev, thomas, david.marchand, ferruh.yigit
Cc: harpreet.anand, nikhil.agarwal, Nipun Gupta
Adding support for AMD CDX devices
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
config/arm/arm64_cdx_linux_gcc | 17 +++++++++++++++++
config/arm/meson.build | 14 ++++++++++++++
2 files changed, 31 insertions(+)
create mode 100644 config/arm/arm64_cdx_linux_gcc
diff --git a/config/arm/arm64_cdx_linux_gcc b/config/arm/arm64_cdx_linux_gcc
new file mode 100644
index 0000000000..8e6d619dae
--- /dev/null
+++ b/config/arm/arm64_cdx_linux_gcc
@@ -0,0 +1,17 @@
+[binaries]
+c = ['ccache', 'aarch64-linux-gnu-gcc']
+cpp = ['ccache', 'aarch64-linux-gnu-g++']
+ar = 'aarch64-linux-gnu-ar'
+as = 'aarch64-linux-gnu-as'
+strip = 'aarch64-linux-gnu-strip'
+pkgconfig = 'aarch64-linux-gnu-pkg-config'
+pcap-config = ''
+
+[host_machine]
+system = 'linux'
+cpu_family = 'aarch64'
+cpu = 'armv8-a'
+endian = 'little'
+
+[properties]
+platform = 'cdx'
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 5213434ca4..39b8929534 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -305,6 +305,18 @@ soc_bluefield = {
'numa': false
}
+soc_cdx = {
+ 'description': 'AMD CDX',
+ 'implementer': '0x41',
+ 'part_number': '0xd42',
+ 'flags': [
+ ['RTE_MACHINE', '"cdx"'],
+ ['RTE_MAX_LCORE', 16],
+ ['RTE_MAX_NUMA_NODES', 1]
+ ],
+ 'numa': false
+}
+
soc_centriq2400 = {
'description': 'Qualcomm Centriq 2400',
'implementer': '0x51',
@@ -463,6 +475,7 @@ generic_aarch32: Generic un-optimized build for armv8 aarch32 execution mode.
armada: Marvell ARMADA
bluefield: NVIDIA BlueField
bluefield3: NVIDIA BlueField-3
+cdx: AMD CDX
centriq2400: Qualcomm Centriq 2400
cn9k: Marvell OCTEON 9
cn10k: Marvell OCTEON 10
@@ -490,6 +503,7 @@ socs = {
'armada': soc_armada,
'bluefield': soc_bluefield,
'bluefield3': soc_bluefield3,
+ 'cdx': soc_cdx,
'centriq2400': soc_centriq2400,
'cn9k': soc_cn9k,
'cn10k' : soc_cn10k,
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v3 5/5] config/arm: add AMD CDX
2023-04-21 14:54 ` [PATCH v3 5/5] config/arm: add AMD CDX Nipun Gupta
@ 2023-05-04 15:28 ` Ferruh Yigit
2023-05-08 10:24 ` Gupta, Nipun
0 siblings, 1 reply; 102+ messages in thread
From: Ferruh Yigit @ 2023-05-04 15:28 UTC (permalink / raw)
To: Nipun Gupta, dev, thomas, david.marchand; +Cc: harpreet.anand, nikhil.agarwal
On 4/21/2023 3:54 PM, Nipun Gupta wrote:
> Adding support for AMD CDX devices
>
> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> ---
> config/arm/arm64_cdx_linux_gcc | 17 +++++++++++++++++
> config/arm/meson.build | 14 ++++++++++++++
> 2 files changed, 31 insertions(+)
> create mode 100644 config/arm/arm64_cdx_linux_gcc
>
> diff --git a/config/arm/arm64_cdx_linux_gcc b/config/arm/arm64_cdx_linux_gcc
> new file mode 100644
> index 0000000000..8e6d619dae
> --- /dev/null
> +++ b/config/arm/arm64_cdx_linux_gcc
> @@ -0,0 +1,17 @@
> +[binaries]
> +c = ['ccache', 'aarch64-linux-gnu-gcc']
> +cpp = ['ccache', 'aarch64-linux-gnu-g++']
> +ar = 'aarch64-linux-gnu-ar'
> +as = 'aarch64-linux-gnu-as'
> +strip = 'aarch64-linux-gnu-strip'
> +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> +pcap-config = ''
> +
> +[host_machine]
> +system = 'linux'
> +cpu_family = 'aarch64'
> +cpu = 'armv8-a'
> +endian = 'little'
> +
> +[properties]
> +platform = 'cdx'
> diff --git a/config/arm/meson.build b/config/arm/meson.build
> index 5213434ca4..39b8929534 100644
> --- a/config/arm/meson.build
> +++ b/config/arm/meson.build
> @@ -305,6 +305,18 @@ soc_bluefield = {
> 'numa': false
> }
>
> +soc_cdx = {
> + 'description': 'AMD CDX',
> + 'implementer': '0x41',
> + 'part_number': '0xd42',
> + 'flags': [
> + ['RTE_MACHINE', '"cdx"'],
> + ['RTE_MAX_LCORE', 16],
> + ['RTE_MAX_NUMA_NODES', 1]
> + ],
> + 'numa': false
> +}
Hi Nipun,
Why we need a new arm platform/config? Is it because of above flags?
If it can work with default values, I think we can drop this patch.
^ permalink raw reply [flat|nested] 102+ messages in thread* RE: [PATCH v3 5/5] config/arm: add AMD CDX
2023-05-04 15:28 ` Ferruh Yigit
@ 2023-05-08 10:24 ` Gupta, Nipun
2023-05-08 10:44 ` Thomas Monjalon
2023-05-08 11:26 ` Ferruh Yigit
0 siblings, 2 replies; 102+ messages in thread
From: Gupta, Nipun @ 2023-05-08 10:24 UTC (permalink / raw)
To: Yigit, Ferruh, dev, thomas, david.marchand
Cc: Anand, Harpreet, Agarwal, Nikhil
> -----Original Message-----
> From: Yigit, Ferruh <Ferruh.Yigit@amd.com>
> Sent: Thursday, May 4, 2023 8:59 PM
> To: Gupta, Nipun <Nipun.Gupta@amd.com>; dev@dpdk.org;
> thomas@monjalon.net; david.marchand@redhat.com
> Cc: Anand, Harpreet <harpreet.anand@amd.com>; Agarwal, Nikhil
> <nikhil.agarwal@amd.com>
> Subject: Re: [PATCH v3 5/5] config/arm: add AMD CDX
>
> On 4/21/2023 3:54 PM, Nipun Gupta wrote:
> > Adding support for AMD CDX devices
> >
> > Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> > ---
> > config/arm/arm64_cdx_linux_gcc | 17 +++++++++++++++++
> > config/arm/meson.build | 14 ++++++++++++++
> > 2 files changed, 31 insertions(+)
> > create mode 100644 config/arm/arm64_cdx_linux_gcc
> >
> > diff --git a/config/arm/arm64_cdx_linux_gcc
> b/config/arm/arm64_cdx_linux_gcc
> > new file mode 100644
> > index 0000000000..8e6d619dae
> > --- /dev/null
> > +++ b/config/arm/arm64_cdx_linux_gcc
> > @@ -0,0 +1,17 @@
> > +[binaries]
> > +c = ['ccache', 'aarch64-linux-gnu-gcc']
> > +cpp = ['ccache', 'aarch64-linux-gnu-g++']
> > +ar = 'aarch64-linux-gnu-ar'
> > +as = 'aarch64-linux-gnu-as'
> > +strip = 'aarch64-linux-gnu-strip'
> > +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> > +pcap-config = ''
> > +
> > +[host_machine]
> > +system = 'linux'
> > +cpu_family = 'aarch64'
> > +cpu = 'armv8-a'
> > +endian = 'little'
> > +
> > +[properties]
> > +platform = 'cdx'
> > diff --git a/config/arm/meson.build b/config/arm/meson.build
> > index 5213434ca4..39b8929534 100644
> > --- a/config/arm/meson.build
> > +++ b/config/arm/meson.build
> > @@ -305,6 +305,18 @@ soc_bluefield = {
> > 'numa': false
> > }
> >
> > +soc_cdx = {
> > + 'description': 'AMD CDX',
> > + 'implementer': '0x41',
> > + 'part_number': '0xd42',
> > + 'flags': [
> > + ['RTE_MACHINE', '"cdx"'],
> > + ['RTE_MAX_LCORE', 16],
> > + ['RTE_MAX_NUMA_NODES', 1]
> > + ],
> > + 'numa': false
> > +}
>
> Hi Nipun,
>
> Why we need a new arm platform/config? Is it because of above flags?
> If it can work with default values, I think we can drop this patch.
Hi Ferruh,
CDX driver works with generic ARM compilation too (arm64_armv8_linux_gcc).
The versal platforms supporting CDX have A78 cores, and adding this cdx config
Helps to provide gcc option "march= armv8.4-a" which is for implementer
"0xd42" (ARM cortex A78 cores)., whereas for generic ARM compilation
"march= armv8-a".
Maybe ARM guys can provide more information regarding if there is any impact
on using generic architecture flag (i.e. march= armv8a) on A78 cores.
Thanks,
Nipun
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v3 5/5] config/arm: add AMD CDX
2023-05-08 10:24 ` Gupta, Nipun
@ 2023-05-08 10:44 ` Thomas Monjalon
2023-05-08 10:48 ` Gupta, Nipun
2023-05-08 11:26 ` Ferruh Yigit
1 sibling, 1 reply; 102+ messages in thread
From: Thomas Monjalon @ 2023-05-08 10:44 UTC (permalink / raw)
To: Yigit, Ferruh, Gupta, Nipun
Cc: dev, david.marchand, Anand, Harpreet, Agarwal, Nikhil
08/05/2023 12:24, Gupta, Nipun:
> From: Yigit, Ferruh <Ferruh.Yigit@amd.com>
> > On 4/21/2023 3:54 PM, Nipun Gupta wrote:
> > > Adding support for AMD CDX devices
> > >
> > > Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> > > ---
> > > config/arm/arm64_cdx_linux_gcc | 17 +++++++++++++++++
> > > config/arm/meson.build | 14 ++++++++++++++
> > > 2 files changed, 31 insertions(+)
> > > create mode 100644 config/arm/arm64_cdx_linux_gcc
> > >
> > > diff --git a/config/arm/arm64_cdx_linux_gcc
> > b/config/arm/arm64_cdx_linux_gcc
> > > new file mode 100644
> > > index 0000000000..8e6d619dae
> > > --- /dev/null
> > > +++ b/config/arm/arm64_cdx_linux_gcc
> > > @@ -0,0 +1,17 @@
> > > +[binaries]
> > > +c = ['ccache', 'aarch64-linux-gnu-gcc']
> > > +cpp = ['ccache', 'aarch64-linux-gnu-g++']
> > > +ar = 'aarch64-linux-gnu-ar'
> > > +as = 'aarch64-linux-gnu-as'
> > > +strip = 'aarch64-linux-gnu-strip'
> > > +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> > > +pcap-config = ''
> > > +
> > > +[host_machine]
> > > +system = 'linux'
> > > +cpu_family = 'aarch64'
> > > +cpu = 'armv8-a'
> > > +endian = 'little'
> > > +
> > > +[properties]
> > > +platform = 'cdx'
> > > diff --git a/config/arm/meson.build b/config/arm/meson.build
> > > index 5213434ca4..39b8929534 100644
> > > --- a/config/arm/meson.build
> > > +++ b/config/arm/meson.build
> > > @@ -305,6 +305,18 @@ soc_bluefield = {
> > > 'numa': false
> > > }
> > >
> > > +soc_cdx = {
> > > + 'description': 'AMD CDX',
> > > + 'implementer': '0x41',
> > > + 'part_number': '0xd42',
> > > + 'flags': [
> > > + ['RTE_MACHINE', '"cdx"'],
> > > + ['RTE_MAX_LCORE', 16],
> > > + ['RTE_MAX_NUMA_NODES', 1]
> > > + ],
> > > + 'numa': false
> > > +}
> >
> > Hi Nipun,
> >
> > Why we need a new arm platform/config? Is it because of above flags?
> > If it can work with default values, I think we can drop this patch.
>
> Hi Ferruh,
>
> CDX driver works with generic ARM compilation too (arm64_armv8_linux_gcc).
>
> The versal platforms supporting CDX have A78 cores, and adding this cdx config
> Helps to provide gcc option "march= armv8.4-a" which is for implementer
> "0xd42" (ARM cortex A78 cores)., whereas for generic ARM compilation
> "march= armv8-a".
>
> Maybe ARM guys can provide more information regarding if there is any impact
> on using generic architecture flag (i.e. march= armv8a) on A78 cores.
Please let's skip this patch for now.
You should send it separately later, with perf testing and HW description in the commit log.
^ permalink raw reply [flat|nested] 102+ messages in thread* RE: [PATCH v3 5/5] config/arm: add AMD CDX
2023-05-08 10:44 ` Thomas Monjalon
@ 2023-05-08 10:48 ` Gupta, Nipun
0 siblings, 0 replies; 102+ messages in thread
From: Gupta, Nipun @ 2023-05-08 10:48 UTC (permalink / raw)
To: Thomas Monjalon, Yigit, Ferruh
Cc: dev, david.marchand, Anand, Harpreet, Agarwal, Nikhil
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Monday, May 8, 2023 4:14 PM
> To: Yigit, Ferruh <Ferruh.Yigit@amd.com>; Gupta, Nipun
> <Nipun.Gupta@amd.com>
> Cc: dev@dpdk.org; david.marchand@redhat.com; Anand, Harpreet
> <harpreet.anand@amd.com>; Agarwal, Nikhil <nikhil.agarwal@amd.com>
> Subject: Re: [PATCH v3 5/5] config/arm: add AMD CDX
>
>
> 08/05/2023 12:24, Gupta, Nipun:
> > From: Yigit, Ferruh <Ferruh.Yigit@amd.com>
> > > On 4/21/2023 3:54 PM, Nipun Gupta wrote:
> > > > Adding support for AMD CDX devices
> > > >
> > > > Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> > > > ---
> > > > config/arm/arm64_cdx_linux_gcc | 17 +++++++++++++++++
> > > > config/arm/meson.build | 14 ++++++++++++++
> > > > 2 files changed, 31 insertions(+)
> > > > create mode 100644 config/arm/arm64_cdx_linux_gcc
> > > >
> > > > diff --git a/config/arm/arm64_cdx_linux_gcc
> > > b/config/arm/arm64_cdx_linux_gcc
> > > > new file mode 100644
> > > > index 0000000000..8e6d619dae
> > > > --- /dev/null
> > > > +++ b/config/arm/arm64_cdx_linux_gcc
> > > > @@ -0,0 +1,17 @@
> > > > +[binaries]
> > > > +c = ['ccache', 'aarch64-linux-gnu-gcc']
> > > > +cpp = ['ccache', 'aarch64-linux-gnu-g++']
> > > > +ar = 'aarch64-linux-gnu-ar'
> > > > +as = 'aarch64-linux-gnu-as'
> > > > +strip = 'aarch64-linux-gnu-strip'
> > > > +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> > > > +pcap-config = ''
> > > > +
> > > > +[host_machine]
> > > > +system = 'linux'
> > > > +cpu_family = 'aarch64'
> > > > +cpu = 'armv8-a'
> > > > +endian = 'little'
> > > > +
> > > > +[properties]
> > > > +platform = 'cdx'
> > > > diff --git a/config/arm/meson.build b/config/arm/meson.build
> > > > index 5213434ca4..39b8929534 100644
> > > > --- a/config/arm/meson.build
> > > > +++ b/config/arm/meson.build
> > > > @@ -305,6 +305,18 @@ soc_bluefield = {
> > > > 'numa': false
> > > > }
> > > >
> > > > +soc_cdx = {
> > > > + 'description': 'AMD CDX',
> > > > + 'implementer': '0x41',
> > > > + 'part_number': '0xd42',
> > > > + 'flags': [
> > > > + ['RTE_MACHINE', '"cdx"'],
> > > > + ['RTE_MAX_LCORE', 16],
> > > > + ['RTE_MAX_NUMA_NODES', 1]
> > > > + ],
> > > > + 'numa': false
> > > > +}
> > >
> > > Hi Nipun,
> > >
> > > Why we need a new arm platform/config? Is it because of above flags?
> > > If it can work with default values, I think we can drop this patch.
> >
> > Hi Ferruh,
> >
> > CDX driver works with generic ARM compilation too
> (arm64_armv8_linux_gcc).
> >
> > The versal platforms supporting CDX have A78 cores, and adding this cdx
> config
> > Helps to provide gcc option "march= armv8.4-a" which is for implementer
> > "0xd42" (ARM cortex A78 cores)., whereas for generic ARM compilation
> > "march= armv8-a".
> >
> > Maybe ARM guys can provide more information regarding if there is any
> impact
> > on using generic architecture flag (i.e. march= armv8a) on A78 cores.
>
> Please let's skip this patch for now.
> You should send it separately later, with perf testing and HW description in the
> commit log.
Sure, I will skip this and send another spin for CDX bus without this change.
Thanks,
Nipun
>
>
^ permalink raw reply [flat|nested] 102+ messages in thread
* Re: [PATCH v3 5/5] config/arm: add AMD CDX
2023-05-08 10:24 ` Gupta, Nipun
2023-05-08 10:44 ` Thomas Monjalon
@ 2023-05-08 11:26 ` Ferruh Yigit
2023-05-08 17:16 ` Honnappa Nagarahalli
2023-05-09 5:55 ` Ruifeng Wang
1 sibling, 2 replies; 102+ messages in thread
From: Ferruh Yigit @ 2023-05-08 11:26 UTC (permalink / raw)
To: Gupta, Nipun, dev, thomas, david.marchand, Honnappa Nagarahalli,
Ruifeng Wang
Cc: Anand, Harpreet, Agarwal, Nikhil
On 5/8/2023 11:24 AM, Gupta, Nipun wrote:
>
>
>> -----Original Message-----
>> From: Yigit, Ferruh <Ferruh.Yigit@amd.com>
>> Sent: Thursday, May 4, 2023 8:59 PM
>> To: Gupta, Nipun <Nipun.Gupta@amd.com>; dev@dpdk.org;
>> thomas@monjalon.net; david.marchand@redhat.com
>> Cc: Anand, Harpreet <harpreet.anand@amd.com>; Agarwal, Nikhil
>> <nikhil.agarwal@amd.com>
>> Subject: Re: [PATCH v3 5/5] config/arm: add AMD CDX
>>
>> On 4/21/2023 3:54 PM, Nipun Gupta wrote:
>>> Adding support for AMD CDX devices
>>>
>>> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
>>> ---
>>> config/arm/arm64_cdx_linux_gcc | 17 +++++++++++++++++
>>> config/arm/meson.build | 14 ++++++++++++++
>>> 2 files changed, 31 insertions(+)
>>> create mode 100644 config/arm/arm64_cdx_linux_gcc
>>>
>>> diff --git a/config/arm/arm64_cdx_linux_gcc
>> b/config/arm/arm64_cdx_linux_gcc
>>> new file mode 100644
>>> index 0000000000..8e6d619dae
>>> --- /dev/null
>>> +++ b/config/arm/arm64_cdx_linux_gcc
>>> @@ -0,0 +1,17 @@
>>> +[binaries]
>>> +c = ['ccache', 'aarch64-linux-gnu-gcc']
>>> +cpp = ['ccache', 'aarch64-linux-gnu-g++']
>>> +ar = 'aarch64-linux-gnu-ar'
>>> +as = 'aarch64-linux-gnu-as'
>>> +strip = 'aarch64-linux-gnu-strip'
>>> +pkgconfig = 'aarch64-linux-gnu-pkg-config'
>>> +pcap-config = ''
>>> +
>>> +[host_machine]
>>> +system = 'linux'
>>> +cpu_family = 'aarch64'
>>> +cpu = 'armv8-a'
>>> +endian = 'little'
>>> +
>>> +[properties]
>>> +platform = 'cdx'
>>> diff --git a/config/arm/meson.build b/config/arm/meson.build
>>> index 5213434ca4..39b8929534 100644
>>> --- a/config/arm/meson.build
>>> +++ b/config/arm/meson.build
>>> @@ -305,6 +305,18 @@ soc_bluefield = {
>>> 'numa': false
>>> }
>>>
>>> +soc_cdx = {
>>> + 'description': 'AMD CDX',
>>> + 'implementer': '0x41',
>>> + 'part_number': '0xd42',
>>> + 'flags': [
>>> + ['RTE_MACHINE', '"cdx"'],
>>> + ['RTE_MAX_LCORE', 16],
>>> + ['RTE_MAX_NUMA_NODES', 1]
>>> + ],
>>> + 'numa': false
>>> +}
>>
>> Hi Nipun,
>>
>> Why we need a new arm platform/config? Is it because of above flags?
>> If it can work with default values, I think we can drop this patch.
>
> Hi Ferruh,
>
> CDX driver works with generic ARM compilation too (arm64_armv8_linux_gcc).
>
> The versal platforms supporting CDX have A78 cores, and adding this cdx config
> Helps to provide gcc option "march= armv8.4-a" which is for implementer
> "0xd42" (ARM cortex A78 cores)., whereas for generic ARM compilation
> "march= armv8-a".
>
> Maybe ARM guys can provide more information regarding if there is any impact
> on using generic architecture flag (i.e. march= armv8a) on A78 cores.
>
Hi Honnappa, Ruifeng,
Can you please support on this question, what is the difference of
'march= armv8-a' flag (comparing march= armv8a)?
Should we consider adding an arm config file to support this flag?
Thanks,
ferruh
^ permalink raw reply [flat|nested] 102+ messages in thread* RE: [PATCH v3 5/5] config/arm: add AMD CDX
2023-05-08 11:26 ` Ferruh Yigit
@ 2023-05-08 17:16 ` Honnappa Nagarahalli
2023-05-08 17:47 ` Ferruh Yigit
2023-05-09 5:55 ` Ruifeng Wang
1 sibling, 1 reply; 102+ messages in thread
From: Honnappa Nagarahalli @ 2023-05-08 17:16 UTC (permalink / raw)
To: Ferruh Yigit, Gupta, Nipun, dev, thomas, david.marchand, Ruifeng Wang
Cc: Anand, Harpreet, Agarwal, Nikhil, nd, nd
<snip>
> >>
> >> On 4/21/2023 3:54 PM, Nipun Gupta wrote:
> >>> Adding support for AMD CDX devices
> >>>
> >>> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> >>> ---
> >>> config/arm/arm64_cdx_linux_gcc | 17 +++++++++++++++++
> >>> config/arm/meson.build | 14 ++++++++++++++
> >>> 2 files changed, 31 insertions(+)
> >>> create mode 100644 config/arm/arm64_cdx_linux_gcc
> >>>
> >>> diff --git a/config/arm/arm64_cdx_linux_gcc
> >> b/config/arm/arm64_cdx_linux_gcc
> >>> new file mode 100644
> >>> index 0000000000..8e6d619dae
> >>> --- /dev/null
> >>> +++ b/config/arm/arm64_cdx_linux_gcc
> >>> @@ -0,0 +1,17 @@
> >>> +[binaries]
> >>> +c = ['ccache', 'aarch64-linux-gnu-gcc'] cpp = ['ccache',
> >>> +'aarch64-linux-gnu-g++'] ar = 'aarch64-linux-gnu-ar'
> >>> +as = 'aarch64-linux-gnu-as'
> >>> +strip = 'aarch64-linux-gnu-strip'
> >>> +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> >>> +pcap-config = ''
> >>> +
> >>> +[host_machine]
> >>> +system = 'linux'
> >>> +cpu_family = 'aarch64'
> >>> +cpu = 'armv8-a'
> >>> +endian = 'little'
> >>> +
> >>> +[properties]
> >>> +platform = 'cdx'
> >>> diff --git a/config/arm/meson.build b/config/arm/meson.build index
> >>> 5213434ca4..39b8929534 100644
> >>> --- a/config/arm/meson.build
> >>> +++ b/config/arm/meson.build
> >>> @@ -305,6 +305,18 @@ soc_bluefield = {
> >>> 'numa': false
> >>> }
> >>>
> >>> +soc_cdx = {
> >>> + 'description': 'AMD CDX',
> >>> + 'implementer': '0x41',
> >>> + 'part_number': '0xd42',
> >>> + 'flags': [
> >>> + ['RTE_MACHINE', '"cdx"'],
> >>> + ['RTE_MAX_LCORE', 16],
> >>> + ['RTE_MAX_NUMA_NODES', 1]
> >>> + ],
> >>> + 'numa': false
> >>> +}
> >>
> >> Hi Nipun,
> >>
> >> Why we need a new arm platform/config? Is it because of above flags?
> >> If it can work with default values, I think we can drop this patch.
> >
> > Hi Ferruh,
> >
> > CDX driver works with generic ARM compilation too
> (arm64_armv8_linux_gcc).
> >
> > The versal platforms supporting CDX have A78 cores, and adding this
> > cdx config Helps to provide gcc option "march= armv8.4-a" which is for
> > implementer "0xd42" (ARM cortex A78 cores)., whereas for generic ARM
> > compilation "march= armv8-a".
> >
> > Maybe ARM guys can provide more information regarding if there is any
> > impact on using generic architecture flag (i.e. march= armv8a) on A78 cores.
> >
>
> Hi Honnappa, Ruifeng,
>
> Can you please support on this question, what is the difference of 'march=
> armv8-a' flag (comparing march= armv8a)?
I am assuming you meant, 'march=armv8.4-a' vs 'march=armv8-a'.
'march=armv8-a' will compile for the ARM V8.0-a ISA which is the base V8-a version of the ISA. This compilation ensures that the binaries can run on any ARM V8-a platforms.
'march=armv8.4-a' will compile the binary for ARM V8.4 ISA. The binary will use additional (possibly instructions that improve performance) instructions which are not supported in ARM V8.0-a. This binary can run only on machines that support V8.4-a and higher. But, it will provide optimized binary for the target platform.
> Should we consider adding an arm config file to support this flag?
We support the optimized binary through the config files. We should keep these config changes.
>
> Thanks,
> ferruh
>
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v3 5/5] config/arm: add AMD CDX
2023-05-08 17:16 ` Honnappa Nagarahalli
@ 2023-05-08 17:47 ` Ferruh Yigit
2023-05-09 4:35 ` Gupta, Nipun
0 siblings, 1 reply; 102+ messages in thread
From: Ferruh Yigit @ 2023-05-08 17:47 UTC (permalink / raw)
To: Honnappa Nagarahalli, Gupta, Nipun, dev, thomas, david.marchand,
Ruifeng Wang
Cc: Anand, Harpreet, Agarwal, Nikhil, nd
On 5/8/2023 6:16 PM, Honnappa Nagarahalli wrote:
> <snip>
>
>>>>
>>>> On 4/21/2023 3:54 PM, Nipun Gupta wrote:
>>>>> Adding support for AMD CDX devices
>>>>>
>>>>> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
>>>>> ---
>>>>> config/arm/arm64_cdx_linux_gcc | 17 +++++++++++++++++
>>>>> config/arm/meson.build | 14 ++++++++++++++
>>>>> 2 files changed, 31 insertions(+)
>>>>> create mode 100644 config/arm/arm64_cdx_linux_gcc
>>>>>
>>>>> diff --git a/config/arm/arm64_cdx_linux_gcc
>>>> b/config/arm/arm64_cdx_linux_gcc
>>>>> new file mode 100644
>>>>> index 0000000000..8e6d619dae
>>>>> --- /dev/null
>>>>> +++ b/config/arm/arm64_cdx_linux_gcc
>>>>> @@ -0,0 +1,17 @@
>>>>> +[binaries]
>>>>> +c = ['ccache', 'aarch64-linux-gnu-gcc'] cpp = ['ccache',
>>>>> +'aarch64-linux-gnu-g++'] ar = 'aarch64-linux-gnu-ar'
>>>>> +as = 'aarch64-linux-gnu-as'
>>>>> +strip = 'aarch64-linux-gnu-strip'
>>>>> +pkgconfig = 'aarch64-linux-gnu-pkg-config'
>>>>> +pcap-config = ''
>>>>> +
>>>>> +[host_machine]
>>>>> +system = 'linux'
>>>>> +cpu_family = 'aarch64'
>>>>> +cpu = 'armv8-a'
>>>>> +endian = 'little'
>>>>> +
>>>>> +[properties]
>>>>> +platform = 'cdx'
>>>>> diff --git a/config/arm/meson.build b/config/arm/meson.build index
>>>>> 5213434ca4..39b8929534 100644
>>>>> --- a/config/arm/meson.build
>>>>> +++ b/config/arm/meson.build
>>>>> @@ -305,6 +305,18 @@ soc_bluefield = {
>>>>> 'numa': false
>>>>> }
>>>>>
>>>>> +soc_cdx = {
>>>>> + 'description': 'AMD CDX',
>>>>> + 'implementer': '0x41',
>>>>> + 'part_number': '0xd42',
>>>>> + 'flags': [
>>>>> + ['RTE_MACHINE', '"cdx"'],
>>>>> + ['RTE_MAX_LCORE', 16],
>>>>> + ['RTE_MAX_NUMA_NODES', 1]
>>>>> + ],
>>>>> + 'numa': false
>>>>> +}
>>>>
>>>> Hi Nipun,
>>>>
>>>> Why we need a new arm platform/config? Is it because of above flags?
>>>> If it can work with default values, I think we can drop this patch.
>>>
>>> Hi Ferruh,
>>>
>>> CDX driver works with generic ARM compilation too
>> (arm64_armv8_linux_gcc).
>>>
>>> The versal platforms supporting CDX have A78 cores, and adding this
>>> cdx config Helps to provide gcc option "march= armv8.4-a" which is for
>>> implementer "0xd42" (ARM cortex A78 cores)., whereas for generic ARM
>>> compilation "march= armv8-a".
>>>
>>> Maybe ARM guys can provide more information regarding if there is any
>>> impact on using generic architecture flag (i.e. march= armv8a) on A78 cores.
>>>
>>
>> Hi Honnappa, Ruifeng,
>>
>> Can you please support on this question, what is the difference of 'march=
>> armv8-a' flag (comparing march= armv8a)?
> I am assuming you meant, 'march=armv8.4-a' vs 'march=armv8-a'.
>
Yes
> 'march=armv8-a' will compile for the ARM V8.0-a ISA which is the base V8-a version of the ISA. This compilation ensures that the binaries can run on any ARM V8-a platforms.
> 'march=armv8.4-a' will compile the binary for ARM V8.4 ISA. The binary will use additional (possibly instructions that improve performance) instructions which are not supported in ARM V8.0-a. This binary can run only on machines that support V8.4-a and higher. But, it will provide optimized binary for the target platform.
>
>> Should we consider adding an arm config file to support this flag?
> We support the optimized binary through the config files. We should keep these config changes.
>
Got it, thanks for the guidance.
^ permalink raw reply [flat|nested] 102+ messages in thread* RE: [PATCH v3 5/5] config/arm: add AMD CDX
2023-05-08 17:47 ` Ferruh Yigit
@ 2023-05-09 4:35 ` Gupta, Nipun
0 siblings, 0 replies; 102+ messages in thread
From: Gupta, Nipun @ 2023-05-09 4:35 UTC (permalink / raw)
To: Yigit, Ferruh, Honnappa Nagarahalli, dev, thomas, david.marchand,
Ruifeng Wang
Cc: Anand, Harpreet, Agarwal, Nikhil, nd
[AMD Official Use Only - General]
> -----Original Message-----
> From: Yigit, Ferruh <Ferruh.Yigit@amd.com>
> Sent: Monday, May 8, 2023 11:17 PM
> To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Gupta, Nipun
> <Nipun.Gupta@amd.com>; dev@dpdk.org; thomas@monjalon.net;
> david.marchand@redhat.com; Ruifeng Wang <Ruifeng.Wang@arm.com>
> Cc: Anand, Harpreet <harpreet.anand@amd.com>; Agarwal, Nikhil
> <nikhil.agarwal@amd.com>; nd <nd@arm.com>
> Subject: Re: [PATCH v3 5/5] config/arm: add AMD CDX
>
> On 5/8/2023 6:16 PM, Honnappa Nagarahalli wrote:
> > <snip>
> >
> >>>>
> >>>> On 4/21/2023 3:54 PM, Nipun Gupta wrote:
> >>>>> Adding support for AMD CDX devices
> >>>>>
> >>>>> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> >>>>> ---
> >>>>> config/arm/arm64_cdx_linux_gcc | 17 +++++++++++++++++
> >>>>> config/arm/meson.build | 14 ++++++++++++++
> >>>>> 2 files changed, 31 insertions(+)
> >>>>> create mode 100644 config/arm/arm64_cdx_linux_gcc
> >>>>>
> >>>>> diff --git a/config/arm/arm64_cdx_linux_gcc
> >>>> b/config/arm/arm64_cdx_linux_gcc
> >>>>> new file mode 100644
> >>>>> index 0000000000..8e6d619dae
> >>>>> --- /dev/null
> >>>>> +++ b/config/arm/arm64_cdx_linux_gcc
> >>>>> @@ -0,0 +1,17 @@
> >>>>> +[binaries]
> >>>>> +c = ['ccache', 'aarch64-linux-gnu-gcc'] cpp = ['ccache',
> >>>>> +'aarch64-linux-gnu-g++'] ar = 'aarch64-linux-gnu-ar'
> >>>>> +as = 'aarch64-linux-gnu-as'
> >>>>> +strip = 'aarch64-linux-gnu-strip'
> >>>>> +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> >>>>> +pcap-config = ''
> >>>>> +
> >>>>> +[host_machine]
> >>>>> +system = 'linux'
> >>>>> +cpu_family = 'aarch64'
> >>>>> +cpu = 'armv8-a'
> >>>>> +endian = 'little'
> >>>>> +
> >>>>> +[properties]
> >>>>> +platform = 'cdx'
> >>>>> diff --git a/config/arm/meson.build b/config/arm/meson.build index
> >>>>> 5213434ca4..39b8929534 100644
> >>>>> --- a/config/arm/meson.build
> >>>>> +++ b/config/arm/meson.build
> >>>>> @@ -305,6 +305,18 @@ soc_bluefield = {
> >>>>> 'numa': false
> >>>>> }
> >>>>>
> >>>>> +soc_cdx = {
> >>>>> + 'description': 'AMD CDX',
> >>>>> + 'implementer': '0x41',
> >>>>> + 'part_number': '0xd42',
> >>>>> + 'flags': [
> >>>>> + ['RTE_MACHINE', '"cdx"'],
> >>>>> + ['RTE_MAX_LCORE', 16],
> >>>>> + ['RTE_MAX_NUMA_NODES', 1]
> >>>>> + ],
> >>>>> + 'numa': false
> >>>>> +}
> >>>>
> >>>> Hi Nipun,
> >>>>
> >>>> Why we need a new arm platform/config? Is it because of above flags?
> >>>> If it can work with default values, I think we can drop this patch.
> >>>
> >>> Hi Ferruh,
> >>>
> >>> CDX driver works with generic ARM compilation too
> >> (arm64_armv8_linux_gcc).
> >>>
> >>> The versal platforms supporting CDX have A78 cores, and adding this
> >>> cdx config Helps to provide gcc option "march= armv8.4-a" which is for
> >>> implementer "0xd42" (ARM cortex A78 cores)., whereas for generic
> ARM
> >>> compilation "march= armv8-a".
> >>>
> >>> Maybe ARM guys can provide more information regarding if there is any
> >>> impact on using generic architecture flag (i.e. march= armv8a) on A78
> cores.
> >>>
> >>
> >> Hi Honnappa, Ruifeng,
> >>
> >> Can you please support on this question, what is the difference of
> 'march=
> >> armv8-a' flag (comparing march= armv8a)?
> > I am assuming you meant, 'march=armv8.4-a' vs 'march=armv8-a'.
> >
>
> Yes
>
> > 'march=armv8-a' will compile for the ARM V8.0-a ISA which is the base V8-a
> version of the ISA. This compilation ensures that the binaries can run on any
> ARM V8-a platforms.
> > 'march=armv8.4-a' will compile the binary for ARM V8.4 ISA. The binary will
> use additional (possibly instructions that improve performance) instructions
> which are not supported in ARM V8.0-a. This binary can run only on machines
> that support V8.4-a and higher. But, it will provide optimized binary for the
> target platform.
> >
> >> Should we consider adding an arm config file to support this flag?
> > We support the optimized binary through the config files. We should keep
> these config changes.
> >
>
> Got it, thanks for the guidance.
Thanks Honnappa, Ferruh,
As discussed with Thomas too, we will add this as a separate platform patch,
with more platform details separated from CDX bus (patch sent out v4).
^ permalink raw reply [flat|nested] 102+ messages in thread
* RE: [PATCH v3 5/5] config/arm: add AMD CDX
2023-05-08 11:26 ` Ferruh Yigit
2023-05-08 17:16 ` Honnappa Nagarahalli
@ 2023-05-09 5:55 ` Ruifeng Wang
2023-06-14 10:30 ` Ferruh Yigit
1 sibling, 1 reply; 102+ messages in thread
From: Ruifeng Wang @ 2023-05-09 5:55 UTC (permalink / raw)
To: Ferruh Yigit, Gupta, Nipun, dev, thomas, david.marchand,
Honnappa Nagarahalli
Cc: Anand, Harpreet, Agarwal, Nikhil, nd
> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit@amd.com>
> Sent: Monday, May 8, 2023 7:27 PM
> To: Gupta, Nipun <Nipun.Gupta@amd.com>; dev@dpdk.org; thomas@monjalon.net;
> david.marchand@redhat.com; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Ruifeng
> Wang <Ruifeng.Wang@arm.com>
> Cc: Anand, Harpreet <harpreet.anand@amd.com>; Agarwal, Nikhil <nikhil.agarwal@amd.com>
> Subject: Re: [PATCH v3 5/5] config/arm: add AMD CDX
>
> On 5/8/2023 11:24 AM, Gupta, Nipun wrote:
> >
> >
> >> -----Original Message-----
> >> From: Yigit, Ferruh <Ferruh.Yigit@amd.com>
> >> Sent: Thursday, May 4, 2023 8:59 PM
> >> To: Gupta, Nipun <Nipun.Gupta@amd.com>; dev@dpdk.org;
> >> thomas@monjalon.net; david.marchand@redhat.com
> >> Cc: Anand, Harpreet <harpreet.anand@amd.com>; Agarwal, Nikhil
> >> <nikhil.agarwal@amd.com>
> >> Subject: Re: [PATCH v3 5/5] config/arm: add AMD CDX
> >>
> >> On 4/21/2023 3:54 PM, Nipun Gupta wrote:
> >>> Adding support for AMD CDX devices
> >>>
> >>> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> >>> ---
> >>> config/arm/arm64_cdx_linux_gcc | 17 +++++++++++++++++
> >>> config/arm/meson.build | 14 ++++++++++++++
> >>> 2 files changed, 31 insertions(+)
> >>> create mode 100644 config/arm/arm64_cdx_linux_gcc
> >>>
> >>> diff --git a/config/arm/arm64_cdx_linux_gcc
> >> b/config/arm/arm64_cdx_linux_gcc
> >>> new file mode 100644
> >>> index 0000000000..8e6d619dae
> >>> --- /dev/null
> >>> +++ b/config/arm/arm64_cdx_linux_gcc
> >>> @@ -0,0 +1,17 @@
> >>> +[binaries]
> >>> +c = ['ccache', 'aarch64-linux-gnu-gcc'] cpp = ['ccache',
> >>> +'aarch64-linux-gnu-g++'] ar = 'aarch64-linux-gnu-ar'
> >>> +as = 'aarch64-linux-gnu-as'
> >>> +strip = 'aarch64-linux-gnu-strip'
> >>> +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> >>> +pcap-config = ''
> >>> +
> >>> +[host_machine]
> >>> +system = 'linux'
> >>> +cpu_family = 'aarch64'
> >>> +cpu = 'armv8-a'
> >>> +endian = 'little'
> >>> +
> >>> +[properties]
> >>> +platform = 'cdx'
> >>> diff --git a/config/arm/meson.build b/config/arm/meson.build index
> >>> 5213434ca4..39b8929534 100644
> >>> --- a/config/arm/meson.build
> >>> +++ b/config/arm/meson.build
> >>> @@ -305,6 +305,18 @@ soc_bluefield = {
> >>> 'numa': false
> >>> }
> >>>
> >>> +soc_cdx = {
> >>> + 'description': 'AMD CDX',
> >>> + 'implementer': '0x41',
> >>> + 'part_number': '0xd42',
> >>> + 'flags': [
> >>> + ['RTE_MACHINE', '"cdx"'],
> >>> + ['RTE_MAX_LCORE', 16],
> >>> + ['RTE_MAX_NUMA_NODES', 1]
> >>> + ],
> >>> + 'numa': false
> >>> +}
> >>
> >> Hi Nipun,
> >>
> >> Why we need a new arm platform/config? Is it because of above flags?
> >> If it can work with default values, I think we can drop this patch.
> >
> > Hi Ferruh,
> >
> > CDX driver works with generic ARM compilation too (arm64_armv8_linux_gcc).
> >
> > The versal platforms supporting CDX have A78 cores, and adding this
> > cdx config Helps to provide gcc option "march= armv8.4-a" which is for
> > implementer "0xd42" (ARM cortex A78 cores)., whereas for generic ARM
> > compilation "march= armv8-a".
> >
> > Maybe ARM guys can provide more information regarding if there is any
> > impact on using generic architecture flag (i.e. march= armv8a) on A78 cores.
> >
>
> Hi Honnappa, Ruifeng,
>
> Can you please support on this question, what is the difference of 'march= armv8-a' flag
> (comparing march= armv8a)?
> Should we consider adding an arm config file to support this flag?
I see there is a new version without change to config file.
FWIW, native build is fine without this change. Because the specific (implementer, part number) flags
are already in place. What enabled by this change are options for soc build (-Dplatform=cdx) and
cross-build (--cross-file arm64_cdx_linux_gcc).
Regards,
Ruifeng
>
> Thanks,
> ferruh
>
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v3 5/5] config/arm: add AMD CDX
2023-05-09 5:55 ` Ruifeng Wang
@ 2023-06-14 10:30 ` Ferruh Yigit
2023-06-15 7:00 ` Ruifeng Wang
0 siblings, 1 reply; 102+ messages in thread
From: Ferruh Yigit @ 2023-06-14 10:30 UTC (permalink / raw)
To: Ruifeng Wang, Gupta, Nipun, dev, thomas, david.marchand,
Honnappa Nagarahalli
Cc: Anand, Harpreet, Agarwal, Nikhil, nd
On 5/9/2023 6:55 AM, Ruifeng Wang wrote:
>> -----Original Message-----
>> From: Ferruh Yigit <ferruh.yigit@amd.com>
>> Sent: Monday, May 8, 2023 7:27 PM
>> To: Gupta, Nipun <Nipun.Gupta@amd.com>; dev@dpdk.org; thomas@monjalon.net;
>> david.marchand@redhat.com; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Ruifeng
>> Wang <Ruifeng.Wang@arm.com>
>> Cc: Anand, Harpreet <harpreet.anand@amd.com>; Agarwal, Nikhil <nikhil.agarwal@amd.com>
>> Subject: Re: [PATCH v3 5/5] config/arm: add AMD CDX
>>
>> On 5/8/2023 11:24 AM, Gupta, Nipun wrote:
>>>
>>>
>>>> -----Original Message-----
>>>> From: Yigit, Ferruh <Ferruh.Yigit@amd.com>
>>>> Sent: Thursday, May 4, 2023 8:59 PM
>>>> To: Gupta, Nipun <Nipun.Gupta@amd.com>; dev@dpdk.org;
>>>> thomas@monjalon.net; david.marchand@redhat.com
>>>> Cc: Anand, Harpreet <harpreet.anand@amd.com>; Agarwal, Nikhil
>>>> <nikhil.agarwal@amd.com>
>>>> Subject: Re: [PATCH v3 5/5] config/arm: add AMD CDX
>>>>
>>>> On 4/21/2023 3:54 PM, Nipun Gupta wrote:
>>>>> Adding support for AMD CDX devices
>>>>>
>>>>> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
>>>>> ---
>>>>> config/arm/arm64_cdx_linux_gcc | 17 +++++++++++++++++
>>>>> config/arm/meson.build | 14 ++++++++++++++
>>>>> 2 files changed, 31 insertions(+)
>>>>> create mode 100644 config/arm/arm64_cdx_linux_gcc
>>>>>
>>>>> diff --git a/config/arm/arm64_cdx_linux_gcc
>>>> b/config/arm/arm64_cdx_linux_gcc
>>>>> new file mode 100644
>>>>> index 0000000000..8e6d619dae
>>>>> --- /dev/null
>>>>> +++ b/config/arm/arm64_cdx_linux_gcc
>>>>> @@ -0,0 +1,17 @@
>>>>> +[binaries]
>>>>> +c = ['ccache', 'aarch64-linux-gnu-gcc'] cpp = ['ccache',
>>>>> +'aarch64-linux-gnu-g++'] ar = 'aarch64-linux-gnu-ar'
>>>>> +as = 'aarch64-linux-gnu-as'
>>>>> +strip = 'aarch64-linux-gnu-strip'
>>>>> +pkgconfig = 'aarch64-linux-gnu-pkg-config'
>>>>> +pcap-config = ''
>>>>> +
>>>>> +[host_machine]
>>>>> +system = 'linux'
>>>>> +cpu_family = 'aarch64'
>>>>> +cpu = 'armv8-a'
>>>>> +endian = 'little'
>>>>> +
>>>>> +[properties]
>>>>> +platform = 'cdx'
>>>>> diff --git a/config/arm/meson.build b/config/arm/meson.build index
>>>>> 5213434ca4..39b8929534 100644
>>>>> --- a/config/arm/meson.build
>>>>> +++ b/config/arm/meson.build
>>>>> @@ -305,6 +305,18 @@ soc_bluefield = {
>>>>> 'numa': false
>>>>> }
>>>>>
>>>>> +soc_cdx = {
>>>>> + 'description': 'AMD CDX',
>>>>> + 'implementer': '0x41',
>>>>> + 'part_number': '0xd42',
>>>>> + 'flags': [
>>>>> + ['RTE_MACHINE', '"cdx"'],
>>>>> + ['RTE_MAX_LCORE', 16],
>>>>> + ['RTE_MAX_NUMA_NODES', 1]
>>>>> + ],
>>>>> + 'numa': false
>>>>> +}
>>>>
>>>> Hi Nipun,
>>>>
>>>> Why we need a new arm platform/config? Is it because of above flags?
>>>> If it can work with default values, I think we can drop this patch.
>>>
>>> Hi Ferruh,
>>>
>>> CDX driver works with generic ARM compilation too (arm64_armv8_linux_gcc).
>>>
>>> The versal platforms supporting CDX have A78 cores, and adding this
>>> cdx config Helps to provide gcc option "march= armv8.4-a" which is for
>>> implementer "0xd42" (ARM cortex A78 cores)., whereas for generic ARM
>>> compilation "march= armv8-a".
>>>
>>> Maybe ARM guys can provide more information regarding if there is any
>>> impact on using generic architecture flag (i.e. march= armv8a) on A78 cores.
>>>
>>
>> Hi Honnappa, Ruifeng,
>>
>> Can you please support on this question, what is the difference of 'march= armv8-a' flag
>> (comparing march= armv8a)?
>> Should we consider adding an arm config file to support this flag?
>
> I see there is a new version without change to config file.
> FWIW, native build is fine without this change. Because the specific (implementer, part number) flags
> are already in place. What enabled by this change are options for soc build (-Dplatform=cdx) and
> cross-build (--cross-file arm64_cdx_linux_gcc).
>
Hi Ruifeng, Honnappa,
Config file will come as standalone patch, it only separated from this set.
And config file is required mainly for '--march=armv8.4-a' parameter.
There are multiple configs using the same parameter, is it a good option
to create a common config for 'armv8.4-a', similar to 'generic' one?
Or is it preferred that each SoC adding its own config, as done in this
patch?
Thanks,
ferruh
^ permalink raw reply [flat|nested] 102+ messages in thread* RE: [PATCH v3 5/5] config/arm: add AMD CDX
2023-06-14 10:30 ` Ferruh Yigit
@ 2023-06-15 7:00 ` Ruifeng Wang
0 siblings, 0 replies; 102+ messages in thread
From: Ruifeng Wang @ 2023-06-15 7:00 UTC (permalink / raw)
To: Ferruh Yigit, Gupta, Nipun, dev, thomas, david.marchand,
Honnappa Nagarahalli
Cc: Anand, Harpreet, Agarwal, Nikhil, nd, nd
> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit@amd.com>
> Sent: Wednesday, June 14, 2023 6:31 PM
> To: Ruifeng Wang <Ruifeng.Wang@arm.com>; Gupta, Nipun <Nipun.Gupta@amd.com>; dev@dpdk.org;
> thomas@monjalon.net; david.marchand@redhat.com; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>
> Cc: Anand, Harpreet <harpreet.anand@amd.com>; Agarwal, Nikhil <nikhil.agarwal@amd.com>; nd
> <nd@arm.com>
> Subject: Re: [PATCH v3 5/5] config/arm: add AMD CDX
>
> On 5/9/2023 6:55 AM, Ruifeng Wang wrote:
> >> -----Original Message-----
> >> From: Ferruh Yigit <ferruh.yigit@amd.com>
> >> Sent: Monday, May 8, 2023 7:27 PM
> >> To: Gupta, Nipun <Nipun.Gupta@amd.com>; dev@dpdk.org;
> >> thomas@monjalon.net; david.marchand@redhat.com; Honnappa Nagarahalli
> >> <Honnappa.Nagarahalli@arm.com>; Ruifeng Wang <Ruifeng.Wang@arm.com>
> >> Cc: Anand, Harpreet <harpreet.anand@amd.com>; Agarwal, Nikhil
> >> <nikhil.agarwal@amd.com>
> >> Subject: Re: [PATCH v3 5/5] config/arm: add AMD CDX
> >>
> >> On 5/8/2023 11:24 AM, Gupta, Nipun wrote:
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Yigit, Ferruh <Ferruh.Yigit@amd.com>
> >>>> Sent: Thursday, May 4, 2023 8:59 PM
> >>>> To: Gupta, Nipun <Nipun.Gupta@amd.com>; dev@dpdk.org;
> >>>> thomas@monjalon.net; david.marchand@redhat.com
> >>>> Cc: Anand, Harpreet <harpreet.anand@amd.com>; Agarwal, Nikhil
> >>>> <nikhil.agarwal@amd.com>
> >>>> Subject: Re: [PATCH v3 5/5] config/arm: add AMD CDX
> >>>>
> >>>> On 4/21/2023 3:54 PM, Nipun Gupta wrote:
> >>>>> Adding support for AMD CDX devices
> >>>>>
> >>>>> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> >>>>> ---
> >>>>> config/arm/arm64_cdx_linux_gcc | 17 +++++++++++++++++
> >>>>> config/arm/meson.build | 14 ++++++++++++++
> >>>>> 2 files changed, 31 insertions(+) create mode 100644
> >>>>> config/arm/arm64_cdx_linux_gcc
> >>>>>
> >>>>> diff --git a/config/arm/arm64_cdx_linux_gcc
> >>>> b/config/arm/arm64_cdx_linux_gcc
> >>>>> new file mode 100644
> >>>>> index 0000000000..8e6d619dae
> >>>>> --- /dev/null
> >>>>> +++ b/config/arm/arm64_cdx_linux_gcc
> >>>>> @@ -0,0 +1,17 @@
> >>>>> +[binaries]
> >>>>> +c = ['ccache', 'aarch64-linux-gnu-gcc'] cpp = ['ccache',
> >>>>> +'aarch64-linux-gnu-g++'] ar = 'aarch64-linux-gnu-ar'
> >>>>> +as = 'aarch64-linux-gnu-as'
> >>>>> +strip = 'aarch64-linux-gnu-strip'
> >>>>> +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> >>>>> +pcap-config = ''
> >>>>> +
> >>>>> +[host_machine]
> >>>>> +system = 'linux'
> >>>>> +cpu_family = 'aarch64'
> >>>>> +cpu = 'armv8-a'
> >>>>> +endian = 'little'
> >>>>> +
> >>>>> +[properties]
> >>>>> +platform = 'cdx'
> >>>>> diff --git a/config/arm/meson.build b/config/arm/meson.build index
> >>>>> 5213434ca4..39b8929534 100644
> >>>>> --- a/config/arm/meson.build
> >>>>> +++ b/config/arm/meson.build
> >>>>> @@ -305,6 +305,18 @@ soc_bluefield = {
> >>>>> 'numa': false
> >>>>> }
> >>>>>
> >>>>> +soc_cdx = {
> >>>>> + 'description': 'AMD CDX',
> >>>>> + 'implementer': '0x41',
> >>>>> + 'part_number': '0xd42',
> >>>>> + 'flags': [
> >>>>> + ['RTE_MACHINE', '"cdx"'],
> >>>>> + ['RTE_MAX_LCORE', 16],
> >>>>> + ['RTE_MAX_NUMA_NODES', 1]
> >>>>> + ],
> >>>>> + 'numa': false
> >>>>> +}
> >>>>
> >>>> Hi Nipun,
> >>>>
> >>>> Why we need a new arm platform/config? Is it because of above flags?
> >>>> If it can work with default values, I think we can drop this patch.
> >>>
> >>> Hi Ferruh,
> >>>
> >>> CDX driver works with generic ARM compilation too (arm64_armv8_linux_gcc).
> >>>
> >>> The versal platforms supporting CDX have A78 cores, and adding this
> >>> cdx config Helps to provide gcc option "march= armv8.4-a" which is
> >>> for implementer "0xd42" (ARM cortex A78 cores)., whereas for generic
> >>> ARM compilation "march= armv8-a".
> >>>
> >>> Maybe ARM guys can provide more information regarding if there is
> >>> any impact on using generic architecture flag (i.e. march= armv8a) on A78 cores.
> >>>
> >>
> >> Hi Honnappa, Ruifeng,
> >>
> >> Can you please support on this question, what is the difference of
> >> 'march= armv8-a' flag (comparing march= armv8a)?
> >> Should we consider adding an arm config file to support this flag?
> >
> > I see there is a new version without change to config file.
> > FWIW, native build is fine without this change. Because the specific
> > (implementer, part number) flags are already in place. What enabled by
> > this change are options for soc build (-Dplatform=cdx) and cross-build (--cross-file
> arm64_cdx_linux_gcc).
> >
>
> Hi Ruifeng, Honnappa,
>
> Config file will come as standalone patch, it only separated from this set.
>
> And config file is required mainly for '--march=armv8.4-a' parameter.
>
> There are multiple configs using the same parameter, is it a good option to create a
> common config for 'armv8.4-a', similar to 'generic' one?
> Or is it preferred that each SoC adding its own config, as done in this patch?
Hi Ferruh,
It is preferred that each SoC adding its own config.
SoCs at the same ISA level (e.g. armv8.4-a) can have different optional features/extensions supported.
Therefore, SoC config is provided alongside with part number config.
Thanks.
>
> Thanks,
> ferruh
>
>
>
>
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v4 0/4] Support AMD CDX bus, for FPGA based CDX devices. The CDX
2023-01-24 14:07 [RFC PATCH 0/6] add support for CDX bus Nipun Gupta
` (8 preceding siblings ...)
2023-04-21 14:54 ` [PATCH v3 0/5] Support AMD CDX bus, for FPGA based CDX devices. The CDX Nipun Gupta
@ 2023-05-08 11:18 ` Nipun Gupta
2023-05-08 11:18 ` [PATCH v4 1/4] bus/cdx: introduce cdx bus Nipun Gupta
` (4 more replies)
2023-05-25 10:08 ` [PATCH v5 0/5] Support AMD CDX bus Nipun Gupta
` (3 subsequent siblings)
13 siblings, 5 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-05-08 11:18 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
It uses sysfs interface and the vfio-cdx driver to discover
and initialize the CDX devices.
The CDX bus and VFIO support is available at Xilinx open source tree:
https://github.com/Xilinx/linux-xlnx (drivers/cdx/ and drivers/vfio/cdx)
Linux CDX bus patches has been added into linux next:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/cdx
VFIO patches are also submitted in upstream:
https://www.spinics.net/lists/kvm/msg310623.html
CDX is a Hardware Architecture designed for AMD FPGA devices. It
consists of mechanism for interaction between FPGA, Firmware and
the APUs (Application CPUs).
Firmware resides on RPU (Realtime CPUs) which interacts with
the FPGA program manager and the APUs. The RPU provides memory-mapped
interface (RPU if) which is used to communicate with APUs.
VFIO CDX driver provides the CDX device resources like MMIO and interrupts
to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
driver to discover and initialize the CDX devices for user-space
applications.
Changes v3->v4:
- removed platform specific patch (adding config for ARM CDX)
from this series
Changes v2->v3:
- merged cdx bus compilation enablement in the first patch
- fixed issue reported by check-git-log.sh
- updated release notes
- updated offset to uint64_t instead of off_t in cdx_map_resource
Changes v1->v2:
- Moved file rte_cdx_bus.h to internal bus_cdx_driver.h
and added this file to deivce_cdx_headers
- Moved cdx.h to private.h
- Removed rte_ prefix from the static symbols in .c files.
Changes RFC->v1:
- Marked few API's as internal which were not required
to be provided to user.
Nipun Gupta (4):
bus/cdx: introduce cdx bus
bus/cdx: add DMA map and unmap support
bus/cdx: add support for MSI
bus/cdx: support plug unplug and dev iterator
MAINTAINERS | 5 +
doc/guides/rel_notes/release_23_07.rst | 6 +
drivers/bus/cdx/bus_cdx_driver.h | 227 ++++++++
drivers/bus/cdx/cdx.c | 693 +++++++++++++++++++++++++
drivers/bus/cdx/cdx_logs.h | 37 ++
drivers/bus/cdx/cdx_vfio.c | 615 ++++++++++++++++++++++
drivers/bus/cdx/meson.build | 13 +
drivers/bus/cdx/private.h | 49 ++
drivers/bus/cdx/version.map | 13 +
drivers/bus/meson.build | 1 +
lib/eal/common/eal_common_interrupts.c | 21 +
lib/eal/common/eal_interrupts.h | 1 +
lib/eal/include/rte_interrupts.h | 32 ++
lib/eal/version.map | 2 +
14 files changed, 1715 insertions(+)
create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
create mode 100644 drivers/bus/cdx/cdx.c
create mode 100644 drivers/bus/cdx/cdx_logs.h
create mode 100644 drivers/bus/cdx/cdx_vfio.c
create mode 100644 drivers/bus/cdx/meson.build
create mode 100644 drivers/bus/cdx/private.h
create mode 100644 drivers/bus/cdx/version.map
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v4 1/4] bus/cdx: introduce cdx bus
2023-05-08 11:18 ` [PATCH v4 0/4] Support AMD CDX bus, for FPGA based CDX devices. The CDX Nipun Gupta
@ 2023-05-08 11:18 ` Nipun Gupta
2023-05-09 6:54 ` Xia, Chenbo
2023-05-24 11:14 ` Thomas Monjalon
2023-05-08 11:18 ` [PATCH v4 2/4] bus/cdx: add DMA map and unmap support Nipun Gupta
` (3 subsequent siblings)
4 siblings, 2 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-05-08 11:18 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
CDX bus supports multiple type of devices, which can be
exposed to user-space via vfio-cdx.
vfio-cdx provides the MMIO IO_MEMORY regions as well as the
DMA interface for the device (IOMMU).
This support aims to enable the DPDK to support the cdx
devices in user-space using VFIO interface.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
MAINTAINERS | 5 +
doc/guides/rel_notes/release_23_07.rst | 6 +
drivers/bus/cdx/bus_cdx_driver.h | 201 ++++++++++
drivers/bus/cdx/cdx.c | 520 +++++++++++++++++++++++++
drivers/bus/cdx/cdx_logs.h | 37 ++
drivers/bus/cdx/cdx_vfio.c | 437 +++++++++++++++++++++
drivers/bus/cdx/meson.build | 13 +
drivers/bus/cdx/private.h | 49 +++
drivers/bus/cdx/version.map | 11 +
drivers/bus/meson.build | 1 +
10 files changed, 1280 insertions(+)
create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
create mode 100644 drivers/bus/cdx/cdx.c
create mode 100644 drivers/bus/cdx/cdx_logs.h
create mode 100644 drivers/bus/cdx/cdx_vfio.c
create mode 100644 drivers/bus/cdx/meson.build
create mode 100644 drivers/bus/cdx/private.h
create mode 100644 drivers/bus/cdx/version.map
diff --git a/MAINTAINERS b/MAINTAINERS
index 8df23e5099..1f9b6af9b9 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -569,6 +569,11 @@ M: Parav Pandit <parav@nvidia.com>
M: Xueming Li <xuemingl@nvidia.com>
F: drivers/bus/auxiliary/
+CDX bus driver
+M: Nipun Gupta <nipun.gupta@amd.com>
+M: Nikhil Agarwal <nikhil.agarwal@amd.com>
+F: drivers/bus/cdx/
+
Intel FPGA bus
M: Rosen Xu <rosen.xu@intel.com>
F: drivers/bus/ifpga/
diff --git a/doc/guides/rel_notes/release_23_07.rst b/doc/guides/rel_notes/release_23_07.rst
index a9b1293689..a5f581dc3d 100644
--- a/doc/guides/rel_notes/release_23_07.rst
+++ b/doc/guides/rel_notes/release_23_07.rst
@@ -55,6 +55,12 @@ New Features
Also, make sure to start the actual text at the margin.
=======================================================
+* **Added CDX bus support.**
+
+ CDX bus driver has been added to support AMD CDX bus, which operates
+ on FPGA based CDX devices. The CDX devices are memory mapped on system
+ bus for embedded CPUs.
+
Removed Items
-------------
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
new file mode 100644
index 0000000000..7edcb019eb
--- /dev/null
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -0,0 +1,201 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef _BUS_CDX_DRIVER_H_
+#define _BUS_CDX_DRIVER_H_
+
+/**
+ * @file
+ *
+ * CDX bus interface
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <limits.h>
+#include <errno.h>
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <bus_driver.h>
+#include <dev_driver.h>
+#include <rte_debug.h>
+#include <rte_interrupts.h>
+#include <rte_dev.h>
+#include <rte_bus.h>
+
+/* Forward declarations */
+struct rte_cdx_device;
+struct rte_cdx_driver;
+
+#define CDX_MAX_RESOURCE 4
+
+/** List of CDX devices */
+RTE_TAILQ_HEAD(rte_cdx_device_list, rte_cdx_device);
+/** List of CDX drivers */
+RTE_TAILQ_HEAD(rte_cdx_driver_list, rte_cdx_driver);
+
+/* CDX Bus iterators */
+#define FOREACH_DEVICE_ON_CDXBUS(p) \
+ RTE_TAILQ_FOREACH(p, &rte_cdx_bus.device_list, next)
+
+#define FOREACH_DRIVER_ON_CDXBUS(p) \
+ RTE_TAILQ_FOREACH(p, &rte_cdx_bus.driver_list, next)
+
+/** Any CDX device identifier (vendor, device) */
+#define RTE_CDX_ANY_ID (0xffff)
+
+#define RTE_PMD_REGISTER_CDX_TABLE(name, table) \
+static const char DRV_EXP_TAG(name, cdx_tbl_export)[] __rte_used = \
+RTE_STR(table)
+
+/**
+ * A structure describing an ID for a CDX driver. Each driver provides a
+ * table of these IDs for each device that it supports.
+ */
+struct rte_cdx_id {
+ uint16_t vendor_id; /**< Vendor ID. */
+ uint16_t device_id; /**< Device ID. */
+};
+
+/**
+ * A structure describing a CDX device.
+ */
+struct rte_cdx_device {
+ RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
+ struct rte_device device; /**< Inherit core device */
+ struct rte_cdx_id id; /**< CDX ID. */
+ struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
+ /**< CDX Memory Resource */
+};
+
+/**
+ * @internal
+ * Helper macro for drivers that need to convert to struct rte_cdx_device.
+ */
+#define RTE_DEV_TO_CDX_DEV(ptr) \
+ container_of(ptr, struct rte_cdx_device, device)
+
+#define RTE_DEV_TO_CDX_DEV_CONST(ptr) \
+ container_of(ptr, const struct rte_cdx_device, device)
+
+#define RTE_ETH_DEV_TO_CDX_DEV(eth_dev) RTE_DEV_TO_CDX_DEV((eth_dev)->device)
+
+#ifdef __cplusplus
+/** C++ macro used to help building up tables of device IDs */
+#define RTE_CDX_DEVICE(vend, dev) \
+ (vend), \
+ (dev)
+#else
+/** Macro used to help building up tables of device IDs */
+#define RTE_CDX_DEVICE(vend, dev) \
+ .vendor_id = (vend), \
+ .device_id = (dev)
+#endif
+
+/**
+ * Initialisation function for the driver called during CDX probing.
+ */
+typedef int (rte_cdx_probe_t)(struct rte_cdx_driver *, struct rte_cdx_device *);
+
+/**
+ * Uninitialisation function for the driver called during hotplugging.
+ */
+typedef int (rte_cdx_remove_t)(struct rte_cdx_device *);
+
+/**
+ * A structure describing a CDX driver.
+ */
+struct rte_cdx_driver {
+ RTE_TAILQ_ENTRY(rte_cdx_driver) next; /**< Next in list. */
+ struct rte_driver driver; /**< Inherit core driver. */
+ struct rte_cdx_bus *bus; /**< CDX bus reference. */
+ rte_cdx_probe_t *probe; /**< Device probe function. */
+ rte_cdx_remove_t *remove; /**< Device remove function. */
+ const struct rte_cdx_id *id_table; /**< ID table, NULL terminated. */
+ uint32_t drv_flags; /**< Flags RTE_CDX_DRV_*. */
+};
+
+/**
+ * Structure describing the CDX bus
+ */
+struct rte_cdx_bus {
+ struct rte_bus bus; /**< Inherit the generic class */
+ struct rte_cdx_device_list device_list; /**< List of CDX devices */
+ struct rte_cdx_driver_list driver_list; /**< List of CDX drivers */
+};
+
+/**
+ * Get Pathname of CDX devices directory.
+ *
+ * @return
+ * sysfs path for CDX devices.
+ */
+__rte_internal
+const char *rte_cdx_get_sysfs_path(void);
+
+/**
+ * Map the CDX device resources in user space virtual memory address
+ *
+ * @param dev
+ * A pointer to a rte_cdx_device structure describing the device
+ * to use
+ *
+ * @return
+ * 0 on success, negative on error and positive if no driver
+ * is found for the device.
+ */
+__rte_internal
+int rte_cdx_map_device(struct rte_cdx_device *dev);
+
+/**
+ * Unmap this device
+ *
+ * @param dev
+ * A pointer to a rte_cdx_device structure describing the device
+ * to use
+ */
+__rte_internal
+void rte_cdx_unmap_device(struct rte_cdx_device *dev);
+
+/**
+ * Register a CDX driver.
+ *
+ * @param driver
+ * A pointer to a rte_cdx_driver structure describing the driver
+ * to be registered.
+ */
+__rte_internal
+void rte_cdx_register(struct rte_cdx_driver *driver);
+
+/**
+ * Helper for CDX device registration from driver (eth, crypto, raw) instance
+ */
+#define RTE_PMD_REGISTER_CDX(nm, cdx_drv) \
+ RTE_INIT(cdxinitfn_ ##nm) \
+ {\
+ (cdx_drv).driver.name = RTE_STR(nm);\
+ rte_cdx_register(&cdx_drv); \
+ } \
+ RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+/**
+ * Unregister a CDX driver.
+ *
+ * @param driver
+ * A pointer to a rte_cdx_driver structure describing the driver
+ * to be unregistered.
+ */
+__rte_internal
+void rte_cdx_unregister(struct rte_cdx_driver *driver);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _BUS_CDX_DRIVER_H_ */
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
new file mode 100644
index 0000000000..d479daa315
--- /dev/null
+++ b/drivers/bus/cdx/cdx.c
@@ -0,0 +1,520 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+/*
+ * Architecture Overview
+ * =====================
+ * CDX is a Hardware Architecture designed for AMD FPGA devices. It
+ * consists of sophisticated mechanism for interaction between FPGA,
+ * Firmware and the APUs (Application CPUs).
+ *
+ * Firmware resides on RPU (Realtime CPUs) which interacts with
+ * the FPGA program manager and the APUs. The RPU provides memory-mapped
+ * interface (RPU if) which is used to communicate with APUs.
+ *
+ * The diagram below shows an overview of the CDX architecture:
+ *
+ * +--------------------------------------+
+ * | DPDK |
+ * | DPDK CDX drivers |
+ * | | |
+ * | DPDK CDX bus |
+ * | | |
+ * +-----------------------------|--------+
+ * |
+ * +-----------------------------|--------+
+ * | Application CPUs (APU) | |
+ * | | |
+ * | VFIO CDX driver |
+ * | Linux OS | |
+ * | Linux CDX bus |
+ * | | |
+ * +-----------------------------|--------+
+ * |
+ * |
+ * +------------------------| RPU if |----+
+ * | | |
+ * | V |
+ * | Realtime CPUs (RPU) |
+ * | |
+ * +--------------------------------------+
+ * |
+ * +---------------------|----------------+
+ * | FPGA | |
+ * | +-----------------------+ |
+ * | | | | |
+ * | +-------+ +-------+ +-------+ |
+ * | | dev 1 | | dev 2 | | dev 3 | |
+ * | +-------+ +-------+ +-------+ |
+ * +--------------------------------------+
+ *
+ * The RPU firmware extracts the device information from the loaded FPGA
+ * image and implements a mechanism that allows the APU drivers to
+ * enumerate such devices (device personality and resource details) via
+ * a dedicated communication channel.
+ *
+ * VFIO CDX driver provides the CDX device resources like MMIO and interrupts
+ * to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
+ * driver to discover and initialize the CDX devices for user-space
+ * applications.
+ */
+
+#include <string.h>
+#include <dirent.h>
+
+#include <rte_log.h>
+#include <rte_bus.h>
+#include <rte_eal_paging.h>
+#include <rte_errno.h>
+#include <rte_devargs.h>
+#include <rte_malloc.h>
+#include <rte_memcpy.h>
+#include <rte_vfio.h>
+
+#include <eal_filesystem.h>
+
+#include "bus_cdx_driver.h"
+#include "cdx_logs.h"
+#include "private.h"
+
+#define SYSFS_CDX_DEVICES "/sys/bus/cdx/devices"
+#define CDX_BUS_NAME cdx
+#define CDX_DEV_PREFIX "cdx-"
+
+/**
+ * @file
+ * CDX probing using Linux sysfs.
+ */
+
+/* Add a device to CDX bus */
+static void
+cdx_add_device(struct rte_cdx_device *cdx_dev)
+{
+ TAILQ_INSERT_TAIL(&rte_cdx_bus.device_list, cdx_dev, next);
+}
+
+static int
+cdx_get_kernel_driver_by_path(const char *filename, char *driver_name,
+ size_t len)
+{
+ int count;
+ char path[PATH_MAX];
+ char *name;
+
+ if (!filename || !driver_name)
+ return -1;
+
+ count = readlink(filename, path, PATH_MAX);
+ if (count >= PATH_MAX)
+ return -1;
+
+ /* For device does not have a driver */
+ if (count < 0)
+ return 1;
+
+ path[count] = '\0';
+
+ name = strrchr(path, '/');
+ if (name) {
+ strlcpy(driver_name, name + 1, len);
+ return 0;
+ }
+
+ return -1;
+}
+
+int rte_cdx_map_device(struct rte_cdx_device *dev)
+{
+ return cdx_vfio_map_resource(dev);
+}
+
+void rte_cdx_unmap_device(struct rte_cdx_device *dev)
+{
+ cdx_vfio_unmap_resource(dev);
+}
+
+static struct rte_devargs *
+cdx_devargs_lookup(const char *dev_name)
+{
+ struct rte_devargs *devargs;
+
+ RTE_EAL_DEVARGS_FOREACH("cdx", devargs) {
+ if (strcmp(devargs->name, dev_name) == 0)
+ return devargs;
+ }
+ return NULL;
+}
+
+static bool
+cdx_ignore_device(const char *dev_name)
+{
+ struct rte_devargs *devargs = cdx_devargs_lookup(dev_name);
+
+ switch (rte_cdx_bus.bus.conf.scan_mode) {
+ case RTE_BUS_SCAN_ALLOWLIST:
+ if (devargs && devargs->policy == RTE_DEV_ALLOWED)
+ return false;
+ break;
+ case RTE_BUS_SCAN_UNDEFINED:
+ case RTE_BUS_SCAN_BLOCKLIST:
+ if (devargs == NULL || devargs->policy != RTE_DEV_BLOCKED)
+ return false;
+ break;
+ }
+ return true;
+}
+
+/*
+ * Scan one cdx sysfs entry, and fill the devices list from it.
+ * It checks if the CDX device is bound to vfio-cdx driver. In case
+ * the device is vfio bound, it reads the vendor and device id and
+ * stores it for device-driver matching.
+ */
+static int
+cdx_scan_one(const char *dirname, const char *dev_name)
+{
+ char filename[PATH_MAX];
+ struct rte_cdx_device *dev = NULL;
+ char driver[PATH_MAX];
+ unsigned long tmp;
+ char *name = NULL;
+ int ret;
+
+ dev = calloc(1, sizeof(*dev));
+ if (!dev)
+ return -ENOMEM;
+
+ name = calloc(1, RTE_DEV_NAME_MAX_LEN);
+ if (!name) {
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ dev->device.bus = &rte_cdx_bus.bus;
+ memcpy(name, dev_name, RTE_DEV_NAME_MAX_LEN);
+ dev->device.name = name;
+
+ /* parse driver */
+ snprintf(filename, sizeof(filename), "%s/driver", dirname);
+ ret = cdx_get_kernel_driver_by_path(filename, driver, sizeof(driver));
+ if (ret < 0) {
+ CDX_BUS_ERR("Fail to get kernel driver");
+ ret = -1;
+ goto err;
+ }
+
+ /*
+ * Check if device is bound to 'vfio-cdx' driver, so that user-space
+ * can gracefully access the device.
+ */
+ if (ret || strcmp(driver, "vfio-cdx")) {
+ ret = 0;
+ goto err;
+ }
+
+ /* get vendor id */
+ snprintf(filename, sizeof(filename), "%s/vendor", dirname);
+ if (eal_parse_sysfs_value(filename, &tmp) < 0) {
+ ret = -1;
+ goto err;
+ }
+ dev->id.vendor_id = (uint16_t)tmp;
+
+ /* get device id */
+ snprintf(filename, sizeof(filename), "%s/device", dirname);
+ if (eal_parse_sysfs_value(filename, &tmp) < 0) {
+ free(dev);
+ return -1;
+ }
+ dev->id.device_id = (uint16_t)tmp;
+
+ cdx_add_device(dev);
+
+ return 0;
+
+err:
+ if (name)
+ free(name);
+ if (dev)
+ free(dev);
+ return ret;
+}
+
+/*
+ * Scan the content of the CDX bus, and the devices in the devices
+ * list.
+ */
+static int
+cdx_scan(void)
+{
+ struct dirent *e;
+ DIR *dir;
+ char dirname[PATH_MAX];
+
+ dir = opendir(rte_cdx_get_sysfs_path());
+ if (dir == NULL) {
+ CDX_BUS_ERR("%s(): opendir failed: %s", __func__,
+ strerror(errno));
+ return -1;
+ }
+
+ while ((e = readdir(dir)) != NULL) {
+ if (e->d_name[0] == '.')
+ continue;
+
+ if (cdx_ignore_device(e->d_name))
+ continue;
+
+ snprintf(dirname, sizeof(dirname), "%s/%s",
+ rte_cdx_get_sysfs_path(), e->d_name);
+
+ if (cdx_scan_one(dirname, e->d_name) < 0)
+ goto error;
+ }
+ closedir(dir);
+ return 0;
+
+error:
+ closedir(dir);
+ return -1;
+}
+
+const char *
+rte_cdx_get_sysfs_path(void)
+{
+ return SYSFS_CDX_DEVICES;
+}
+
+/* map a particular resource from a file */
+void *
+cdx_map_resource(void *requested_addr, int fd, uint64_t offset, size_t size,
+ int additional_flags)
+{
+ void *mapaddr;
+
+ /* Map the cdx MMIO memory resource of device */
+ mapaddr = rte_mem_map(requested_addr, size,
+ RTE_PROT_READ | RTE_PROT_WRITE,
+ RTE_MAP_SHARED | additional_flags, fd, offset);
+ if (mapaddr == NULL) {
+ CDX_BUS_ERR("%s(): cannot map resource(%d, %p, 0x%zx, 0x%"PRIx64"): %s (%p)",
+ __func__, fd, requested_addr, size, offset,
+ rte_strerror(rte_errno), mapaddr);
+ }
+ CDX_BUS_DEBUG("CDX MMIO memory mapped at %p", mapaddr);
+
+ return mapaddr;
+}
+
+/* unmap a particular resource */
+void
+cdx_unmap_resource(void *requested_addr, size_t size)
+{
+ if (requested_addr == NULL)
+ return;
+
+ /* Unmap the CDX memory resource of device */
+ if (rte_mem_unmap(requested_addr, size)) {
+ CDX_BUS_ERR("%s(): cannot mem unmap(%p, %#zx): %s", __func__,
+ requested_addr, size, rte_strerror(rte_errno));
+ }
+ CDX_BUS_DEBUG("CDX memory unmapped at %p", requested_addr);
+}
+/*
+ * Match the CDX Driver and Device using device id and vendor id.
+ */
+static int
+cdx_match(const struct rte_cdx_driver *cdx_drv,
+ const struct rte_cdx_device *cdx_dev)
+{
+ const struct rte_cdx_id *id_table;
+
+ for (id_table = cdx_drv->id_table; id_table->vendor_id != 0;
+ id_table++) {
+ /* check if device's identifiers match the driver's ones */
+ if (id_table->vendor_id != cdx_dev->id.vendor_id &&
+ id_table->vendor_id != RTE_CDX_ANY_ID)
+ continue;
+ if (id_table->device_id != cdx_dev->id.device_id &&
+ id_table->device_id != RTE_CDX_ANY_ID)
+ continue;
+
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * If vendor id and device id match, call the probe() function of the
+ * driver.
+ */
+static int
+cdx_probe_one_driver(struct rte_cdx_driver *dr,
+ struct rte_cdx_device *dev)
+{
+ const char *dev_name = dev->device.name;
+ bool already_probed;
+ int ret;
+
+ if ((dr == NULL) || (dev == NULL))
+ return -EINVAL;
+
+ /* The device is not blocked; Check if driver supports it */
+ if (!cdx_match(dr, dev))
+ /* Match of device and driver failed */
+ return 1;
+
+ already_probed = rte_dev_is_probed(&dev->device);
+ if (already_probed) {
+ CDX_BUS_INFO("Device %s is already probed", dev->device.name);
+ return -EEXIST;
+ }
+
+ CDX_BUS_DEBUG(" probe device %s using driver: %s", dev_name,
+ dr->driver.name);
+
+ ret = cdx_vfio_map_resource(dev);
+ if (ret != 0) {
+ CDX_BUS_ERR("CDX map device failed: %d", ret);
+ goto error_map_device;
+ }
+
+ /* call the driver probe() function */
+ ret = dr->probe(dr, dev);
+ if (ret) {
+ CDX_BUS_ERR("Probe CDX driver: %s device: %s failed: %d",
+ dr->driver.name, dev_name, ret);
+ goto error_probe;
+ } else {
+ dev->device.driver = &dr->driver;
+ }
+
+ return ret;
+
+error_probe:
+ cdx_vfio_unmap_resource(dev);
+error_map_device:
+ return ret;
+}
+
+/*
+ * If vendor/device ID match, call the probe() function of all
+ * registered driver for the given device. Return < 0 if initialization
+ * failed, return 1 if no driver is found for this device.
+ */
+static int
+cdx_probe_all_drivers(struct rte_cdx_device *dev)
+{
+ struct rte_cdx_driver *dr = NULL;
+ int rc = 0;
+
+ if (dev == NULL)
+ return -EINVAL;
+
+ FOREACH_DRIVER_ON_CDXBUS(dr) {
+ rc = cdx_probe_one_driver(dr, dev);
+ if (rc < 0)
+ /* negative value is an error */
+ return rc;
+ if (rc > 0)
+ /* positive value means driver doesn't support it */
+ continue;
+ return 0;
+ }
+ return 1;
+}
+
+/*
+ * Scan the content of the CDX bus, and call the probe() function for
+ * all registered drivers that have a matching entry in its id_table
+ * for discovered devices.
+ */
+static int
+cdx_probe(void)
+{
+ struct rte_cdx_device *dev = NULL;
+ size_t probed = 0, failed = 0;
+ int ret = 0;
+
+ FOREACH_DEVICE_ON_CDXBUS(dev) {
+ probed++;
+
+ ret = cdx_probe_all_drivers(dev);
+ if (ret < 0) {
+ CDX_BUS_ERR("Requested device %s cannot be used",
+ dev->device.name);
+ rte_errno = errno;
+ failed++;
+ ret = 0;
+ }
+ }
+
+ return (probed && probed == failed) ? -1 : 0;
+}
+
+static int
+cdx_parse(const char *name, void *addr)
+{
+ const char **out = addr;
+ int ret;
+
+ ret = strncmp(name, CDX_DEV_PREFIX, strlen(CDX_DEV_PREFIX));
+
+ if (ret == 0 && addr)
+ *out = name;
+
+ return ret;
+}
+
+/* register a driver */
+void
+rte_cdx_register(struct rte_cdx_driver *driver)
+{
+ TAILQ_INSERT_TAIL(&rte_cdx_bus.driver_list, driver, next);
+ driver->bus = &rte_cdx_bus;
+}
+
+/* unregister a driver */
+void
+rte_cdx_unregister(struct rte_cdx_driver *driver)
+{
+ TAILQ_REMOVE(&rte_cdx_bus.driver_list, driver, next);
+ driver->bus = NULL;
+}
+
+static struct rte_device *
+cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
+ const void *data)
+{
+ const struct rte_cdx_device *cdx_start;
+ struct rte_cdx_device *cdx_dev;
+
+ if (start != NULL) {
+ cdx_start = RTE_DEV_TO_CDX_DEV_CONST(start);
+ cdx_dev = TAILQ_NEXT(cdx_start, next);
+ } else {
+ cdx_dev = TAILQ_FIRST(&rte_cdx_bus.device_list);
+ }
+ while (cdx_dev != NULL) {
+ if (cmp(&cdx_dev->device, data) == 0)
+ return &cdx_dev->device;
+ cdx_dev = TAILQ_NEXT(cdx_dev, next);
+ }
+ return NULL;
+}
+
+struct rte_cdx_bus rte_cdx_bus = {
+ .bus = {
+ .scan = cdx_scan,
+ .probe = cdx_probe,
+ .find_device = cdx_find_device,
+ .parse = cdx_parse,
+ },
+ .device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
+ .driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
+};
+
+RTE_REGISTER_BUS(cdx, rte_cdx_bus.bus);
+RTE_LOG_REGISTER_DEFAULT(cdx_logtype_bus, NOTICE);
diff --git a/drivers/bus/cdx/cdx_logs.h b/drivers/bus/cdx/cdx_logs.h
new file mode 100644
index 0000000000..e5e23d8b94
--- /dev/null
+++ b/drivers/bus/cdx/cdx_logs.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef _CDX_LOGS_H_
+#define _CDX_LOGS_H_
+
+extern int cdx_logtype_bus;
+
+#define CDX_BUS_LOG(level, fmt, args...) \
+ rte_log(RTE_LOG_ ## level, cdx_logtype_bus, "cdx: " fmt "\n", \
+ ##args)
+
+/* Debug logs are with Function names */
+#define CDX_BUS_DEBUG(fmt, args...) \
+ rte_log(RTE_LOG_DEBUG, cdx_logtype_bus, "cdx: %s(): " fmt "\n", \
+ __func__, ##args)
+
+#define CDX_BUS_INFO(fmt, args...) \
+ CDX_BUS_LOG(INFO, fmt, ## args)
+#define CDX_BUS_ERR(fmt, args...) \
+ CDX_BUS_LOG(ERR, fmt, ## args)
+#define CDX_BUS_WARN(fmt, args...) \
+ CDX_BUS_LOG(WARNING, fmt, ## args)
+
+/* DP Logs, toggled out at compile time if level lower than current level */
+#define CDX_BUS_DP_LOG(level, fmt, args...) \
+ RTE_LOG_DP(level, PMD, fmt, ## args)
+
+#define CDX_BUS_DP_DEBUG(fmt, args...) \
+ CDX_BUS_DP_LOG(DEBUG, fmt, ## args)
+#define CDX_BUS_DP_INFO(fmt, args...) \
+ CDX_BUS_DP_LOG(INFO, fmt, ## args)
+#define CDX_BUS_DP_WARN(fmt, args...) \
+ CDX_BUS_DP_LOG(WARNING, fmt, ## args)
+
+#endif /* _CDX_LOGS_H_ */
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
new file mode 100644
index 0000000000..ae11f589b3
--- /dev/null
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -0,0 +1,437 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+/**
+ * @file
+ * CDX probing using Linux VFIO.
+ *
+ * This code tries to determine if the CDX device is bound to VFIO driver,
+ * and initialize it (map MMIO regions, set up interrupts) if that's the case.
+ *
+ */
+
+#include <string.h>
+#include <fcntl.h>
+#include <sys/eventfd.h>
+#include <sys/socket.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <stdbool.h>
+
+#include <rte_log.h>
+#include <rte_eal_paging.h>
+#include <rte_malloc.h>
+#include <rte_vfio.h>
+#include <rte_eal.h>
+#include <rte_bus.h>
+#include <rte_spinlock.h>
+#include <rte_tailq.h>
+
+#include "eal_filesystem.h"
+
+#include "bus_cdx_driver.h"
+#include "cdx_logs.h"
+#include "private.h"
+
+/**
+ * A structure describing a CDX mapping.
+ */
+struct cdx_map {
+ void *addr;
+ char *path;
+ uint64_t offset;
+ uint64_t size;
+};
+
+/**
+ * A structure describing a mapped CDX resource.
+ * For multi-process we need to reproduce all CDX mappings in secondary
+ * processes, so save them in a tailq.
+ */
+struct mapped_cdx_resource {
+ TAILQ_ENTRY(mapped_cdx_resource) next;
+ char name[RTE_DEV_NAME_MAX_LEN]; /**< CDX device name */
+ char path[PATH_MAX];
+ int nb_maps;
+ struct cdx_map maps[CDX_MAX_RESOURCE];
+};
+
+/** mapped cdx device list */
+TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
+
+static struct rte_tailq_elem cdx_vfio_tailq = {
+ .name = "VFIO_CDX_RESOURCE_LIST",
+};
+EAL_REGISTER_TAILQ(cdx_vfio_tailq)
+
+static struct mapped_cdx_resource *
+cdx_vfio_find_and_unmap_resource(struct mapped_cdx_res_list *vfio_res_list,
+ struct rte_cdx_device *dev)
+{
+ struct mapped_cdx_resource *vfio_res = NULL;
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+ int i;
+
+ /* Get vfio_res */
+ TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+ if (strcmp(vfio_res->name, dev_name))
+ continue;
+ break;
+ }
+
+ if (vfio_res == NULL)
+ return vfio_res;
+
+ CDX_BUS_INFO("Releasing CDX mapped resource for %s", dev_name);
+
+ maps = vfio_res->maps;
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ if (maps[i].addr) {
+ CDX_BUS_DEBUG("Calling cdx_unmap_resource for %s at %p",
+ dev_name, maps[i].addr);
+ cdx_unmap_resource(maps[i].addr, maps[i].size);
+ }
+ }
+
+ return vfio_res;
+}
+
+static int
+cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
+{
+ char cdx_addr[PATH_MAX] = {0};
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list;
+
+ vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ vfio_res = cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for cdx device!",
+ cdx_addr);
+ return -1;
+ }
+
+ TAILQ_REMOVE(vfio_res_list, vfio_res, next);
+ rte_free(vfio_res);
+ return 0;
+}
+
+static int
+cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
+{
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list;
+
+ vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ vfio_res = cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for CDX device!",
+ dev->device.name);
+ return -1;
+ }
+
+ return 0;
+}
+
+int
+cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
+{
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return cdx_vfio_unmap_resource_primary(dev);
+ else
+ return cdx_vfio_unmap_resource_secondary(dev);
+}
+
+static int
+cdx_rte_vfio_setup_device(int vfio_dev_fd)
+{
+ /*
+ * Reset the device. If the device is not capable of resetting,
+ * then it updates errno as EINVAL.
+ */
+ if (ioctl(vfio_dev_fd, VFIO_DEVICE_RESET) && errno != EINVAL) {
+ CDX_BUS_ERR("Unable to reset device! Error: %d (%s)", errno,
+ strerror(errno));
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+cdx_vfio_mmap_resource(int vfio_dev_fd, struct mapped_cdx_resource *vfio_res,
+ int index, int additional_flags)
+{
+ struct cdx_map *map = &vfio_res->maps[index];
+ void *vaddr;
+
+ if (map->size == 0) {
+ CDX_BUS_DEBUG("map size is 0, skip region %d", index);
+ return 0;
+ }
+
+ /* reserve the address using an inaccessible mapping */
+ vaddr = mmap(map->addr, map->size, 0, MAP_PRIVATE |
+ MAP_ANONYMOUS | additional_flags, -1, 0);
+ if (vaddr != MAP_FAILED) {
+ void *map_addr = NULL;
+
+ if (map->size) {
+ /* actual map of first part */
+ map_addr = cdx_map_resource(vaddr, vfio_dev_fd,
+ map->offset, map->size,
+ RTE_MAP_FORCE_ADDRESS);
+ }
+
+ if (map_addr == NULL) {
+ munmap(vaddr, map->size);
+ vaddr = MAP_FAILED;
+ CDX_BUS_ERR("Failed to map cdx MMIO region %d", index);
+ return -1;
+ }
+ } else {
+ CDX_BUS_ERR("Failed to create inaccessible mapping for MMIO region %d",
+ index);
+ return -1;
+ }
+
+ map->addr = vaddr;
+ return 0;
+}
+
+/*
+ * region info may contain capability headers, so we need to keep reallocating
+ * the memory until we match allocated memory size with argsz.
+ */
+static int
+cdx_vfio_get_region_info(int vfio_dev_fd, struct vfio_region_info **info,
+ int region)
+{
+ struct vfio_region_info *ri;
+ size_t argsz = sizeof(*ri);
+ int ret;
+
+ ri = malloc(sizeof(*ri));
+ if (ri == NULL) {
+ CDX_BUS_ERR("Cannot allocate memory for VFIO region info");
+ return -1;
+ }
+again:
+ memset(ri, 0, argsz);
+ ri->argsz = argsz;
+ ri->index = region;
+
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_REGION_INFO, ri);
+ if (ret < 0) {
+ free(ri);
+ return ret;
+ }
+ if (ri->argsz != argsz) {
+ struct vfio_region_info *tmp;
+
+ argsz = ri->argsz;
+ tmp = realloc(ri, argsz);
+
+ if (tmp == NULL) {
+ /* realloc failed but the ri is still there */
+ free(ri);
+ CDX_BUS_ERR("Cannot reallocate memory for VFIO region info");
+ return -1;
+ }
+ ri = tmp;
+ goto again;
+ }
+ *info = ri;
+
+ return 0;
+}
+
+static int
+find_max_end_va(const struct rte_memseg_list *msl, void *arg)
+{
+ size_t sz = msl->len;
+ void *end_va = RTE_PTR_ADD(msl->base_va, sz);
+ void **max_va = arg;
+
+ if (*max_va < end_va)
+ *max_va = end_va;
+ return 0;
+}
+
+static void *
+cdx_find_max_end_va(void)
+{
+ void *va = NULL;
+
+ rte_memseg_list_walk(find_max_end_va, &va);
+ return va;
+}
+
+static int
+cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ char cdx_addr[PATH_MAX] = {0};
+ static void *cdx_map_addr;
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+ int vfio_dev_fd, i, ret;
+
+ ret = rte_vfio_setup_device(rte_cdx_get_sysfs_path(), dev_name,
+ &vfio_dev_fd, &device_info);
+ if (ret)
+ return ret;
+
+ /* allocate vfio_res and get region info */
+ vfio_res = rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0);
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("Cannot store VFIO mmap details");
+ goto err_vfio_dev_fd;
+ }
+ memcpy(vfio_res->name, dev_name, RTE_DEV_NAME_MAX_LEN);
+
+ /* get number of registers */
+ vfio_res->nb_maps = device_info.num_regions;
+
+ /* map memory regions */
+ maps = vfio_res->maps;
+
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ struct vfio_region_info *reg = NULL;
+ void *vaddr;
+
+ ret = cdx_vfio_get_region_info(vfio_dev_fd, ®, i);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s cannot get device region info error %i (%s)",
+ dev_name, errno, strerror(errno));
+ goto err_vfio_res;
+ }
+
+ /* skip non-mmappable regions */
+ if ((reg->flags & VFIO_REGION_INFO_FLAG_MMAP) == 0) {
+ free(reg);
+ continue;
+ }
+
+ /* try mapping somewhere close to the end of hugepages */
+ if (cdx_map_addr == NULL)
+ cdx_map_addr = cdx_find_max_end_va();
+
+ vaddr = cdx_map_addr;
+ cdx_map_addr = RTE_PTR_ADD(vaddr, (size_t)reg->size);
+
+ cdx_map_addr = RTE_PTR_ALIGN(cdx_map_addr,
+ sysconf(_SC_PAGE_SIZE));
+
+ maps[i].addr = vaddr;
+ maps[i].offset = reg->offset;
+ maps[i].size = reg->size;
+ maps[i].path = NULL; /* vfio doesn't have per-resource paths */
+
+ ret = cdx_vfio_mmap_resource(vfio_dev_fd, vfio_res, i, 0);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s mapping region %i failed: %s",
+ cdx_addr, i, strerror(errno));
+ free(reg);
+ goto err_vfio_res;
+ }
+
+ dev->mem_resource[i].addr = maps[i].addr;
+ dev->mem_resource[i].len = maps[i].size;
+
+ free(reg);
+ }
+
+ if (cdx_rte_vfio_setup_device(vfio_dev_fd) < 0) {
+ CDX_BUS_ERR("%s setup device failed", dev_name);
+ goto err_vfio_res;
+ }
+
+ TAILQ_INSERT_TAIL(vfio_res_list, vfio_res, next);
+
+ return 0;
+err_vfio_res:
+ cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+ rte_free(vfio_res);
+err_vfio_dev_fd:
+ rte_vfio_release_device(rte_cdx_get_sysfs_path(),
+ dev_name, vfio_dev_fd);
+ return -1;
+}
+
+static int
+cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ char cdx_addr[PATH_MAX] = {0};
+ int vfio_dev_fd;
+ int i, ret;
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+
+ /* if we're in a secondary process, just find our tailq entry */
+ TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+ if (strcmp(vfio_res->name, dev_name))
+ continue;
+ break;
+ }
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for cdx device!",
+ dev_name);
+ return -1;
+ }
+
+ ret = rte_vfio_setup_device(rte_cdx_get_sysfs_path(), dev_name,
+ &vfio_dev_fd, &device_info);
+ if (ret)
+ return ret;
+
+ /* map MMIO regions */
+ maps = vfio_res->maps;
+
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ ret = cdx_vfio_mmap_resource(vfio_dev_fd, vfio_res, i, MAP_FIXED);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s mapping MMIO region %i failed: %s",
+ dev_name, i, strerror(errno));
+ goto err_vfio_dev_fd;
+ }
+
+ dev->mem_resource[i].addr = maps[i].addr;
+ dev->mem_resource[i].len = maps[i].size;
+ }
+
+ return 0;
+err_vfio_dev_fd:
+ rte_vfio_release_device(rte_cdx_get_sysfs_path(),
+ cdx_addr, vfio_dev_fd);
+ return -1;
+}
+
+/*
+ * map the CDX resources of a CDX device in virtual memory (VFIO version).
+ * primary and secondary processes follow almost exactly the same path
+ */
+int
+cdx_vfio_map_resource(struct rte_cdx_device *dev)
+{
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return cdx_vfio_map_resource_primary(dev);
+ else
+ return cdx_vfio_map_resource_secondary(dev);
+}
diff --git a/drivers/bus/cdx/meson.build b/drivers/bus/cdx/meson.build
new file mode 100644
index 0000000000..f2ca104d34
--- /dev/null
+++ b/drivers/bus/cdx/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+
+if not is_linux
+ build = false
+ reason = 'only supported on Linux'
+endif
+
+driver_sdk_headers = files('bus_cdx_driver.h')
+sources = files(
+ 'cdx.c',
+ 'cdx_vfio.c',
+)
diff --git a/drivers/bus/cdx/private.h b/drivers/bus/cdx/private.h
new file mode 100644
index 0000000000..c4ca76dc1b
--- /dev/null
+++ b/drivers/bus/cdx/private.h
@@ -0,0 +1,49 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef _CDX_PRIVATE_H_
+#define _CDX_PRIVATE_H_
+
+#include <stdbool.h>
+#include <stdio.h>
+
+#include "bus_cdx_driver.h"
+
+extern struct rte_cdx_bus rte_cdx_bus;
+
+/**
+ * Map a particular resource from a file.
+ *
+ * @param requested_addr
+ * The starting address for the new mapping range.
+ * @param fd
+ * The file descriptor.
+ * @param offset
+ * The offset for the mapping range.
+ * @param size
+ * The size for the mapping range.
+ * @param additional_flags
+ * The additional rte_mem_map() flags for the mapping range.
+ * @return
+ * - On success, the function returns a pointer to the mapped area.
+ * - On error, NULL is returned.
+ */
+void *cdx_map_resource(void *requested_addr, int fd, uint64_t offset,
+ size_t size, int additional_flags);
+
+/**
+ * Unmap a particular resource.
+ *
+ * @param requested_addr
+ * The address for the unmapping range.
+ * @param size
+ * The size for the unmapping range.
+ */
+void cdx_unmap_resource(void *requested_addr, size_t size);
+
+/* map/unmap VFIO resource */
+int cdx_vfio_map_resource(struct rte_cdx_device *dev);
+int cdx_vfio_unmap_resource(struct rte_cdx_device *dev);
+
+#endif /* _CDX_PRIVATE_H_ */
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
new file mode 100644
index 0000000000..957fcab978
--- /dev/null
+++ b/drivers/bus/cdx/version.map
@@ -0,0 +1,11 @@
+INTERNAL {
+ global:
+
+ rte_cdx_get_sysfs_path;
+ rte_cdx_map_device;
+ rte_cdx_register;
+ rte_cdx_unmap_device;
+ rte_cdx_unregister;
+
+ local: *;
+};
diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
index 6d2520c543..a78b4283bf 100644
--- a/drivers/bus/meson.build
+++ b/drivers/bus/meson.build
@@ -3,6 +3,7 @@
drivers = [
'auxiliary',
+ 'cdx',
'dpaa',
'fslmc',
'ifpga',
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* RE: [PATCH v4 1/4] bus/cdx: introduce cdx bus
2023-05-08 11:18 ` [PATCH v4 1/4] bus/cdx: introduce cdx bus Nipun Gupta
@ 2023-05-09 6:54 ` Xia, Chenbo
2023-05-09 11:09 ` Gupta, Nipun
2023-05-24 11:14 ` Thomas Monjalon
1 sibling, 1 reply; 102+ messages in thread
From: Xia, Chenbo @ 2023-05-09 6:54 UTC (permalink / raw)
To: Nipun Gupta, dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal
> -----Original Message-----
> From: Nipun Gupta <nipun.gupta@amd.com>
> Sent: Monday, May 8, 2023 7:18 PM
> To: dev@dpdk.org; thomas@monjalon.net; david.marchand@redhat.com
> Cc: ferruh.yigit@amd.com; harpreet.anand@amd.com; nikhil.agarwal@amd.com;
> Nipun Gupta <nipun.gupta@amd.com>
> Subject: [PATCH v4 1/4] bus/cdx: introduce cdx bus
>
> CDX bus supports multiple type of devices, which can be
> exposed to user-space via vfio-cdx.
>
> vfio-cdx provides the MMIO IO_MEMORY regions as well as the
> DMA interface for the device (IOMMU).
>
> This support aims to enable the DPDK to support the cdx
> devices in user-space using VFIO interface.
>
> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> ---
> MAINTAINERS | 5 +
> doc/guides/rel_notes/release_23_07.rst | 6 +
> drivers/bus/cdx/bus_cdx_driver.h | 201 ++++++++++
> drivers/bus/cdx/cdx.c | 520 +++++++++++++++++++++++++
> drivers/bus/cdx/cdx_logs.h | 37 ++
> drivers/bus/cdx/cdx_vfio.c | 437 +++++++++++++++++++++
> drivers/bus/cdx/meson.build | 13 +
> drivers/bus/cdx/private.h | 49 +++
> drivers/bus/cdx/version.map | 11 +
> drivers/bus/meson.build | 1 +
> 10 files changed, 1280 insertions(+)
> create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
> create mode 100644 drivers/bus/cdx/cdx.c
> create mode 100644 drivers/bus/cdx/cdx_logs.h
> create mode 100644 drivers/bus/cdx/cdx_vfio.c
> create mode 100644 drivers/bus/cdx/meson.build
> create mode 100644 drivers/bus/cdx/private.h
> create mode 100644 drivers/bus/cdx/version.map
>
...
> --- /dev/null
> +++ b/drivers/bus/cdx/cdx.c
> @@ -0,0 +1,520 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
> + */
> +
> +/*
> + * Architecture Overview
> + * =====================
> + * CDX is a Hardware Architecture designed for AMD FPGA devices. It
> + * consists of sophisticated mechanism for interaction between FPGA,
> + * Firmware and the APUs (Application CPUs).
> + *
> + * Firmware resides on RPU (Realtime CPUs) which interacts with
> + * the FPGA program manager and the APUs. The RPU provides memory-mapped
> + * interface (RPU if) which is used to communicate with APUs.
> + *
> + * The diagram below shows an overview of the CDX architecture:
> + *
> + * +--------------------------------------+
> + * | DPDK |
> + * | DPDK CDX drivers |
> + * | | |
> + * | DPDK CDX bus |
> + * | | |
> + * +-----------------------------|--------+
> + * |
> + * +-----------------------------|--------+
> + * | Application CPUs (APU) | |
> + * | | |
> + * | VFIO CDX driver |
> + * | Linux OS | |
> + * | Linux CDX bus |
> + * | | |
> + * +-----------------------------|--------+
> + * |
> + * |
> + * +------------------------| RPU if |----+
> + * | | |
> + * | V |
> + * | Realtime CPUs (RPU) |
> + * | |
> + * +--------------------------------------+
> + * |
> + * +---------------------|----------------+
> + * | FPGA | |
> + * | +-----------------------+ |
> + * | | | | |
> + * | +-------+ +-------+ +-------+ |
> + * | | dev 1 | | dev 2 | | dev 3 | |
> + * | +-------+ +-------+ +-------+ |
> + * +--------------------------------------+
> + *
> + * The RPU firmware extracts the device information from the loaded FPGA
> + * image and implements a mechanism that allows the APU drivers to
> + * enumerate such devices (device personality and resource details) via
> + * a dedicated communication channel.
What is APU? CPU resources that application uses? Then why cpu resources that DPDK
Uses are not part of it?
I also notice that there are more and more vfio-XXX devices in DPDK now: pci/platform/cdx.
Some vfio related code are very similar. In the future we may need to think about make them
a common lib or something.
Thanks,
Chenbo
> + *
> + * VFIO CDX driver provides the CDX device resources like MMIO and
> interrupts
> + * to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-
> cdx
> + * driver to discover and initialize the CDX devices for user-space
> + * applications.
> + */
> +
^ permalink raw reply [flat|nested] 102+ messages in thread
* RE: [PATCH v4 1/4] bus/cdx: introduce cdx bus
2023-05-09 6:54 ` Xia, Chenbo
@ 2023-05-09 11:09 ` Gupta, Nipun
2023-05-09 11:25 ` Ferruh Yigit
0 siblings, 1 reply; 102+ messages in thread
From: Gupta, Nipun @ 2023-05-09 11:09 UTC (permalink / raw)
To: Xia, Chenbo, dev, thomas, david.marchand
Cc: Yigit, Ferruh, Anand, Harpreet, Agarwal, Nikhil
[AMD Official Use Only - General]
> -----Original Message-----
> From: Xia, Chenbo <chenbo.xia@intel.com>
> Sent: Tuesday, May 9, 2023 12:25 PM
> To: Gupta, Nipun <Nipun.Gupta@amd.com>; dev@dpdk.org;
> thomas@monjalon.net; david.marchand@redhat.com
> Cc: Yigit, Ferruh <Ferruh.Yigit@amd.com>; Anand, Harpreet
> <harpreet.anand@amd.com>; Agarwal, Nikhil <nikhil.agarwal@amd.com>
> Subject: RE: [PATCH v4 1/4] bus/cdx: introduce cdx bus
>
> Caution: This message originated from an External Source. Use proper
> caution when opening attachments, clicking links, or responding.
>
>
> > -----Original Message-----
> > From: Nipun Gupta <nipun.gupta@amd.com>
> > Sent: Monday, May 8, 2023 7:18 PM
> > To: dev@dpdk.org; thomas@monjalon.net; david.marchand@redhat.com
> > Cc: ferruh.yigit@amd.com; harpreet.anand@amd.com;
> nikhil.agarwal@amd.com;
> > Nipun Gupta <nipun.gupta@amd.com>
> > Subject: [PATCH v4 1/4] bus/cdx: introduce cdx bus
> >
> > CDX bus supports multiple type of devices, which can be
> > exposed to user-space via vfio-cdx.
> >
> > vfio-cdx provides the MMIO IO_MEMORY regions as well as the
> > DMA interface for the device (IOMMU).
> >
> > This support aims to enable the DPDK to support the cdx
> > devices in user-space using VFIO interface.
> >
> > Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> > ---
> > MAINTAINERS | 5 +
> > doc/guides/rel_notes/release_23_07.rst | 6 +
> > drivers/bus/cdx/bus_cdx_driver.h | 201 ++++++++++
> > drivers/bus/cdx/cdx.c | 520 +++++++++++++++++++++++++
> > drivers/bus/cdx/cdx_logs.h | 37 ++
> > drivers/bus/cdx/cdx_vfio.c | 437 +++++++++++++++++++++
> > drivers/bus/cdx/meson.build | 13 +
> > drivers/bus/cdx/private.h | 49 +++
> > drivers/bus/cdx/version.map | 11 +
> > drivers/bus/meson.build | 1 +
> > 10 files changed, 1280 insertions(+)
> > create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
> > create mode 100644 drivers/bus/cdx/cdx.c
> > create mode 100644 drivers/bus/cdx/cdx_logs.h
> > create mode 100644 drivers/bus/cdx/cdx_vfio.c
> > create mode 100644 drivers/bus/cdx/meson.build
> > create mode 100644 drivers/bus/cdx/private.h
> > create mode 100644 drivers/bus/cdx/version.map
> >
>
> ...
>
> > --- /dev/null
> > +++ b/drivers/bus/cdx/cdx.c
> > @@ -0,0 +1,520 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
> > + */
> > +
> > +/*
> > + * Architecture Overview
> > + * =====================
> > + * CDX is a Hardware Architecture designed for AMD FPGA devices. It
> > + * consists of sophisticated mechanism for interaction between FPGA,
> > + * Firmware and the APUs (Application CPUs).
> > + *
> > + * Firmware resides on RPU (Realtime CPUs) which interacts with
> > + * the FPGA program manager and the APUs. The RPU provides memory-
> mapped
> > + * interface (RPU if) which is used to communicate with APUs.
> > + *
> > + * The diagram below shows an overview of the CDX architecture:
> > + *
> > + * +--------------------------------------+
> > + * | DPDK |
> > + * | DPDK CDX drivers |
> > + * | | |
> > + * | DPDK CDX bus |
> > + * | | |
> > + * +-----------------------------|--------+
> > + * |
> > + * +-----------------------------|--------+
> > + * | Application CPUs (APU) | |
> > + * | | |
> > + * | VFIO CDX driver |
> > + * | Linux OS | |
> > + * | Linux CDX bus |
> > + * | | |
> > + * +-----------------------------|--------+
> > + * |
> > + * |
> > + * +------------------------| RPU if |----+
> > + * | | |
> > + * | V |
> > + * | Realtime CPUs (RPU) |
> > + * | |
> > + * +--------------------------------------+
> > + * |
> > + * +---------------------|----------------+
> > + * | FPGA | |
> > + * | +-----------------------+ |
> > + * | | | | |
> > + * | +-------+ +-------+ +-------+ |
> > + * | | dev 1 | | dev 2 | | dev 3 | |
> > + * | +-------+ +-------+ +-------+ |
> > + * +--------------------------------------+
> > + *
> > + * The RPU firmware extracts the device information from the loaded
> FPGA
> > + * image and implements a mechanism that allows the APU drivers to
> > + * enumerate such devices (device personality and resource details) via
> > + * a dedicated communication channel.
>
> What is APU? CPU resources that application uses? Then why cpu resources
> that DPDK
> Uses are not part of it?
Hi Chenbo,
APU's are application processor unit and are on-chip CPU's. So in short APUs
are CPUs only on which applications like DPDK are running.
Thanks,
Nipun
^ permalink raw reply [flat|nested] 102+ messages in thread
* Re: [PATCH v4 1/4] bus/cdx: introduce cdx bus
2023-05-09 11:09 ` Gupta, Nipun
@ 2023-05-09 11:25 ` Ferruh Yigit
2023-05-10 1:29 ` Xia, Chenbo
0 siblings, 1 reply; 102+ messages in thread
From: Ferruh Yigit @ 2023-05-09 11:25 UTC (permalink / raw)
To: Gupta, Nipun, Xia, Chenbo, dev, thomas, david.marchand
Cc: Anand, Harpreet, Agarwal, Nikhil
On 5/9/2023 12:09 PM, Gupta, Nipun wrote:
> [AMD Official Use Only - General]
>
>
>
>> -----Original Message-----
>> From: Xia, Chenbo <chenbo.xia@intel.com>
>> Sent: Tuesday, May 9, 2023 12:25 PM
>> To: Gupta, Nipun <Nipun.Gupta@amd.com>; dev@dpdk.org;
>> thomas@monjalon.net; david.marchand@redhat.com
>> Cc: Yigit, Ferruh <Ferruh.Yigit@amd.com>; Anand, Harpreet
>> <harpreet.anand@amd.com>; Agarwal, Nikhil <nikhil.agarwal@amd.com>
>> Subject: RE: [PATCH v4 1/4] bus/cdx: introduce cdx bus
>>
>> Caution: This message originated from an External Source. Use proper
>> caution when opening attachments, clicking links, or responding.
>>
>>
>>> -----Original Message-----
>>> From: Nipun Gupta <nipun.gupta@amd.com>
>>> Sent: Monday, May 8, 2023 7:18 PM
>>> To: dev@dpdk.org; thomas@monjalon.net; david.marchand@redhat.com
>>> Cc: ferruh.yigit@amd.com; harpreet.anand@amd.com;
>> nikhil.agarwal@amd.com;
>>> Nipun Gupta <nipun.gupta@amd.com>
>>> Subject: [PATCH v4 1/4] bus/cdx: introduce cdx bus
>>>
>>> CDX bus supports multiple type of devices, which can be
>>> exposed to user-space via vfio-cdx.
>>>
>>> vfio-cdx provides the MMIO IO_MEMORY regions as well as the
>>> DMA interface for the device (IOMMU).
>>>
>>> This support aims to enable the DPDK to support the cdx
>>> devices in user-space using VFIO interface.
>>>
>>> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
>>> ---
>>> MAINTAINERS | 5 +
>>> doc/guides/rel_notes/release_23_07.rst | 6 +
>>> drivers/bus/cdx/bus_cdx_driver.h | 201 ++++++++++
>>> drivers/bus/cdx/cdx.c | 520 +++++++++++++++++++++++++
>>> drivers/bus/cdx/cdx_logs.h | 37 ++
>>> drivers/bus/cdx/cdx_vfio.c | 437 +++++++++++++++++++++
>>> drivers/bus/cdx/meson.build | 13 +
>>> drivers/bus/cdx/private.h | 49 +++
>>> drivers/bus/cdx/version.map | 11 +
>>> drivers/bus/meson.build | 1 +
>>> 10 files changed, 1280 insertions(+)
>>> create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
>>> create mode 100644 drivers/bus/cdx/cdx.c
>>> create mode 100644 drivers/bus/cdx/cdx_logs.h
>>> create mode 100644 drivers/bus/cdx/cdx_vfio.c
>>> create mode 100644 drivers/bus/cdx/meson.build
>>> create mode 100644 drivers/bus/cdx/private.h
>>> create mode 100644 drivers/bus/cdx/version.map
>>>
>>
>> ...
>>
>>> --- /dev/null
>>> +++ b/drivers/bus/cdx/cdx.c
>>> @@ -0,0 +1,520 @@
>>> +/* SPDX-License-Identifier: BSD-3-Clause
>>> + * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
>>> + */
>>> +
>>> +/*
>>> + * Architecture Overview
>>> + * =====================
>>> + * CDX is a Hardware Architecture designed for AMD FPGA devices. It
>>> + * consists of sophisticated mechanism for interaction between FPGA,
>>> + * Firmware and the APUs (Application CPUs).
>>> + *
>>> + * Firmware resides on RPU (Realtime CPUs) which interacts with
>>> + * the FPGA program manager and the APUs. The RPU provides memory-
>> mapped
>>> + * interface (RPU if) which is used to communicate with APUs.
>>> + *
>>> + * The diagram below shows an overview of the CDX architecture:
>>> + *
>>> + * +--------------------------------------+
>>> + * | DPDK |
>>> + * | DPDK CDX drivers |
>>> + * | | |
>>> + * | DPDK CDX bus |
>>> + * | | |
>>> + * +-----------------------------|--------+
>>> + * |
>>> + * +-----------------------------|--------+
>>> + * | Application CPUs (APU) | |
>>> + * | | |
>>> + * | VFIO CDX driver |
>>> + * | Linux OS | |
>>> + * | Linux CDX bus |
>>> + * | | |
>>> + * +-----------------------------|--------+
>>> + * |
>>> + * |
>>> + * +------------------------| RPU if |----+
>>> + * | | |
>>> + * | V |
>>> + * | Realtime CPUs (RPU) |
>>> + * | |
>>> + * +--------------------------------------+
>>> + * |
>>> + * +---------------------|----------------+
>>> + * | FPGA | |
>>> + * | +-----------------------+ |
>>> + * | | | | |
>>> + * | +-------+ +-------+ +-------+ |
>>> + * | | dev 1 | | dev 2 | | dev 3 | |
>>> + * | +-------+ +-------+ +-------+ |
>>> + * +--------------------------------------+
>>> + *
>>> + * The RPU firmware extracts the device information from the loaded
>> FPGA
>>> + * image and implements a mechanism that allows the APU drivers to
>>> + * enumerate such devices (device personality and resource details) via
>>> + * a dedicated communication channel.
>>
>> What is APU? CPU resources that application uses? Then why cpu resources
>> that DPDK
>> Uses are not part of it?
>
> Hi Chenbo,
>
> APU's are application processor unit and are on-chip CPU's. So in short APUs
> are CPUs only on which applications like DPDK are running.
>
There are two sets of processing unit in the SoC. FW runs on RPU,
everything else runs on APU (Arm cores), including OS, DPDK etc..
^ permalink raw reply [flat|nested] 102+ messages in thread
* RE: [PATCH v4 1/4] bus/cdx: introduce cdx bus
2023-05-09 11:25 ` Ferruh Yigit
@ 2023-05-10 1:29 ` Xia, Chenbo
0 siblings, 0 replies; 102+ messages in thread
From: Xia, Chenbo @ 2023-05-10 1:29 UTC (permalink / raw)
To: Ferruh Yigit, Gupta, Nipun, dev, thomas, david.marchand
Cc: Anand, Harpreet, Agarwal, Nikhil
> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit@amd.com>
> Sent: Tuesday, May 9, 2023 7:26 PM
> To: Gupta, Nipun <Nipun.Gupta@amd.com>; Xia, Chenbo <chenbo.xia@intel.com>;
> dev@dpdk.org; thomas@monjalon.net; david.marchand@redhat.com
> Cc: Anand, Harpreet <harpreet.anand@amd.com>; Agarwal, Nikhil
> <nikhil.agarwal@amd.com>
> Subject: Re: [PATCH v4 1/4] bus/cdx: introduce cdx bus
>
> On 5/9/2023 12:09 PM, Gupta, Nipun wrote:
> > [AMD Official Use Only - General]
> >
> >
> >
> >> -----Original Message-----
> >> From: Xia, Chenbo <chenbo.xia@intel.com>
> >> Sent: Tuesday, May 9, 2023 12:25 PM
> >> To: Gupta, Nipun <Nipun.Gupta@amd.com>; dev@dpdk.org;
> >> thomas@monjalon.net; david.marchand@redhat.com
> >> Cc: Yigit, Ferruh <Ferruh.Yigit@amd.com>; Anand, Harpreet
> >> <harpreet.anand@amd.com>; Agarwal, Nikhil <nikhil.agarwal@amd.com>
> >> Subject: RE: [PATCH v4 1/4] bus/cdx: introduce cdx bus
> >>
> >> Caution: This message originated from an External Source. Use proper
> >> caution when opening attachments, clicking links, or responding.
> >>
> >>
> >>> -----Original Message-----
> >>> From: Nipun Gupta <nipun.gupta@amd.com>
> >>> Sent: Monday, May 8, 2023 7:18 PM
> >>> To: dev@dpdk.org; thomas@monjalon.net; david.marchand@redhat.com
> >>> Cc: ferruh.yigit@amd.com; harpreet.anand@amd.com;
> >> nikhil.agarwal@amd.com;
> >>> Nipun Gupta <nipun.gupta@amd.com>
> >>> Subject: [PATCH v4 1/4] bus/cdx: introduce cdx bus
> >>>
> >>> CDX bus supports multiple type of devices, which can be
> >>> exposed to user-space via vfio-cdx.
> >>>
> >>> vfio-cdx provides the MMIO IO_MEMORY regions as well as the
> >>> DMA interface for the device (IOMMU).
> >>>
> >>> This support aims to enable the DPDK to support the cdx
> >>> devices in user-space using VFIO interface.
> >>>
> >>> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> >>> ---
> >>> MAINTAINERS | 5 +
> >>> doc/guides/rel_notes/release_23_07.rst | 6 +
> >>> drivers/bus/cdx/bus_cdx_driver.h | 201 ++++++++++
> >>> drivers/bus/cdx/cdx.c | 520
> +++++++++++++++++++++++++
> >>> drivers/bus/cdx/cdx_logs.h | 37 ++
> >>> drivers/bus/cdx/cdx_vfio.c | 437 +++++++++++++++++++++
> >>> drivers/bus/cdx/meson.build | 13 +
> >>> drivers/bus/cdx/private.h | 49 +++
> >>> drivers/bus/cdx/version.map | 11 +
> >>> drivers/bus/meson.build | 1 +
> >>> 10 files changed, 1280 insertions(+)
> >>> create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
> >>> create mode 100644 drivers/bus/cdx/cdx.c
> >>> create mode 100644 drivers/bus/cdx/cdx_logs.h
> >>> create mode 100644 drivers/bus/cdx/cdx_vfio.c
> >>> create mode 100644 drivers/bus/cdx/meson.build
> >>> create mode 100644 drivers/bus/cdx/private.h
> >>> create mode 100644 drivers/bus/cdx/version.map
> >>>
> >>
> >> ...
> >>
> >>> --- /dev/null
> >>> +++ b/drivers/bus/cdx/cdx.c
> >>> @@ -0,0 +1,520 @@
> >>> +/* SPDX-License-Identifier: BSD-3-Clause
> >>> + * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
> >>> + */
> >>> +
> >>> +/*
> >>> + * Architecture Overview
> >>> + * =====================
> >>> + * CDX is a Hardware Architecture designed for AMD FPGA devices. It
> >>> + * consists of sophisticated mechanism for interaction between FPGA,
> >>> + * Firmware and the APUs (Application CPUs).
> >>> + *
> >>> + * Firmware resides on RPU (Realtime CPUs) which interacts with
> >>> + * the FPGA program manager and the APUs. The RPU provides memory-
> >> mapped
> >>> + * interface (RPU if) which is used to communicate with APUs.
> >>> + *
> >>> + * The diagram below shows an overview of the CDX architecture:
> >>> + *
> >>> + * +--------------------------------------+
> >>> + * | DPDK |
> >>> + * | DPDK CDX drivers |
> >>> + * | | |
> >>> + * | DPDK CDX bus |
> >>> + * | | |
> >>> + * +-----------------------------|--------+
> >>> + * |
> >>> + * +-----------------------------|--------+
> >>> + * | Application CPUs (APU) | |
> >>> + * | | |
> >>> + * | VFIO CDX driver |
> >>> + * | Linux OS | |
> >>> + * | Linux CDX bus |
> >>> + * | | |
> >>> + * +-----------------------------|--------+
> >>> + * |
> >>> + * |
> >>> + * +------------------------| RPU if |----+
> >>> + * | | |
> >>> + * | V |
> >>> + * | Realtime CPUs (RPU) |
> >>> + * | |
> >>> + * +--------------------------------------+
> >>> + * |
> >>> + * +---------------------|----------------+
> >>> + * | FPGA | |
> >>> + * | +-----------------------+ |
> >>> + * | | | | |
> >>> + * | +-------+ +-------+ +-------+ |
> >>> + * | | dev 1 | | dev 2 | | dev 3 | |
> >>> + * | +-------+ +-------+ +-------+ |
> >>> + * +--------------------------------------+
> >>> + *
> >>> + * The RPU firmware extracts the device information from the loaded
> >> FPGA
> >>> + * image and implements a mechanism that allows the APU drivers to
> >>> + * enumerate such devices (device personality and resource details)
> via
> >>> + * a dedicated communication channel.
> >>
> >> What is APU? CPU resources that application uses? Then why cpu
> resources
> >> that DPDK
> >> Uses are not part of it?
> >
> > Hi Chenbo,
> >
> > APU's are application processor unit and are on-chip CPU's. So in short
> APUs
> > are CPUs only on which applications like DPDK are running.
> >
>
> There are two sets of processing unit in the SoC. FW runs on RPU,
> everything else runs on APU (Arm cores), including OS, DPDK etc..
Cool, thanks Nipun and Ferruh for the explanation!
^ permalink raw reply [flat|nested] 102+ messages in thread
* Re: [PATCH v4 1/4] bus/cdx: introduce cdx bus
2023-05-08 11:18 ` [PATCH v4 1/4] bus/cdx: introduce cdx bus Nipun Gupta
2023-05-09 6:54 ` Xia, Chenbo
@ 2023-05-24 11:14 ` Thomas Monjalon
2023-05-24 17:04 ` Gupta, Nipun
1 sibling, 1 reply; 102+ messages in thread
From: Thomas Monjalon @ 2023-05-24 11:14 UTC (permalink / raw)
To: Nipun Gupta
Cc: dev, david.marchand, ferruh.yigit, harpreet.anand, nikhil.agarwal
Hello,
If I understand well, it is very specific to AMD devices.
So I suggest adding "AMD" in title and descriptions.
08/05/2023 13:18, Nipun Gupta:
> CDX bus supports multiple type of devices, which can be
> exposed to user-space via vfio-cdx.
>
> vfio-cdx provides the MMIO IO_MEMORY regions as well as the
> DMA interface for the device (IOMMU).
>
> This support aims to enable the DPDK to support the cdx
> devices in user-space using VFIO interface.
>
> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
[...]
> +CDX bus driver
Can we name it "AMD CDX bus"?
> +M: Nipun Gupta <nipun.gupta@amd.com>
> +M: Nikhil Agarwal <nikhil.agarwal@amd.com>
> +F: drivers/bus/cdx/
[...]
> +* **Added CDX bus support.**
Here as well and in other places, would be more precise to say "AMD CDX".
> +
> + CDX bus driver has been added to support AMD CDX bus, which operates
> + on FPGA based CDX devices. The CDX devices are memory mapped on system
> + bus for embedded CPUs.
[...]
> +#ifndef _BUS_CDX_DRIVER_H_
> +#define _BUS_CDX_DRIVER_H_
In general I prefer not adding underscores,
as it is not required, and not really private.
[...]
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <limits.h>
> +#include <errno.h>
> +#include <stdint.h>
> +#include <inttypes.h>
I would bet some includes are not needed for this header file.
> +
> +#include <bus_driver.h>
> +#include <dev_driver.h>
> +#include <rte_debug.h>
> +#include <rte_interrupts.h>
> +#include <rte_dev.h>
> +#include <rte_bus.h>
[...]
^ permalink raw reply [flat|nested] 102+ messages in thread
* RE: [PATCH v4 1/4] bus/cdx: introduce cdx bus
2023-05-24 11:14 ` Thomas Monjalon
@ 2023-05-24 17:04 ` Gupta, Nipun
0 siblings, 0 replies; 102+ messages in thread
From: Gupta, Nipun @ 2023-05-24 17:04 UTC (permalink / raw)
To: Thomas Monjalon
Cc: dev, david.marchand, Yigit, Ferruh, Anand, Harpreet, Agarwal, Nikhil
[AMD Official Use Only - General]
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, May 24, 2023 4:45 PM
> To: Gupta, Nipun <Nipun.Gupta@amd.com>
> Cc: dev@dpdk.org; david.marchand@redhat.com; Yigit, Ferruh
> <Ferruh.Yigit@amd.com>; Anand, Harpreet <harpreet.anand@amd.com>;
> Agarwal, Nikhil <nikhil.agarwal@amd.com>
> Subject: Re: [PATCH v4 1/4] bus/cdx: introduce cdx bus
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> Hello,
>
> If I understand well, it is very specific to AMD devices.
> So I suggest adding "AMD" in title and descriptions.
>
> 08/05/2023 13:18, Nipun Gupta:
> > CDX bus supports multiple type of devices, which can be
> > exposed to user-space via vfio-cdx.
> >
> > vfio-cdx provides the MMIO IO_MEMORY regions as well as the
> > DMA interface for the device (IOMMU).
> >
> > This support aims to enable the DPDK to support the cdx
> > devices in user-space using VFIO interface.
> >
> > Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> [...]
> > +CDX bus driver
>
> Can we name it "AMD CDX bus"?
Sure will update.
>
> > +M: Nipun Gupta <nipun.gupta@amd.com>
> > +M: Nikhil Agarwal <nikhil.agarwal@amd.com>
> > +F: drivers/bus/cdx/
>
> [...]
> > +* **Added CDX bus support.**
>
> Here as well and in other places, would be more precise to say "AMD CDX".
>
> > +
> > + CDX bus driver has been added to support AMD CDX bus, which operates
> > + on FPGA based CDX devices. The CDX devices are memory mapped on
> system
> > + bus for embedded CPUs.
> [...]
> > +#ifndef _BUS_CDX_DRIVER_H_
> > +#define _BUS_CDX_DRIVER_H_
>
> In general I prefer not adding underscores,
> as it is not required, and not really private.
Okay.. I will update for all the header files.
>
> [...]
> > +#include <stdio.h>
> > +#include <stdlib.h>
> > +#include <limits.h>
> > +#include <errno.h>
> > +#include <stdint.h>
> > +#include <inttypes.h>
>
> I would bet some includes are not needed for this header file.
Will cut them short.
Thanks,
Nipun
>
> > +
> > +#include <bus_driver.h>
> > +#include <dev_driver.h>
> > +#include <rte_debug.h>
> > +#include <rte_interrupts.h>
> > +#include <rte_dev.h>
> > +#include <rte_bus.h>
> [...]
>
>
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v4 2/4] bus/cdx: add DMA map and unmap support
2023-05-08 11:18 ` [PATCH v4 0/4] Support AMD CDX bus, for FPGA based CDX devices. The CDX Nipun Gupta
2023-05-08 11:18 ` [PATCH v4 1/4] bus/cdx: introduce cdx bus Nipun Gupta
@ 2023-05-08 11:18 ` Nipun Gupta
2023-05-08 11:18 ` [PATCH v4 3/4] bus/cdx: add support for MSI Nipun Gupta
` (2 subsequent siblings)
4 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-05-08 11:18 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
CDX bus can use VFIO interface for mapping and unmapping
of DMA addresses in the IOMMU. This change adds the callback
support for map and unmap APIs as well as fetching the IOMMU
class.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
drivers/bus/cdx/cdx.c | 40 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index d479daa315..8cc273336e 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -505,12 +505,52 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
return NULL;
}
+static int
+cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+ struct rte_cdx_device *cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+
+ if (!cdx_dev) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ return rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ (uintptr_t)addr, iova, len);
+}
+
+static int
+cdx_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+ struct rte_cdx_device *cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+
+ if (!cdx_dev) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ return rte_vfio_container_dma_unmap(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ (uintptr_t)addr, iova, len);
+}
+
+static enum rte_iova_mode
+cdx_get_iommu_class(void)
+{
+ if (TAILQ_EMPTY(&rte_cdx_bus.device_list))
+ return RTE_IOVA_DC;
+
+ return RTE_IOVA_VA;
+}
+
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
.scan = cdx_scan,
.probe = cdx_probe,
.find_device = cdx_find_device,
.parse = cdx_parse,
+ .dma_map = cdx_dma_map,
+ .dma_unmap = cdx_dma_unmap,
+ .get_iommu_class = cdx_get_iommu_class,
},
.device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
.driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v4 3/4] bus/cdx: add support for MSI
2023-05-08 11:18 ` [PATCH v4 0/4] Support AMD CDX bus, for FPGA based CDX devices. The CDX Nipun Gupta
2023-05-08 11:18 ` [PATCH v4 1/4] bus/cdx: introduce cdx bus Nipun Gupta
2023-05-08 11:18 ` [PATCH v4 2/4] bus/cdx: add DMA map and unmap support Nipun Gupta
@ 2023-05-08 11:18 ` Nipun Gupta
2023-05-24 11:06 ` Thomas Monjalon
2023-05-08 11:18 ` [PATCH v4 4/4] bus/cdx: support plug unplug and dev iterator Nipun Gupta
2023-05-12 10:25 ` [PATCH v4 0/4] Support AMD CDX bus, for FPGA based CDX devices. The CDX Ferruh Yigit
4 siblings, 1 reply; 102+ messages in thread
From: Nipun Gupta @ 2023-05-08 11:18 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
MSI's are exposed to the devices using VFIO (vfio-cdx). This
patch uses the same to add support for MSI for the devices on
the cdx bus.
A couple of API's have been introduced in the EAL interrupt
framework:
- rte_intr_irq_count_set: This API is used to set the total
interrupts on the interrupt handle. This would be provided
by VFIO (irq.count) for VFIO enabled devices.
- rte_intr_irq_count_get: This API returns the total number
interrupts which were set.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
drivers/bus/cdx/bus_cdx_driver.h | 25 ++++
drivers/bus/cdx/cdx.c | 11 ++
drivers/bus/cdx/cdx_vfio.c | 182 ++++++++++++++++++++++++-
drivers/bus/cdx/version.map | 2 +
lib/eal/common/eal_common_interrupts.c | 21 +++
lib/eal/common/eal_interrupts.h | 1 +
lib/eal/include/rte_interrupts.h | 32 +++++
lib/eal/version.map | 2 +
8 files changed, 274 insertions(+), 2 deletions(-)
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index 7edcb019eb..fdeaf46664 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -72,6 +72,7 @@ struct rte_cdx_device {
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
/**< CDX Memory Resource */
+ struct rte_intr_handle *intr_handle; /**< Interrupt handle */
};
/**
@@ -173,6 +174,30 @@ void rte_cdx_unmap_device(struct rte_cdx_device *dev);
__rte_internal
void rte_cdx_register(struct rte_cdx_driver *driver);
+/**
+ * Enables VFIO Interrupts for CDX bus devices.
+ *
+ * @param intr_handle
+ * Pointer to the interrupt handle.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle);
+
+/**
+ * Disable VFIO Interrupts for CDX bus devices.
+ *
+ * @param intr_handle
+ * Pointer to the interrupt handle.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle);
+
/**
* Helper for CDX device registration from driver (eth, crypto, raw) instance
*/
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index 8cc273336e..6c9ceaaf7f 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -204,6 +204,15 @@ cdx_scan_one(const char *dirname, const char *dev_name)
goto err;
}
+ /* Allocate interrupt instance for cdx device */
+ dev->intr_handle =
+ rte_intr_instance_alloc(RTE_INTR_INSTANCE_F_PRIVATE);
+ if (dev->intr_handle == NULL) {
+ CDX_BUS_ERR("Failed to create interrupt instance for %s\n",
+ dev->device.name);
+ return -ENOMEM;
+ }
+
/*
* Check if device is bound to 'vfio-cdx' driver, so that user-space
* can gracefully access the device.
@@ -394,6 +403,8 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr,
return ret;
error_probe:
+ rte_intr_instance_free(dev->intr_handle);
+ dev->intr_handle = NULL;
cdx_vfio_unmap_resource(dev);
error_map_device:
return ret;
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index ae11f589b3..1422b98503 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -60,6 +60,10 @@ struct mapped_cdx_resource {
/** mapped cdx device list */
TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
+/* irq set buffer length for MSI interrupts */
+#define MSI_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \
+ sizeof(int) * (RTE_MAX_RXTX_INTR_VEC_ID + 1))
+
static struct rte_tailq_elem cdx_vfio_tailq = {
.name = "VFIO_CDX_RESOURCE_LIST",
};
@@ -104,6 +108,27 @@ cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
char cdx_addr[PATH_MAX] = {0};
struct mapped_cdx_resource *vfio_res = NULL;
struct mapped_cdx_res_list *vfio_res_list;
+ int ret, vfio_dev_fd;
+
+ if (rte_intr_fd_get(dev->intr_handle) < 0)
+ return -1;
+
+ if (close(rte_intr_fd_get(dev->intr_handle)) < 0) {
+ CDX_BUS_ERR("Error when closing eventfd file descriptor for %s",
+ dev->device.name);
+ return -1;
+ }
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ ret = rte_vfio_release_device(rte_cdx_get_sysfs_path(), dev->device.name,
+ vfio_dev_fd);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot release VFIO device");
+ return ret;
+ }
vfio_res_list =
RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
@@ -126,6 +151,18 @@ cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
{
struct mapped_cdx_resource *vfio_res = NULL;
struct mapped_cdx_res_list *vfio_res_list;
+ int ret, vfio_dev_fd;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ ret = rte_vfio_release_device(rte_cdx_get_sysfs_path(), dev->device.name,
+ vfio_dev_fd);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot release VFIO device");
+ return ret;
+ }
vfio_res_list =
RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
@@ -150,9 +187,80 @@ cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
return cdx_vfio_unmap_resource_secondary(dev);
}
+/* set up interrupt support (but not enable interrupts) */
static int
-cdx_rte_vfio_setup_device(int vfio_dev_fd)
+cdx_vfio_setup_interrupts(struct rte_cdx_device *dev, int vfio_dev_fd,
+ int num_irqs)
{
+ int i, ret;
+
+ if (num_irqs == 0)
+ return 0;
+
+ /* start from MSI interrupt type */
+ for (i = 0; i < num_irqs; i++) {
+ struct vfio_irq_info irq = { .argsz = sizeof(irq) };
+ int fd = -1;
+
+ irq.index = i;
+
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_IRQ_INFO, &irq);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot get VFIO IRQ info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /* if this vector cannot be used with eventfd, fail if we explicitly
+ * specified interrupt type, otherwise continue
+ */
+ if ((irq.flags & VFIO_IRQ_INFO_EVENTFD) == 0)
+ continue;
+
+ if (rte_intr_irq_count_set(dev->intr_handle, irq.count))
+ return -1;
+
+ /* Reallocate the efds and elist fields of intr_handle based
+ * on CDX device MSI size.
+ */
+ if ((uint32_t)rte_intr_nb_intr_get(dev->intr_handle) < irq.count &&
+ rte_intr_event_list_update(dev->intr_handle, irq.count))
+ return -1;
+
+ /* set up an eventfd for interrupts */
+ fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
+ if (fd < 0) {
+ CDX_BUS_ERR("Cannot set up eventfd, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ if (rte_intr_fd_set(dev->intr_handle, fd))
+ return -1;
+
+ /* DPDK CDX bus currently supports only MSI-X */
+ if (rte_intr_type_set(dev->intr_handle, RTE_INTR_HANDLE_VFIO_MSIX))
+ return -1;
+
+ if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
+ return -1;
+
+ return 0;
+ }
+
+ /* if we're here, we haven't found a suitable interrupt vector */
+ return -1;
+}
+
+static int
+cdx_vfio_setup_device(struct rte_cdx_device *dev, int vfio_dev_fd,
+ int num_irqs)
+{
+ if (cdx_vfio_setup_interrupts(dev, vfio_dev_fd, num_irqs) != 0) {
+ CDX_BUS_ERR("Error setting up interrupts!");
+ return -1;
+ }
+
/*
* Reset the device. If the device is not capable of resetting,
* then it updates errno as EINVAL.
@@ -288,6 +396,9 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
struct cdx_map *maps;
int vfio_dev_fd, i, ret;
+ if (rte_intr_fd_set(dev->intr_handle, -1))
+ return -1;
+
ret = rte_vfio_setup_device(rte_cdx_get_sysfs_path(), dev_name,
&vfio_dev_fd, &device_info);
if (ret)
@@ -353,7 +464,7 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
free(reg);
}
- if (cdx_rte_vfio_setup_device(vfio_dev_fd) < 0) {
+ if (cdx_vfio_setup_device(dev, vfio_dev_fd, device_info.num_irqs) < 0) {
CDX_BUS_ERR("%s setup device failed", dev_name);
goto err_vfio_res;
}
@@ -383,6 +494,9 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
const char *dev_name = dev->device.name;
struct cdx_map *maps;
+ if (rte_intr_fd_set(dev->intr_handle, -1))
+ return -1;
+
/* if we're in a secondary process, just find our tailq entry */
TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
if (strcmp(vfio_res->name, dev_name))
@@ -416,6 +530,10 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
dev->mem_resource[i].len = maps[i].size;
}
+ /* we need save vfio_dev_fd, so it can be used during release */
+ if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
+ goto err_vfio_dev_fd;
+
return 0;
err_vfio_dev_fd:
rte_vfio_release_device(rte_cdx_get_sysfs_path(),
@@ -435,3 +553,63 @@ cdx_vfio_map_resource(struct rte_cdx_device *dev)
else
return cdx_vfio_map_resource_secondary(dev);
}
+
+int
+rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle)
+{
+ char irq_set_buf[MSI_IRQ_SET_BUF_LEN];
+ struct vfio_irq_set *irq_set;
+ int *fd_ptr, vfio_dev_fd, i;
+ int ret;
+
+ irq_set = (struct vfio_irq_set *) irq_set_buf;
+ irq_set->count = rte_intr_irq_count_get(intr_handle);
+ irq_set->argsz = sizeof(struct vfio_irq_set) +
+ (sizeof(int) * irq_set->count);
+
+ irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER;
+ irq_set->index = 0;
+ irq_set->start = 0;
+ fd_ptr = (int *) &irq_set->data;
+
+ for (i = 0; i < rte_intr_nb_efd_get(intr_handle); i++)
+ fd_ptr[i] = rte_intr_efds_index_get(intr_handle, i);
+
+ vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
+
+ if (ret) {
+ CDX_BUS_ERR("Error enabling MSI interrupts for fd %d",
+ rte_intr_fd_get(intr_handle));
+ return -1;
+ }
+
+ return 0;
+}
+
+/* disable MSI interrupts */
+int
+rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle)
+{
+ struct vfio_irq_set *irq_set;
+ char irq_set_buf[MSI_IRQ_SET_BUF_LEN];
+ int len, ret, vfio_dev_fd;
+
+ len = sizeof(struct vfio_irq_set);
+
+ irq_set = (struct vfio_irq_set *) irq_set_buf;
+ irq_set->argsz = len;
+ irq_set->count = 0;
+ irq_set->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_TRIGGER;
+ irq_set->index = 0;
+ irq_set->start = 0;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
+
+ if (ret)
+ CDX_BUS_ERR("Error disabling MSI interrupts for fd %d",
+ rte_intr_fd_get(intr_handle));
+
+ return ret;
+}
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
index 957fcab978..2f3d484ebd 100644
--- a/drivers/bus/cdx/version.map
+++ b/drivers/bus/cdx/version.map
@@ -6,6 +6,8 @@ INTERNAL {
rte_cdx_register;
rte_cdx_unmap_device;
rte_cdx_unregister;
+ rte_cdx_vfio_intr_disable;
+ rte_cdx_vfio_intr_enable;
local: *;
};
diff --git a/lib/eal/common/eal_common_interrupts.c b/lib/eal/common/eal_common_interrupts.c
index 97b64fed58..a0167d9ad4 100644
--- a/lib/eal/common/eal_common_interrupts.c
+++ b/lib/eal/common/eal_common_interrupts.c
@@ -398,6 +398,27 @@ int rte_intr_elist_index_set(struct rte_intr_handle *intr_handle,
return -rte_errno;
}
+int rte_intr_irq_count_set(struct rte_intr_handle *intr_handle,
+ int irq_count)
+{
+ CHECK_VALID_INTR_HANDLE(intr_handle);
+
+ intr_handle->irq_count = irq_count;
+
+ return 0;
+fail:
+ return -rte_errno;
+}
+
+int rte_intr_irq_count_get(const struct rte_intr_handle *intr_handle)
+{
+ CHECK_VALID_INTR_HANDLE(intr_handle);
+
+ return intr_handle->irq_count;
+fail:
+ return -rte_errno;
+}
+
int rte_intr_vec_list_alloc(struct rte_intr_handle *intr_handle,
const char *name, int size)
{
diff --git a/lib/eal/common/eal_interrupts.h b/lib/eal/common/eal_interrupts.h
index 482781b862..237f471a76 100644
--- a/lib/eal/common/eal_interrupts.h
+++ b/lib/eal/common/eal_interrupts.h
@@ -16,6 +16,7 @@ struct rte_intr_handle {
};
uint32_t alloc_flags; /**< flags passed at allocation */
enum rte_intr_handle_type type; /**< handle type */
+ uint32_t irq_count; /**< IRQ count provided via VFIO */
uint32_t max_intr; /**< max interrupt requested */
uint32_t nb_efd; /**< number of available efd(event fd) */
uint8_t efd_counter_size; /**< size of efd counter, used for vdev */
diff --git a/lib/eal/include/rte_interrupts.h b/lib/eal/include/rte_interrupts.h
index 487e3c8875..bc477a483f 100644
--- a/lib/eal/include/rte_interrupts.h
+++ b/lib/eal/include/rte_interrupts.h
@@ -506,6 +506,38 @@ __rte_internal
int
rte_intr_max_intr_get(const struct rte_intr_handle *intr_handle);
+/**
+ * @internal
+ * Set the irq count field of interrupt handle with user
+ * provided irq count value.
+ *
+ * @param intr_handle
+ * pointer to the interrupt handle.
+ * @param irq_count
+ * IRQ count
+ *
+ * @return
+ * - On success, zero.
+ * - On failure, a negative value and rte_errno is set.
+ */
+__rte_internal
+int
+rte_intr_irq_count_set(struct rte_intr_handle *intr_handle, int irq_count);
+
+/**
+ * @internal
+ * Returns the irq count field of the given interrupt handle instance.
+ *
+ * @param intr_handle
+ * pointer to the interrupt handle.
+ *
+ * @return
+ * - On success, ir count.
+ * - On failure, a negative value and rte_errno is set.
+ */
+__rte_internal
+int rte_intr_irq_count_get(const struct rte_intr_handle *intr_handle);
+
/**
* @internal
* Set the number of event fd field of interrupt handle
diff --git a/lib/eal/version.map b/lib/eal/version.map
index 6d6978f108..14bf7ade77 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -458,6 +458,8 @@ INTERNAL {
rte_intr_instance_dup;
rte_intr_instance_windows_handle_get;
rte_intr_instance_windows_handle_set;
+ rte_intr_irq_count_get;
+ rte_intr_irq_count_set;
rte_intr_max_intr_get;
rte_intr_max_intr_set;
rte_intr_nb_efd_get;
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v4 3/4] bus/cdx: add support for MSI
2023-05-08 11:18 ` [PATCH v4 3/4] bus/cdx: add support for MSI Nipun Gupta
@ 2023-05-24 11:06 ` Thomas Monjalon
2023-05-24 17:06 ` Gupta, Nipun
0 siblings, 1 reply; 102+ messages in thread
From: Thomas Monjalon @ 2023-05-24 11:06 UTC (permalink / raw)
To: Nipun Gupta
Cc: dev, david.marchand, ferruh.yigit, harpreet.anand, nikhil.agarwal
08/05/2023 13:18, Nipun Gupta:
> MSI's are exposed to the devices using VFIO (vfio-cdx). This
> patch uses the same to add support for MSI for the devices on
> the cdx bus.
>
> A couple of API's have been introduced in the EAL interrupt
> framework:
> - rte_intr_irq_count_set: This API is used to set the total
> interrupts on the interrupt handle. This would be provided
> by VFIO (irq.count) for VFIO enabled devices.
> - rte_intr_irq_count_get: This API returns the total number
> interrupts which were set.
[...]
> --- a/lib/eal/common/eal_interrupts.h
> +++ b/lib/eal/common/eal_interrupts.h
> @@ -16,6 +16,7 @@ struct rte_intr_handle {
> };
> uint32_t alloc_flags; /**< flags passed at allocation */
> enum rte_intr_handle_type type; /**< handle type */
> + uint32_t irq_count; /**< IRQ count provided via VFIO */
Why only via VFIO?
[...]
> +/**
> + * @internal
> + * Set the irq count field of interrupt handle with user
> + * provided irq count value.
> + *
> + * @param intr_handle
> + * pointer to the interrupt handle.
> + * @param irq_count
> + * IRQ count
Please write IRQ all uppercase consistently.
Same for CDX.
> + rte_intr_irq_count_get;
> + rte_intr_irq_count_set;
Adding a new API in EAL deserves a separate commit with a different audience.
It looks like it has been hidden from EAL reviewers eyes so far.
^ permalink raw reply [flat|nested] 102+ messages in thread* RE: [PATCH v4 3/4] bus/cdx: add support for MSI
2023-05-24 11:06 ` Thomas Monjalon
@ 2023-05-24 17:06 ` Gupta, Nipun
0 siblings, 0 replies; 102+ messages in thread
From: Gupta, Nipun @ 2023-05-24 17:06 UTC (permalink / raw)
To: Thomas Monjalon
Cc: dev, david.marchand, Yigit, Ferruh, Anand, Harpreet, Agarwal, Nikhil
[AMD Official Use Only - General]
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, May 24, 2023 4:37 PM
> To: Gupta, Nipun <Nipun.Gupta@amd.com>
> Cc: dev@dpdk.org; david.marchand@redhat.com; Yigit, Ferruh
> <Ferruh.Yigit@amd.com>; Anand, Harpreet <harpreet.anand@amd.com>;
> Agarwal, Nikhil <nikhil.agarwal@amd.com>
> Subject: Re: [PATCH v4 3/4] bus/cdx: add support for MSI
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> 08/05/2023 13:18, Nipun Gupta:
> > MSI's are exposed to the devices using VFIO (vfio-cdx). This
> > patch uses the same to add support for MSI for the devices on
> > the cdx bus.
> >
> > A couple of API's have been introduced in the EAL interrupt
> > framework:
> > - rte_intr_irq_count_set: This API is used to set the total
> > interrupts on the interrupt handle. This would be provided
> > by VFIO (irq.count) for VFIO enabled devices.
> > - rte_intr_irq_count_get: This API returns the total number
> > interrupts which were set.
> [...]
> > --- a/lib/eal/common/eal_interrupts.h
> > +++ b/lib/eal/common/eal_interrupts.h
> > @@ -16,6 +16,7 @@ struct rte_intr_handle {
> > };
> > uint32_t alloc_flags; /**< flags passed at allocation */
> > enum rte_intr_handle_type type; /**< handle type */
> > + uint32_t irq_count; /**< IRQ count provided via VFIO */
>
> Why only via VFIO?
Though this represents total number of irq count, for VFIO it is returned by
VFIO_DEVICE_GET_IRQ_INFO ioctl call. I am not sure about UIO, so added
Comment w.r.t. VFIO only. I will make it generic in next spin.
>
> [...]
> > +/**
> > + * @internal
> > + * Set the irq count field of interrupt handle with user
> > + * provided irq count value.
> > + *
> > + * @param intr_handle
> > + * pointer to the interrupt handle.
> > + * @param irq_count
> > + * IRQ count
>
> Please write IRQ all uppercase consistently.
> Same for CDX.
Sure.
>
> > + rte_intr_irq_count_get;
> > + rte_intr_irq_count_set;
>
> Adding a new API in EAL deserves a separate commit with a different audience.
> It looks like it has been hidden from EAL reviewers eyes so far.
Will make a separate commit for this.
Thanks,
Nipun
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v4 4/4] bus/cdx: support plug unplug and dev iterator
2023-05-08 11:18 ` [PATCH v4 0/4] Support AMD CDX bus, for FPGA based CDX devices. The CDX Nipun Gupta
` (2 preceding siblings ...)
2023-05-08 11:18 ` [PATCH v4 3/4] bus/cdx: add support for MSI Nipun Gupta
@ 2023-05-08 11:18 ` Nipun Gupta
2023-05-12 10:25 ` [PATCH v4 0/4] Support AMD CDX bus, for FPGA based CDX devices. The CDX Ferruh Yigit
4 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-05-08 11:18 UTC (permalink / raw)
To: dev, thomas, david.marchand
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
This change adds support for plugging and unplugging
CDX devices on the CDX bus. Also, CDX dev iterator support
has been added for the CDX bus.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
---
drivers/bus/cdx/bus_cdx_driver.h | 1 +
drivers/bus/cdx/cdx.c | 122 +++++++++++++++++++++++++++++++
2 files changed, 123 insertions(+)
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index fdeaf46664..3d89e7c054 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -69,6 +69,7 @@ struct rte_cdx_id {
struct rte_cdx_device {
RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
struct rte_device device; /**< Inherit core device */
+ struct rte_cdx_driver *driver; /**< CDX driver used in probing */
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
/**< CDX Memory Resource */
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index 6c9ceaaf7f..0a30b8648a 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -68,6 +68,7 @@
#include <rte_eal_paging.h>
#include <rte_errno.h>
#include <rte_devargs.h>
+#include <rte_kvargs.h>
#include <rte_malloc.h>
#include <rte_memcpy.h>
#include <rte_vfio.h>
@@ -82,6 +83,15 @@
#define CDX_BUS_NAME cdx
#define CDX_DEV_PREFIX "cdx-"
+enum cdx_params {
+ RTE_CDX_PARAM_NAME,
+};
+
+static const char * const cdx_params_keys[] = {
+ [RTE_CDX_PARAM_NAME] = "name",
+ NULL,
+};
+
/**
* @file
* CDX probing using Linux sysfs.
@@ -399,6 +409,7 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr,
} else {
dev->device.driver = &dr->driver;
}
+ dev->driver = dr;
return ret;
@@ -516,6 +527,71 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
return NULL;
}
+/* Remove a device from CDX bus */
+static void
+cdx_remove_device(struct rte_cdx_device *cdx_dev)
+{
+ TAILQ_REMOVE(&rte_cdx_bus.device_list, cdx_dev, next);
+}
+
+/*
+ * If vendor/device ID match, call the remove() function of the
+ * driver.
+ */
+static int
+cdx_detach_dev(struct rte_cdx_device *dev)
+{
+ struct rte_cdx_driver *dr;
+ int ret = 0;
+
+ if (dev == NULL)
+ return -EINVAL;
+
+ dr = dev->driver;
+
+ CDX_BUS_DEBUG("detach device %s using driver: %s",
+ dev->device.name, dr->driver.name);
+
+ if (dr->remove) {
+ ret = dr->remove(dev);
+ if (ret < 0)
+ return ret;
+ }
+
+ /* clear driver structure */
+ dev->driver = NULL;
+ dev->device.driver = NULL;
+
+ rte_cdx_unmap_device(dev);
+
+ rte_intr_instance_free(dev->intr_handle);
+ dev->intr_handle = NULL;
+
+ return 0;
+}
+
+static int
+cdx_plug(struct rte_device *dev)
+{
+ return cdx_probe_all_drivers(RTE_DEV_TO_CDX_DEV(dev));
+}
+
+static int
+cdx_unplug(struct rte_device *dev)
+{
+ struct rte_cdx_device *cdx_dev;
+ int ret;
+
+ cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+ ret = cdx_detach_dev(cdx_dev);
+ if (ret == 0) {
+ cdx_remove_device(cdx_dev);
+ rte_devargs_remove(dev->devargs);
+ free(cdx_dev);
+ }
+ return ret;
+}
+
static int
cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
{
@@ -553,15 +629,61 @@ cdx_get_iommu_class(void)
return RTE_IOVA_VA;
}
+static int
+cdx_dev_match(const struct rte_device *dev,
+ const void *_kvlist)
+{
+ const struct rte_kvargs *kvlist = _kvlist;
+ const char *key = cdx_params_keys[RTE_CDX_PARAM_NAME];
+ const char *name;
+
+ /* no kvlist arg, all devices match */
+ if (kvlist == NULL)
+ return 0;
+
+ /* if key is present in kvlist and does not match, filter device */
+ name = rte_kvargs_get(kvlist, key);
+ if (name != NULL && strcmp(name, dev->name))
+ return -1;
+
+ return 0;
+}
+
+static void *
+cdx_dev_iterate(const void *start,
+ const char *str,
+ const struct rte_dev_iterator *it __rte_unused)
+{
+ rte_bus_find_device_t find_device;
+ struct rte_kvargs *kvargs = NULL;
+ struct rte_device *dev;
+
+ if (str != NULL) {
+ kvargs = rte_kvargs_parse(str, cdx_params_keys);
+ if (kvargs == NULL) {
+ CDX_BUS_ERR("cannot parse argument list %s", str);
+ rte_errno = EINVAL;
+ return NULL;
+ }
+ }
+ find_device = rte_cdx_bus.bus.find_device;
+ dev = find_device(start, cdx_dev_match, kvargs);
+ rte_kvargs_free(kvargs);
+ return dev;
+}
+
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
.scan = cdx_scan,
.probe = cdx_probe,
.find_device = cdx_find_device,
+ .plug = cdx_plug,
+ .unplug = cdx_unplug,
.parse = cdx_parse,
.dma_map = cdx_dma_map,
.dma_unmap = cdx_dma_unmap,
.get_iommu_class = cdx_get_iommu_class,
+ .dev_iterate = cdx_dev_iterate,
},
.device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
.driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v4 0/4] Support AMD CDX bus, for FPGA based CDX devices. The CDX
2023-05-08 11:18 ` [PATCH v4 0/4] Support AMD CDX bus, for FPGA based CDX devices. The CDX Nipun Gupta
` (3 preceding siblings ...)
2023-05-08 11:18 ` [PATCH v4 4/4] bus/cdx: support plug unplug and dev iterator Nipun Gupta
@ 2023-05-12 10:25 ` Ferruh Yigit
4 siblings, 0 replies; 102+ messages in thread
From: Ferruh Yigit @ 2023-05-12 10:25 UTC (permalink / raw)
To: Nipun Gupta, dev, thomas, david.marchand; +Cc: harpreet.anand, nikhil.agarwal
On 5/8/2023 12:18 PM, Nipun Gupta wrote:
> It uses sysfs interface and the vfio-cdx driver to discover
> and initialize the CDX devices.
>
> The CDX bus and VFIO support is available at Xilinx open source tree:
> https://github.com/Xilinx/linux-xlnx (drivers/cdx/ and drivers/vfio/cdx)
>
> Linux CDX bus patches has been added into linux next:
> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/cdx
>
> VFIO patches are also submitted in upstream:
> https://www.spinics.net/lists/kvm/msg310623.html
>
> CDX is a Hardware Architecture designed for AMD FPGA devices. It
> consists of mechanism for interaction between FPGA, Firmware and
> the APUs (Application CPUs).
> Firmware resides on RPU (Realtime CPUs) which interacts with
> the FPGA program manager and the APUs. The RPU provides memory-mapped
> interface (RPU if) which is used to communicate with APUs.
>
> VFIO CDX driver provides the CDX device resources like MMIO and interrupts
> to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
> driver to discover and initialize the CDX devices for user-space
> applications.
>
> Changes v3->v4:
> - removed platform specific patch (adding config for ARM CDX)
> from this series
>
> Changes v2->v3:
> - merged cdx bus compilation enablement in the first patch
> - fixed issue reported by check-git-log.sh
> - updated release notes
> - updated offset to uint64_t instead of off_t in cdx_map_resource
>
> Changes v1->v2:
> - Moved file rte_cdx_bus.h to internal bus_cdx_driver.h
> and added this file to deivce_cdx_headers
> - Moved cdx.h to private.h
> - Removed rte_ prefix from the static symbols in .c files.
>
> Changes RFC->v1:
> - Marked few API's as internal which were not required
> to be provided to user.
>
> Nipun Gupta (4):
> bus/cdx: introduce cdx bus
> bus/cdx: add DMA map and unmap support
> bus/cdx: add support for MSI
> bus/cdx: support plug unplug and dev iterator
For series,
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v5 0/5] Support AMD CDX bus
2023-01-24 14:07 [RFC PATCH 0/6] add support for CDX bus Nipun Gupta
` (9 preceding siblings ...)
2023-05-08 11:18 ` [PATCH v4 0/4] Support AMD CDX bus, for FPGA based CDX devices. The CDX Nipun Gupta
@ 2023-05-25 10:08 ` Nipun Gupta
2023-05-25 10:08 ` [PATCH v5 1/5] bus/cdx: introduce " Nipun Gupta
` (4 more replies)
2023-06-05 13:26 ` [PATCH v6 0/4] Support AMD CDX bus Nipun Gupta
` (2 subsequent siblings)
13 siblings, 5 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-05-25 10:08 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
The CDX devices are memory mapped on system bus for embedded CPUs.
It uses sysfs interface and the vfio-cdx driver to discover
and initialize the CDX devices.
The CDX bus and VFIO support is available at Xilinx/AMD open source tree:
https://github.com/Xilinx/linux-xlnx (drivers/cdx/ and drivers/vfio/cdx)
Linux AMD CDX bus patches has been added into linux tree:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/cdx
VFIO patches are also submitted in upstream:
https://www.spinics.net/lists/kvm/msg310623.html
CDX is a Hardware Architecture designed for AMD FPGA devices. It
consists of mechanism for interaction between FPGA, Firmware and
the APUs (Application CPUs).
Firmware resides on RPU (Realtime CPUs) which interacts with
the FPGA program manager and the APUs. The RPU provides memory-mapped
interface (RPU if) which is used to communicate with APUs.
VFIO CDX driver provides the CDX device resources like MMIO and interrupts
to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
driver to discover and initialize the CDX devices for user-space
applications.
Changes v4->v5
- Split patch and have a separate commit for eal interrupt field
(irq_count) and corresponding set/get APIs (rte_intr_irq_count_set,
rte_intr_irq_count_get)
- Renamed "CDX bus" to "AMD CDX bus" in documentation/commit headings
- Removed unnecessary headers
Changes v3->v4:
- removed platform specific patch (adding config for ARM CDX)
from this series
Changes v2->v3:
- merged cdx bus compilation enablement in the first patch
- fixed issue reported by check-git-log.sh
- updated release notes
- updated offset to uint64_t instead of off_t in cdx_map_resource
Changes v1->v2:
- Moved file rte_cdx_bus.h to internal bus_cdx_driver.h
and added this file to deivce_cdx_headers
- Moved cdx.h to private.h
- Removed rte_ prefix from the static symbols in .c files.
Changes RFC->v1:
- Marked few API's as internal which were not required
to be provided to user.
Nipun Gupta (5):
bus/cdx: introduce AMD CDX bus
bus/cdx: add DMA map and unmap support
eal/interrupts: add IRQ count in interrupt handle
bus/cdx: add support for MSI
bus/cdx: support plug unplug and dev iterator
MAINTAINERS | 5 +
doc/guides/rel_notes/release_23_07.rst | 6 +
drivers/bus/cdx/bus_cdx_driver.h | 222 ++++++++
drivers/bus/cdx/cdx.c | 690 +++++++++++++++++++++++++
drivers/bus/cdx/cdx_logs.h | 37 ++
drivers/bus/cdx/cdx_vfio.c | 605 ++++++++++++++++++++++
drivers/bus/cdx/meson.build | 13 +
drivers/bus/cdx/private.h | 46 ++
drivers/bus/cdx/version.map | 13 +
drivers/bus/meson.build | 1 +
lib/eal/common/eal_common_interrupts.c | 21 +
lib/eal/common/eal_interrupts.h | 1 +
lib/eal/include/rte_interrupts.h | 32 ++
lib/eal/version.map | 2 +
14 files changed, 1694 insertions(+)
create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
create mode 100644 drivers/bus/cdx/cdx.c
create mode 100644 drivers/bus/cdx/cdx_logs.h
create mode 100644 drivers/bus/cdx/cdx_vfio.c
create mode 100644 drivers/bus/cdx/meson.build
create mode 100644 drivers/bus/cdx/private.h
create mode 100644 drivers/bus/cdx/version.map
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v5 1/5] bus/cdx: introduce AMD CDX bus
2023-05-25 10:08 ` [PATCH v5 0/5] Support AMD CDX bus Nipun Gupta
@ 2023-05-25 10:08 ` Nipun Gupta
2023-06-01 15:00 ` David Marchand
2023-05-25 10:08 ` [PATCH v5 2/5] bus/cdx: add DMA map and unmap support Nipun Gupta
` (3 subsequent siblings)
4 siblings, 1 reply; 102+ messages in thread
From: Nipun Gupta @ 2023-05-25 10:08 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
AMD CDX bus supports multiple type of devices, which can be
exposed to user-space via vfio-cdx.
vfio-cdx provides the MMIO IO_MEMORY regions as well as the
DMA interface for the device (IOMMU).
This support aims to enable the DPDK to support the cdx
devices in user-space using VFIO interface.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
MAINTAINERS | 5 +
doc/guides/rel_notes/release_23_07.rst | 6 +
drivers/bus/cdx/bus_cdx_driver.h | 196 ++++++++++
drivers/bus/cdx/cdx.c | 517 +++++++++++++++++++++++++
drivers/bus/cdx/cdx_logs.h | 37 ++
drivers/bus/cdx/cdx_vfio.c | 427 ++++++++++++++++++++
drivers/bus/cdx/meson.build | 13 +
drivers/bus/cdx/private.h | 46 +++
drivers/bus/cdx/version.map | 11 +
drivers/bus/meson.build | 1 +
10 files changed, 1259 insertions(+)
create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
create mode 100644 drivers/bus/cdx/cdx.c
create mode 100644 drivers/bus/cdx/cdx_logs.h
create mode 100644 drivers/bus/cdx/cdx_vfio.c
create mode 100644 drivers/bus/cdx/meson.build
create mode 100644 drivers/bus/cdx/private.h
create mode 100644 drivers/bus/cdx/version.map
diff --git a/MAINTAINERS b/MAINTAINERS
index a5219926ab..c4b2b3565b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -569,6 +569,11 @@ M: Parav Pandit <parav@nvidia.com>
M: Xueming Li <xuemingl@nvidia.com>
F: drivers/bus/auxiliary/
+AMD CDX bus driver
+M: Nipun Gupta <nipun.gupta@amd.com>
+M: Nikhil Agarwal <nikhil.agarwal@amd.com>
+F: drivers/bus/cdx/
+
Intel FPGA bus
M: Rosen Xu <rosen.xu@intel.com>
F: drivers/bus/ifpga/
diff --git a/doc/guides/rel_notes/release_23_07.rst b/doc/guides/rel_notes/release_23_07.rst
index a9b1293689..7c6bb2b894 100644
--- a/doc/guides/rel_notes/release_23_07.rst
+++ b/doc/guides/rel_notes/release_23_07.rst
@@ -55,6 +55,12 @@ New Features
Also, make sure to start the actual text at the margin.
=======================================================
+* **Added AMD CDX bus support.**
+
+ CDX bus driver has been added to support AMD CDX bus, which operates
+ on FPGA based CDX devices. The CDX devices are memory mapped on system
+ bus for embedded CPUs.
+
Removed Items
-------------
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
new file mode 100644
index 0000000000..f1dce06a16
--- /dev/null
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -0,0 +1,196 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef BUS_CDX_DRIVER_H
+#define BUS_CDX_DRIVER_H
+
+/**
+ * @file
+ *
+ * AMD CDX bus interface
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdlib.h>
+#include <inttypes.h>
+
+#include <bus_driver.h>
+#include <dev_driver.h>
+#include <rte_interrupts.h>
+#include <rte_dev.h>
+#include <rte_bus.h>
+
+/* Forward declarations */
+struct rte_cdx_device;
+struct rte_cdx_driver;
+
+#define CDX_MAX_RESOURCE 4
+
+/** List of CDX devices */
+RTE_TAILQ_HEAD(rte_cdx_device_list, rte_cdx_device);
+/** List of CDX drivers */
+RTE_TAILQ_HEAD(rte_cdx_driver_list, rte_cdx_driver);
+
+/* CDX Bus iterators */
+#define FOREACH_DEVICE_ON_CDXBUS(p) \
+ RTE_TAILQ_FOREACH(p, &rte_cdx_bus.device_list, next)
+
+#define FOREACH_DRIVER_ON_CDXBUS(p) \
+ RTE_TAILQ_FOREACH(p, &rte_cdx_bus.driver_list, next)
+
+/** Any CDX device identifier (vendor, device) */
+#define RTE_CDX_ANY_ID (0xffff)
+
+#define RTE_PMD_REGISTER_CDX_TABLE(name, table) \
+static const char DRV_EXP_TAG(name, cdx_tbl_export)[] __rte_used = \
+RTE_STR(table)
+
+/**
+ * A structure describing an ID for a CDX driver. Each driver provides a
+ * table of these IDs for each device that it supports.
+ */
+struct rte_cdx_id {
+ uint16_t vendor_id; /**< Vendor ID. */
+ uint16_t device_id; /**< Device ID. */
+};
+
+/**
+ * A structure describing a CDX device.
+ */
+struct rte_cdx_device {
+ RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
+ struct rte_device device; /**< Inherit core device */
+ struct rte_cdx_id id; /**< CDX ID. */
+ struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
+ /**< CDX Memory Resource */
+};
+
+/**
+ * @internal
+ * Helper macro for drivers that need to convert to struct rte_cdx_device.
+ */
+#define RTE_DEV_TO_CDX_DEV(ptr) \
+ container_of(ptr, struct rte_cdx_device, device)
+
+#define RTE_DEV_TO_CDX_DEV_CONST(ptr) \
+ container_of(ptr, const struct rte_cdx_device, device)
+
+#define RTE_ETH_DEV_TO_CDX_DEV(eth_dev) RTE_DEV_TO_CDX_DEV((eth_dev)->device)
+
+#ifdef __cplusplus
+/** C++ macro used to help building up tables of device IDs */
+#define RTE_CDX_DEVICE(vend, dev) \
+ (vend), \
+ (dev)
+#else
+/** Macro used to help building up tables of device IDs */
+#define RTE_CDX_DEVICE(vend, dev) \
+ .vendor_id = (vend), \
+ .device_id = (dev)
+#endif
+
+/**
+ * Initialisation function for the driver called during CDX probing.
+ */
+typedef int (rte_cdx_probe_t)(struct rte_cdx_driver *, struct rte_cdx_device *);
+
+/**
+ * Uninitialisation function for the driver called during hotplugging.
+ */
+typedef int (rte_cdx_remove_t)(struct rte_cdx_device *);
+
+/**
+ * A structure describing a CDX driver.
+ */
+struct rte_cdx_driver {
+ RTE_TAILQ_ENTRY(rte_cdx_driver) next; /**< Next in list. */
+ struct rte_driver driver; /**< Inherit core driver. */
+ struct rte_cdx_bus *bus; /**< CDX bus reference. */
+ rte_cdx_probe_t *probe; /**< Device probe function. */
+ rte_cdx_remove_t *remove; /**< Device remove function. */
+ const struct rte_cdx_id *id_table; /**< ID table, NULL terminated. */
+ uint32_t drv_flags; /**< Flags RTE_CDX_DRV_*. */
+};
+
+/**
+ * Structure describing the CDX bus
+ */
+struct rte_cdx_bus {
+ struct rte_bus bus; /**< Inherit the generic class */
+ struct rte_cdx_device_list device_list; /**< List of CDX devices */
+ struct rte_cdx_driver_list driver_list; /**< List of CDX drivers */
+};
+
+/**
+ * Get Pathname of CDX devices directory.
+ *
+ * @return
+ * sysfs path for CDX devices.
+ */
+__rte_internal
+const char *rte_cdx_get_sysfs_path(void);
+
+/**
+ * Map the CDX device resources in user space virtual memory address
+ *
+ * @param dev
+ * A pointer to a rte_cdx_device structure describing the device
+ * to use
+ *
+ * @return
+ * 0 on success, negative on error and positive if no driver
+ * is found for the device.
+ */
+__rte_internal
+int rte_cdx_map_device(struct rte_cdx_device *dev);
+
+/**
+ * Unmap this device
+ *
+ * @param dev
+ * A pointer to a rte_cdx_device structure describing the device
+ * to use
+ */
+__rte_internal
+void rte_cdx_unmap_device(struct rte_cdx_device *dev);
+
+/**
+ * Register a CDX driver.
+ *
+ * @param driver
+ * A pointer to a rte_cdx_driver structure describing the driver
+ * to be registered.
+ */
+__rte_internal
+void rte_cdx_register(struct rte_cdx_driver *driver);
+
+/**
+ * Helper for CDX device registration from driver (eth, crypto, raw) instance
+ */
+#define RTE_PMD_REGISTER_CDX(nm, cdx_drv) \
+ RTE_INIT(cdxinitfn_ ##nm) \
+ {\
+ (cdx_drv).driver.name = RTE_STR(nm);\
+ rte_cdx_register(&cdx_drv); \
+ } \
+ RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+/**
+ * Unregister a CDX driver.
+ *
+ * @param driver
+ * A pointer to a rte_cdx_driver structure describing the driver
+ * to be unregistered.
+ */
+__rte_internal
+void rte_cdx_unregister(struct rte_cdx_driver *driver);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* BUS_CDX_DRIVER_H */
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
new file mode 100644
index 0000000000..1ddb5a92f7
--- /dev/null
+++ b/drivers/bus/cdx/cdx.c
@@ -0,0 +1,517 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+/*
+ * Architecture Overview
+ * =====================
+ * CDX is a Hardware Architecture designed for AMD FPGA devices. It
+ * consists of sophisticated mechanism for interaction between FPGA,
+ * Firmware and the APUs (Application CPUs).
+ *
+ * Firmware resides on RPU (Realtime CPUs) which interacts with
+ * the FPGA program manager and the APUs. The RPU provides memory-mapped
+ * interface (RPU if) which is used to communicate with APUs.
+ *
+ * The diagram below shows an overview of the AMD CDX architecture:
+ *
+ * +--------------------------------------+
+ * | DPDK |
+ * | DPDK CDX drivers |
+ * | | |
+ * | DPDK AMD CDX bus |
+ * | | |
+ * +-----------------------------|--------+
+ * |
+ * +-----------------------------|--------+
+ * | Application CPUs (APU) | |
+ * | | |
+ * | VFIO CDX driver |
+ * | Linux OS | |
+ * | Linux AMD CDX bus |
+ * | | |
+ * +-----------------------------|--------+
+ * |
+ * |
+ * +------------------------| RPU if |----+
+ * | | |
+ * | V |
+ * | Realtime CPUs (RPU) |
+ * | |
+ * +--------------------------------------+
+ * |
+ * +---------------------|----------------+
+ * | FPGA | |
+ * | +-----------------------+ |
+ * | | | | |
+ * | +-------+ +-------+ +-------+ |
+ * | | dev 1 | | dev 2 | | dev 3 | |
+ * | +-------+ +-------+ +-------+ |
+ * +--------------------------------------+
+ *
+ * The RPU firmware extracts the device information from the loaded FPGA
+ * image and implements a mechanism that allows the APU drivers to
+ * enumerate such devices (device personality and resource details) via
+ * a dedicated communication channel.
+ *
+ * VFIO CDX driver provides the CDX device resources like MMIO and interrupts
+ * to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
+ * driver to discover and initialize the CDX devices for user-space
+ * applications.
+ */
+
+#include <string.h>
+#include <dirent.h>
+
+#include <rte_eal_paging.h>
+#include <rte_errno.h>
+#include <rte_devargs.h>
+#include <rte_malloc.h>
+#include <rte_vfio.h>
+
+#include <eal_filesystem.h>
+
+#include "bus_cdx_driver.h"
+#include "cdx_logs.h"
+#include "private.h"
+
+#define SYSFS_CDX_DEVICES "/sys/bus/cdx/devices"
+#define CDX_BUS_NAME cdx
+#define CDX_DEV_PREFIX "cdx-"
+
+/**
+ * @file
+ * CDX probing using Linux sysfs.
+ */
+
+/* Add a device to CDX bus */
+static void
+cdx_add_device(struct rte_cdx_device *cdx_dev)
+{
+ TAILQ_INSERT_TAIL(&rte_cdx_bus.device_list, cdx_dev, next);
+}
+
+static int
+cdx_get_kernel_driver_by_path(const char *filename, char *driver_name,
+ size_t len)
+{
+ int count;
+ char path[PATH_MAX];
+ char *name;
+
+ if (!filename || !driver_name)
+ return -1;
+
+ count = readlink(filename, path, PATH_MAX);
+ if (count >= PATH_MAX)
+ return -1;
+
+ /* For device does not have a driver */
+ if (count < 0)
+ return 1;
+
+ path[count] = '\0';
+
+ name = strrchr(path, '/');
+ if (name) {
+ strlcpy(driver_name, name + 1, len);
+ return 0;
+ }
+
+ return -1;
+}
+
+int rte_cdx_map_device(struct rte_cdx_device *dev)
+{
+ return cdx_vfio_map_resource(dev);
+}
+
+void rte_cdx_unmap_device(struct rte_cdx_device *dev)
+{
+ cdx_vfio_unmap_resource(dev);
+}
+
+static struct rte_devargs *
+cdx_devargs_lookup(const char *dev_name)
+{
+ struct rte_devargs *devargs;
+
+ RTE_EAL_DEVARGS_FOREACH("cdx", devargs) {
+ if (strcmp(devargs->name, dev_name) == 0)
+ return devargs;
+ }
+ return NULL;
+}
+
+static bool
+cdx_ignore_device(const char *dev_name)
+{
+ struct rte_devargs *devargs = cdx_devargs_lookup(dev_name);
+
+ switch (rte_cdx_bus.bus.conf.scan_mode) {
+ case RTE_BUS_SCAN_ALLOWLIST:
+ if (devargs && devargs->policy == RTE_DEV_ALLOWED)
+ return false;
+ break;
+ case RTE_BUS_SCAN_UNDEFINED:
+ case RTE_BUS_SCAN_BLOCKLIST:
+ if (devargs == NULL || devargs->policy != RTE_DEV_BLOCKED)
+ return false;
+ break;
+ }
+ return true;
+}
+
+/*
+ * Scan one cdx sysfs entry, and fill the devices list from it.
+ * It checks if the CDX device is bound to vfio-cdx driver. In case
+ * the device is vfio bound, it reads the vendor and device id and
+ * stores it for device-driver matching.
+ */
+static int
+cdx_scan_one(const char *dirname, const char *dev_name)
+{
+ char filename[PATH_MAX];
+ struct rte_cdx_device *dev = NULL;
+ char driver[PATH_MAX];
+ unsigned long tmp;
+ char *name = NULL;
+ int ret;
+
+ dev = calloc(1, sizeof(*dev));
+ if (!dev)
+ return -ENOMEM;
+
+ name = calloc(1, RTE_DEV_NAME_MAX_LEN);
+ if (!name) {
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ dev->device.bus = &rte_cdx_bus.bus;
+ memcpy(name, dev_name, RTE_DEV_NAME_MAX_LEN);
+ dev->device.name = name;
+
+ /* parse driver */
+ snprintf(filename, sizeof(filename), "%s/driver", dirname);
+ ret = cdx_get_kernel_driver_by_path(filename, driver, sizeof(driver));
+ if (ret < 0) {
+ CDX_BUS_ERR("Fail to get kernel driver");
+ ret = -1;
+ goto err;
+ }
+
+ /*
+ * Check if device is bound to 'vfio-cdx' driver, so that user-space
+ * can gracefully access the device.
+ */
+ if (ret || strcmp(driver, "vfio-cdx")) {
+ ret = 0;
+ goto err;
+ }
+
+ /* get vendor id */
+ snprintf(filename, sizeof(filename), "%s/vendor", dirname);
+ if (eal_parse_sysfs_value(filename, &tmp) < 0) {
+ ret = -1;
+ goto err;
+ }
+ dev->id.vendor_id = (uint16_t)tmp;
+
+ /* get device id */
+ snprintf(filename, sizeof(filename), "%s/device", dirname);
+ if (eal_parse_sysfs_value(filename, &tmp) < 0) {
+ free(dev);
+ return -1;
+ }
+ dev->id.device_id = (uint16_t)tmp;
+
+ cdx_add_device(dev);
+
+ return 0;
+
+err:
+ if (name)
+ free(name);
+ if (dev)
+ free(dev);
+ return ret;
+}
+
+/*
+ * Scan the content of the CDX bus, and the devices in the devices
+ * list.
+ */
+static int
+cdx_scan(void)
+{
+ struct dirent *e;
+ DIR *dir;
+ char dirname[PATH_MAX];
+
+ dir = opendir(rte_cdx_get_sysfs_path());
+ if (dir == NULL) {
+ CDX_BUS_ERR("%s(): opendir failed: %s", __func__,
+ strerror(errno));
+ return -1;
+ }
+
+ while ((e = readdir(dir)) != NULL) {
+ if (e->d_name[0] == '.')
+ continue;
+
+ if (cdx_ignore_device(e->d_name))
+ continue;
+
+ snprintf(dirname, sizeof(dirname), "%s/%s",
+ rte_cdx_get_sysfs_path(), e->d_name);
+
+ if (cdx_scan_one(dirname, e->d_name) < 0)
+ goto error;
+ }
+ closedir(dir);
+ return 0;
+
+error:
+ closedir(dir);
+ return -1;
+}
+
+const char *
+rte_cdx_get_sysfs_path(void)
+{
+ return SYSFS_CDX_DEVICES;
+}
+
+/* map a particular resource from a file */
+void *
+cdx_map_resource(void *requested_addr, int fd, uint64_t offset, size_t size,
+ int additional_flags)
+{
+ void *mapaddr;
+
+ /* Map the cdx MMIO memory resource of device */
+ mapaddr = rte_mem_map(requested_addr, size,
+ RTE_PROT_READ | RTE_PROT_WRITE,
+ RTE_MAP_SHARED | additional_flags, fd, offset);
+ if (mapaddr == NULL) {
+ CDX_BUS_ERR("%s(): cannot map resource(%d, %p, 0x%zx, 0x%"PRIx64"): %s (%p)",
+ __func__, fd, requested_addr, size, offset,
+ rte_strerror(rte_errno), mapaddr);
+ }
+ CDX_BUS_DEBUG("CDX MMIO memory mapped at %p", mapaddr);
+
+ return mapaddr;
+}
+
+/* unmap a particular resource */
+void
+cdx_unmap_resource(void *requested_addr, size_t size)
+{
+ if (requested_addr == NULL)
+ return;
+
+ /* Unmap the CDX memory resource of device */
+ if (rte_mem_unmap(requested_addr, size)) {
+ CDX_BUS_ERR("%s(): cannot mem unmap(%p, %#zx): %s", __func__,
+ requested_addr, size, rte_strerror(rte_errno));
+ }
+ CDX_BUS_DEBUG("CDX memory unmapped at %p", requested_addr);
+}
+/*
+ * Match the CDX Driver and Device using device id and vendor id.
+ */
+static int
+cdx_match(const struct rte_cdx_driver *cdx_drv,
+ const struct rte_cdx_device *cdx_dev)
+{
+ const struct rte_cdx_id *id_table;
+
+ for (id_table = cdx_drv->id_table; id_table->vendor_id != 0;
+ id_table++) {
+ /* check if device's identifiers match the driver's ones */
+ if (id_table->vendor_id != cdx_dev->id.vendor_id &&
+ id_table->vendor_id != RTE_CDX_ANY_ID)
+ continue;
+ if (id_table->device_id != cdx_dev->id.device_id &&
+ id_table->device_id != RTE_CDX_ANY_ID)
+ continue;
+
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * If vendor id and device id match, call the probe() function of the
+ * driver.
+ */
+static int
+cdx_probe_one_driver(struct rte_cdx_driver *dr,
+ struct rte_cdx_device *dev)
+{
+ const char *dev_name = dev->device.name;
+ bool already_probed;
+ int ret;
+
+ if ((dr == NULL) || (dev == NULL))
+ return -EINVAL;
+
+ /* The device is not blocked; Check if driver supports it */
+ if (!cdx_match(dr, dev))
+ /* Match of device and driver failed */
+ return 1;
+
+ already_probed = rte_dev_is_probed(&dev->device);
+ if (already_probed) {
+ CDX_BUS_INFO("Device %s is already probed", dev->device.name);
+ return -EEXIST;
+ }
+
+ CDX_BUS_DEBUG(" probe device %s using driver: %s", dev_name,
+ dr->driver.name);
+
+ ret = cdx_vfio_map_resource(dev);
+ if (ret != 0) {
+ CDX_BUS_ERR("CDX map device failed: %d", ret);
+ goto error_map_device;
+ }
+
+ /* call the driver probe() function */
+ ret = dr->probe(dr, dev);
+ if (ret) {
+ CDX_BUS_ERR("Probe CDX driver: %s device: %s failed: %d",
+ dr->driver.name, dev_name, ret);
+ goto error_probe;
+ } else {
+ dev->device.driver = &dr->driver;
+ }
+
+ return ret;
+
+error_probe:
+ cdx_vfio_unmap_resource(dev);
+error_map_device:
+ return ret;
+}
+
+/*
+ * If vendor/device ID match, call the probe() function of all
+ * registered driver for the given device. Return < 0 if initialization
+ * failed, return 1 if no driver is found for this device.
+ */
+static int
+cdx_probe_all_drivers(struct rte_cdx_device *dev)
+{
+ struct rte_cdx_driver *dr = NULL;
+ int rc = 0;
+
+ if (dev == NULL)
+ return -EINVAL;
+
+ FOREACH_DRIVER_ON_CDXBUS(dr) {
+ rc = cdx_probe_one_driver(dr, dev);
+ if (rc < 0)
+ /* negative value is an error */
+ return rc;
+ if (rc > 0)
+ /* positive value means driver doesn't support it */
+ continue;
+ return 0;
+ }
+ return 1;
+}
+
+/*
+ * Scan the content of the CDX bus, and call the probe() function for
+ * all registered drivers that have a matching entry in its id_table
+ * for discovered devices.
+ */
+static int
+cdx_probe(void)
+{
+ struct rte_cdx_device *dev = NULL;
+ size_t probed = 0, failed = 0;
+ int ret = 0;
+
+ FOREACH_DEVICE_ON_CDXBUS(dev) {
+ probed++;
+
+ ret = cdx_probe_all_drivers(dev);
+ if (ret < 0) {
+ CDX_BUS_ERR("Requested device %s cannot be used",
+ dev->device.name);
+ rte_errno = errno;
+ failed++;
+ ret = 0;
+ }
+ }
+
+ return (probed && probed == failed) ? -1 : 0;
+}
+
+static int
+cdx_parse(const char *name, void *addr)
+{
+ const char **out = addr;
+ int ret;
+
+ ret = strncmp(name, CDX_DEV_PREFIX, strlen(CDX_DEV_PREFIX));
+
+ if (ret == 0 && addr)
+ *out = name;
+
+ return ret;
+}
+
+/* register a driver */
+void
+rte_cdx_register(struct rte_cdx_driver *driver)
+{
+ TAILQ_INSERT_TAIL(&rte_cdx_bus.driver_list, driver, next);
+ driver->bus = &rte_cdx_bus;
+}
+
+/* unregister a driver */
+void
+rte_cdx_unregister(struct rte_cdx_driver *driver)
+{
+ TAILQ_REMOVE(&rte_cdx_bus.driver_list, driver, next);
+ driver->bus = NULL;
+}
+
+static struct rte_device *
+cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
+ const void *data)
+{
+ const struct rte_cdx_device *cdx_start;
+ struct rte_cdx_device *cdx_dev;
+
+ if (start != NULL) {
+ cdx_start = RTE_DEV_TO_CDX_DEV_CONST(start);
+ cdx_dev = TAILQ_NEXT(cdx_start, next);
+ } else {
+ cdx_dev = TAILQ_FIRST(&rte_cdx_bus.device_list);
+ }
+ while (cdx_dev != NULL) {
+ if (cmp(&cdx_dev->device, data) == 0)
+ return &cdx_dev->device;
+ cdx_dev = TAILQ_NEXT(cdx_dev, next);
+ }
+ return NULL;
+}
+
+struct rte_cdx_bus rte_cdx_bus = {
+ .bus = {
+ .scan = cdx_scan,
+ .probe = cdx_probe,
+ .find_device = cdx_find_device,
+ .parse = cdx_parse,
+ },
+ .device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
+ .driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
+};
+
+RTE_REGISTER_BUS(cdx, rte_cdx_bus.bus);
+RTE_LOG_REGISTER_DEFAULT(cdx_logtype_bus, NOTICE);
diff --git a/drivers/bus/cdx/cdx_logs.h b/drivers/bus/cdx/cdx_logs.h
new file mode 100644
index 0000000000..929c83f1e6
--- /dev/null
+++ b/drivers/bus/cdx/cdx_logs.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef CDX_LOGS_H
+#define CDX_LOGS_H
+
+extern int cdx_logtype_bus;
+
+#define CDX_BUS_LOG(level, fmt, args...) \
+ rte_log(RTE_LOG_ ## level, cdx_logtype_bus, "cdx: " fmt "\n", \
+ ##args)
+
+/* Debug logs are with Function names */
+#define CDX_BUS_DEBUG(fmt, args...) \
+ rte_log(RTE_LOG_DEBUG, cdx_logtype_bus, "cdx: %s(): " fmt "\n", \
+ __func__, ##args)
+
+#define CDX_BUS_INFO(fmt, args...) \
+ CDX_BUS_LOG(INFO, fmt, ## args)
+#define CDX_BUS_ERR(fmt, args...) \
+ CDX_BUS_LOG(ERR, fmt, ## args)
+#define CDX_BUS_WARN(fmt, args...) \
+ CDX_BUS_LOG(WARNING, fmt, ## args)
+
+/* DP Logs, toggled out at compile time if level lower than current level */
+#define CDX_BUS_DP_LOG(level, fmt, args...) \
+ RTE_LOG_DP(level, PMD, fmt, ## args)
+
+#define CDX_BUS_DP_DEBUG(fmt, args...) \
+ CDX_BUS_DP_LOG(DEBUG, fmt, ## args)
+#define CDX_BUS_DP_INFO(fmt, args...) \
+ CDX_BUS_DP_LOG(INFO, fmt, ## args)
+#define CDX_BUS_DP_WARN(fmt, args...) \
+ CDX_BUS_DP_LOG(WARNING, fmt, ## args)
+
+#endif /* CDX_LOGS_H */
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
new file mode 100644
index 0000000000..e54432de5b
--- /dev/null
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -0,0 +1,427 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+/**
+ * @file
+ * CDX probing using Linux VFIO.
+ *
+ * This code tries to determine if the CDX device is bound to VFIO driver,
+ * and initialize it (map MMIO regions, set up interrupts) if that's the case.
+ *
+ */
+
+#include <fcntl.h>
+#include <unistd.h>
+#include <sys/eventfd.h>
+#include <sys/socket.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <rte_eal_paging.h>
+#include <rte_malloc.h>
+#include <rte_vfio.h>
+
+#include "bus_cdx_driver.h"
+#include "cdx_logs.h"
+#include "private.h"
+/**
+ * A structure describing a CDX mapping.
+ */
+struct cdx_map {
+ void *addr;
+ char *path;
+ uint64_t offset;
+ uint64_t size;
+};
+
+/**
+ * A structure describing a mapped CDX resource.
+ * For multi-process we need to reproduce all CDX mappings in secondary
+ * processes, so save them in a tailq.
+ */
+struct mapped_cdx_resource {
+ TAILQ_ENTRY(mapped_cdx_resource) next;
+ char name[RTE_DEV_NAME_MAX_LEN]; /**< CDX device name */
+ char path[PATH_MAX];
+ int nb_maps;
+ struct cdx_map maps[CDX_MAX_RESOURCE];
+};
+
+/** mapped cdx device list */
+TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
+
+static struct rte_tailq_elem cdx_vfio_tailq = {
+ .name = "VFIO_CDX_RESOURCE_LIST",
+};
+EAL_REGISTER_TAILQ(cdx_vfio_tailq)
+
+static struct mapped_cdx_resource *
+cdx_vfio_find_and_unmap_resource(struct mapped_cdx_res_list *vfio_res_list,
+ struct rte_cdx_device *dev)
+{
+ struct mapped_cdx_resource *vfio_res = NULL;
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+ int i;
+
+ /* Get vfio_res */
+ TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+ if (strcmp(vfio_res->name, dev_name))
+ continue;
+ break;
+ }
+
+ if (vfio_res == NULL)
+ return vfio_res;
+
+ CDX_BUS_INFO("Releasing CDX mapped resource for %s", dev_name);
+
+ maps = vfio_res->maps;
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ if (maps[i].addr) {
+ CDX_BUS_DEBUG("Calling cdx_unmap_resource for %s at %p",
+ dev_name, maps[i].addr);
+ cdx_unmap_resource(maps[i].addr, maps[i].size);
+ }
+ }
+
+ return vfio_res;
+}
+
+static int
+cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
+{
+ char cdx_addr[PATH_MAX] = {0};
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list;
+
+ vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ vfio_res = cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for cdx device!",
+ cdx_addr);
+ return -1;
+ }
+
+ TAILQ_REMOVE(vfio_res_list, vfio_res, next);
+ rte_free(vfio_res);
+ return 0;
+}
+
+static int
+cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
+{
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list;
+
+ vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ vfio_res = cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for CDX device!",
+ dev->device.name);
+ return -1;
+ }
+
+ return 0;
+}
+
+int
+cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
+{
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return cdx_vfio_unmap_resource_primary(dev);
+ else
+ return cdx_vfio_unmap_resource_secondary(dev);
+}
+
+static int
+cdx_rte_vfio_setup_device(int vfio_dev_fd)
+{
+ /*
+ * Reset the device. If the device is not capable of resetting,
+ * then it updates errno as EINVAL.
+ */
+ if (ioctl(vfio_dev_fd, VFIO_DEVICE_RESET) && errno != EINVAL) {
+ CDX_BUS_ERR("Unable to reset device! Error: %d (%s)", errno,
+ strerror(errno));
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+cdx_vfio_mmap_resource(int vfio_dev_fd, struct mapped_cdx_resource *vfio_res,
+ int index, int additional_flags)
+{
+ struct cdx_map *map = &vfio_res->maps[index];
+ void *vaddr;
+
+ if (map->size == 0) {
+ CDX_BUS_DEBUG("map size is 0, skip region %d", index);
+ return 0;
+ }
+
+ /* reserve the address using an inaccessible mapping */
+ vaddr = mmap(map->addr, map->size, 0, MAP_PRIVATE |
+ MAP_ANONYMOUS | additional_flags, -1, 0);
+ if (vaddr != MAP_FAILED) {
+ void *map_addr = NULL;
+
+ if (map->size) {
+ /* actual map of first part */
+ map_addr = cdx_map_resource(vaddr, vfio_dev_fd,
+ map->offset, map->size,
+ RTE_MAP_FORCE_ADDRESS);
+ }
+
+ if (map_addr == NULL) {
+ munmap(vaddr, map->size);
+ vaddr = MAP_FAILED;
+ CDX_BUS_ERR("Failed to map cdx MMIO region %d", index);
+ return -1;
+ }
+ } else {
+ CDX_BUS_ERR("Failed to create inaccessible mapping for MMIO region %d",
+ index);
+ return -1;
+ }
+
+ map->addr = vaddr;
+ return 0;
+}
+
+/*
+ * region info may contain capability headers, so we need to keep reallocating
+ * the memory until we match allocated memory size with argsz.
+ */
+static int
+cdx_vfio_get_region_info(int vfio_dev_fd, struct vfio_region_info **info,
+ int region)
+{
+ struct vfio_region_info *ri;
+ size_t argsz = sizeof(*ri);
+ int ret;
+
+ ri = malloc(sizeof(*ri));
+ if (ri == NULL) {
+ CDX_BUS_ERR("Cannot allocate memory for VFIO region info");
+ return -1;
+ }
+again:
+ memset(ri, 0, argsz);
+ ri->argsz = argsz;
+ ri->index = region;
+
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_REGION_INFO, ri);
+ if (ret < 0) {
+ free(ri);
+ return ret;
+ }
+ if (ri->argsz != argsz) {
+ struct vfio_region_info *tmp;
+
+ argsz = ri->argsz;
+ tmp = realloc(ri, argsz);
+
+ if (tmp == NULL) {
+ /* realloc failed but the ri is still there */
+ free(ri);
+ CDX_BUS_ERR("Cannot reallocate memory for VFIO region info");
+ return -1;
+ }
+ ri = tmp;
+ goto again;
+ }
+ *info = ri;
+
+ return 0;
+}
+
+static int
+find_max_end_va(const struct rte_memseg_list *msl, void *arg)
+{
+ size_t sz = msl->len;
+ void *end_va = RTE_PTR_ADD(msl->base_va, sz);
+ void **max_va = arg;
+
+ if (*max_va < end_va)
+ *max_va = end_va;
+ return 0;
+}
+
+static void *
+cdx_find_max_end_va(void)
+{
+ void *va = NULL;
+
+ rte_memseg_list_walk(find_max_end_va, &va);
+ return va;
+}
+
+static int
+cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ char cdx_addr[PATH_MAX] = {0};
+ static void *cdx_map_addr;
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+ int vfio_dev_fd, i, ret;
+
+ ret = rte_vfio_setup_device(rte_cdx_get_sysfs_path(), dev_name,
+ &vfio_dev_fd, &device_info);
+ if (ret)
+ return ret;
+
+ /* allocate vfio_res and get region info */
+ vfio_res = rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0);
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("Cannot store VFIO mmap details");
+ goto err_vfio_dev_fd;
+ }
+ memcpy(vfio_res->name, dev_name, RTE_DEV_NAME_MAX_LEN);
+
+ /* get number of registers */
+ vfio_res->nb_maps = device_info.num_regions;
+
+ /* map memory regions */
+ maps = vfio_res->maps;
+
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ struct vfio_region_info *reg = NULL;
+ void *vaddr;
+
+ ret = cdx_vfio_get_region_info(vfio_dev_fd, ®, i);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s cannot get device region info error %i (%s)",
+ dev_name, errno, strerror(errno));
+ goto err_vfio_res;
+ }
+
+ /* skip non-mmappable regions */
+ if ((reg->flags & VFIO_REGION_INFO_FLAG_MMAP) == 0) {
+ free(reg);
+ continue;
+ }
+
+ /* try mapping somewhere close to the end of hugepages */
+ if (cdx_map_addr == NULL)
+ cdx_map_addr = cdx_find_max_end_va();
+
+ vaddr = cdx_map_addr;
+ cdx_map_addr = RTE_PTR_ADD(vaddr, (size_t)reg->size);
+
+ cdx_map_addr = RTE_PTR_ALIGN(cdx_map_addr,
+ sysconf(_SC_PAGE_SIZE));
+
+ maps[i].addr = vaddr;
+ maps[i].offset = reg->offset;
+ maps[i].size = reg->size;
+ maps[i].path = NULL; /* vfio doesn't have per-resource paths */
+
+ ret = cdx_vfio_mmap_resource(vfio_dev_fd, vfio_res, i, 0);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s mapping region %i failed: %s",
+ cdx_addr, i, strerror(errno));
+ free(reg);
+ goto err_vfio_res;
+ }
+
+ dev->mem_resource[i].addr = maps[i].addr;
+ dev->mem_resource[i].len = maps[i].size;
+
+ free(reg);
+ }
+
+ if (cdx_rte_vfio_setup_device(vfio_dev_fd) < 0) {
+ CDX_BUS_ERR("%s setup device failed", dev_name);
+ goto err_vfio_res;
+ }
+
+ TAILQ_INSERT_TAIL(vfio_res_list, vfio_res, next);
+
+ return 0;
+err_vfio_res:
+ cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+ rte_free(vfio_res);
+err_vfio_dev_fd:
+ rte_vfio_release_device(rte_cdx_get_sysfs_path(),
+ dev_name, vfio_dev_fd);
+ return -1;
+}
+
+static int
+cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ char cdx_addr[PATH_MAX] = {0};
+ int vfio_dev_fd;
+ int i, ret;
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+
+ /* if we're in a secondary process, just find our tailq entry */
+ TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+ if (strcmp(vfio_res->name, dev_name))
+ continue;
+ break;
+ }
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for cdx device!",
+ dev_name);
+ return -1;
+ }
+
+ ret = rte_vfio_setup_device(rte_cdx_get_sysfs_path(), dev_name,
+ &vfio_dev_fd, &device_info);
+ if (ret)
+ return ret;
+
+ /* map MMIO regions */
+ maps = vfio_res->maps;
+
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ ret = cdx_vfio_mmap_resource(vfio_dev_fd, vfio_res, i, MAP_FIXED);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s mapping MMIO region %i failed: %s",
+ dev_name, i, strerror(errno));
+ goto err_vfio_dev_fd;
+ }
+
+ dev->mem_resource[i].addr = maps[i].addr;
+ dev->mem_resource[i].len = maps[i].size;
+ }
+
+ return 0;
+err_vfio_dev_fd:
+ rte_vfio_release_device(rte_cdx_get_sysfs_path(),
+ cdx_addr, vfio_dev_fd);
+ return -1;
+}
+
+/*
+ * map the CDX resources of a CDX device in virtual memory (VFIO version).
+ * primary and secondary processes follow almost exactly the same path
+ */
+int
+cdx_vfio_map_resource(struct rte_cdx_device *dev)
+{
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return cdx_vfio_map_resource_primary(dev);
+ else
+ return cdx_vfio_map_resource_secondary(dev);
+}
diff --git a/drivers/bus/cdx/meson.build b/drivers/bus/cdx/meson.build
new file mode 100644
index 0000000000..f2ca104d34
--- /dev/null
+++ b/drivers/bus/cdx/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+
+if not is_linux
+ build = false
+ reason = 'only supported on Linux'
+endif
+
+driver_sdk_headers = files('bus_cdx_driver.h')
+sources = files(
+ 'cdx.c',
+ 'cdx_vfio.c',
+)
diff --git a/drivers/bus/cdx/private.h b/drivers/bus/cdx/private.h
new file mode 100644
index 0000000000..c5ce5a46b8
--- /dev/null
+++ b/drivers/bus/cdx/private.h
@@ -0,0 +1,46 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef CDX_PRIVATE_H
+#define CDX_PRIVATE_H
+
+#include "bus_cdx_driver.h"
+
+extern struct rte_cdx_bus rte_cdx_bus;
+
+/**
+ * Map a particular resource from a file.
+ *
+ * @param requested_addr
+ * The starting address for the new mapping range.
+ * @param fd
+ * The file descriptor.
+ * @param offset
+ * The offset for the mapping range.
+ * @param size
+ * The size for the mapping range.
+ * @param additional_flags
+ * The additional rte_mem_map() flags for the mapping range.
+ * @return
+ * - On success, the function returns a pointer to the mapped area.
+ * - On error, NULL is returned.
+ */
+void *cdx_map_resource(void *requested_addr, int fd, uint64_t offset,
+ size_t size, int additional_flags);
+
+/**
+ * Unmap a particular resource.
+ *
+ * @param requested_addr
+ * The address for the unmapping range.
+ * @param size
+ * The size for the unmapping range.
+ */
+void cdx_unmap_resource(void *requested_addr, size_t size);
+
+/* map/unmap VFIO resource */
+int cdx_vfio_map_resource(struct rte_cdx_device *dev);
+int cdx_vfio_unmap_resource(struct rte_cdx_device *dev);
+
+#endif /* CDX_PRIVATE_H */
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
new file mode 100644
index 0000000000..957fcab978
--- /dev/null
+++ b/drivers/bus/cdx/version.map
@@ -0,0 +1,11 @@
+INTERNAL {
+ global:
+
+ rte_cdx_get_sysfs_path;
+ rte_cdx_map_device;
+ rte_cdx_register;
+ rte_cdx_unmap_device;
+ rte_cdx_unregister;
+
+ local: *;
+};
diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
index 6d2520c543..a78b4283bf 100644
--- a/drivers/bus/meson.build
+++ b/drivers/bus/meson.build
@@ -3,6 +3,7 @@
drivers = [
'auxiliary',
+ 'cdx',
'dpaa',
'fslmc',
'ifpga',
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v5 1/5] bus/cdx: introduce AMD CDX bus
2023-05-25 10:08 ` [PATCH v5 1/5] bus/cdx: introduce " Nipun Gupta
@ 2023-06-01 15:00 ` David Marchand
2023-06-01 18:10 ` Nipun Gupta
2023-06-05 8:04 ` Nipun Gupta
0 siblings, 2 replies; 102+ messages in thread
From: David Marchand @ 2023-06-01 15:00 UTC (permalink / raw)
To: Nipun Gupta
Cc: dev, thomas, hkalra, anatoly.burakov, stephen, ferruh.yigit,
harpreet.anand, nikhil.agarwal
Hello,
On Thu, May 25, 2023 at 12:08 PM Nipun Gupta <nipun.gupta@amd.com> wrote:
>
> AMD CDX bus supports multiple type of devices, which can be
> exposed to user-space via vfio-cdx.
>
> vfio-cdx provides the MMIO IO_MEMORY regions as well as the
> DMA interface for the device (IOMMU).
>
> This support aims to enable the DPDK to support the cdx
> devices in user-space using VFIO interface.
>
> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
> ---
> MAINTAINERS | 5 +
> doc/guides/rel_notes/release_23_07.rst | 6 +
> drivers/bus/cdx/bus_cdx_driver.h | 196 ++++++++++
> drivers/bus/cdx/cdx.c | 517 +++++++++++++++++++++++++
> drivers/bus/cdx/cdx_logs.h | 37 ++
> drivers/bus/cdx/cdx_vfio.c | 427 ++++++++++++++++++++
> drivers/bus/cdx/meson.build | 13 +
> drivers/bus/cdx/private.h | 46 +++
> drivers/bus/cdx/version.map | 11 +
> drivers/bus/meson.build | 1 +
> 10 files changed, 1259 insertions(+)
> create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
> create mode 100644 drivers/bus/cdx/cdx.c
> create mode 100644 drivers/bus/cdx/cdx_logs.h
> create mode 100644 drivers/bus/cdx/cdx_vfio.c
> create mode 100644 drivers/bus/cdx/meson.build
> create mode 100644 drivers/bus/cdx/private.h
> create mode 100644 drivers/bus/cdx/version.map
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index a5219926ab..c4b2b3565b 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -569,6 +569,11 @@ M: Parav Pandit <parav@nvidia.com>
> M: Xueming Li <xuemingl@nvidia.com>
> F: drivers/bus/auxiliary/
>
> +AMD CDX bus driver
> +M: Nipun Gupta <nipun.gupta@amd.com>
> +M: Nikhil Agarwal <nikhil.agarwal@amd.com>
> +F: drivers/bus/cdx/
> +
> Intel FPGA bus
> M: Rosen Xu <rosen.xu@intel.com>
> F: drivers/bus/ifpga/
> diff --git a/doc/guides/rel_notes/release_23_07.rst b/doc/guides/rel_notes/release_23_07.rst
> index a9b1293689..7c6bb2b894 100644
> --- a/doc/guides/rel_notes/release_23_07.rst
> +++ b/doc/guides/rel_notes/release_23_07.rst
> @@ -55,6 +55,12 @@ New Features
> Also, make sure to start the actual text at the margin.
> =======================================================
>
> +* **Added AMD CDX bus support.**
> +
> + CDX bus driver has been added to support AMD CDX bus, which operates
> + on FPGA based CDX devices. The CDX devices are memory mapped on system
> + bus for embedded CPUs.
> +
>
> Removed Items
> -------------
> diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
> new file mode 100644
> index 0000000000..f1dce06a16
> --- /dev/null
> +++ b/drivers/bus/cdx/bus_cdx_driver.h
> @@ -0,0 +1,196 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
> + */
> +
> +#ifndef BUS_CDX_DRIVER_H
> +#define BUS_CDX_DRIVER_H
> +
> +/**
> + * @file
> + *
No need for empty line here.
> + * AMD CDX bus interface
> + */
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
Do you expect some out of tree drivers written in C++?
Otherwise, this is unneeded.
> +
> +#include <stdlib.h>
> +#include <inttypes.h>
> +
> +#include <bus_driver.h>
> +#include <dev_driver.h>
> +#include <rte_interrupts.h>
> +#include <rte_dev.h>
dev_driver.h is the "driver" header embedding rte_dev.h, no need to
include rte_dev.h again.
> +#include <rte_bus.h>
Idem, including bus_driver.h is enough.
> +
> +/* Forward declarations */
> +struct rte_cdx_device;
> +struct rte_cdx_driver;
> +
> +#define CDX_MAX_RESOURCE 4
> +
> +/** List of CDX devices */
> +RTE_TAILQ_HEAD(rte_cdx_device_list, rte_cdx_device);
> +/** List of CDX drivers */
> +RTE_TAILQ_HEAD(rte_cdx_driver_list, rte_cdx_driver);
You don't need to name/expose those types, this is only used in the
singleton object for the bus (see below with comment on rte_cdx_bus
struct).
> +
> +/* CDX Bus iterators */
> +#define FOREACH_DEVICE_ON_CDXBUS(p) \
> + RTE_TAILQ_FOREACH(p, &rte_cdx_bus.device_list, next)
> +
> +#define FOREACH_DRIVER_ON_CDXBUS(p) \
> + RTE_TAILQ_FOREACH(p, &rte_cdx_bus.driver_list, next)
These iterators can probably be hidden in the bus code.
I see no need to expose them to drivers.
So either move this to cdx.c, or if multiple code units need them,
move to private.h.
> +
> +/** Any CDX device identifier (vendor, device) */
> +#define RTE_CDX_ANY_ID (0xffff)
> +
> +#define RTE_PMD_REGISTER_CDX_TABLE(name, table) \
> +static const char DRV_EXP_TAG(name, cdx_tbl_export)[] __rte_used = \
> +RTE_STR(table)
> +
> +/**
> + * A structure describing an ID for a CDX driver. Each driver provides a
> + * table of these IDs for each device that it supports.
> + */
> +struct rte_cdx_id {
> + uint16_t vendor_id; /**< Vendor ID. */
> + uint16_t device_id; /**< Device ID. */
> +};
> +
> +/**
> + * A structure describing a CDX device.
> + */
> +struct rte_cdx_device {
> + RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
> + struct rte_device device; /**< Inherit core device */
> + struct rte_cdx_id id; /**< CDX ID. */
> + struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
> + /**< CDX Memory Resource */
> +};
> +
> +/**
> + * @internal
> + * Helper macro for drivers that need to convert to struct rte_cdx_device.
> + */
> +#define RTE_DEV_TO_CDX_DEV(ptr) \
> + container_of(ptr, struct rte_cdx_device, device)
> +
> +#define RTE_DEV_TO_CDX_DEV_CONST(ptr) \
> + container_of(ptr, const struct rte_cdx_device, device)
> +
> +#define RTE_ETH_DEV_TO_CDX_DEV(eth_dev) RTE_DEV_TO_CDX_DEV((eth_dev)->device)
> +
> +#ifdef __cplusplus
> +/** C++ macro used to help building up tables of device IDs */
> +#define RTE_CDX_DEVICE(vend, dev) \
> + (vend), \
> + (dev)
> +#else
> +/** Macro used to help building up tables of device IDs */
> +#define RTE_CDX_DEVICE(vend, dev) \
> + .vendor_id = (vend), \
> + .device_id = (dev)
> +#endif
> +
> +/**
> + * Initialisation function for the driver called during CDX probing.
> + */
> +typedef int (rte_cdx_probe_t)(struct rte_cdx_driver *, struct rte_cdx_device *);
> +
> +/**
> + * Uninitialisation function for the driver called during hotplugging.
> + */
> +typedef int (rte_cdx_remove_t)(struct rte_cdx_device *);
> +
> +/**
> + * A structure describing a CDX driver.
> + */
> +struct rte_cdx_driver {
> + RTE_TAILQ_ENTRY(rte_cdx_driver) next; /**< Next in list. */
> + struct rte_driver driver; /**< Inherit core driver. */
> + struct rte_cdx_bus *bus; /**< CDX bus reference. */
> + rte_cdx_probe_t *probe; /**< Device probe function. */
> + rte_cdx_remove_t *remove; /**< Device remove function. */
> + const struct rte_cdx_id *id_table; /**< ID table, NULL terminated. */
> + uint32_t drv_flags; /**< Flags RTE_CDX_DRV_*. */
> +};
> +
> +/**
> + * Structure describing the CDX bus
> + */
> +struct rte_cdx_bus {
> + struct rte_bus bus; /**< Inherit the generic class */
> + struct rte_cdx_device_list device_list; /**< List of CDX devices */
> + struct rte_cdx_driver_list driver_list; /**< List of CDX drivers */
As I mentionned above, we can go with a simple:
RTE_TAILQ_HEAD(, rte_cdx_device) device_list;
RTE_TAILQ_HEAD(, rte_cdx_driver) driver_list;
> +};
And I don't see a need to expose the object rte_cdx_bus object in the
drivers API, so the rte_cdx_bus structure can probably move to
private.h.
> +
> +/**
> + * Get Pathname of CDX devices directory.
> + *
> + * @return
> + * sysfs path for CDX devices.
> + */
> +__rte_internal
> +const char *rte_cdx_get_sysfs_path(void);
Hard to tell without a first CDX driver.. will there be a user for this symbol?
> +
> +/**
> + * Map the CDX device resources in user space virtual memory address
> + *
> + * @param dev
> + * A pointer to a rte_cdx_device structure describing the device
> + * to use
> + *
> + * @return
> + * 0 on success, negative on error and positive if no driver
> + * is found for the device.
positive if no driver is found?
This sounds like a copy/paste.
> + */
> +__rte_internal
> +int rte_cdx_map_device(struct rte_cdx_device *dev);
> +
> +/**
> + * Unmap this device
> + *
> + * @param dev
> + * A pointer to a rte_cdx_device structure describing the device
> + * to use
> + */
> +__rte_internal
> +void rte_cdx_unmap_device(struct rte_cdx_device *dev);
> +
> +/**
> + * Register a CDX driver.
> + *
> + * @param driver
> + * A pointer to a rte_cdx_driver structure describing the driver
> + * to be registered.
> + */
> +__rte_internal
> +void rte_cdx_register(struct rte_cdx_driver *driver);
> +
> +/**
> + * Helper for CDX device registration from driver (eth, crypto, raw) instance
> + */
> +#define RTE_PMD_REGISTER_CDX(nm, cdx_drv) \
> + RTE_INIT(cdxinitfn_ ##nm) \
> + {\
> + (cdx_drv).driver.name = RTE_STR(nm);\
> + rte_cdx_register(&cdx_drv); \
> + } \
> + RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
> +
> +/**
> + * Unregister a CDX driver.
> + *
> + * @param driver
> + * A pointer to a rte_cdx_driver structure describing the driver
> + * to be unregistered.
> + */
> +__rte_internal
> +void rte_cdx_unregister(struct rte_cdx_driver *driver);
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#endif /* BUS_CDX_DRIVER_H */
> diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
> new file mode 100644
> index 0000000000..1ddb5a92f7
> --- /dev/null
> +++ b/drivers/bus/cdx/cdx.c
> @@ -0,0 +1,517 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
> + */
> +
> +/*
> + * Architecture Overview
> + * =====================
> + * CDX is a Hardware Architecture designed for AMD FPGA devices. It
> + * consists of sophisticated mechanism for interaction between FPGA,
> + * Firmware and the APUs (Application CPUs).
> + *
> + * Firmware resides on RPU (Realtime CPUs) which interacts with
> + * the FPGA program manager and the APUs. The RPU provides memory-mapped
> + * interface (RPU if) which is used to communicate with APUs.
> + *
> + * The diagram below shows an overview of the AMD CDX architecture:
> + *
> + * +--------------------------------------+
> + * | DPDK |
> + * | DPDK CDX drivers |
> + * | | |
> + * | DPDK AMD CDX bus |
> + * | | |
> + * +-----------------------------|--------+
> + * |
> + * +-----------------------------|--------+
> + * | Application CPUs (APU) | |
> + * | | |
> + * | VFIO CDX driver |
> + * | Linux OS | |
> + * | Linux AMD CDX bus |
> + * | | |
> + * +-----------------------------|--------+
> + * |
> + * |
> + * +------------------------| RPU if |----+
> + * | | |
> + * | V |
> + * | Realtime CPUs (RPU) |
> + * | |
> + * +--------------------------------------+
> + * |
> + * +---------------------|----------------+
> + * | FPGA | |
> + * | +-----------------------+ |
> + * | | | | |
> + * | +-------+ +-------+ +-------+ |
> + * | | dev 1 | | dev 2 | | dev 3 | |
> + * | +-------+ +-------+ +-------+ |
> + * +--------------------------------------+
> + *
> + * The RPU firmware extracts the device information from the loaded FPGA
> + * image and implements a mechanism that allows the APU drivers to
> + * enumerate such devices (device personality and resource details) via
> + * a dedicated communication channel.
> + *
> + * VFIO CDX driver provides the CDX device resources like MMIO and interrupts
> + * to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
> + * driver to discover and initialize the CDX devices for user-space
> + * applications.
> + */
> +
> +#include <string.h>
> +#include <dirent.h>
> +
> +#include <rte_eal_paging.h>
> +#include <rte_errno.h>
> +#include <rte_devargs.h>
> +#include <rte_malloc.h>
> +#include <rte_vfio.h>
> +
> +#include <eal_filesystem.h>
> +
> +#include "bus_cdx_driver.h"
> +#include "cdx_logs.h"
> +#include "private.h"
> +
> +#define SYSFS_CDX_DEVICES "/sys/bus/cdx/devices"
> +#define CDX_BUS_NAME cdx
> +#define CDX_DEV_PREFIX "cdx-"
> +
> +/**
> + * @file
> + * CDX probing using Linux sysfs.
> + */
> +
> +/* Add a device to CDX bus */
> +static void
> +cdx_add_device(struct rte_cdx_device *cdx_dev)
> +{
> + TAILQ_INSERT_TAIL(&rte_cdx_bus.device_list, cdx_dev, next);
> +}
> +
> +static int
> +cdx_get_kernel_driver_by_path(const char *filename, char *driver_name,
> + size_t len)
> +{
> + int count;
> + char path[PATH_MAX];
> + char *name;
> +
> + if (!filename || !driver_name)
> + return -1;
> +
> + count = readlink(filename, path, PATH_MAX);
> + if (count >= PATH_MAX)
> + return -1;
> +
> + /* For device does not have a driver */
> + if (count < 0)
> + return 1;
> +
> + path[count] = '\0';
> +
> + name = strrchr(path, '/');
> + if (name) {
> + strlcpy(driver_name, name + 1, len);
> + return 0;
> + }
> +
> + return -1;
> +}
> +
> +int rte_cdx_map_device(struct rte_cdx_device *dev)
> +{
> + return cdx_vfio_map_resource(dev);
> +}
> +
> +void rte_cdx_unmap_device(struct rte_cdx_device *dev)
> +{
> + cdx_vfio_unmap_resource(dev);
> +}
> +
> +static struct rte_devargs *
> +cdx_devargs_lookup(const char *dev_name)
> +{
> + struct rte_devargs *devargs;
> +
> + RTE_EAL_DEVARGS_FOREACH("cdx", devargs) {
> + if (strcmp(devargs->name, dev_name) == 0)
> + return devargs;
> + }
> + return NULL;
> +}
> +
> +static bool
> +cdx_ignore_device(const char *dev_name)
> +{
> + struct rte_devargs *devargs = cdx_devargs_lookup(dev_name);
> +
> + switch (rte_cdx_bus.bus.conf.scan_mode) {
> + case RTE_BUS_SCAN_ALLOWLIST:
> + if (devargs && devargs->policy == RTE_DEV_ALLOWED)
> + return false;
> + break;
> + case RTE_BUS_SCAN_UNDEFINED:
> + case RTE_BUS_SCAN_BLOCKLIST:
> + if (devargs == NULL || devargs->policy != RTE_DEV_BLOCKED)
> + return false;
> + break;
> + }
> + return true;
> +}
> +
> +/*
> + * Scan one cdx sysfs entry, and fill the devices list from it.
> + * It checks if the CDX device is bound to vfio-cdx driver. In case
> + * the device is vfio bound, it reads the vendor and device id and
> + * stores it for device-driver matching.
> + */
> +static int
> +cdx_scan_one(const char *dirname, const char *dev_name)
> +{
> + char filename[PATH_MAX];
> + struct rte_cdx_device *dev = NULL;
> + char driver[PATH_MAX];
> + unsigned long tmp;
> + char *name = NULL;
> + int ret;
> +
> + dev = calloc(1, sizeof(*dev));
> + if (!dev)
> + return -ENOMEM;
> +
> + name = calloc(1, RTE_DEV_NAME_MAX_LEN);
> + if (!name) {
> + ret = -ENOMEM;
> + goto err;
> + }
> +
> + dev->device.bus = &rte_cdx_bus.bus;
> + memcpy(name, dev_name, RTE_DEV_NAME_MAX_LEN);
> + dev->device.name = name;
If you dynamically allocate the name, you need to keep a reference to
it in the rte_cdx_device object for freeing in a cleanup() later.
The reason is that the rte_device object "name" field is const.
So rather than dynamically allocate to a fixed size, why not have a
name[RTE_DEV_NAME_MAX_LEN] in rte_cdx_device?
> +
> + /* parse driver */
> + snprintf(filename, sizeof(filename), "%s/driver", dirname);
> + ret = cdx_get_kernel_driver_by_path(filename, driver, sizeof(driver));
> + if (ret < 0) {
> + CDX_BUS_ERR("Fail to get kernel driver");
> + ret = -1;
> + goto err;
> + }
> +
> + /*
> + * Check if device is bound to 'vfio-cdx' driver, so that user-space
> + * can gracefully access the device.
> + */
> + if (ret || strcmp(driver, "vfio-cdx")) {
> + ret = 0;
> + goto err;
> + }
> +
> + /* get vendor id */
> + snprintf(filename, sizeof(filename), "%s/vendor", dirname);
> + if (eal_parse_sysfs_value(filename, &tmp) < 0) {
> + ret = -1;
> + goto err;
> + }
> + dev->id.vendor_id = (uint16_t)tmp;
> +
> + /* get device id */
> + snprintf(filename, sizeof(filename), "%s/device", dirname);
> + if (eal_parse_sysfs_value(filename, &tmp) < 0) {
> + free(dev);
> + return -1;
> + }
> + dev->id.device_id = (uint16_t)tmp;
> +
> + cdx_add_device(dev);
> +
> + return 0;
> +
> +err:
> + if (name)
> + free(name);
> + if (dev)
> + free(dev);
free() handles NULL fine, no need to check.
> + return ret;
> +}
> +
> +/*
> + * Scan the content of the CDX bus, and the devices in the devices
> + * list.
> + */
> +static int
> +cdx_scan(void)
> +{
> + struct dirent *e;
> + DIR *dir;
> + char dirname[PATH_MAX];
> +
> + dir = opendir(rte_cdx_get_sysfs_path());
> + if (dir == NULL) {
> + CDX_BUS_ERR("%s(): opendir failed: %s", __func__,
> + strerror(errno));
> + return -1;
> + }
> +
> + while ((e = readdir(dir)) != NULL) {
> + if (e->d_name[0] == '.')
> + continue;
> +
> + if (cdx_ignore_device(e->d_name))
> + continue;
> +
> + snprintf(dirname, sizeof(dirname), "%s/%s",
> + rte_cdx_get_sysfs_path(), e->d_name);
> +
> + if (cdx_scan_one(dirname, e->d_name) < 0)
> + goto error;
> + }
> + closedir(dir);
> + return 0;
> +
> +error:
> + closedir(dir);
> + return -1;
> +}
> +
> +const char *
> +rte_cdx_get_sysfs_path(void)
> +{
> + return SYSFS_CDX_DEVICES;
> +}
> +
> +/* map a particular resource from a file */
> +void *
> +cdx_map_resource(void *requested_addr, int fd, uint64_t offset, size_t size,
> + int additional_flags)
> +{
> + void *mapaddr;
> +
> + /* Map the cdx MMIO memory resource of device */
> + mapaddr = rte_mem_map(requested_addr, size,
> + RTE_PROT_READ | RTE_PROT_WRITE,
> + RTE_MAP_SHARED | additional_flags, fd, offset);
> + if (mapaddr == NULL) {
> + CDX_BUS_ERR("%s(): cannot map resource(%d, %p, 0x%zx, 0x%"PRIx64"): %s (%p)",
> + __func__, fd, requested_addr, size, offset,
> + rte_strerror(rte_errno), mapaddr);
> + }
> + CDX_BUS_DEBUG("CDX MMIO memory mapped at %p", mapaddr);
> +
> + return mapaddr;
> +}
> +
> +/* unmap a particular resource */
> +void
> +cdx_unmap_resource(void *requested_addr, size_t size)
> +{
> + if (requested_addr == NULL)
> + return;
> +
> + /* Unmap the CDX memory resource of device */
> + if (rte_mem_unmap(requested_addr, size)) {
> + CDX_BUS_ERR("%s(): cannot mem unmap(%p, %#zx): %s", __func__,
> + requested_addr, size, rte_strerror(rte_errno));
> + }
> + CDX_BUS_DEBUG("CDX memory unmapped at %p", requested_addr);
> +}
> +/*
> + * Match the CDX Driver and Device using device id and vendor id.
> + */
> +static int
bool
> +cdx_match(const struct rte_cdx_driver *cdx_drv,
> + const struct rte_cdx_device *cdx_dev)
> +{
> + const struct rte_cdx_id *id_table;
> +
> + for (id_table = cdx_drv->id_table; id_table->vendor_id != 0;
> + id_table++) {
> + /* check if device's identifiers match the driver's ones */
> + if (id_table->vendor_id != cdx_dev->id.vendor_id &&
> + id_table->vendor_id != RTE_CDX_ANY_ID)
> + continue;
> + if (id_table->device_id != cdx_dev->id.device_id &&
> + id_table->device_id != RTE_CDX_ANY_ID)
> + continue;
> +
> + return 1;
> + }
> +
> + return 0;
> +}
> +
> +/*
> + * If vendor id and device id match, call the probe() function of the
> + * driver.
> + */
> +static int
> +cdx_probe_one_driver(struct rte_cdx_driver *dr,
> + struct rte_cdx_device *dev)
> +{
> + const char *dev_name = dev->device.name;
> + bool already_probed;
> + int ret;
> +
> + if ((dr == NULL) || (dev == NULL))
> + return -EINVAL;
This symbol can't be called with dr or dev == NULL.
> +
> + /* The device is not blocked; Check if driver supports it */
> + if (!cdx_match(dr, dev))
> + /* Match of device and driver failed */
> + return 1;
> +
> + already_probed = rte_dev_is_probed(&dev->device);
> + if (already_probed) {
> + CDX_BUS_INFO("Device %s is already probed", dev->device.name);
> + return -EEXIST;
> + }
> +
> + CDX_BUS_DEBUG(" probe device %s using driver: %s", dev_name,
> + dr->driver.name);
> +
> + ret = cdx_vfio_map_resource(dev);
> + if (ret != 0) {
> + CDX_BUS_ERR("CDX map device failed: %d", ret);
> + goto error_map_device;
> + }
> +
> + /* call the driver probe() function */
> + ret = dr->probe(dr, dev);
> + if (ret) {
> + CDX_BUS_ERR("Probe CDX driver: %s device: %s failed: %d",
> + dr->driver.name, dev_name, ret);
> + goto error_probe;
> + } else {
> + dev->device.driver = &dr->driver;
> + }
> +
> + return ret;
> +
> +error_probe:
> + cdx_vfio_unmap_resource(dev);
> +error_map_device:
> + return ret;
> +}
> +
> +/*
> + * If vendor/device ID match, call the probe() function of all
> + * registered driver for the given device. Return < 0 if initialization
> + * failed, return 1 if no driver is found for this device.
> + */
> +static int
> +cdx_probe_all_drivers(struct rte_cdx_device *dev)
> +{
> + struct rte_cdx_driver *dr = NULL;
> + int rc = 0;
> +
> + if (dev == NULL)
> + return -EINVAL;
This symbol can't be called with dev == NULL.
> +
> + FOREACH_DRIVER_ON_CDXBUS(dr) {
> + rc = cdx_probe_one_driver(dr, dev);
> + if (rc < 0)
> + /* negative value is an error */
> + return rc;
> + if (rc > 0)
> + /* positive value means driver doesn't support it */
> + continue;
> + return 0;
> + }
> + return 1;
> +}
> +
> +/*
> + * Scan the content of the CDX bus, and call the probe() function for
> + * all registered drivers that have a matching entry in its id_table
> + * for discovered devices.
> + */
> +static int
> +cdx_probe(void)
> +{
> + struct rte_cdx_device *dev = NULL;
> + size_t probed = 0, failed = 0;
> + int ret = 0;
> +
> + FOREACH_DEVICE_ON_CDXBUS(dev) {
> + probed++;
> +
> + ret = cdx_probe_all_drivers(dev);
> + if (ret < 0) {
> + CDX_BUS_ERR("Requested device %s cannot be used",
> + dev->device.name);
> + rte_errno = errno;
> + failed++;
> + ret = 0;
> + }
> + }
> +
> + return (probed && probed == failed) ? -1 : 0;
> +}
> +
> +static int
> +cdx_parse(const char *name, void *addr)
> +{
> + const char **out = addr;
> + int ret;
> +
> + ret = strncmp(name, CDX_DEV_PREFIX, strlen(CDX_DEV_PREFIX));
> +
> + if (ret == 0 && addr)
> + *out = name;
> +
> + return ret;
> +}
> +
> +/* register a driver */
> +void
> +rte_cdx_register(struct rte_cdx_driver *driver)
> +{
> + TAILQ_INSERT_TAIL(&rte_cdx_bus.driver_list, driver, next);
> + driver->bus = &rte_cdx_bus;
> +}
> +
> +/* unregister a driver */
> +void
> +rte_cdx_unregister(struct rte_cdx_driver *driver)
> +{
> + TAILQ_REMOVE(&rte_cdx_bus.driver_list, driver, next);
> + driver->bus = NULL;
> +}
> +
> +static struct rte_device *
> +cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
> + const void *data)
> +{
> + const struct rte_cdx_device *cdx_start;
> + struct rte_cdx_device *cdx_dev;
> +
> + if (start != NULL) {
> + cdx_start = RTE_DEV_TO_CDX_DEV_CONST(start);
> + cdx_dev = TAILQ_NEXT(cdx_start, next);
> + } else {
> + cdx_dev = TAILQ_FIRST(&rte_cdx_bus.device_list);
> + }
> + while (cdx_dev != NULL) {
> + if (cmp(&cdx_dev->device, data) == 0)
> + return &cdx_dev->device;
> + cdx_dev = TAILQ_NEXT(cdx_dev, next);
> + }
> + return NULL;
> +}
> +
> +struct rte_cdx_bus rte_cdx_bus = {
> + .bus = {
> + .scan = cdx_scan,
> + .probe = cdx_probe,
> + .find_device = cdx_find_device,
> + .parse = cdx_parse,
I see neither unplug, nor cleanup op which is strange because this API
provides a way to unregister drivers.
> + },
> + .device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
> + .driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
> +};
> +
> +RTE_REGISTER_BUS(cdx, rte_cdx_bus.bus);
> +RTE_LOG_REGISTER_DEFAULT(cdx_logtype_bus, NOTICE);
[snip]
> diff --git a/drivers/bus/cdx/meson.build b/drivers/bus/cdx/meson.build
> new file mode 100644
> index 0000000000..f2ca104d34
> --- /dev/null
> +++ b/drivers/bus/cdx/meson.build
> @@ -0,0 +1,13 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
> +
> +if not is_linux
> + build = false
> + reason = 'only supported on Linux'
> +endif
> +
> +driver_sdk_headers = files('bus_cdx_driver.h')
> +sources = files(
> + 'cdx.c',
> + 'cdx_vfio.c',
> +)
> diff --git a/drivers/bus/cdx/private.h b/drivers/bus/cdx/private.h
> new file mode 100644
> index 0000000000..c5ce5a46b8
> --- /dev/null
> +++ b/drivers/bus/cdx/private.h
> @@ -0,0 +1,46 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
> + */
> +
> +#ifndef CDX_PRIVATE_H
> +#define CDX_PRIVATE_H
> +
> +#include "bus_cdx_driver.h"
> +
> +extern struct rte_cdx_bus rte_cdx_bus;
This could move to cdx.c.
> +
> +/**
> + * Map a particular resource from a file.
> + *
> + * @param requested_addr
> + * The starting address for the new mapping range.
> + * @param fd
> + * The file descriptor.
> + * @param offset
> + * The offset for the mapping range.
> + * @param size
> + * The size for the mapping range.
> + * @param additional_flags
> + * The additional rte_mem_map() flags for the mapping range.
> + * @return
> + * - On success, the function returns a pointer to the mapped area.
> + * - On error, NULL is returned.
> + */
> +void *cdx_map_resource(void *requested_addr, int fd, uint64_t offset,
> + size_t size, int additional_flags);
> +
> +/**
> + * Unmap a particular resource.
> + *
> + * @param requested_addr
> + * The address for the unmapping range.
> + * @param size
> + * The size for the unmapping range.
> + */
> +void cdx_unmap_resource(void *requested_addr, size_t size);
> +
> +/* map/unmap VFIO resource */
> +int cdx_vfio_map_resource(struct rte_cdx_device *dev);
> +int cdx_vfio_unmap_resource(struct rte_cdx_device *dev);
> +
> +#endif /* CDX_PRIVATE_H */
--
David Marchand
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v5 1/5] bus/cdx: introduce AMD CDX bus
2023-06-01 15:00 ` David Marchand
@ 2023-06-01 18:10 ` Nipun Gupta
2023-06-05 8:04 ` Nipun Gupta
1 sibling, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-06-01 18:10 UTC (permalink / raw)
To: David Marchand
Cc: dev, thomas, hkalra, anatoly.burakov, stephen, ferruh.yigit,
harpreet.anand, nikhil.agarwal
On 6/1/2023 8:30 PM, David Marchand wrote:
<snip>
>> +
>> +struct rte_cdx_bus rte_cdx_bus = {
>> + .bus = {
>> + .scan = cdx_scan,
>> + .probe = cdx_probe,
>> + .find_device = cdx_find_device,
>> + .parse = cdx_parse,
>
> I see neither unplug, nor cleanup op which is strange because this API
> provides a way to unregister drivers.
plug/unplug are part of patch 5/5.
Nipun
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v5 1/5] bus/cdx: introduce AMD CDX bus
2023-06-01 15:00 ` David Marchand
2023-06-01 18:10 ` Nipun Gupta
@ 2023-06-05 8:04 ` Nipun Gupta
1 sibling, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-06-05 8:04 UTC (permalink / raw)
To: David Marchand
Cc: dev, thomas, hkalra, anatoly.burakov, stephen, ferruh.yigit,
harpreet.anand, nikhil.agarwal
On 6/1/2023 8:30 PM, David Marchand wrote:
>
> Hello,
>
> On Thu, May 25, 2023 at 12:08 PM Nipun Gupta <nipun.gupta@amd.com> wrote:
>>
<snip>
>> + * AMD CDX bus interface
>> + */
>> +
>> +#ifdef __cplusplus
>> +extern "C" {
>> +#endif
>
> Do you expect some out of tree drivers written in C++?
> Otherwise, this is unneeded.
As the CDX devices are FPGA based, customers would be using CDX bus
framework and adding CDX drivers on top of it, so we do expect some out
of tree drivers here and would like to keep it.
Thanks,
Nipun
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v5 2/5] bus/cdx: add DMA map and unmap support
2023-05-25 10:08 ` [PATCH v5 0/5] Support AMD CDX bus Nipun Gupta
2023-05-25 10:08 ` [PATCH v5 1/5] bus/cdx: introduce " Nipun Gupta
@ 2023-05-25 10:08 ` Nipun Gupta
2023-06-01 15:07 ` David Marchand
2023-05-25 10:08 ` [PATCH v5 3/5] eal/interrupts: add IRQ count in interrupt handle Nipun Gupta
` (2 subsequent siblings)
4 siblings, 1 reply; 102+ messages in thread
From: Nipun Gupta @ 2023-05-25 10:08 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
AMD CDX bus can use VFIO interface for mapping and unmapping
of DMA addresses in the IOMMU. This change adds the callback
support for map and unmap APIs as well as fetching the IOMMU
class.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
drivers/bus/cdx/cdx.c | 40 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index 1ddb5a92f7..64ea879f3b 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -502,12 +502,52 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
return NULL;
}
+static int
+cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+ struct rte_cdx_device *cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+
+ if (!cdx_dev) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ return rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ (uintptr_t)addr, iova, len);
+}
+
+static int
+cdx_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+ struct rte_cdx_device *cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+
+ if (!cdx_dev) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ return rte_vfio_container_dma_unmap(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ (uintptr_t)addr, iova, len);
+}
+
+static enum rte_iova_mode
+cdx_get_iommu_class(void)
+{
+ if (TAILQ_EMPTY(&rte_cdx_bus.device_list))
+ return RTE_IOVA_DC;
+
+ return RTE_IOVA_VA;
+}
+
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
.scan = cdx_scan,
.probe = cdx_probe,
.find_device = cdx_find_device,
.parse = cdx_parse,
+ .dma_map = cdx_dma_map,
+ .dma_unmap = cdx_dma_unmap,
+ .get_iommu_class = cdx_get_iommu_class,
},
.device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
.driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v5 2/5] bus/cdx: add DMA map and unmap support
2023-05-25 10:08 ` [PATCH v5 2/5] bus/cdx: add DMA map and unmap support Nipun Gupta
@ 2023-06-01 15:07 ` David Marchand
0 siblings, 0 replies; 102+ messages in thread
From: David Marchand @ 2023-06-01 15:07 UTC (permalink / raw)
To: Nipun Gupta
Cc: dev, thomas, hkalra, anatoly.burakov, stephen, ferruh.yigit,
harpreet.anand, nikhil.agarwal
On Thu, May 25, 2023 at 12:09 PM Nipun Gupta <nipun.gupta@amd.com> wrote:
>
> AMD CDX bus can use VFIO interface for mapping and unmapping
> of DMA addresses in the IOMMU. This change adds the callback
> support for map and unmap APIs as well as fetching the IOMMU
> class.
>
> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
> ---
> drivers/bus/cdx/cdx.c | 40 ++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 40 insertions(+)
>
> diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
> index 1ddb5a92f7..64ea879f3b 100644
> --- a/drivers/bus/cdx/cdx.c
> +++ b/drivers/bus/cdx/cdx.c
> @@ -502,12 +502,52 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
> return NULL;
> }
>
> +static int
> +cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
> +{
> + struct rte_cdx_device *cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
> +
> + if (!cdx_dev) {
> + rte_errno = EINVAL;
> + return -1;
> + }
RTE_DEV_TO_CDX_DEV() applies an offset to dev.
Checking dev != NULL is probably a better check.
But on the other hand, calling this dma_map op will be done through
dev->bus->dma_map.
So checking dev is useless too.
> +
> + return rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD,
> + (uintptr_t)addr, iova, len);
> +}
> +
> +static int
> +cdx_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
> +{
> + struct rte_cdx_device *cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
> +
> + if (!cdx_dev) {
> + rte_errno = EINVAL;
> + return -1;
> + }
Idem.
> +
> + return rte_vfio_container_dma_unmap(RTE_VFIO_DEFAULT_CONTAINER_FD,
> + (uintptr_t)addr, iova, len);
> +}
> +
> +static enum rte_iova_mode
> +cdx_get_iommu_class(void)
> +{
> + if (TAILQ_EMPTY(&rte_cdx_bus.device_list))
> + return RTE_IOVA_DC;
> +
> + return RTE_IOVA_VA;
> +}
> +
> struct rte_cdx_bus rte_cdx_bus = {
> .bus = {
> .scan = cdx_scan,
> .probe = cdx_probe,
> .find_device = cdx_find_device,
> .parse = cdx_parse,
> + .dma_map = cdx_dma_map,
> + .dma_unmap = cdx_dma_unmap,
> + .get_iommu_class = cdx_get_iommu_class,
> },
> .device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
> .driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
> --
> 2.17.1
>
--
David Marchand
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v5 3/5] eal/interrupts: add IRQ count in interrupt handle
2023-05-25 10:08 ` [PATCH v5 0/5] Support AMD CDX bus Nipun Gupta
2023-05-25 10:08 ` [PATCH v5 1/5] bus/cdx: introduce " Nipun Gupta
2023-05-25 10:08 ` [PATCH v5 2/5] bus/cdx: add DMA map and unmap support Nipun Gupta
@ 2023-05-25 10:08 ` Nipun Gupta
2023-06-01 15:25 ` David Marchand
2023-06-06 7:18 ` [EXT] " Harman Kalra
2023-05-25 10:08 ` [PATCH v5 4/5] bus/cdx: add support for MSI Nipun Gupta
2023-05-25 10:08 ` [PATCH v5 5/5] bus/cdx: support plug unplug and dev iterator Nipun Gupta
4 siblings, 2 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-05-25 10:08 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
Have total number of IRQ count support in interrupt handle.
In case of VFIO this IRQ count is returned when
VFIO_DEVICE_GET_IRQ_INFO ioctl is invoked. This IRQ_count can
used by the devices to store/provide total number of interrupts
available and to enable or disable these interrupts.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
lib/eal/common/eal_common_interrupts.c | 21 +++++++++++++++++
lib/eal/common/eal_interrupts.h | 1 +
lib/eal/include/rte_interrupts.h | 32 ++++++++++++++++++++++++++
lib/eal/version.map | 2 ++
4 files changed, 56 insertions(+)
diff --git a/lib/eal/common/eal_common_interrupts.c b/lib/eal/common/eal_common_interrupts.c
index 97b64fed58..a0167d9ad4 100644
--- a/lib/eal/common/eal_common_interrupts.c
+++ b/lib/eal/common/eal_common_interrupts.c
@@ -398,6 +398,27 @@ int rte_intr_elist_index_set(struct rte_intr_handle *intr_handle,
return -rte_errno;
}
+int rte_intr_irq_count_set(struct rte_intr_handle *intr_handle,
+ int irq_count)
+{
+ CHECK_VALID_INTR_HANDLE(intr_handle);
+
+ intr_handle->irq_count = irq_count;
+
+ return 0;
+fail:
+ return -rte_errno;
+}
+
+int rte_intr_irq_count_get(const struct rte_intr_handle *intr_handle)
+{
+ CHECK_VALID_INTR_HANDLE(intr_handle);
+
+ return intr_handle->irq_count;
+fail:
+ return -rte_errno;
+}
+
int rte_intr_vec_list_alloc(struct rte_intr_handle *intr_handle,
const char *name, int size)
{
diff --git a/lib/eal/common/eal_interrupts.h b/lib/eal/common/eal_interrupts.h
index 482781b862..eaf8e20187 100644
--- a/lib/eal/common/eal_interrupts.h
+++ b/lib/eal/common/eal_interrupts.h
@@ -16,6 +16,7 @@ struct rte_intr_handle {
};
uint32_t alloc_flags; /**< flags passed at allocation */
enum rte_intr_handle_type type; /**< handle type */
+ uint32_t irq_count; /**< Total IRQ count */
uint32_t max_intr; /**< max interrupt requested */
uint32_t nb_efd; /**< number of available efd(event fd) */
uint8_t efd_counter_size; /**< size of efd counter, used for vdev */
diff --git a/lib/eal/include/rte_interrupts.h b/lib/eal/include/rte_interrupts.h
index 487e3c8875..415d1fcac0 100644
--- a/lib/eal/include/rte_interrupts.h
+++ b/lib/eal/include/rte_interrupts.h
@@ -506,6 +506,38 @@ __rte_internal
int
rte_intr_max_intr_get(const struct rte_intr_handle *intr_handle);
+/**
+ * @internal
+ * Set the IRQ count field of interrupt handle with user
+ * provided IRQ count value.
+ *
+ * @param intr_handle
+ * pointer to the interrupt handle.
+ * @param irq_count
+ * IRQ count
+ *
+ * @return
+ * - On success, zero.
+ * - On failure, a negative value and rte_errno is set.
+ */
+__rte_internal
+int
+rte_intr_irq_count_set(struct rte_intr_handle *intr_handle, int irq_count);
+
+/**
+ * @internal
+ * Returns the IRQ count field of the given interrupt handle instance.
+ *
+ * @param intr_handle
+ * pointer to the interrupt handle.
+ *
+ * @return
+ * - On success, ir count.
+ * - On failure, a negative value and rte_errno is set.
+ */
+__rte_internal
+int rte_intr_irq_count_get(const struct rte_intr_handle *intr_handle);
+
/**
* @internal
* Set the number of event fd field of interrupt handle
diff --git a/lib/eal/version.map b/lib/eal/version.map
index 51a820d829..d31c942e04 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -458,6 +458,8 @@ INTERNAL {
rte_intr_instance_dup;
rte_intr_instance_windows_handle_get;
rte_intr_instance_windows_handle_set;
+ rte_intr_irq_count_get;
+ rte_intr_irq_count_set;
rte_intr_max_intr_get;
rte_intr_max_intr_set;
rte_intr_nb_efd_get;
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v5 3/5] eal/interrupts: add IRQ count in interrupt handle
2023-05-25 10:08 ` [PATCH v5 3/5] eal/interrupts: add IRQ count in interrupt handle Nipun Gupta
@ 2023-06-01 15:25 ` David Marchand
2023-06-01 18:04 ` Nipun Gupta
2023-06-06 7:18 ` [EXT] " Harman Kalra
1 sibling, 1 reply; 102+ messages in thread
From: David Marchand @ 2023-06-01 15:25 UTC (permalink / raw)
To: Nipun Gupta
Cc: dev, thomas, hkalra, anatoly.burakov, stephen, ferruh.yigit,
harpreet.anand, nikhil.agarwal
On Thu, May 25, 2023 at 12:08 PM Nipun Gupta <nipun.gupta@amd.com> wrote:
>
> Have total number of IRQ count support in interrupt handle.
> In case of VFIO this IRQ count is returned when
> VFIO_DEVICE_GET_IRQ_INFO ioctl is invoked. This IRQ_count can
> used by the devices to store/provide total number of interrupts
> available and to enable or disable these interrupts.
>
> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
> ---
> lib/eal/common/eal_common_interrupts.c | 21 +++++++++++++++++
> lib/eal/common/eal_interrupts.h | 1 +
> lib/eal/include/rte_interrupts.h | 32 ++++++++++++++++++++++++++
> lib/eal/version.map | 2 ++
> 4 files changed, 56 insertions(+)
>
> diff --git a/lib/eal/common/eal_common_interrupts.c b/lib/eal/common/eal_common_interrupts.c
> index 97b64fed58..a0167d9ad4 100644
> --- a/lib/eal/common/eal_common_interrupts.c
> +++ b/lib/eal/common/eal_common_interrupts.c
> @@ -398,6 +398,27 @@ int rte_intr_elist_index_set(struct rte_intr_handle *intr_handle,
> return -rte_errno;
> }
>
> +int rte_intr_irq_count_set(struct rte_intr_handle *intr_handle,
> + int irq_count)
> +{
> + CHECK_VALID_INTR_HANDLE(intr_handle);
> +
> + intr_handle->irq_count = irq_count;
> +
> + return 0;
> +fail:
> + return -rte_errno;
> +}
> +
> +int rte_intr_irq_count_get(const struct rte_intr_handle *intr_handle)
> +{
> + CHECK_VALID_INTR_HANDLE(intr_handle);
> +
> + return intr_handle->irq_count;
> +fail:
> + return -rte_errno;
> +}
> +
> int rte_intr_vec_list_alloc(struct rte_intr_handle *intr_handle,
> const char *name, int size)
> {
> diff --git a/lib/eal/common/eal_interrupts.h b/lib/eal/common/eal_interrupts.h
> index 482781b862..eaf8e20187 100644
> --- a/lib/eal/common/eal_interrupts.h
> +++ b/lib/eal/common/eal_interrupts.h
> @@ -16,6 +16,7 @@ struct rte_intr_handle {
> };
> uint32_t alloc_flags; /**< flags passed at allocation */
> enum rte_intr_handle_type type; /**< handle type */
> + uint32_t irq_count; /**< Total IRQ count */
> uint32_t max_intr; /**< max interrupt requested */
> uint32_t nb_efd; /**< number of available efd(event fd) */
> uint8_t efd_counter_size; /**< size of efd counter, used for vdev */
> diff --git a/lib/eal/include/rte_interrupts.h b/lib/eal/include/rte_interrupts.h
> index 487e3c8875..415d1fcac0 100644
> --- a/lib/eal/include/rte_interrupts.h
> +++ b/lib/eal/include/rte_interrupts.h
> @@ -506,6 +506,38 @@ __rte_internal
> int
> rte_intr_max_intr_get(const struct rte_intr_handle *intr_handle);
>
> +/**
> + * @internal
> + * Set the IRQ count field of interrupt handle with user
> + * provided IRQ count value.
I am intrigued by this new notion.
We already have different sizes in the intr_handle, why do we need a new one?
Plus, in the cdx patch using this new API, I see that an fd array is
filled based on nb_efd.
So it seems to me that this new irq_count is just a duplicate of nb_efd.
> + * @param intr_handle
> + * pointer to the interrupt handle.
> + * @param irq_count
> + * IRQ count
> + *
> + * @return
> + * - On success, zero.
> + * - On failure, a negative value and rte_errno is set.
> + */
> +__rte_internal
> +int
> +rte_intr_irq_count_set(struct rte_intr_handle *intr_handle, int irq_count);
> +
--
David Marchand
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v5 3/5] eal/interrupts: add IRQ count in interrupt handle
2023-06-01 15:25 ` David Marchand
@ 2023-06-01 18:04 ` Nipun Gupta
2023-06-01 18:18 ` Gupta, Nipun
0 siblings, 1 reply; 102+ messages in thread
From: Nipun Gupta @ 2023-06-01 18:04 UTC (permalink / raw)
To: David Marchand
Cc: dev, thomas, hkalra, anatoly.burakov, stephen, ferruh.yigit,
harpreet.anand, nikhil.agarwal
Hi David,
On 6/1/2023 8:55 PM, David Marchand wrote:
>
> On Thu, May 25, 2023 at 12:08 PM Nipun Gupta <nipun.gupta@amd.com> wrote:
>>
>> Have total number of IRQ count support in interrupt handle.
>> In case of VFIO this IRQ count is returned when
>> VFIO_DEVICE_GET_IRQ_INFO ioctl is invoked. This IRQ_count can
>> used by the devices to store/provide total number of interrupts
>> available and to enable or disable these interrupts.
>>
>> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
>> Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
>> ---
>> lib/eal/common/eal_common_interrupts.c | 21 +++++++++++++++++
>> lib/eal/common/eal_interrupts.h | 1 +
>> lib/eal/include/rte_interrupts.h | 32 ++++++++++++++++++++++++++
>> lib/eal/version.map | 2 ++
>> 4 files changed, 56 insertions(+)
>>
>> diff --git a/lib/eal/common/eal_common_interrupts.c b/lib/eal/common/eal_common_interrupts.c
>> index 97b64fed58..a0167d9ad4 100644
>> --- a/lib/eal/common/eal_common_interrupts.c
>> +++ b/lib/eal/common/eal_common_interrupts.c
>> @@ -398,6 +398,27 @@ int rte_intr_elist_index_set(struct rte_intr_handle *intr_handle,
>> return -rte_errno;
>> }
>>
>> +int rte_intr_irq_count_set(struct rte_intr_handle *intr_handle,
>> + int irq_count)
>> +{
>> + CHECK_VALID_INTR_HANDLE(intr_handle);
>> +
>> + intr_handle->irq_count = irq_count;
>> +
>> + return 0;
>> +fail:
>> + return -rte_errno;
>> +}
>> +
>> +int rte_intr_irq_count_get(const struct rte_intr_handle *intr_handle)
>> +{
>> + CHECK_VALID_INTR_HANDLE(intr_handle);
>> +
>> + return intr_handle->irq_count;
>> +fail:
>> + return -rte_errno;
>> +}
>> +
>> int rte_intr_vec_list_alloc(struct rte_intr_handle *intr_handle,
>> const char *name, int size)
>> {
>> diff --git a/lib/eal/common/eal_interrupts.h b/lib/eal/common/eal_interrupts.h
>> index 482781b862..eaf8e20187 100644
>> --- a/lib/eal/common/eal_interrupts.h
>> +++ b/lib/eal/common/eal_interrupts.h
>> @@ -16,6 +16,7 @@ struct rte_intr_handle {
>> };
>> uint32_t alloc_flags; /**< flags passed at allocation */
>> enum rte_intr_handle_type type; /**< handle type */
>> + uint32_t irq_count; /**< Total IRQ count */
>> uint32_t max_intr; /**< max interrupt requested */
>> uint32_t nb_efd; /**< number of available efd(event fd) */
>> uint8_t efd_counter_size; /**< size of efd counter, used for vdev */
>> diff --git a/lib/eal/include/rte_interrupts.h b/lib/eal/include/rte_interrupts.h
>> index 487e3c8875..415d1fcac0 100644
>> --- a/lib/eal/include/rte_interrupts.h
>> +++ b/lib/eal/include/rte_interrupts.h
>> @@ -506,6 +506,38 @@ __rte_internal
>> int
>> rte_intr_max_intr_get(const struct rte_intr_handle *intr_handle);
>>
>> +/**
>> + * @internal
>> + * Set the IRQ count field of interrupt handle with user
>> + * provided IRQ count value.
>
> I am intrigued by this new notion.
> We already have different sizes in the intr_handle, why do we need a new one?
>
> Plus, in the cdx patch using this new API, I see that an fd array is
> filled based on nb_efd.
> So it seems to me that this new irq_count is just a duplicate of nb_efd.
The API rte_intr_efd_enable() sets the nb_efd and max_intr to values
provided by the caller (+ NB_OTHER_INTR for max_intr). These values are
dependent on the number of interrupts which are enabled by any DPDK
driver rather than the actual number of interrupts supported or provided
by the Linux driver.
With CDX bus the devices can have interrupts which are not really bound
by the number of queues, and as these devices are programmable (FPGA
based), users are free to implement the interrupts as per the need. I
need to provide the total number of interrupts supported by the device
to the drivers.
nb_intr is also there, which is default initialized by
RTE_MAX_RXTX_INTR_VEC_ID, due to which this does not seem to be best
fit. Though as per the meaning it shall represent the total number of
interrupts supported by the device. Maybe we can remove the default
value and use this as a total interrupt count supported for the device?
w.r.t. comments on the other patches, I will fix them in the next respin.
Regards,
Nipun
>
>
>> + * @param intr_handle
>> + * pointer to the interrupt handle.
>> + * @param irq_count
>> + * IRQ count
>> + *
>> + * @return
>> + * - On success, zero.
>> + * - On failure, a negative value and rte_errno is set.
>> + */
>> +__rte_internal
>> +int
>> +rte_intr_irq_count_set(struct rte_intr_handle *intr_handle, int irq_count);
>> +
>
>
> --
> David Marchand
>
^ permalink raw reply [flat|nested] 102+ messages in thread* RE: [PATCH v5 3/5] eal/interrupts: add IRQ count in interrupt handle
2023-06-01 18:04 ` Nipun Gupta
@ 2023-06-01 18:18 ` Gupta, Nipun
0 siblings, 0 replies; 102+ messages in thread
From: Gupta, Nipun @ 2023-06-01 18:18 UTC (permalink / raw)
To: David Marchand
Cc: dev, thomas, hkalra, anatoly.burakov, stephen, Yigit, Ferruh,
Anand, Harpreet, Agarwal, Nikhil
> -----Original Message-----
> From: Gupta, Nipun
> Sent: Thursday, June 1, 2023 11:35 PM
> To: David Marchand <david.marchand@redhat.com>
> Cc: dev@dpdk.org; thomas@monjalon.net; hkalra@marvell.com;
> anatoly.burakov@intel.com; stephen@networkplumber.org; Yigit, Ferruh
> <Ferruh.Yigit@amd.com>; Anand, Harpreet <harpreet.anand@amd.com>;
> Agarwal, Nikhil <nikhil.agarwal@amd.com>
> Subject: Re: [PATCH v5 3/5] eal/interrupts: add IRQ count in interrupt handle
>
> Hi David,
>
> On 6/1/2023 8:55 PM, David Marchand wrote:
> >
> > On Thu, May 25, 2023 at 12:08 PM Nipun Gupta <nipun.gupta@amd.com>
> wrote:
> >>
> >> Have total number of IRQ count support in interrupt handle.
> >> In case of VFIO this IRQ count is returned when
> >> VFIO_DEVICE_GET_IRQ_INFO ioctl is invoked. This IRQ_count can
> >> used by the devices to store/provide total number of interrupts
> >> available and to enable or disable these interrupts.
> >>
> >> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> >> Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
> >> ---
> >> lib/eal/common/eal_common_interrupts.c | 21 +++++++++++++++++
> >> lib/eal/common/eal_interrupts.h | 1 +
> >> lib/eal/include/rte_interrupts.h | 32 ++++++++++++++++++++++++++
> >> lib/eal/version.map | 2 ++
> >> 4 files changed, 56 insertions(+)
> >>
> >> diff --git a/lib/eal/common/eal_common_interrupts.c
> b/lib/eal/common/eal_common_interrupts.c
> >> index 97b64fed58..a0167d9ad4 100644
> >> --- a/lib/eal/common/eal_common_interrupts.c
> >> +++ b/lib/eal/common/eal_common_interrupts.c
> >> @@ -398,6 +398,27 @@ int rte_intr_elist_index_set(struct rte_intr_handle
> *intr_handle,
> >> return -rte_errno;
> >> }
> >>
> >> +int rte_intr_irq_count_set(struct rte_intr_handle *intr_handle,
> >> + int irq_count)
> >> +{
> >> + CHECK_VALID_INTR_HANDLE(intr_handle);
> >> +
> >> + intr_handle->irq_count = irq_count;
> >> +
> >> + return 0;
> >> +fail:
> >> + return -rte_errno;
> >> +}
> >> +
> >> +int rte_intr_irq_count_get(const struct rte_intr_handle *intr_handle)
> >> +{
> >> + CHECK_VALID_INTR_HANDLE(intr_handle);
> >> +
> >> + return intr_handle->irq_count;
> >> +fail:
> >> + return -rte_errno;
> >> +}
> >> +
> >> int rte_intr_vec_list_alloc(struct rte_intr_handle *intr_handle,
> >> const char *name, int size)
> >> {
> >> diff --git a/lib/eal/common/eal_interrupts.h
> b/lib/eal/common/eal_interrupts.h
> >> index 482781b862..eaf8e20187 100644
> >> --- a/lib/eal/common/eal_interrupts.h
> >> +++ b/lib/eal/common/eal_interrupts.h
> >> @@ -16,6 +16,7 @@ struct rte_intr_handle {
> >> };
> >> uint32_t alloc_flags; /**< flags passed at allocation */
> >> enum rte_intr_handle_type type; /**< handle type */
> >> + uint32_t irq_count; /**< Total IRQ count */
> >> uint32_t max_intr; /**< max interrupt requested */
> >> uint32_t nb_efd; /**< number of available efd(event fd) */
> >> uint8_t efd_counter_size; /**< size of efd counter, used for vdev */
> >> diff --git a/lib/eal/include/rte_interrupts.h b/lib/eal/include/rte_interrupts.h
> >> index 487e3c8875..415d1fcac0 100644
> >> --- a/lib/eal/include/rte_interrupts.h
> >> +++ b/lib/eal/include/rte_interrupts.h
> >> @@ -506,6 +506,38 @@ __rte_internal
> >> int
> >> rte_intr_max_intr_get(const struct rte_intr_handle *intr_handle);
> >>
> >> +/**
> >> + * @internal
> >> + * Set the IRQ count field of interrupt handle with user
> >> + * provided IRQ count value.
> >
> > I am intrigued by this new notion.
> > We already have different sizes in the intr_handle, why do we need a new one?
> >
> > Plus, in the cdx patch using this new API, I see that an fd array is
> > filled based on nb_efd.
> > So it seems to me that this new irq_count is just a duplicate of nb_efd.
>
> The API rte_intr_efd_enable() sets the nb_efd and max_intr to values
> provided by the caller (+ NB_OTHER_INTR for max_intr). These values are
> dependent on the number of interrupts which are enabled by any DPDK
> driver rather than the actual number of interrupts supported or provided
> by the Linux driver.
>
> With CDX bus the devices can have interrupts which are not really bound
> by the number of queues, and as these devices are programmable (FPGA
> based), users are free to implement the interrupts as per the need. I
> need to provide the total number of interrupts supported by the device
> to the drivers.
>
> nb_intr is also there, which is default initialized by
> RTE_MAX_RXTX_INTR_VEC_ID, due to which this does not seem to be best
> fit. Though as per the meaning it shall represent the total number of
> interrupts supported by the device. Maybe we can remove the default
> value and use this as a total interrupt count supported for the device?
On more look nb_intr seems to be more aligned towards the number of efd's
allocated, rather than the total number of interrupts supported on the device.
So, it should have a default value, but yes then I am not sure it this can be used
to represent total number of interrupt count. Please let me know if your views
on this.
Regards,
Nipun
^ permalink raw reply [flat|nested] 102+ messages in thread
* RE: [EXT] [PATCH v5 3/5] eal/interrupts: add IRQ count in interrupt handle
2023-05-25 10:08 ` [PATCH v5 3/5] eal/interrupts: add IRQ count in interrupt handle Nipun Gupta
2023-06-01 15:25 ` David Marchand
@ 2023-06-06 7:18 ` Harman Kalra
2023-06-06 7:27 ` Nipun Gupta
1 sibling, 1 reply; 102+ messages in thread
From: Harman Kalra @ 2023-06-06 7:18 UTC (permalink / raw)
To: Nipun Gupta, dev, thomas, david.marchand, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal
> -----Original Message-----
> From: Nipun Gupta <nipun.gupta@amd.com>
> Sent: Thursday, May 25, 2023 3:38 PM
> To: dev@dpdk.org; thomas@monjalon.net; david.marchand@redhat.com;
> Harman Kalra <hkalra@marvell.com>; anatoly.burakov@intel.com;
> stephen@networkplumber.org
> Cc: ferruh.yigit@amd.com; harpreet.anand@amd.com;
> nikhil.agarwal@amd.com; Nipun Gupta <nipun.gupta@amd.com>
> Subject: [EXT] [PATCH v5 3/5] eal/interrupts: add IRQ count in interrupt
> handle
>
> External Email
>
> ----------------------------------------------------------------------
> Have total number of IRQ count support in interrupt handle.
> In case of VFIO this IRQ count is returned when
> VFIO_DEVICE_GET_IRQ_INFO ioctl is invoked. This IRQ_count can used by
> the devices to store/provide total number of interrupts available and to
> enable or disable these interrupts.
>
Hi Nipun,
We already have "max_intr" field for the same purpose and its respective APIs
plt_intr_max_intr_set()/plt_intr_max_intr_get()
Thanks
Harman
^ permalink raw reply [flat|nested] 102+ messages in thread
* Re: [EXT] [PATCH v5 3/5] eal/interrupts: add IRQ count in interrupt handle
2023-06-06 7:18 ` [EXT] " Harman Kalra
@ 2023-06-06 7:27 ` Nipun Gupta
2023-06-07 5:45 ` Harman Kalra
0 siblings, 1 reply; 102+ messages in thread
From: Nipun Gupta @ 2023-06-06 7:27 UTC (permalink / raw)
To: Harman Kalra, dev, thomas, david.marchand, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal
On 6/6/2023 12:48 PM, Harman Kalra wrote:
>
>> -----Original Message-----
>> From: Nipun Gupta <nipun.gupta@amd.com>
>> Sent: Thursday, May 25, 2023 3:38 PM
>> To: dev@dpdk.org; thomas@monjalon.net; david.marchand@redhat.com;
>> Harman Kalra <hkalra@marvell.com>; anatoly.burakov@intel.com;
>> stephen@networkplumber.org
>> Cc: ferruh.yigit@amd.com; harpreet.anand@amd.com;
>> nikhil.agarwal@amd.com; Nipun Gupta <nipun.gupta@amd.com>
>> Subject: [EXT] [PATCH v5 3/5] eal/interrupts: add IRQ count in interrupt
>> handle
>>
>> External Email
>>
>> ----------------------------------------------------------------------
>> Have total number of IRQ count support in interrupt handle.
>> In case of VFIO this IRQ count is returned when
>> VFIO_DEVICE_GET_IRQ_INFO ioctl is invoked. This IRQ_count can used by
>> the devices to store/provide total number of interrupts available and to
>> enable or disable these interrupts.
>>
>
> Hi Nipun,
>
> We already have "max_intr" field for the same purpose and its respective APIs
> plt_intr_max_intr_set()/plt_intr_max_intr_get()
Hi Harman,
If we have a look into rte_intr_efd_enable() API, 'max_intr' being set
in this API. So once a driver is using the interrupts the 'max_intr'
would be overwritten. 'nb_intr' which is described as "Max vector count"
seems more relevant to me here and I have used 'nb_intr' to have the
total interrupt count available and sent out the updated series for CDX
bus. Please let me know in case you have separate thoughts on this.
Thanks,
Nipun
>
> Thanks
> Harman
^ permalink raw reply [flat|nested] 102+ messages in thread
* RE: [EXT] [PATCH v5 3/5] eal/interrupts: add IRQ count in interrupt handle
2023-06-06 7:27 ` Nipun Gupta
@ 2023-06-07 5:45 ` Harman Kalra
0 siblings, 0 replies; 102+ messages in thread
From: Harman Kalra @ 2023-06-07 5:45 UTC (permalink / raw)
To: Nipun Gupta, dev, thomas, david.marchand, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal
> -----Original Message-----
> From: Nipun Gupta <nipun.gupta@amd.com>
> Sent: Tuesday, June 6, 2023 12:57 PM
> To: Harman Kalra <hkalra@marvell.com>; dev@dpdk.org;
> thomas@monjalon.net; david.marchand@redhat.com;
> anatoly.burakov@intel.com; stephen@networkplumber.org
> Cc: ferruh.yigit@amd.com; harpreet.anand@amd.com;
> nikhil.agarwal@amd.com
> Subject: Re: [EXT] [PATCH v5 3/5] eal/interrupts: add IRQ count in interrupt
> handle
>
>
>
> On 6/6/2023 12:48 PM, Harman Kalra wrote:
> >
> >> -----Original Message-----
> >> From: Nipun Gupta <nipun.gupta@amd.com>
> >> Sent: Thursday, May 25, 2023 3:38 PM
> >> To: dev@dpdk.org; thomas@monjalon.net;
> david.marchand@redhat.com;
> >> Harman Kalra <hkalra@marvell.com>; anatoly.burakov@intel.com;
> >> stephen@networkplumber.org
> >> Cc: ferruh.yigit@amd.com; harpreet.anand@amd.com;
> >> nikhil.agarwal@amd.com; Nipun Gupta <nipun.gupta@amd.com>
> >> Subject: [EXT] [PATCH v5 3/5] eal/interrupts: add IRQ count in
> >> interrupt handle
> >>
> >> External Email
> >>
> >> ---------------------------------------------------------------------
> >> - Have total number of IRQ count support in interrupt handle.
> >> In case of VFIO this IRQ count is returned when
> >> VFIO_DEVICE_GET_IRQ_INFO ioctl is invoked. This IRQ_count can used by
> >> the devices to store/provide total number of interrupts available and
> >> to enable or disable these interrupts.
> >>
> >
> > Hi Nipun,
> >
> > We already have "max_intr" field for the same purpose and its
> > respective APIs
> > plt_intr_max_intr_set()/plt_intr_max_intr_get()
>
> Hi Harman,
>
> If we have a look into rte_intr_efd_enable() API, 'max_intr' being set in this
> API. So once a driver is using the interrupts the 'max_intr'
> would be overwritten. 'nb_intr' which is described as "Max vector count"
> seems more relevant to me here and I have used 'nb_intr' to have the total
> interrupt count available and sent out the updated series for CDX bus. Please
> let me know in case you have separate thoughts on this.
>
Hi Nipun,
Got your point, we are aligned on this.
Just to bring to your notice, In pci_vfio_setup_interrupts(), if irq.count > nb_intr
we are updating nb_intr as part of rte_intr_event_list_update()
Thanks
Harman
> Thanks,
> Nipun
>
> >
> > Thanks
> > Harman
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v5 4/5] bus/cdx: add support for MSI
2023-05-25 10:08 ` [PATCH v5 0/5] Support AMD CDX bus Nipun Gupta
` (2 preceding siblings ...)
2023-05-25 10:08 ` [PATCH v5 3/5] eal/interrupts: add IRQ count in interrupt handle Nipun Gupta
@ 2023-05-25 10:08 ` Nipun Gupta
2023-06-01 15:09 ` David Marchand
2023-05-25 10:08 ` [PATCH v5 5/5] bus/cdx: support plug unplug and dev iterator Nipun Gupta
4 siblings, 1 reply; 102+ messages in thread
From: Nipun Gupta @ 2023-05-25 10:08 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
MSI's are exposed to the devices using VFIO (vfio-cdx). This
patch uses the same to add support for MSI for the devices on
the cdx bus.
A couple of API's have been introduced in the EAL interrupt
framework:
- rte_intr_irq_count_set: This API is used to set the total
interrupts on the interrupt handle. This would be provided
by VFIO (irq.count) for VFIO enabled devices.
- rte_intr_irq_count_get: This API returns the total number
interrupts which were set.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
drivers/bus/cdx/bus_cdx_driver.h | 25 +++++
drivers/bus/cdx/cdx.c | 11 ++
drivers/bus/cdx/cdx_vfio.c | 182 ++++++++++++++++++++++++++++++-
drivers/bus/cdx/version.map | 2 +
4 files changed, 218 insertions(+), 2 deletions(-)
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index f1dce06a16..a8c54d728e 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -67,6 +67,7 @@ struct rte_cdx_device {
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
/**< CDX Memory Resource */
+ struct rte_intr_handle *intr_handle; /**< Interrupt handle */
};
/**
@@ -168,6 +169,30 @@ void rte_cdx_unmap_device(struct rte_cdx_device *dev);
__rte_internal
void rte_cdx_register(struct rte_cdx_driver *driver);
+/**
+ * Enables VFIO Interrupts for CDX bus devices.
+ *
+ * @param intr_handle
+ * Pointer to the interrupt handle.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle);
+
+/**
+ * Disable VFIO Interrupts for CDX bus devices.
+ *
+ * @param intr_handle
+ * Pointer to the interrupt handle.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle);
+
/**
* Helper for CDX device registration from driver (eth, crypto, raw) instance
*/
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index 64ea879f3b..c691c38e04 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -201,6 +201,15 @@ cdx_scan_one(const char *dirname, const char *dev_name)
goto err;
}
+ /* Allocate interrupt instance for cdx device */
+ dev->intr_handle =
+ rte_intr_instance_alloc(RTE_INTR_INSTANCE_F_PRIVATE);
+ if (dev->intr_handle == NULL) {
+ CDX_BUS_ERR("Failed to create interrupt instance for %s\n",
+ dev->device.name);
+ return -ENOMEM;
+ }
+
/*
* Check if device is bound to 'vfio-cdx' driver, so that user-space
* can gracefully access the device.
@@ -391,6 +400,8 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr,
return ret;
error_probe:
+ rte_intr_instance_free(dev->intr_handle);
+ dev->intr_handle = NULL;
cdx_vfio_unmap_resource(dev);
error_map_device:
return ret;
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index e54432de5b..ec6512e158 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -50,6 +50,10 @@ struct mapped_cdx_resource {
/** mapped cdx device list */
TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
+/* IRQ set buffer length for MSI interrupts */
+#define MSI_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \
+ sizeof(int) * (RTE_MAX_RXTX_INTR_VEC_ID + 1))
+
static struct rte_tailq_elem cdx_vfio_tailq = {
.name = "VFIO_CDX_RESOURCE_LIST",
};
@@ -94,6 +98,27 @@ cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
char cdx_addr[PATH_MAX] = {0};
struct mapped_cdx_resource *vfio_res = NULL;
struct mapped_cdx_res_list *vfio_res_list;
+ int ret, vfio_dev_fd;
+
+ if (rte_intr_fd_get(dev->intr_handle) < 0)
+ return -1;
+
+ if (close(rte_intr_fd_get(dev->intr_handle)) < 0) {
+ CDX_BUS_ERR("Error when closing eventfd file descriptor for %s",
+ dev->device.name);
+ return -1;
+ }
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ ret = rte_vfio_release_device(rte_cdx_get_sysfs_path(), dev->device.name,
+ vfio_dev_fd);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot release VFIO device");
+ return ret;
+ }
vfio_res_list =
RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
@@ -116,6 +141,18 @@ cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
{
struct mapped_cdx_resource *vfio_res = NULL;
struct mapped_cdx_res_list *vfio_res_list;
+ int ret, vfio_dev_fd;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ ret = rte_vfio_release_device(rte_cdx_get_sysfs_path(), dev->device.name,
+ vfio_dev_fd);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot release VFIO device");
+ return ret;
+ }
vfio_res_list =
RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
@@ -140,9 +177,80 @@ cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
return cdx_vfio_unmap_resource_secondary(dev);
}
+/* set up interrupt support (but not enable interrupts) */
static int
-cdx_rte_vfio_setup_device(int vfio_dev_fd)
+cdx_vfio_setup_interrupts(struct rte_cdx_device *dev, int vfio_dev_fd,
+ int num_irqs)
{
+ int i, ret;
+
+ if (num_irqs == 0)
+ return 0;
+
+ /* start from MSI interrupt type */
+ for (i = 0; i < num_irqs; i++) {
+ struct vfio_irq_info irq = { .argsz = sizeof(irq) };
+ int fd = -1;
+
+ irq.index = i;
+
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_IRQ_INFO, &irq);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot get VFIO IRQ info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /* if this vector cannot be used with eventfd, fail if we explicitly
+ * specified interrupt type, otherwise continue
+ */
+ if ((irq.flags & VFIO_IRQ_INFO_EVENTFD) == 0)
+ continue;
+
+ if (rte_intr_irq_count_set(dev->intr_handle, irq.count))
+ return -1;
+
+ /* Reallocate the efds and elist fields of intr_handle based
+ * on CDX device MSI size.
+ */
+ if ((uint32_t)rte_intr_nb_intr_get(dev->intr_handle) < irq.count &&
+ rte_intr_event_list_update(dev->intr_handle, irq.count))
+ return -1;
+
+ /* set up an eventfd for interrupts */
+ fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
+ if (fd < 0) {
+ CDX_BUS_ERR("Cannot set up eventfd, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ if (rte_intr_fd_set(dev->intr_handle, fd))
+ return -1;
+
+ /* DPDK CDX bus currently supports only MSI-X */
+ if (rte_intr_type_set(dev->intr_handle, RTE_INTR_HANDLE_VFIO_MSIX))
+ return -1;
+
+ if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
+ return -1;
+
+ return 0;
+ }
+
+ /* if we're here, we haven't found a suitable interrupt vector */
+ return -1;
+}
+
+static int
+cdx_vfio_setup_device(struct rte_cdx_device *dev, int vfio_dev_fd,
+ int num_irqs)
+{
+ if (cdx_vfio_setup_interrupts(dev, vfio_dev_fd, num_irqs) != 0) {
+ CDX_BUS_ERR("Error setting up interrupts!");
+ return -1;
+ }
+
/*
* Reset the device. If the device is not capable of resetting,
* then it updates errno as EINVAL.
@@ -278,6 +386,9 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
struct cdx_map *maps;
int vfio_dev_fd, i, ret;
+ if (rte_intr_fd_set(dev->intr_handle, -1))
+ return -1;
+
ret = rte_vfio_setup_device(rte_cdx_get_sysfs_path(), dev_name,
&vfio_dev_fd, &device_info);
if (ret)
@@ -343,7 +454,7 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
free(reg);
}
- if (cdx_rte_vfio_setup_device(vfio_dev_fd) < 0) {
+ if (cdx_vfio_setup_device(dev, vfio_dev_fd, device_info.num_irqs) < 0) {
CDX_BUS_ERR("%s setup device failed", dev_name);
goto err_vfio_res;
}
@@ -373,6 +484,9 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
const char *dev_name = dev->device.name;
struct cdx_map *maps;
+ if (rte_intr_fd_set(dev->intr_handle, -1))
+ return -1;
+
/* if we're in a secondary process, just find our tailq entry */
TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
if (strcmp(vfio_res->name, dev_name))
@@ -406,6 +520,10 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
dev->mem_resource[i].len = maps[i].size;
}
+ /* we need save vfio_dev_fd, so it can be used during release */
+ if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
+ goto err_vfio_dev_fd;
+
return 0;
err_vfio_dev_fd:
rte_vfio_release_device(rte_cdx_get_sysfs_path(),
@@ -425,3 +543,63 @@ cdx_vfio_map_resource(struct rte_cdx_device *dev)
else
return cdx_vfio_map_resource_secondary(dev);
}
+
+int
+rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle)
+{
+ char irq_set_buf[MSI_IRQ_SET_BUF_LEN];
+ struct vfio_irq_set *irq_set;
+ int *fd_ptr, vfio_dev_fd, i;
+ int ret;
+
+ irq_set = (struct vfio_irq_set *) irq_set_buf;
+ irq_set->count = rte_intr_irq_count_get(intr_handle);
+ irq_set->argsz = sizeof(struct vfio_irq_set) +
+ (sizeof(int) * irq_set->count);
+
+ irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER;
+ irq_set->index = 0;
+ irq_set->start = 0;
+ fd_ptr = (int *) &irq_set->data;
+
+ for (i = 0; i < rte_intr_nb_efd_get(intr_handle); i++)
+ fd_ptr[i] = rte_intr_efds_index_get(intr_handle, i);
+
+ vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
+
+ if (ret) {
+ CDX_BUS_ERR("Error enabling MSI interrupts for fd %d",
+ rte_intr_fd_get(intr_handle));
+ return -1;
+ }
+
+ return 0;
+}
+
+/* disable MSI interrupts */
+int
+rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle)
+{
+ struct vfio_irq_set *irq_set;
+ char irq_set_buf[MSI_IRQ_SET_BUF_LEN];
+ int len, ret, vfio_dev_fd;
+
+ len = sizeof(struct vfio_irq_set);
+
+ irq_set = (struct vfio_irq_set *) irq_set_buf;
+ irq_set->argsz = len;
+ irq_set->count = 0;
+ irq_set->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_TRIGGER;
+ irq_set->index = 0;
+ irq_set->start = 0;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
+
+ if (ret)
+ CDX_BUS_ERR("Error disabling MSI interrupts for fd %d",
+ rte_intr_fd_get(intr_handle));
+
+ return ret;
+}
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
index 957fcab978..2f3d484ebd 100644
--- a/drivers/bus/cdx/version.map
+++ b/drivers/bus/cdx/version.map
@@ -6,6 +6,8 @@ INTERNAL {
rte_cdx_register;
rte_cdx_unmap_device;
rte_cdx_unregister;
+ rte_cdx_vfio_intr_disable;
+ rte_cdx_vfio_intr_enable;
local: *;
};
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v5 4/5] bus/cdx: add support for MSI
2023-05-25 10:08 ` [PATCH v5 4/5] bus/cdx: add support for MSI Nipun Gupta
@ 2023-06-01 15:09 ` David Marchand
2023-06-05 8:05 ` Nipun Gupta
0 siblings, 1 reply; 102+ messages in thread
From: David Marchand @ 2023-06-01 15:09 UTC (permalink / raw)
To: Nipun Gupta
Cc: dev, thomas, hkalra, anatoly.burakov, stephen, ferruh.yigit,
harpreet.anand, nikhil.agarwal
On Thu, May 25, 2023 at 12:09 PM Nipun Gupta <nipun.gupta@amd.com> wrote:
>
> MSI's are exposed to the devices using VFIO (vfio-cdx). This
> patch uses the same to add support for MSI for the devices on
> the cdx bus.
>
> A couple of API's have been introduced in the EAL interrupt
> framework:
> - rte_intr_irq_count_set: This API is used to set the total
> interrupts on the interrupt handle. This would be provided
> by VFIO (irq.count) for VFIO enabled devices.
> - rte_intr_irq_count_get: This API returns the total number
> interrupts which were set.
>
> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
> ---
> drivers/bus/cdx/bus_cdx_driver.h | 25 +++++
> drivers/bus/cdx/cdx.c | 11 ++
> drivers/bus/cdx/cdx_vfio.c | 182 ++++++++++++++++++++++++++++++-
> drivers/bus/cdx/version.map | 2 +
> 4 files changed, 218 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
> index f1dce06a16..a8c54d728e 100644
> --- a/drivers/bus/cdx/bus_cdx_driver.h
> +++ b/drivers/bus/cdx/bus_cdx_driver.h
> @@ -67,6 +67,7 @@ struct rte_cdx_device {
> struct rte_cdx_id id; /**< CDX ID. */
> struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
> /**< CDX Memory Resource */
> + struct rte_intr_handle *intr_handle; /**< Interrupt handle */
> };
>
> /**
> @@ -168,6 +169,30 @@ void rte_cdx_unmap_device(struct rte_cdx_device *dev);
> __rte_internal
> void rte_cdx_register(struct rte_cdx_driver *driver);
>
> +/**
> + * Enables VFIO Interrupts for CDX bus devices.
> + *
> + * @param intr_handle
> + * Pointer to the interrupt handle.
> + *
> + * @return
> + * 0 on success, -1 on error.
> + */
> +__rte_internal
> +int rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle);
> +
> +/**
> + * Disable VFIO Interrupts for CDX bus devices.
> + *
> + * @param intr_handle
> + * Pointer to the interrupt handle.
> + *
> + * @return
> + * 0 on success, -1 on error.
> + */
> +__rte_internal
> +int rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle);
> +
> /**
> * Helper for CDX device registration from driver (eth, crypto, raw) instance
> */
> diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
> index 64ea879f3b..c691c38e04 100644
> --- a/drivers/bus/cdx/cdx.c
> +++ b/drivers/bus/cdx/cdx.c
> @@ -201,6 +201,15 @@ cdx_scan_one(const char *dirname, const char *dev_name)
> goto err;
> }
>
> + /* Allocate interrupt instance for cdx device */
> + dev->intr_handle =
> + rte_intr_instance_alloc(RTE_INTR_INSTANCE_F_PRIVATE);
> + if (dev->intr_handle == NULL) {
> + CDX_BUS_ERR("Failed to create interrupt instance for %s\n",
CDX_BUS_ERR already appends a \n.
> + dev->device.name);
> + return -ENOMEM;
> + }
> +
> /*
> * Check if device is bound to 'vfio-cdx' driver, so that user-space
> * can gracefully access the device.
> @@ -391,6 +400,8 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr,
> return ret;
>
> error_probe:
> + rte_intr_instance_free(dev->intr_handle);
> + dev->intr_handle = NULL;
> cdx_vfio_unmap_resource(dev);
> error_map_device:
> return ret;
> diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
> index e54432de5b..ec6512e158 100644
> --- a/drivers/bus/cdx/cdx_vfio.c
> +++ b/drivers/bus/cdx/cdx_vfio.c
> @@ -50,6 +50,10 @@ struct mapped_cdx_resource {
> /** mapped cdx device list */
> TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
>
> +/* IRQ set buffer length for MSI interrupts */
> +#define MSI_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \
> + sizeof(int) * (RTE_MAX_RXTX_INTR_VEC_ID + 1))
> +
> static struct rte_tailq_elem cdx_vfio_tailq = {
> .name = "VFIO_CDX_RESOURCE_LIST",
> };
> @@ -94,6 +98,27 @@ cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
> char cdx_addr[PATH_MAX] = {0};
> struct mapped_cdx_resource *vfio_res = NULL;
> struct mapped_cdx_res_list *vfio_res_list;
> + int ret, vfio_dev_fd;
> +
> + if (rte_intr_fd_get(dev->intr_handle) < 0)
> + return -1;
> +
> + if (close(rte_intr_fd_get(dev->intr_handle)) < 0) {
> + CDX_BUS_ERR("Error when closing eventfd file descriptor for %s",
> + dev->device.name);
> + return -1;
> + }
> +
> + vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
> + if (vfio_dev_fd < 0)
> + return -1;
> +
> + ret = rte_vfio_release_device(rte_cdx_get_sysfs_path(), dev->device.name,
> + vfio_dev_fd);
> + if (ret < 0) {
> + CDX_BUS_ERR("Cannot release VFIO device");
> + return ret;
> + }
>
> vfio_res_list =
> RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
> @@ -116,6 +141,18 @@ cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
> {
> struct mapped_cdx_resource *vfio_res = NULL;
> struct mapped_cdx_res_list *vfio_res_list;
> + int ret, vfio_dev_fd;
> +
> + vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
> + if (vfio_dev_fd < 0)
> + return -1;
> +
> + ret = rte_vfio_release_device(rte_cdx_get_sysfs_path(), dev->device.name,
> + vfio_dev_fd);
> + if (ret < 0) {
> + CDX_BUS_ERR("Cannot release VFIO device");
> + return ret;
> + }
>
> vfio_res_list =
> RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
> @@ -140,9 +177,80 @@ cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
> return cdx_vfio_unmap_resource_secondary(dev);
> }
>
> +/* set up interrupt support (but not enable interrupts) */
> static int
> -cdx_rte_vfio_setup_device(int vfio_dev_fd)
> +cdx_vfio_setup_interrupts(struct rte_cdx_device *dev, int vfio_dev_fd,
Why rename this function?
> + int num_irqs)
> {
> + int i, ret;
> +
> + if (num_irqs == 0)
> + return 0;
> +
> + /* start from MSI interrupt type */
> + for (i = 0; i < num_irqs; i++) {
[snip]
--
David Marchand
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v5 4/5] bus/cdx: add support for MSI
2023-06-01 15:09 ` David Marchand
@ 2023-06-05 8:05 ` Nipun Gupta
0 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-06-05 8:05 UTC (permalink / raw)
To: David Marchand
Cc: dev, thomas, hkalra, anatoly.burakov, stephen, ferruh.yigit,
harpreet.anand, nikhil.agarwal
On 6/1/2023 8:39 PM, David Marchand wrote:
>
> On Thu, May 25, 2023 at 12:09 PM Nipun Gupta <nipun.gupta@amd.com> wrote:
>>
<snip>
>> @@ -116,6 +141,18 @@ cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
>> {
>> struct mapped_cdx_resource *vfio_res = NULL;
>> struct mapped_cdx_res_list *vfio_res_list;
>> + int ret, vfio_dev_fd;
>> +
>> + vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
>> + if (vfio_dev_fd < 0)
>> + return -1;
>> +
>> + ret = rte_vfio_release_device(rte_cdx_get_sysfs_path(), dev->device.name,
>> + vfio_dev_fd);
>> + if (ret < 0) {
>> + CDX_BUS_ERR("Cannot release VFIO device");
>> + return ret;
>> + }
>>
>> vfio_res_list =
>> RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
>> @@ -140,9 +177,80 @@ cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
>> return cdx_vfio_unmap_resource_secondary(dev);
>> }
>>
>> +/* set up interrupt support (but not enable interrupts) */
>> static int
>> -cdx_rte_vfio_setup_device(int vfio_dev_fd)
>> +cdx_vfio_setup_interrupts(struct rte_cdx_device *dev, int vfio_dev_fd,
>
> Why rename this function?
This is actually not renamed. git commit is showing this, because
parameters of cdx_rte_vfio_setup_device() has been updated and a new API
has also been added cdx_vfio_setup_interrupts(). It shows parameter
update as "+cdx_rte_vfio_setup_device()" just after this API code.
Thanks,
Nipun
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v5 5/5] bus/cdx: support plug unplug and dev iterator
2023-05-25 10:08 ` [PATCH v5 0/5] Support AMD CDX bus Nipun Gupta
` (3 preceding siblings ...)
2023-05-25 10:08 ` [PATCH v5 4/5] bus/cdx: add support for MSI Nipun Gupta
@ 2023-05-25 10:08 ` Nipun Gupta
4 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-05-25 10:08 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
This change adds support for plugging and unplugging
CDX devices on AMD CDX bus. Also, CDX dev iterator support
has been added for the CDX bus.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
drivers/bus/cdx/bus_cdx_driver.h | 1 +
drivers/bus/cdx/cdx.c | 122 +++++++++++++++++++++++++++++++
2 files changed, 123 insertions(+)
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index a8c54d728e..1e86f55788 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -64,6 +64,7 @@ struct rte_cdx_id {
struct rte_cdx_device {
RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
struct rte_device device; /**< Inherit core device */
+ struct rte_cdx_driver *driver; /**< CDX driver used in probing */
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
/**< CDX Memory Resource */
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index c691c38e04..b3d07fd638 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -66,6 +66,7 @@
#include <rte_eal_paging.h>
#include <rte_errno.h>
#include <rte_devargs.h>
+#include <rte_kvargs.h>
#include <rte_malloc.h>
#include <rte_vfio.h>
@@ -79,6 +80,15 @@
#define CDX_BUS_NAME cdx
#define CDX_DEV_PREFIX "cdx-"
+enum cdx_params {
+ RTE_CDX_PARAM_NAME,
+};
+
+static const char * const cdx_params_keys[] = {
+ [RTE_CDX_PARAM_NAME] = "name",
+ NULL,
+};
+
/**
* @file
* CDX probing using Linux sysfs.
@@ -396,6 +406,7 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr,
} else {
dev->device.driver = &dr->driver;
}
+ dev->driver = dr;
return ret;
@@ -513,6 +524,71 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
return NULL;
}
+/* Remove a device from CDX bus */
+static void
+cdx_remove_device(struct rte_cdx_device *cdx_dev)
+{
+ TAILQ_REMOVE(&rte_cdx_bus.device_list, cdx_dev, next);
+}
+
+/*
+ * If vendor/device ID match, call the remove() function of the
+ * driver.
+ */
+static int
+cdx_detach_dev(struct rte_cdx_device *dev)
+{
+ struct rte_cdx_driver *dr;
+ int ret = 0;
+
+ if (dev == NULL)
+ return -EINVAL;
+
+ dr = dev->driver;
+
+ CDX_BUS_DEBUG("detach device %s using driver: %s",
+ dev->device.name, dr->driver.name);
+
+ if (dr->remove) {
+ ret = dr->remove(dev);
+ if (ret < 0)
+ return ret;
+ }
+
+ /* clear driver structure */
+ dev->driver = NULL;
+ dev->device.driver = NULL;
+
+ rte_cdx_unmap_device(dev);
+
+ rte_intr_instance_free(dev->intr_handle);
+ dev->intr_handle = NULL;
+
+ return 0;
+}
+
+static int
+cdx_plug(struct rte_device *dev)
+{
+ return cdx_probe_all_drivers(RTE_DEV_TO_CDX_DEV(dev));
+}
+
+static int
+cdx_unplug(struct rte_device *dev)
+{
+ struct rte_cdx_device *cdx_dev;
+ int ret;
+
+ cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+ ret = cdx_detach_dev(cdx_dev);
+ if (ret == 0) {
+ cdx_remove_device(cdx_dev);
+ rte_devargs_remove(dev->devargs);
+ free(cdx_dev);
+ }
+ return ret;
+}
+
static int
cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
{
@@ -550,15 +626,61 @@ cdx_get_iommu_class(void)
return RTE_IOVA_VA;
}
+static int
+cdx_dev_match(const struct rte_device *dev,
+ const void *_kvlist)
+{
+ const struct rte_kvargs *kvlist = _kvlist;
+ const char *key = cdx_params_keys[RTE_CDX_PARAM_NAME];
+ const char *name;
+
+ /* no kvlist arg, all devices match */
+ if (kvlist == NULL)
+ return 0;
+
+ /* if key is present in kvlist and does not match, filter device */
+ name = rte_kvargs_get(kvlist, key);
+ if (name != NULL && strcmp(name, dev->name))
+ return -1;
+
+ return 0;
+}
+
+static void *
+cdx_dev_iterate(const void *start,
+ const char *str,
+ const struct rte_dev_iterator *it __rte_unused)
+{
+ rte_bus_find_device_t find_device;
+ struct rte_kvargs *kvargs = NULL;
+ struct rte_device *dev;
+
+ if (str != NULL) {
+ kvargs = rte_kvargs_parse(str, cdx_params_keys);
+ if (kvargs == NULL) {
+ CDX_BUS_ERR("cannot parse argument list %s", str);
+ rte_errno = EINVAL;
+ return NULL;
+ }
+ }
+ find_device = rte_cdx_bus.bus.find_device;
+ dev = find_device(start, cdx_dev_match, kvargs);
+ rte_kvargs_free(kvargs);
+ return dev;
+}
+
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
.scan = cdx_scan,
.probe = cdx_probe,
.find_device = cdx_find_device,
+ .plug = cdx_plug,
+ .unplug = cdx_unplug,
.parse = cdx_parse,
.dma_map = cdx_dma_map,
.dma_unmap = cdx_dma_unmap,
.get_iommu_class = cdx_get_iommu_class,
+ .dev_iterate = cdx_dev_iterate,
},
.device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
.driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v6 0/4] Support AMD CDX bus
2023-01-24 14:07 [RFC PATCH 0/6] add support for CDX bus Nipun Gupta
` (10 preceding siblings ...)
2023-05-25 10:08 ` [PATCH v5 0/5] Support AMD CDX bus Nipun Gupta
@ 2023-06-05 13:26 ` Nipun Gupta
2023-06-05 13:26 ` [PATCH v6 1/4] bus/cdx: introduce " Nipun Gupta
` (3 more replies)
2023-06-06 10:02 ` [PATCH v7 0/4] Support AMD CDX bus Nipun Gupta
2023-06-07 4:24 ` [PATCH v8 0/4] Support AMD CDX bus Nipun Gupta
13 siblings, 4 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-06-05 13:26 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
The CDX devices are memory mapped on system bus for embedded CPUs.
It uses sysfs interface and the vfio-cdx driver to discover
and initialize the CDX devices.
The CDX bus and VFIO support is available at Xilinx/AMD open source tree:
https://github.com/Xilinx/linux-xlnx (drivers/cdx/ and drivers/vfio/cdx)
Linux AMD CDX bus patches has been added into linux tree:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/cdx
VFIO patches are also submitted in upstream:
https://www.spinics.net/lists/kvm/msg310623.html
CDX is a Hardware Architecture designed for AMD FPGA devices. It
consists of mechanism for interaction between FPGA, Firmware and
the APUs (Application CPUs).
Firmware resides on RPU (Realtime CPUs) which interacts with
the FPGA program manager and the APUs. The RPU provides memory-mapped
interface (RPU if) which is used to communicate with APUs.
VFIO CDX driver provides the CDX device resources like MMIO and interrupts
to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
driver to discover and initialize the CDX devices for user-space
applications.
Changes v5->v6
- Removed newly introduce irq_count parameter, and use num_intr
instead.
- Removed rte_cdx_sysfs_path(), and directly used the macro for
the sysfs path.
- Removed few unrequired param checks in multiple APIs
- add name variable in rte_cdx_device
- moved rte_cdx_bus from bus_cdx_driver.h to private.h
- Other code cleanup
Changes v4->v5
- Split patch and have a separate commit for eal interrupt field
(irq_count) and corresponding set/get APIs (rte_intr_irq_count_set,
rte_intr_irq_count_get)
- Renamed "CDX bus" to "AMD CDX bus" in documentation/commit headings
- Removed unnecessary headers
Changes v3->v4:
- removed platform specific patch (adding config for ARM CDX)
from this series
Changes v2->v3:
- merged cdx bus compilation enablement in the first patch
- fixed issue reported by check-git-log.sh
- updated release notes
- updated offset to uint64_t instead of off_t in cdx_map_resource
Changes v1->v2:
- Moved file rte_cdx_bus.h to internal bus_cdx_driver.h
and added this file to deivce_cdx_headers
- Moved cdx.h to private.h
- Removed rte_ prefix from the static symbols in .c files.
Changes RFC->v1:
- Marked few API's as internal which were not required
to be provided to user.
Nipun Gupta (4):
bus/cdx: introduce AMD CDX bus
bus/cdx: add DMA map and unmap support
bus/cdx: add support for MSI
bus/cdx: support plug unplug and dev iterator
MAINTAINERS | 5 +
doc/guides/rel_notes/release_23_07.rst | 6 +
drivers/bus/cdx/bus_cdx_driver.h | 201 ++++++++
drivers/bus/cdx/cdx.c | 666 +++++++++++++++++++++++++
drivers/bus/cdx/cdx_logs.h | 37 ++
drivers/bus/cdx/cdx_vfio.c | 598 ++++++++++++++++++++++
drivers/bus/cdx/meson.build | 13 +
drivers/bus/cdx/private.h | 53 ++
drivers/bus/cdx/version.map | 13 +
drivers/bus/meson.build | 1 +
10 files changed, 1593 insertions(+)
create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
create mode 100644 drivers/bus/cdx/cdx.c
create mode 100644 drivers/bus/cdx/cdx_logs.h
create mode 100644 drivers/bus/cdx/cdx_vfio.c
create mode 100644 drivers/bus/cdx/meson.build
create mode 100644 drivers/bus/cdx/private.h
create mode 100644 drivers/bus/cdx/version.map
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v6 1/4] bus/cdx: introduce AMD CDX bus
2023-06-05 13:26 ` [PATCH v6 0/4] Support AMD CDX bus Nipun Gupta
@ 2023-06-05 13:26 ` Nipun Gupta
2023-06-06 8:55 ` Thomas Monjalon
2023-06-05 13:26 ` [PATCH v6 2/4] bus/cdx: add DMA map and unmap support Nipun Gupta
` (2 subsequent siblings)
3 siblings, 1 reply; 102+ messages in thread
From: Nipun Gupta @ 2023-06-05 13:26 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
AMD CDX bus supports multiple type of devices, which can be
exposed to user-space via vfio-cdx.
vfio-cdx provides the MMIO IO_MEMORY regions as well as the
DMA interface for the device (IOMMU).
This support aims to enable the DPDK to support the cdx
devices in user-space using VFIO interface.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
MAINTAINERS | 5 +
doc/guides/rel_notes/release_23_07.rst | 6 +
drivers/bus/cdx/bus_cdx_driver.h | 175 +++++++++
drivers/bus/cdx/cdx.c | 503 +++++++++++++++++++++++++
drivers/bus/cdx/cdx_logs.h | 37 ++
drivers/bus/cdx/cdx_vfio.c | 426 +++++++++++++++++++++
drivers/bus/cdx/meson.build | 13 +
drivers/bus/cdx/private.h | 53 +++
drivers/bus/cdx/version.map | 11 +
drivers/bus/meson.build | 1 +
10 files changed, 1230 insertions(+)
create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
create mode 100644 drivers/bus/cdx/cdx.c
create mode 100644 drivers/bus/cdx/cdx_logs.h
create mode 100644 drivers/bus/cdx/cdx_vfio.c
create mode 100644 drivers/bus/cdx/meson.build
create mode 100644 drivers/bus/cdx/private.h
create mode 100644 drivers/bus/cdx/version.map
diff --git a/MAINTAINERS b/MAINTAINERS
index a5219926ab..c4b2b3565b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -569,6 +569,11 @@ M: Parav Pandit <parav@nvidia.com>
M: Xueming Li <xuemingl@nvidia.com>
F: drivers/bus/auxiliary/
+AMD CDX bus driver
+M: Nipun Gupta <nipun.gupta@amd.com>
+M: Nikhil Agarwal <nikhil.agarwal@amd.com>
+F: drivers/bus/cdx/
+
Intel FPGA bus
M: Rosen Xu <rosen.xu@intel.com>
F: drivers/bus/ifpga/
diff --git a/doc/guides/rel_notes/release_23_07.rst b/doc/guides/rel_notes/release_23_07.rst
index a9b1293689..7c6bb2b894 100644
--- a/doc/guides/rel_notes/release_23_07.rst
+++ b/doc/guides/rel_notes/release_23_07.rst
@@ -55,6 +55,12 @@ New Features
Also, make sure to start the actual text at the margin.
=======================================================
+* **Added AMD CDX bus support.**
+
+ CDX bus driver has been added to support AMD CDX bus, which operates
+ on FPGA based CDX devices. The CDX devices are memory mapped on system
+ bus for embedded CPUs.
+
Removed Items
-------------
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
new file mode 100644
index 0000000000..dbbd1bbdf2
--- /dev/null
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -0,0 +1,175 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef BUS_CDX_DRIVER_H
+#define BUS_CDX_DRIVER_H
+
+/**
+ * @file
+ * AMD CDX bus interface
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdlib.h>
+#include <inttypes.h>
+
+#include <bus_driver.h>
+#include <dev_driver.h>
+#include <rte_interrupts.h>
+
+/* Forward declarations */
+struct rte_cdx_device;
+struct rte_cdx_driver;
+struct rte_cdx_bus;
+
+#define CDX_BUS_DEVICES_PATH "/sys/bus/cdx/devices"
+
+#define CDX_MAX_RESOURCE 4
+
+/** Any CDX device identifier (vendor, device) */
+#define RTE_CDX_ANY_ID (0xffff)
+
+#define RTE_PMD_REGISTER_CDX_TABLE(name, table) \
+static const char DRV_EXP_TAG(name, cdx_tbl_export)[] __rte_used = \
+RTE_STR(table)
+
+/**
+ * A structure describing an ID for a CDX driver. Each driver provides a
+ * table of these IDs for each device that it supports.
+ */
+struct rte_cdx_id {
+ uint16_t vendor_id; /**< Vendor ID. */
+ uint16_t device_id; /**< Device ID. */
+};
+
+/**
+ * A structure describing a CDX device.
+ */
+struct rte_cdx_device {
+ RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
+ struct rte_device device; /**< Inherit core device */
+ char name[RTE_DEV_NAME_MAX_LEN]; /**< Device name */
+ struct rte_cdx_id id; /**< CDX ID. */
+ struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
+ /**< CDX Memory Resource */
+};
+
+/**
+ * @internal
+ * Helper macro for drivers that need to convert to struct rte_cdx_device.
+ */
+#define RTE_DEV_TO_CDX_DEV(ptr) \
+ container_of(ptr, struct rte_cdx_device, device)
+
+#define RTE_DEV_TO_CDX_DEV_CONST(ptr) \
+ container_of(ptr, const struct rte_cdx_device, device)
+
+#define RTE_ETH_DEV_TO_CDX_DEV(eth_dev) RTE_DEV_TO_CDX_DEV((eth_dev)->device)
+
+#ifdef __cplusplus
+/** C++ macro used to help building up tables of device IDs */
+#define RTE_CDX_DEVICE(vend, dev) \
+ (vend), \
+ (dev)
+#else
+/** Macro used to help building up tables of device IDs */
+#define RTE_CDX_DEVICE(vend, dev) \
+ .vendor_id = (vend), \
+ .device_id = (dev)
+#endif
+
+/**
+ * Initialisation function for the driver called during CDX probing.
+ */
+typedef int (rte_cdx_probe_t)(struct rte_cdx_driver *, struct rte_cdx_device *);
+
+/**
+ * Uninitialisation function for the driver called during hotplugging.
+ */
+typedef int (rte_cdx_remove_t)(struct rte_cdx_device *);
+
+/**
+ * A structure describing a CDX driver.
+ */
+struct rte_cdx_driver {
+ RTE_TAILQ_ENTRY(rte_cdx_driver) next; /**< Next in list. */
+ struct rte_driver driver; /**< Inherit core driver. */
+ struct rte_cdx_bus *bus; /**< CDX bus reference. */
+ rte_cdx_probe_t *probe; /**< Device probe function. */
+ rte_cdx_remove_t *remove; /**< Device remove function. */
+ const struct rte_cdx_id *id_table; /**< ID table, NULL terminated. */
+ uint32_t drv_flags; /**< Flags RTE_CDX_DRV_*. */
+};
+
+/**
+ * Get Pathname of CDX devices directory.
+ *
+ * @return
+ * sysfs path for CDX devices.
+ */
+__rte_internal
+const char *rte_cdx_get_sysfs_path(void);
+
+/**
+ * Map the CDX device resources in user space virtual memory address
+ *
+ * @param dev
+ * A pointer to a rte_cdx_device structure describing the device
+ * to use
+ *
+ * @return
+ * 0 on success, <0 on error.
+ */
+__rte_internal
+int rte_cdx_map_device(struct rte_cdx_device *dev);
+
+/**
+ * Unmap this device
+ *
+ * @param dev
+ * A pointer to a rte_cdx_device structure describing the device
+ * to use
+ */
+__rte_internal
+void rte_cdx_unmap_device(struct rte_cdx_device *dev);
+
+/**
+ * Register a CDX driver.
+ *
+ * @param driver
+ * A pointer to a rte_cdx_driver structure describing the driver
+ * to be registered.
+ */
+__rte_internal
+void rte_cdx_register(struct rte_cdx_driver *driver);
+
+/**
+ * Helper for CDX device registration from driver (eth, crypto, raw) instance
+ */
+#define RTE_PMD_REGISTER_CDX(nm, cdx_drv) \
+ RTE_INIT(cdxinitfn_ ##nm) \
+ {\
+ (cdx_drv).driver.name = RTE_STR(nm);\
+ rte_cdx_register(&cdx_drv); \
+ } \
+ RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+/**
+ * Unregister a CDX driver.
+ *
+ * @param driver
+ * A pointer to a rte_cdx_driver structure describing the driver
+ * to be unregistered.
+ */
+__rte_internal
+void rte_cdx_unregister(struct rte_cdx_driver *driver);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* BUS_CDX_DRIVER_H */
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
new file mode 100644
index 0000000000..9386cb7a0a
--- /dev/null
+++ b/drivers/bus/cdx/cdx.c
@@ -0,0 +1,503 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+/*
+ * Architecture Overview
+ * =====================
+ * CDX is a Hardware Architecture designed for AMD FPGA devices. It
+ * consists of sophisticated mechanism for interaction between FPGA,
+ * Firmware and the APUs (Application CPUs).
+ *
+ * Firmware resides on RPU (Realtime CPUs) which interacts with
+ * the FPGA program manager and the APUs. The RPU provides memory-mapped
+ * interface (RPU if) which is used to communicate with APUs.
+ *
+ * The diagram below shows an overview of the AMD CDX architecture:
+ *
+ * +--------------------------------------+
+ * | DPDK |
+ * | DPDK CDX drivers |
+ * | | |
+ * | DPDK AMD CDX bus |
+ * | | |
+ * +-----------------------------|--------+
+ * |
+ * +-----------------------------|--------+
+ * | Application CPUs (APU) | |
+ * | | |
+ * | VFIO CDX driver |
+ * | Linux OS | |
+ * | Linux AMD CDX bus |
+ * | | |
+ * +-----------------------------|--------+
+ * |
+ * |
+ * +------------------------| RPU if |----+
+ * | | |
+ * | V |
+ * | Realtime CPUs (RPU) |
+ * | |
+ * +--------------------------------------+
+ * |
+ * +---------------------|----------------+
+ * | FPGA | |
+ * | +-----------------------+ |
+ * | | | | |
+ * | +-------+ +-------+ +-------+ |
+ * | | dev 1 | | dev 2 | | dev 3 | |
+ * | +-------+ +-------+ +-------+ |
+ * +--------------------------------------+
+ *
+ * The RPU firmware extracts the device information from the loaded FPGA
+ * image and implements a mechanism that allows the APU drivers to
+ * enumerate such devices (device personality and resource details) via
+ * a dedicated communication channel.
+ *
+ * VFIO CDX driver provides the CDX device resources like MMIO and interrupts
+ * to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
+ * driver to discover and initialize the CDX devices for user-space
+ * applications.
+ */
+
+/**
+ * @file
+ * CDX probing using Linux sysfs.
+ */
+
+#include <string.h>
+#include <dirent.h>
+
+#include <rte_eal_paging.h>
+#include <rte_errno.h>
+#include <rte_devargs.h>
+#include <rte_malloc.h>
+#include <rte_vfio.h>
+
+#include <eal_filesystem.h>
+
+#include "bus_cdx_driver.h"
+#include "cdx_logs.h"
+#include "private.h"
+
+#define CDX_BUS_NAME cdx
+#define CDX_DEV_PREFIX "cdx-"
+
+/* CDX Bus iterators */
+#define FOREACH_DEVICE_ON_CDXBUS(p) \
+ RTE_TAILQ_FOREACH(p, &rte_cdx_bus.device_list, next)
+
+#define FOREACH_DRIVER_ON_CDXBUS(p) \
+ RTE_TAILQ_FOREACH(p, &rte_cdx_bus.driver_list, next)
+
+struct rte_cdx_bus rte_cdx_bus;
+
+/* Add a device to CDX bus */
+static void
+cdx_add_device(struct rte_cdx_device *cdx_dev)
+{
+ TAILQ_INSERT_TAIL(&rte_cdx_bus.device_list, cdx_dev, next);
+}
+
+static int
+cdx_get_kernel_driver_by_path(const char *filename, char *driver_name,
+ size_t len)
+{
+ int count;
+ char path[PATH_MAX];
+ char *name;
+
+ if (!filename || !driver_name)
+ return -1;
+
+ count = readlink(filename, path, PATH_MAX);
+ if (count >= PATH_MAX)
+ return -1;
+
+ /* For device does not have a driver */
+ if (count < 0)
+ return 1;
+
+ path[count] = '\0';
+
+ name = strrchr(path, '/');
+ if (name) {
+ strlcpy(driver_name, name + 1, len);
+ return 0;
+ }
+
+ return -1;
+}
+
+int rte_cdx_map_device(struct rte_cdx_device *dev)
+{
+ return cdx_vfio_map_resource(dev);
+}
+
+void rte_cdx_unmap_device(struct rte_cdx_device *dev)
+{
+ cdx_vfio_unmap_resource(dev);
+}
+
+static struct rte_devargs *
+cdx_devargs_lookup(const char *dev_name)
+{
+ struct rte_devargs *devargs;
+
+ RTE_EAL_DEVARGS_FOREACH("cdx", devargs) {
+ if (strcmp(devargs->name, dev_name) == 0)
+ return devargs;
+ }
+ return NULL;
+}
+
+static bool
+cdx_ignore_device(const char *dev_name)
+{
+ struct rte_devargs *devargs = cdx_devargs_lookup(dev_name);
+
+ switch (rte_cdx_bus.bus.conf.scan_mode) {
+ case RTE_BUS_SCAN_ALLOWLIST:
+ if (devargs && devargs->policy == RTE_DEV_ALLOWED)
+ return false;
+ break;
+ case RTE_BUS_SCAN_UNDEFINED:
+ case RTE_BUS_SCAN_BLOCKLIST:
+ if (devargs == NULL || devargs->policy != RTE_DEV_BLOCKED)
+ return false;
+ break;
+ }
+ return true;
+}
+
+/*
+ * Scan one cdx sysfs entry, and fill the devices list from it.
+ * It checks if the CDX device is bound to vfio-cdx driver. In case
+ * the device is vfio bound, it reads the vendor and device id and
+ * stores it for device-driver matching.
+ */
+static int
+cdx_scan_one(const char *dirname, const char *dev_name)
+{
+ char filename[PATH_MAX];
+ struct rte_cdx_device *dev = NULL;
+ char driver[PATH_MAX];
+ unsigned long tmp;
+ int ret;
+
+ dev = calloc(1, sizeof(*dev));
+ if (!dev)
+ return -ENOMEM;
+
+ dev->device.bus = &rte_cdx_bus.bus;
+ memcpy(dev->name, dev_name, RTE_DEV_NAME_MAX_LEN);
+ dev->device.name = dev->name;
+
+ /* parse driver */
+ snprintf(filename, sizeof(filename), "%s/driver", dirname);
+ ret = cdx_get_kernel_driver_by_path(filename, driver, sizeof(driver));
+ if (ret < 0) {
+ CDX_BUS_ERR("Fail to get kernel driver");
+ ret = -1;
+ goto err;
+ }
+
+ /*
+ * Check if device is bound to 'vfio-cdx' driver, so that user-space
+ * can gracefully access the device.
+ */
+ if (ret || strcmp(driver, "vfio-cdx")) {
+ ret = 0;
+ goto err;
+ }
+
+ /* get vendor id */
+ snprintf(filename, sizeof(filename), "%s/vendor", dirname);
+ if (eal_parse_sysfs_value(filename, &tmp) < 0) {
+ ret = -1;
+ goto err;
+ }
+ dev->id.vendor_id = (uint16_t)tmp;
+
+ /* get device id */
+ snprintf(filename, sizeof(filename), "%s/device", dirname);
+ if (eal_parse_sysfs_value(filename, &tmp) < 0) {
+ free(dev);
+ return -1;
+ }
+ dev->id.device_id = (uint16_t)tmp;
+
+ cdx_add_device(dev);
+
+ return 0;
+
+err:
+ free(dev);
+ return ret;
+}
+
+/*
+ * Scan the content of the CDX bus, and the devices in the devices
+ * list.
+ */
+static int
+cdx_scan(void)
+{
+ struct dirent *e;
+ DIR *dir;
+ char dirname[PATH_MAX];
+
+ dir = opendir(CDX_BUS_DEVICES_PATH);
+ if (dir == NULL) {
+ CDX_BUS_ERR("%s(): opendir failed: %s", __func__,
+ strerror(errno));
+ return -1;
+ }
+
+ while ((e = readdir(dir)) != NULL) {
+ if (e->d_name[0] == '.')
+ continue;
+
+ if (cdx_ignore_device(e->d_name))
+ continue;
+
+ snprintf(dirname, sizeof(dirname), "%s/%s",
+ CDX_BUS_DEVICES_PATH, e->d_name);
+
+ if (cdx_scan_one(dirname, e->d_name) < 0)
+ goto error;
+ }
+ closedir(dir);
+ return 0;
+
+error:
+ closedir(dir);
+ return -1;
+}
+
+/* map a particular resource from a file */
+void *
+cdx_map_resource(void *requested_addr, int fd, uint64_t offset, size_t size,
+ int additional_flags)
+{
+ void *mapaddr;
+
+ /* Map the cdx MMIO memory resource of device */
+ mapaddr = rte_mem_map(requested_addr, size,
+ RTE_PROT_READ | RTE_PROT_WRITE,
+ RTE_MAP_SHARED | additional_flags, fd, offset);
+ if (mapaddr == NULL) {
+ CDX_BUS_ERR("%s(): cannot map resource(%d, %p, 0x%zx, 0x%"PRIx64"): %s (%p)",
+ __func__, fd, requested_addr, size, offset,
+ rte_strerror(rte_errno), mapaddr);
+ }
+ CDX_BUS_DEBUG("CDX MMIO memory mapped at %p", mapaddr);
+
+ return mapaddr;
+}
+
+/* unmap a particular resource */
+void
+cdx_unmap_resource(void *requested_addr, size_t size)
+{
+ if (requested_addr == NULL)
+ return;
+
+ /* Unmap the CDX memory resource of device */
+ if (rte_mem_unmap(requested_addr, size)) {
+ CDX_BUS_ERR("%s(): cannot mem unmap(%p, %#zx): %s", __func__,
+ requested_addr, size, rte_strerror(rte_errno));
+ }
+ CDX_BUS_DEBUG("CDX memory unmapped at %p", requested_addr);
+}
+/*
+ * Match the CDX Driver and Device using device id and vendor id.
+ */
+static bool
+cdx_match(const struct rte_cdx_driver *cdx_drv,
+ const struct rte_cdx_device *cdx_dev)
+{
+ const struct rte_cdx_id *id_table;
+
+ for (id_table = cdx_drv->id_table; id_table->vendor_id != 0;
+ id_table++) {
+ /* check if device's identifiers match the driver's ones */
+ if (id_table->vendor_id != cdx_dev->id.vendor_id &&
+ id_table->vendor_id != RTE_CDX_ANY_ID)
+ continue;
+ if (id_table->device_id != cdx_dev->id.device_id &&
+ id_table->device_id != RTE_CDX_ANY_ID)
+ continue;
+
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * If vendor id and device id match, call the probe() function of the
+ * driver.
+ */
+static int
+cdx_probe_one_driver(struct rte_cdx_driver *dr,
+ struct rte_cdx_device *dev)
+{
+ const char *dev_name = dev->name;
+ bool already_probed;
+ int ret;
+
+ /* The device is not blocked; Check if driver supports it */
+ if (!cdx_match(dr, dev))
+ /* Match of device and driver failed */
+ return 1;
+
+ already_probed = rte_dev_is_probed(&dev->device);
+ if (already_probed) {
+ CDX_BUS_INFO("Device %s is already probed", dev_name);
+ return -EEXIST;
+ }
+
+ CDX_BUS_DEBUG(" probe device %s using driver: %s", dev_name,
+ dr->driver.name);
+
+ ret = cdx_vfio_map_resource(dev);
+ if (ret != 0) {
+ CDX_BUS_ERR("CDX map device failed: %d", ret);
+ goto error_map_device;
+ }
+
+ /* call the driver probe() function */
+ ret = dr->probe(dr, dev);
+ if (ret) {
+ CDX_BUS_ERR("Probe CDX driver: %s device: %s failed: %d",
+ dr->driver.name, dev_name, ret);
+ goto error_probe;
+ } else {
+ dev->device.driver = &dr->driver;
+ }
+
+ return ret;
+
+error_probe:
+ cdx_vfio_unmap_resource(dev);
+error_map_device:
+ return ret;
+}
+
+/*
+ * If vendor/device ID match, call the probe() function of all
+ * registered driver for the given device. Return < 0 if initialization
+ * failed, return 1 if no driver is found for this device.
+ */
+static int
+cdx_probe_all_drivers(struct rte_cdx_device *dev)
+{
+ struct rte_cdx_driver *dr = NULL;
+ int rc = 0;
+
+ FOREACH_DRIVER_ON_CDXBUS(dr) {
+ rc = cdx_probe_one_driver(dr, dev);
+ if (rc < 0)
+ /* negative value is an error */
+ return rc;
+ if (rc > 0)
+ /* positive value means driver doesn't support it */
+ continue;
+ return 0;
+ }
+ return 1;
+}
+
+/*
+ * Scan the content of the CDX bus, and call the probe() function for
+ * all registered drivers that have a matching entry in its id_table
+ * for discovered devices.
+ */
+static int
+cdx_probe(void)
+{
+ struct rte_cdx_device *dev = NULL;
+ size_t probed = 0, failed = 0;
+ int ret = 0;
+
+ FOREACH_DEVICE_ON_CDXBUS(dev) {
+ probed++;
+
+ ret = cdx_probe_all_drivers(dev);
+ if (ret < 0) {
+ CDX_BUS_ERR("Requested device %s cannot be used",
+ dev->name);
+ rte_errno = errno;
+ failed++;
+ ret = 0;
+ }
+ }
+
+ return (probed && probed == failed) ? -1 : 0;
+}
+
+static int
+cdx_parse(const char *name, void *addr)
+{
+ const char **out = addr;
+ int ret;
+
+ ret = strncmp(name, CDX_DEV_PREFIX, strlen(CDX_DEV_PREFIX));
+
+ if (ret == 0 && addr)
+ *out = name;
+
+ return ret;
+}
+
+/* register a driver */
+void
+rte_cdx_register(struct rte_cdx_driver *driver)
+{
+ TAILQ_INSERT_TAIL(&rte_cdx_bus.driver_list, driver, next);
+ driver->bus = &rte_cdx_bus;
+}
+
+/* unregister a driver */
+void
+rte_cdx_unregister(struct rte_cdx_driver *driver)
+{
+ TAILQ_REMOVE(&rte_cdx_bus.driver_list, driver, next);
+ driver->bus = NULL;
+}
+
+static struct rte_device *
+cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
+ const void *data)
+{
+ const struct rte_cdx_device *cdx_start;
+ struct rte_cdx_device *cdx_dev;
+
+ if (start != NULL) {
+ cdx_start = RTE_DEV_TO_CDX_DEV_CONST(start);
+ cdx_dev = TAILQ_NEXT(cdx_start, next);
+ } else {
+ cdx_dev = TAILQ_FIRST(&rte_cdx_bus.device_list);
+ }
+ while (cdx_dev != NULL) {
+ if (cmp(&cdx_dev->device, data) == 0)
+ return &cdx_dev->device;
+ cdx_dev = TAILQ_NEXT(cdx_dev, next);
+ }
+ return NULL;
+}
+
+struct rte_cdx_bus rte_cdx_bus = {
+ .bus = {
+ .scan = cdx_scan,
+ .probe = cdx_probe,
+ .find_device = cdx_find_device,
+ .parse = cdx_parse,
+ },
+ .device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
+ .driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
+};
+
+RTE_REGISTER_BUS(cdx, rte_cdx_bus.bus);
+RTE_LOG_REGISTER_DEFAULT(cdx_logtype_bus, NOTICE);
diff --git a/drivers/bus/cdx/cdx_logs.h b/drivers/bus/cdx/cdx_logs.h
new file mode 100644
index 0000000000..929c83f1e6
--- /dev/null
+++ b/drivers/bus/cdx/cdx_logs.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef CDX_LOGS_H
+#define CDX_LOGS_H
+
+extern int cdx_logtype_bus;
+
+#define CDX_BUS_LOG(level, fmt, args...) \
+ rte_log(RTE_LOG_ ## level, cdx_logtype_bus, "cdx: " fmt "\n", \
+ ##args)
+
+/* Debug logs are with Function names */
+#define CDX_BUS_DEBUG(fmt, args...) \
+ rte_log(RTE_LOG_DEBUG, cdx_logtype_bus, "cdx: %s(): " fmt "\n", \
+ __func__, ##args)
+
+#define CDX_BUS_INFO(fmt, args...) \
+ CDX_BUS_LOG(INFO, fmt, ## args)
+#define CDX_BUS_ERR(fmt, args...) \
+ CDX_BUS_LOG(ERR, fmt, ## args)
+#define CDX_BUS_WARN(fmt, args...) \
+ CDX_BUS_LOG(WARNING, fmt, ## args)
+
+/* DP Logs, toggled out at compile time if level lower than current level */
+#define CDX_BUS_DP_LOG(level, fmt, args...) \
+ RTE_LOG_DP(level, PMD, fmt, ## args)
+
+#define CDX_BUS_DP_DEBUG(fmt, args...) \
+ CDX_BUS_DP_LOG(DEBUG, fmt, ## args)
+#define CDX_BUS_DP_INFO(fmt, args...) \
+ CDX_BUS_DP_LOG(INFO, fmt, ## args)
+#define CDX_BUS_DP_WARN(fmt, args...) \
+ CDX_BUS_DP_LOG(WARNING, fmt, ## args)
+
+#endif /* CDX_LOGS_H */
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
new file mode 100644
index 0000000000..c545bcd9f5
--- /dev/null
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -0,0 +1,426 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+/**
+ * @file
+ * CDX probing using Linux VFIO.
+ *
+ * This code tries to determine if the CDX device is bound to VFIO driver,
+ * and initialize it (map MMIO regions, set up interrupts) if that's the case.
+ *
+ */
+
+#include <fcntl.h>
+#include <unistd.h>
+#include <sys/eventfd.h>
+#include <sys/socket.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <rte_eal_paging.h>
+#include <rte_malloc.h>
+#include <rte_vfio.h>
+
+#include "bus_cdx_driver.h"
+#include "cdx_logs.h"
+#include "private.h"
+
+/**
+ * A structure describing a CDX mapping.
+ */
+struct cdx_map {
+ void *addr;
+ char *path;
+ uint64_t offset;
+ uint64_t size;
+};
+
+/**
+ * A structure describing a mapped CDX resource.
+ * For multi-process we need to reproduce all CDX mappings in secondary
+ * processes, so save them in a tailq.
+ */
+struct mapped_cdx_resource {
+ TAILQ_ENTRY(mapped_cdx_resource) next;
+ char name[RTE_DEV_NAME_MAX_LEN]; /**< CDX device name */
+ char path[PATH_MAX];
+ int nb_maps;
+ struct cdx_map maps[CDX_MAX_RESOURCE];
+};
+
+/** mapped cdx device list */
+TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
+
+static struct rte_tailq_elem cdx_vfio_tailq = {
+ .name = "VFIO_CDX_RESOURCE_LIST",
+};
+EAL_REGISTER_TAILQ(cdx_vfio_tailq)
+
+static struct mapped_cdx_resource *
+cdx_vfio_find_and_unmap_resource(struct mapped_cdx_res_list *vfio_res_list,
+ struct rte_cdx_device *dev)
+{
+ struct mapped_cdx_resource *vfio_res = NULL;
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+ int i;
+
+ /* Get vfio_res */
+ TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+ if (strcmp(vfio_res->name, dev_name))
+ continue;
+ break;
+ }
+
+ if (vfio_res == NULL)
+ return vfio_res;
+
+ CDX_BUS_INFO("Releasing CDX mapped resource for %s", dev_name);
+
+ maps = vfio_res->maps;
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ if (maps[i].addr) {
+ CDX_BUS_DEBUG("Calling cdx_unmap_resource for %s at %p",
+ dev_name, maps[i].addr);
+ cdx_unmap_resource(maps[i].addr, maps[i].size);
+ }
+ }
+
+ return vfio_res;
+}
+
+static int
+cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
+{
+ char cdx_addr[PATH_MAX] = {0};
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list;
+
+ vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ vfio_res = cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for cdx device!",
+ cdx_addr);
+ return -1;
+ }
+
+ TAILQ_REMOVE(vfio_res_list, vfio_res, next);
+ rte_free(vfio_res);
+ return 0;
+}
+
+static int
+cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
+{
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list;
+
+ vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ vfio_res = cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for CDX device!",
+ dev->device.name);
+ return -1;
+ }
+
+ return 0;
+}
+
+int
+cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
+{
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return cdx_vfio_unmap_resource_primary(dev);
+ else
+ return cdx_vfio_unmap_resource_secondary(dev);
+}
+
+static int
+cdx_rte_vfio_setup_device(int vfio_dev_fd)
+{
+ /*
+ * Reset the device. If the device is not capable of resetting,
+ * then it updates errno as EINVAL.
+ */
+ if (ioctl(vfio_dev_fd, VFIO_DEVICE_RESET) && errno != EINVAL) {
+ CDX_BUS_ERR("Unable to reset device! Error: %d (%s)", errno,
+ strerror(errno));
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+cdx_vfio_mmap_resource(int vfio_dev_fd, struct mapped_cdx_resource *vfio_res,
+ int index, int additional_flags)
+{
+ struct cdx_map *map = &vfio_res->maps[index];
+ void *vaddr;
+
+ if (map->size == 0) {
+ CDX_BUS_DEBUG("map size is 0, skip region %d", index);
+ return 0;
+ }
+
+ /* reserve the address using an inaccessible mapping */
+ vaddr = mmap(map->addr, map->size, 0, MAP_PRIVATE |
+ MAP_ANONYMOUS | additional_flags, -1, 0);
+ if (vaddr != MAP_FAILED) {
+ void *map_addr = NULL;
+
+ if (map->size) {
+ /* actual map of first part */
+ map_addr = cdx_map_resource(vaddr, vfio_dev_fd,
+ map->offset, map->size,
+ RTE_MAP_FORCE_ADDRESS);
+ }
+
+ if (map_addr == NULL) {
+ munmap(vaddr, map->size);
+ vaddr = MAP_FAILED;
+ CDX_BUS_ERR("Failed to map cdx MMIO region %d", index);
+ return -1;
+ }
+ } else {
+ CDX_BUS_ERR("Failed to create inaccessible mapping for MMIO region %d",
+ index);
+ return -1;
+ }
+
+ map->addr = vaddr;
+ return 0;
+}
+
+/*
+ * region info may contain capability headers, so we need to keep reallocating
+ * the memory until we match allocated memory size with argsz.
+ */
+static int
+cdx_vfio_get_region_info(int vfio_dev_fd, struct vfio_region_info **info,
+ int region)
+{
+ struct vfio_region_info *ri;
+ size_t argsz = sizeof(*ri);
+ int ret;
+
+ ri = malloc(sizeof(*ri));
+ if (ri == NULL) {
+ CDX_BUS_ERR("Cannot allocate memory for VFIO region info");
+ return -1;
+ }
+again:
+ memset(ri, 0, argsz);
+ ri->argsz = argsz;
+ ri->index = region;
+
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_REGION_INFO, ri);
+ if (ret < 0) {
+ free(ri);
+ return ret;
+ }
+ if (ri->argsz != argsz) {
+ struct vfio_region_info *tmp;
+
+ argsz = ri->argsz;
+ tmp = realloc(ri, argsz);
+
+ if (tmp == NULL) {
+ /* realloc failed but the ri is still there */
+ free(ri);
+ CDX_BUS_ERR("Cannot reallocate memory for VFIO region info");
+ return -1;
+ }
+ ri = tmp;
+ goto again;
+ }
+ *info = ri;
+
+ return 0;
+}
+
+static int
+find_max_end_va(const struct rte_memseg_list *msl, void *arg)
+{
+ size_t sz = msl->len;
+ void *end_va = RTE_PTR_ADD(msl->base_va, sz);
+ void **max_va = arg;
+
+ if (*max_va < end_va)
+ *max_va = end_va;
+ return 0;
+}
+
+static void *
+cdx_find_max_end_va(void)
+{
+ void *va = NULL;
+
+ rte_memseg_list_walk(find_max_end_va, &va);
+ return va;
+}
+
+static int
+cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ char cdx_addr[PATH_MAX] = {0};
+ static void *cdx_map_addr;
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+ int vfio_dev_fd, i, ret;
+
+ ret = rte_vfio_setup_device(CDX_BUS_DEVICES_PATH, dev_name,
+ &vfio_dev_fd, &device_info);
+ if (ret)
+ return ret;
+
+ /* allocate vfio_res and get region info */
+ vfio_res = rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0);
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("Cannot store VFIO mmap details");
+ goto err_vfio_dev_fd;
+ }
+ memcpy(vfio_res->name, dev_name, RTE_DEV_NAME_MAX_LEN);
+
+ /* get number of registers */
+ vfio_res->nb_maps = device_info.num_regions;
+
+ /* map memory regions */
+ maps = vfio_res->maps;
+
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ struct vfio_region_info *reg = NULL;
+ void *vaddr;
+
+ ret = cdx_vfio_get_region_info(vfio_dev_fd, ®, i);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s cannot get device region info error %i (%s)",
+ dev_name, errno, strerror(errno));
+ goto err_vfio_res;
+ }
+
+ /* skip non-mmappable regions */
+ if ((reg->flags & VFIO_REGION_INFO_FLAG_MMAP) == 0) {
+ free(reg);
+ continue;
+ }
+
+ /* try mapping somewhere close to the end of hugepages */
+ if (cdx_map_addr == NULL)
+ cdx_map_addr = cdx_find_max_end_va();
+
+ vaddr = cdx_map_addr;
+ cdx_map_addr = RTE_PTR_ADD(vaddr, (size_t)reg->size);
+
+ cdx_map_addr = RTE_PTR_ALIGN(cdx_map_addr,
+ sysconf(_SC_PAGE_SIZE));
+
+ maps[i].addr = vaddr;
+ maps[i].offset = reg->offset;
+ maps[i].size = reg->size;
+ maps[i].path = NULL; /* vfio doesn't have per-resource paths */
+
+ ret = cdx_vfio_mmap_resource(vfio_dev_fd, vfio_res, i, 0);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s mapping region %i failed: %s",
+ cdx_addr, i, strerror(errno));
+ free(reg);
+ goto err_vfio_res;
+ }
+
+ dev->mem_resource[i].addr = maps[i].addr;
+ dev->mem_resource[i].len = maps[i].size;
+
+ free(reg);
+ }
+
+ if (cdx_rte_vfio_setup_device(vfio_dev_fd) < 0) {
+ CDX_BUS_ERR("%s setup device failed", dev_name);
+ goto err_vfio_res;
+ }
+
+ TAILQ_INSERT_TAIL(vfio_res_list, vfio_res, next);
+
+ return 0;
+err_vfio_res:
+ cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+ rte_free(vfio_res);
+err_vfio_dev_fd:
+ rte_vfio_release_device(CDX_BUS_DEVICES_PATH, dev_name, vfio_dev_fd);
+ return -1;
+}
+
+static int
+cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ char cdx_addr[PATH_MAX] = {0};
+ int vfio_dev_fd;
+ int i, ret;
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+
+ /* if we're in a secondary process, just find our tailq entry */
+ TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+ if (strcmp(vfio_res->name, dev_name))
+ continue;
+ break;
+ }
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for cdx device!",
+ dev_name);
+ return -1;
+ }
+
+ ret = rte_vfio_setup_device(CDX_BUS_DEVICES_PATH, dev_name,
+ &vfio_dev_fd, &device_info);
+ if (ret)
+ return ret;
+
+ /* map MMIO regions */
+ maps = vfio_res->maps;
+
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ ret = cdx_vfio_mmap_resource(vfio_dev_fd, vfio_res, i, MAP_FIXED);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s mapping MMIO region %i failed: %s",
+ dev_name, i, strerror(errno));
+ goto err_vfio_dev_fd;
+ }
+
+ dev->mem_resource[i].addr = maps[i].addr;
+ dev->mem_resource[i].len = maps[i].size;
+ }
+
+ return 0;
+err_vfio_dev_fd:
+ rte_vfio_release_device(CDX_BUS_DEVICES_PATH, cdx_addr, vfio_dev_fd);
+ return -1;
+}
+
+/*
+ * map the CDX resources of a CDX device in virtual memory (VFIO version).
+ * primary and secondary processes follow almost exactly the same path
+ */
+int
+cdx_vfio_map_resource(struct rte_cdx_device *dev)
+{
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return cdx_vfio_map_resource_primary(dev);
+ else
+ return cdx_vfio_map_resource_secondary(dev);
+}
diff --git a/drivers/bus/cdx/meson.build b/drivers/bus/cdx/meson.build
new file mode 100644
index 0000000000..f2ca104d34
--- /dev/null
+++ b/drivers/bus/cdx/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+
+if not is_linux
+ build = false
+ reason = 'only supported on Linux'
+endif
+
+driver_sdk_headers = files('bus_cdx_driver.h')
+sources = files(
+ 'cdx.c',
+ 'cdx_vfio.c',
+)
diff --git a/drivers/bus/cdx/private.h b/drivers/bus/cdx/private.h
new file mode 100644
index 0000000000..c89f098acd
--- /dev/null
+++ b/drivers/bus/cdx/private.h
@@ -0,0 +1,53 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef CDX_PRIVATE_H
+#define CDX_PRIVATE_H
+
+#include "bus_cdx_driver.h"
+
+/**
+ * Structure describing the CDX bus
+ */
+struct rte_cdx_bus {
+ struct rte_bus bus; /**< Inherit the generic class */
+ RTE_TAILQ_HEAD(, rte_cdx_device) device_list; /**< List of CDX devices */
+ RTE_TAILQ_HEAD(, rte_cdx_driver) driver_list; /**< List of CDX drivers */
+};
+
+/**
+ * Map a particular resource from a file.
+ *
+ * @param requested_addr
+ * The starting address for the new mapping range.
+ * @param fd
+ * The file descriptor.
+ * @param offset
+ * The offset for the mapping range.
+ * @param size
+ * The size for the mapping range.
+ * @param additional_flags
+ * The additional rte_mem_map() flags for the mapping range.
+ * @return
+ * - On success, the function returns a pointer to the mapped area.
+ * - On error, NULL is returned.
+ */
+void *cdx_map_resource(void *requested_addr, int fd, uint64_t offset,
+ size_t size, int additional_flags);
+
+/**
+ * Unmap a particular resource.
+ *
+ * @param requested_addr
+ * The address for the unmapping range.
+ * @param size
+ * The size for the unmapping range.
+ */
+void cdx_unmap_resource(void *requested_addr, size_t size);
+
+/* map/unmap VFIO resource */
+int cdx_vfio_map_resource(struct rte_cdx_device *dev);
+int cdx_vfio_unmap_resource(struct rte_cdx_device *dev);
+
+#endif /* CDX_PRIVATE_H */
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
new file mode 100644
index 0000000000..957fcab978
--- /dev/null
+++ b/drivers/bus/cdx/version.map
@@ -0,0 +1,11 @@
+INTERNAL {
+ global:
+
+ rte_cdx_get_sysfs_path;
+ rte_cdx_map_device;
+ rte_cdx_register;
+ rte_cdx_unmap_device;
+ rte_cdx_unregister;
+
+ local: *;
+};
diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
index 6d2520c543..a78b4283bf 100644
--- a/drivers/bus/meson.build
+++ b/drivers/bus/meson.build
@@ -3,6 +3,7 @@
drivers = [
'auxiliary',
+ 'cdx',
'dpaa',
'fslmc',
'ifpga',
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v6 1/4] bus/cdx: introduce AMD CDX bus
2023-06-05 13:26 ` [PATCH v6 1/4] bus/cdx: introduce " Nipun Gupta
@ 2023-06-06 8:55 ` Thomas Monjalon
0 siblings, 0 replies; 102+ messages in thread
From: Thomas Monjalon @ 2023-06-06 8:55 UTC (permalink / raw)
To: Nipun Gupta, Nipun Gupta
Cc: dev, david.marchand, hkalra, anatoly.burakov, stephen,
ferruh.yigit, harpreet.anand, nikhil.agarwal
05/06/2023 15:26, Nipun Gupta:
> +INTERNAL {
> + global:
> +
> + rte_cdx_get_sysfs_path;
Why exporting this function to the drivers?
Another minor comment: please make sure all doxygen comments are ended with a dot.
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v6 2/4] bus/cdx: add DMA map and unmap support
2023-06-05 13:26 ` [PATCH v6 0/4] Support AMD CDX bus Nipun Gupta
2023-06-05 13:26 ` [PATCH v6 1/4] bus/cdx: introduce " Nipun Gupta
@ 2023-06-05 13:26 ` Nipun Gupta
2023-06-05 13:26 ` [PATCH v6 3/4] bus/cdx: add support for MSI Nipun Gupta
2023-06-05 13:26 ` [PATCH v6 4/4] bus/cdx: support plug unplug and dev iterator Nipun Gupta
3 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-06-05 13:26 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
AMD CDX bus can use VFIO interface for mapping and unmapping
of DMA addresses in the IOMMU. This change adds the callback
support for map and unmap APIs as well as fetching the IOMMU
class.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
drivers/bus/cdx/cdx.c | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index 9386cb7a0a..9ad8f73424 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -488,12 +488,42 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
return NULL;
}
+static int
+cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+ RTE_SET_USED(dev);
+
+ return rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ (uintptr_t)addr, iova, len);
+}
+
+static int
+cdx_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+ RTE_SET_USED(dev);
+
+ return rte_vfio_container_dma_unmap(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ (uintptr_t)addr, iova, len);
+}
+
+static enum rte_iova_mode
+cdx_get_iommu_class(void)
+{
+ if (TAILQ_EMPTY(&rte_cdx_bus.device_list))
+ return RTE_IOVA_DC;
+
+ return RTE_IOVA_VA;
+}
+
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
.scan = cdx_scan,
.probe = cdx_probe,
.find_device = cdx_find_device,
.parse = cdx_parse,
+ .dma_map = cdx_dma_map,
+ .dma_unmap = cdx_dma_unmap,
+ .get_iommu_class = cdx_get_iommu_class,
},
.device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
.driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v6 3/4] bus/cdx: add support for MSI
2023-06-05 13:26 ` [PATCH v6 0/4] Support AMD CDX bus Nipun Gupta
2023-06-05 13:26 ` [PATCH v6 1/4] bus/cdx: introduce " Nipun Gupta
2023-06-05 13:26 ` [PATCH v6 2/4] bus/cdx: add DMA map and unmap support Nipun Gupta
@ 2023-06-05 13:26 ` Nipun Gupta
2023-06-06 9:30 ` David Marchand
2023-06-05 13:26 ` [PATCH v6 4/4] bus/cdx: support plug unplug and dev iterator Nipun Gupta
3 siblings, 1 reply; 102+ messages in thread
From: Nipun Gupta @ 2023-06-05 13:26 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
MSI's are exposed to the devices using VFIO (vfio-cdx). This
patch uses the same to add support for MSI for the devices on
the cdx bus.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
drivers/bus/cdx/bus_cdx_driver.h | 25 +++++
drivers/bus/cdx/cdx.c | 11 ++
drivers/bus/cdx/cdx_vfio.c | 176 ++++++++++++++++++++++++++++++-
drivers/bus/cdx/version.map | 2 +
4 files changed, 212 insertions(+), 2 deletions(-)
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index dbbd1bbdf2..5745429fa7 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -56,6 +56,7 @@ struct rte_cdx_device {
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
/**< CDX Memory Resource */
+ struct rte_intr_handle *intr_handle; /**< Interrupt handle */
};
/**
@@ -147,6 +148,30 @@ void rte_cdx_unmap_device(struct rte_cdx_device *dev);
__rte_internal
void rte_cdx_register(struct rte_cdx_driver *driver);
+/**
+ * Enables VFIO Interrupts for CDX bus devices.
+ *
+ * @param intr_handle
+ * Pointer to the interrupt handle.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle);
+
+/**
+ * Disable VFIO Interrupts for CDX bus devices.
+ *
+ * @param intr_handle
+ * Pointer to the interrupt handle.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle);
+
/**
* Helper for CDX device registration from driver (eth, crypto, raw) instance
*/
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index 9ad8f73424..e5e6a665c3 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -202,6 +202,15 @@ cdx_scan_one(const char *dirname, const char *dev_name)
goto err;
}
+ /* Allocate interrupt instance for cdx device */
+ dev->intr_handle =
+ rte_intr_instance_alloc(RTE_INTR_INSTANCE_F_PRIVATE);
+ if (dev->intr_handle == NULL) {
+ CDX_BUS_ERR("Failed to create interrupt instance for %s",
+ dev->device.name);
+ return -ENOMEM;
+ }
+
/*
* Check if device is bound to 'vfio-cdx' driver, so that user-space
* can gracefully access the device.
@@ -380,6 +389,8 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr,
return ret;
error_probe:
+ rte_intr_instance_free(dev->intr_handle);
+ dev->intr_handle = NULL;
cdx_vfio_unmap_resource(dev);
error_map_device:
return ret;
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index c545bcd9f5..e8ccf7fb94 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -51,6 +51,10 @@ struct mapped_cdx_resource {
/** mapped cdx device list */
TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
+/* IRQ set buffer length for MSI interrupts */
+#define MSI_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \
+ sizeof(int) * (RTE_MAX_RXTX_INTR_VEC_ID + 1))
+
static struct rte_tailq_elem cdx_vfio_tailq = {
.name = "VFIO_CDX_RESOURCE_LIST",
};
@@ -95,6 +99,27 @@ cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
char cdx_addr[PATH_MAX] = {0};
struct mapped_cdx_resource *vfio_res = NULL;
struct mapped_cdx_res_list *vfio_res_list;
+ int ret, vfio_dev_fd;
+
+ if (rte_intr_fd_get(dev->intr_handle) < 0)
+ return -1;
+
+ if (close(rte_intr_fd_get(dev->intr_handle)) < 0) {
+ CDX_BUS_ERR("Error when closing eventfd file descriptor for %s",
+ dev->device.name);
+ return -1;
+ }
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ ret = rte_vfio_release_device(CDX_BUS_DEVICES_PATH, dev->device.name,
+ vfio_dev_fd);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot release VFIO device");
+ return ret;
+ }
vfio_res_list =
RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
@@ -117,6 +142,18 @@ cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
{
struct mapped_cdx_resource *vfio_res = NULL;
struct mapped_cdx_res_list *vfio_res_list;
+ int ret, vfio_dev_fd;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ ret = rte_vfio_release_device(CDX_BUS_DEVICES_PATH, dev->device.name,
+ vfio_dev_fd);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot release VFIO device");
+ return ret;
+ }
vfio_res_list =
RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
@@ -141,9 +178,74 @@ cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
return cdx_vfio_unmap_resource_secondary(dev);
}
+/* set up interrupt support (but not enable interrupts) */
static int
-cdx_rte_vfio_setup_device(int vfio_dev_fd)
+cdx_vfio_setup_interrupts(struct rte_cdx_device *dev, int vfio_dev_fd,
+ int num_irqs)
{
+ int i, ret;
+
+ if (num_irqs == 0)
+ return 0;
+
+ /* start from MSI interrupt type */
+ for (i = 0; i < num_irqs; i++) {
+ struct vfio_irq_info irq = { .argsz = sizeof(irq) };
+ int fd = -1;
+
+ irq.index = i;
+
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_IRQ_INFO, &irq);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot get VFIO IRQ info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /* if this vector cannot be used with eventfd, fail if we explicitly
+ * specified interrupt type, otherwise continue
+ */
+ if ((irq.flags & VFIO_IRQ_INFO_EVENTFD) == 0)
+ continue;
+
+ /* Set nb_intr to the total number of interrupts */
+ if (rte_intr_event_list_update(dev->intr_handle, irq.count))
+ return -1;
+
+ /* set up an eventfd for interrupts */
+ fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
+ if (fd < 0) {
+ CDX_BUS_ERR("Cannot set up eventfd, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ if (rte_intr_fd_set(dev->intr_handle, fd))
+ return -1;
+
+ /* DPDK CDX bus currently supports only MSI-X */
+ if (rte_intr_type_set(dev->intr_handle, RTE_INTR_HANDLE_VFIO_MSIX))
+ return -1;
+
+ if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
+ return -1;
+
+ return 0;
+ }
+
+ /* if we're here, we haven't found a suitable interrupt vector */
+ return -1;
+}
+
+static int
+cdx_vfio_setup_device(struct rte_cdx_device *dev, int vfio_dev_fd,
+ int num_irqs)
+{
+ if (cdx_vfio_setup_interrupts(dev, vfio_dev_fd, num_irqs) != 0) {
+ CDX_BUS_ERR("Error setting up interrupts!");
+ return -1;
+ }
+
/*
* Reset the device. If the device is not capable of resetting,
* then it updates errno as EINVAL.
@@ -279,6 +381,9 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
struct cdx_map *maps;
int vfio_dev_fd, i, ret;
+ if (rte_intr_fd_set(dev->intr_handle, -1))
+ return -1;
+
ret = rte_vfio_setup_device(CDX_BUS_DEVICES_PATH, dev_name,
&vfio_dev_fd, &device_info);
if (ret)
@@ -344,7 +449,7 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
free(reg);
}
- if (cdx_rte_vfio_setup_device(vfio_dev_fd) < 0) {
+ if (cdx_vfio_setup_device(dev, vfio_dev_fd, device_info.num_irqs) < 0) {
CDX_BUS_ERR("%s setup device failed", dev_name);
goto err_vfio_res;
}
@@ -373,6 +478,9 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
const char *dev_name = dev->device.name;
struct cdx_map *maps;
+ if (rte_intr_fd_set(dev->intr_handle, -1))
+ return -1;
+
/* if we're in a secondary process, just find our tailq entry */
TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
if (strcmp(vfio_res->name, dev_name))
@@ -406,6 +514,10 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
dev->mem_resource[i].len = maps[i].size;
}
+ /* we need save vfio_dev_fd, so it can be used during release */
+ if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
+ goto err_vfio_dev_fd;
+
return 0;
err_vfio_dev_fd:
rte_vfio_release_device(CDX_BUS_DEVICES_PATH, cdx_addr, vfio_dev_fd);
@@ -424,3 +536,63 @@ cdx_vfio_map_resource(struct rte_cdx_device *dev)
else
return cdx_vfio_map_resource_secondary(dev);
}
+
+int
+rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle)
+{
+ char irq_set_buf[MSI_IRQ_SET_BUF_LEN];
+ struct vfio_irq_set *irq_set;
+ int *fd_ptr, vfio_dev_fd, i;
+ int ret;
+
+ irq_set = (struct vfio_irq_set *) irq_set_buf;
+ irq_set->count = rte_intr_nb_intr_get(intr_handle);
+ irq_set->argsz = sizeof(struct vfio_irq_set) +
+ (sizeof(int) * irq_set->count);
+
+ irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER;
+ irq_set->index = 0;
+ irq_set->start = 0;
+ fd_ptr = (int *) &irq_set->data;
+
+ for (i = 0; i < rte_intr_nb_efd_get(intr_handle); i++)
+ fd_ptr[i] = rte_intr_efds_index_get(intr_handle, i);
+
+ vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
+
+ if (ret) {
+ CDX_BUS_ERR("Error enabling MSI interrupts for fd %d",
+ rte_intr_fd_get(intr_handle));
+ return -1;
+ }
+
+ return 0;
+}
+
+/* disable MSI interrupts */
+int
+rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle)
+{
+ struct vfio_irq_set *irq_set;
+ char irq_set_buf[MSI_IRQ_SET_BUF_LEN];
+ int len, ret, vfio_dev_fd;
+
+ len = sizeof(struct vfio_irq_set);
+
+ irq_set = (struct vfio_irq_set *) irq_set_buf;
+ irq_set->argsz = len;
+ irq_set->count = 0;
+ irq_set->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_TRIGGER;
+ irq_set->index = 0;
+ irq_set->start = 0;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
+
+ if (ret)
+ CDX_BUS_ERR("Error disabling MSI interrupts for fd %d",
+ rte_intr_fd_get(intr_handle));
+
+ return ret;
+}
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
index 957fcab978..2f3d484ebd 100644
--- a/drivers/bus/cdx/version.map
+++ b/drivers/bus/cdx/version.map
@@ -6,6 +6,8 @@ INTERNAL {
rte_cdx_register;
rte_cdx_unmap_device;
rte_cdx_unregister;
+ rte_cdx_vfio_intr_disable;
+ rte_cdx_vfio_intr_enable;
local: *;
};
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v6 3/4] bus/cdx: add support for MSI
2023-06-05 13:26 ` [PATCH v6 3/4] bus/cdx: add support for MSI Nipun Gupta
@ 2023-06-06 9:30 ` David Marchand
2023-06-06 9:42 ` Nipun Gupta
0 siblings, 1 reply; 102+ messages in thread
From: David Marchand @ 2023-06-06 9:30 UTC (permalink / raw)
To: Nipun Gupta
Cc: dev, thomas, hkalra, anatoly.burakov, stephen, ferruh.yigit,
harpreet.anand, nikhil.agarwal
On Mon, Jun 5, 2023 at 3:27 PM Nipun Gupta <nipun.gupta@amd.com> wrote:
> +/* set up interrupt support (but not enable interrupts) */
> static int
> -cdx_rte_vfio_setup_device(int vfio_dev_fd)
> +cdx_vfio_setup_interrupts(struct rte_cdx_device *dev, int vfio_dev_fd,
I had dropped a bit too much of the hunk in my previous comment, but see below:
[snip]
> +static int
> +cdx_vfio_setup_device(struct rte_cdx_device *dev, int vfio_dev_fd,
> + int num_irqs)
cdx_rte_vfio_setup_device() is renamed to cdx_vfio_setup_device().
Such a rename is unneeded noise, cdx_vfio_setup_device() should be
used from the start, in patch1.
If there is no further comment on the series, I can fix this when applying.
--
David Marchand
^ permalink raw reply [flat|nested] 102+ messages in thread
* Re: [PATCH v6 3/4] bus/cdx: add support for MSI
2023-06-06 9:30 ` David Marchand
@ 2023-06-06 9:42 ` Nipun Gupta
0 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-06-06 9:42 UTC (permalink / raw)
To: David Marchand
Cc: dev, thomas, hkalra, anatoly.burakov, stephen, ferruh.yigit,
harpreet.anand, nikhil.agarwal
On 6/6/2023 3:00 PM, David Marchand wrote:
>
> On Mon, Jun 5, 2023 at 3:27 PM Nipun Gupta <nipun.gupta@amd.com> wrote:
>> +/* set up interrupt support (but not enable interrupts) */
>> static int
>> -cdx_rte_vfio_setup_device(int vfio_dev_fd)
>> +cdx_vfio_setup_interrupts(struct rte_cdx_device *dev, int vfio_dev_fd,
>
> I had dropped a bit too much of the hunk in my previous comment, but see below:
>
> [snip]
>
>> +static int
>> +cdx_vfio_setup_device(struct rte_cdx_device *dev, int vfio_dev_fd,
>> + int num_irqs)
>
> cdx_rte_vfio_setup_device() is renamed to cdx_vfio_setup_device().
> Such a rename is unneeded noise, cdx_vfio_setup_device() should be
> used from the start, in patch1.
>
> If there is no further comment on the series, I can fix this when applying.
Thanks for pointing. I missed this part. There are some minor changes
requested by Thomas and I would send out a version today with this
change you mentioned as well.
Regards,
Nipun
>
>
> --
> David Marchand
>
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v6 4/4] bus/cdx: support plug unplug and dev iterator
2023-06-05 13:26 ` [PATCH v6 0/4] Support AMD CDX bus Nipun Gupta
` (2 preceding siblings ...)
2023-06-05 13:26 ` [PATCH v6 3/4] bus/cdx: add support for MSI Nipun Gupta
@ 2023-06-05 13:26 ` Nipun Gupta
3 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-06-05 13:26 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
This change adds support for plugging and unplugging
CDX devices on AMD CDX bus. Also, CDX dev iterator support
has been added for the CDX bus.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
drivers/bus/cdx/bus_cdx_driver.h | 1 +
drivers/bus/cdx/cdx.c | 122 +++++++++++++++++++++++++++++++
2 files changed, 123 insertions(+)
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index 5745429fa7..288cadf31b 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -52,6 +52,7 @@ struct rte_cdx_id {
struct rte_cdx_device {
RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
struct rte_device device; /**< Inherit core device */
+ struct rte_cdx_driver *driver; /**< CDX driver used in probing */
char name[RTE_DEV_NAME_MAX_LEN]; /**< Device name */
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index e5e6a665c3..2165486bc3 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -71,6 +71,7 @@
#include <rte_eal_paging.h>
#include <rte_errno.h>
#include <rte_devargs.h>
+#include <rte_kvargs.h>
#include <rte_malloc.h>
#include <rte_vfio.h>
@@ -92,6 +93,15 @@
struct rte_cdx_bus rte_cdx_bus;
+enum cdx_params {
+ RTE_CDX_PARAM_NAME,
+};
+
+static const char * const cdx_params_keys[] = {
+ [RTE_CDX_PARAM_NAME] = "name",
+ NULL,
+};
+
/* Add a device to CDX bus */
static void
cdx_add_device(struct rte_cdx_device *cdx_dev)
@@ -385,6 +395,7 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr,
} else {
dev->device.driver = &dr->driver;
}
+ dev->driver = dr;
return ret;
@@ -499,6 +510,71 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
return NULL;
}
+/* Remove a device from CDX bus */
+static void
+cdx_remove_device(struct rte_cdx_device *cdx_dev)
+{
+ TAILQ_REMOVE(&rte_cdx_bus.device_list, cdx_dev, next);
+}
+
+/*
+ * If vendor/device ID match, call the remove() function of the
+ * driver.
+ */
+static int
+cdx_detach_dev(struct rte_cdx_device *dev)
+{
+ struct rte_cdx_driver *dr;
+ int ret = 0;
+
+ if (dev == NULL)
+ return -EINVAL;
+
+ dr = dev->driver;
+
+ CDX_BUS_DEBUG("detach device %s using driver: %s",
+ dev->device.name, dr->driver.name);
+
+ if (dr->remove) {
+ ret = dr->remove(dev);
+ if (ret < 0)
+ return ret;
+ }
+
+ /* clear driver structure */
+ dev->driver = NULL;
+ dev->device.driver = NULL;
+
+ rte_cdx_unmap_device(dev);
+
+ rte_intr_instance_free(dev->intr_handle);
+ dev->intr_handle = NULL;
+
+ return 0;
+}
+
+static int
+cdx_plug(struct rte_device *dev)
+{
+ return cdx_probe_all_drivers(RTE_DEV_TO_CDX_DEV(dev));
+}
+
+static int
+cdx_unplug(struct rte_device *dev)
+{
+ struct rte_cdx_device *cdx_dev;
+ int ret;
+
+ cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+ ret = cdx_detach_dev(cdx_dev);
+ if (ret == 0) {
+ cdx_remove_device(cdx_dev);
+ rte_devargs_remove(dev->devargs);
+ free(cdx_dev);
+ }
+ return ret;
+}
+
static int
cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
{
@@ -526,15 +602,61 @@ cdx_get_iommu_class(void)
return RTE_IOVA_VA;
}
+static int
+cdx_dev_match(const struct rte_device *dev,
+ const void *_kvlist)
+{
+ const struct rte_kvargs *kvlist = _kvlist;
+ const char *key = cdx_params_keys[RTE_CDX_PARAM_NAME];
+ const char *name;
+
+ /* no kvlist arg, all devices match */
+ if (kvlist == NULL)
+ return 0;
+
+ /* if key is present in kvlist and does not match, filter device */
+ name = rte_kvargs_get(kvlist, key);
+ if (name != NULL && strcmp(name, dev->name))
+ return -1;
+
+ return 0;
+}
+
+static void *
+cdx_dev_iterate(const void *start,
+ const char *str,
+ const struct rte_dev_iterator *it __rte_unused)
+{
+ rte_bus_find_device_t find_device;
+ struct rte_kvargs *kvargs = NULL;
+ struct rte_device *dev;
+
+ if (str != NULL) {
+ kvargs = rte_kvargs_parse(str, cdx_params_keys);
+ if (kvargs == NULL) {
+ CDX_BUS_ERR("cannot parse argument list %s", str);
+ rte_errno = EINVAL;
+ return NULL;
+ }
+ }
+ find_device = rte_cdx_bus.bus.find_device;
+ dev = find_device(start, cdx_dev_match, kvargs);
+ rte_kvargs_free(kvargs);
+ return dev;
+}
+
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
.scan = cdx_scan,
.probe = cdx_probe,
.find_device = cdx_find_device,
+ .plug = cdx_plug,
+ .unplug = cdx_unplug,
.parse = cdx_parse,
.dma_map = cdx_dma_map,
.dma_unmap = cdx_dma_unmap,
.get_iommu_class = cdx_get_iommu_class,
+ .dev_iterate = cdx_dev_iterate,
},
.device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
.driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v7 0/4] Support AMD CDX bus
2023-01-24 14:07 [RFC PATCH 0/6] add support for CDX bus Nipun Gupta
` (11 preceding siblings ...)
2023-06-05 13:26 ` [PATCH v6 0/4] Support AMD CDX bus Nipun Gupta
@ 2023-06-06 10:02 ` Nipun Gupta
2023-06-06 10:02 ` [PATCH v7 1/4] bus/cdx: introduce " Nipun Gupta
` (3 more replies)
2023-06-07 4:24 ` [PATCH v8 0/4] Support AMD CDX bus Nipun Gupta
13 siblings, 4 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-06-06 10:02 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
The CDX devices are memory mapped on system bus for embedded CPUs.
It uses sysfs interface and the vfio-cdx driver to discover
and initialize the CDX devices.
The CDX bus and VFIO support is available at Xilinx/AMD open source tree:
https://github.com/Xilinx/linux-xlnx (drivers/cdx/ and drivers/vfio/cdx)
Linux AMD CDX bus patches has been added into linux tree:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/cdx
VFIO patches are also submitted in upstream:
https://www.spinics.net/lists/kvm/msg310623.html
CDX is a Hardware Architecture designed for AMD FPGA devices. It
consists of mechanism for interaction between FPGA, Firmware and
the APUs (Application CPUs).
Firmware resides on RPU (Realtime CPUs) which interacts with
the FPGA program manager and the APUs. The RPU provides memory-mapped
interface (RPU if) which is used to communicate with APUs.
VFIO CDX driver provides the CDX device resources like MMIO and interrupts
to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
driver to discover and initialize the CDX devices for user-space
applications.
Changes v6->v7
- removed redundant references to rte_cdx_sysfs_path().
- Fixed naming of cdx_vfio_setup_device() API from the
first patch itself.
- Added dot after few sentences in comments.
Changes v5->v6
- Removed newly introduce irq_count parameter, and use num_intr
instead.
- Removed rte_cdx_sysfs_path(), and directly used the macro for
the sysfs path.
- Removed few unrequired param checks in multiple APIs
- add name variable in rte_cdx_device
- moved rte_cdx_bus from bus_cdx_driver.h to private.h
- Other code cleanup
Changes v4->v5
- Split patch and have a separate commit for eal interrupt field
(irq_count) and corresponding set/get APIs (rte_intr_irq_count_set,
rte_intr_irq_count_get)
- Renamed "CDX bus" to "AMD CDX bus" in documentation/commit headings
- Removed unnecessary headers
Changes v3->v4:
- removed platform specific patch (adding config for ARM CDX)
from this series
Changes v2->v3:
- merged cdx bus compilation enablement in the first patch
- fixed issue reported by check-git-log.sh
- updated release notes
- updated offset to uint64_t instead of off_t in cdx_map_resource
Changes v1->v2:
- Moved file rte_cdx_bus.h to internal bus_cdx_driver.h
and added this file to deivce_cdx_headers
- Moved cdx.h to private.h
- Removed rte_ prefix from the static symbols in .c files.
Changes RFC->v1:
- Marked few API's as internal which were not required
to be provided to user.
Nipun Gupta (4):
bus/cdx: introduce AMD CDX bus
bus/cdx: add DMA map and unmap support
bus/cdx: add support for MSI
bus/cdx: support plug unplug and dev iterator
MAINTAINERS | 5 +
doc/guides/rel_notes/release_23_07.rst | 6 +
drivers/bus/cdx/bus_cdx_driver.h | 192 +++++++
drivers/bus/cdx/cdx.c | 666 +++++++++++++++++++++++++
drivers/bus/cdx/cdx_logs.h | 37 ++
drivers/bus/cdx/cdx_vfio.c | 598 ++++++++++++++++++++++
drivers/bus/cdx/meson.build | 13 +
drivers/bus/cdx/private.h | 53 ++
drivers/bus/cdx/version.map | 12 +
drivers/bus/meson.build | 1 +
10 files changed, 1583 insertions(+)
create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
create mode 100644 drivers/bus/cdx/cdx.c
create mode 100644 drivers/bus/cdx/cdx_logs.h
create mode 100644 drivers/bus/cdx/cdx_vfio.c
create mode 100644 drivers/bus/cdx/meson.build
create mode 100644 drivers/bus/cdx/private.h
create mode 100644 drivers/bus/cdx/version.map
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v7 1/4] bus/cdx: introduce AMD CDX bus
2023-06-06 10:02 ` [PATCH v7 0/4] Support AMD CDX bus Nipun Gupta
@ 2023-06-06 10:02 ` Nipun Gupta
2023-06-06 13:00 ` Thomas Monjalon
2023-06-06 10:02 ` [PATCH v7 2/4] bus/cdx: add DMA map and unmap support Nipun Gupta
` (2 subsequent siblings)
3 siblings, 1 reply; 102+ messages in thread
From: Nipun Gupta @ 2023-06-06 10:02 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
AMD CDX bus supports multiple type of devices, which can be
exposed to user-space via vfio-cdx.
vfio-cdx provides the MMIO IO_MEMORY regions as well as the
DMA interface for the device (IOMMU).
This support aims to enable the DPDK to support the cdx
devices in user-space using VFIO interface.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
MAINTAINERS | 5 +
doc/guides/rel_notes/release_23_07.rst | 6 +
drivers/bus/cdx/bus_cdx_driver.h | 166 ++++++++
drivers/bus/cdx/cdx.c | 503 +++++++++++++++++++++++++
drivers/bus/cdx/cdx_logs.h | 37 ++
drivers/bus/cdx/cdx_vfio.c | 426 +++++++++++++++++++++
drivers/bus/cdx/meson.build | 13 +
drivers/bus/cdx/private.h | 53 +++
drivers/bus/cdx/version.map | 10 +
drivers/bus/meson.build | 1 +
10 files changed, 1220 insertions(+)
create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
create mode 100644 drivers/bus/cdx/cdx.c
create mode 100644 drivers/bus/cdx/cdx_logs.h
create mode 100644 drivers/bus/cdx/cdx_vfio.c
create mode 100644 drivers/bus/cdx/meson.build
create mode 100644 drivers/bus/cdx/private.h
create mode 100644 drivers/bus/cdx/version.map
diff --git a/MAINTAINERS b/MAINTAINERS
index a5219926ab..c4b2b3565b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -569,6 +569,11 @@ M: Parav Pandit <parav@nvidia.com>
M: Xueming Li <xuemingl@nvidia.com>
F: drivers/bus/auxiliary/
+AMD CDX bus driver
+M: Nipun Gupta <nipun.gupta@amd.com>
+M: Nikhil Agarwal <nikhil.agarwal@amd.com>
+F: drivers/bus/cdx/
+
Intel FPGA bus
M: Rosen Xu <rosen.xu@intel.com>
F: drivers/bus/ifpga/
diff --git a/doc/guides/rel_notes/release_23_07.rst b/doc/guides/rel_notes/release_23_07.rst
index a9b1293689..7c6bb2b894 100644
--- a/doc/guides/rel_notes/release_23_07.rst
+++ b/doc/guides/rel_notes/release_23_07.rst
@@ -55,6 +55,12 @@ New Features
Also, make sure to start the actual text at the margin.
=======================================================
+* **Added AMD CDX bus support.**
+
+ CDX bus driver has been added to support AMD CDX bus, which operates
+ on FPGA based CDX devices. The CDX devices are memory mapped on system
+ bus for embedded CPUs.
+
Removed Items
-------------
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
new file mode 100644
index 0000000000..3515fcddf2
--- /dev/null
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -0,0 +1,166 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef BUS_CDX_DRIVER_H
+#define BUS_CDX_DRIVER_H
+
+/**
+ * @file
+ * AMD CDX bus interface
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdlib.h>
+#include <inttypes.h>
+
+#include <bus_driver.h>
+#include <dev_driver.h>
+#include <rte_interrupts.h>
+
+/* Forward declarations */
+struct rte_cdx_device;
+struct rte_cdx_driver;
+struct rte_cdx_bus;
+
+#define CDX_BUS_DEVICES_PATH "/sys/bus/cdx/devices"
+
+#define CDX_MAX_RESOURCE 4
+
+/** Any CDX device identifier (vendor, device). */
+#define RTE_CDX_ANY_ID (0xffff)
+
+#define RTE_PMD_REGISTER_CDX_TABLE(name, table) \
+static const char DRV_EXP_TAG(name, cdx_tbl_export)[] __rte_used = \
+RTE_STR(table)
+
+/**
+ * A structure describing an ID for a CDX driver. Each driver provides a
+ * table of these IDs for each device that it supports.
+ */
+struct rte_cdx_id {
+ uint16_t vendor_id; /**< Vendor ID. */
+ uint16_t device_id; /**< Device ID. */
+};
+
+/**
+ * A structure describing a CDX device.
+ */
+struct rte_cdx_device {
+ RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
+ struct rte_device device; /**< Inherit core device */
+ char name[RTE_DEV_NAME_MAX_LEN]; /**< Device name */
+ struct rte_cdx_id id; /**< CDX ID. */
+ struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
+ /**< CDX Memory Resource */
+};
+
+/**
+ * @internal
+ * Helper macro for drivers that need to convert to struct rte_cdx_device.
+ */
+#define RTE_DEV_TO_CDX_DEV(ptr) \
+ container_of(ptr, struct rte_cdx_device, device)
+
+#define RTE_DEV_TO_CDX_DEV_CONST(ptr) \
+ container_of(ptr, const struct rte_cdx_device, device)
+
+#define RTE_ETH_DEV_TO_CDX_DEV(eth_dev) RTE_DEV_TO_CDX_DEV((eth_dev)->device)
+
+#ifdef __cplusplus
+/** C++ macro used to help building up tables of device IDs. */
+#define RTE_CDX_DEVICE(vend, dev) \
+ (vend), \
+ (dev)
+#else
+/** Macro used to help building up tables of device IDs. */
+#define RTE_CDX_DEVICE(vend, dev) \
+ .vendor_id = (vend), \
+ .device_id = (dev)
+#endif
+
+/**
+ * Initialisation function for the driver called during CDX probing.
+ */
+typedef int (rte_cdx_probe_t)(struct rte_cdx_driver *, struct rte_cdx_device *);
+
+/**
+ * Uninitialisation function for the driver called during hotplugging.
+ */
+typedef int (rte_cdx_remove_t)(struct rte_cdx_device *);
+
+/**
+ * A structure describing a CDX driver.
+ */
+struct rte_cdx_driver {
+ RTE_TAILQ_ENTRY(rte_cdx_driver) next; /**< Next in list. */
+ struct rte_driver driver; /**< Inherit core driver. */
+ struct rte_cdx_bus *bus; /**< CDX bus reference. */
+ rte_cdx_probe_t *probe; /**< Device probe function. */
+ rte_cdx_remove_t *remove; /**< Device remove function. */
+ const struct rte_cdx_id *id_table; /**< ID table, NULL terminated. */
+ uint32_t drv_flags; /**< Flags RTE_CDX_DRV_*. */
+};
+
+/**
+ * Map the CDX device resources in user space virtual memory address.
+ *
+ * @param dev
+ * A pointer to a rte_cdx_device structure describing the device
+ * to use.
+ *
+ * @return
+ * 0 on success, <0 on error.
+ */
+__rte_internal
+int rte_cdx_map_device(struct rte_cdx_device *dev);
+
+/**
+ * Unmap this device.
+ *
+ * @param dev
+ * A pointer to a rte_cdx_device structure describing the device
+ * to use.
+ */
+__rte_internal
+void rte_cdx_unmap_device(struct rte_cdx_device *dev);
+
+/**
+ * Register a CDX driver.
+ *
+ * @param driver
+ * A pointer to a rte_cdx_driver structure describing the driver
+ * to be registered.
+ */
+__rte_internal
+void rte_cdx_register(struct rte_cdx_driver *driver);
+
+/**
+ * Helper for CDX device registration from driver (eth, crypto, raw) instance.
+ */
+#define RTE_PMD_REGISTER_CDX(nm, cdx_drv) \
+ RTE_INIT(cdxinitfn_ ##nm) \
+ {\
+ (cdx_drv).driver.name = RTE_STR(nm);\
+ rte_cdx_register(&cdx_drv); \
+ } \
+ RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+/**
+ * Unregister a CDX driver.
+ *
+ * @param driver
+ * A pointer to a rte_cdx_driver structure describing the driver
+ * to be unregistered.
+ */
+__rte_internal
+void rte_cdx_unregister(struct rte_cdx_driver *driver);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* BUS_CDX_DRIVER_H */
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
new file mode 100644
index 0000000000..9386cb7a0a
--- /dev/null
+++ b/drivers/bus/cdx/cdx.c
@@ -0,0 +1,503 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+/*
+ * Architecture Overview
+ * =====================
+ * CDX is a Hardware Architecture designed for AMD FPGA devices. It
+ * consists of sophisticated mechanism for interaction between FPGA,
+ * Firmware and the APUs (Application CPUs).
+ *
+ * Firmware resides on RPU (Realtime CPUs) which interacts with
+ * the FPGA program manager and the APUs. The RPU provides memory-mapped
+ * interface (RPU if) which is used to communicate with APUs.
+ *
+ * The diagram below shows an overview of the AMD CDX architecture:
+ *
+ * +--------------------------------------+
+ * | DPDK |
+ * | DPDK CDX drivers |
+ * | | |
+ * | DPDK AMD CDX bus |
+ * | | |
+ * +-----------------------------|--------+
+ * |
+ * +-----------------------------|--------+
+ * | Application CPUs (APU) | |
+ * | | |
+ * | VFIO CDX driver |
+ * | Linux OS | |
+ * | Linux AMD CDX bus |
+ * | | |
+ * +-----------------------------|--------+
+ * |
+ * |
+ * +------------------------| RPU if |----+
+ * | | |
+ * | V |
+ * | Realtime CPUs (RPU) |
+ * | |
+ * +--------------------------------------+
+ * |
+ * +---------------------|----------------+
+ * | FPGA | |
+ * | +-----------------------+ |
+ * | | | | |
+ * | +-------+ +-------+ +-------+ |
+ * | | dev 1 | | dev 2 | | dev 3 | |
+ * | +-------+ +-------+ +-------+ |
+ * +--------------------------------------+
+ *
+ * The RPU firmware extracts the device information from the loaded FPGA
+ * image and implements a mechanism that allows the APU drivers to
+ * enumerate such devices (device personality and resource details) via
+ * a dedicated communication channel.
+ *
+ * VFIO CDX driver provides the CDX device resources like MMIO and interrupts
+ * to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
+ * driver to discover and initialize the CDX devices for user-space
+ * applications.
+ */
+
+/**
+ * @file
+ * CDX probing using Linux sysfs.
+ */
+
+#include <string.h>
+#include <dirent.h>
+
+#include <rte_eal_paging.h>
+#include <rte_errno.h>
+#include <rte_devargs.h>
+#include <rte_malloc.h>
+#include <rte_vfio.h>
+
+#include <eal_filesystem.h>
+
+#include "bus_cdx_driver.h"
+#include "cdx_logs.h"
+#include "private.h"
+
+#define CDX_BUS_NAME cdx
+#define CDX_DEV_PREFIX "cdx-"
+
+/* CDX Bus iterators */
+#define FOREACH_DEVICE_ON_CDXBUS(p) \
+ RTE_TAILQ_FOREACH(p, &rte_cdx_bus.device_list, next)
+
+#define FOREACH_DRIVER_ON_CDXBUS(p) \
+ RTE_TAILQ_FOREACH(p, &rte_cdx_bus.driver_list, next)
+
+struct rte_cdx_bus rte_cdx_bus;
+
+/* Add a device to CDX bus */
+static void
+cdx_add_device(struct rte_cdx_device *cdx_dev)
+{
+ TAILQ_INSERT_TAIL(&rte_cdx_bus.device_list, cdx_dev, next);
+}
+
+static int
+cdx_get_kernel_driver_by_path(const char *filename, char *driver_name,
+ size_t len)
+{
+ int count;
+ char path[PATH_MAX];
+ char *name;
+
+ if (!filename || !driver_name)
+ return -1;
+
+ count = readlink(filename, path, PATH_MAX);
+ if (count >= PATH_MAX)
+ return -1;
+
+ /* For device does not have a driver */
+ if (count < 0)
+ return 1;
+
+ path[count] = '\0';
+
+ name = strrchr(path, '/');
+ if (name) {
+ strlcpy(driver_name, name + 1, len);
+ return 0;
+ }
+
+ return -1;
+}
+
+int rte_cdx_map_device(struct rte_cdx_device *dev)
+{
+ return cdx_vfio_map_resource(dev);
+}
+
+void rte_cdx_unmap_device(struct rte_cdx_device *dev)
+{
+ cdx_vfio_unmap_resource(dev);
+}
+
+static struct rte_devargs *
+cdx_devargs_lookup(const char *dev_name)
+{
+ struct rte_devargs *devargs;
+
+ RTE_EAL_DEVARGS_FOREACH("cdx", devargs) {
+ if (strcmp(devargs->name, dev_name) == 0)
+ return devargs;
+ }
+ return NULL;
+}
+
+static bool
+cdx_ignore_device(const char *dev_name)
+{
+ struct rte_devargs *devargs = cdx_devargs_lookup(dev_name);
+
+ switch (rte_cdx_bus.bus.conf.scan_mode) {
+ case RTE_BUS_SCAN_ALLOWLIST:
+ if (devargs && devargs->policy == RTE_DEV_ALLOWED)
+ return false;
+ break;
+ case RTE_BUS_SCAN_UNDEFINED:
+ case RTE_BUS_SCAN_BLOCKLIST:
+ if (devargs == NULL || devargs->policy != RTE_DEV_BLOCKED)
+ return false;
+ break;
+ }
+ return true;
+}
+
+/*
+ * Scan one cdx sysfs entry, and fill the devices list from it.
+ * It checks if the CDX device is bound to vfio-cdx driver. In case
+ * the device is vfio bound, it reads the vendor and device id and
+ * stores it for device-driver matching.
+ */
+static int
+cdx_scan_one(const char *dirname, const char *dev_name)
+{
+ char filename[PATH_MAX];
+ struct rte_cdx_device *dev = NULL;
+ char driver[PATH_MAX];
+ unsigned long tmp;
+ int ret;
+
+ dev = calloc(1, sizeof(*dev));
+ if (!dev)
+ return -ENOMEM;
+
+ dev->device.bus = &rte_cdx_bus.bus;
+ memcpy(dev->name, dev_name, RTE_DEV_NAME_MAX_LEN);
+ dev->device.name = dev->name;
+
+ /* parse driver */
+ snprintf(filename, sizeof(filename), "%s/driver", dirname);
+ ret = cdx_get_kernel_driver_by_path(filename, driver, sizeof(driver));
+ if (ret < 0) {
+ CDX_BUS_ERR("Fail to get kernel driver");
+ ret = -1;
+ goto err;
+ }
+
+ /*
+ * Check if device is bound to 'vfio-cdx' driver, so that user-space
+ * can gracefully access the device.
+ */
+ if (ret || strcmp(driver, "vfio-cdx")) {
+ ret = 0;
+ goto err;
+ }
+
+ /* get vendor id */
+ snprintf(filename, sizeof(filename), "%s/vendor", dirname);
+ if (eal_parse_sysfs_value(filename, &tmp) < 0) {
+ ret = -1;
+ goto err;
+ }
+ dev->id.vendor_id = (uint16_t)tmp;
+
+ /* get device id */
+ snprintf(filename, sizeof(filename), "%s/device", dirname);
+ if (eal_parse_sysfs_value(filename, &tmp) < 0) {
+ free(dev);
+ return -1;
+ }
+ dev->id.device_id = (uint16_t)tmp;
+
+ cdx_add_device(dev);
+
+ return 0;
+
+err:
+ free(dev);
+ return ret;
+}
+
+/*
+ * Scan the content of the CDX bus, and the devices in the devices
+ * list.
+ */
+static int
+cdx_scan(void)
+{
+ struct dirent *e;
+ DIR *dir;
+ char dirname[PATH_MAX];
+
+ dir = opendir(CDX_BUS_DEVICES_PATH);
+ if (dir == NULL) {
+ CDX_BUS_ERR("%s(): opendir failed: %s", __func__,
+ strerror(errno));
+ return -1;
+ }
+
+ while ((e = readdir(dir)) != NULL) {
+ if (e->d_name[0] == '.')
+ continue;
+
+ if (cdx_ignore_device(e->d_name))
+ continue;
+
+ snprintf(dirname, sizeof(dirname), "%s/%s",
+ CDX_BUS_DEVICES_PATH, e->d_name);
+
+ if (cdx_scan_one(dirname, e->d_name) < 0)
+ goto error;
+ }
+ closedir(dir);
+ return 0;
+
+error:
+ closedir(dir);
+ return -1;
+}
+
+/* map a particular resource from a file */
+void *
+cdx_map_resource(void *requested_addr, int fd, uint64_t offset, size_t size,
+ int additional_flags)
+{
+ void *mapaddr;
+
+ /* Map the cdx MMIO memory resource of device */
+ mapaddr = rte_mem_map(requested_addr, size,
+ RTE_PROT_READ | RTE_PROT_WRITE,
+ RTE_MAP_SHARED | additional_flags, fd, offset);
+ if (mapaddr == NULL) {
+ CDX_BUS_ERR("%s(): cannot map resource(%d, %p, 0x%zx, 0x%"PRIx64"): %s (%p)",
+ __func__, fd, requested_addr, size, offset,
+ rte_strerror(rte_errno), mapaddr);
+ }
+ CDX_BUS_DEBUG("CDX MMIO memory mapped at %p", mapaddr);
+
+ return mapaddr;
+}
+
+/* unmap a particular resource */
+void
+cdx_unmap_resource(void *requested_addr, size_t size)
+{
+ if (requested_addr == NULL)
+ return;
+
+ /* Unmap the CDX memory resource of device */
+ if (rte_mem_unmap(requested_addr, size)) {
+ CDX_BUS_ERR("%s(): cannot mem unmap(%p, %#zx): %s", __func__,
+ requested_addr, size, rte_strerror(rte_errno));
+ }
+ CDX_BUS_DEBUG("CDX memory unmapped at %p", requested_addr);
+}
+/*
+ * Match the CDX Driver and Device using device id and vendor id.
+ */
+static bool
+cdx_match(const struct rte_cdx_driver *cdx_drv,
+ const struct rte_cdx_device *cdx_dev)
+{
+ const struct rte_cdx_id *id_table;
+
+ for (id_table = cdx_drv->id_table; id_table->vendor_id != 0;
+ id_table++) {
+ /* check if device's identifiers match the driver's ones */
+ if (id_table->vendor_id != cdx_dev->id.vendor_id &&
+ id_table->vendor_id != RTE_CDX_ANY_ID)
+ continue;
+ if (id_table->device_id != cdx_dev->id.device_id &&
+ id_table->device_id != RTE_CDX_ANY_ID)
+ continue;
+
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * If vendor id and device id match, call the probe() function of the
+ * driver.
+ */
+static int
+cdx_probe_one_driver(struct rte_cdx_driver *dr,
+ struct rte_cdx_device *dev)
+{
+ const char *dev_name = dev->name;
+ bool already_probed;
+ int ret;
+
+ /* The device is not blocked; Check if driver supports it */
+ if (!cdx_match(dr, dev))
+ /* Match of device and driver failed */
+ return 1;
+
+ already_probed = rte_dev_is_probed(&dev->device);
+ if (already_probed) {
+ CDX_BUS_INFO("Device %s is already probed", dev_name);
+ return -EEXIST;
+ }
+
+ CDX_BUS_DEBUG(" probe device %s using driver: %s", dev_name,
+ dr->driver.name);
+
+ ret = cdx_vfio_map_resource(dev);
+ if (ret != 0) {
+ CDX_BUS_ERR("CDX map device failed: %d", ret);
+ goto error_map_device;
+ }
+
+ /* call the driver probe() function */
+ ret = dr->probe(dr, dev);
+ if (ret) {
+ CDX_BUS_ERR("Probe CDX driver: %s device: %s failed: %d",
+ dr->driver.name, dev_name, ret);
+ goto error_probe;
+ } else {
+ dev->device.driver = &dr->driver;
+ }
+
+ return ret;
+
+error_probe:
+ cdx_vfio_unmap_resource(dev);
+error_map_device:
+ return ret;
+}
+
+/*
+ * If vendor/device ID match, call the probe() function of all
+ * registered driver for the given device. Return < 0 if initialization
+ * failed, return 1 if no driver is found for this device.
+ */
+static int
+cdx_probe_all_drivers(struct rte_cdx_device *dev)
+{
+ struct rte_cdx_driver *dr = NULL;
+ int rc = 0;
+
+ FOREACH_DRIVER_ON_CDXBUS(dr) {
+ rc = cdx_probe_one_driver(dr, dev);
+ if (rc < 0)
+ /* negative value is an error */
+ return rc;
+ if (rc > 0)
+ /* positive value means driver doesn't support it */
+ continue;
+ return 0;
+ }
+ return 1;
+}
+
+/*
+ * Scan the content of the CDX bus, and call the probe() function for
+ * all registered drivers that have a matching entry in its id_table
+ * for discovered devices.
+ */
+static int
+cdx_probe(void)
+{
+ struct rte_cdx_device *dev = NULL;
+ size_t probed = 0, failed = 0;
+ int ret = 0;
+
+ FOREACH_DEVICE_ON_CDXBUS(dev) {
+ probed++;
+
+ ret = cdx_probe_all_drivers(dev);
+ if (ret < 0) {
+ CDX_BUS_ERR("Requested device %s cannot be used",
+ dev->name);
+ rte_errno = errno;
+ failed++;
+ ret = 0;
+ }
+ }
+
+ return (probed && probed == failed) ? -1 : 0;
+}
+
+static int
+cdx_parse(const char *name, void *addr)
+{
+ const char **out = addr;
+ int ret;
+
+ ret = strncmp(name, CDX_DEV_PREFIX, strlen(CDX_DEV_PREFIX));
+
+ if (ret == 0 && addr)
+ *out = name;
+
+ return ret;
+}
+
+/* register a driver */
+void
+rte_cdx_register(struct rte_cdx_driver *driver)
+{
+ TAILQ_INSERT_TAIL(&rte_cdx_bus.driver_list, driver, next);
+ driver->bus = &rte_cdx_bus;
+}
+
+/* unregister a driver */
+void
+rte_cdx_unregister(struct rte_cdx_driver *driver)
+{
+ TAILQ_REMOVE(&rte_cdx_bus.driver_list, driver, next);
+ driver->bus = NULL;
+}
+
+static struct rte_device *
+cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
+ const void *data)
+{
+ const struct rte_cdx_device *cdx_start;
+ struct rte_cdx_device *cdx_dev;
+
+ if (start != NULL) {
+ cdx_start = RTE_DEV_TO_CDX_DEV_CONST(start);
+ cdx_dev = TAILQ_NEXT(cdx_start, next);
+ } else {
+ cdx_dev = TAILQ_FIRST(&rte_cdx_bus.device_list);
+ }
+ while (cdx_dev != NULL) {
+ if (cmp(&cdx_dev->device, data) == 0)
+ return &cdx_dev->device;
+ cdx_dev = TAILQ_NEXT(cdx_dev, next);
+ }
+ return NULL;
+}
+
+struct rte_cdx_bus rte_cdx_bus = {
+ .bus = {
+ .scan = cdx_scan,
+ .probe = cdx_probe,
+ .find_device = cdx_find_device,
+ .parse = cdx_parse,
+ },
+ .device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
+ .driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
+};
+
+RTE_REGISTER_BUS(cdx, rte_cdx_bus.bus);
+RTE_LOG_REGISTER_DEFAULT(cdx_logtype_bus, NOTICE);
diff --git a/drivers/bus/cdx/cdx_logs.h b/drivers/bus/cdx/cdx_logs.h
new file mode 100644
index 0000000000..a1046ce544
--- /dev/null
+++ b/drivers/bus/cdx/cdx_logs.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef CDX_LOGS_H
+#define CDX_LOGS_H
+
+extern int cdx_logtype_bus;
+
+#define CDX_BUS_LOG(level, fmt, args...) \
+ rte_log(RTE_LOG_ ## level, cdx_logtype_bus, "cdx: " fmt "\n", \
+ ##args)
+
+/* Debug logs with Function names */
+#define CDX_BUS_DEBUG(fmt, args...) \
+ rte_log(RTE_LOG_DEBUG, cdx_logtype_bus, "cdx: %s(): " fmt "\n", \
+ __func__, ##args)
+
+#define CDX_BUS_INFO(fmt, args...) \
+ CDX_BUS_LOG(INFO, fmt, ## args)
+#define CDX_BUS_ERR(fmt, args...) \
+ CDX_BUS_LOG(ERR, fmt, ## args)
+#define CDX_BUS_WARN(fmt, args...) \
+ CDX_BUS_LOG(WARNING, fmt, ## args)
+
+/* DP Logs, toggled out at compile time if level lower than current level */
+#define CDX_BUS_DP_LOG(level, fmt, args...) \
+ RTE_LOG_DP(level, PMD, fmt, ## args)
+
+#define CDX_BUS_DP_DEBUG(fmt, args...) \
+ CDX_BUS_DP_LOG(DEBUG, fmt, ## args)
+#define CDX_BUS_DP_INFO(fmt, args...) \
+ CDX_BUS_DP_LOG(INFO, fmt, ## args)
+#define CDX_BUS_DP_WARN(fmt, args...) \
+ CDX_BUS_DP_LOG(WARNING, fmt, ## args)
+
+#endif /* CDX_LOGS_H */
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
new file mode 100644
index 0000000000..d320867ea8
--- /dev/null
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -0,0 +1,426 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+/**
+ * @file
+ * CDX probing using Linux VFIO.
+ *
+ * This code tries to determine if the CDX device is bound to VFIO driver,
+ * and initialize it (map MMIO regions, set up interrupts) if that's the case.
+ *
+ */
+
+#include <fcntl.h>
+#include <unistd.h>
+#include <sys/eventfd.h>
+#include <sys/socket.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <rte_eal_paging.h>
+#include <rte_malloc.h>
+#include <rte_vfio.h>
+
+#include "bus_cdx_driver.h"
+#include "cdx_logs.h"
+#include "private.h"
+
+/**
+ * A structure describing a CDX mapping.
+ */
+struct cdx_map {
+ void *addr;
+ char *path;
+ uint64_t offset;
+ uint64_t size;
+};
+
+/**
+ * A structure describing a mapped CDX resource.
+ * For multi-process we need to reproduce all CDX mappings in secondary
+ * processes, so save them in a tailq.
+ */
+struct mapped_cdx_resource {
+ TAILQ_ENTRY(mapped_cdx_resource) next;
+ char name[RTE_DEV_NAME_MAX_LEN]; /**< CDX device name */
+ char path[PATH_MAX];
+ int nb_maps;
+ struct cdx_map maps[CDX_MAX_RESOURCE];
+};
+
+/** mapped cdx device list */
+TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
+
+static struct rte_tailq_elem cdx_vfio_tailq = {
+ .name = "VFIO_CDX_RESOURCE_LIST",
+};
+EAL_REGISTER_TAILQ(cdx_vfio_tailq)
+
+static struct mapped_cdx_resource *
+cdx_vfio_find_and_unmap_resource(struct mapped_cdx_res_list *vfio_res_list,
+ struct rte_cdx_device *dev)
+{
+ struct mapped_cdx_resource *vfio_res = NULL;
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+ int i;
+
+ /* Get vfio_res */
+ TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+ if (strcmp(vfio_res->name, dev_name))
+ continue;
+ break;
+ }
+
+ if (vfio_res == NULL)
+ return vfio_res;
+
+ CDX_BUS_INFO("Releasing CDX mapped resource for %s", dev_name);
+
+ maps = vfio_res->maps;
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ if (maps[i].addr) {
+ CDX_BUS_DEBUG("Calling cdx_unmap_resource for %s at %p",
+ dev_name, maps[i].addr);
+ cdx_unmap_resource(maps[i].addr, maps[i].size);
+ }
+ }
+
+ return vfio_res;
+}
+
+static int
+cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
+{
+ char cdx_addr[PATH_MAX] = {0};
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list;
+
+ vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ vfio_res = cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for cdx device!",
+ cdx_addr);
+ return -1;
+ }
+
+ TAILQ_REMOVE(vfio_res_list, vfio_res, next);
+ rte_free(vfio_res);
+ return 0;
+}
+
+static int
+cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
+{
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list;
+
+ vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ vfio_res = cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for CDX device!",
+ dev->device.name);
+ return -1;
+ }
+
+ return 0;
+}
+
+int
+cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
+{
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return cdx_vfio_unmap_resource_primary(dev);
+ else
+ return cdx_vfio_unmap_resource_secondary(dev);
+}
+
+static int
+cdx_vfio_setup_device(int vfio_dev_fd)
+{
+ /*
+ * Reset the device. If the device is not capable of resetting,
+ * then it updates errno as EINVAL.
+ */
+ if (ioctl(vfio_dev_fd, VFIO_DEVICE_RESET) && errno != EINVAL) {
+ CDX_BUS_ERR("Unable to reset device! Error: %d (%s)", errno,
+ strerror(errno));
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+cdx_vfio_mmap_resource(int vfio_dev_fd, struct mapped_cdx_resource *vfio_res,
+ int index, int additional_flags)
+{
+ struct cdx_map *map = &vfio_res->maps[index];
+ void *vaddr;
+
+ if (map->size == 0) {
+ CDX_BUS_DEBUG("map size is 0, skip region %d", index);
+ return 0;
+ }
+
+ /* reserve the address using an inaccessible mapping */
+ vaddr = mmap(map->addr, map->size, 0, MAP_PRIVATE |
+ MAP_ANONYMOUS | additional_flags, -1, 0);
+ if (vaddr != MAP_FAILED) {
+ void *map_addr = NULL;
+
+ if (map->size) {
+ /* actual map of first part */
+ map_addr = cdx_map_resource(vaddr, vfio_dev_fd,
+ map->offset, map->size,
+ RTE_MAP_FORCE_ADDRESS);
+ }
+
+ if (map_addr == NULL) {
+ munmap(vaddr, map->size);
+ vaddr = MAP_FAILED;
+ CDX_BUS_ERR("Failed to map cdx MMIO region %d", index);
+ return -1;
+ }
+ } else {
+ CDX_BUS_ERR("Failed to create inaccessible mapping for MMIO region %d",
+ index);
+ return -1;
+ }
+
+ map->addr = vaddr;
+ return 0;
+}
+
+/*
+ * region info may contain capability headers, so we need to keep reallocating
+ * the memory until we match allocated memory size with argsz.
+ */
+static int
+cdx_vfio_get_region_info(int vfio_dev_fd, struct vfio_region_info **info,
+ int region)
+{
+ struct vfio_region_info *ri;
+ size_t argsz = sizeof(*ri);
+ int ret;
+
+ ri = malloc(sizeof(*ri));
+ if (ri == NULL) {
+ CDX_BUS_ERR("Cannot allocate memory for VFIO region info");
+ return -1;
+ }
+again:
+ memset(ri, 0, argsz);
+ ri->argsz = argsz;
+ ri->index = region;
+
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_REGION_INFO, ri);
+ if (ret < 0) {
+ free(ri);
+ return ret;
+ }
+ if (ri->argsz != argsz) {
+ struct vfio_region_info *tmp;
+
+ argsz = ri->argsz;
+ tmp = realloc(ri, argsz);
+
+ if (tmp == NULL) {
+ /* realloc failed but the ri is still there */
+ free(ri);
+ CDX_BUS_ERR("Cannot reallocate memory for VFIO region info");
+ return -1;
+ }
+ ri = tmp;
+ goto again;
+ }
+ *info = ri;
+
+ return 0;
+}
+
+static int
+find_max_end_va(const struct rte_memseg_list *msl, void *arg)
+{
+ size_t sz = msl->len;
+ void *end_va = RTE_PTR_ADD(msl->base_va, sz);
+ void **max_va = arg;
+
+ if (*max_va < end_va)
+ *max_va = end_va;
+ return 0;
+}
+
+static void *
+cdx_find_max_end_va(void)
+{
+ void *va = NULL;
+
+ rte_memseg_list_walk(find_max_end_va, &va);
+ return va;
+}
+
+static int
+cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ char cdx_addr[PATH_MAX] = {0};
+ static void *cdx_map_addr;
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+ int vfio_dev_fd, i, ret;
+
+ ret = rte_vfio_setup_device(CDX_BUS_DEVICES_PATH, dev_name,
+ &vfio_dev_fd, &device_info);
+ if (ret)
+ return ret;
+
+ /* allocate vfio_res and get region info */
+ vfio_res = rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0);
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("Cannot store VFIO mmap details");
+ goto err_vfio_dev_fd;
+ }
+ memcpy(vfio_res->name, dev_name, RTE_DEV_NAME_MAX_LEN);
+
+ /* get number of registers */
+ vfio_res->nb_maps = device_info.num_regions;
+
+ /* map memory regions */
+ maps = vfio_res->maps;
+
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ struct vfio_region_info *reg = NULL;
+ void *vaddr;
+
+ ret = cdx_vfio_get_region_info(vfio_dev_fd, ®, i);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s cannot get device region info error %i (%s)",
+ dev_name, errno, strerror(errno));
+ goto err_vfio_res;
+ }
+
+ /* skip non-mmappable regions */
+ if ((reg->flags & VFIO_REGION_INFO_FLAG_MMAP) == 0) {
+ free(reg);
+ continue;
+ }
+
+ /* try mapping somewhere close to the end of hugepages */
+ if (cdx_map_addr == NULL)
+ cdx_map_addr = cdx_find_max_end_va();
+
+ vaddr = cdx_map_addr;
+ cdx_map_addr = RTE_PTR_ADD(vaddr, (size_t)reg->size);
+
+ cdx_map_addr = RTE_PTR_ALIGN(cdx_map_addr,
+ sysconf(_SC_PAGE_SIZE));
+
+ maps[i].addr = vaddr;
+ maps[i].offset = reg->offset;
+ maps[i].size = reg->size;
+ maps[i].path = NULL; /* vfio doesn't have per-resource paths */
+
+ ret = cdx_vfio_mmap_resource(vfio_dev_fd, vfio_res, i, 0);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s mapping region %i failed: %s",
+ cdx_addr, i, strerror(errno));
+ free(reg);
+ goto err_vfio_res;
+ }
+
+ dev->mem_resource[i].addr = maps[i].addr;
+ dev->mem_resource[i].len = maps[i].size;
+
+ free(reg);
+ }
+
+ if (cdx_vfio_setup_device(vfio_dev_fd) < 0) {
+ CDX_BUS_ERR("%s setup device failed", dev_name);
+ goto err_vfio_res;
+ }
+
+ TAILQ_INSERT_TAIL(vfio_res_list, vfio_res, next);
+
+ return 0;
+err_vfio_res:
+ cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+ rte_free(vfio_res);
+err_vfio_dev_fd:
+ rte_vfio_release_device(CDX_BUS_DEVICES_PATH, dev_name, vfio_dev_fd);
+ return -1;
+}
+
+static int
+cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ char cdx_addr[PATH_MAX] = {0};
+ int vfio_dev_fd;
+ int i, ret;
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+
+ /* if we're in a secondary process, just find our tailq entry */
+ TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+ if (strcmp(vfio_res->name, dev_name))
+ continue;
+ break;
+ }
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for cdx device!",
+ dev_name);
+ return -1;
+ }
+
+ ret = rte_vfio_setup_device(CDX_BUS_DEVICES_PATH, dev_name,
+ &vfio_dev_fd, &device_info);
+ if (ret)
+ return ret;
+
+ /* map MMIO regions */
+ maps = vfio_res->maps;
+
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ ret = cdx_vfio_mmap_resource(vfio_dev_fd, vfio_res, i, MAP_FIXED);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s mapping MMIO region %i failed: %s",
+ dev_name, i, strerror(errno));
+ goto err_vfio_dev_fd;
+ }
+
+ dev->mem_resource[i].addr = maps[i].addr;
+ dev->mem_resource[i].len = maps[i].size;
+ }
+
+ return 0;
+err_vfio_dev_fd:
+ rte_vfio_release_device(CDX_BUS_DEVICES_PATH, cdx_addr, vfio_dev_fd);
+ return -1;
+}
+
+/*
+ * map the CDX resources of a CDX device in virtual memory (VFIO version).
+ * primary and secondary processes follow almost exactly the same path
+ */
+int
+cdx_vfio_map_resource(struct rte_cdx_device *dev)
+{
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return cdx_vfio_map_resource_primary(dev);
+ else
+ return cdx_vfio_map_resource_secondary(dev);
+}
diff --git a/drivers/bus/cdx/meson.build b/drivers/bus/cdx/meson.build
new file mode 100644
index 0000000000..f2ca104d34
--- /dev/null
+++ b/drivers/bus/cdx/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+
+if not is_linux
+ build = false
+ reason = 'only supported on Linux'
+endif
+
+driver_sdk_headers = files('bus_cdx_driver.h')
+sources = files(
+ 'cdx.c',
+ 'cdx_vfio.c',
+)
diff --git a/drivers/bus/cdx/private.h b/drivers/bus/cdx/private.h
new file mode 100644
index 0000000000..81987d0cfe
--- /dev/null
+++ b/drivers/bus/cdx/private.h
@@ -0,0 +1,53 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef CDX_PRIVATE_H
+#define CDX_PRIVATE_H
+
+#include "bus_cdx_driver.h"
+
+/**
+ * Structure describing the CDX bus.
+ */
+struct rte_cdx_bus {
+ struct rte_bus bus; /**< Inherit the generic class */
+ RTE_TAILQ_HEAD(, rte_cdx_device) device_list; /**< List of CDX devices */
+ RTE_TAILQ_HEAD(, rte_cdx_driver) driver_list; /**< List of CDX drivers */
+};
+
+/**
+ * Map a particular resource from a file.
+ *
+ * @param requested_addr
+ * The starting address for the new mapping range.
+ * @param fd
+ * The file descriptor.
+ * @param offset
+ * The offset for the mapping range.
+ * @param size
+ * The size for the mapping range.
+ * @param additional_flags
+ * The additional rte_mem_map() flags for the mapping range.
+ * @return
+ * - On success, the function returns a pointer to the mapped area.
+ * - On error, NULL is returned.
+ */
+void *cdx_map_resource(void *requested_addr, int fd, uint64_t offset,
+ size_t size, int additional_flags);
+
+/**
+ * Unmap a particular resource.
+ *
+ * @param requested_addr
+ * The address for the unmapping range.
+ * @param size
+ * The size for the unmapping range.
+ */
+void cdx_unmap_resource(void *requested_addr, size_t size);
+
+/* map/unmap VFIO resource */
+int cdx_vfio_map_resource(struct rte_cdx_device *dev);
+int cdx_vfio_unmap_resource(struct rte_cdx_device *dev);
+
+#endif /* CDX_PRIVATE_H */
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
new file mode 100644
index 0000000000..360460da18
--- /dev/null
+++ b/drivers/bus/cdx/version.map
@@ -0,0 +1,10 @@
+INTERNAL {
+ global:
+
+ rte_cdx_map_device;
+ rte_cdx_register;
+ rte_cdx_unmap_device;
+ rte_cdx_unregister;
+
+ local: *;
+};
diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
index 6d2520c543..a78b4283bf 100644
--- a/drivers/bus/meson.build
+++ b/drivers/bus/meson.build
@@ -3,6 +3,7 @@
drivers = [
'auxiliary',
+ 'cdx',
'dpaa',
'fslmc',
'ifpga',
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v7 1/4] bus/cdx: introduce AMD CDX bus
2023-06-06 10:02 ` [PATCH v7 1/4] bus/cdx: introduce " Nipun Gupta
@ 2023-06-06 13:00 ` Thomas Monjalon
2023-06-06 13:38 ` Nipun Gupta
0 siblings, 1 reply; 102+ messages in thread
From: Thomas Monjalon @ 2023-06-06 13:00 UTC (permalink / raw)
To: Nipun Gupta, Nipun Gupta
Cc: dev, david.marchand, hkalra, anatoly.burakov, stephen,
ferruh.yigit, harpreet.anand, nikhil.agarwal
06/06/2023 12:02, Nipun Gupta:
> +/* Forward declarations */
> +struct rte_cdx_device;
> +struct rte_cdx_driver;
> +struct rte_cdx_bus;
You should be very careful about what you export.
> +
> +#define CDX_BUS_DEVICES_PATH "/sys/bus/cdx/devices"
This define is not needed in the .h file.
> +
> +#define CDX_MAX_RESOURCE 4
This size looks required for rte_cdx_device,
but as an exported constant, it should be prefixed with RTE_
> +
> +/** Any CDX device identifier (vendor, device). */
> +#define RTE_CDX_ANY_ID (0xffff)
^ permalink raw reply [flat|nested] 102+ messages in thread
* Re: [PATCH v7 1/4] bus/cdx: introduce AMD CDX bus
2023-06-06 13:00 ` Thomas Monjalon
@ 2023-06-06 13:38 ` Nipun Gupta
2023-06-06 13:43 ` Thomas Monjalon
0 siblings, 1 reply; 102+ messages in thread
From: Nipun Gupta @ 2023-06-06 13:38 UTC (permalink / raw)
To: Thomas Monjalon
Cc: dev, david.marchand, hkalra, anatoly.burakov, stephen,
ferruh.yigit, harpreet.anand, nikhil.agarwal
On 6/6/2023 6:30 PM, Thomas Monjalon wrote:
>
> 06/06/2023 12:02, Nipun Gupta:
>> +/* Forward declarations */
>> +struct rte_cdx_device;
>> +struct rte_cdx_driver;
>> +struct rte_cdx_bus;
>
> You should be very careful about what you export.
>
>> +
>> +#define CDX_BUS_DEVICES_PATH "/sys/bus/cdx/devices"
>
> This define is not needed in the .h file.
This is required and used by some of the out of tree CDX drivers which
are developed by the customers, so it is needed here.
>
>> +
>> +#define CDX_MAX_RESOURCE 4
>
> This size looks required for rte_cdx_device,
> but as an exported constant, it should be prefixed with RTE_
Sure, will update.
Thanks,
Nipun
>
>> +
>> +/** Any CDX device identifier (vendor, device). */
>> +#define RTE_CDX_ANY_ID (0xffff)
>
>
>
^ permalink raw reply [flat|nested] 102+ messages in thread
* Re: [PATCH v7 1/4] bus/cdx: introduce AMD CDX bus
2023-06-06 13:38 ` Nipun Gupta
@ 2023-06-06 13:43 ` Thomas Monjalon
0 siblings, 0 replies; 102+ messages in thread
From: Thomas Monjalon @ 2023-06-06 13:43 UTC (permalink / raw)
To: Nipun Gupta
Cc: dev, david.marchand, hkalra, anatoly.burakov, stephen,
ferruh.yigit, harpreet.anand, nikhil.agarwal
06/06/2023 15:38, Nipun Gupta:
>
> On 6/6/2023 6:30 PM, Thomas Monjalon wrote:
> >
> > 06/06/2023 12:02, Nipun Gupta:
> >> +/* Forward declarations */
> >> +struct rte_cdx_device;
> >> +struct rte_cdx_driver;
> >> +struct rte_cdx_bus;
> >
> > You should be very careful about what you export.
> >
> >> +
> >> +#define CDX_BUS_DEVICES_PATH "/sys/bus/cdx/devices"
> >
> > This define is not needed in the .h file.
>
> This is required and used by some of the out of tree CDX drivers which
> are developed by the customers, so it is needed here.
In this case, it should be prefixed with RTE_
> >
> >> +
> >> +#define CDX_MAX_RESOURCE 4
> >
> > This size looks required for rte_cdx_device,
> > but as an exported constant, it should be prefixed with RTE_
>
> Sure, will update.
>
> Thanks,
> Nipun
>
> >
> >> +
> >> +/** Any CDX device identifier (vendor, device). */
> >> +#define RTE_CDX_ANY_ID (0xffff)
> >
> >
> >
>
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v7 2/4] bus/cdx: add DMA map and unmap support
2023-06-06 10:02 ` [PATCH v7 0/4] Support AMD CDX bus Nipun Gupta
2023-06-06 10:02 ` [PATCH v7 1/4] bus/cdx: introduce " Nipun Gupta
@ 2023-06-06 10:02 ` Nipun Gupta
2023-06-06 10:02 ` [PATCH v7 3/4] bus/cdx: add support for MSI Nipun Gupta
2023-06-06 10:02 ` [PATCH v7 4/4] bus/cdx: support plug unplug and dev iterator Nipun Gupta
3 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-06-06 10:02 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
AMD CDX bus can use VFIO interface for mapping and unmapping
of DMA addresses in the IOMMU. This change adds the callback
support for map and unmap APIs as well as fetching the IOMMU
class.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
drivers/bus/cdx/cdx.c | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index 9386cb7a0a..9ad8f73424 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -488,12 +488,42 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
return NULL;
}
+static int
+cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+ RTE_SET_USED(dev);
+
+ return rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ (uintptr_t)addr, iova, len);
+}
+
+static int
+cdx_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+ RTE_SET_USED(dev);
+
+ return rte_vfio_container_dma_unmap(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ (uintptr_t)addr, iova, len);
+}
+
+static enum rte_iova_mode
+cdx_get_iommu_class(void)
+{
+ if (TAILQ_EMPTY(&rte_cdx_bus.device_list))
+ return RTE_IOVA_DC;
+
+ return RTE_IOVA_VA;
+}
+
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
.scan = cdx_scan,
.probe = cdx_probe,
.find_device = cdx_find_device,
.parse = cdx_parse,
+ .dma_map = cdx_dma_map,
+ .dma_unmap = cdx_dma_unmap,
+ .get_iommu_class = cdx_get_iommu_class,
},
.device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
.driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v7 3/4] bus/cdx: add support for MSI
2023-06-06 10:02 ` [PATCH v7 0/4] Support AMD CDX bus Nipun Gupta
2023-06-06 10:02 ` [PATCH v7 1/4] bus/cdx: introduce " Nipun Gupta
2023-06-06 10:02 ` [PATCH v7 2/4] bus/cdx: add DMA map and unmap support Nipun Gupta
@ 2023-06-06 10:02 ` Nipun Gupta
2023-06-06 10:02 ` [PATCH v7 4/4] bus/cdx: support plug unplug and dev iterator Nipun Gupta
3 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-06-06 10:02 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
MSI's are exposed to the devices using VFIO (vfio-cdx). This
patch uses the same to add support for MSI for the devices on
the cdx bus.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
drivers/bus/cdx/bus_cdx_driver.h | 25 +++++
drivers/bus/cdx/cdx.c | 11 ++
drivers/bus/cdx/cdx_vfio.c | 176 ++++++++++++++++++++++++++++++-
drivers/bus/cdx/version.map | 2 +
4 files changed, 212 insertions(+), 2 deletions(-)
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index 3515fcddf2..cb99c13636 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -56,6 +56,7 @@ struct rte_cdx_device {
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
/**< CDX Memory Resource */
+ struct rte_intr_handle *intr_handle; /**< Interrupt handle */
};
/**
@@ -149,6 +150,30 @@ void rte_cdx_register(struct rte_cdx_driver *driver);
} \
RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+/**
+ * Enables VFIO Interrupts for CDX bus devices.
+ *
+ * @param intr_handle
+ * Pointer to the interrupt handle.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle);
+
+/**
+ * Disable VFIO Interrupts for CDX bus devices.
+ *
+ * @param intr_handle
+ * Pointer to the interrupt handle.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle);
+
/**
* Unregister a CDX driver.
*
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index 9ad8f73424..e5e6a665c3 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -202,6 +202,15 @@ cdx_scan_one(const char *dirname, const char *dev_name)
goto err;
}
+ /* Allocate interrupt instance for cdx device */
+ dev->intr_handle =
+ rte_intr_instance_alloc(RTE_INTR_INSTANCE_F_PRIVATE);
+ if (dev->intr_handle == NULL) {
+ CDX_BUS_ERR("Failed to create interrupt instance for %s",
+ dev->device.name);
+ return -ENOMEM;
+ }
+
/*
* Check if device is bound to 'vfio-cdx' driver, so that user-space
* can gracefully access the device.
@@ -380,6 +389,8 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr,
return ret;
error_probe:
+ rte_intr_instance_free(dev->intr_handle);
+ dev->intr_handle = NULL;
cdx_vfio_unmap_resource(dev);
error_map_device:
return ret;
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index d320867ea8..e8ccf7fb94 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -51,6 +51,10 @@ struct mapped_cdx_resource {
/** mapped cdx device list */
TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
+/* IRQ set buffer length for MSI interrupts */
+#define MSI_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \
+ sizeof(int) * (RTE_MAX_RXTX_INTR_VEC_ID + 1))
+
static struct rte_tailq_elem cdx_vfio_tailq = {
.name = "VFIO_CDX_RESOURCE_LIST",
};
@@ -95,6 +99,27 @@ cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
char cdx_addr[PATH_MAX] = {0};
struct mapped_cdx_resource *vfio_res = NULL;
struct mapped_cdx_res_list *vfio_res_list;
+ int ret, vfio_dev_fd;
+
+ if (rte_intr_fd_get(dev->intr_handle) < 0)
+ return -1;
+
+ if (close(rte_intr_fd_get(dev->intr_handle)) < 0) {
+ CDX_BUS_ERR("Error when closing eventfd file descriptor for %s",
+ dev->device.name);
+ return -1;
+ }
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ ret = rte_vfio_release_device(CDX_BUS_DEVICES_PATH, dev->device.name,
+ vfio_dev_fd);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot release VFIO device");
+ return ret;
+ }
vfio_res_list =
RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
@@ -117,6 +142,18 @@ cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
{
struct mapped_cdx_resource *vfio_res = NULL;
struct mapped_cdx_res_list *vfio_res_list;
+ int ret, vfio_dev_fd;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ ret = rte_vfio_release_device(CDX_BUS_DEVICES_PATH, dev->device.name,
+ vfio_dev_fd);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot release VFIO device");
+ return ret;
+ }
vfio_res_list =
RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
@@ -141,9 +178,74 @@ cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
return cdx_vfio_unmap_resource_secondary(dev);
}
+/* set up interrupt support (but not enable interrupts) */
static int
-cdx_vfio_setup_device(int vfio_dev_fd)
+cdx_vfio_setup_interrupts(struct rte_cdx_device *dev, int vfio_dev_fd,
+ int num_irqs)
{
+ int i, ret;
+
+ if (num_irqs == 0)
+ return 0;
+
+ /* start from MSI interrupt type */
+ for (i = 0; i < num_irqs; i++) {
+ struct vfio_irq_info irq = { .argsz = sizeof(irq) };
+ int fd = -1;
+
+ irq.index = i;
+
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_IRQ_INFO, &irq);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot get VFIO IRQ info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /* if this vector cannot be used with eventfd, fail if we explicitly
+ * specified interrupt type, otherwise continue
+ */
+ if ((irq.flags & VFIO_IRQ_INFO_EVENTFD) == 0)
+ continue;
+
+ /* Set nb_intr to the total number of interrupts */
+ if (rte_intr_event_list_update(dev->intr_handle, irq.count))
+ return -1;
+
+ /* set up an eventfd for interrupts */
+ fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
+ if (fd < 0) {
+ CDX_BUS_ERR("Cannot set up eventfd, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ if (rte_intr_fd_set(dev->intr_handle, fd))
+ return -1;
+
+ /* DPDK CDX bus currently supports only MSI-X */
+ if (rte_intr_type_set(dev->intr_handle, RTE_INTR_HANDLE_VFIO_MSIX))
+ return -1;
+
+ if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
+ return -1;
+
+ return 0;
+ }
+
+ /* if we're here, we haven't found a suitable interrupt vector */
+ return -1;
+}
+
+static int
+cdx_vfio_setup_device(struct rte_cdx_device *dev, int vfio_dev_fd,
+ int num_irqs)
+{
+ if (cdx_vfio_setup_interrupts(dev, vfio_dev_fd, num_irqs) != 0) {
+ CDX_BUS_ERR("Error setting up interrupts!");
+ return -1;
+ }
+
/*
* Reset the device. If the device is not capable of resetting,
* then it updates errno as EINVAL.
@@ -279,6 +381,9 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
struct cdx_map *maps;
int vfio_dev_fd, i, ret;
+ if (rte_intr_fd_set(dev->intr_handle, -1))
+ return -1;
+
ret = rte_vfio_setup_device(CDX_BUS_DEVICES_PATH, dev_name,
&vfio_dev_fd, &device_info);
if (ret)
@@ -344,7 +449,7 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
free(reg);
}
- if (cdx_vfio_setup_device(vfio_dev_fd) < 0) {
+ if (cdx_vfio_setup_device(dev, vfio_dev_fd, device_info.num_irqs) < 0) {
CDX_BUS_ERR("%s setup device failed", dev_name);
goto err_vfio_res;
}
@@ -373,6 +478,9 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
const char *dev_name = dev->device.name;
struct cdx_map *maps;
+ if (rte_intr_fd_set(dev->intr_handle, -1))
+ return -1;
+
/* if we're in a secondary process, just find our tailq entry */
TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
if (strcmp(vfio_res->name, dev_name))
@@ -406,6 +514,10 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
dev->mem_resource[i].len = maps[i].size;
}
+ /* we need save vfio_dev_fd, so it can be used during release */
+ if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
+ goto err_vfio_dev_fd;
+
return 0;
err_vfio_dev_fd:
rte_vfio_release_device(CDX_BUS_DEVICES_PATH, cdx_addr, vfio_dev_fd);
@@ -424,3 +536,63 @@ cdx_vfio_map_resource(struct rte_cdx_device *dev)
else
return cdx_vfio_map_resource_secondary(dev);
}
+
+int
+rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle)
+{
+ char irq_set_buf[MSI_IRQ_SET_BUF_LEN];
+ struct vfio_irq_set *irq_set;
+ int *fd_ptr, vfio_dev_fd, i;
+ int ret;
+
+ irq_set = (struct vfio_irq_set *) irq_set_buf;
+ irq_set->count = rte_intr_nb_intr_get(intr_handle);
+ irq_set->argsz = sizeof(struct vfio_irq_set) +
+ (sizeof(int) * irq_set->count);
+
+ irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER;
+ irq_set->index = 0;
+ irq_set->start = 0;
+ fd_ptr = (int *) &irq_set->data;
+
+ for (i = 0; i < rte_intr_nb_efd_get(intr_handle); i++)
+ fd_ptr[i] = rte_intr_efds_index_get(intr_handle, i);
+
+ vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
+
+ if (ret) {
+ CDX_BUS_ERR("Error enabling MSI interrupts for fd %d",
+ rte_intr_fd_get(intr_handle));
+ return -1;
+ }
+
+ return 0;
+}
+
+/* disable MSI interrupts */
+int
+rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle)
+{
+ struct vfio_irq_set *irq_set;
+ char irq_set_buf[MSI_IRQ_SET_BUF_LEN];
+ int len, ret, vfio_dev_fd;
+
+ len = sizeof(struct vfio_irq_set);
+
+ irq_set = (struct vfio_irq_set *) irq_set_buf;
+ irq_set->argsz = len;
+ irq_set->count = 0;
+ irq_set->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_TRIGGER;
+ irq_set->index = 0;
+ irq_set->start = 0;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
+
+ if (ret)
+ CDX_BUS_ERR("Error disabling MSI interrupts for fd %d",
+ rte_intr_fd_get(intr_handle));
+
+ return ret;
+}
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
index 360460da18..0a15d39ae8 100644
--- a/drivers/bus/cdx/version.map
+++ b/drivers/bus/cdx/version.map
@@ -5,6 +5,8 @@ INTERNAL {
rte_cdx_register;
rte_cdx_unmap_device;
rte_cdx_unregister;
+ rte_cdx_vfio_intr_disable;
+ rte_cdx_vfio_intr_enable;
local: *;
};
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v7 4/4] bus/cdx: support plug unplug and dev iterator
2023-06-06 10:02 ` [PATCH v7 0/4] Support AMD CDX bus Nipun Gupta
` (2 preceding siblings ...)
2023-06-06 10:02 ` [PATCH v7 3/4] bus/cdx: add support for MSI Nipun Gupta
@ 2023-06-06 10:02 ` Nipun Gupta
3 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-06-06 10:02 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
This change adds support for plugging and unplugging
CDX devices on AMD CDX bus. Also, CDX dev iterator support
has been added for the CDX bus.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
drivers/bus/cdx/bus_cdx_driver.h | 1 +
drivers/bus/cdx/cdx.c | 122 +++++++++++++++++++++++++++++++
2 files changed, 123 insertions(+)
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index cb99c13636..04ccdf7c53 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -52,6 +52,7 @@ struct rte_cdx_id {
struct rte_cdx_device {
RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
struct rte_device device; /**< Inherit core device */
+ struct rte_cdx_driver *driver; /**< CDX driver used in probing */
char name[RTE_DEV_NAME_MAX_LEN]; /**< Device name */
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[CDX_MAX_RESOURCE];
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index e5e6a665c3..2165486bc3 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -71,6 +71,7 @@
#include <rte_eal_paging.h>
#include <rte_errno.h>
#include <rte_devargs.h>
+#include <rte_kvargs.h>
#include <rte_malloc.h>
#include <rte_vfio.h>
@@ -92,6 +93,15 @@
struct rte_cdx_bus rte_cdx_bus;
+enum cdx_params {
+ RTE_CDX_PARAM_NAME,
+};
+
+static const char * const cdx_params_keys[] = {
+ [RTE_CDX_PARAM_NAME] = "name",
+ NULL,
+};
+
/* Add a device to CDX bus */
static void
cdx_add_device(struct rte_cdx_device *cdx_dev)
@@ -385,6 +395,7 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr,
} else {
dev->device.driver = &dr->driver;
}
+ dev->driver = dr;
return ret;
@@ -499,6 +510,71 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
return NULL;
}
+/* Remove a device from CDX bus */
+static void
+cdx_remove_device(struct rte_cdx_device *cdx_dev)
+{
+ TAILQ_REMOVE(&rte_cdx_bus.device_list, cdx_dev, next);
+}
+
+/*
+ * If vendor/device ID match, call the remove() function of the
+ * driver.
+ */
+static int
+cdx_detach_dev(struct rte_cdx_device *dev)
+{
+ struct rte_cdx_driver *dr;
+ int ret = 0;
+
+ if (dev == NULL)
+ return -EINVAL;
+
+ dr = dev->driver;
+
+ CDX_BUS_DEBUG("detach device %s using driver: %s",
+ dev->device.name, dr->driver.name);
+
+ if (dr->remove) {
+ ret = dr->remove(dev);
+ if (ret < 0)
+ return ret;
+ }
+
+ /* clear driver structure */
+ dev->driver = NULL;
+ dev->device.driver = NULL;
+
+ rte_cdx_unmap_device(dev);
+
+ rte_intr_instance_free(dev->intr_handle);
+ dev->intr_handle = NULL;
+
+ return 0;
+}
+
+static int
+cdx_plug(struct rte_device *dev)
+{
+ return cdx_probe_all_drivers(RTE_DEV_TO_CDX_DEV(dev));
+}
+
+static int
+cdx_unplug(struct rte_device *dev)
+{
+ struct rte_cdx_device *cdx_dev;
+ int ret;
+
+ cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+ ret = cdx_detach_dev(cdx_dev);
+ if (ret == 0) {
+ cdx_remove_device(cdx_dev);
+ rte_devargs_remove(dev->devargs);
+ free(cdx_dev);
+ }
+ return ret;
+}
+
static int
cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
{
@@ -526,15 +602,61 @@ cdx_get_iommu_class(void)
return RTE_IOVA_VA;
}
+static int
+cdx_dev_match(const struct rte_device *dev,
+ const void *_kvlist)
+{
+ const struct rte_kvargs *kvlist = _kvlist;
+ const char *key = cdx_params_keys[RTE_CDX_PARAM_NAME];
+ const char *name;
+
+ /* no kvlist arg, all devices match */
+ if (kvlist == NULL)
+ return 0;
+
+ /* if key is present in kvlist and does not match, filter device */
+ name = rte_kvargs_get(kvlist, key);
+ if (name != NULL && strcmp(name, dev->name))
+ return -1;
+
+ return 0;
+}
+
+static void *
+cdx_dev_iterate(const void *start,
+ const char *str,
+ const struct rte_dev_iterator *it __rte_unused)
+{
+ rte_bus_find_device_t find_device;
+ struct rte_kvargs *kvargs = NULL;
+ struct rte_device *dev;
+
+ if (str != NULL) {
+ kvargs = rte_kvargs_parse(str, cdx_params_keys);
+ if (kvargs == NULL) {
+ CDX_BUS_ERR("cannot parse argument list %s", str);
+ rte_errno = EINVAL;
+ return NULL;
+ }
+ }
+ find_device = rte_cdx_bus.bus.find_device;
+ dev = find_device(start, cdx_dev_match, kvargs);
+ rte_kvargs_free(kvargs);
+ return dev;
+}
+
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
.scan = cdx_scan,
.probe = cdx_probe,
.find_device = cdx_find_device,
+ .plug = cdx_plug,
+ .unplug = cdx_unplug,
.parse = cdx_parse,
.dma_map = cdx_dma_map,
.dma_unmap = cdx_dma_unmap,
.get_iommu_class = cdx_get_iommu_class,
+ .dev_iterate = cdx_dev_iterate,
},
.device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
.driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread
* [PATCH v8 0/4] Support AMD CDX bus
2023-01-24 14:07 [RFC PATCH 0/6] add support for CDX bus Nipun Gupta
` (12 preceding siblings ...)
2023-06-06 10:02 ` [PATCH v7 0/4] Support AMD CDX bus Nipun Gupta
@ 2023-06-07 4:24 ` Nipun Gupta
2023-06-07 4:24 ` [PATCH v8 1/4] bus/cdx: introduce " Nipun Gupta
` (4 more replies)
13 siblings, 5 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-06-07 4:24 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
The CDX devices are memory mapped on system bus for embedded CPUs.
It uses sysfs interface and the vfio-cdx driver to discover
and initialize the CDX devices.
The CDX bus and VFIO support is available at Xilinx/AMD open source tree:
https://github.com/Xilinx/linux-xlnx (drivers/cdx/ and drivers/vfio/cdx)
Linux AMD CDX bus patches has been added into linux tree:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/cdx
VFIO patches are also submitted in upstream:
https://www.spinics.net/lists/kvm/msg310623.html
CDX is a Hardware Architecture designed for AMD FPGA devices. It
consists of mechanism for interaction between FPGA, Firmware and
the APUs (Application CPUs).
Firmware resides on RPU (Realtime CPUs) which interacts with
the FPGA program manager and the APUs. The RPU provides memory-mapped
interface (RPU if) which is used to communicate with APUs.
VFIO CDX driver provides the CDX device resources like MMIO and interrupts
to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
driver to discover and initialize the CDX devices for user-space
applications.
Changes v7->v8
- Renamed CDX_BUS_DEVICES_PATH to RTE_CDX_BUS_DEVICES_PATH
- Renamed CDX_MAX_RESOURCE to RTE_CDX_MAX_RESOURCE
Changes v6->v7
- removed redundant references to rte_cdx_sysfs_path().
- Fixed naming of cdx_vfio_setup_device() API from the
first patch itself.
- Added dot after few sentences in comments.
Changes v5->v6
- Removed newly introduce irq_count parameter, and use num_intr
instead.
- Removed rte_cdx_sysfs_path(), and directly used the macro for
the sysfs path.
- Removed few unrequired param checks in multiple APIs
- add name variable in rte_cdx_device
- moved rte_cdx_bus from bus_cdx_driver.h to private.h
- Other code cleanup
Changes v4->v5
- Split patch and have a separate commit for eal interrupt field
(irq_count) and corresponding set/get APIs (rte_intr_irq_count_set,
rte_intr_irq_count_get)
- Renamed "CDX bus" to "AMD CDX bus" in documentation/commit headings
- Removed unnecessary headers
Changes v3->v4:
- removed platform specific patch (adding config for ARM CDX)
from this series
Changes v2->v3:
- merged cdx bus compilation enablement in the first patch
- fixed issue reported by check-git-log.sh
- updated release notes
- updated offset to uint64_t instead of off_t in cdx_map_resource
Changes v1->v2:
- Moved file rte_cdx_bus.h to internal bus_cdx_driver.h
and added this file to deivce_cdx_headers
- Moved cdx.h to private.h
- Removed rte_ prefix from the static symbols in .c files.
Changes RFC->v1:
- Marked few API's as internal which were not required
to be provided to user.
Nipun Gupta (4):
bus/cdx: introduce AMD CDX bus
bus/cdx: add DMA map and unmap support
bus/cdx: add support for MSI
bus/cdx: support plug unplug and dev iterator
MAINTAINERS | 5 +
doc/guides/rel_notes/release_23_07.rst | 6 +
drivers/bus/cdx/bus_cdx_driver.h | 192 +++++++
drivers/bus/cdx/cdx.c | 666 +++++++++++++++++++++++++
drivers/bus/cdx/cdx_logs.h | 37 ++
drivers/bus/cdx/cdx_vfio.c | 598 ++++++++++++++++++++++
drivers/bus/cdx/meson.build | 13 +
drivers/bus/cdx/private.h | 53 ++
drivers/bus/cdx/version.map | 12 +
drivers/bus/meson.build | 1 +
10 files changed, 1583 insertions(+)
create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
create mode 100644 drivers/bus/cdx/cdx.c
create mode 100644 drivers/bus/cdx/cdx_logs.h
create mode 100644 drivers/bus/cdx/cdx_vfio.c
create mode 100644 drivers/bus/cdx/meson.build
create mode 100644 drivers/bus/cdx/private.h
create mode 100644 drivers/bus/cdx/version.map
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v8 1/4] bus/cdx: introduce AMD CDX bus
2023-06-07 4:24 ` [PATCH v8 0/4] Support AMD CDX bus Nipun Gupta
@ 2023-06-07 4:24 ` Nipun Gupta
2023-06-07 4:24 ` [PATCH v8 2/4] bus/cdx: add DMA map and unmap support Nipun Gupta
` (3 subsequent siblings)
4 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-06-07 4:24 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
AMD CDX bus supports multiple type of devices, which can be
exposed to user-space via vfio-cdx.
vfio-cdx provides the MMIO IO_MEMORY regions as well as the
DMA interface for the device (IOMMU).
This support aims to enable the DPDK to support the cdx
devices in user-space using VFIO interface.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
MAINTAINERS | 5 +
doc/guides/rel_notes/release_23_07.rst | 6 +
drivers/bus/cdx/bus_cdx_driver.h | 166 ++++++++
drivers/bus/cdx/cdx.c | 503 +++++++++++++++++++++++++
drivers/bus/cdx/cdx_logs.h | 37 ++
drivers/bus/cdx/cdx_vfio.c | 426 +++++++++++++++++++++
drivers/bus/cdx/meson.build | 13 +
drivers/bus/cdx/private.h | 53 +++
drivers/bus/cdx/version.map | 10 +
drivers/bus/meson.build | 1 +
10 files changed, 1220 insertions(+)
create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
create mode 100644 drivers/bus/cdx/cdx.c
create mode 100644 drivers/bus/cdx/cdx_logs.h
create mode 100644 drivers/bus/cdx/cdx_vfio.c
create mode 100644 drivers/bus/cdx/meson.build
create mode 100644 drivers/bus/cdx/private.h
create mode 100644 drivers/bus/cdx/version.map
diff --git a/MAINTAINERS b/MAINTAINERS
index a5219926ab..c4b2b3565b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -569,6 +569,11 @@ M: Parav Pandit <parav@nvidia.com>
M: Xueming Li <xuemingl@nvidia.com>
F: drivers/bus/auxiliary/
+AMD CDX bus driver
+M: Nipun Gupta <nipun.gupta@amd.com>
+M: Nikhil Agarwal <nikhil.agarwal@amd.com>
+F: drivers/bus/cdx/
+
Intel FPGA bus
M: Rosen Xu <rosen.xu@intel.com>
F: drivers/bus/ifpga/
diff --git a/doc/guides/rel_notes/release_23_07.rst b/doc/guides/rel_notes/release_23_07.rst
index a9b1293689..7c6bb2b894 100644
--- a/doc/guides/rel_notes/release_23_07.rst
+++ b/doc/guides/rel_notes/release_23_07.rst
@@ -55,6 +55,12 @@ New Features
Also, make sure to start the actual text at the margin.
=======================================================
+* **Added AMD CDX bus support.**
+
+ CDX bus driver has been added to support AMD CDX bus, which operates
+ on FPGA based CDX devices. The CDX devices are memory mapped on system
+ bus for embedded CPUs.
+
Removed Items
-------------
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
new file mode 100644
index 0000000000..9a2be6481a
--- /dev/null
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -0,0 +1,166 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef BUS_CDX_DRIVER_H
+#define BUS_CDX_DRIVER_H
+
+/**
+ * @file
+ * AMD CDX bus interface
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdlib.h>
+#include <inttypes.h>
+
+#include <bus_driver.h>
+#include <dev_driver.h>
+#include <rte_interrupts.h>
+
+/* Forward declarations */
+struct rte_cdx_device;
+struct rte_cdx_driver;
+struct rte_cdx_bus;
+
+#define RTE_CDX_BUS_DEVICES_PATH "/sys/bus/cdx/devices"
+
+#define RTE_CDX_MAX_RESOURCE 4
+
+/** Any CDX device identifier (vendor, device). */
+#define RTE_CDX_ANY_ID (0xffff)
+
+#define RTE_PMD_REGISTER_CDX_TABLE(name, table) \
+static const char DRV_EXP_TAG(name, cdx_tbl_export)[] __rte_used = \
+RTE_STR(table)
+
+/**
+ * A structure describing an ID for a CDX driver. Each driver provides a
+ * table of these IDs for each device that it supports.
+ */
+struct rte_cdx_id {
+ uint16_t vendor_id; /**< Vendor ID. */
+ uint16_t device_id; /**< Device ID. */
+};
+
+/**
+ * A structure describing a CDX device.
+ */
+struct rte_cdx_device {
+ RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
+ struct rte_device device; /**< Inherit core device */
+ char name[RTE_DEV_NAME_MAX_LEN]; /**< Device name */
+ struct rte_cdx_id id; /**< CDX ID. */
+ struct rte_mem_resource mem_resource[RTE_CDX_MAX_RESOURCE];
+ /**< CDX Memory Resource */
+};
+
+/**
+ * @internal
+ * Helper macro for drivers that need to convert to struct rte_cdx_device.
+ */
+#define RTE_DEV_TO_CDX_DEV(ptr) \
+ container_of(ptr, struct rte_cdx_device, device)
+
+#define RTE_DEV_TO_CDX_DEV_CONST(ptr) \
+ container_of(ptr, const struct rte_cdx_device, device)
+
+#define RTE_ETH_DEV_TO_CDX_DEV(eth_dev) RTE_DEV_TO_CDX_DEV((eth_dev)->device)
+
+#ifdef __cplusplus
+/** C++ macro used to help building up tables of device IDs. */
+#define RTE_CDX_DEVICE(vend, dev) \
+ (vend), \
+ (dev)
+#else
+/** Macro used to help building up tables of device IDs. */
+#define RTE_CDX_DEVICE(vend, dev) \
+ .vendor_id = (vend), \
+ .device_id = (dev)
+#endif
+
+/**
+ * Initialisation function for the driver called during CDX probing.
+ */
+typedef int (rte_cdx_probe_t)(struct rte_cdx_driver *, struct rte_cdx_device *);
+
+/**
+ * Uninitialisation function for the driver called during hotplugging.
+ */
+typedef int (rte_cdx_remove_t)(struct rte_cdx_device *);
+
+/**
+ * A structure describing a CDX driver.
+ */
+struct rte_cdx_driver {
+ RTE_TAILQ_ENTRY(rte_cdx_driver) next; /**< Next in list. */
+ struct rte_driver driver; /**< Inherit core driver. */
+ struct rte_cdx_bus *bus; /**< CDX bus reference. */
+ rte_cdx_probe_t *probe; /**< Device probe function. */
+ rte_cdx_remove_t *remove; /**< Device remove function. */
+ const struct rte_cdx_id *id_table; /**< ID table, NULL terminated. */
+ uint32_t drv_flags; /**< Flags RTE_CDX_DRV_*. */
+};
+
+/**
+ * Map the CDX device resources in user space virtual memory address.
+ *
+ * @param dev
+ * A pointer to a rte_cdx_device structure describing the device
+ * to use.
+ *
+ * @return
+ * 0 on success, <0 on error.
+ */
+__rte_internal
+int rte_cdx_map_device(struct rte_cdx_device *dev);
+
+/**
+ * Unmap this device.
+ *
+ * @param dev
+ * A pointer to a rte_cdx_device structure describing the device
+ * to use.
+ */
+__rte_internal
+void rte_cdx_unmap_device(struct rte_cdx_device *dev);
+
+/**
+ * Register a CDX driver.
+ *
+ * @param driver
+ * A pointer to a rte_cdx_driver structure describing the driver
+ * to be registered.
+ */
+__rte_internal
+void rte_cdx_register(struct rte_cdx_driver *driver);
+
+/**
+ * Helper for CDX device registration from driver (eth, crypto, raw) instance.
+ */
+#define RTE_PMD_REGISTER_CDX(nm, cdx_drv) \
+ RTE_INIT(cdxinitfn_ ##nm) \
+ {\
+ (cdx_drv).driver.name = RTE_STR(nm);\
+ rte_cdx_register(&cdx_drv); \
+ } \
+ RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+/**
+ * Unregister a CDX driver.
+ *
+ * @param driver
+ * A pointer to a rte_cdx_driver structure describing the driver
+ * to be unregistered.
+ */
+__rte_internal
+void rte_cdx_unregister(struct rte_cdx_driver *driver);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* BUS_CDX_DRIVER_H */
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
new file mode 100644
index 0000000000..38515e7fda
--- /dev/null
+++ b/drivers/bus/cdx/cdx.c
@@ -0,0 +1,503 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+/*
+ * Architecture Overview
+ * =====================
+ * CDX is a Hardware Architecture designed for AMD FPGA devices. It
+ * consists of sophisticated mechanism for interaction between FPGA,
+ * Firmware and the APUs (Application CPUs).
+ *
+ * Firmware resides on RPU (Realtime CPUs) which interacts with
+ * the FPGA program manager and the APUs. The RPU provides memory-mapped
+ * interface (RPU if) which is used to communicate with APUs.
+ *
+ * The diagram below shows an overview of the AMD CDX architecture:
+ *
+ * +--------------------------------------+
+ * | DPDK |
+ * | DPDK CDX drivers |
+ * | | |
+ * | DPDK AMD CDX bus |
+ * | | |
+ * +-----------------------------|--------+
+ * |
+ * +-----------------------------|--------+
+ * | Application CPUs (APU) | |
+ * | | |
+ * | VFIO CDX driver |
+ * | Linux OS | |
+ * | Linux AMD CDX bus |
+ * | | |
+ * +-----------------------------|--------+
+ * |
+ * |
+ * +------------------------| RPU if |----+
+ * | | |
+ * | V |
+ * | Realtime CPUs (RPU) |
+ * | |
+ * +--------------------------------------+
+ * |
+ * +---------------------|----------------+
+ * | FPGA | |
+ * | +-----------------------+ |
+ * | | | | |
+ * | +-------+ +-------+ +-------+ |
+ * | | dev 1 | | dev 2 | | dev 3 | |
+ * | +-------+ +-------+ +-------+ |
+ * +--------------------------------------+
+ *
+ * The RPU firmware extracts the device information from the loaded FPGA
+ * image and implements a mechanism that allows the APU drivers to
+ * enumerate such devices (device personality and resource details) via
+ * a dedicated communication channel.
+ *
+ * VFIO CDX driver provides the CDX device resources like MMIO and interrupts
+ * to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
+ * driver to discover and initialize the CDX devices for user-space
+ * applications.
+ */
+
+/**
+ * @file
+ * CDX probing using Linux sysfs.
+ */
+
+#include <string.h>
+#include <dirent.h>
+
+#include <rte_eal_paging.h>
+#include <rte_errno.h>
+#include <rte_devargs.h>
+#include <rte_malloc.h>
+#include <rte_vfio.h>
+
+#include <eal_filesystem.h>
+
+#include "bus_cdx_driver.h"
+#include "cdx_logs.h"
+#include "private.h"
+
+#define CDX_BUS_NAME cdx
+#define CDX_DEV_PREFIX "cdx-"
+
+/* CDX Bus iterators */
+#define FOREACH_DEVICE_ON_CDXBUS(p) \
+ RTE_TAILQ_FOREACH(p, &rte_cdx_bus.device_list, next)
+
+#define FOREACH_DRIVER_ON_CDXBUS(p) \
+ RTE_TAILQ_FOREACH(p, &rte_cdx_bus.driver_list, next)
+
+struct rte_cdx_bus rte_cdx_bus;
+
+/* Add a device to CDX bus */
+static void
+cdx_add_device(struct rte_cdx_device *cdx_dev)
+{
+ TAILQ_INSERT_TAIL(&rte_cdx_bus.device_list, cdx_dev, next);
+}
+
+static int
+cdx_get_kernel_driver_by_path(const char *filename, char *driver_name,
+ size_t len)
+{
+ int count;
+ char path[PATH_MAX];
+ char *name;
+
+ if (!filename || !driver_name)
+ return -1;
+
+ count = readlink(filename, path, PATH_MAX);
+ if (count >= PATH_MAX)
+ return -1;
+
+ /* For device does not have a driver */
+ if (count < 0)
+ return 1;
+
+ path[count] = '\0';
+
+ name = strrchr(path, '/');
+ if (name) {
+ strlcpy(driver_name, name + 1, len);
+ return 0;
+ }
+
+ return -1;
+}
+
+int rte_cdx_map_device(struct rte_cdx_device *dev)
+{
+ return cdx_vfio_map_resource(dev);
+}
+
+void rte_cdx_unmap_device(struct rte_cdx_device *dev)
+{
+ cdx_vfio_unmap_resource(dev);
+}
+
+static struct rte_devargs *
+cdx_devargs_lookup(const char *dev_name)
+{
+ struct rte_devargs *devargs;
+
+ RTE_EAL_DEVARGS_FOREACH("cdx", devargs) {
+ if (strcmp(devargs->name, dev_name) == 0)
+ return devargs;
+ }
+ return NULL;
+}
+
+static bool
+cdx_ignore_device(const char *dev_name)
+{
+ struct rte_devargs *devargs = cdx_devargs_lookup(dev_name);
+
+ switch (rte_cdx_bus.bus.conf.scan_mode) {
+ case RTE_BUS_SCAN_ALLOWLIST:
+ if (devargs && devargs->policy == RTE_DEV_ALLOWED)
+ return false;
+ break;
+ case RTE_BUS_SCAN_UNDEFINED:
+ case RTE_BUS_SCAN_BLOCKLIST:
+ if (devargs == NULL || devargs->policy != RTE_DEV_BLOCKED)
+ return false;
+ break;
+ }
+ return true;
+}
+
+/*
+ * Scan one cdx sysfs entry, and fill the devices list from it.
+ * It checks if the CDX device is bound to vfio-cdx driver. In case
+ * the device is vfio bound, it reads the vendor and device id and
+ * stores it for device-driver matching.
+ */
+static int
+cdx_scan_one(const char *dirname, const char *dev_name)
+{
+ char filename[PATH_MAX];
+ struct rte_cdx_device *dev = NULL;
+ char driver[PATH_MAX];
+ unsigned long tmp;
+ int ret;
+
+ dev = calloc(1, sizeof(*dev));
+ if (!dev)
+ return -ENOMEM;
+
+ dev->device.bus = &rte_cdx_bus.bus;
+ memcpy(dev->name, dev_name, RTE_DEV_NAME_MAX_LEN);
+ dev->device.name = dev->name;
+
+ /* parse driver */
+ snprintf(filename, sizeof(filename), "%s/driver", dirname);
+ ret = cdx_get_kernel_driver_by_path(filename, driver, sizeof(driver));
+ if (ret < 0) {
+ CDX_BUS_ERR("Fail to get kernel driver");
+ ret = -1;
+ goto err;
+ }
+
+ /*
+ * Check if device is bound to 'vfio-cdx' driver, so that user-space
+ * can gracefully access the device.
+ */
+ if (ret || strcmp(driver, "vfio-cdx")) {
+ ret = 0;
+ goto err;
+ }
+
+ /* get vendor id */
+ snprintf(filename, sizeof(filename), "%s/vendor", dirname);
+ if (eal_parse_sysfs_value(filename, &tmp) < 0) {
+ ret = -1;
+ goto err;
+ }
+ dev->id.vendor_id = (uint16_t)tmp;
+
+ /* get device id */
+ snprintf(filename, sizeof(filename), "%s/device", dirname);
+ if (eal_parse_sysfs_value(filename, &tmp) < 0) {
+ free(dev);
+ return -1;
+ }
+ dev->id.device_id = (uint16_t)tmp;
+
+ cdx_add_device(dev);
+
+ return 0;
+
+err:
+ free(dev);
+ return ret;
+}
+
+/*
+ * Scan the content of the CDX bus, and the devices in the devices
+ * list.
+ */
+static int
+cdx_scan(void)
+{
+ struct dirent *e;
+ DIR *dir;
+ char dirname[PATH_MAX];
+
+ dir = opendir(RTE_CDX_BUS_DEVICES_PATH);
+ if (dir == NULL) {
+ CDX_BUS_ERR("%s(): opendir failed: %s", __func__,
+ strerror(errno));
+ return -1;
+ }
+
+ while ((e = readdir(dir)) != NULL) {
+ if (e->d_name[0] == '.')
+ continue;
+
+ if (cdx_ignore_device(e->d_name))
+ continue;
+
+ snprintf(dirname, sizeof(dirname), "%s/%s",
+ RTE_CDX_BUS_DEVICES_PATH, e->d_name);
+
+ if (cdx_scan_one(dirname, e->d_name) < 0)
+ goto error;
+ }
+ closedir(dir);
+ return 0;
+
+error:
+ closedir(dir);
+ return -1;
+}
+
+/* map a particular resource from a file */
+void *
+cdx_map_resource(void *requested_addr, int fd, uint64_t offset, size_t size,
+ int additional_flags)
+{
+ void *mapaddr;
+
+ /* Map the cdx MMIO memory resource of device */
+ mapaddr = rte_mem_map(requested_addr, size,
+ RTE_PROT_READ | RTE_PROT_WRITE,
+ RTE_MAP_SHARED | additional_flags, fd, offset);
+ if (mapaddr == NULL) {
+ CDX_BUS_ERR("%s(): cannot map resource(%d, %p, 0x%zx, 0x%"PRIx64"): %s (%p)",
+ __func__, fd, requested_addr, size, offset,
+ rte_strerror(rte_errno), mapaddr);
+ }
+ CDX_BUS_DEBUG("CDX MMIO memory mapped at %p", mapaddr);
+
+ return mapaddr;
+}
+
+/* unmap a particular resource */
+void
+cdx_unmap_resource(void *requested_addr, size_t size)
+{
+ if (requested_addr == NULL)
+ return;
+
+ /* Unmap the CDX memory resource of device */
+ if (rte_mem_unmap(requested_addr, size)) {
+ CDX_BUS_ERR("%s(): cannot mem unmap(%p, %#zx): %s", __func__,
+ requested_addr, size, rte_strerror(rte_errno));
+ }
+ CDX_BUS_DEBUG("CDX memory unmapped at %p", requested_addr);
+}
+/*
+ * Match the CDX Driver and Device using device id and vendor id.
+ */
+static bool
+cdx_match(const struct rte_cdx_driver *cdx_drv,
+ const struct rte_cdx_device *cdx_dev)
+{
+ const struct rte_cdx_id *id_table;
+
+ for (id_table = cdx_drv->id_table; id_table->vendor_id != 0;
+ id_table++) {
+ /* check if device's identifiers match the driver's ones */
+ if (id_table->vendor_id != cdx_dev->id.vendor_id &&
+ id_table->vendor_id != RTE_CDX_ANY_ID)
+ continue;
+ if (id_table->device_id != cdx_dev->id.device_id &&
+ id_table->device_id != RTE_CDX_ANY_ID)
+ continue;
+
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * If vendor id and device id match, call the probe() function of the
+ * driver.
+ */
+static int
+cdx_probe_one_driver(struct rte_cdx_driver *dr,
+ struct rte_cdx_device *dev)
+{
+ const char *dev_name = dev->name;
+ bool already_probed;
+ int ret;
+
+ /* The device is not blocked; Check if driver supports it */
+ if (!cdx_match(dr, dev))
+ /* Match of device and driver failed */
+ return 1;
+
+ already_probed = rte_dev_is_probed(&dev->device);
+ if (already_probed) {
+ CDX_BUS_INFO("Device %s is already probed", dev_name);
+ return -EEXIST;
+ }
+
+ CDX_BUS_DEBUG(" probe device %s using driver: %s", dev_name,
+ dr->driver.name);
+
+ ret = cdx_vfio_map_resource(dev);
+ if (ret != 0) {
+ CDX_BUS_ERR("CDX map device failed: %d", ret);
+ goto error_map_device;
+ }
+
+ /* call the driver probe() function */
+ ret = dr->probe(dr, dev);
+ if (ret) {
+ CDX_BUS_ERR("Probe CDX driver: %s device: %s failed: %d",
+ dr->driver.name, dev_name, ret);
+ goto error_probe;
+ } else {
+ dev->device.driver = &dr->driver;
+ }
+
+ return ret;
+
+error_probe:
+ cdx_vfio_unmap_resource(dev);
+error_map_device:
+ return ret;
+}
+
+/*
+ * If vendor/device ID match, call the probe() function of all
+ * registered driver for the given device. Return < 0 if initialization
+ * failed, return 1 if no driver is found for this device.
+ */
+static int
+cdx_probe_all_drivers(struct rte_cdx_device *dev)
+{
+ struct rte_cdx_driver *dr = NULL;
+ int rc = 0;
+
+ FOREACH_DRIVER_ON_CDXBUS(dr) {
+ rc = cdx_probe_one_driver(dr, dev);
+ if (rc < 0)
+ /* negative value is an error */
+ return rc;
+ if (rc > 0)
+ /* positive value means driver doesn't support it */
+ continue;
+ return 0;
+ }
+ return 1;
+}
+
+/*
+ * Scan the content of the CDX bus, and call the probe() function for
+ * all registered drivers that have a matching entry in its id_table
+ * for discovered devices.
+ */
+static int
+cdx_probe(void)
+{
+ struct rte_cdx_device *dev = NULL;
+ size_t probed = 0, failed = 0;
+ int ret = 0;
+
+ FOREACH_DEVICE_ON_CDXBUS(dev) {
+ probed++;
+
+ ret = cdx_probe_all_drivers(dev);
+ if (ret < 0) {
+ CDX_BUS_ERR("Requested device %s cannot be used",
+ dev->name);
+ rte_errno = errno;
+ failed++;
+ ret = 0;
+ }
+ }
+
+ return (probed && probed == failed) ? -1 : 0;
+}
+
+static int
+cdx_parse(const char *name, void *addr)
+{
+ const char **out = addr;
+ int ret;
+
+ ret = strncmp(name, CDX_DEV_PREFIX, strlen(CDX_DEV_PREFIX));
+
+ if (ret == 0 && addr)
+ *out = name;
+
+ return ret;
+}
+
+/* register a driver */
+void
+rte_cdx_register(struct rte_cdx_driver *driver)
+{
+ TAILQ_INSERT_TAIL(&rte_cdx_bus.driver_list, driver, next);
+ driver->bus = &rte_cdx_bus;
+}
+
+/* unregister a driver */
+void
+rte_cdx_unregister(struct rte_cdx_driver *driver)
+{
+ TAILQ_REMOVE(&rte_cdx_bus.driver_list, driver, next);
+ driver->bus = NULL;
+}
+
+static struct rte_device *
+cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
+ const void *data)
+{
+ const struct rte_cdx_device *cdx_start;
+ struct rte_cdx_device *cdx_dev;
+
+ if (start != NULL) {
+ cdx_start = RTE_DEV_TO_CDX_DEV_CONST(start);
+ cdx_dev = TAILQ_NEXT(cdx_start, next);
+ } else {
+ cdx_dev = TAILQ_FIRST(&rte_cdx_bus.device_list);
+ }
+ while (cdx_dev != NULL) {
+ if (cmp(&cdx_dev->device, data) == 0)
+ return &cdx_dev->device;
+ cdx_dev = TAILQ_NEXT(cdx_dev, next);
+ }
+ return NULL;
+}
+
+struct rte_cdx_bus rte_cdx_bus = {
+ .bus = {
+ .scan = cdx_scan,
+ .probe = cdx_probe,
+ .find_device = cdx_find_device,
+ .parse = cdx_parse,
+ },
+ .device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
+ .driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
+};
+
+RTE_REGISTER_BUS(cdx, rte_cdx_bus.bus);
+RTE_LOG_REGISTER_DEFAULT(cdx_logtype_bus, NOTICE);
diff --git a/drivers/bus/cdx/cdx_logs.h b/drivers/bus/cdx/cdx_logs.h
new file mode 100644
index 0000000000..a1046ce544
--- /dev/null
+++ b/drivers/bus/cdx/cdx_logs.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef CDX_LOGS_H
+#define CDX_LOGS_H
+
+extern int cdx_logtype_bus;
+
+#define CDX_BUS_LOG(level, fmt, args...) \
+ rte_log(RTE_LOG_ ## level, cdx_logtype_bus, "cdx: " fmt "\n", \
+ ##args)
+
+/* Debug logs with Function names */
+#define CDX_BUS_DEBUG(fmt, args...) \
+ rte_log(RTE_LOG_DEBUG, cdx_logtype_bus, "cdx: %s(): " fmt "\n", \
+ __func__, ##args)
+
+#define CDX_BUS_INFO(fmt, args...) \
+ CDX_BUS_LOG(INFO, fmt, ## args)
+#define CDX_BUS_ERR(fmt, args...) \
+ CDX_BUS_LOG(ERR, fmt, ## args)
+#define CDX_BUS_WARN(fmt, args...) \
+ CDX_BUS_LOG(WARNING, fmt, ## args)
+
+/* DP Logs, toggled out at compile time if level lower than current level */
+#define CDX_BUS_DP_LOG(level, fmt, args...) \
+ RTE_LOG_DP(level, PMD, fmt, ## args)
+
+#define CDX_BUS_DP_DEBUG(fmt, args...) \
+ CDX_BUS_DP_LOG(DEBUG, fmt, ## args)
+#define CDX_BUS_DP_INFO(fmt, args...) \
+ CDX_BUS_DP_LOG(INFO, fmt, ## args)
+#define CDX_BUS_DP_WARN(fmt, args...) \
+ CDX_BUS_DP_LOG(WARNING, fmt, ## args)
+
+#endif /* CDX_LOGS_H */
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
new file mode 100644
index 0000000000..86290f0450
--- /dev/null
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -0,0 +1,426 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+/**
+ * @file
+ * CDX probing using Linux VFIO.
+ *
+ * This code tries to determine if the CDX device is bound to VFIO driver,
+ * and initialize it (map MMIO regions, set up interrupts) if that's the case.
+ *
+ */
+
+#include <fcntl.h>
+#include <unistd.h>
+#include <sys/eventfd.h>
+#include <sys/socket.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <rte_eal_paging.h>
+#include <rte_malloc.h>
+#include <rte_vfio.h>
+
+#include "bus_cdx_driver.h"
+#include "cdx_logs.h"
+#include "private.h"
+
+/**
+ * A structure describing a CDX mapping.
+ */
+struct cdx_map {
+ void *addr;
+ char *path;
+ uint64_t offset;
+ uint64_t size;
+};
+
+/**
+ * A structure describing a mapped CDX resource.
+ * For multi-process we need to reproduce all CDX mappings in secondary
+ * processes, so save them in a tailq.
+ */
+struct mapped_cdx_resource {
+ TAILQ_ENTRY(mapped_cdx_resource) next;
+ char name[RTE_DEV_NAME_MAX_LEN]; /**< CDX device name */
+ char path[PATH_MAX];
+ int nb_maps;
+ struct cdx_map maps[RTE_CDX_MAX_RESOURCE];
+};
+
+/** mapped cdx device list */
+TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
+
+static struct rte_tailq_elem cdx_vfio_tailq = {
+ .name = "VFIO_CDX_RESOURCE_LIST",
+};
+EAL_REGISTER_TAILQ(cdx_vfio_tailq)
+
+static struct mapped_cdx_resource *
+cdx_vfio_find_and_unmap_resource(struct mapped_cdx_res_list *vfio_res_list,
+ struct rte_cdx_device *dev)
+{
+ struct mapped_cdx_resource *vfio_res = NULL;
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+ int i;
+
+ /* Get vfio_res */
+ TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+ if (strcmp(vfio_res->name, dev_name))
+ continue;
+ break;
+ }
+
+ if (vfio_res == NULL)
+ return vfio_res;
+
+ CDX_BUS_INFO("Releasing CDX mapped resource for %s", dev_name);
+
+ maps = vfio_res->maps;
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ if (maps[i].addr) {
+ CDX_BUS_DEBUG("Calling cdx_unmap_resource for %s at %p",
+ dev_name, maps[i].addr);
+ cdx_unmap_resource(maps[i].addr, maps[i].size);
+ }
+ }
+
+ return vfio_res;
+}
+
+static int
+cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
+{
+ char cdx_addr[PATH_MAX] = {0};
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list;
+
+ vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ vfio_res = cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for cdx device!",
+ cdx_addr);
+ return -1;
+ }
+
+ TAILQ_REMOVE(vfio_res_list, vfio_res, next);
+ rte_free(vfio_res);
+ return 0;
+}
+
+static int
+cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
+{
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list;
+
+ vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ vfio_res = cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for CDX device!",
+ dev->device.name);
+ return -1;
+ }
+
+ return 0;
+}
+
+int
+cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
+{
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return cdx_vfio_unmap_resource_primary(dev);
+ else
+ return cdx_vfio_unmap_resource_secondary(dev);
+}
+
+static int
+cdx_vfio_setup_device(int vfio_dev_fd)
+{
+ /*
+ * Reset the device. If the device is not capable of resetting,
+ * then it updates errno as EINVAL.
+ */
+ if (ioctl(vfio_dev_fd, VFIO_DEVICE_RESET) && errno != EINVAL) {
+ CDX_BUS_ERR("Unable to reset device! Error: %d (%s)", errno,
+ strerror(errno));
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+cdx_vfio_mmap_resource(int vfio_dev_fd, struct mapped_cdx_resource *vfio_res,
+ int index, int additional_flags)
+{
+ struct cdx_map *map = &vfio_res->maps[index];
+ void *vaddr;
+
+ if (map->size == 0) {
+ CDX_BUS_DEBUG("map size is 0, skip region %d", index);
+ return 0;
+ }
+
+ /* reserve the address using an inaccessible mapping */
+ vaddr = mmap(map->addr, map->size, 0, MAP_PRIVATE |
+ MAP_ANONYMOUS | additional_flags, -1, 0);
+ if (vaddr != MAP_FAILED) {
+ void *map_addr = NULL;
+
+ if (map->size) {
+ /* actual map of first part */
+ map_addr = cdx_map_resource(vaddr, vfio_dev_fd,
+ map->offset, map->size,
+ RTE_MAP_FORCE_ADDRESS);
+ }
+
+ if (map_addr == NULL) {
+ munmap(vaddr, map->size);
+ vaddr = MAP_FAILED;
+ CDX_BUS_ERR("Failed to map cdx MMIO region %d", index);
+ return -1;
+ }
+ } else {
+ CDX_BUS_ERR("Failed to create inaccessible mapping for MMIO region %d",
+ index);
+ return -1;
+ }
+
+ map->addr = vaddr;
+ return 0;
+}
+
+/*
+ * region info may contain capability headers, so we need to keep reallocating
+ * the memory until we match allocated memory size with argsz.
+ */
+static int
+cdx_vfio_get_region_info(int vfio_dev_fd, struct vfio_region_info **info,
+ int region)
+{
+ struct vfio_region_info *ri;
+ size_t argsz = sizeof(*ri);
+ int ret;
+
+ ri = malloc(sizeof(*ri));
+ if (ri == NULL) {
+ CDX_BUS_ERR("Cannot allocate memory for VFIO region info");
+ return -1;
+ }
+again:
+ memset(ri, 0, argsz);
+ ri->argsz = argsz;
+ ri->index = region;
+
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_REGION_INFO, ri);
+ if (ret < 0) {
+ free(ri);
+ return ret;
+ }
+ if (ri->argsz != argsz) {
+ struct vfio_region_info *tmp;
+
+ argsz = ri->argsz;
+ tmp = realloc(ri, argsz);
+
+ if (tmp == NULL) {
+ /* realloc failed but the ri is still there */
+ free(ri);
+ CDX_BUS_ERR("Cannot reallocate memory for VFIO region info");
+ return -1;
+ }
+ ri = tmp;
+ goto again;
+ }
+ *info = ri;
+
+ return 0;
+}
+
+static int
+find_max_end_va(const struct rte_memseg_list *msl, void *arg)
+{
+ size_t sz = msl->len;
+ void *end_va = RTE_PTR_ADD(msl->base_va, sz);
+ void **max_va = arg;
+
+ if (*max_va < end_va)
+ *max_va = end_va;
+ return 0;
+}
+
+static void *
+cdx_find_max_end_va(void)
+{
+ void *va = NULL;
+
+ rte_memseg_list_walk(find_max_end_va, &va);
+ return va;
+}
+
+static int
+cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ char cdx_addr[PATH_MAX] = {0};
+ static void *cdx_map_addr;
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+ int vfio_dev_fd, i, ret;
+
+ ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
+ &vfio_dev_fd, &device_info);
+ if (ret)
+ return ret;
+
+ /* allocate vfio_res and get region info */
+ vfio_res = rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0);
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("Cannot store VFIO mmap details");
+ goto err_vfio_dev_fd;
+ }
+ memcpy(vfio_res->name, dev_name, RTE_DEV_NAME_MAX_LEN);
+
+ /* get number of registers */
+ vfio_res->nb_maps = device_info.num_regions;
+
+ /* map memory regions */
+ maps = vfio_res->maps;
+
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ struct vfio_region_info *reg = NULL;
+ void *vaddr;
+
+ ret = cdx_vfio_get_region_info(vfio_dev_fd, ®, i);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s cannot get device region info error %i (%s)",
+ dev_name, errno, strerror(errno));
+ goto err_vfio_res;
+ }
+
+ /* skip non-mmappable regions */
+ if ((reg->flags & VFIO_REGION_INFO_FLAG_MMAP) == 0) {
+ free(reg);
+ continue;
+ }
+
+ /* try mapping somewhere close to the end of hugepages */
+ if (cdx_map_addr == NULL)
+ cdx_map_addr = cdx_find_max_end_va();
+
+ vaddr = cdx_map_addr;
+ cdx_map_addr = RTE_PTR_ADD(vaddr, (size_t)reg->size);
+
+ cdx_map_addr = RTE_PTR_ALIGN(cdx_map_addr,
+ sysconf(_SC_PAGE_SIZE));
+
+ maps[i].addr = vaddr;
+ maps[i].offset = reg->offset;
+ maps[i].size = reg->size;
+ maps[i].path = NULL; /* vfio doesn't have per-resource paths */
+
+ ret = cdx_vfio_mmap_resource(vfio_dev_fd, vfio_res, i, 0);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s mapping region %i failed: %s",
+ cdx_addr, i, strerror(errno));
+ free(reg);
+ goto err_vfio_res;
+ }
+
+ dev->mem_resource[i].addr = maps[i].addr;
+ dev->mem_resource[i].len = maps[i].size;
+
+ free(reg);
+ }
+
+ if (cdx_vfio_setup_device(vfio_dev_fd) < 0) {
+ CDX_BUS_ERR("%s setup device failed", dev_name);
+ goto err_vfio_res;
+ }
+
+ TAILQ_INSERT_TAIL(vfio_res_list, vfio_res, next);
+
+ return 0;
+err_vfio_res:
+ cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
+ rte_free(vfio_res);
+err_vfio_dev_fd:
+ rte_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev_name, vfio_dev_fd);
+ return -1;
+}
+
+static int
+cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ char cdx_addr[PATH_MAX] = {0};
+ int vfio_dev_fd;
+ int i, ret;
+ struct mapped_cdx_resource *vfio_res = NULL;
+ struct mapped_cdx_res_list *vfio_res_list =
+ RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
+ const char *dev_name = dev->device.name;
+ struct cdx_map *maps;
+
+ /* if we're in a secondary process, just find our tailq entry */
+ TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+ if (strcmp(vfio_res->name, dev_name))
+ continue;
+ break;
+ }
+ /* if we haven't found our tailq entry, something's wrong */
+ if (vfio_res == NULL) {
+ CDX_BUS_ERR("%s cannot find TAILQ entry for cdx device!",
+ dev_name);
+ return -1;
+ }
+
+ ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
+ &vfio_dev_fd, &device_info);
+ if (ret)
+ return ret;
+
+ /* map MMIO regions */
+ maps = vfio_res->maps;
+
+ for (i = 0; i < vfio_res->nb_maps; i++) {
+ ret = cdx_vfio_mmap_resource(vfio_dev_fd, vfio_res, i, MAP_FIXED);
+ if (ret < 0) {
+ CDX_BUS_ERR("%s mapping MMIO region %i failed: %s",
+ dev_name, i, strerror(errno));
+ goto err_vfio_dev_fd;
+ }
+
+ dev->mem_resource[i].addr = maps[i].addr;
+ dev->mem_resource[i].len = maps[i].size;
+ }
+
+ return 0;
+err_vfio_dev_fd:
+ rte_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, cdx_addr, vfio_dev_fd);
+ return -1;
+}
+
+/*
+ * map the CDX resources of a CDX device in virtual memory (VFIO version).
+ * primary and secondary processes follow almost exactly the same path
+ */
+int
+cdx_vfio_map_resource(struct rte_cdx_device *dev)
+{
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return cdx_vfio_map_resource_primary(dev);
+ else
+ return cdx_vfio_map_resource_secondary(dev);
+}
diff --git a/drivers/bus/cdx/meson.build b/drivers/bus/cdx/meson.build
new file mode 100644
index 0000000000..f2ca104d34
--- /dev/null
+++ b/drivers/bus/cdx/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+
+if not is_linux
+ build = false
+ reason = 'only supported on Linux'
+endif
+
+driver_sdk_headers = files('bus_cdx_driver.h')
+sources = files(
+ 'cdx.c',
+ 'cdx_vfio.c',
+)
diff --git a/drivers/bus/cdx/private.h b/drivers/bus/cdx/private.h
new file mode 100644
index 0000000000..81987d0cfe
--- /dev/null
+++ b/drivers/bus/cdx/private.h
@@ -0,0 +1,53 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
+ */
+
+#ifndef CDX_PRIVATE_H
+#define CDX_PRIVATE_H
+
+#include "bus_cdx_driver.h"
+
+/**
+ * Structure describing the CDX bus.
+ */
+struct rte_cdx_bus {
+ struct rte_bus bus; /**< Inherit the generic class */
+ RTE_TAILQ_HEAD(, rte_cdx_device) device_list; /**< List of CDX devices */
+ RTE_TAILQ_HEAD(, rte_cdx_driver) driver_list; /**< List of CDX drivers */
+};
+
+/**
+ * Map a particular resource from a file.
+ *
+ * @param requested_addr
+ * The starting address for the new mapping range.
+ * @param fd
+ * The file descriptor.
+ * @param offset
+ * The offset for the mapping range.
+ * @param size
+ * The size for the mapping range.
+ * @param additional_flags
+ * The additional rte_mem_map() flags for the mapping range.
+ * @return
+ * - On success, the function returns a pointer to the mapped area.
+ * - On error, NULL is returned.
+ */
+void *cdx_map_resource(void *requested_addr, int fd, uint64_t offset,
+ size_t size, int additional_flags);
+
+/**
+ * Unmap a particular resource.
+ *
+ * @param requested_addr
+ * The address for the unmapping range.
+ * @param size
+ * The size for the unmapping range.
+ */
+void cdx_unmap_resource(void *requested_addr, size_t size);
+
+/* map/unmap VFIO resource */
+int cdx_vfio_map_resource(struct rte_cdx_device *dev);
+int cdx_vfio_unmap_resource(struct rte_cdx_device *dev);
+
+#endif /* CDX_PRIVATE_H */
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
new file mode 100644
index 0000000000..360460da18
--- /dev/null
+++ b/drivers/bus/cdx/version.map
@@ -0,0 +1,10 @@
+INTERNAL {
+ global:
+
+ rte_cdx_map_device;
+ rte_cdx_register;
+ rte_cdx_unmap_device;
+ rte_cdx_unregister;
+
+ local: *;
+};
diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
index 6d2520c543..a78b4283bf 100644
--- a/drivers/bus/meson.build
+++ b/drivers/bus/meson.build
@@ -3,6 +3,7 @@
drivers = [
'auxiliary',
+ 'cdx',
'dpaa',
'fslmc',
'ifpga',
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v8 2/4] bus/cdx: add DMA map and unmap support
2023-06-07 4:24 ` [PATCH v8 0/4] Support AMD CDX bus Nipun Gupta
2023-06-07 4:24 ` [PATCH v8 1/4] bus/cdx: introduce " Nipun Gupta
@ 2023-06-07 4:24 ` Nipun Gupta
2023-06-07 4:24 ` [PATCH v8 3/4] bus/cdx: add support for MSI Nipun Gupta
` (2 subsequent siblings)
4 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-06-07 4:24 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
AMD CDX bus can use VFIO interface for mapping and unmapping
of DMA addresses in the IOMMU. This change adds the callback
support for map and unmap APIs as well as fetching the IOMMU
class.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
drivers/bus/cdx/cdx.c | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index 38515e7fda..0c6bac3a44 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -488,12 +488,42 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
return NULL;
}
+static int
+cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+ RTE_SET_USED(dev);
+
+ return rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ (uintptr_t)addr, iova, len);
+}
+
+static int
+cdx_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+ RTE_SET_USED(dev);
+
+ return rte_vfio_container_dma_unmap(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ (uintptr_t)addr, iova, len);
+}
+
+static enum rte_iova_mode
+cdx_get_iommu_class(void)
+{
+ if (TAILQ_EMPTY(&rte_cdx_bus.device_list))
+ return RTE_IOVA_DC;
+
+ return RTE_IOVA_VA;
+}
+
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
.scan = cdx_scan,
.probe = cdx_probe,
.find_device = cdx_find_device,
.parse = cdx_parse,
+ .dma_map = cdx_dma_map,
+ .dma_unmap = cdx_dma_unmap,
+ .get_iommu_class = cdx_get_iommu_class,
},
.device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
.driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v8 3/4] bus/cdx: add support for MSI
2023-06-07 4:24 ` [PATCH v8 0/4] Support AMD CDX bus Nipun Gupta
2023-06-07 4:24 ` [PATCH v8 1/4] bus/cdx: introduce " Nipun Gupta
2023-06-07 4:24 ` [PATCH v8 2/4] bus/cdx: add DMA map and unmap support Nipun Gupta
@ 2023-06-07 4:24 ` Nipun Gupta
2023-06-07 4:24 ` [PATCH v8 4/4] bus/cdx: support plug unplug and dev iterator Nipun Gupta
2023-06-07 13:36 ` [PATCH v8 0/4] Support AMD CDX bus Thomas Monjalon
4 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-06-07 4:24 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
MSI's are exposed to the devices using VFIO (vfio-cdx). This
patch uses the same to add support for MSI for the devices on
the cdx bus.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
drivers/bus/cdx/bus_cdx_driver.h | 25 +++++
drivers/bus/cdx/cdx.c | 11 ++
drivers/bus/cdx/cdx_vfio.c | 176 ++++++++++++++++++++++++++++++-
drivers/bus/cdx/version.map | 2 +
4 files changed, 212 insertions(+), 2 deletions(-)
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index 9a2be6481a..95c266bccb 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -56,6 +56,7 @@ struct rte_cdx_device {
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[RTE_CDX_MAX_RESOURCE];
/**< CDX Memory Resource */
+ struct rte_intr_handle *intr_handle; /**< Interrupt handle */
};
/**
@@ -149,6 +150,30 @@ void rte_cdx_register(struct rte_cdx_driver *driver);
} \
RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+/**
+ * Enables VFIO Interrupts for CDX bus devices.
+ *
+ * @param intr_handle
+ * Pointer to the interrupt handle.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle);
+
+/**
+ * Disable VFIO Interrupts for CDX bus devices.
+ *
+ * @param intr_handle
+ * Pointer to the interrupt handle.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle);
+
/**
* Unregister a CDX driver.
*
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index 0c6bac3a44..88505cfbfb 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -202,6 +202,15 @@ cdx_scan_one(const char *dirname, const char *dev_name)
goto err;
}
+ /* Allocate interrupt instance for cdx device */
+ dev->intr_handle =
+ rte_intr_instance_alloc(RTE_INTR_INSTANCE_F_PRIVATE);
+ if (dev->intr_handle == NULL) {
+ CDX_BUS_ERR("Failed to create interrupt instance for %s",
+ dev->device.name);
+ return -ENOMEM;
+ }
+
/*
* Check if device is bound to 'vfio-cdx' driver, so that user-space
* can gracefully access the device.
@@ -380,6 +389,8 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr,
return ret;
error_probe:
+ rte_intr_instance_free(dev->intr_handle);
+ dev->intr_handle = NULL;
cdx_vfio_unmap_resource(dev);
error_map_device:
return ret;
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 86290f0450..8a3ac0b995 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -51,6 +51,10 @@ struct mapped_cdx_resource {
/** mapped cdx device list */
TAILQ_HEAD(mapped_cdx_res_list, mapped_cdx_resource);
+/* IRQ set buffer length for MSI interrupts */
+#define MSI_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \
+ sizeof(int) * (RTE_MAX_RXTX_INTR_VEC_ID + 1))
+
static struct rte_tailq_elem cdx_vfio_tailq = {
.name = "VFIO_CDX_RESOURCE_LIST",
};
@@ -95,6 +99,27 @@ cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
char cdx_addr[PATH_MAX] = {0};
struct mapped_cdx_resource *vfio_res = NULL;
struct mapped_cdx_res_list *vfio_res_list;
+ int ret, vfio_dev_fd;
+
+ if (rte_intr_fd_get(dev->intr_handle) < 0)
+ return -1;
+
+ if (close(rte_intr_fd_get(dev->intr_handle)) < 0) {
+ CDX_BUS_ERR("Error when closing eventfd file descriptor for %s",
+ dev->device.name);
+ return -1;
+ }
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ ret = rte_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev->device.name,
+ vfio_dev_fd);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot release VFIO device");
+ return ret;
+ }
vfio_res_list =
RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
@@ -117,6 +142,18 @@ cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
{
struct mapped_cdx_resource *vfio_res = NULL;
struct mapped_cdx_res_list *vfio_res_list;
+ int ret, vfio_dev_fd;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ ret = rte_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev->device.name,
+ vfio_dev_fd);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot release VFIO device");
+ return ret;
+ }
vfio_res_list =
RTE_TAILQ_CAST(cdx_vfio_tailq.head, mapped_cdx_res_list);
@@ -141,9 +178,74 @@ cdx_vfio_unmap_resource(struct rte_cdx_device *dev)
return cdx_vfio_unmap_resource_secondary(dev);
}
+/* set up interrupt support (but not enable interrupts) */
static int
-cdx_vfio_setup_device(int vfio_dev_fd)
+cdx_vfio_setup_interrupts(struct rte_cdx_device *dev, int vfio_dev_fd,
+ int num_irqs)
{
+ int i, ret;
+
+ if (num_irqs == 0)
+ return 0;
+
+ /* start from MSI interrupt type */
+ for (i = 0; i < num_irqs; i++) {
+ struct vfio_irq_info irq = { .argsz = sizeof(irq) };
+ int fd = -1;
+
+ irq.index = i;
+
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_IRQ_INFO, &irq);
+ if (ret < 0) {
+ CDX_BUS_ERR("Cannot get VFIO IRQ info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /* if this vector cannot be used with eventfd, fail if we explicitly
+ * specified interrupt type, otherwise continue
+ */
+ if ((irq.flags & VFIO_IRQ_INFO_EVENTFD) == 0)
+ continue;
+
+ /* Set nb_intr to the total number of interrupts */
+ if (rte_intr_event_list_update(dev->intr_handle, irq.count))
+ return -1;
+
+ /* set up an eventfd for interrupts */
+ fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
+ if (fd < 0) {
+ CDX_BUS_ERR("Cannot set up eventfd, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ if (rte_intr_fd_set(dev->intr_handle, fd))
+ return -1;
+
+ /* DPDK CDX bus currently supports only MSI-X */
+ if (rte_intr_type_set(dev->intr_handle, RTE_INTR_HANDLE_VFIO_MSIX))
+ return -1;
+
+ if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
+ return -1;
+
+ return 0;
+ }
+
+ /* if we're here, we haven't found a suitable interrupt vector */
+ return -1;
+}
+
+static int
+cdx_vfio_setup_device(struct rte_cdx_device *dev, int vfio_dev_fd,
+ int num_irqs)
+{
+ if (cdx_vfio_setup_interrupts(dev, vfio_dev_fd, num_irqs) != 0) {
+ CDX_BUS_ERR("Error setting up interrupts!");
+ return -1;
+ }
+
/*
* Reset the device. If the device is not capable of resetting,
* then it updates errno as EINVAL.
@@ -279,6 +381,9 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
struct cdx_map *maps;
int vfio_dev_fd, i, ret;
+ if (rte_intr_fd_set(dev->intr_handle, -1))
+ return -1;
+
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
&vfio_dev_fd, &device_info);
if (ret)
@@ -344,7 +449,7 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
free(reg);
}
- if (cdx_vfio_setup_device(vfio_dev_fd) < 0) {
+ if (cdx_vfio_setup_device(dev, vfio_dev_fd, device_info.num_irqs) < 0) {
CDX_BUS_ERR("%s setup device failed", dev_name);
goto err_vfio_res;
}
@@ -373,6 +478,9 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
const char *dev_name = dev->device.name;
struct cdx_map *maps;
+ if (rte_intr_fd_set(dev->intr_handle, -1))
+ return -1;
+
/* if we're in a secondary process, just find our tailq entry */
TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
if (strcmp(vfio_res->name, dev_name))
@@ -406,6 +514,10 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
dev->mem_resource[i].len = maps[i].size;
}
+ /* we need save vfio_dev_fd, so it can be used during release */
+ if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
+ goto err_vfio_dev_fd;
+
return 0;
err_vfio_dev_fd:
rte_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, cdx_addr, vfio_dev_fd);
@@ -424,3 +536,63 @@ cdx_vfio_map_resource(struct rte_cdx_device *dev)
else
return cdx_vfio_map_resource_secondary(dev);
}
+
+int
+rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle)
+{
+ char irq_set_buf[MSI_IRQ_SET_BUF_LEN];
+ struct vfio_irq_set *irq_set;
+ int *fd_ptr, vfio_dev_fd, i;
+ int ret;
+
+ irq_set = (struct vfio_irq_set *) irq_set_buf;
+ irq_set->count = rte_intr_nb_intr_get(intr_handle);
+ irq_set->argsz = sizeof(struct vfio_irq_set) +
+ (sizeof(int) * irq_set->count);
+
+ irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER;
+ irq_set->index = 0;
+ irq_set->start = 0;
+ fd_ptr = (int *) &irq_set->data;
+
+ for (i = 0; i < rte_intr_nb_efd_get(intr_handle); i++)
+ fd_ptr[i] = rte_intr_efds_index_get(intr_handle, i);
+
+ vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
+
+ if (ret) {
+ CDX_BUS_ERR("Error enabling MSI interrupts for fd %d",
+ rte_intr_fd_get(intr_handle));
+ return -1;
+ }
+
+ return 0;
+}
+
+/* disable MSI interrupts */
+int
+rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle)
+{
+ struct vfio_irq_set *irq_set;
+ char irq_set_buf[MSI_IRQ_SET_BUF_LEN];
+ int len, ret, vfio_dev_fd;
+
+ len = sizeof(struct vfio_irq_set);
+
+ irq_set = (struct vfio_irq_set *) irq_set_buf;
+ irq_set->argsz = len;
+ irq_set->count = 0;
+ irq_set->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_TRIGGER;
+ irq_set->index = 0;
+ irq_set->start = 0;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
+
+ if (ret)
+ CDX_BUS_ERR("Error disabling MSI interrupts for fd %d",
+ rte_intr_fd_get(intr_handle));
+
+ return ret;
+}
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
index 360460da18..0a15d39ae8 100644
--- a/drivers/bus/cdx/version.map
+++ b/drivers/bus/cdx/version.map
@@ -5,6 +5,8 @@ INTERNAL {
rte_cdx_register;
rte_cdx_unmap_device;
rte_cdx_unregister;
+ rte_cdx_vfio_intr_disable;
+ rte_cdx_vfio_intr_enable;
local: *;
};
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* [PATCH v8 4/4] bus/cdx: support plug unplug and dev iterator
2023-06-07 4:24 ` [PATCH v8 0/4] Support AMD CDX bus Nipun Gupta
` (2 preceding siblings ...)
2023-06-07 4:24 ` [PATCH v8 3/4] bus/cdx: add support for MSI Nipun Gupta
@ 2023-06-07 4:24 ` Nipun Gupta
2023-06-07 13:36 ` [PATCH v8 0/4] Support AMD CDX bus Thomas Monjalon
4 siblings, 0 replies; 102+ messages in thread
From: Nipun Gupta @ 2023-06-07 4:24 UTC (permalink / raw)
To: dev, thomas, david.marchand, hkalra, anatoly.burakov, stephen
Cc: ferruh.yigit, harpreet.anand, nikhil.agarwal, Nipun Gupta
This change adds support for plugging and unplugging
CDX devices on AMD CDX bus. Also, CDX dev iterator support
has been added for the CDX bus.
Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
drivers/bus/cdx/bus_cdx_driver.h | 1 +
drivers/bus/cdx/cdx.c | 122 +++++++++++++++++++++++++++++++
2 files changed, 123 insertions(+)
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index 95c266bccb..fcacdb5896 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -52,6 +52,7 @@ struct rte_cdx_id {
struct rte_cdx_device {
RTE_TAILQ_ENTRY(rte_cdx_device) next; /**< Next probed CDX device. */
struct rte_device device; /**< Inherit core device */
+ struct rte_cdx_driver *driver; /**< CDX driver used in probing */
char name[RTE_DEV_NAME_MAX_LEN]; /**< Device name */
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[RTE_CDX_MAX_RESOURCE];
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index 88505cfbfb..9607096d18 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -71,6 +71,7 @@
#include <rte_eal_paging.h>
#include <rte_errno.h>
#include <rte_devargs.h>
+#include <rte_kvargs.h>
#include <rte_malloc.h>
#include <rte_vfio.h>
@@ -92,6 +93,15 @@
struct rte_cdx_bus rte_cdx_bus;
+enum cdx_params {
+ RTE_CDX_PARAM_NAME,
+};
+
+static const char * const cdx_params_keys[] = {
+ [RTE_CDX_PARAM_NAME] = "name",
+ NULL,
+};
+
/* Add a device to CDX bus */
static void
cdx_add_device(struct rte_cdx_device *cdx_dev)
@@ -385,6 +395,7 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr,
} else {
dev->device.driver = &dr->driver;
}
+ dev->driver = dr;
return ret;
@@ -499,6 +510,71 @@ cdx_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
return NULL;
}
+/* Remove a device from CDX bus */
+static void
+cdx_remove_device(struct rte_cdx_device *cdx_dev)
+{
+ TAILQ_REMOVE(&rte_cdx_bus.device_list, cdx_dev, next);
+}
+
+/*
+ * If vendor/device ID match, call the remove() function of the
+ * driver.
+ */
+static int
+cdx_detach_dev(struct rte_cdx_device *dev)
+{
+ struct rte_cdx_driver *dr;
+ int ret = 0;
+
+ if (dev == NULL)
+ return -EINVAL;
+
+ dr = dev->driver;
+
+ CDX_BUS_DEBUG("detach device %s using driver: %s",
+ dev->device.name, dr->driver.name);
+
+ if (dr->remove) {
+ ret = dr->remove(dev);
+ if (ret < 0)
+ return ret;
+ }
+
+ /* clear driver structure */
+ dev->driver = NULL;
+ dev->device.driver = NULL;
+
+ rte_cdx_unmap_device(dev);
+
+ rte_intr_instance_free(dev->intr_handle);
+ dev->intr_handle = NULL;
+
+ return 0;
+}
+
+static int
+cdx_plug(struct rte_device *dev)
+{
+ return cdx_probe_all_drivers(RTE_DEV_TO_CDX_DEV(dev));
+}
+
+static int
+cdx_unplug(struct rte_device *dev)
+{
+ struct rte_cdx_device *cdx_dev;
+ int ret;
+
+ cdx_dev = RTE_DEV_TO_CDX_DEV(dev);
+ ret = cdx_detach_dev(cdx_dev);
+ if (ret == 0) {
+ cdx_remove_device(cdx_dev);
+ rte_devargs_remove(dev->devargs);
+ free(cdx_dev);
+ }
+ return ret;
+}
+
static int
cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
{
@@ -526,15 +602,61 @@ cdx_get_iommu_class(void)
return RTE_IOVA_VA;
}
+static int
+cdx_dev_match(const struct rte_device *dev,
+ const void *_kvlist)
+{
+ const struct rte_kvargs *kvlist = _kvlist;
+ const char *key = cdx_params_keys[RTE_CDX_PARAM_NAME];
+ const char *name;
+
+ /* no kvlist arg, all devices match */
+ if (kvlist == NULL)
+ return 0;
+
+ /* if key is present in kvlist and does not match, filter device */
+ name = rte_kvargs_get(kvlist, key);
+ if (name != NULL && strcmp(name, dev->name))
+ return -1;
+
+ return 0;
+}
+
+static void *
+cdx_dev_iterate(const void *start,
+ const char *str,
+ const struct rte_dev_iterator *it __rte_unused)
+{
+ rte_bus_find_device_t find_device;
+ struct rte_kvargs *kvargs = NULL;
+ struct rte_device *dev;
+
+ if (str != NULL) {
+ kvargs = rte_kvargs_parse(str, cdx_params_keys);
+ if (kvargs == NULL) {
+ CDX_BUS_ERR("cannot parse argument list %s", str);
+ rte_errno = EINVAL;
+ return NULL;
+ }
+ }
+ find_device = rte_cdx_bus.bus.find_device;
+ dev = find_device(start, cdx_dev_match, kvargs);
+ rte_kvargs_free(kvargs);
+ return dev;
+}
+
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
.scan = cdx_scan,
.probe = cdx_probe,
.find_device = cdx_find_device,
+ .plug = cdx_plug,
+ .unplug = cdx_unplug,
.parse = cdx_parse,
.dma_map = cdx_dma_map,
.dma_unmap = cdx_dma_unmap,
.get_iommu_class = cdx_get_iommu_class,
+ .dev_iterate = cdx_dev_iterate,
},
.device_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.device_list),
.driver_list = TAILQ_HEAD_INITIALIZER(rte_cdx_bus.driver_list),
--
2.17.1
^ permalink raw reply [flat|nested] 102+ messages in thread* Re: [PATCH v8 0/4] Support AMD CDX bus
2023-06-07 4:24 ` [PATCH v8 0/4] Support AMD CDX bus Nipun Gupta
` (3 preceding siblings ...)
2023-06-07 4:24 ` [PATCH v8 4/4] bus/cdx: support plug unplug and dev iterator Nipun Gupta
@ 2023-06-07 13:36 ` Thomas Monjalon
4 siblings, 0 replies; 102+ messages in thread
From: Thomas Monjalon @ 2023-06-07 13:36 UTC (permalink / raw)
To: Nipun Gupta
Cc: dev, david.marchand, hkalra, anatoly.burakov, stephen,
ferruh.yigit, harpreet.anand, nikhil.agarwal
07/06/2023 06:24, Nipun Gupta:
> The CDX devices are memory mapped on system bus for embedded CPUs.
>
> It uses sysfs interface and the vfio-cdx driver to discover
> and initialize the CDX devices.
>
> The CDX bus and VFIO support is available at Xilinx/AMD open source tree:
> https://github.com/Xilinx/linux-xlnx (drivers/cdx/ and drivers/vfio/cdx)
>
> Linux AMD CDX bus patches has been added into linux tree:
> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/cdx
>
> VFIO patches are also submitted in upstream:
> https://www.spinics.net/lists/kvm/msg310623.html
>
> CDX is a Hardware Architecture designed for AMD FPGA devices. It
> consists of mechanism for interaction between FPGA, Firmware and
> the APUs (Application CPUs).
> Firmware resides on RPU (Realtime CPUs) which interacts with
> the FPGA program manager and the APUs. The RPU provides memory-mapped
> interface (RPU if) which is used to communicate with APUs.
>
> VFIO CDX driver provides the CDX device resources like MMIO and interrupts
> to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx
> driver to discover and initialize the CDX devices for user-space
> applications.
Applied, thanks.
^ permalink raw reply [flat|nested] 102+ messages in thread