From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 37F68A0C43; Sun, 4 Jul 2021 18:13:20 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D5C3940686; Sun, 4 Jul 2021 18:13:19 +0200 (CEST) Received: from shelob.oktetlabs.ru (shelob.oktetlabs.ru [91.220.146.113]) by mails.dpdk.org (Postfix) with ESMTP id B010540040 for ; Sun, 4 Jul 2021 18:13:17 +0200 (CEST) Received: from [192.168.38.17] (aros.oktetlabs.ru [192.168.38.17]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by shelob.oktetlabs.ru (Postfix) with ESMTPSA id 23A657F4E2; Sun, 4 Jul 2021 19:13:17 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 shelob.oktetlabs.ru 23A657F4E2 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=oktetlabs.ru; s=default; t=1625415197; bh=TB7K7NXAOzGTkqFK4iFS4hnXLw2V2ziPlKSLbgIeldQ=; h=Subject:To:Cc:References:From:Date:In-Reply-To; b=k8WPnmfwu4j097+MzkjsUvTW+MyBR4zW9Fay7CwunYAVg+yiRW89K+HAKPuDbfn4b JP46zDhxnGfzFafl1DLyeGhB2LC9wTqXE45vV1ky2iPidn3swcoinMtEpEc1hzaYMQ +D9OtwMXUea51KwY/ASYhrQ6pIaO7IXtYPEHRVGs= To: Xueming Li Cc: dev@dpdk.org, Wang Haiyue , Thomas Monjalon , Kinsella Ray , Neil Horman References: <20210623000349.631468-2-xuemingl@nvidia.com> <20210625114726.776425-2-xuemingl@nvidia.com> From: Andrew Rybchenko Organization: OKTET Labs Message-ID: <68c206e6-f090-b303-ff12-fa09ac80f5fd@oktetlabs.ru> Date: Sun, 4 Jul 2021 19:13:16 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <20210625114726.776425-2-xuemingl@nvidia.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH v6 2/2] bus/auxiliary: introduce auxiliary bus X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 6/25/21 2:47 PM, Xueming Li wrote: > Auxiliary bus [1] provides a way to split function into child-devices > representing sub-domains of functionality. Each auxiliary device > represents a part of its parent functionality. > > Auxiliary device is identified by unique device name, sysfs path: > /sys/bus/auxiliary/devices/ > > Devargs legacy syntax ofauxiliary device: Missing space after 'of' > -a auxiliary:[,args...] > Devargs generic syntax of auxiliary device: > -a bus=auxiliary,name=,,/class=,,/driver=,, Are two commas above intentionall? What for? > > [1] kernel auxiliary bus document: > https://www.kernel.org/doc/html/latest/driver-api/auxiliary_bus.html > > Signed-off-by: Xueming Li With my below notes fixed: Acked-by: Andrew Rybchenko > Cc: Wang Haiyue > Cc: Thomas Monjalon > Cc: Kinsella Ray > --- > MAINTAINERS | 5 + > doc/guides/rel_notes/release_21_08.rst | 6 + > drivers/bus/auxiliary/auxiliary_common.c | 411 ++++++++++++++++++++++ > drivers/bus/auxiliary/auxiliary_params.c | 59 ++++ > drivers/bus/auxiliary/linux/auxiliary.c | 141 ++++++++ > drivers/bus/auxiliary/meson.build | 16 + > drivers/bus/auxiliary/private.h | 74 ++++ > drivers/bus/auxiliary/rte_bus_auxiliary.h | 201 +++++++++++ > drivers/bus/auxiliary/version.map | 7 + > drivers/bus/meson.build | 1 + > 10 files changed, 921 insertions(+) > create mode 100644 drivers/bus/auxiliary/auxiliary_common.c > create mode 100644 drivers/bus/auxiliary/auxiliary_params.c > create mode 100644 drivers/bus/auxiliary/linux/auxiliary.c > create mode 100644 drivers/bus/auxiliary/meson.build > create mode 100644 drivers/bus/auxiliary/private.h > create mode 100644 drivers/bus/auxiliary/rte_bus_auxiliary.h > create mode 100644 drivers/bus/auxiliary/version.map > > diff --git a/MAINTAINERS b/MAINTAINERS > index 5877a16971..eaf691ca6a 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -525,6 +525,11 @@ F: doc/guides/mempool/octeontx2.rst > Bus Drivers > ----------- > > +Auxiliary bus driver Shouldn't it be EXPERIMENTAL? > +M: Parav Pandit > +M: Xueming Li > +F: drivers/bus/auxiliary/ > + > Intel FPGA bus > M: Rosen Xu > F: drivers/bus/ifpga/ > diff --git a/doc/guides/rel_notes/release_21_08.rst b/doc/guides/rel_notes/release_21_08.rst > index a6ecfdf3ce..e7ef4c8a05 100644 > --- a/doc/guides/rel_notes/release_21_08.rst > +++ b/doc/guides/rel_notes/release_21_08.rst > @@ -55,6 +55,12 @@ New Features > Also, make sure to start the actual text at the margin. > ======================================================= > > +* **Added auxiliary bus support.** > + > + Auxiliary bus provides a way to split function into child-devices > + representing sub-domains of functionality. Each auxiliary device > + represents a part of its parent functionality. > + > > Removed Items > ------------- > diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c > new file mode 100644 > index 0000000000..8a75306da5 > --- /dev/null > +++ b/drivers/bus/auxiliary/auxiliary_common.c > @@ -0,0 +1,411 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright (c) 2021 NVIDIA Corporation & Affiliates > + */ > + > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > + > +#include "private.h" > +#include "rte_bus_auxiliary.h" > + > +static struct rte_devargs * > +auxiliary_devargs_lookup(const char *name) > +{ > + struct rte_devargs *devargs; > + > + RTE_EAL_DEVARGS_FOREACH(RTE_BUS_AUXILIARY_NAME, devargs) { > + if (strcmp(devargs->name, name) == 0) > + return devargs; > + } > + return NULL; > +} > + > +/* > + * Test whether the auxiliary device exist Missing full stop above. > + * > + * Stub for OS not supporting auxiliary bus. > + */ > +__rte_weak bool > +auxiliary_dev_exists(const char *name) > +{ > + RTE_SET_USED(name); > + return false; > +} > + > +/* > + * Scan the devices in the auxiliary bus. > + * > + * Stub for OS not supporting auxiliary bus. > + */ > +__rte_weak int > +auxiliary_scan(void) > +{ > + return 0; > +} > + > +/* > + * Update a device's devargs being scanned. > + * > + * @param aux_dev > + * AUXILIARY device. > + */ > +void > +auxiliary_on_scan(struct rte_auxiliary_device *aux_dev) > +{ > + aux_dev->device.devargs = auxiliary_devargs_lookup(aux_dev->name); > +} > + > +/* > + * Match the auxiliary driver and device using driver function. > + */ > +bool > +auxiliary_match(const struct rte_auxiliary_driver *aux_drv, > + const struct rte_auxiliary_device *aux_dev) > +{ > + if (aux_drv->match == NULL) > + return false; > + return aux_drv->match(aux_dev->name); > +} > + > +/* > + * Call the probe() function of the driver. > + */ > +static int > +rte_auxiliary_probe_one_driver(struct rte_auxiliary_driver *drv, > + struct rte_auxiliary_device *dev) > +{ > + enum rte_iova_mode iova_mode; > + int ret; > + > + if ((drv == NULL) || (dev == NULL)) Unnecessary internal parenthesis. > + return -EINVAL; > + > + /* Check if driver supports it. */ > + if (!auxiliary_match(drv, dev)) > + /* Match of device and driver failed */ > + return 1; > + > + /* No initialization when marked as blocked, return without error. */ > + if (dev->device.devargs != NULL && > + dev->device.devargs->policy == RTE_DEV_BLOCKED) { > + AUXILIARY_LOG(INFO, "Device is blocked, not initializing"); > + return -1; > + } > + > + if (dev->device.numa_node < 0) { > + AUXILIARY_LOG(INFO, "Device is not NUMA-aware, defaulting socket to 0"); socket -> NUMA node > + dev->device.numa_node = 0; > + } > + > + iova_mode = rte_eal_iova_mode(); > + if ((drv->drv_flags & RTE_AUXILIARY_DRV_NEED_IOVA_AS_VA) > 0 && > + iova_mode != RTE_IOVA_VA) { > + AUXILIARY_LOG(ERR, "Driver %s expecting VA IOVA mode but current mode is PA, not initializing", > + drv->driver.name); > + return -EINVAL; > + } > + > + dev->driver = drv; > + > + AUXILIARY_LOG(INFO, "Probe auxiliary driver: %s device: %s (socket %i)", socket -> NUMA node > + drv->driver.name, dev->name, dev->device.numa_node); > + ret = drv->probe(drv, dev); > + if (ret != 0) > + dev->driver = NULL; > + else > + dev->device.driver = &drv->driver; > + > + return ret; > +} > + > +/* > + * Call the remove() function of the driver. > + */ > +static int > +rte_auxiliary_driver_remove_dev(struct rte_auxiliary_device *dev) > +{ > + struct rte_auxiliary_driver *drv; > + int ret = 0; > + > + if (dev == NULL) > + return -EINVAL; > + > + drv = dev->driver; > + > + AUXILIARY_LOG(DEBUG, "Driver %s remove auxiliary device %s on NUMA socket %i", socket -> node > + drv->driver.name, dev->name, dev->device.numa_node); > + > + if (drv->remove != NULL) { > + ret = drv->remove(dev); > + if (ret < 0) > + return ret; > + } > + > + /* clear driver structure */ > + dev->driver = NULL; > + dev->device.driver = NULL; > + > + return 0; > +} > + > +/* > + * Call the probe() function of all registered driver for the given device. > + * Return < 0 if initialization failed. > + * Return 1 if no driver is found for this device. > + */ > +static int > +auxiliary_probe_all_drivers(struct rte_auxiliary_device *dev) > +{ > + struct rte_auxiliary_driver *drv; > + int rc; > + > + if (dev == NULL) > + return -EINVAL; > + > + FOREACH_DRIVER_ON_AUXILIARY_BUS(drv) { > + if (!drv->match(dev->name)) > + continue; > + > + rc = rte_auxiliary_probe_one_driver(drv, dev); > + if (rc < 0) > + /* negative value is an error */ > + return rc; > + if (rc > 0) > + /* positive value means driver doesn't support it */ > + continue; > + return 0; > + } > + return 1; > +} > + > +/* > + * Scan the content of the auxiliary bus, and call the probe function for > + * all registered drivers to try to probe discovered devices. > + */ > +static int > +auxiliary_probe(void) > +{ > + struct rte_auxiliary_device *dev = NULL; > + size_t probed = 0, failed = 0; > + int ret = 0; > + > + FOREACH_DEVICE_ON_AUXILIARY_BUS(dev) { > + probed++; > + > + ret = auxiliary_probe_all_drivers(dev); > + if (ret < 0) { > + if (ret != -EEXIST) { > + AUXILIARY_LOG(ERR, "Requested device %s cannot be used", > + dev->name); > + rte_errno = errno; > + failed++; > + } > + ret = 0; > + } > + } > + > + return (probed && probed == failed) ? -1 : 0; > +} > + > +static int > +auxiliary_parse(const char *name, void *addr) > +{ > + struct rte_auxiliary_driver *drv = NULL; > + const char **out = addr; > + > + /* Allow empty device name "auxiliary:" to bypass entire bus scan. */ > + if (strlen(name) == 0) > + return 0; > + > + FOREACH_DRIVER_ON_AUXILIARY_BUS(drv) { > + if (drv->match(name)) > + break; > + } > + if (drv != NULL && addr != NULL) > + *out = name; > + return drv != NULL ? 0 : -1; > +} > + > +/* Register a driver */ > +void > +rte_auxiliary_register(struct rte_auxiliary_driver *driver) > +{ > + TAILQ_INSERT_TAIL(&auxiliary_bus.driver_list, driver, next); > + driver->bus = &auxiliary_bus; > +} > + > +/* Unregister a driver */ > +void > +rte_auxiliary_unregister(struct rte_auxiliary_driver *driver) > +{ > + TAILQ_REMOVE(&auxiliary_bus.driver_list, driver, next); > + driver->bus = NULL; > +} > + > +/* Add a device to auxiliary bus */ > +void > +auxiliary_add_device(struct rte_auxiliary_device *aux_dev) > +{ > + TAILQ_INSERT_TAIL(&auxiliary_bus.device_list, aux_dev, next); > +} > + > +/* Insert a device into a predefined position in auxiliary bus */ > +void > +auxiliary_insert_device(struct rte_auxiliary_device *exist_aux_dev, > + struct rte_auxiliary_device *new_aux_dev) > +{ > + TAILQ_INSERT_BEFORE(exist_aux_dev, new_aux_dev, next); > +} > + > +/* Remove a device from auxiliary bus */ > +static void > +rte_auxiliary_remove_device(struct rte_auxiliary_device *auxiliary_dev) > +{ > + TAILQ_REMOVE(&auxiliary_bus.device_list, auxiliary_dev, next); > +} > + > +static struct rte_device * > +auxiliary_find_device(const struct rte_device *start, rte_dev_cmp_t cmp, > + const void *data) > +{ > + const struct rte_auxiliary_device *pstart; > + struct rte_auxiliary_device *adev; > + > + if (start != NULL) { > + pstart = RTE_DEV_TO_AUXILIARY_CONST(start); > + adev = TAILQ_NEXT(pstart, next); > + } else { > + adev = TAILQ_FIRST(&auxiliary_bus.device_list); > + } > + while (adev != NULL) { > + if (cmp(&adev->device, data) == 0) > + return &adev->device; > + adev = TAILQ_NEXT(adev, next); > + } > + return NULL; > +} > + > +static int > +auxiliary_plug(struct rte_device *dev) > +{ > + if (!auxiliary_dev_exists(dev->name)) > + return -ENOENT; > + return auxiliary_probe_all_drivers(RTE_DEV_TO_AUXILIARY(dev)); > +} > + > +static int > +auxiliary_unplug(struct rte_device *dev) > +{ > + struct rte_auxiliary_device *adev; > + int ret; > + > + adev = RTE_DEV_TO_AUXILIARY(dev); > + ret = rte_auxiliary_driver_remove_dev(adev); > + if (ret == 0) { > + rte_auxiliary_remove_device(adev); > + rte_devargs_remove(dev->devargs); > + free(adev); > + } > + return ret; > +} > + > +static int > +auxiliary_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len) > +{ > + struct rte_auxiliary_device *aux_dev = RTE_DEV_TO_AUXILIARY(dev); > + > + if (dev == NULL || aux_dev->driver == NULL) { > + rte_errno = EINVAL; > + return -1; > + } > + if (aux_dev->driver->dma_map == NULL) { > + rte_errno = ENOTSUP; > + return -1; > + } > + return aux_dev->driver->dma_map(aux_dev, addr, iova, len); > +} > + > +static int > +auxiliary_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, > + size_t len) > +{ > + struct rte_auxiliary_device *aux_dev = RTE_DEV_TO_AUXILIARY(dev); > + > + if (dev == NULL || aux_dev->driver == NULL) { > + rte_errno = EINVAL; > + return -1; > + } > + if (aux_dev->driver->dma_unmap == NULL) { > + rte_errno = ENOTSUP; > + return -1; > + } > + return aux_dev->driver->dma_unmap(aux_dev, addr, iova, len); > +} > + > +bool > +auxiliary_is_ignored_device(const char *name) > +{ > + struct rte_devargs *devargs = auxiliary_devargs_lookup(name); > + > + switch (auxiliary_bus.bus.conf.scan_mode) { > + case RTE_BUS_SCAN_ALLOWLIST: > + if (devargs && devargs->policy == RTE_DEV_ALLOWED) > + return false; > + break; > + case RTE_BUS_SCAN_UNDEFINED: > + case RTE_BUS_SCAN_BLOCKLIST: > + if (devargs == NULL || devargs->policy != RTE_DEV_BLOCKED) > + return false; > + break; > + } > + return true; > +} > + > +static enum rte_iova_mode > +auxiliary_get_iommu_class(void) > +{ > + const struct rte_auxiliary_driver *drv; > + > + FOREACH_DRIVER_ON_AUXILIARY_BUS(drv) { > + if ((drv->drv_flags & RTE_AUXILIARY_DRV_NEED_IOVA_AS_VA) > 0) > + return RTE_IOVA_VA; > + } > + > + return RTE_IOVA_DC; > +} > + > +struct rte_auxiliary_bus auxiliary_bus = { > + .bus = { > + .scan = auxiliary_scan, > + .probe = auxiliary_probe, > + .find_device = auxiliary_find_device, > + .plug = auxiliary_plug, > + .unplug = auxiliary_unplug, > + .parse = auxiliary_parse, > + .dma_map = auxiliary_dma_map, > + .dma_unmap = auxiliary_dma_unmap, > + .get_iommu_class = auxiliary_get_iommu_class, > + .dev_iterate = auxiliary_dev_iterate, > + }, > + .device_list = TAILQ_HEAD_INITIALIZER(auxiliary_bus.device_list), > + .driver_list = TAILQ_HEAD_INITIALIZER(auxiliary_bus.driver_list), > +}; > + > +RTE_REGISTER_BUS(auxiliary, auxiliary_bus.bus); > +RTE_LOG_REGISTER_DEFAULT(auxiliary_bus_logtype, NOTICE); > diff --git a/drivers/bus/auxiliary/auxiliary_params.c b/drivers/bus/auxiliary/auxiliary_params.c > new file mode 100644 > index 0000000000..cd3fa56cb4 > --- /dev/null > +++ b/drivers/bus/auxiliary/auxiliary_params.c > @@ -0,0 +1,59 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright (c) 2021 NVIDIA Corporation & Affiliates > + */ > + > +#include > + > +#include > +#include > +#include > +#include > + > +#include "private.h" > +#include "rte_bus_auxiliary.h" > + > +enum auxiliary_params { > + RTE_AUXILIARY_PARAM_NAME, > +}; > + > +static const char * const auxiliary_params_keys[] = { > + [RTE_AUXILIARY_PARAM_NAME] = "name", > +}; > + > +static int > +auxiliary_dev_match(const struct rte_device *dev, > + const void *_kvlist) > +{ > + const struct rte_kvargs *kvlist = _kvlist; > + int ret; > + > + ret = rte_kvargs_process(kvlist, > + auxiliary_params_keys[RTE_AUXILIARY_PARAM_NAME], > + rte_kvargs_strcmp, (void *)(uintptr_t)dev->name); > + > + return ret != 0 ? -1 : 0; > +} > + > +void * > +auxiliary_dev_iterate(const void *start, > + const char *str, > + const struct rte_dev_iterator *it __rte_unused) > +{ > + rte_bus_find_device_t find_device; > + struct rte_kvargs *kvargs = NULL; > + struct rte_device *dev; > + > + if (str != NULL) { > + kvargs = rte_kvargs_parse(str, auxiliary_params_keys); > + if (kvargs == NULL) { > + AUXILIARY_LOG(ERR, "cannot parse argument list %s", > + str); > + rte_errno = EINVAL; > + return NULL; > + } > + } > + find_device = auxiliary_bus.bus.find_device; > + dev = find_device(start, auxiliary_dev_match, kvargs); > + rte_kvargs_free(kvargs); > + return dev; > +} > diff --git a/drivers/bus/auxiliary/linux/auxiliary.c b/drivers/bus/auxiliary/linux/auxiliary.c > new file mode 100644 > index 0000000000..8464487971 > --- /dev/null > +++ b/drivers/bus/auxiliary/linux/auxiliary.c > @@ -0,0 +1,141 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright (c) 2021 NVIDIA Corporation & Affiliates > + */ > + > +#include > +#include > + > +#include > +#include > +#include > +#include > +#include > +#include > + > +#include "../rte_bus_auxiliary.h" > +#include "../private.h" > + > +#define AUXILIARY_SYSFS_PATH "/sys/bus/auxiliary/devices" > + > +/* Scan one auxiliary sysfs entry, and fill the devices list from it. */ > +static int > +auxiliary_scan_one(const char *dirname, const char *name) > +{ > + struct rte_auxiliary_device *dev; > + struct rte_auxiliary_device *dev2; > + char filename[PATH_MAX]; > + unsigned long tmp; > + int ret; > + > + dev = malloc(sizeof(*dev)); > + if (dev == NULL) > + return -1; > + > + memset(dev, 0, sizeof(*dev)); > + if (rte_strscpy(dev->name, name, sizeof(dev->name)) < 0) { > + free(dev); > + return -1; > + } > + dev->device.name = dev->name; > + dev->device.bus = &auxiliary_bus.bus; > + > + /* Get NUMA node, default to 0 if not present */ > + snprintf(filename, sizeof(filename), "%s/%s/numa_node", > + dirname, name); > + if (access(filename, F_OK) != -1) { > + if (eal_parse_sysfs_value(filename, &tmp) == 0) > + dev->device.numa_node = tmp; > + else > + dev->device.numa_node = -1; > + } else { > + dev->device.numa_node = 0; > + } > + > + auxiliary_on_scan(dev); > + > + /* Device is valid, add in list (sorted) */ > + TAILQ_FOREACH(dev2, &auxiliary_bus.device_list, next) { > + ret = strcmp(dev->name, dev2->name); > + if (ret > 0) > + continue; > + if (ret < 0) { > + auxiliary_insert_device(dev2, dev); > + } else { /* already registered */ > + if (rte_dev_is_probed(&dev2->device) && > + dev2->device.devargs != dev->device.devargs) { > + /* To probe device with new devargs. */ > + rte_devargs_remove(dev2->device.devargs); > + auxiliary_on_scan(dev2); > + } > + free(dev); > + } > + return 0; > + } > + auxiliary_add_device(dev); > + return 0; > +} > + > +/* > + * Test whether the auxiliary device exist Missing full stop above. > + */ > +bool > +auxiliary_dev_exists(const char *name) > +{ > + DIR *dir; > + char dirname[PATH_MAX]; > + > + snprintf(dirname, sizeof(dirname), "%s/%s", > + AUXILIARY_SYSFS_PATH, name); > + dir = opendir(dirname); > + if (dir == NULL) > + return false; > + closedir(dir); > + return true; > +} > + > +/* > + * Scan the devices in the auxiliary bus Missing full stop above. > + */ > +int > +auxiliary_scan(void) > +{ > + struct dirent *e; > + DIR *dir; > + char dirname[PATH_MAX]; > + struct rte_auxiliary_driver *drv; > + > + dir = opendir(AUXILIARY_SYSFS_PATH); > + if (dir == NULL) { > + AUXILIARY_LOG(INFO, "%s not found, is auxiliary module loaded?", > + AUXILIARY_SYSFS_PATH); > + return 0; > + } > + > + while ((e = readdir(dir)) != NULL) { > + if (e->d_name[0] == '.') > + continue; > + > + if (auxiliary_is_ignored_device(e->d_name)) > + continue; > + > + snprintf(dirname, sizeof(dirname), "%s/%s", > + AUXILIARY_SYSFS_PATH, e->d_name); > + > + /* Ignore if no driver can handle. */ > + FOREACH_DRIVER_ON_AUXILIARY_BUS(drv) { > + if (drv->match(e->d_name)) > + break; > + } > + if (drv == NULL) > + continue; > + > + if (auxiliary_scan_one(dirname, e->d_name) < 0) > + goto error; > + } > + closedir(dir); > + return 0; > + > +error: > + closedir(dir); > + return -1; > +} > diff --git a/drivers/bus/auxiliary/meson.build b/drivers/bus/auxiliary/meson.build > new file mode 100644 > index 0000000000..357550eff7 > --- /dev/null > +++ b/drivers/bus/auxiliary/meson.build > @@ -0,0 +1,16 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright (c) 2021 NVIDIA Corporation & Affiliates > + > +headers = files( > + 'rte_bus_auxiliary.h', > +) > +sources = files( > + 'auxiliary_common.c', > + 'auxiliary_params.c', > +) > +if is_linux > + sources += files( > + 'linux/auxiliary.c', > + ) > +endif > +deps += ['kvargs'] > diff --git a/drivers/bus/auxiliary/private.h b/drivers/bus/auxiliary/private.h > new file mode 100644 > index 0000000000..cb3e849993 > --- /dev/null > +++ b/drivers/bus/auxiliary/private.h > @@ -0,0 +1,74 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright (c) 2021 NVIDIA Corporation & Affiliates > + */ > + > +#ifndef AUXILIARY_PRIVATE_H May be add BUS_ prefix at leaat? > +#define AUXILIARY_PRIVATE_H > + > +#include > +#include > + > +#include "rte_bus_auxiliary.h" > + > +extern struct rte_auxiliary_bus auxiliary_bus; > +extern int auxiliary_bus_logtype; > + > +#define AUXILIARY_LOG(level, ...) \ > + rte_log(RTE_LOG_ ## level, auxiliary_bus_logtype, \ > + RTE_FMT("auxiliary bus: " RTE_FMT_HEAD(__VA_ARGS__,) "\n", \ > + RTE_FMT_TAIL(__VA_ARGS__,))) > + > +/* Auxiliary bus iterators */ > +#define FOREACH_DEVICE_ON_AUXILIARY_BUS(p) \ > + TAILQ_FOREACH(p, &(auxiliary_bus.device_list), next) > + > +#define FOREACH_DRIVER_ON_AUXILIARY_BUS(p) \ > + TAILQ_FOREACH(p, &(auxiliary_bus.driver_list), next) > + > +bool auxiliary_dev_exists(const char *name); > + > +/* > + * Scan the content of the auxiliary bus, and the devices in the devices > + * list. > + */ > +int auxiliary_scan(void); > + > +/* > + * Update a device being scanned. > + */ > +void auxiliary_on_scan(struct rte_auxiliary_device *aux_dev); > + > +/* > + * Validate whether a device with given auxiliary device should be ignored > + * or not. > + */ > +bool auxiliary_is_ignored_device(const char *name); > + > +/* > + * Add an auxiliary device to the auxiliary bus (append to auxiliary device > + * list). This function also updates the bus references of the auxiliary > + * device and the generic device object embedded within. > + */ > +void auxiliary_add_device(struct rte_auxiliary_device *aux_dev); > + > +/* > + * Insert an auxiliary device in the auxiliary bus at a particular location > + * in the device list. It also updates the auxiliary bus reference of the > + * new devices to be inserted. > + */ > +void auxiliary_insert_device(struct rte_auxiliary_device *exist_aux_dev, > + struct rte_auxiliary_device *new_aux_dev); > + > +/* > + * Match the auxiliary driver and device by driver function Missing full stop. > + */ > +bool auxiliary_match(const struct rte_auxiliary_driver *aux_drv, > + const struct rte_auxiliary_device *aux_dev); > + > +/* > + * Iterate over devices, matching any device against the provided string Missing full stop. > + */ > +void *auxiliary_dev_iterate(const void *start, const char *str, > + const struct rte_dev_iterator *it); > + > +#endif /* AUXILIARY_PRIVATE_H */ > diff --git a/drivers/bus/auxiliary/rte_bus_auxiliary.h b/drivers/bus/auxiliary/rte_bus_auxiliary.h > new file mode 100644 > index 0000000000..16b147e387 > --- /dev/null > +++ b/drivers/bus/auxiliary/rte_bus_auxiliary.h > @@ -0,0 +1,201 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright (c) 2021 NVIDIA Corporation & Affiliates > + */ > + > +#ifndef RTE_BUS_AUXILIARY_H > +#define RTE_BUS_AUXILIARY_H > + > +/** > + * @file > + * > + * Auxiliary Bus Interface. > + */ > + > +#ifdef __cplusplus > +extern "C" { > +#endif > + > +#include > +#include > +#include > +#include > +#include > +#include > +#include > + > +#include > +#include > +#include > +#include > +#include > + > +#define RTE_BUS_AUXILIARY_NAME "auxiliary" > + > +/* Forward declarations */ > +struct rte_auxiliary_driver; > +struct rte_auxiliary_bus; > +struct rte_auxiliary_device; > + > +/** > + * Match function for the driver to decide if device can be handled. > + * > + * @param name > + * Pointer to the auxiliary device name. > + * @return > + * Whether the driver can handle the auxiliary device. > + */ > +typedef bool(rte_auxiliary_match_t)(const char *name); > + > +/** > + * Initialization function for the driver called during auxiliary probing. > + * > + * @param drv > + * Pointer to the auxiliary driver. > + * @param dev > + * Pointer to the auxiliary device. > + * @return > + * - 0 On success. > + * - Negative value and rte_errno is set otherwise. > + */ > +typedef int(rte_auxiliary_probe_t)(struct rte_auxiliary_driver *drv, > + struct rte_auxiliary_device *dev); > + > +/** > + * Uninitialization function for the driver called during hotplugging. > + * > + * @param dev > + * Pointer to the auxiliary device. > + * @return > + * - 0 On success. > + * - Negative value and rte_errno is set otherwise. > + */ > +typedef int (rte_auxiliary_remove_t)(struct rte_auxiliary_device *dev); > + > +/** > + * Driver-specific DMA mapping. After a successful call the device > + * will be able to read/write from/to this segment. > + * > + * @param dev > + * Pointer to the auxiliary device. > + * @param addr > + * Starting virtual address of memory to be mapped. > + * @param iova > + * Starting IOVA address of memory to be mapped. > + * @param len > + * Length of memory segment being mapped. > + * @return > + * - 0 On success. > + * - Negative value and rte_errno is set otherwise. > + */ > +typedef int (rte_auxiliary_dma_map_t)(struct rte_auxiliary_device *dev, > + void *addr, uint64_t iova, size_t len); > + > +/** > + * Driver-specific DMA un-mapping. After a successful call the device > + * will not be able to read/write from/to this segment. > + * > + * @param dev > + * Pointer to the auxiliary device. > + * @param addr > + * Starting virtual address of memory to be unmapped. > + * @param iova > + * Starting IOVA address of memory to be unmapped. > + * @param len > + * Length of memory segment being unmapped. > + * @return > + * - 0 On success. > + * - Negative value and rte_errno is set otherwise. > + */ > +typedef int (rte_auxiliary_dma_unmap_t)(struct rte_auxiliary_device *dev, > + void *addr, uint64_t iova, size_t len); > + > +/** > + * A structure describing an auxiliary device. > + */ > +struct rte_auxiliary_device { > + TAILQ_ENTRY(rte_auxiliary_device) next; /**< Next probed device. */ > + struct rte_device device; /**< Inherit core device */ > + char name[RTE_DEV_NAME_MAX_LEN + 1]; /**< ASCII device name */ > + struct rte_intr_handle intr_handle; /**< Interrupt handle */ > + struct rte_auxiliary_driver *driver; /**< Device driver */ > +}; > + > +/** List of auxiliary devices */ > +TAILQ_HEAD(rte_auxiliary_device_list, rte_auxiliary_device); > +/** List of auxiliary drivers */ > +TAILQ_HEAD(rte_auxiliary_driver_list, rte_auxiliary_driver); Shouldn't we hide rte_auxiliary_device inside the library take API/ABI stability into account? Or will be it DPDK internal anyway? If so, it should be done INTERNAL from the very beginning. > + > +/** > + * Structure describing the auxiliary bus > + */ > +struct rte_auxiliary_bus { > + struct rte_bus bus; /**< Inherit the generic class */ > + struct rte_auxiliary_device_list device_list; /**< List of devices */ > + struct rte_auxiliary_driver_list driver_list; /**< List of drivers */ > +}; It looks internal. The following forward declaration should be sufficient to build. struct rte_auxiliary_bus; > + > +/** > + * A structure describing an auxiliary driver. > + */ > +struct rte_auxiliary_driver { > + TAILQ_ENTRY(rte_auxiliary_driver) next; /**< Next in list. */ > + struct rte_driver driver; /**< Inherit core driver. */ > + struct rte_auxiliary_bus *bus; /**< Auxiliary bus reference. */ > + rte_auxiliary_match_t *match; /**< Device match function. */ > + rte_auxiliary_probe_t *probe; /**< Device probe function. */ > + rte_auxiliary_remove_t *remove; /**< Device remove function. */ > + rte_auxiliary_dma_map_t *dma_map; /**< Device DMA map function. */ > + rte_auxiliary_dma_unmap_t *dma_unmap; /**< Device DMA unmap function. */ > + uint32_t drv_flags; /**< Flags RTE_AUXILIARY_DRV_*. */ > +}; > + > +/** > + * @internal > + * Helper macro for drivers that need to convert to struct rte_auxiliary_device. > + */ > +#define RTE_DEV_TO_AUXILIARY(ptr) \ > + container_of(ptr, struct rte_auxiliary_device, device) > + > +#define RTE_DEV_TO_AUXILIARY_CONST(ptr) \ > + container_of(ptr, const struct rte_auxiliary_device, device) > + > +#define RTE_ETH_DEV_TO_AUXILIARY(eth_dev) \ > + RTE_DEV_TO_AUXILIARY((eth_dev)->device) > + > +/** Device driver needs IOVA as VA and cannot work with IOVA as PA */ > +#define RTE_AUXILIARY_DRV_NEED_IOVA_AS_VA 0x002 > + > +/** Don't we need EXPERIMENTAL notice here? > + * Register an auxiliary driver. > + * > + * @param driver > + * A pointer to a rte_auxiliary_driver structure describing the driver > + * to be registered. > + */ > +__rte_experimental > +void rte_auxiliary_register(struct rte_auxiliary_driver *driver); > + > +/** Helper for auxiliary device registration from driver instance */ > +#define RTE_PMD_REGISTER_AUXILIARY(nm, auxiliary_drv) \ > + RTE_INIT(auxiliaryinitfn_ ##nm) \ > + { \ > + (auxiliary_drv).driver.name = RTE_STR(nm); \ > + rte_auxiliary_register(&(auxiliary_drv)); \ > + } \ > + RTE_PMD_EXPORT_NAME(nm, __COUNTER__) > + > +/** Don't we need EXPERIMENTAL notice here? > + * Unregister an auxiliary driver. > + * > + * @param driver > + * A pointer to a rte_auxiliary_driver structure describing the driver > + * to be unregistered. > + */ > +__rte_experimental > +void rte_auxiliary_unregister(struct rte_auxiliary_driver *driver); > + > +#ifdef __cplusplus > +} > +#endif > + > +#endif /* RTE_BUS_AUXILIARY_H */ > diff --git a/drivers/bus/auxiliary/version.map b/drivers/bus/auxiliary/version.map > new file mode 100644 > index 0000000000..a52260657c > --- /dev/null > +++ b/drivers/bus/auxiliary/version.map > @@ -0,0 +1,7 @@ > +EXPERIMENTAL { > + global: > + > + # added in 21.08 > + rte_auxiliary_register; > + rte_auxiliary_unregister; > +}; > diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build > index 410058de3a..45eab5233d 100644 > --- a/drivers/bus/meson.build > +++ b/drivers/bus/meson.build > @@ -2,6 +2,7 @@ > # Copyright(c) 2017 Intel Corporation > > drivers = [ > + 'auxiliary', > 'dpaa', > 'fslmc', > 'ifpga', >