[RFC v3 0/3] Import Kernel uAPI header files

[RFC v3 0/3] Import Kernel uAPI header files

[Maxime Coquelin]

This series enables importing Linux Kernel uAPI headers
into the DPDK repository. It aims at solving alignment
issues between the build system and the system applications
linked ot DPDK libraries are run on.

It can also help simplify spaghetti code done to support
different versions of the Linux Kernel headers used by
the build system.

Guidelines and import script are also part of first patch.
A uAPI checker script is added in this 3rd RFC, goals is to
use it in CI for any patch touching linux-headers/uapi.

Follow-up patches are an example of imported uAPI inclusion
of VDUSE header into the Vhost library.

Morten, I did not (again) apply your Ack on patch 1, as it
has some significant changes and additions.

Changes in RFC v3:
==================
- Support nested headers include
- Interactive mode to select which headers to include
- Store version in a file instead of git commit messages
- All imported headers aligned on same version
- Improve loops in scripts (David)
- Update documentation


Changes in RFC v2:
==================
- Fix typos in documentation and commit messags (David, Morten)
- Add uAPI checker script
- Add uAPI to global_inc
- Fix build issues on FreeBSD and documentation (CI, David)
- Simplify import script (David)

Maxime Coquelin (3):
  uapi: introduce kernel uAPI headers import
  uapi: import VDUSE header
  vduse: use imported VDUSE uAPI header

 devtools/check-linux-uapi.sh           |  85 ++++++
 devtools/import-linux-uapi.sh          | 119 +++++++++
 doc/guides/contributing/index.rst      |   1 +
 doc/guides/contributing/linux_uapi.rst |  71 +++++
 lib/vhost/meson.build                  |   5 +-
 lib/vhost/vduse.c                      |   2 +-
 lib/vhost/vduse.h                      |  22 --
 linux-headers/uapi/.gitignore          |   4 +
 linux-headers/uapi/linux/vduse.h       | 353 +++++++++++++++++++++++++
 linux-headers/uapi/version             |   1 +
 meson.build                            |   8 +-
 11 files changed, 642 insertions(+), 29 deletions(-)
 create mode 100755 devtools/check-linux-uapi.sh
 create mode 100755 devtools/import-linux-uapi.sh
 create mode 100644 doc/guides/contributing/linux_uapi.rst
 create mode 100644 linux-headers/uapi/.gitignore
 create mode 100644 linux-headers/uapi/linux/vduse.h
 create mode 100644 linux-headers/uapi/version

-- 
2.46.0

[RFC v3 1/3] uapi: introduce kernel uAPI headers import

[Maxime Coquelin]

This patch introduces uAPI headers import into the DPDK
repository. This import is possible thanks to Linux Kernel
licence exception for syscalls:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/LICENSES/exceptions/Linux-syscall-note

Header files are have to be explicitly imported.

Guidelines are provided in the documentation, and helper
scripts are also provided to ensure proper importation of the
header (unmodified content from a released Kernel version):
 - import-linux-uapi.sh: used to add and update headers and
 their dependencies to linux-headers/uapi/
 - check-linux-uapi.sh: used to check all headers are valid

Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 devtools/check-linux-uapi.sh           |  85 ++++++++++++++++++
 devtools/import-linux-uapi.sh          | 119 +++++++++++++++++++++++++
 doc/guides/contributing/index.rst      |   1 +
 doc/guides/contributing/linux_uapi.rst |  71 +++++++++++++++
 linux-headers/uapi/.gitignore          |   4 +
 linux-headers/uapi/version             |   1 +
 meson.build                            |   8 +-
 7 files changed, 287 insertions(+), 2 deletions(-)
 create mode 100755 devtools/check-linux-uapi.sh
 create mode 100755 devtools/import-linux-uapi.sh
 create mode 100644 doc/guides/contributing/linux_uapi.rst
 create mode 100644 linux-headers/uapi/.gitignore
 create mode 100644 linux-headers/uapi/version

diff --git a/devtools/check-linux-uapi.sh b/devtools/check-linux-uapi.sh
new file mode 100755
index 0000000000..fc42c03169
--- /dev/null
+++ b/devtools/check-linux-uapi.sh
@@ -0,0 +1,85 @@
+#!/bin/sh
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2024 Red Hat, Inc.
+
+#
+# Check Linux Kernel uAPI header files
+#
+
+base_url="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/include/uapi/"
+base_path="linux-headers/uapi/"
+errors=0
+version=""
+
+print_usage()
+{
+	echo "Usage: $(basename $0) [-h]"
+}
+
+check_uapi_header() {
+	path=$1
+	file=${1#$base_path}
+
+	cp -f $path $tmpinput1
+
+	# Restore includes fixups done by import-linux-uapi.sh
+	sed -i "s|//#include <linux/compiler.h>|#include <linux/compiler.h>|g" $tmpinput1
+	sed -i "s|#include <uapi/|#include <|g" $tmpinput1
+
+	url="${base_url}${file}?h=${version}"
+	echo -n "Checking $file... "
+	curl -s -f -o $tmpinput2 $url
+	if [ $? -ne 0 ]; then
+		echo "Failed to download $url"
+		exit 1
+	fi
+
+	diff -q $tmpinput1 $tmpinput2 >/dev/null
+	if [ $? -ne 0 ]; then
+		echo "KO"
+		diff -u $tmpinput1 $tmpinput2
+		errors=$((errors+1))
+	else
+		echo "OK"
+	fi
+}
+
+while getopts hv ARG ; do
+	case $ARG in
+		h )
+			print_usage
+			exit 0
+			;;
+		? )
+			print_usage
+			exit 1
+			;;
+	esac
+done
+
+shift $(($OPTIND - 1))
+if [ $# -ne 0 ]; then
+	print_usage
+	exit 1
+fi
+
+cd $(dirname $0)/..
+
+tmpinput1=$(mktemp -t dpdk.checkuapi.XXXXXX)
+tmpinput2=$(mktemp -t dpdk.checkuapi.XXXXXX)
+trap "rm -f '$tmpinput1 $tmpinput2'" INT
+
+version=$(< ${base_path}/version)
+
+echo "Checking imported headers for version ${version}"
+
+for filename in $(find $base_path -name "*.h" -type f); do
+	check_uapi_header "${filename}" </dev/null
+done
+
+echo "$errors error(s) found"
+
+rm -f $tmpinput1 $tmpinput2
+trap - INT
+
+exit $errors
diff --git a/devtools/import-linux-uapi.sh b/devtools/import-linux-uapi.sh
new file mode 100755
index 0000000000..ff15e23821
--- /dev/null
+++ b/devtools/import-linux-uapi.sh
@@ -0,0 +1,119 @@
+#!/bin/sh
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2024 Red Hat, Inc.
+
+#
+# Import Linux Kernel uAPI header file
+#
+
+base_url="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/include/uapi/"
+base_path="linux-headers/uapi/"
+version=""
+file=""
+
+print_usage()
+{
+	echo "Usage: $(basename $0) [-h] [-a FILE] [-u VERSION]"
+	echo "-a FILE      import Linux header file. E.g. linux/vfio.h"
+	echo "-u VERSION   update imported list of Linux headers to a given version. E.g. v6.10"
+}
+
+version_valid() {
+    printf '%s\n%s' "$1" "$2" | sort -C -V
+}
+
+update_headers()
+{
+	local header=${filename//"$base_path"}
+	local url
+	local path
+
+	echo "Updating to $version"
+	while IFS= read -d '' -r filename; do
+		header=${filename//"$base_path"/}
+		url="${base_url}${header}?h=${version}"
+		path="${base_path}${header}"
+		curl -s -f -o $path $url
+	done < <(find $base_path -name "*.h" -type f -print0)
+}
+
+import_header()
+{
+	local include
+	local import
+	local header=$1
+
+	local url="${base_url}${header}?h=${version}"
+	local path="${base_path}${header}"
+
+	curl -s -f --create-dirs -o $path $url
+
+	for include in $(sed -ne 's/^#include <\(.*\)>$/\1/p' $path); do
+		if [ ! -f "${base_path}${include}" ]; then
+			read -p "Import $include (y/n): " import < /dev/tty && [ "$import" = 'y' ] || continue
+			echo "Importing $include for $path"
+			import_header "$include"
+		fi
+	done
+}
+
+fixup_includes()
+{
+	local path=$1
+
+	# Do not include linux/compiler.h as done by make headers
+	sed -i "s|^#include <linux/compiler.h>|//#include <linux/compiler.h>|g" $path
+
+	# Prepend include path with "uapi/" if the header is imported
+	for include in $(sed -ne 's/^#include <\(.*\)>$/\1/p' $path); do
+		if [ -f "${base_path}${include}" ]; then
+			sed -i "s|${include}|uapi/${include}|g" $path
+		fi
+	done
+}
+
+while getopts a:u:hv opt ; do
+	case ${opt} in
+		a )
+			file=$OPTARG
+			;;
+		u )
+			version=$OPTARG
+			;;
+		h )
+			print_usage
+			exit 0
+			;;
+		? )
+			print_usage
+			exit 1
+			;;
+	esac
+done
+
+cd $(dirname $0)/..
+
+current_version=$(< ${base_path}/version)
+
+if [ -n "${version}" ]; then
+	version_valid "$version" "$current_version"
+	if [ $? -eq 0 ]; then
+		echo "Headers already up to date ($current_version >= $version)"
+		version=$current_version
+	else
+		update_headers
+	fi
+else
+	echo "Version not specified, using current version ($current_version)"
+	version=$current_version
+fi
+
+if [ -n "${file}" ]; then
+	import_header $file
+fi
+
+for filename in $(find $base_path -name "*.h" -type f); do
+	fixup_includes "${filename}" </dev/null
+done
+
+echo $version > ${base_path}/version
diff --git a/doc/guides/contributing/index.rst b/doc/guides/contributing/index.rst
index dcb9b1fbf0..603dc72654 100644
--- a/doc/guides/contributing/index.rst
+++ b/doc/guides/contributing/index.rst
@@ -19,3 +19,4 @@ Contributor's Guidelines
     vulnerability
     stable
     cheatsheet
+    linux_uapi
diff --git a/doc/guides/contributing/linux_uapi.rst b/doc/guides/contributing/linux_uapi.rst
new file mode 100644
index 0000000000..d4255eb22e
--- /dev/null
+++ b/doc/guides/contributing/linux_uapi.rst
@@ -0,0 +1,71 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+   Copyright(c) 2024 Red Hat, Inc.
+
+Linux uAPI header files
+=======================
+
+Rationale
+---------
+
+The system a DPDK library or driver is built on is not necessarily running the
+same Kernel version than the system that will run it.
+Importing Linux Kernel uAPI headers enable to build features that are not
+supported yet by the build system.
+
+For example, the build system runs upstream Kernel v5.19 and we would like to
+build a VDUSE application that will use VDUSE_IOTLB_GET_INFO ioctl() introduced
+in Linux Kernel v6.0.
+
+`Linux Kernel licence exception regarding syscalls
+<https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/LICENSES/exceptions/Linux-syscall-note>`_
+enable importing unmodified Linux Kernel uAPI header files.
+
+Importing or updating an uAPI header file
+-----------------------------------------
+
+In order to ensure the imported uAPI headers are both unmodified and from a
+released version of the linux Kernel, a helper script is made available and
+MUST be used.
+Below is an example to import ``linux/vduse.h`` file from Linux ``v6.10``:
+
+.. code-block:: console
+
+   ./devtools/import-linux-uapi.sh -a linux/vduse.h -u v6.10
+
+Once imported, the header files should be committed without any other change.
+Note that all the imported headers will be updated to the requested version.
+
+Updating imported headers to a newer released version should only be done on
+a need basis, it can be achieved using the same script:
+
+.. code-block:: console
+
+   ./devtools/import-linux-uapi.sh -u v6.10
+
+The commit message should reflect why updating the headers is necessary.
+
+Once committed, user can check headers are valid by using the Linux uAPI
+checker tool:
+
+.. code-block:: console
+
+   ./devtools/check-linux-uapi.sh
+
+Header inclusion into library or driver
+---------------------------------------
+
+The library or driver willing to make use of imported uAPI headers needs to
+explicitly add uAPI headers path to the ``includes`` var in its ``meson.build``
+file:
+
+.. code-block:: python
+
+   includes += linux_uapi_inc
+
+Then, it can be included with ``uapi/`` prefix in C files.
+For example to include VDUSE uAPI:
+
+.. code-block:: c
+
+   #include <uapi/linux/vduse.h>
+
diff --git a/linux-headers/uapi/.gitignore b/linux-headers/uapi/.gitignore
new file mode 100644
index 0000000000..558ba597d6
--- /dev/null
+++ b/linux-headers/uapi/.gitignore
@@ -0,0 +1,4 @@
+**
+!**/
+!**/*.h
+!version
diff --git a/linux-headers/uapi/version b/linux-headers/uapi/version
new file mode 100644
index 0000000000..2b256fd48f
--- /dev/null
+++ b/linux-headers/uapi/version
@@ -0,0 +1 @@
+v0.0
diff --git a/meson.build b/meson.build
index 8b248d4505..aca47334ab 100644
--- a/meson.build
+++ b/meson.build
@@ -67,16 +67,20 @@ endif
 # configure the build, and make sure configs here and in config folder are
 # able to be included in any file. We also store a global array of include dirs
 # for passing to pmdinfogen scripts
-global_inc = include_directories('.', 'config',
+global_inc = [include_directories('.', 'config',
     'lib/eal/include',
     'lib/eal/@0@/include'.format(host_machine.system()),
     'lib/eal/@0@/include'.format(arch_subdir),
-)
+)]
 
 # do configuration and get tool paths
 subdir('buildtools')
 subdir('config')
 
+if is_linux
+    global_inc += include_directories('linux-headers')
+endif
+
 # build libs and drivers
 subdir('lib')
 subdir('drivers')
-- 
2.46.0

[RFC v3 2/3] uapi: import VDUSE header

[Maxime Coquelin]

Import VDUSE headers to be included in the Vhost library.
Set header version to v6.10 Linux Kernel release.

Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 linux-headers/uapi/linux/vduse.h | 353 +++++++++++++++++++++++++++++++
 linux-headers/uapi/version       |   2 +-
 2 files changed, 354 insertions(+), 1 deletion(-)
 create mode 100644 linux-headers/uapi/linux/vduse.h

diff --git a/linux-headers/uapi/linux/vduse.h b/linux-headers/uapi/linux/vduse.h
new file mode 100644
index 0000000000..11bd48c72c
--- /dev/null
+++ b/linux-headers/uapi/linux/vduse.h
@@ -0,0 +1,353 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+#ifndef _UAPI_VDUSE_H_
+#define _UAPI_VDUSE_H_
+
+#include <linux/types.h>
+
+#define VDUSE_BASE	0x81
+
+/* The ioctls for control device (/dev/vduse/control) */
+
+#define VDUSE_API_VERSION	0
+
+/*
+ * Get the version of VDUSE API that kernel supported (VDUSE_API_VERSION).
+ * This is used for future extension.
+ */
+#define VDUSE_GET_API_VERSION	_IOR(VDUSE_BASE, 0x00, __u64)
+
+/* Set the version of VDUSE API that userspace supported. */
+#define VDUSE_SET_API_VERSION	_IOW(VDUSE_BASE, 0x01, __u64)
+
+/**
+ * struct vduse_dev_config - basic configuration of a VDUSE device
+ * @name: VDUSE device name, needs to be NUL terminated
+ * @vendor_id: virtio vendor id
+ * @device_id: virtio device id
+ * @features: virtio features
+ * @vq_num: the number of virtqueues
+ * @vq_align: the allocation alignment of virtqueue's metadata
+ * @reserved: for future use, needs to be initialized to zero
+ * @config_size: the size of the configuration space
+ * @config: the buffer of the configuration space
+ *
+ * Structure used by VDUSE_CREATE_DEV ioctl to create VDUSE device.
+ */
+struct vduse_dev_config {
+#define VDUSE_NAME_MAX	256
+	char name[VDUSE_NAME_MAX];
+	__u32 vendor_id;
+	__u32 device_id;
+	__u64 features;
+	__u32 vq_num;
+	__u32 vq_align;
+	__u32 reserved[13];
+	__u32 config_size;
+	__u8 config[];
+};
+
+/* Create a VDUSE device which is represented by a char device (/dev/vduse/$NAME) */
+#define VDUSE_CREATE_DEV	_IOW(VDUSE_BASE, 0x02, struct vduse_dev_config)
+
+/*
+ * Destroy a VDUSE device. Make sure there are no more references
+ * to the char device (/dev/vduse/$NAME).
+ */
+#define VDUSE_DESTROY_DEV	_IOW(VDUSE_BASE, 0x03, char[VDUSE_NAME_MAX])
+
+/* The ioctls for VDUSE device (/dev/vduse/$NAME) */
+
+/**
+ * struct vduse_iotlb_entry - entry of IOTLB to describe one IOVA region [start, last]
+ * @offset: the mmap offset on returned file descriptor
+ * @start: start of the IOVA region
+ * @last: last of the IOVA region
+ * @perm: access permission of the IOVA region
+ *
+ * Structure used by VDUSE_IOTLB_GET_FD ioctl to find an overlapped IOVA region.
+ */
+struct vduse_iotlb_entry {
+	__u64 offset;
+	__u64 start;
+	__u64 last;
+#define VDUSE_ACCESS_RO 0x1
+#define VDUSE_ACCESS_WO 0x2
+#define VDUSE_ACCESS_RW 0x3
+	__u8 perm;
+};
+
+/*
+ * Find the first IOVA region that overlaps with the range [start, last]
+ * and return the corresponding file descriptor. Return -EINVAL means the
+ * IOVA region doesn't exist. Caller should set start and last fields.
+ */
+#define VDUSE_IOTLB_GET_FD	_IOWR(VDUSE_BASE, 0x10, struct vduse_iotlb_entry)
+
+/*
+ * Get the negotiated virtio features. It's a subset of the features in
+ * struct vduse_dev_config which can be accepted by virtio driver. It's
+ * only valid after FEATURES_OK status bit is set.
+ */
+#define VDUSE_DEV_GET_FEATURES	_IOR(VDUSE_BASE, 0x11, __u64)
+
+/**
+ * struct vduse_config_data - data used to update configuration space
+ * @offset: the offset from the beginning of configuration space
+ * @length: the length to write to configuration space
+ * @buffer: the buffer used to write from
+ *
+ * Structure used by VDUSE_DEV_SET_CONFIG ioctl to update device
+ * configuration space.
+ */
+struct vduse_config_data {
+	__u32 offset;
+	__u32 length;
+	__u8 buffer[];
+};
+
+/* Set device configuration space */
+#define VDUSE_DEV_SET_CONFIG	_IOW(VDUSE_BASE, 0x12, struct vduse_config_data)
+
+/*
+ * Inject a config interrupt. It's usually used to notify virtio driver
+ * that device configuration space has changed.
+ */
+#define VDUSE_DEV_INJECT_CONFIG_IRQ	_IO(VDUSE_BASE, 0x13)
+
+/**
+ * struct vduse_vq_config - basic configuration of a virtqueue
+ * @index: virtqueue index
+ * @max_size: the max size of virtqueue
+ * @reserved: for future use, needs to be initialized to zero
+ *
+ * Structure used by VDUSE_VQ_SETUP ioctl to setup a virtqueue.
+ */
+struct vduse_vq_config {
+	__u32 index;
+	__u16 max_size;
+	__u16 reserved[13];
+};
+
+/*
+ * Setup the specified virtqueue. Make sure all virtqueues have been
+ * configured before the device is attached to vDPA bus.
+ */
+#define VDUSE_VQ_SETUP		_IOW(VDUSE_BASE, 0x14, struct vduse_vq_config)
+
+/**
+ * struct vduse_vq_state_split - split virtqueue state
+ * @avail_index: available index
+ */
+struct vduse_vq_state_split {
+	__u16 avail_index;
+};
+
+/**
+ * struct vduse_vq_state_packed - packed virtqueue state
+ * @last_avail_counter: last driver ring wrap counter observed by device
+ * @last_avail_idx: device available index
+ * @last_used_counter: device ring wrap counter
+ * @last_used_idx: used index
+ */
+struct vduse_vq_state_packed {
+	__u16 last_avail_counter;
+	__u16 last_avail_idx;
+	__u16 last_used_counter;
+	__u16 last_used_idx;
+};
+
+/**
+ * struct vduse_vq_info - information of a virtqueue
+ * @index: virtqueue index
+ * @num: the size of virtqueue
+ * @desc_addr: address of desc area
+ * @driver_addr: address of driver area
+ * @device_addr: address of device area
+ * @split: split virtqueue state
+ * @packed: packed virtqueue state
+ * @ready: ready status of virtqueue
+ *
+ * Structure used by VDUSE_VQ_GET_INFO ioctl to get virtqueue's information.
+ */
+struct vduse_vq_info {
+	__u32 index;
+	__u32 num;
+	__u64 desc_addr;
+	__u64 driver_addr;
+	__u64 device_addr;
+	union {
+		struct vduse_vq_state_split split;
+		struct vduse_vq_state_packed packed;
+	};
+	__u8 ready;
+};
+
+/* Get the specified virtqueue's information. Caller should set index field. */
+#define VDUSE_VQ_GET_INFO	_IOWR(VDUSE_BASE, 0x15, struct vduse_vq_info)
+
+/**
+ * struct vduse_vq_eventfd - eventfd configuration for a virtqueue
+ * @index: virtqueue index
+ * @fd: eventfd, -1 means de-assigning the eventfd
+ *
+ * Structure used by VDUSE_VQ_SETUP_KICKFD ioctl to setup kick eventfd.
+ */
+struct vduse_vq_eventfd {
+	__u32 index;
+#define VDUSE_EVENTFD_DEASSIGN -1
+	int fd;
+};
+
+/*
+ * Setup kick eventfd for specified virtqueue. The kick eventfd is used
+ * by VDUSE kernel module to notify userspace to consume the avail vring.
+ */
+#define VDUSE_VQ_SETUP_KICKFD	_IOW(VDUSE_BASE, 0x16, struct vduse_vq_eventfd)
+
+/*
+ * Inject an interrupt for specific virtqueue. It's used to notify virtio driver
+ * to consume the used vring.
+ */
+#define VDUSE_VQ_INJECT_IRQ	_IOW(VDUSE_BASE, 0x17, __u32)
+
+/**
+ * struct vduse_iova_umem - userspace memory configuration for one IOVA region
+ * @uaddr: start address of userspace memory, it must be aligned to page size
+ * @iova: start of the IOVA region
+ * @size: size of the IOVA region
+ * @reserved: for future use, needs to be initialized to zero
+ *
+ * Structure used by VDUSE_IOTLB_REG_UMEM and VDUSE_IOTLB_DEREG_UMEM
+ * ioctls to register/de-register userspace memory for IOVA regions
+ */
+struct vduse_iova_umem {
+	__u64 uaddr;
+	__u64 iova;
+	__u64 size;
+	__u64 reserved[3];
+};
+
+/* Register userspace memory for IOVA regions */
+#define VDUSE_IOTLB_REG_UMEM	_IOW(VDUSE_BASE, 0x18, struct vduse_iova_umem)
+
+/* De-register the userspace memory. Caller should set iova and size field. */
+#define VDUSE_IOTLB_DEREG_UMEM	_IOW(VDUSE_BASE, 0x19, struct vduse_iova_umem)
+
+/**
+ * struct vduse_iova_info - information of one IOVA region
+ * @start: start of the IOVA region
+ * @last: last of the IOVA region
+ * @capability: capability of the IOVA regsion
+ * @reserved: for future use, needs to be initialized to zero
+ *
+ * Structure used by VDUSE_IOTLB_GET_INFO ioctl to get information of
+ * one IOVA region.
+ */
+struct vduse_iova_info {
+	__u64 start;
+	__u64 last;
+#define VDUSE_IOVA_CAP_UMEM (1 << 0)
+	__u64 capability;
+	__u64 reserved[3];
+};
+
+/*
+ * Find the first IOVA region that overlaps with the range [start, last]
+ * and return some information on it. Caller should set start and last fields.
+ */
+#define VDUSE_IOTLB_GET_INFO	_IOWR(VDUSE_BASE, 0x1a, struct vduse_iova_info)
+
+/* The control messages definition for read(2)/write(2) on /dev/vduse/$NAME */
+
+/**
+ * enum vduse_req_type - request type
+ * @VDUSE_GET_VQ_STATE: get the state for specified virtqueue from userspace
+ * @VDUSE_SET_STATUS: set the device status
+ * @VDUSE_UPDATE_IOTLB: Notify userspace to update the memory mapping for
+ *                      specified IOVA range via VDUSE_IOTLB_GET_FD ioctl
+ */
+enum vduse_req_type {
+	VDUSE_GET_VQ_STATE,
+	VDUSE_SET_STATUS,
+	VDUSE_UPDATE_IOTLB,
+};
+
+/**
+ * struct vduse_vq_state - virtqueue state
+ * @index: virtqueue index
+ * @split: split virtqueue state
+ * @packed: packed virtqueue state
+ */
+struct vduse_vq_state {
+	__u32 index;
+	union {
+		struct vduse_vq_state_split split;
+		struct vduse_vq_state_packed packed;
+	};
+};
+
+/**
+ * struct vduse_dev_status - device status
+ * @status: device status
+ */
+struct vduse_dev_status {
+	__u8 status;
+};
+
+/**
+ * struct vduse_iova_range - IOVA range [start, last]
+ * @start: start of the IOVA range
+ * @last: last of the IOVA range
+ */
+struct vduse_iova_range {
+	__u64 start;
+	__u64 last;
+};
+
+/**
+ * struct vduse_dev_request - control request
+ * @type: request type
+ * @request_id: request id
+ * @reserved: for future use
+ * @vq_state: virtqueue state, only index field is available
+ * @s: device status
+ * @iova: IOVA range for updating
+ * @padding: padding
+ *
+ * Structure used by read(2) on /dev/vduse/$NAME.
+ */
+struct vduse_dev_request {
+	__u32 type;
+	__u32 request_id;
+	__u32 reserved[4];
+	union {
+		struct vduse_vq_state vq_state;
+		struct vduse_dev_status s;
+		struct vduse_iova_range iova;
+		__u32 padding[32];
+	};
+};
+
+/**
+ * struct vduse_dev_response - response to control request
+ * @request_id: corresponding request id
+ * @result: the result of request
+ * @reserved: for future use, needs to be initialized to zero
+ * @vq_state: virtqueue state
+ * @padding: padding
+ *
+ * Structure used by write(2) on /dev/vduse/$NAME.
+ */
+struct vduse_dev_response {
+	__u32 request_id;
+#define VDUSE_REQ_RESULT_OK	0x00
+#define VDUSE_REQ_RESULT_FAILED	0x01
+	__u32 result;
+	__u32 reserved[4];
+	union {
+		struct vduse_vq_state vq_state;
+		__u32 padding[32];
+	};
+};
+
+#endif /* _UAPI_VDUSE_H_ */
diff --git a/linux-headers/uapi/version b/linux-headers/uapi/version
index 2b256fd48f..d06407735c 100644
--- a/linux-headers/uapi/version
+++ b/linux-headers/uapi/version
@@ -1 +1 @@
-v0.0
+v6.10
-- 
2.46.0

[RFC v3 3/3] vduse: use imported VDUSE uAPI header

[Maxime Coquelin]

This patch makes use of the imported VDUSE headers.
The VDUSE support is now systematically built on Linux
systems, even if the build system does not support its
ioctl().

Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 lib/vhost/meson.build |  5 +----
 lib/vhost/vduse.c     |  2 +-
 lib/vhost/vduse.h     | 22 ----------------------
 3 files changed, 2 insertions(+), 27 deletions(-)

diff --git a/lib/vhost/meson.build b/lib/vhost/meson.build
index 41b622a9be..5c2528ff8c 100644
--- a/lib/vhost/meson.build
+++ b/lib/vhost/meson.build
@@ -23,16 +23,13 @@ sources = files(
         'iotlb.c',
         'socket.c',
         'vdpa.c',
+        'vduse.c',
         'vhost.c',
         'vhost_crypto.c',
         'vhost_user.c',
         'virtio_net.c',
         'virtio_net_ctrl.c',
 )
-if cc.has_header('linux/vduse.h')
-    sources += files('vduse.c')
-    cflags += '-DVHOST_HAS_VDUSE'
-endif
 headers = files(
         'rte_vdpa.h',
         'rte_vhost.h',
diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c
index c66602905c..a41ff2024c 100644
--- a/lib/vhost/vduse.c
+++ b/lib/vhost/vduse.c
@@ -8,7 +8,7 @@
 #include <fcntl.h>
 
 
-#include <linux/vduse.h>
+#include <uapi/linux/vduse.h>
 #include <linux/virtio_net.h>
 
 #include <sys/ioctl.h>
diff --git a/lib/vhost/vduse.h b/lib/vhost/vduse.h
index 0d8f3f1205..47ca97a064 100644
--- a/lib/vhost/vduse.h
+++ b/lib/vhost/vduse.h
@@ -9,29 +9,7 @@
 
 #define VDUSE_NET_SUPPORTED_FEATURES VIRTIO_NET_SUPPORTED_FEATURES
 
-#ifdef VHOST_HAS_VDUSE
-
 int vduse_device_create(const char *path, bool compliant_ol_flags);
 int vduse_device_destroy(const char *path);
 
-#else
-
-static inline int
-vduse_device_create(const char *path, bool compliant_ol_flags)
-{
-	RTE_SET_USED(compliant_ol_flags);
-
-	VHOST_CONFIG_LOG(path, ERR, "VDUSE support disabled at build time");
-	return -1;
-}
-
-static inline int
-vduse_device_destroy(const char *path)
-{
-	VHOST_CONFIG_LOG(path, ERR, "VDUSE support disabled at build time");
-	return -1;
-}
-
-#endif /* VHOST_HAS_VDUSE */
-
 #endif /* _VDUSE_H */
-- 
2.46.0

Re: [RFC v3 0/3] Import Kernel uAPI header files

[Ferruh Yigit]

On 9/11/2024 8:32 PM, Maxime Coquelin wrote:
> This series enables importing Linux Kernel uAPI headers
> into the DPDK repository. It aims at solving alignment
> issues between the build system and the system applications
> linked ot DPDK libraries are run on.
> 

Hi Maxime,

Is the imported Linux Kernel uAPI headers are always backward compatible?

If I build and run on identical platform, lets say has kernel v5.4,
and DPDK has the kernel uAPI header from v6.10, is this has any chance
to create backward compatibility issues?
Or can it enable some features which are indeed not exist/supported in
the platform that has old version of kernel?

> It can also help simplify spaghetti code done to support
> different versions of the Linux Kernel headers used by
> the build system.
> 
> Guidelines and import script are also part of first patch.
> A uAPI checker script is added in this 3rd RFC, goals is to
> use it in CI for any patch touching linux-headers/uapi.
> 
> Follow-up patches are an example of imported uAPI inclusion
> of VDUSE header into the Vhost library.
> 
> Morten, I did not (again) apply your Ack on patch 1, as it
> has some significant changes and additions.
> 
> Changes in RFC v3:
> ==================
> - Support nested headers include
> - Interactive mode to select which headers to include
> - Store version in a file instead of git commit messages
> - All imported headers aligned on same version
> - Improve loops in scripts (David)
> - Update documentation
> 
> 
> Changes in RFC v2:
> ==================
> - Fix typos in documentation and commit messags (David, Morten)
> - Add uAPI checker script
> - Add uAPI to global_inc
> - Fix build issues on FreeBSD and documentation (CI, David)
> - Simplify import script (David)
> 
> Maxime Coquelin (3):
>   uapi: introduce kernel uAPI headers import
>   uapi: import VDUSE header
>   vduse: use imported VDUSE uAPI header
> 
>  devtools/check-linux-uapi.sh           |  85 ++++++
>  devtools/import-linux-uapi.sh          | 119 +++++++++
>  doc/guides/contributing/index.rst      |   1 +
>  doc/guides/contributing/linux_uapi.rst |  71 +++++
>  lib/vhost/meson.build                  |   5 +-
>  lib/vhost/vduse.c                      |   2 +-
>  lib/vhost/vduse.h                      |  22 --
>  linux-headers/uapi/.gitignore          |   4 +
>  linux-headers/uapi/linux/vduse.h       | 353 +++++++++++++++++++++++++
>  linux-headers/uapi/version             |   1 +
>  meson.build                            |   8 +-
>  11 files changed, 642 insertions(+), 29 deletions(-)
>  create mode 100755 devtools/check-linux-uapi.sh
>  create mode 100755 devtools/import-linux-uapi.sh
>  create mode 100644 doc/guides/contributing/linux_uapi.rst
>  create mode 100644 linux-headers/uapi/.gitignore
>  create mode 100644 linux-headers/uapi/linux/vduse.h
>  create mode 100644 linux-headers/uapi/version
> 

Re: [RFC v3 0/3] Import Kernel uAPI header files

[Maxime Coquelin]

Hi Ferruh,

On 9/12/24 10:30, Ferruh Yigit wrote:
> On 9/11/2024 8:32 PM, Maxime Coquelin wrote:
>> This series enables importing Linux Kernel uAPI headers
>> into the DPDK repository. It aims at solving alignment
>> issues between the build system and the system applications
>> linked ot DPDK libraries are run on.
>>
> 
> Hi Maxime,
> 
> Is the imported Linux Kernel uAPI headers are always backward compatible?

Yes, at least that's what is advertised! :)

"
The binary interface provided by the kernel to user space cannot be 
broken except under the most severe circumstances.
"


> If I build and run on identical platform, lets say has kernel v5.4,
> and DPDK has the kernel uAPI header from v6.10, is this has any chance
> to create backward compatibility issues?

It should not if backward compatibility is preserved as advertised.

> Or can it enable some features which are indeed not exist/supported in
> the platform that has old version of kernel?

It can enable new feature, like for example a new ioctl command.

In this case, it should return something like ENOIOCTLCMD if the Kernel
does not support it.

> 
>> It can also help simplify spaghetti code done to support
>> different versions of the Linux Kernel headers used by
>> the build system.
>>
>> Guidelines and import script are also part of first patch.
>> A uAPI checker script is added in this 3rd RFC, goals is to
>> use it in CI for any patch touching linux-headers/uapi.
>>
>> Follow-up patches are an example of imported uAPI inclusion
>> of VDUSE header into the Vhost library.
>>
>> Morten, I did not (again) apply your Ack on patch 1, as it
>> has some significant changes and additions.
>>
>> Changes in RFC v3:
>> ==================
>> - Support nested headers include
>> - Interactive mode to select which headers to include
>> - Store version in a file instead of git commit messages
>> - All imported headers aligned on same version
>> - Improve loops in scripts (David)
>> - Update documentation
>>
>>
>> Changes in RFC v2:
>> ==================
>> - Fix typos in documentation and commit messags (David, Morten)
>> - Add uAPI checker script
>> - Add uAPI to global_inc
>> - Fix build issues on FreeBSD and documentation (CI, David)
>> - Simplify import script (David)
>>
>> Maxime Coquelin (3):
>>    uapi: introduce kernel uAPI headers import
>>    uapi: import VDUSE header
>>    vduse: use imported VDUSE uAPI header
>>
>>   devtools/check-linux-uapi.sh           |  85 ++++++
>>   devtools/import-linux-uapi.sh          | 119 +++++++++
>>   doc/guides/contributing/index.rst      |   1 +
>>   doc/guides/contributing/linux_uapi.rst |  71 +++++
>>   lib/vhost/meson.build                  |   5 +-
>>   lib/vhost/vduse.c                      |   2 +-
>>   lib/vhost/vduse.h                      |  22 --
>>   linux-headers/uapi/.gitignore          |   4 +
>>   linux-headers/uapi/linux/vduse.h       | 353 +++++++++++++++++++++++++
>>   linux-headers/uapi/version             |   1 +
>>   meson.build                            |   8 +-
>>   11 files changed, 642 insertions(+), 29 deletions(-)
>>   create mode 100755 devtools/check-linux-uapi.sh
>>   create mode 100755 devtools/import-linux-uapi.sh
>>   create mode 100644 doc/guides/contributing/linux_uapi.rst
>>   create mode 100644 linux-headers/uapi/.gitignore
>>   create mode 100644 linux-headers/uapi/linux/vduse.h
>>   create mode 100644 linux-headers/uapi/version
>>
> 

Re: [RFC v3 0/3] Import Kernel uAPI header files

[Ferruh Yigit]

On 9/12/2024 1:08 PM, Maxime Coquelin wrote:
> Hi Ferruh,
> 
> On 9/12/24 10:30, Ferruh Yigit wrote:
>> On 9/11/2024 8:32 PM, Maxime Coquelin wrote:
>>> This series enables importing Linux Kernel uAPI headers
>>> into the DPDK repository. It aims at solving alignment
>>> issues between the build system and the system applications
>>> linked ot DPDK libraries are run on.
>>>
>>
>> Hi Maxime,
>>
>> Is the imported Linux Kernel uAPI headers are always backward compatible?
> 
> Yes, at least that's what is advertised! :)
> 
> "
> The binary interface provided by the kernel to user space cannot be
> broken except under the most severe circumstances.
> "
> 

Ack.

> 
>> If I build and run on identical platform, lets say has kernel v5.4,
>> and DPDK has the kernel uAPI header from v6.10, is this has any chance
>> to create backward compatibility issues?
> 
> It should not if backward compatibility is preserved as advertised.
> 
>> Or can it enable some features which are indeed not exist/supported in
>> the platform that has old version of kernel?
> 
> It can enable new feature, like for example a new ioctl command.
> 
> In this case, it should return something like ENOIOCTLCMD if the Kernel
> does not support it.
> 

OK. I think this has the same risk with building dpdk application in a
platform with newer kernel version, and distribute it to the another
platform with older kernel. If interface is backward compatible, this
should be OK.


In this approach we are duplicating the kernel headers, which is not a
great option. Also this requires maintenance, updating kernel headers in
DPDK as needed, which is not great in long run.

I guess intention is to enable a feature that comes with newer kernel
versions, if so why not force enable it in DPDK, like:
#ifndef FEATURE_MACRO
#define FEATURE_MACRO
#define X
#define Y
#define Z
#endif

If the kernel uAPI is stable, moving these defines to library should be
OK. When this feature hits the distribution kernels we can remove
defines from the DPDK library.


OR, won't it help if whoever building DPDK, download kernel headers
package for necessary version, and build DPDK with them?
For this documenting required versions, and a how to documentation to
explain meson commands etc.. may be sufficient.


>>
>>> It can also help simplify spaghetti code done to support
>>> different versions of the Linux Kernel headers used by
>>> the build system.
>>>
>>> Guidelines and import script are also part of first patch.
>>> A uAPI checker script is added in this 3rd RFC, goals is to
>>> use it in CI for any patch touching linux-headers/uapi.
>>>
>>> Follow-up patches are an example of imported uAPI inclusion
>>> of VDUSE header into the Vhost library.
>>>
>>> Morten, I did not (again) apply your Ack on patch 1, as it
>>> has some significant changes and additions.
>>>
>>> Changes in RFC v3:
>>> ==================
>>> - Support nested headers include
>>> - Interactive mode to select which headers to include
>>> - Store version in a file instead of git commit messages
>>> - All imported headers aligned on same version
>>> - Improve loops in scripts (David)
>>> - Update documentation
>>>
>>>
>>> Changes in RFC v2:
>>> ==================
>>> - Fix typos in documentation and commit messags (David, Morten)
>>> - Add uAPI checker script
>>> - Add uAPI to global_inc
>>> - Fix build issues on FreeBSD and documentation (CI, David)
>>> - Simplify import script (David)
>>>
>>> Maxime Coquelin (3):
>>>    uapi: introduce kernel uAPI headers import
>>>    uapi: import VDUSE header
>>>    vduse: use imported VDUSE uAPI header
>>>
>>>   devtools/check-linux-uapi.sh           |  85 ++++++
>>>   devtools/import-linux-uapi.sh          | 119 +++++++++
>>>   doc/guides/contributing/index.rst      |   1 +
>>>   doc/guides/contributing/linux_uapi.rst |  71 +++++
>>>   lib/vhost/meson.build                  |   5 +-
>>>   lib/vhost/vduse.c                      |   2 +-
>>>   lib/vhost/vduse.h                      |  22 --
>>>   linux-headers/uapi/.gitignore          |   4 +
>>>   linux-headers/uapi/linux/vduse.h       | 353 +++++++++++++++++++++++++
>>>   linux-headers/uapi/version             |   1 +
>>>   meson.build                            |   8 +-
>>>   11 files changed, 642 insertions(+), 29 deletions(-)
>>>   create mode 100755 devtools/check-linux-uapi.sh
>>>   create mode 100755 devtools/import-linux-uapi.sh
>>>   create mode 100644 doc/guides/contributing/linux_uapi.rst
>>>   create mode 100644 linux-headers/uapi/.gitignore
>>>   create mode 100644 linux-headers/uapi/linux/vduse.h
>>>   create mode 100644 linux-headers/uapi/version
>>>
>>
> 

Re: [RFC v3 0/3] Import Kernel uAPI header files

[Maxime Coquelin]

On 9/12/24 15:16, Ferruh Yigit wrote:
> On 9/12/2024 1:08 PM, Maxime Coquelin wrote:
>> Hi Ferruh,
>>
>> On 9/12/24 10:30, Ferruh Yigit wrote:
>>> On 9/11/2024 8:32 PM, Maxime Coquelin wrote:
>>>> This series enables importing Linux Kernel uAPI headers
>>>> into the DPDK repository. It aims at solving alignment
>>>> issues between the build system and the system applications
>>>> linked ot DPDK libraries are run on.
>>>>
>>>
>>> Hi Maxime,
>>>
>>> Is the imported Linux Kernel uAPI headers are always backward compatible?
>>
>> Yes, at least that's what is advertised! :)
>>
>> "
>> The binary interface provided by the kernel to user space cannot be
>> broken except under the most severe circumstances.
>> "
>>
> 
> Ack.
> 
>>
>>> If I build and run on identical platform, lets say has kernel v5.4,
>>> and DPDK has the kernel uAPI header from v6.10, is this has any chance
>>> to create backward compatibility issues?
>>
>> It should not if backward compatibility is preserved as advertised.
>>
>>> Or can it enable some features which are indeed not exist/supported in
>>> the platform that has old version of kernel?
>>
>> It can enable new feature, like for example a new ioctl command.
>>
>> In this case, it should return something like ENOIOCTLCMD if the Kernel
>> does not support it.
>>
> 
> OK. I think this has the same risk with building dpdk application in a
> platform with newer kernel version, and distribute it to the another
> platform with older kernel. If interface is backward compatible, this
> should be OK.

Yes.

> 
> In this approach we are duplicating the kernel headers, which is not a
> great option. Also this requires maintenance, updating kernel headers in
> DPDK as needed, which is not great in long run.
> 
> I guess intention is to enable a feature that comes with newer kernel
> versions, if so why not force enable it in DPDK, like:
> #ifndef FEATURE_MACRO
> #define FEATURE_MACRO
> #define X
> #define Y
> #define Z
> #endif

That's exactly what my proposal aims at fixing!
We do this a lot in Vhost library, and this is a mess.

> If the kernel uAPI is stable, moving these defines to library should be
> OK.

I see some issues with such solution:

1. This is error prone (e.g. involuntary modifications to redefined
    code). It can be also quite complex/big to redefine (look at
    linux/userfaultfd.h)
2. We copy "GPL v2 WITH Linux-syscall-note" definitions into BSD
    licensed code (but IANAL).

> When this feature hits the distribution kernels we can remove
> defines from the DPDK library.

I think this is easier said than done.
Distribution Kernels backport feature on a need basis, it is difficult 
to ensure all the distribution Kernels have a given feature included.

> 
> OR, won't it help if whoever building DPDK, download kernel headers
> package for necessary version, and build DPDK with them?
> For this documenting required versions, and a how to documentation to
> explain meson commands etc.. may be sufficient.

That would add another step to building DPDK.
It would need to be a mandatory step, otherwise we would have to 
continue with all these #ifndef #define..

I think my proposal is the best compromise:
1. No need for extra pre-requisites for building DPDK
2. UAPI included on a need basis (e.g. only vduse.h, only virtio
    headers, etc..)
3. UAPI headers isolated in a dedicated directory, ensuring no
    involuntary modifications are done to these files thanks to provided
    checker that can be integrated into CI.License of the UAPI headers
    are preseved.
4. Helps with builds reproducibility.

> 
>>>
>>>> It can also help simplify spaghetti code done to support
>>>> different versions of the Linux Kernel headers used by
>>>> the build system.
>>>>
>>>> Guidelines and import script are also part of first patch.
>>>> A uAPI checker script is added in this 3rd RFC, goals is to
>>>> use it in CI for any patch touching linux-headers/uapi.
>>>>
>>>> Follow-up patches are an example of imported uAPI inclusion
>>>> of VDUSE header into the Vhost library.
>>>>
>>>> Morten, I did not (again) apply your Ack on patch 1, as it
>>>> has some significant changes and additions.
>>>>
>>>> Changes in RFC v3:
>>>> ==================
>>>> - Support nested headers include
>>>> - Interactive mode to select which headers to include
>>>> - Store version in a file instead of git commit messages
>>>> - All imported headers aligned on same version
>>>> - Improve loops in scripts (David)
>>>> - Update documentation
>>>>
>>>>
>>>> Changes in RFC v2:
>>>> ==================
>>>> - Fix typos in documentation and commit messags (David, Morten)
>>>> - Add uAPI checker script
>>>> - Add uAPI to global_inc
>>>> - Fix build issues on FreeBSD and documentation (CI, David)
>>>> - Simplify import script (David)
>>>>
>>>> Maxime Coquelin (3):
>>>>     uapi: introduce kernel uAPI headers import
>>>>     uapi: import VDUSE header
>>>>     vduse: use imported VDUSE uAPI header
>>>>
>>>>    devtools/check-linux-uapi.sh           |  85 ++++++
>>>>    devtools/import-linux-uapi.sh          | 119 +++++++++
>>>>    doc/guides/contributing/index.rst      |   1 +
>>>>    doc/guides/contributing/linux_uapi.rst |  71 +++++
>>>>    lib/vhost/meson.build                  |   5 +-
>>>>    lib/vhost/vduse.c                      |   2 +-
>>>>    lib/vhost/vduse.h                      |  22 --
>>>>    linux-headers/uapi/.gitignore          |   4 +
>>>>    linux-headers/uapi/linux/vduse.h       | 353 +++++++++++++++++++++++++
>>>>    linux-headers/uapi/version             |   1 +
>>>>    meson.build                            |   8 +-
>>>>    11 files changed, 642 insertions(+), 29 deletions(-)
>>>>    create mode 100755 devtools/check-linux-uapi.sh
>>>>    create mode 100755 devtools/import-linux-uapi.sh
>>>>    create mode 100644 doc/guides/contributing/linux_uapi.rst
>>>>    create mode 100644 linux-headers/uapi/.gitignore
>>>>    create mode 100644 linux-headers/uapi/linux/vduse.h
>>>>    create mode 100644 linux-headers/uapi/version
>>>>
>>>
>>
> 

Re: [RFC v3 1/3] uapi: introduce kernel uAPI headers import

[David Marchand]

On Wed, Sep 11, 2024 at 9:32 PM Maxime Coquelin
<maxime.coquelin@redhat.com> wrote:
>
> This patch introduces uAPI headers import into the DPDK
> repository. This import is possible thanks to Linux Kernel
> licence exception for syscalls:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/LICENSES/exceptions/Linux-syscall-note
>
> Header files are have to be explicitly imported.
>
> Guidelines are provided in the documentation, and helper
> scripts are also provided to ensure proper importation of the
> header (unmodified content from a released Kernel version):
>  - import-linux-uapi.sh: used to add and update headers and
>  their dependencies to linux-headers/uapi/
>  - check-linux-uapi.sh: used to check all headers are valid
>
> Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>

I have been trying this script to import linux/vfio.h and cleanup its
usage in DPDK.

There is one issue that was raised.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/vfio.h#n1573

struct vfio_bitmap {
__u64        pgsize; /* page size for bitmap in bytes */
__u64        size; /* in bytes */
__u64 __user *data; /* one bit per page */
};

The __user annotation is sanitized by the headers install script in
the kernel, but the dpdk import script is missing this part.
Such sanitizations breaks the check script.

We could invert the logic in the check script: instead of "restoring"
an imported header, the check would convert a freshly downloaded
header and compare it against the imported header in dpdk.
One thing though is that we would need a copy of the "conversion"
function in the two scripts.

One idea.. can we have a single script?

# Interactive mode, with questions about what to import if dependencies exist:
$ devtools/linux-uapi.sh import linux/vfio.h v6.10

# Non interactive mode, the script uses the version file and imported headers:
$ devtools/linux-uapi.sh check


Regardless of this suggestion, I have some nits about the shell scripts below:

> ---
>  devtools/check-linux-uapi.sh           |  85 ++++++++++++++++++
>  devtools/import-linux-uapi.sh          | 119 +++++++++++++++++++++++++
>  doc/guides/contributing/index.rst      |   1 +
>  doc/guides/contributing/linux_uapi.rst |  71 +++++++++++++++
>  linux-headers/uapi/.gitignore          |   4 +
>  linux-headers/uapi/version             |   1 +
>  meson.build                            |   8 +-
>  7 files changed, 287 insertions(+), 2 deletions(-)
>  create mode 100755 devtools/check-linux-uapi.sh
>  create mode 100755 devtools/import-linux-uapi.sh
>  create mode 100644 doc/guides/contributing/linux_uapi.rst
>  create mode 100644 linux-headers/uapi/.gitignore
>  create mode 100644 linux-headers/uapi/version
>
> diff --git a/devtools/check-linux-uapi.sh b/devtools/check-linux-uapi.sh
> new file mode 100755
> index 0000000000..fc42c03169
> --- /dev/null
> +++ b/devtools/check-linux-uapi.sh
> @@ -0,0 +1,85 @@
> +#!/bin/sh

It should not be too hard to pass -e, see suggestions.


> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2024 Red Hat, Inc.
> +
> +#
> +# Check Linux Kernel uAPI header files
> +#
> +
> +base_url="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/include/uapi/"
> +base_path="linux-headers/uapi/"
> +errors=0
> +version=""
> +
> +print_usage()
> +{
> +       echo "Usage: $(basename $0) [-h]"
> +}
> +
> +check_uapi_header() {
> +       path=$1
> +       file=${1#$base_path}
> +
> +       cp -f $path $tmpinput1
> +
> +       # Restore includes fixups done by import-linux-uapi.sh
> +       sed -i "s|//#include <linux/compiler.h>|#include <linux/compiler.h>|g" $tmpinput1
> +       sed -i "s|#include <uapi/|#include <|g" $tmpinput1
> +
> +       url="${base_url}${file}?h=${version}"
> +       echo -n "Checking $file... "
> +       curl -s -f -o $tmpinput2 $url
> +       if [ $? -ne 0 ]; then

I always have trouble remembering the value of $?... maybe simpler:
if ! curl -s -f -o $tmpinput2 $url; then

Doing so makes it possible to pass a -e in the shebang.

> +               echo "Failed to download $url"
> +               exit 1

And this is a shell function, prefer return 1, rather than make the
whole script exit.

> +       fi
> +
> +       diff -q $tmpinput1 $tmpinput2 >/dev/null
> +       if [ $? -ne 0 ]; then

Idem:
if ! diff -q $tmpinput1 $tmpinput2 >/dev/null

> +               echo "KO"
> +               diff -u $tmpinput1 $tmpinput2
> +               errors=$((errors+1))

Touching the global variable errors should be done by the caller from
my pov, so here, return 1.

> +       else
> +               echo "OK"
> +       fi
> +}
> +
> +while getopts hv ARG ; do
> +       case $ARG in
> +               h )
> +                       print_usage
> +                       exit 0
> +                       ;;
> +               ? )
> +                       print_usage
> +                       exit 1
> +                       ;;
> +       esac
> +done
> +
> +shift $(($OPTIND - 1))
> +if [ $# -ne 0 ]; then
> +       print_usage
> +       exit 1
> +fi
> +
> +cd $(dirname $0)/..
> +
> +tmpinput1=$(mktemp -t dpdk.checkuapi.XXXXXX)
> +tmpinput2=$(mktemp -t dpdk.checkuapi.XXXXXX)
> +trap "rm -f '$tmpinput1 $tmpinput2'" INT
> +
> +version=$(< ${base_path}/version)
> +
> +echo "Checking imported headers for version ${version}"
> +
> +for filename in $(find $base_path -name "*.h" -type f); do
> +       check_uapi_header "${filename}" </dev/null

No need for </dev/null afaics.

And I would increment the global var errors here, so something like:
check_uapi_header "${filename}" || errors=$((errors + 1))

> +done
> +
> +echo "$errors error(s) found"
> +
> +rm -f $tmpinput1 $tmpinput2
> +trap - INT
> +
> +exit $errors
> diff --git a/devtools/import-linux-uapi.sh b/devtools/import-linux-uapi.sh
> new file mode 100755
> index 0000000000..ff15e23821
> --- /dev/null
> +++ b/devtools/import-linux-uapi.sh
> @@ -0,0 +1,119 @@
> +#!/bin/sh

Similarly, I would try to make this script runable with -e.


> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2024 Red Hat, Inc.
> +
> +#
> +# Import Linux Kernel uAPI header file
> +#
> +
> +base_url="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/include/uapi/"
> +base_path="linux-headers/uapi/"
> +version=""
> +file=""
> +
> +print_usage()
> +{
> +       echo "Usage: $(basename $0) [-h] [-a FILE] [-u VERSION]"
> +       echo "-a FILE      import Linux header file. E.g. linux/vfio.h"
> +       echo "-u VERSION   update imported list of Linux headers to a given version. E.g. v6.10"
> +}
> +
> +version_valid() {

The name is strange, this is comparing versions iiuc.
version_older_than() maybe ?


> +    printf '%s\n%s' "$1" "$2" | sort -C -V
> +}
> +
> +update_headers()
> +{
> +       local header=${filename//"$base_path"}

${filename%$base_path}

> +       local url
> +       local path
> +
> +       echo "Updating to $version"
> +       while IFS= read -d '' -r filename; do

for filename in $(find $base_path -name "*.h" -type f)

> +               header=${filename//"$base_path"/}

header=${filename%$base_path}

> +               url="${base_url}${header}?h=${version}"
> +               path="${base_path}${header}"
> +               curl -s -f -o $path $url

Maybe log an explicit error if curl fails, like in the check script.


> +       done < <(find $base_path -name "*.h" -type f -print0)
> +}
> +
> +import_header()
> +{
> +       local include
> +       local import
> +       local header=$1
> +
> +       local url="${base_url}${header}?h=${version}"
> +       local path="${base_path}${header}"
> +
> +       curl -s -f --create-dirs -o $path $url

Log an error if failing.


> +
> +       for include in $(sed -ne 's/^#include <\(.*\)>$/\1/p' $path); do
> +               if [ ! -f "${base_path}${include}" ]; then
> +                       read -p "Import $include (y/n): " import < /dev/tty && [ "$import" = 'y' ] || continue

Do we really need to force /dev/tty ?

> +                       echo "Importing $include for $path"
> +                       import_header "$include"
> +               fi
> +       done
> +}
> +
> +fixup_includes()
> +{
> +       local path=$1
> +
> +       # Do not include linux/compiler.h as done by make headers
> +       sed -i "s|^#include <linux/compiler.h>|//#include <linux/compiler.h>|g" $path
> +
> +       # Prepend include path with "uapi/" if the header is imported
> +       for include in $(sed -ne 's/^#include <\(.*\)>$/\1/p' $path); do
> +               if [ -f "${base_path}${include}" ]; then
> +                       sed -i "s|${include}|uapi/${include}|g" $path
> +               fi
> +       done
> +}
> +
> +while getopts a:u:hv opt ; do
> +       case ${opt} in
> +               a )
> +                       file=$OPTARG
> +                       ;;
> +               u )
> +                       version=$OPTARG
> +                       ;;
> +               h )
> +                       print_usage
> +                       exit 0
> +                       ;;
> +               ? )
> +                       print_usage
> +                       exit 1
> +                       ;;
> +       esac
> +done
> +
> +cd $(dirname $0)/..
> +
> +current_version=$(< ${base_path}/version)
> +
> +if [ -n "${version}" ]; then
> +       version_valid "$version" "$current_version"
> +       if [ $? -eq 0 ]; then

if version_older_than "$version" "$current_version"; then

> +               echo "Headers already up to date ($current_version >= $version)"
> +               version=$current_version
> +       else
> +               update_headers
> +       fi
> +else
> +       echo "Version not specified, using current version ($current_version)"
> +       version=$current_version
> +fi
> +
> +if [ -n "${file}" ]; then
> +       import_header $file
> +fi
> +
> +for filename in $(find $base_path -name "*.h" -type f); do
> +       fixup_includes "${filename}" </dev/null

No need for </dev/null

> +done
> +
> +echo $version > ${base_path}/version


-- 
David Marchand

Re: [RFC v3 1/3] uapi: introduce kernel uAPI headers import

[Maxime Coquelin]

On 9/17/24 13:36, David Marchand wrote:
> On Wed, Sep 11, 2024 at 9:32 PM Maxime Coquelin
> <maxime.coquelin@redhat.com> wrote:
>>
>> This patch introduces uAPI headers import into the DPDK
>> repository. This import is possible thanks to Linux Kernel
>> licence exception for syscalls:
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/LICENSES/exceptions/Linux-syscall-note
>>
>> Header files are have to be explicitly imported.
>>
>> Guidelines are provided in the documentation, and helper
>> scripts are also provided to ensure proper importation of the
>> header (unmodified content from a released Kernel version):
>>   - import-linux-uapi.sh: used to add and update headers and
>>   their dependencies to linux-headers/uapi/
>>   - check-linux-uapi.sh: used to check all headers are valid
>>
>> Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> 
> I have been trying this script to import linux/vfio.h and cleanup its
> usage in DPDK.
> 
> There is one issue that was raised.
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/vfio.h#n1573
> 
> struct vfio_bitmap {
> __u64        pgsize; /* page size for bitmap in bytes */
> __u64        size; /* in bytes */
> __u64 __user *data; /* one bit per page */
> };
> 
> The __user annotation is sanitized by the headers install script in
> the kernel, but the dpdk import script is missing this part.
> Such sanitizations breaks the check script.
> 
> We could invert the logic in the check script: instead of "restoring"
> an imported header, the check would convert a freshly downloaded
> header and compare it against the imported header in dpdk.
> One thing though is that we would need a copy of the "conversion"
> function in the two scripts.
> 
> One idea.. can we have a single script?

I think your suggestion makes sense.
Will implement it in v1 with other nits you suggest below.

Thanks,
Maxime

> # Interactive mode, with questions about what to import if dependencies exist:
> $ devtools/linux-uapi.sh import linux/vfio.h v6.10
> 
> # Non interactive mode, the script uses the version file and imported headers:
> $ devtools/linux-uapi.sh check
> 
> 
> Regardless of this suggestion, I have some nits about the shell scripts below:
> 
>> ---
>>   devtools/check-linux-uapi.sh           |  85 ++++++++++++++++++
>>   devtools/import-linux-uapi.sh          | 119 +++++++++++++++++++++++++
>>   doc/guides/contributing/index.rst      |   1 +
>>   doc/guides/contributing/linux_uapi.rst |  71 +++++++++++++++
>>   linux-headers/uapi/.gitignore          |   4 +
>>   linux-headers/uapi/version             |   1 +
>>   meson.build                            |   8 +-
>>   7 files changed, 287 insertions(+), 2 deletions(-)
>>   create mode 100755 devtools/check-linux-uapi.sh
>>   create mode 100755 devtools/import-linux-uapi.sh
>>   create mode 100644 doc/guides/contributing/linux_uapi.rst
>>   create mode 100644 linux-headers/uapi/.gitignore
>>   create mode 100644 linux-headers/uapi/version
>>
>> diff --git a/devtools/check-linux-uapi.sh b/devtools/check-linux-uapi.sh
>> new file mode 100755
>> index 0000000000..fc42c03169
>> --- /dev/null
>> +++ b/devtools/check-linux-uapi.sh
>> @@ -0,0 +1,85 @@
>> +#!/bin/sh
> 
> It should not be too hard to pass -e, see suggestions.
> 
> 
>> +# SPDX-License-Identifier: BSD-3-Clause
>> +# Copyright (c) 2024 Red Hat, Inc.
>> +
>> +#
>> +# Check Linux Kernel uAPI header files
>> +#
>> +
>> +base_url="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/include/uapi/"
>> +base_path="linux-headers/uapi/"
>> +errors=0
>> +version=""
>> +
>> +print_usage()
>> +{
>> +       echo "Usage: $(basename $0) [-h]"
>> +}
>> +
>> +check_uapi_header() {
>> +       path=$1
>> +       file=${1#$base_path}
>> +
>> +       cp -f $path $tmpinput1
>> +
>> +       # Restore includes fixups done by import-linux-uapi.sh
>> +       sed -i "s|//#include <linux/compiler.h>|#include <linux/compiler.h>|g" $tmpinput1
>> +       sed -i "s|#include <uapi/|#include <|g" $tmpinput1
>> +
>> +       url="${base_url}${file}?h=${version}"
>> +       echo -n "Checking $file... "
>> +       curl -s -f -o $tmpinput2 $url
>> +       if [ $? -ne 0 ]; then
> 
> I always have trouble remembering the value of $?... maybe simpler:
> if ! curl -s -f -o $tmpinput2 $url; then
> 
> Doing so makes it possible to pass a -e in the shebang.
> 
>> +               echo "Failed to download $url"
>> +               exit 1
> 
> And this is a shell function, prefer return 1, rather than make the
> whole script exit.
> 
>> +       fi
>> +
>> +       diff -q $tmpinput1 $tmpinput2 >/dev/null
>> +       if [ $? -ne 0 ]; then
> 
> Idem:
> if ! diff -q $tmpinput1 $tmpinput2 >/dev/null
> 
>> +               echo "KO"
>> +               diff -u $tmpinput1 $tmpinput2
>> +               errors=$((errors+1))
> 
> Touching the global variable errors should be done by the caller from
> my pov, so here, return 1.
> 
>> +       else
>> +               echo "OK"
>> +       fi
>> +}
>> +
>> +while getopts hv ARG ; do
>> +       case $ARG in
>> +               h )
>> +                       print_usage
>> +                       exit 0
>> +                       ;;
>> +               ? )
>> +                       print_usage
>> +                       exit 1
>> +                       ;;
>> +       esac
>> +done
>> +
>> +shift $(($OPTIND - 1))
>> +if [ $# -ne 0 ]; then
>> +       print_usage
>> +       exit 1
>> +fi
>> +
>> +cd $(dirname $0)/..
>> +
>> +tmpinput1=$(mktemp -t dpdk.checkuapi.XXXXXX)
>> +tmpinput2=$(mktemp -t dpdk.checkuapi.XXXXXX)
>> +trap "rm -f '$tmpinput1 $tmpinput2'" INT
>> +
>> +version=$(< ${base_path}/version)
>> +
>> +echo "Checking imported headers for version ${version}"
>> +
>> +for filename in $(find $base_path -name "*.h" -type f); do
>> +       check_uapi_header "${filename}" </dev/null
> 
> No need for </dev/null afaics.
> 
> And I would increment the global var errors here, so something like:
> check_uapi_header "${filename}" || errors=$((errors + 1))
> 
>> +done
>> +
>> +echo "$errors error(s) found"
>> +
>> +rm -f $tmpinput1 $tmpinput2
>> +trap - INT
>> +
>> +exit $errors
>> diff --git a/devtools/import-linux-uapi.sh b/devtools/import-linux-uapi.sh
>> new file mode 100755
>> index 0000000000..ff15e23821
>> --- /dev/null
>> +++ b/devtools/import-linux-uapi.sh
>> @@ -0,0 +1,119 @@
>> +#!/bin/sh
> 
> Similarly, I would try to make this script runable with -e.
> 
> 
>> +# SPDX-License-Identifier: BSD-3-Clause
>> +# Copyright (c) 2024 Red Hat, Inc.
>> +
>> +#
>> +# Import Linux Kernel uAPI header file
>> +#
>> +
>> +base_url="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/include/uapi/"
>> +base_path="linux-headers/uapi/"
>> +version=""
>> +file=""
>> +
>> +print_usage()
>> +{
>> +       echo "Usage: $(basename $0) [-h] [-a FILE] [-u VERSION]"
>> +       echo "-a FILE      import Linux header file. E.g. linux/vfio.h"
>> +       echo "-u VERSION   update imported list of Linux headers to a given version. E.g. v6.10"
>> +}
>> +
>> +version_valid() {
> 
> The name is strange, this is comparing versions iiuc.
> version_older_than() maybe ?
> 
> 
>> +    printf '%s\n%s' "$1" "$2" | sort -C -V
>> +}
>> +
>> +update_headers()
>> +{
>> +       local header=${filename//"$base_path"}
> 
> ${filename%$base_path}
> 
>> +       local url
>> +       local path
>> +
>> +       echo "Updating to $version"
>> +       while IFS= read -d '' -r filename; do
> 
> for filename in $(find $base_path -name "*.h" -type f)
> 
>> +               header=${filename//"$base_path"/}
> 
> header=${filename%$base_path}
> 
>> +               url="${base_url}${header}?h=${version}"
>> +               path="${base_path}${header}"
>> +               curl -s -f -o $path $url
> 
> Maybe log an explicit error if curl fails, like in the check script.
> 
> 
>> +       done < <(find $base_path -name "*.h" -type f -print0)
>> +}
>> +
>> +import_header()
>> +{
>> +       local include
>> +       local import
>> +       local header=$1
>> +
>> +       local url="${base_url}${header}?h=${version}"
>> +       local path="${base_path}${header}"
>> +
>> +       curl -s -f --create-dirs -o $path $url
> 
> Log an error if failing.
> 
> 
>> +
>> +       for include in $(sed -ne 's/^#include <\(.*\)>$/\1/p' $path); do
>> +               if [ ! -f "${base_path}${include}" ]; then
>> +                       read -p "Import $include (y/n): " import < /dev/tty && [ "$import" = 'y' ] || continue
> 
> Do we really need to force /dev/tty ?
> 
>> +                       echo "Importing $include for $path"
>> +                       import_header "$include"
>> +               fi
>> +       done
>> +}
>> +
>> +fixup_includes()
>> +{
>> +       local path=$1
>> +
>> +       # Do not include linux/compiler.h as done by make headers
>> +       sed -i "s|^#include <linux/compiler.h>|//#include <linux/compiler.h>|g" $path
>> +
>> +       # Prepend include path with "uapi/" if the header is imported
>> +       for include in $(sed -ne 's/^#include <\(.*\)>$/\1/p' $path); do
>> +               if [ -f "${base_path}${include}" ]; then
>> +                       sed -i "s|${include}|uapi/${include}|g" $path
>> +               fi
>> +       done
>> +}
>> +
>> +while getopts a:u:hv opt ; do
>> +       case ${opt} in
>> +               a )
>> +                       file=$OPTARG
>> +                       ;;
>> +               u )
>> +                       version=$OPTARG
>> +                       ;;
>> +               h )
>> +                       print_usage
>> +                       exit 0
>> +                       ;;
>> +               ? )
>> +                       print_usage
>> +                       exit 1
>> +                       ;;
>> +       esac
>> +done
>> +
>> +cd $(dirname $0)/..
>> +
>> +current_version=$(< ${base_path}/version)
>> +
>> +if [ -n "${version}" ]; then
>> +       version_valid "$version" "$current_version"
>> +       if [ $? -eq 0 ]; then
> 
> if version_older_than "$version" "$current_version"; then
> 
>> +               echo "Headers already up to date ($current_version >= $version)"
>> +               version=$current_version
>> +       else
>> +               update_headers
>> +       fi
>> +else
>> +       echo "Version not specified, using current version ($current_version)"
>> +       version=$current_version
>> +fi
>> +
>> +if [ -n "${file}" ]; then
>> +       import_header $file
>> +fi
>> +
>> +for filename in $(find $base_path -name "*.h" -type f); do
>> +       fixup_includes "${filename}" </dev/null
> 
> No need for </dev/null
> 
>> +done
>> +
>> +echo $version > ${base_path}/version
> 
> 

Re: [RFC v3 1/3] uapi: introduce kernel uAPI headers import

[Thomas Monjalon]

11/09/2024 21:32, Maxime Coquelin:
> This patch introduces uAPI headers import into the DPDK
> repository. This import is possible thanks to Linux Kernel
> licence exception for syscalls:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/LICENSES/exceptions/Linux-syscall-note
> 
> Header files are have to be explicitly imported.
[...]
>  linux-headers/uapi/.gitignore          |   4 +
>  linux-headers/uapi/version             |   1 +

I propose to host these new files in kernel/linux/uapi/