[RFC 0/6] Add a stricter headers check

[RFC 0/6] Add a stricter headers check

[David Marchand]

As explained in patch 6, the current headers check can not catch
issues when a public header includes an internal header.
Fixing this from meson does not seem an easy task.

The approach of this RFC is to reimplement the check in a Makefile invoked
out of DPDK (like what is done for external compilation of examples).
This has the advantage of being simple, and avoiding any (non intentional)
implicit include path coming from the meson framework.

As there was no easy way to distinguish "indirect" headers in an
installed DPDK, I chose to move those headers in a new sub directory
(patch 5).

Patch 1-4 fixes have not been marked as backport material as those bugs
seems minor/easy to fix externally (by either including missing headers,
or enabling enable_driver_sdk option).

For now, I left the check_includes= option untouched, as there may be
users of this check and this check still catches issues without
requiring to install DPDK.


-- 
David Marchand

David Marchand (6):
  baseband/acc: fix exported header
  drivers: drop export of driver headers
  eventdev: do not include driver header in DMA adapter
  drivers: fix exported headers
  build: install indirect headers to a dedicated directory
  buildtools: externally check exported headers

 .ci/linux-build.sh                            |  7 +-
 buildtools/chkincs/Makefile                   | 77 +++++++++++++++++++
 buildtools/pkg-config/meson.build             |  8 +-
 devtools/test-meson-builds.sh                 |  6 +-
 drivers/baseband/acc/meson.build              |  2 +-
 drivers/bus/vmbus/rte_vmbus_reg.h             |  6 ++
 drivers/crypto/cnxk/rte_pmd_cnxk_crypto.h     |  4 +
 drivers/net/dpaa/rte_pmd_dpaa.h               |  2 +
 drivers/net/iavf/rte_pmd_iavf.h               |  6 ++
 drivers/net/mlx5/rte_pmd_mlx5.h               |  3 +
 drivers/raw/cnxk_bphy/rte_pmd_bphy.h          | 16 ++++
 drivers/raw/cnxk_gpio/rte_pmd_cnxk_gpio.h     |  3 +
 drivers/raw/dpaa2_cmdif/rte_pmd_dpaa2_cmdif.h |  2 +
 drivers/raw/ntb/rte_pmd_ntb.h                 |  2 +
 lib/bbdev/meson.build                         |  5 +-
 lib/eal/x86/include/meson.build               |  3 +-
 lib/ethdev/meson.build                        |  6 +-
 lib/eventdev/rte_event_dma_adapter.h          |  2 +-
 lib/meson.build                               |  2 +-
 lib/mldev/meson.build                         |  5 +-
 lib/rawdev/meson.build                        |  3 +-
 lib/regexdev/meson.build                      |  3 +-
 lib/security/meson.build                      |  3 +-
 23 files changed, 153 insertions(+), 23 deletions(-)
 create mode 100644 buildtools/chkincs/Makefile

-- 
2.47.0

[RFC 1/6] baseband/acc: fix exported header

[David Marchand]

rte_acc_cfg.h relies on rte_acc_common_cfg.h.

Fixes: 32e8b7ea35dd ("baseband/acc100: refactor to segregate common code")

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 drivers/baseband/acc/meson.build | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/baseband/acc/meson.build b/drivers/baseband/acc/meson.build
index 64fcf1537a..d9fb947eaa 100644
--- a/drivers/baseband/acc/meson.build
+++ b/drivers/baseband/acc/meson.build
@@ -26,4 +26,4 @@ deps += ['bus_pci']
 
 sources = files('acc_common.c', 'rte_acc100_pmd.c', 'rte_vrb_pmd.c')
 
-headers = files('rte_acc_cfg.h')
+headers = files('rte_acc_cfg.h', 'rte_acc_common_cfg.h')
-- 
2.47.0

[RFC 2/6] drivers: drop export of driver headers

[David Marchand]

Many classes are exposing driver only headers as public headers.
Move them to the driver_sdk_headers list.

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 lib/bbdev/meson.build    | 5 ++---
 lib/ethdev/meson.build   | 6 +++---
 lib/mldev/meson.build    | 5 +----
 lib/rawdev/meson.build   | 3 ++-
 lib/regexdev/meson.build | 3 ++-
 lib/security/meson.build | 3 ++-
 6 files changed, 12 insertions(+), 13 deletions(-)

diff --git a/lib/bbdev/meson.build b/lib/bbdev/meson.build
index 07685e7578..7d035065f1 100644
--- a/lib/bbdev/meson.build
+++ b/lib/bbdev/meson.build
@@ -8,7 +8,6 @@ if is_windows
 endif
 
 sources = files('rte_bbdev.c')
-headers = files('rte_bbdev.h',
-        'rte_bbdev_pmd.h',
-        'rte_bbdev_op.h')
+headers = files('rte_bbdev.h', 'rte_bbdev_op.h')
+driver_sdk_headers = files('rte_bbdev_pmd.h')
 deps += ['mbuf']
diff --git a/lib/ethdev/meson.build b/lib/ethdev/meson.build
index f1d2586591..8ba6c708a2 100644
--- a/lib/ethdev/meson.build
+++ b/lib/ethdev/meson.build
@@ -26,11 +26,8 @@ headers = files(
         'rte_ethdev_trace_fp.h',
         'rte_dev_info.h',
         'rte_flow.h',
-        'rte_flow_driver.h',
         'rte_mtr.h',
-        'rte_mtr_driver.h',
         'rte_tm.h',
-        'rte_tm_driver.h',
 )
 
 indirect_headers += files(
@@ -42,6 +39,9 @@ driver_sdk_headers += files(
         'ethdev_driver.h',
         'ethdev_pci.h',
         'ethdev_vdev.h',
+        'rte_flow_driver.h',
+        'rte_mtr_driver.h',
+        'rte_tm_driver.h',
 )
 
 if is_linux
diff --git a/lib/mldev/meson.build b/lib/mldev/meson.build
index 2c933baad6..efc3edd288 100644
--- a/lib/mldev/meson.build
+++ b/lib/mldev/meson.build
@@ -32,11 +32,8 @@ headers = files(
         'rte_mldev.h',
 )
 
-indirect_headers += files(
-        'rte_mldev_core.h',
-)
-
 driver_sdk_headers += files(
+        'rte_mldev_core.h',
         'rte_mldev_pmd.h',
         'mldev_utils.h',
 )
diff --git a/lib/rawdev/meson.build b/lib/rawdev/meson.build
index 7dfc3d5cf9..ccfd922fda 100644
--- a/lib/rawdev/meson.build
+++ b/lib/rawdev/meson.build
@@ -8,6 +8,7 @@ if is_windows
 endif
 
 sources = files('rte_rawdev.c')
-headers = files('rte_rawdev.h', 'rte_rawdev_pmd.h')
+headers = files('rte_rawdev.h')
+driver_sdk_headers = files('rte_rawdev_pmd.h')
 
 deps += ['telemetry']
diff --git a/lib/regexdev/meson.build b/lib/regexdev/meson.build
index 426e764ece..05040051c5 100644
--- a/lib/regexdev/meson.build
+++ b/lib/regexdev/meson.build
@@ -8,6 +8,7 @@ if is_windows
 endif
 
 sources = files('rte_regexdev.c')
-headers = files('rte_regexdev.h', 'rte_regexdev_driver.h')
+headers = files('rte_regexdev.h')
 indirect_headers += files('rte_regexdev_core.h')
+driver_sdk_headers = files('rte_regexdev_driver.h')
 deps += ['mbuf']
diff --git a/lib/security/meson.build b/lib/security/meson.build
index 1034a7a299..d5431d472c 100644
--- a/lib/security/meson.build
+++ b/lib/security/meson.build
@@ -2,5 +2,6 @@
 # Copyright(c) 2017-2019 Intel Corporation
 
 sources = files('rte_security.c')
-headers = files('rte_security.h', 'rte_security_driver.h')
+headers = files('rte_security.h')
+driver_sdk_headers = files('rte_security_driver.h')
 deps += ['mempool', 'cryptodev', 'net']
-- 
2.47.0

[RFC 3/6] eventdev: do not include driver header in DMA adapter

[David Marchand]

The dma adapter header does not require including rte_dmadev_pmd.h which
is a driver header.

Fixes: 66a30a29387a ("eventdev/dma: introduce DMA adapter")

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 lib/eventdev/rte_event_dma_adapter.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/eventdev/rte_event_dma_adapter.h b/lib/eventdev/rte_event_dma_adapter.h
index 5c480b82ff..34142f26db 100644
--- a/lib/eventdev/rte_event_dma_adapter.h
+++ b/lib/eventdev/rte_event_dma_adapter.h
@@ -144,7 +144,7 @@
 #include <stdint.h>
 
 #include <rte_common.h>
-#include <rte_dmadev_pmd.h>
+#include <rte_dmadev.h>
 #include <rte_eventdev.h>
 
 #ifdef __cplusplus
-- 
2.47.0

[RFC 4/6] drivers: fix exported headers

[David Marchand]

Those headers could not be included individually as they were not
including their dependencies, or were subject to some build warnings.

Fixes: 831dba47bd36 ("bus/vmbus: add Hyper-V virtual bus support")
Fixes: 5b2a1a02dcaf ("crypto/cnxk: fix experimental version for PMD API")
Fixes: e5abbeeeefa5 ("crypto/cnxk: add PMD API for getting CPTR")
Fixes: 3ca607402c4d ("crypto/cnxk: add PMD API to flush CTX")
Fixes: 8c3495f5d2dd ("net/dpaa: support loopback API")
Fixes: 12b435bf8f2f ("net/iavf: support flex desc metadata extraction")
Fixes: 23f627e0ed28 ("net/mlx5: add flow sync API")
Fixes: f5177bdc8b76 ("net/mlx5: add GENEVE TLV options parser API")
Fixes: 7cf197684589 ("raw/cnxk_bphy: support interrupt init and cleanup")
Fixes: 633dae698070 ("raw/cnxk_gpio: add standard GPIO operations")
Fixes: 53c71586c789 ("raw/dpaa2_cmdif: support enqueue/dequeue operations")
Fixes: c39d1e082a4b ("raw/ntb: setup queues")

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 drivers/bus/vmbus/rte_vmbus_reg.h             |  6 ++++++
 drivers/crypto/cnxk/rte_pmd_cnxk_crypto.h     |  4 ++++
 drivers/net/dpaa/rte_pmd_dpaa.h               |  2 ++
 drivers/net/iavf/rte_pmd_iavf.h               |  6 ++++++
 drivers/net/mlx5/rte_pmd_mlx5.h               |  3 +++
 drivers/raw/cnxk_bphy/rte_pmd_bphy.h          | 16 ++++++++++++++++
 drivers/raw/cnxk_gpio/rte_pmd_cnxk_gpio.h     |  3 +++
 drivers/raw/dpaa2_cmdif/rte_pmd_dpaa2_cmdif.h |  2 ++
 drivers/raw/ntb/rte_pmd_ntb.h                 |  2 ++
 9 files changed, 44 insertions(+)

diff --git a/drivers/bus/vmbus/rte_vmbus_reg.h b/drivers/bus/vmbus/rte_vmbus_reg.h
index e3299aa871..95c8eb29b4 100644
--- a/drivers/bus/vmbus/rte_vmbus_reg.h
+++ b/drivers/bus/vmbus/rte_vmbus_reg.h
@@ -6,6 +6,12 @@
 #ifndef _VMBUS_REG_H_
 #define _VMBUS_REG_H_
 
+#include <stdint.h>
+
+#include <rte_common.h>
+#include <rte_stdatomic.h>
+#include <rte_uuid.h>
+
 /*
  * Hyper-V SynIC message format.
  */
diff --git a/drivers/crypto/cnxk/rte_pmd_cnxk_crypto.h b/drivers/crypto/cnxk/rte_pmd_cnxk_crypto.h
index 02278605a2..2bb0ff9e95 100644
--- a/drivers/crypto/cnxk/rte_pmd_cnxk_crypto.h
+++ b/drivers/crypto/cnxk/rte_pmd_cnxk_crypto.h
@@ -11,8 +11,12 @@
 #ifndef _PMD_CNXK_CRYPTO_H_
 #define _PMD_CNXK_CRYPTO_H_
 
+#include <stdbool.h>
 #include <stdint.h>
 
+#include <rte_compat.h>
+#include <rte_crypto.h>
+
 /* Forward declarations */
 
 /**
diff --git a/drivers/net/dpaa/rte_pmd_dpaa.h b/drivers/net/dpaa/rte_pmd_dpaa.h
index ec45633ba2..0a57e2097a 100644
--- a/drivers/net/dpaa/rte_pmd_dpaa.h
+++ b/drivers/net/dpaa/rte_pmd_dpaa.h
@@ -5,6 +5,8 @@
 #ifndef _PMD_DPAA_H_
 #define _PMD_DPAA_H_
 
+#include <stdint.h>
+
 /**
  * @file rte_pmd_dpaa.h
  *
diff --git a/drivers/net/iavf/rte_pmd_iavf.h b/drivers/net/iavf/rte_pmd_iavf.h
index 56d453fc4c..04b86a5dd7 100644
--- a/drivers/net/iavf/rte_pmd_iavf.h
+++ b/drivers/net/iavf/rte_pmd_iavf.h
@@ -15,6 +15,7 @@
  */
 
 #include <stdio.h>
+
 #include <rte_compat.h>
 #include <rte_mbuf.h>
 #include <rte_mbuf_dyn.h>
@@ -184,6 +185,7 @@ __rte_experimental
 static inline void
 rte_pmd_ifd_dump_proto_xtr_metadata(struct rte_mbuf *m)
 {
+#ifdef ALLOW_EXPERIMENTAL_API
 	union rte_pmd_ifd_proto_xtr_metadata data;
 
 	if (!rte_pmd_ifd_dynf_proto_xtr_metadata_avail())
@@ -243,6 +245,10 @@ rte_pmd_ifd_dump_proto_xtr_metadata(struct rte_mbuf *m)
 	else if (m->ol_flags & RTE_IAVF_PKT_RX_DYNF_PROTO_XTR_IP_OFFSET)
 		printf(" - Flexible descriptor's Extraction: ip_offset=%u",
 		       data.ip_ofs);
+#else
+	RTE_SET_USED(m);
+	RTE_VERIFY(false);
+#endif
 }
 
 #ifdef __cplusplus
diff --git a/drivers/net/mlx5/rte_pmd_mlx5.h b/drivers/net/mlx5/rte_pmd_mlx5.h
index fdd2f65888..f2c6aebe0b 100644
--- a/drivers/net/mlx5/rte_pmd_mlx5.h
+++ b/drivers/net/mlx5/rte_pmd_mlx5.h
@@ -5,6 +5,9 @@
 #ifndef RTE_PMD_PRIVATE_MLX5_H_
 #define RTE_PMD_PRIVATE_MLX5_H_
 
+#include <stdint.h>
+
+#include <rte_byteorder.h>
 #include <rte_compat.h>
 
 /**
diff --git a/drivers/raw/cnxk_bphy/rte_pmd_bphy.h b/drivers/raw/cnxk_bphy/rte_pmd_bphy.h
index f668e6ea82..c200c935ff 100644
--- a/drivers/raw/cnxk_bphy/rte_pmd_bphy.h
+++ b/drivers/raw/cnxk_bphy/rte_pmd_bphy.h
@@ -391,6 +391,7 @@ rte_pmd_bphy_intr_init(uint16_t dev_id)
 {
 	struct cnxk_bphy_irq_msg msg = {
 		.type = CNXK_BPHY_IRQ_MSG_TYPE_INIT,
+		.data = NULL,
 	};
 
 	return __rte_pmd_bphy_enq_deq(dev_id, CNXK_BPHY_DEF_QUEUE, &msg,
@@ -411,6 +412,7 @@ rte_pmd_bphy_intr_fini(uint16_t dev_id)
 {
 	struct cnxk_bphy_irq_msg msg = {
 		.type = CNXK_BPHY_IRQ_MSG_TYPE_FINI,
+		.data = NULL,
 	};
 
 	return __rte_pmd_bphy_enq_deq(dev_id, CNXK_BPHY_DEF_QUEUE, &msg,
@@ -470,6 +472,9 @@ rte_pmd_bphy_intr_unregister(uint16_t dev_id, int irq_num)
 {
 	struct cnxk_bphy_irq_info info = {
 		.irq_num = irq_num,
+		.handler = NULL,
+		.data = NULL,
+		.cpu = -1,
 	};
 	struct cnxk_bphy_irq_msg msg = {
 		.type = CNXK_BPHY_IRQ_MSG_TYPE_UNREGISTER,
@@ -496,6 +501,7 @@ rte_pmd_bphy_intr_mem_get(uint16_t dev_id, struct cnxk_bphy_mem *mem)
 {
 	struct cnxk_bphy_irq_msg msg = {
 		.type = CNXK_BPHY_IRQ_MSG_TYPE_MEM_GET,
+		.data = NULL,
 	};
 
 	return __rte_pmd_bphy_enq_deq(dev_id, CNXK_BPHY_DEF_QUEUE, &msg,
@@ -518,6 +524,7 @@ rte_pmd_bphy_npa_pf_func_get(uint16_t dev_id, uint16_t *pf_func)
 {
 	struct cnxk_bphy_irq_msg msg = {
 		.type = CNXK_BPHY_MSG_TYPE_NPA_PF_FUNC,
+		.data = NULL,
 	};
 
 	return __rte_pmd_bphy_enq_deq(dev_id, CNXK_BPHY_DEF_QUEUE, &msg,
@@ -540,6 +547,7 @@ rte_pmd_bphy_sso_pf_func_get(uint16_t dev_id, uint16_t *pf_func)
 {
 	struct cnxk_bphy_irq_msg msg = {
 		.type = CNXK_BPHY_MSG_TYPE_SSO_PF_FUNC,
+		.data = NULL,
 	};
 
 	return __rte_pmd_bphy_enq_deq(dev_id, CNXK_BPHY_DEF_QUEUE, &msg,
@@ -565,6 +573,7 @@ rte_pmd_bphy_cgx_get_link_info(uint16_t dev_id, uint16_t lmac,
 {
 	struct cnxk_bphy_cgx_msg msg = {
 		.type = CNXK_BPHY_CGX_MSG_TYPE_GET_LINKINFO,
+		.data = NULL,
 	};
 
 	return __rte_pmd_bphy_enq_deq(dev_id, lmac, &msg, info, sizeof(*info));
@@ -586,6 +595,7 @@ rte_pmd_bphy_cgx_intlbk_disable(uint16_t dev_id, uint16_t lmac)
 {
 	struct cnxk_bphy_cgx_msg msg = {
 		.type = CNXK_BPHY_CGX_MSG_TYPE_INTLBK_DISABLE,
+		.data = NULL,
 	};
 
 	return __rte_pmd_bphy_enq_deq(dev_id, lmac, &msg, NULL, 0);
@@ -607,6 +617,7 @@ rte_pmd_bphy_cgx_intlbk_enable(uint16_t dev_id, uint16_t lmac)
 {
 	struct cnxk_bphy_cgx_msg msg = {
 		.type = CNXK_BPHY_CGX_MSG_TYPE_INTLBK_ENABLE,
+		.data = NULL,
 	};
 
 	return __rte_pmd_bphy_enq_deq(dev_id, lmac, &msg, NULL, 0);
@@ -628,6 +639,7 @@ rte_pmd_bphy_cgx_ptp_rx_disable(uint16_t dev_id, uint16_t lmac)
 {
 	struct cnxk_bphy_cgx_msg msg = {
 		.type = CNXK_BPHY_CGX_MSG_TYPE_PTP_RX_DISABLE,
+		.data = NULL,
 	};
 
 	return __rte_pmd_bphy_enq_deq(dev_id, lmac, &msg, NULL, 0);
@@ -649,6 +661,7 @@ rte_pmd_bphy_cgx_ptp_rx_enable(uint16_t dev_id, uint16_t lmac)
 {
 	struct cnxk_bphy_cgx_msg msg = {
 		.type = CNXK_BPHY_CGX_MSG_TYPE_PTP_RX_ENABLE,
+		.data = NULL,
 	};
 
 	return __rte_pmd_bphy_enq_deq(dev_id, lmac, &msg, NULL, 0);
@@ -720,6 +733,7 @@ rte_pmd_bphy_cgx_start_rxtx(uint16_t dev_id, uint16_t lmac)
 {
 	struct cnxk_bphy_cgx_msg msg = {
 		.type = CNXK_BPHY_CGX_MSG_TYPE_START_RXTX,
+		.data = NULL,
 	};
 
 	return __rte_pmd_bphy_enq_deq(dev_id, lmac, &msg, NULL, 0);
@@ -741,6 +755,7 @@ rte_pmd_bphy_cgx_stop_rxtx(uint16_t dev_id, uint16_t lmac)
 {
 	struct cnxk_bphy_cgx_msg msg = {
 		.type = CNXK_BPHY_CGX_MSG_TYPE_STOP_RXTX,
+		.data = NULL,
 	};
 
 	return __rte_pmd_bphy_enq_deq(dev_id, lmac, &msg, NULL, 0);
@@ -765,6 +780,7 @@ rte_pmd_bphy_cgx_get_supported_fec(uint16_t dev_id, uint16_t lmac,
 {
 	struct cnxk_bphy_cgx_msg msg = {
 		.type = CNXK_BPHY_CGX_MSG_TYPE_GET_SUPPORTED_FEC,
+		.data = NULL,
 	};
 
 	return __rte_pmd_bphy_enq_deq(dev_id, lmac, &msg, fec, sizeof(*fec));
diff --git a/drivers/raw/cnxk_gpio/rte_pmd_cnxk_gpio.h b/drivers/raw/cnxk_gpio/rte_pmd_cnxk_gpio.h
index 80a37be9c7..72d138ab1d 100644
--- a/drivers/raw/cnxk_gpio/rte_pmd_cnxk_gpio.h
+++ b/drivers/raw/cnxk_gpio/rte_pmd_cnxk_gpio.h
@@ -261,6 +261,7 @@ rte_pmd_gpio_get_pin_value(uint16_t dev_id, int gpio, int *val)
 {
 	struct cnxk_gpio_msg msg = {
 		.type = CNXK_GPIO_MSG_TYPE_GET_PIN_VALUE,
+		.data = NULL,
 	};
 
 	return __rte_pmd_gpio_enq_deq(dev_id, gpio, &msg, val, sizeof(*val));
@@ -285,6 +286,7 @@ rte_pmd_gpio_get_pin_edge(uint16_t dev_id, int gpio,
 {
 	struct cnxk_gpio_msg msg = {
 		.type = CNXK_GPIO_MSG_TYPE_GET_PIN_EDGE,
+		.data = NULL,
 	};
 
 	return __rte_pmd_gpio_enq_deq(dev_id, gpio, &msg, edge, sizeof(*edge));
@@ -308,6 +310,7 @@ rte_pmd_gpio_get_pin_dir(uint16_t dev_id, int gpio, enum cnxk_gpio_pin_dir *dir)
 {
 	struct cnxk_gpio_msg msg = {
 		.type = CNXK_GPIO_MSG_TYPE_GET_PIN_DIR,
+		.data = NULL,
 	};
 
 	return __rte_pmd_gpio_enq_deq(dev_id, gpio, &msg, dir, sizeof(*dir));
diff --git a/drivers/raw/dpaa2_cmdif/rte_pmd_dpaa2_cmdif.h b/drivers/raw/dpaa2_cmdif/rte_pmd_dpaa2_cmdif.h
index 483b66eaae..7731fc6363 100644
--- a/drivers/raw/dpaa2_cmdif/rte_pmd_dpaa2_cmdif.h
+++ b/drivers/raw/dpaa2_cmdif/rte_pmd_dpaa2_cmdif.h
@@ -12,6 +12,8 @@
  *
  */
 
+#include <stdint.h>
+
 #ifdef __cplusplus
 extern "C" {
 #endif
diff --git a/drivers/raw/ntb/rte_pmd_ntb.h b/drivers/raw/ntb/rte_pmd_ntb.h
index 6591ce7931..76da3be026 100644
--- a/drivers/raw/ntb/rte_pmd_ntb.h
+++ b/drivers/raw/ntb/rte_pmd_ntb.h
@@ -5,6 +5,8 @@
 #ifndef _RTE_PMD_NTB_H_
 #define _RTE_PMD_NTB_H_
 
+#include <stdint.h>
+
 /* App needs to set/get these attrs */
 #define NTB_QUEUE_SZ_NAME           "queue_size"
 #define NTB_QUEUE_NUM_NAME          "queue_num"
-- 
2.47.0

[RFC 5/6] build: install indirect headers to a dedicated directory

[David Marchand]

The headers check currently skips "indirect" headers as instrusted via
the indirect_headers meson variable.

This headers check has some limitation that will be addressed in a next
change by inspected all exported headers.
However, exported headers lack the information about "indirect" quality.

Separate "indirect" headers by exporting them in a internal/ sub directory.
This also makes it more obvious which headers are not to be directly used
by an application.

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 buildtools/pkg-config/meson.build | 8 +++++++-
 lib/eal/x86/include/meson.build   | 3 ++-
 lib/meson.build                   | 2 +-
 3 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/buildtools/pkg-config/meson.build b/buildtools/pkg-config/meson.build
index b36add17e3..809706fe3e 100644
--- a/buildtools/pkg-config/meson.build
+++ b/buildtools/pkg-config/meson.build
@@ -27,12 +27,18 @@ endif
 # are skipped in the case of static linkage thanks to the flag --as-needed.
 
 
+subdirs = [ '.', 'internal' ]
+if get_option('include_subdir_arch') != ''
+    subdirs = [ subdirs, get_option('include_subdir_arch') ]
+    subdirs = [ subdirs, join_paths(get_option('include_subdir_arch'), 'internal')]
+endif
+
 pkg.generate(name: 'dpdk-libs',
         filebase: 'libdpdk-libs',
         description: '''Internal-only DPDK pkgconfig file. Not for direct use.
 Use libdpdk.pc instead of this file to query DPDK compile/link arguments''',
         version: meson.project_version(),
-        subdirs: [get_option('include_subdir_arch'), '.'],
+        subdirs: subdirs,
         extra_cflags: pkg_extra_cflags,
         libraries: ['-Wl,--as-needed'] + dpdk_libraries,
         libraries_private: dpdk_extra_ldflags)
diff --git a/lib/eal/x86/include/meson.build b/lib/eal/x86/include/meson.build
index 52d2f8e969..a100330208 100644
--- a/lib/eal/x86/include/meson.build
+++ b/lib/eal/x86/include/meson.build
@@ -22,5 +22,6 @@ arch_indirect_headers = files(
         'rte_byteorder_32.h',
         'rte_byteorder_64.h',
 )
-install_headers(arch_headers + arch_indirect_headers, subdir: get_option('include_subdir_arch'))
+install_headers(arch_headers, subdir: get_option('include_subdir_arch'))
+install_headers(arch_indirect_headers, subdir: join_paths(get_option('include_subdir_arch'), 'internal'))
 dpdk_chkinc_headers += arch_headers
diff --git a/lib/meson.build b/lib/meson.build
index ce92cb5537..78ada7782e 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -202,7 +202,7 @@ foreach l:libraries
     dpdk_libs_enabled += name
     dpdk_conf.set('RTE_LIB_' + name.to_upper(), 1)
     install_headers(headers)
-    install_headers(indirect_headers)
+    install_headers(indirect_headers, subdir: 'internal')
     if get_option('enable_driver_sdk')
         install_headers(driver_sdk_headers)
     endif
-- 
2.47.0

[RFC 6/6] buildtools: externally check exported headers

[David Marchand]

At the moment, the headers check (triggered via the check_includes meson
option) is run "internally", iow with compilation flags and include path
coming from the meson components.

One issue is that both internal and public headers are usually stored
in a single directory in the DPDK components.
If a lib/foo library exports a header rte_foo.h (iow rte_foo.h is part
of the headers list in lib/foo/meson.build) and this rte_foo.h includes
an internal header foo_internal.h, then the headers check won't detect
such an issue as rte_foo.h is compiled with -Ilib/foo.

Reimplement the headers check by inspecting (compiling) all DPDK headers
installed in a staging directory.
To identify the directory where DPDK headers are, this check relies on
pkg-config and skips EAL generic/ and internal/ headers.

The existing headers check (though less accurate) is kept as is,
as it provides a first level of check and may have existing users.

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 .ci/linux-build.sh            |  7 ++--
 buildtools/chkincs/Makefile   | 77 +++++++++++++++++++++++++++++++++++
 devtools/test-meson-builds.sh |  6 ++-
 3 files changed, 85 insertions(+), 5 deletions(-)
 create mode 100644 buildtools/chkincs/Makefile

diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index fdb5787621..3146d9ccd8 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -95,8 +95,6 @@ OPTS="$OPTS -Ddefault_library=$DEF_LIB"
 OPTS="$OPTS -Dbuildtype=$buildtype"
 if [ "$STDATOMIC" = "true" ]; then
 	OPTS="$OPTS -Denable_stdatomic=true"
-else
-	OPTS="$OPTS -Dcheck_includes=true"
 fi
 if [ "$MINI" = "true" ]; then
     OPTS="$OPTS -Denable_drivers=net/null"
@@ -176,13 +174,16 @@ if [ "$RUN_TESTS" = "true" ]; then
     [ "$failed" != "true" ]
 fi
 
-# Test examples compilation with an installed dpdk
+# Test headers and examples compilation with an installed dpdk
 if [ "$BUILD_EXAMPLES" = "true" ]; then
     [ -d install ] || DESTDIR=$(pwd)/install meson install -C build
     export LD_LIBRARY_PATH=$(dirname $(find $(pwd)/install -name librte_eal.so)):$LD_LIBRARY_PATH
     export PATH=$(dirname $(find $(pwd)/install -name dpdk-devbind.py)):$PATH
     export PKG_CONFIG_PATH=$(dirname $(find $(pwd)/install -name libdpdk.pc)):$PKG_CONFIG_PATH
     export PKGCONF="pkg-config --define-prefix"
+
+    make -C buildtools/chkincs O=build/buildtools/chkincs
+
     find build/examples -maxdepth 1 -type f -name "dpdk-*" |
     while read target; do
         target=${target%%:*}
diff --git a/buildtools/chkincs/Makefile b/buildtools/chkincs/Makefile
new file mode 100644
index 0000000000..838819c617
--- /dev/null
+++ b/buildtools/chkincs/Makefile
@@ -0,0 +1,77 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2024 Red Hat, Inc.
+
+ifeq ($(V),)
+Q ?= @
+else
+Q =
+endif
+
+O ?= build
+
+PKGCONF ?= pkg-config
+
+ifneq ($(shell $(PKGCONF) --exists libdpdk && echo 0),0)
+$(error "no installation of DPDK found")
+endif
+
+ifeq ($(I),)
+
+.PHONY: headers_list
+.ONESHELL:
+headers_list:
+	$(Q)for dir in $$($(PKGCONF) --cflags-only-I libdpdk); do
+		dir=$${dir##-I}
+		[ -e $$dir/rte_build_config.h ] || continue
+		$(MAKE) I="$$dir" O="$(O)"
+		break
+	done
+else
+
+HEADERS := $(shell find $(I) -name "*.h" | grep -vE $(I)'/(generic|internal)/')
+SRCS := $(patsubst $(I)/%.h, $(O)/%.c, $(HEADERS))
+.PRECIOUS: $(SRCS)
+
+PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
+CFLAGS = $(shell $(PKGCONF) --cflags libdpdk) -Wall -Wextra -Werror
+LDFLAGS = $(shell $(PKGCONF) --libs libdpdk)
+
+OBJS := $(patsubst $(I)/%.h, $(O)/%.o, $(HEADERS))
+OBJS_EXP := $(patsubst $(I)/%.h, $(O)/%+exp.o, $(HEADERS))
+OBJS_ALL := $(patsubst $(I)/%.h, $(O)/%+all.o, $(HEADERS))
+
+all: $(OBJS) $(OBJS_EXP) $(OBJS_ALL)
+
+$(O):
+	$(Q)mkdir -p $@
+
+$(O)/%.c: $(I)/%.h $(O) gen_c_file_for_header.py Makefile
+	$(Q)python3 gen_c_file_for_header.py $< $@
+
+$(O)/%.o: $(O)/%.c Makefile $(PC_FILE)
+	$(Q)$(CC) $(CFLAGS) -o $@ -c $<
+
+$(O)/%+exp.o: $(O)/%.c Makefile $(PC_FILE)
+	$(Q)$(CC) $(CFLAGS) -DALLOW_EXPERIMENTAL_API -o $@ -c $<
+
+$(O)/%+all.o: $(O)/%.c Makefile $(PC_FILE)
+	$(Q)$(CC) $(CFLAGS) -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -o $@ -c $<
+
+CXXFLAGS = $(shell $(PKGCONF) --cflags libdpdk) -Wall -Wextra -Werror
+
+OBJS_CPP := $(patsubst $(I)/%.h, $(O)/%.cpp.o, $(HEADERS))
+OBJS_CPP_EXP := $(patsubst $(I)/%.h, $(O)/%.cpp+exp.o, $(HEADERS))
+OBJS_CPP_ALL := $(patsubst $(I)/%.h, $(O)/%.cpp+all.o, $(HEADERS))
+
+all: $(OBJS_CPP) $(OBJS_CPP_EXP) $(OBJS_CPP_ALL)
+
+$(O)/%.cpp.o: $(O)/%.c Makefile $(PC_FILE)
+	$(Q)$(CXX) $(CXXFLAGS) -o $@ -c $<
+
+$(O)/%.cpp+exp.o: $(O)/%.c Makefile $(PC_FILE)
+	$(Q)$(CXX) $(CXXFLAGS) -DALLOW_EXPERIMENTAL_API -o $@ -c $<
+
+$(O)/%.cpp+all.o: $(O)/%.c Makefile $(PC_FILE)
+	$(Q)$(CXX) $(CXXFLAGS) -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -o $@ -c $<
+
+endif
diff --git a/devtools/test-meson-builds.sh b/devtools/test-meson-builds.sh
index 4fff1f7177..c3cebead70 100755
--- a/devtools/test-meson-builds.sh
+++ b/devtools/test-meson-builds.sh
@@ -248,7 +248,7 @@ generic_isa='nehalem'
 if ! check_cc_flags "-march=$generic_isa" ; then
 	generic_isa='corei7'
 fi
-build build-x86-generic cc skipABI -Dcheck_includes=true \
+build build-x86-generic cc skipABI \
 	-Dlibdir=lib -Dcpu_instruction_set=$generic_isa $use_shared
 
 # 32-bit with default compiler
@@ -318,9 +318,11 @@ if [ "$examples" = 'all' ]; then
 	done | sort -u |
 	tr '\n' ' ')
 fi
-# if pkg-config defines the necessary flags, test building some examples
+# if pkg-config defines the necessary flags, check headers and test building some examples
 if pkg-config --define-prefix libdpdk >/dev/null 2>&1; then
 	export PKGCONF="pkg-config --define-prefix"
+	echo "## Checking exported headers"
+	$MAKE -C buildtools/chkincs O=$build_path/buildtools/chkincs -j
 	for example in $examples; do
 		echo "## Building $example"
 		[ $example = helloworld ] && static=static || static= # save disk space
-- 
2.47.0

Re: [RFC 5/6] build: install indirect headers to a dedicated directory

[Bruce Richardson]

On Wed, Nov 27, 2024 at 12:26:15PM +0100, David Marchand wrote:
> The headers check currently skips "indirect" headers as instrusted via
> the indirect_headers meson variable.
> 
> This headers check has some limitation that will be addressed in a next
> change by inspected all exported headers.
> However, exported headers lack the information about "indirect" quality.
> 
> Separate "indirect" headers by exporting them in a internal/ sub directory.
> This also makes it more obvious which headers are not to be directly used
> by an application.
> 
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
>  buildtools/pkg-config/meson.build | 8 +++++++-
>  lib/eal/x86/include/meson.build   | 3 ++-
>  lib/meson.build                   | 2 +-
>  3 files changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/buildtools/pkg-config/meson.build b/buildtools/pkg-config/meson.build
> index b36add17e3..809706fe3e 100644
> --- a/buildtools/pkg-config/meson.build
> +++ b/buildtools/pkg-config/meson.build
> @@ -27,12 +27,18 @@ endif
>  # are skipped in the case of static linkage thanks to the flag --as-needed.
>  
>  
> +subdirs = [ '.', 'internal' ]
> +if get_option('include_subdir_arch') != ''
> +    subdirs = [ subdirs, get_option('include_subdir_arch') ]
> +    subdirs = [ subdirs, join_paths(get_option('include_subdir_arch'), 'internal')]

minor nit, I tend to prefer using "+=" rather than relying on flattening to
extend the arrays.


> +endif
> +
>  pkg.generate(name: 'dpdk-libs',
>          filebase: 'libdpdk-libs',
>          description: '''Internal-only DPDK pkgconfig file. Not for direct use.
>  Use libdpdk.pc instead of this file to query DPDK compile/link arguments''',
>          version: meson.project_version(),
> -        subdirs: [get_option('include_subdir_arch'), '.'],
> +        subdirs: subdirs,
>          extra_cflags: pkg_extra_cflags,
>          libraries: ['-Wl,--as-needed'] + dpdk_libraries,
>          libraries_private: dpdk_extra_ldflags)
> diff --git a/lib/eal/x86/include/meson.build b/lib/eal/x86/include/meson.build
> index 52d2f8e969..a100330208 100644
> --- a/lib/eal/x86/include/meson.build
> +++ b/lib/eal/x86/include/meson.build
> @@ -22,5 +22,6 @@ arch_indirect_headers = files(
>          'rte_byteorder_32.h',
>          'rte_byteorder_64.h',
>  )
> -install_headers(arch_headers + arch_indirect_headers, subdir: get_option('include_subdir_arch'))
> +install_headers(arch_headers, subdir: get_option('include_subdir_arch'))
> +install_headers(arch_indirect_headers, subdir: join_paths(get_option('include_subdir_arch'), 'internal'))
>  dpdk_chkinc_headers += arch_headers
> diff --git a/lib/meson.build b/lib/meson.build
> index ce92cb5537..78ada7782e 100644
> --- a/lib/meson.build
> +++ b/lib/meson.build
> @@ -202,7 +202,7 @@ foreach l:libraries
>      dpdk_libs_enabled += name
>      dpdk_conf.set('RTE_LIB_' + name.to_upper(), 1)
>      install_headers(headers)
> -    install_headers(indirect_headers)
> +    install_headers(indirect_headers, subdir: 'internal')
>      if get_option('enable_driver_sdk')
>          install_headers(driver_sdk_headers)
>      endif
> -- 
> 2.47.0
>