[dpdk-dev] [PATCH 0/8] Removal of PCI bus ABIs

[dpdk-dev] [PATCH 0/8] Removal of PCI bus ABIs

[Chenbo Xia]

As announced in the deprecation notice, most ABIs in PCI bus will be removed.

As there exist some applications that want to access PCI memory resource,
two new APIs are defined in Patch 1 and corresponding changes are applied
to testpmd in Patch 2.

Patch 3-4 clean up the unnecessary usage of PCI bus header in examples.

Patch 5-7 clean up the unused PCI related structure in kni library and related
tests and examples.

Patch 8 finally removes most of ABIs in PCI bus.

Chenbo Xia (8):
  bus/pci: add new memory resource access APIs
  app/testpmd: use PCI memory resource access APIs
  examples/ethtool: use PCI library API to get PCI address
  examples/kni: remove unused PCI bus header
  test/kni: remove setting of PCI ID and address
  examples/ip_pipeline: remove setting of PCI ID and address
  kni: replace unused variable definition with reserved bytes
  bus/pci: remove ABIs in PCI bus

 app/test-pmd/config.c                         |  38 +-
 app/test-pmd/testpmd.h                        |  46 +-
 app/test/test_kni.c                           |  57 ---
 app/test/virtual_pmd.c                        |   2 +-
 doc/guides/rel_notes/release_21_11.rst        |   7 +
 drivers/baseband/acc100/rte_acc100_pmd.c      |   2 +-
 .../fpga_5gnr_fec/rte_fpga_5gnr_fec.c         |   2 +-
 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c  |   2 +-
 drivers/bus/pci/bsd/pci.c                     |   1 -
 drivers/bus/pci/linux/pci.c                   |   1 -
 drivers/bus/pci/linux/pci_uio.c               |   1 -
 drivers/bus/pci/linux/pci_vfio.c              |   1 -
 drivers/bus/pci/meson.build                   |   4 +
 drivers/bus/pci/pci_common.c                  |  78 ++++
 drivers/bus/pci/pci_common_uio.c              |   1 -
 drivers/bus/pci/pci_driver.h                  | 402 ++++++++++++++++++
 drivers/bus/pci/pci_params.c                  |   1 -
 drivers/bus/pci/private.h                     |   3 +-
 drivers/bus/pci/rte_bus_pci.h                 | 387 ++---------------
 drivers/bus/pci/version.map                   |  28 +-
 drivers/common/cnxk/roc_platform.h            |   2 +-
 drivers/common/mlx5/linux/mlx5_common_verbs.c |   2 +-
 drivers/common/mlx5/mlx5_common_pci.c         |   2 +-
 drivers/common/octeontx2/otx2_dev.h           |   2 +-
 drivers/common/octeontx2/otx2_sec_idev.c      |   2 +-
 drivers/common/qat/qat_device.h               |   2 +-
 drivers/common/qat/qat_qp.c                   |   2 +-
 drivers/common/sfc_efx/sfc_efx.h              |   2 +-
 drivers/compress/mlx5/mlx5_compress.c         |   2 +-
 drivers/compress/octeontx/otx_zip.h           |   2 +-
 drivers/compress/qat/qat_comp.c               |   2 +-
 drivers/crypto/ccp/ccp_dev.h                  |   2 +-
 drivers/crypto/ccp/ccp_pci.h                  |   2 +-
 drivers/crypto/ccp/rte_ccp_pmd.c              |   2 +-
 drivers/crypto/cnxk/cn10k_cryptodev.c         |   2 +-
 drivers/crypto/cnxk/cn9k_cryptodev.c          |   2 +-
 drivers/crypto/mlx5/mlx5_crypto.c             |   2 +-
 drivers/crypto/nitrox/nitrox_device.h         |   2 +-
 drivers/crypto/octeontx/otx_cryptodev.c       |   2 +-
 drivers/crypto/octeontx/otx_cryptodev_ops.c   |   2 +-
 drivers/crypto/octeontx2/otx2_cryptodev.c     |   2 +-
 drivers/crypto/qat/qat_sym.c                  |   2 +-
 drivers/crypto/qat/qat_sym_pmd.c              |   2 +-
 drivers/crypto/virtio/virtio_cryptodev.c      |   2 +-
 drivers/crypto/virtio/virtio_pci.h            |   2 +-
 drivers/event/dlb2/pf/dlb2_main.h             |   2 +-
 drivers/event/dlb2/pf/dlb2_pf.c               |   2 +-
 drivers/event/octeontx/ssovf_probe.c          |   2 +-
 drivers/event/octeontx/timvf_probe.c          |   2 +-
 drivers/event/octeontx2/otx2_evdev.c          |   2 +-
 drivers/mempool/cnxk/cnxk_mempool.c           |   2 +-
 drivers/mempool/octeontx/octeontx_fpavf.c     |   2 +-
 drivers/mempool/octeontx2/otx2_mempool.c      |   2 +-
 drivers/mempool/octeontx2/otx2_mempool.h      |   2 +-
 drivers/mempool/octeontx2/otx2_mempool_irq.c  |   2 +-
 drivers/meson.build                           |   4 +
 drivers/net/ark/ark_ethdev.c                  |   2 +-
 drivers/net/avp/avp_ethdev.c                  |   2 +-
 drivers/net/bnx2x/bnx2x.h                     |   2 +-
 drivers/net/bnxt/bnxt.h                       |   2 +-
 drivers/net/bonding/rte_eth_bond_args.c       |   2 +-
 drivers/net/cxgbe/base/adapter.h              |   2 +-
 drivers/net/cxgbe/cxgbe_ethdev.c              |   2 +-
 drivers/net/e1000/em_ethdev.c                 |   2 +-
 drivers/net/e1000/em_rxtx.c                   |   2 +-
 drivers/net/e1000/igb_ethdev.c                |   2 +-
 drivers/net/e1000/igb_pf.c                    |   2 +-
 drivers/net/ena/ena_ethdev.h                  |   2 +-
 drivers/net/enic/base/vnic_dev.h              |   2 +-
 drivers/net/enic/enic_ethdev.c                |   2 +-
 drivers/net/enic/enic_main.c                  |   2 +-
 drivers/net/enic/enic_vf_representor.c        |   2 +-
 drivers/net/hinic/base/hinic_pmd_hwdev.c      |   2 +-
 drivers/net/hinic/base/hinic_pmd_hwif.c       |   2 +-
 drivers/net/hinic/base/hinic_pmd_nicio.c      |   2 +-
 drivers/net/hinic/hinic_pmd_ethdev.c          |   2 +-
 drivers/net/hns3/hns3_ethdev.c                |   2 +-
 drivers/net/hns3/hns3_rxtx.c                  |   2 +-
 drivers/net/i40e/i40e_ethdev.c                |   2 +-
 drivers/net/i40e/i40e_ethdev_vf.c             |   2 +-
 drivers/net/i40e/i40e_vf_representor.c        |   2 +-
 drivers/net/igc/igc_ethdev.c                  |   2 +-
 drivers/net/ionic/ionic.h                     |   2 +-
 drivers/net/ionic/ionic_ethdev.c              |   2 +-
 drivers/net/ipn3ke/ipn3ke_ethdev.c            |   2 +-
 drivers/net/ipn3ke/ipn3ke_representor.c       |   2 +-
 drivers/net/ipn3ke/ipn3ke_tm.c                |   2 +-
 drivers/net/ixgbe/ixgbe_ethdev.c              |   2 +-
 drivers/net/ixgbe/ixgbe_ethdev.h              |   2 +-
 drivers/net/mlx4/mlx4_ethdev.c                |   2 +-
 drivers/net/mlx5/linux/mlx5_ethdev_os.c       |   2 +-
 drivers/net/mlx5/linux/mlx5_os.c              |   2 +-
 drivers/net/mlx5/mlx5.c                       |   2 +-
 drivers/net/mlx5/mlx5_ethdev.c                |   2 +-
 drivers/net/mlx5/mlx5_txq.c                   |   2 +-
 drivers/net/netvsc/hn_vf.c                    |   2 +-
 drivers/net/octeontx/base/octeontx_pkivf.c    |   2 +-
 drivers/net/octeontx/base/octeontx_pkovf.c    |   2 +-
 drivers/net/octeontx2/otx2_ethdev_irq.c       |   2 +-
 drivers/net/qede/base/bcm_osal.h              |   2 +-
 drivers/net/sfc/sfc.h                         |   2 +-
 drivers/net/sfc/sfc_ethdev.c                  |   2 +-
 drivers/net/sfc/sfc_sriov.c                   |   2 +-
 drivers/net/thunderx/nicvf_ethdev.c           |   2 +-
 drivers/net/txgbe/txgbe_ethdev.h              |   2 +-
 drivers/net/txgbe/txgbe_flow.c                |   2 +-
 drivers/net/txgbe/txgbe_pf.c                  |   2 +-
 drivers/net/virtio/virtio_pci.h               |   2 +-
 drivers/net/virtio/virtio_pci_ethdev.c        |   2 +-
 drivers/net/vmxnet3/vmxnet3_ethdev.c          |   2 +-
 drivers/raw/cnxk_bphy/cnxk_bphy.c             |   2 +-
 drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c         |   2 +-
 drivers/raw/cnxk_bphy/cnxk_bphy_irq.c         |   2 +-
 drivers/raw/cnxk_bphy/cnxk_bphy_irq.h         |   2 +-
 drivers/raw/ifpga/ifpga_rawdev.c              |   2 +-
 drivers/raw/ifpga/rte_pmd_ifpga.c             |   2 +-
 drivers/raw/ioat/idxd_pci.c                   |   2 +-
 drivers/raw/ioat/ioat_rawdev.c                |   2 +-
 drivers/raw/ntb/ntb.c                         |   2 +-
 drivers/raw/ntb/ntb_hw_intel.c                |   2 +-
 drivers/raw/octeontx2_dma/otx2_dpi_rawdev.c   |   2 +-
 drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c     |   2 +-
 drivers/raw/octeontx2_ep/otx2_ep_rawdev.c     |   2 +-
 drivers/regex/mlx5/mlx5_regex.c               |   2 +-
 drivers/regex/mlx5/mlx5_regex_fastpath.c      |   2 +-
 drivers/vdpa/ifc/base/ifcvf_osdep.h           |   2 +-
 drivers/vdpa/ifc/ifcvf_vdpa.c                 |   2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |   2 +-
 examples/ethtool/lib/rte_ethtool.c            |  14 +-
 examples/ethtool/meson.build                  |   2 +-
 examples/ip_pipeline/kni.c                    |  16 -
 examples/kni/main.c                           |   1 -
 lib/ethdev/ethdev_pci.h                       |   2 +-
 lib/eventdev/eventdev_pmd_pci.h               |   2 +-
 lib/kni/rte_kni.h                             |   4 +-
 135 files changed, 699 insertions(+), 624 deletions(-)
 create mode 100644 drivers/bus/pci/pci_driver.h

-- 
2.17.1

[dpdk-dev] [PATCH 1/8] bus/pci: add new memory resource access APIs

[Chenbo Xia]

Some applications wants to access PCI memory resource. Currently
applications use struct rte_pci_device to access it. Since the
structure will be made internal later, this patch adds two APIs
for memory resource access.

Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
---
 doc/guides/rel_notes/release_21_11.rst |  5 ++
 drivers/bus/pci/pci_common.c           | 78 ++++++++++++++++++++++++++
 drivers/bus/pci/rte_bus_pci.h          | 36 ++++++++++++
 drivers/bus/pci/version.map            |  4 ++
 4 files changed, 123 insertions(+)

diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst
index 675b573834..1c9abb74ec 100644
--- a/doc/guides/rel_notes/release_21_11.rst
+++ b/doc/guides/rel_notes/release_21_11.rst
@@ -62,6 +62,11 @@ New Features
   * Added bus-level parsing of the devargs syntax.
   * Kept compatibility with the legacy syntax as parsing fallback.
 
+* **Added new memory resource read/write APIs in PCI bus.**
+
+  Added new memory resource read/write APIs ``rte_pci_mem_rd32`` and
+  ``rte_pci_mem_wr32`` for applications to read/write PCI memory
+  resource.
 
 Removed Items
 -------------
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index 3406e03b29..944288b132 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -25,6 +25,7 @@
 #include <rte_common.h>
 #include <rte_devargs.h>
 #include <rte_vfio.h>
+#include <rte_io.h>
 
 #include "private.h"
 
@@ -777,6 +778,83 @@ rte_pci_set_bus_master(struct rte_pci_device *dev, bool enable)
 	return 0;
 }
 
+static void *
+get_pci_mem_addr(const char *name, uint16_t idx, uint64_t offset)
+{
+	struct rte_pci_device *dev = NULL;
+	struct rte_pci_addr addr = {0};
+	struct rte_mem_resource *res = NULL;
+	bool found = false;
+
+	if (rte_pci_addr_parse(name, &addr)) {
+		RTE_LOG(ERR, EAL, "Wrong name format of PCI device (%s)", name);
+		return NULL;
+	}
+
+	FOREACH_DEVICE_ON_PCIBUS(dev) {
+		if (rte_pci_addr_cmp(&dev->addr, &addr)) {
+			continue;
+		} else {
+			found = true;
+			break;
+		}
+	}
+
+	if (!found) {
+		RTE_LOG(ERR, EAL, "Can not find the device (%s)", name);
+		return NULL;
+	}
+
+	res = &dev->mem_resource[idx];
+	if (idx >= PCI_MAX_RESOURCE || res->len == 0 || res->addr == NULL) {
+		RTE_LOG(ERR, EAL, "Invalid index of a mapped memory resourse");
+		return NULL;
+	}
+
+	if (offset >= res->len || offset + 4 > res->len) {
+		RTE_LOG(ERR, EAL, "Invalid offset of a memory resourse");
+		return NULL;
+	}
+
+	return (void *)((char *)res->addr + offset);
+}
+
+int
+rte_pci_mem_rd32(const char *name, uint16_t idx, uint32_t *data, uint64_t offset)
+{
+	void *reg_addr = NULL;
+
+	if (data == NULL) {
+		RTE_LOG(ERR, EAL, "NULL data buffer for PCI memory access");
+		return -EINVAL;
+	}
+
+	reg_addr = get_pci_mem_addr(name, idx, offset);
+	if (reg_addr == NULL)
+		return -EINVAL;
+
+	*data = rte_read32(reg_addr);
+	return 0;
+}
+
+int
+rte_pci_mem_wr32(const char *name, uint16_t idx, const uint32_t *data, uint64_t offset)
+{
+	void *reg_addr = NULL;
+
+	if (data == NULL) {
+		RTE_LOG(ERR, EAL, "NULL data buffer for PCI memory access");
+		return -EINVAL;
+	}
+
+	reg_addr = get_pci_mem_addr(name, idx, offset);
+	if (reg_addr == NULL)
+		return -EINVAL;
+
+	rte_write32(*data, reg_addr);
+	return 0;
+}
+
 struct rte_pci_bus rte_pci_bus = {
 	.bus = {
 		.scan = rte_pci_scan,
diff --git a/drivers/bus/pci/rte_bus_pci.h b/drivers/bus/pci/rte_bus_pci.h
index 583470e831..21d9dd4289 100644
--- a/drivers/bus/pci/rte_bus_pci.h
+++ b/drivers/bus/pci/rte_bus_pci.h
@@ -392,6 +392,42 @@ void rte_pci_ioport_read(struct rte_pci_ioport *p,
 void rte_pci_ioport_write(struct rte_pci_ioport *p,
 		const void *data, size_t len, off_t offset);
 
+/**
+ * Read 4 bytes from PCI memory resource.
+ *
+ * @param name
+ *   PCI device name (e.g., 0000:18:00.0).
+ * @param idx
+ *   Memory resource index.
+ * @param data
+ *   Data buffer where the bytes should be read into.
+ * @param offset
+ *   The offset into the PCI memory resource.
+ * @return
+ *  0 on success, negative value on error.
+ */
+__rte_experimental
+int
+rte_pci_mem_rd32(const char *name, uint16_t idx, uint32_t *data, uint64_t offset);
+
+/**
+ * Write 4 bytes to PCI memory resource.
+ *
+ * @param name
+ *   PCI device name (e.g., 0000:18:00.0).
+ * @param idx
+ *   Memory resource index.
+ * @param data
+ *   Buffer of data that should be written to PCI memory.
+ * @param offset
+ *   The offset into the PCI memory resource.
+ * @return
+ *  0 on success, negative value on error.
+ */
+__rte_experimental
+int
+rte_pci_mem_wr32(const char *name, uint16_t idx, const uint32_t *data, uint64_t offset);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/drivers/bus/pci/version.map b/drivers/bus/pci/version.map
index aa56439c2b..01ec836559 100644
--- a/drivers/bus/pci/version.map
+++ b/drivers/bus/pci/version.map
@@ -24,4 +24,8 @@ EXPERIMENTAL {
 
 	# added in 21.08
 	rte_pci_set_bus_master;
+
+	# added in 21.11
+	rte_pci_mem_rd32;
+	rte_pci_mem_wr32;
 };
-- 
2.17.1

Re: [dpdk-dev] [PATCH 1/8] bus/pci: add new memory resource access APIs

[Kinsella, Ray]

On 10/09/2021 03:23, Chenbo Xia wrote:
> Some applications wants to access PCI memory resource. Currently
> applications use struct rte_pci_device to access it. Since the
> structure will be made internal later, this patch adds two APIs
> for memory resource access.
> 
> Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
> ---
>  doc/guides/rel_notes/release_21_11.rst |  5 ++
>  drivers/bus/pci/pci_common.c           | 78 ++++++++++++++++++++++++++
>  drivers/bus/pci/rte_bus_pci.h          | 36 ++++++++++++
>  drivers/bus/pci/version.map            |  4 ++
>  4 files changed, 123 insertions(+)
> 
Acked-by: Ray Kinsella <mdr@ashroe.eu>

[dpdk-dev] [PATCH 2/8] app/testpmd: use PCI memory resource access APIs

[Chenbo Xia]

Currently testpmd uses struct rte_pci_device to access PCI memory
resource. Since this structure will be internal later, this patch
replaces use of rte_pci_device with new PCI memory resource access
APIs to read/write BAR 0.

Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
---
 app/test-pmd/config.c  | 38 +++++++++++-----------------------
 app/test-pmd/testpmd.h | 46 +++++++++++++++++++++---------------------
 2 files changed, 35 insertions(+), 49 deletions(-)

diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index 31d8ba1b91..a0ecda3a8e 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -950,10 +950,6 @@ vlan_id_is_invalid(uint16_t vlan_id)
 static int
 port_reg_off_is_invalid(portid_t port_id, uint32_t reg_off)
 {
-	const struct rte_pci_device *pci_dev;
-	const struct rte_bus *bus;
-	uint64_t pci_len;
-
 	if (reg_off & 0x3) {
 		fprintf(stderr,
 			"Port register offset 0x%X not aligned on a 4-byte boundary\n",
@@ -966,22 +962,6 @@ port_reg_off_is_invalid(portid_t port_id, uint32_t reg_off)
 		return 0;
 	}
 
-	bus = rte_bus_find_by_device(ports[port_id].dev_info.device);
-	if (bus && !strcmp(bus->name, "pci")) {
-		pci_dev = RTE_DEV_TO_PCI(ports[port_id].dev_info.device);
-	} else {
-		fprintf(stderr, "Not a PCI device\n");
-		return 1;
-	}
-
-	pci_len = pci_dev->mem_resource[0].len;
-	if (reg_off >= pci_len) {
-		fprintf(stderr,
-			"Port %d: register offset %u (0x%X) out of port PCI resource (length=%"PRIu64")\n",
-			port_id, (unsigned int)reg_off, (unsigned int)reg_off,
-			pci_len);
-		return 1;
-	}
 	return 0;
 }
 
@@ -1009,14 +989,14 @@ port_reg_bit_display(portid_t port_id, uint32_t reg_off, uint8_t bit_x)
 {
 	uint32_t reg_v;
 
-
 	if (port_id_is_invalid(port_id, ENABLED_WARN))
 		return;
 	if (port_reg_off_is_invalid(port_id, reg_off))
 		return;
 	if (reg_bit_pos_is_invalid(bit_x))
 		return;
-	reg_v = port_id_pci_reg_read(port_id, reg_off);
+	if (port_id_pci_reg_read(port_id, reg_off, &reg_v))
+		return;
 	display_port_and_reg_off(port_id, (unsigned)reg_off);
 	printf("bit %d=%d\n", bit_x, (int) ((reg_v & (1 << bit_x)) >> bit_x));
 }
@@ -1042,7 +1022,8 @@ port_reg_bit_field_display(portid_t port_id, uint32_t reg_off,
 	else
 		l_bit = bit1_pos, h_bit = bit2_pos;
 
-	reg_v = port_id_pci_reg_read(port_id, reg_off);
+	if (port_id_pci_reg_read(port_id, reg_off, &reg_v))
+		return;
 	reg_v >>= l_bit;
 	if (h_bit < 31)
 		reg_v &= ((1 << (h_bit - l_bit + 1)) - 1);
@@ -1060,7 +1041,8 @@ port_reg_display(portid_t port_id, uint32_t reg_off)
 		return;
 	if (port_reg_off_is_invalid(port_id, reg_off))
 		return;
-	reg_v = port_id_pci_reg_read(port_id, reg_off);
+	if (port_id_pci_reg_read(port_id, reg_off, &reg_v))
+		return;
 	display_port_reg_value(port_id, reg_off, reg_v);
 }
 
@@ -1081,7 +1063,9 @@ port_reg_bit_set(portid_t port_id, uint32_t reg_off, uint8_t bit_pos,
 			(int) bit_v);
 		return;
 	}
-	reg_v = port_id_pci_reg_read(port_id, reg_off);
+	if (port_id_pci_reg_read(port_id, reg_off, &reg_v))
+		return;
+
 	if (bit_v == 0)
 		reg_v &= ~(1 << bit_pos);
 	else
@@ -1123,7 +1107,9 @@ port_reg_bit_field_set(portid_t port_id, uint32_t reg_off,
 				(unsigned)max_v, (unsigned)max_v);
 		return;
 	}
-	reg_v = port_id_pci_reg_read(port_id, reg_off);
+	if (port_id_pci_reg_read(port_id, reg_off, &reg_v))
+		return;
+
 	reg_v &= ~(max_v << l_bit); /* Keep unchanged bits */
 	reg_v |= (value << l_bit); /* Set changed bits */
 	port_id_pci_reg_write(port_id, reg_off, reg_v);
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 16a3598e48..7922807a6e 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -678,41 +678,41 @@ mbuf_pool_find(unsigned int sock_id, uint16_t idx)
 /**
  * Read/Write operations on a PCI register of a port.
  */
-static inline uint32_t
-port_pci_reg_read(struct rte_port *port, uint32_t reg_off)
+static inline int
+port_id_pci_reg_read(portid_t pt_id, uint32_t reg_off, uint32_t *reg_v)
 {
-	const struct rte_pci_device *pci_dev;
+	struct rte_port *port = &ports[(pt_id)];
+	char name[RTE_ETH_NAME_MAX_LEN];
 	const struct rte_bus *bus;
-	void *reg_addr;
-	uint32_t reg_v;
 
 	if (!port->dev_info.device) {
 		fprintf(stderr, "Invalid device\n");
-		return 0;
+		return -1;
 	}
 
 	bus = rte_bus_find_by_device(port->dev_info.device);
 	if (bus && !strcmp(bus->name, "pci")) {
-		pci_dev = RTE_DEV_TO_PCI(port->dev_info.device);
+		rte_eth_dev_get_name_by_port(pt_id, name);
 	} else {
 		fprintf(stderr, "Not a PCI device\n");
-		return 0;
+		return -1;
 	}
 
-	reg_addr = ((char *)pci_dev->mem_resource[0].addr + reg_off);
-	reg_v = *((volatile uint32_t *)reg_addr);
-	return rte_le_to_cpu_32(reg_v);
-}
+	if (rte_pci_mem_rd32(name, 0, reg_v, reg_off)) {
+		fprintf(stderr, "Failed to read register\n");
+		return -1;
+	}
 
-#define port_id_pci_reg_read(pt_id, reg_off) \
-	port_pci_reg_read(&ports[(pt_id)], (reg_off))
+	*reg_v = rte_le_to_cpu_32(*reg_v);
+	return 0;
+}
 
 static inline void
-port_pci_reg_write(struct rte_port *port, uint32_t reg_off, uint32_t reg_v)
+port_id_pci_reg_write(portid_t pt_id, uint32_t reg_off, uint32_t reg_v)
 {
-	const struct rte_pci_device *pci_dev;
+	struct rte_port *port = &ports[(pt_id)];
+	char name[RTE_ETH_NAME_MAX_LEN];
 	const struct rte_bus *bus;
-	void *reg_addr;
 
 	if (!port->dev_info.device) {
 		fprintf(stderr, "Invalid device\n");
@@ -721,19 +721,19 @@ port_pci_reg_write(struct rte_port *port, uint32_t reg_off, uint32_t reg_v)
 
 	bus = rte_bus_find_by_device(port->dev_info.device);
 	if (bus && !strcmp(bus->name, "pci")) {
-		pci_dev = RTE_DEV_TO_PCI(port->dev_info.device);
+		rte_eth_dev_get_name_by_port(pt_id, name);
 	} else {
 		fprintf(stderr, "Not a PCI device\n");
 		return;
 	}
 
-	reg_addr = ((char *)pci_dev->mem_resource[0].addr + reg_off);
-	*((volatile uint32_t *)reg_addr) = rte_cpu_to_le_32(reg_v);
+	reg_v = rte_cpu_to_le_32(reg_v);
+	if (rte_pci_mem_wr32(name, 0, &reg_v, reg_off)) {
+		fprintf(stderr, "Failed to write register\n");
+		return;
+	}
 }
 
-#define port_id_pci_reg_write(pt_id, reg_off, reg_value) \
-	port_pci_reg_write(&ports[(pt_id)], (reg_off), (reg_value))
-
 static inline void
 get_start_cycles(uint64_t *start_tsc)
 {
-- 
2.17.1

Re: [dpdk-dev] [PATCH 2/8] app/testpmd: use PCI memory resource access APIs

[Li, Xiaoyun]

Hi

> -----Original Message-----
> From: Xia, Chenbo <chenbo.xia@intel.com>
> Sent: Friday, September 10, 2021 10:24
> To: dev@dpdk.org
> Cc: Li, Xiaoyun <xiaoyun.li@intel.com>
> Subject: [PATCH 2/8] app/testpmd: use PCI memory resource access APIs
> 
> Currently testpmd uses struct rte_pci_device to access PCI memory resource.
> Since this structure will be internal later, this patch replaces use of
> rte_pci_device with new PCI memory resource access APIs to read/write BAR 0.
> 
> Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
> ---
>  app/test-pmd/config.c  | 38 +++++++++++-----------------------
>  app/test-pmd/testpmd.h | 46 +++++++++++++++++++++---------------------
>  2 files changed, 35 insertions(+), 49 deletions(-)
> 
<snip>
> 
> @@ -1081,7 +1063,9 @@ port_reg_bit_set(portid_t port_id, uint32_t reg_off,
> uint8_t bit_pos,
>  			(int) bit_v);
>  		return;
>  	}
> -	reg_v = port_id_pci_reg_read(port_id, reg_off);
> +	if (port_id_pci_reg_read(port_id, reg_off, &reg_v))
> +		return;
> +
>  	if (bit_v == 0)
>  		reg_v &= ~(1 << bit_pos);
>  	else
> @@ -1123,7 +1107,9 @@ port_reg_bit_field_set(portid_t port_id, uint32_t
> reg_off,
>  				(unsigned)max_v, (unsigned)max_v);
>  		return;
>  	}
> -	reg_v = port_id_pci_reg_read(port_id, reg_off);
> +	if (port_id_pci_reg_read(port_id, reg_off, &reg_v))
> +		return;
> +
>  	reg_v &= ~(max_v << l_bit); /* Keep unchanged bits */
>  	reg_v |= (value << l_bit); /* Set changed bits */
>  	port_id_pci_reg_write(port_id, reg_off, reg_v); diff --git a/app/test-

In your implementation, port_id_pci_reg_write() can fail. Then the following display shouldn't be called.
So change this function to have return value and check it before use it will be better.

> pmd/testpmd.h b/app/test-pmd/testpmd.h index 16a3598e48..7922807a6e
> 100644
> --- a/app/test-pmd/testpmd.h
> +++ b/app/test-pmd/testpmd.h
<snip>
>  static inline void
> -port_pci_reg_write(struct rte_port *port, uint32_t reg_off, uint32_t reg_v)
> +port_id_pci_reg_write(portid_t pt_id, uint32_t reg_off, uint32_t reg_v)
>  {
> -	const struct rte_pci_device *pci_dev;
> +	struct rte_port *port = &ports[(pt_id)];
> +	char name[RTE_ETH_NAME_MAX_LEN];
>  	const struct rte_bus *bus;
> -	void *reg_addr;
> 
>  	if (!port->dev_info.device) {
>  		fprintf(stderr, "Invalid device\n");
> @@ -721,19 +721,19 @@ port_pci_reg_write(struct rte_port *port, uint32_t
> reg_off, uint32_t reg_v)
> 
>  	bus = rte_bus_find_by_device(port->dev_info.device);
>  	if (bus && !strcmp(bus->name, "pci")) {
> -		pci_dev = RTE_DEV_TO_PCI(port->dev_info.device);
> +		rte_eth_dev_get_name_by_port(pt_id, name);
>  	} else {
>  		fprintf(stderr, "Not a PCI device\n");
>  		return;
>  	}
> 
> -	reg_addr = ((char *)pci_dev->mem_resource[0].addr + reg_off);
> -	*((volatile uint32_t *)reg_addr) = rte_cpu_to_le_32(reg_v);
> +	reg_v = rte_cpu_to_le_32(reg_v);
> +	if (rte_pci_mem_wr32(name, 0, &reg_v, reg_off)) {
> +		fprintf(stderr, "Failed to write register\n");
> +		return;
> +	}
>  }
> 
> -#define port_id_pci_reg_write(pt_id, reg_off, reg_value) \
> -	port_pci_reg_write(&ports[(pt_id)], (reg_off), (reg_value))
> -
>  static inline void
>  get_start_cycles(uint64_t *start_tsc)
>  {
> --
> 2.17.1

Re: [dpdk-dev] [PATCH 2/8] app/testpmd: use PCI memory resource access APIs

[Xia, Chenbo]

Hi,

> -----Original Message-----
> From: Li, Xiaoyun <xiaoyun.li@intel.com>
> Sent: Thursday, September 16, 2021 2:10 PM
> To: Xia, Chenbo <chenbo.xia@intel.com>; dev@dpdk.org
> Subject: RE: [PATCH 2/8] app/testpmd: use PCI memory resource access APIs
> 
> Hi
> 
> > -----Original Message-----
> > From: Xia, Chenbo <chenbo.xia@intel.com>
> > Sent: Friday, September 10, 2021 10:24
> > To: dev@dpdk.org
> > Cc: Li, Xiaoyun <xiaoyun.li@intel.com>
> > Subject: [PATCH 2/8] app/testpmd: use PCI memory resource access APIs
> >
> > Currently testpmd uses struct rte_pci_device to access PCI memory resource.
> > Since this structure will be internal later, this patch replaces use of
> > rte_pci_device with new PCI memory resource access APIs to read/write BAR 0.
> >
> > Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
> > ---
> >  app/test-pmd/config.c  | 38 +++++++++++-----------------------
> >  app/test-pmd/testpmd.h | 46 +++++++++++++++++++++---------------------
> >  2 files changed, 35 insertions(+), 49 deletions(-)
> >
> <snip>
> >
> > @@ -1081,7 +1063,9 @@ port_reg_bit_set(portid_t port_id, uint32_t reg_off,
> > uint8_t bit_pos,
> >  			(int) bit_v);
> >  		return;
> >  	}
> > -	reg_v = port_id_pci_reg_read(port_id, reg_off);
> > +	if (port_id_pci_reg_read(port_id, reg_off, &reg_v))
> > +		return;
> > +
> >  	if (bit_v == 0)
> >  		reg_v &= ~(1 << bit_pos);
> >  	else
> > @@ -1123,7 +1107,9 @@ port_reg_bit_field_set(portid_t port_id, uint32_t
> > reg_off,
> >  				(unsigned)max_v, (unsigned)max_v);
> >  		return;
> >  	}
> > -	reg_v = port_id_pci_reg_read(port_id, reg_off);
> > +	if (port_id_pci_reg_read(port_id, reg_off, &reg_v))
> > +		return;
> > +
> >  	reg_v &= ~(max_v << l_bit); /* Keep unchanged bits */
> >  	reg_v |= (value << l_bit); /* Set changed bits */
> >  	port_id_pci_reg_write(port_id, reg_off, reg_v); diff --git a/app/test-
> 
> In your implementation, port_id_pci_reg_write() can fail. Then the following
> display shouldn't be called.
> So change this function to have return value and check it before use it will
> be better.

Make sense. Will fix in v2.

Thanks,
Chenbo

> 
> > pmd/testpmd.h b/app/test-pmd/testpmd.h index 16a3598e48..7922807a6e
> > 100644
> > --- a/app/test-pmd/testpmd.h
> > +++ b/app/test-pmd/testpmd.h
> <snip>
> >  static inline void
> > -port_pci_reg_write(struct rte_port *port, uint32_t reg_off, uint32_t reg_v)
> > +port_id_pci_reg_write(portid_t pt_id, uint32_t reg_off, uint32_t reg_v)
> >  {
> > -	const struct rte_pci_device *pci_dev;
> > +	struct rte_port *port = &ports[(pt_id)];
> > +	char name[RTE_ETH_NAME_MAX_LEN];
> >  	const struct rte_bus *bus;
> > -	void *reg_addr;
> >
> >  	if (!port->dev_info.device) {
> >  		fprintf(stderr, "Invalid device\n");
> > @@ -721,19 +721,19 @@ port_pci_reg_write(struct rte_port *port, uint32_t
> > reg_off, uint32_t reg_v)
> >
> >  	bus = rte_bus_find_by_device(port->dev_info.device);
> >  	if (bus && !strcmp(bus->name, "pci")) {
> > -		pci_dev = RTE_DEV_TO_PCI(port->dev_info.device);
> > +		rte_eth_dev_get_name_by_port(pt_id, name);
> >  	} else {
> >  		fprintf(stderr, "Not a PCI device\n");
> >  		return;
> >  	}
> >
> > -	reg_addr = ((char *)pci_dev->mem_resource[0].addr + reg_off);
> > -	*((volatile uint32_t *)reg_addr) = rte_cpu_to_le_32(reg_v);
> > +	reg_v = rte_cpu_to_le_32(reg_v);
> > +	if (rte_pci_mem_wr32(name, 0, &reg_v, reg_off)) {
> > +		fprintf(stderr, "Failed to write register\n");
> > +		return;
> > +	}
> >  }
> >
> > -#define port_id_pci_reg_write(pt_id, reg_off, reg_value) \
> > -	port_pci_reg_write(&ports[(pt_id)], (reg_off), (reg_value))
> > -
> >  static inline void
> >  get_start_cycles(uint64_t *start_tsc)
> >  {
> > --
> > 2.17.1

[dpdk-dev] [PATCH 3/8] examples/ethtool: use PCI library API to get PCI address

[Chenbo Xia]

Currently ethtool example uses struct rte_pci_device to know PCI
address of a device. As this API will be removed later in PCI bus,
this patch uses PCI library API to get the PCI address.

Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
---
 examples/ethtool/lib/rte_ethtool.c | 14 +++++++++-----
 examples/ethtool/meson.build       |  2 +-
 2 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/examples/ethtool/lib/rte_ethtool.c b/examples/ethtool/lib/rte_ethtool.c
index 4132516307..89727c9f72 100644
--- a/examples/ethtool/lib/rte_ethtool.c
+++ b/examples/ethtool/lib/rte_ethtool.c
@@ -8,7 +8,7 @@
 #include <rte_version.h>
 #include <rte_ethdev.h>
 #include <rte_ether.h>
-#include <rte_bus_pci.h>
+#include <rte_pci.h>
 #ifdef RTE_NET_IXGBE
 #include <rte_pmd_ixgbe.h>
 #endif
@@ -23,8 +23,9 @@ rte_ethtool_get_drvinfo(uint16_t port_id, struct ethtool_drvinfo *drvinfo)
 {
 	struct rte_eth_dev_info dev_info;
 	struct rte_dev_reg_info reg_info;
-	const struct rte_pci_device *pci_dev;
 	const struct rte_bus *bus = NULL;
+	char name[RTE_ETH_NAME_MAX_LEN];
+	struct rte_pci_addr addr;
 	int n;
 	int ret;
 
@@ -56,11 +57,14 @@ rte_ethtool_get_drvinfo(uint16_t port_id, struct ethtool_drvinfo *drvinfo)
 	if (dev_info.device)
 		bus = rte_bus_find_by_device(dev_info.device);
 	if (bus && !strcmp(bus->name, "pci")) {
-		pci_dev = RTE_DEV_TO_PCI(dev_info.device);
+		rte_eth_dev_get_name_by_port(port_id, name);
+		if (rte_pci_addr_parse(name, &addr)) {
+			printf("Failed to parse pci address\n");
+			return -1;
+		}
 		snprintf(drvinfo->bus_info, sizeof(drvinfo->bus_info),
 			"%04x:%02x:%02x.%x",
-			pci_dev->addr.domain, pci_dev->addr.bus,
-			pci_dev->addr.devid, pci_dev->addr.function);
+			addr.domain, addr.bus, addr.devid, addr.function);
 	} else {
 		snprintf(drvinfo->bus_info, sizeof(drvinfo->bus_info), "N/A");
 	}
diff --git a/examples/ethtool/meson.build b/examples/ethtool/meson.build
index d7f63d48af..c2dbf8ae5a 100644
--- a/examples/ethtool/meson.build
+++ b/examples/ethtool/meson.build
@@ -18,7 +18,7 @@ sources = files(
 )
 includes = include_directories('lib', 'ethtool-app')
 
-deps += 'bus_pci'
+deps += 'pci'
 if dpdk_conf.has('RTE_NET_IXGBE')
     deps += 'net_ixgbe'
 endif
-- 
2.17.1

[dpdk-dev] [PATCH 4/8] examples/kni: remove unused PCI bus header

[Chenbo Xia]

The header rte_bus_pci.h is included in kni example but nothing
in it is used. So remove it.

Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
---
 examples/kni/main.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/examples/kni/main.c b/examples/kni/main.c
index beabb3c848..6dc335c0b5 100644
--- a/examples/kni/main.c
+++ b/examples/kni/main.c
@@ -31,7 +31,6 @@
 #include <rte_lcore.h>
 #include <rte_branch_prediction.h>
 #include <rte_interrupts.h>
-#include <rte_bus_pci.h>
 #include <rte_debug.h>
 #include <rte_ether.h>
 #include <rte_ethdev.h>
-- 
2.17.1

Re: [dpdk-dev] [PATCH 4/8] examples/kni: remove unused PCI bus header

[Ferruh Yigit]

On 9/10/2021 3:23 AM, Chenbo Xia wrote:
> The header rte_bus_pci.h is included in kni example but nothing
> in it is used. So remove it.
> 
> Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>

Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>

[dpdk-dev] [PATCH 5/8] test/kni: remove setting of PCI ID and address

[Chenbo Xia]

PCI device id and address in structure rte_kni_conf are never used
in the test and kni library. So remove the related code.

Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
---
 app/test/test_kni.c | 57 ---------------------------------------------
 1 file changed, 57 deletions(-)

diff --git a/app/test/test_kni.c b/app/test/test_kni.c
index 96733554b6..aa9a316c50 100644
--- a/app/test/test_kni.c
+++ b/app/test/test_kni.c
@@ -25,7 +25,6 @@ test_kni(void)
 #include <rte_string_fns.h>
 #include <rte_mempool.h>
 #include <rte_ethdev.h>
-#include <rte_bus_pci.h>
 #include <rte_cycles.h>
 #include <rte_kni.h>
 
@@ -424,32 +423,14 @@ test_kni_processing(uint16_t port_id, struct rte_mempool *mp)
 	unsigned i;
 	struct rte_kni *kni;
 	struct rte_kni_conf conf;
-	struct rte_eth_dev_info info;
 	struct rte_kni_ops ops;
-	const struct rte_pci_device *pci_dev;
-	const struct rte_bus *bus = NULL;
 
 	if (!mp)
 		return -1;
 
 	memset(&conf, 0, sizeof(conf));
-	memset(&info, 0, sizeof(info));
 	memset(&ops, 0, sizeof(ops));
 
-	ret = rte_eth_dev_info_get(port_id, &info);
-	if (ret != 0) {
-		printf("Error during getting device (port %u) info: %s\n",
-				port_id, strerror(-ret));
-		return -1;
-	}
-
-	if (info.device)
-		bus = rte_bus_find_by_device(info.device);
-	if (bus && !strcmp(bus->name, "pci")) {
-		pci_dev = RTE_DEV_TO_PCI(info.device);
-		conf.addr = pci_dev->addr;
-		conf.id = pci_dev->id;
-	}
 	snprintf(conf.name, sizeof(conf.name), TEST_KNI_PORT);
 
 	/* core id 1 configured for kernel thread */
@@ -543,10 +524,7 @@ test_kni(void)
 	struct rte_kni *kni;
 	struct rte_mempool *mp;
 	struct rte_kni_conf conf;
-	struct rte_eth_dev_info info;
 	struct rte_kni_ops ops;
-	const struct rte_pci_device *pci_dev;
-	const struct rte_bus *bus;
 	FILE *fd;
 	DIR *dir;
 	char buf[16];
@@ -634,26 +612,9 @@ test_kni(void)
 	fclose(fd);
 
 	/* test of allocating KNI with NULL mempool pointer */
-	memset(&info, 0, sizeof(info));
 	memset(&conf, 0, sizeof(conf));
 	memset(&ops, 0, sizeof(ops));
 
-	ret = rte_eth_dev_info_get(port_id, &info);
-	if (ret != 0) {
-		printf("Error during getting device (port %u) info: %s\n",
-				port_id, strerror(-ret));
-		return -1;
-	}
-
-	if (info.device)
-		bus = rte_bus_find_by_device(info.device);
-	else
-		bus = NULL;
-	if (bus && !strcmp(bus->name, "pci")) {
-		pci_dev = RTE_DEV_TO_PCI(info.device);
-		conf.addr = pci_dev->addr;
-		conf.id = pci_dev->id;
-	}
 	conf.group_id = port_id;
 	conf.mbuf_size = MAX_PACKET_SZ;
 
@@ -678,26 +639,8 @@ test_kni(void)
 
 	/* test of allocating KNI without a name */
 	memset(&conf, 0, sizeof(conf));
-	memset(&info, 0, sizeof(info));
 	memset(&ops, 0, sizeof(ops));
 
-	ret = rte_eth_dev_info_get(port_id, &info);
-	if (ret != 0) {
-		printf("Error during getting device (port %u) info: %s\n",
-				port_id, strerror(-ret));
-		ret = -1;
-		goto fail;
-	}
-
-	if (info.device)
-		bus = rte_bus_find_by_device(info.device);
-	else
-		bus = NULL;
-	if (bus && !strcmp(bus->name, "pci")) {
-		pci_dev = RTE_DEV_TO_PCI(info.device);
-		conf.addr = pci_dev->addr;
-		conf.id = pci_dev->id;
-	}
 	conf.group_id = port_id;
 	conf.mbuf_size = MAX_PACKET_SZ;
 
-- 
2.17.1

Re: [dpdk-dev] [PATCH 5/8] test/kni: remove setting of PCI ID and address

[David Marchand]

On Fri, Sep 10, 2021 at 4:38 AM Chenbo Xia <chenbo.xia@intel.com> wrote:
>
> PCI device id and address in structure rte_kni_conf are never used
> in the test and kni library. So remove the related code.

This is probably a leftover after ethtool support removal.
ea6b39b5b847 ("kni: remove ethtool support")
ee3cac929ede ("kni: remove PCI related information")


-- 
David Marchand

Re: [dpdk-dev] [PATCH 5/8] test/kni: remove setting of PCI ID and address

[Ferruh Yigit]

On 9/10/2021 3:23 AM, Chenbo Xia wrote:
> PCI device id and address in structure rte_kni_conf are never used
> in the test and kni library. So remove the related code.
> 
> Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>

Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>


When/If you merge, please feel free to keep the ack.

[dpdk-dev] [PATCH 6/8] examples/ip_pipeline: remove setting of PCI ID and address

[Chenbo Xia]

PCI ID and address in structure rte_kni_conf are never used and
will be removed in kni library. So remove the setting of them
first in the example.

Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
---
 examples/ip_pipeline/kni.c | 16 ----------------
 1 file changed, 16 deletions(-)

diff --git a/examples/ip_pipeline/kni.c b/examples/ip_pipeline/kni.c
index a2d3331cb0..fccecc3dc6 100644
--- a/examples/ip_pipeline/kni.c
+++ b/examples/ip_pipeline/kni.c
@@ -6,7 +6,6 @@
 #include <string.h>
 
 #include <rte_ethdev.h>
-#include <rte_bus_pci.h>
 #include <rte_string_fns.h>
 
 #include "kni.h"
@@ -100,16 +99,12 @@ kni_change_mtu(uint16_t port_id, unsigned int new_mtu)
 struct kni *
 kni_create(const char *name, struct kni_params *params)
 {
-	struct rte_eth_dev_info dev_info;
 	struct rte_kni_conf kni_conf;
 	struct rte_kni_ops kni_ops;
 	struct kni *kni;
 	struct mempool *mempool;
 	struct link *link;
 	struct rte_kni *k;
-	const struct rte_pci_device *pci_dev;
-	const struct rte_bus *bus = NULL;
-	int ret;
 
 	/* Check input params */
 	if ((name == NULL) ||
@@ -124,23 +119,12 @@ kni_create(const char *name, struct kni_params *params)
 		return NULL;
 
 	/* Resource create */
-	ret = rte_eth_dev_info_get(link->port_id, &dev_info);
-	if (ret != 0)
-		return NULL;
-
 	memset(&kni_conf, 0, sizeof(kni_conf));
 	strlcpy(kni_conf.name, name, RTE_KNI_NAMESIZE);
 	kni_conf.force_bind = params->force_bind;
 	kni_conf.core_id = params->thread_id;
 	kni_conf.group_id = link->port_id;
 	kni_conf.mbuf_size = mempool->buffer_size;
-	if (dev_info.device)
-		bus = rte_bus_find_by_device(dev_info.device);
-	if (bus && !strcmp(bus->name, "pci")) {
-		pci_dev = RTE_DEV_TO_PCI(dev_info.device);
-		kni_conf.addr = pci_dev->addr;
-		kni_conf.id = pci_dev->id;
-	}
 
 	memset(&kni_ops, 0, sizeof(kni_ops));
 	kni_ops.port_id = link->port_id;
-- 
2.17.1

Re: [dpdk-dev] [PATCH 6/8] examples/ip_pipeline: remove setting of PCI ID and address

[David Marchand]

On Fri, Sep 10, 2021 at 4:38 AM Chenbo Xia <chenbo.xia@intel.com> wrote:
>
> PCI ID and address in structure rte_kni_conf are never used and
> will be removed in kni library. So remove the setting of them
> first in the example.
>
> Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>

(forgot to Cc: Ferruh in previous comment)
Ditto patch 5.

I would tag with a Fixes: on ethtool removal.
Plus, this is the same kind of cleanup wrt kni, you can squash those
two patches together.


Too bad we do not have a deprecation notice.
lib/kni/rte_kni.h:      struct rte_pci_addr addr; /* depreciated */
lib/kni/rte_kni.h:      struct rte_pci_id id; /* depreciated */

Can we plan for this cleanup?


-- 
David Marchand

Re: [dpdk-dev] [PATCH 6/8] examples/ip_pipeline: remove setting of PCI ID and address

[Xia, Chenbo]

Hi David,

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Friday, September 10, 2021 3:18 PM
> To: Xia, Chenbo <chenbo.xia@intel.com>
> Cc: dev <dev@dpdk.org>; Dumitrescu, Cristian <cristian.dumitrescu@intel.com>;
> Yigit, Ferruh <ferruh.yigit@intel.com>
> Subject: Re: [dpdk-dev] [PATCH 6/8] examples/ip_pipeline: remove setting of
> PCI ID and address
> 
> On Fri, Sep 10, 2021 at 4:38 AM Chenbo Xia <chenbo.xia@intel.com> wrote:
> >
> > PCI ID and address in structure rte_kni_conf are never used and
> > will be removed in kni library. So remove the setting of them
> > first in the example.
> >
> > Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
> 
> (forgot to Cc: Ferruh in previous comment)
> Ditto patch 5.
> 
> I would tag with a Fixes: on ethtool removal.
> Plus, this is the same kind of cleanup wrt kni, you can squash those
> two patches together.

OK. Will do.

> 
> 
> Too bad we do not have a deprecation notice.
> lib/kni/rte_kni.h:      struct rte_pci_addr addr; /* depreciated */
> lib/kni/rte_kni.h:      struct rte_pci_id id; /* depreciated */
> 
> Can we plan for this cleanup?

My bad. I did't notice this when I send the deprecation notice for PCI bus
last release.

If we all agree, we can make this happen 22.11 (Or this release if tech-board
agrees).

Thanks,
Chenbo

> 
> 
> --
> David Marchand

Re: [dpdk-dev] [PATCH 6/8] examples/ip_pipeline: remove setting of PCI ID and address

[Xia, Chenbo]

Hi David,

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Friday, September 10, 2021 3:18 PM
> To: Xia, Chenbo <chenbo.xia@intel.com>
> Cc: dev <dev@dpdk.org>; Dumitrescu, Cristian <cristian.dumitrescu@intel.com>;
> Yigit, Ferruh <ferruh.yigit@intel.com>
> Subject: Re: [dpdk-dev] [PATCH 6/8] examples/ip_pipeline: remove setting of
> PCI ID and address
> 
> On Fri, Sep 10, 2021 at 4:38 AM Chenbo Xia <chenbo.xia@intel.com> wrote:
> >
> > PCI ID and address in structure rte_kni_conf are never used and
> > will be removed in kni library. So remove the setting of them
> > first in the example.
> >
> > Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
> 
> (forgot to Cc: Ferruh in previous comment)
> Ditto patch 5.
> 
> I would tag with a Fixes: on ethtool removal.
> Plus, this is the same kind of cleanup wrt kni, you can squash those
> two patches together.

A question about squash patch 5 and 6, then how should I title the combined
patch? Because there're both test change and example change. I'd like know
your suggestion.

Thanks,
Chenbo

> 
> 
> Too bad we do not have a deprecation notice.
> lib/kni/rte_kni.h:      struct rte_pci_addr addr; /* depreciated */
> lib/kni/rte_kni.h:      struct rte_pci_id id; /* depreciated */
> 
> Can we plan for this cleanup?
> 
> 
> --
> David Marchand

Re: [dpdk-dev] [PATCH 6/8] examples/ip_pipeline: remove setting of PCI ID and address

[David Marchand]

Hi Chenbo,

On Fri, Sep 17, 2021 at 5:09 AM Xia, Chenbo <chenbo.xia@intel.com> wrote:
> > > PCI ID and address in structure rte_kni_conf are never used and
> > > will be removed in kni library. So remove the setting of them
> > > first in the example.
> > >
> > > Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
> >
> > (forgot to Cc: Ferruh in previous comment)
> > Ditto patch 5.
> >
> > I would tag with a Fixes: on ethtool removal.
> > Plus, this is the same kind of cleanup wrt kni, you can squash those
> > two patches together.
>
> A question about squash patch 5 and 6, then how should I title the combined
> patch? Because there're both test change and example change. I'd like know
> your suggestion.
> >

This is a cleanup on unused fields wrt KNI.
"kni: remove unused PCI info from test and example" lgtm.

Ferruh, opinion?

And mark with:
Fixes: ea6b39b5b847 ("kni: remove ethtool support")


-- 
David Marchand

Re: [dpdk-dev] [PATCH 6/8] examples/ip_pipeline: remove setting of PCI ID and address

[Ferruh Yigit]

On 9/17/2021 12:55 PM, David Marchand wrote:
> Hi Chenbo,
> 
> On Fri, Sep 17, 2021 at 5:09 AM Xia, Chenbo <chenbo.xia@intel.com> wrote:
>>>> PCI ID and address in structure rte_kni_conf are never used and
>>>> will be removed in kni library. So remove the setting of them
>>>> first in the example.
>>>>
>>>> Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
>>>
>>> (forgot to Cc: Ferruh in previous comment)
>>> Ditto patch 5.
>>>
>>> I would tag with a Fixes: on ethtool removal.
>>> Plus, this is the same kind of cleanup wrt kni, you can squash those
>>> two patches together.
>>
>> A question about squash patch 5 and 6, then how should I title the combined
>> patch? Because there're both test change and example change. I'd like know
>> your suggestion.
>>>
> 
> This is a cleanup on unused fields wrt KNI.
> "kni: remove unused PCI info from test and example" lgtm.
> 
> Ferruh, opinion?
> 

Make sense to merge them and above suggestion looks good. Thanks.

> And mark with:
> Fixes: ea6b39b5b847 ("kni: remove ethtool support")
> 
> 

[dpdk-dev] [PATCH 7/8] kni: replace unused variable definition with reserved bytes

[Chenbo Xia]

PCI ID and address in structure rte_kni_conf are never used. And in
order not to break ABI, replace these variables with reserved bytes.

Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
---
 lib/kni/rte_kni.h | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/lib/kni/rte_kni.h b/lib/kni/rte_kni.h
index b0eaf46104..2281abbf6a 100644
--- a/lib/kni/rte_kni.h
+++ b/lib/kni/rte_kni.h
@@ -17,7 +17,6 @@
  * and burst transmit packets to KNI interfaces.
  */
 
-#include <rte_pci.h>
 #include <rte_memory.h>
 #include <rte_mempool.h>
 #include <rte_ether.h>
@@ -66,8 +65,7 @@ struct rte_kni_conf {
 	uint32_t core_id;   /* Core ID to bind kernel thread on */
 	uint16_t group_id;  /* Group ID */
 	unsigned mbuf_size; /* mbuf size */
-	struct rte_pci_addr addr; /* depreciated */
-	struct rte_pci_id id; /* depreciated */
+	uint8_t rsvd[20];
 
 	__extension__
 	uint8_t force_bind : 1; /* Flag to bind kernel thread */
-- 
2.17.1

[dpdk-dev] [PATCH 8/8] bus/pci: remove ABIs in PCI bus

[Chenbo Xia]

As announced in the deprecation note, most of ABIs in PCI bus are
removed in this patch. Only the function rte_pci_dump is still ABI
and experimental APIs are kept for future promotion.

This patch creates a new file named pci_driver.h and moves most of
the content in original rte_bus_pci.h to it. After that, pci_driver.h
is considered the interface for drivers and rte_bus_pci.h for
applications. pci_driver.h is defined as driver_sdk_headers so that
out-of-tree drivers can use it.

Then this patch replaces the including of rte_bus_pci.h with pci_driver.h
in all related drivers.

Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
---
 app/test/virtual_pmd.c                        |   2 +-
 doc/guides/rel_notes/release_21_11.rst        |   2 +
 drivers/baseband/acc100/rte_acc100_pmd.c      |   2 +-
 .../fpga_5gnr_fec/rte_fpga_5gnr_fec.c         |   2 +-
 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c  |   2 +-
 drivers/bus/pci/bsd/pci.c                     |   1 -
 drivers/bus/pci/linux/pci.c                   |   1 -
 drivers/bus/pci/linux/pci_uio.c               |   1 -
 drivers/bus/pci/linux/pci_vfio.c              |   1 -
 drivers/bus/pci/meson.build                   |   4 +
 drivers/bus/pci/pci_common_uio.c              |   1 -
 drivers/bus/pci/pci_driver.h                  | 402 ++++++++++++++++++
 drivers/bus/pci/pci_params.c                  |   1 -
 drivers/bus/pci/private.h                     |   3 +-
 drivers/bus/pci/rte_bus_pci.h                 | 375 +---------------
 drivers/bus/pci/version.map                   |  32 +-
 drivers/common/cnxk/roc_platform.h            |   2 +-
 drivers/common/mlx5/linux/mlx5_common_verbs.c |   2 +-
 drivers/common/mlx5/mlx5_common_pci.c         |   2 +-
 drivers/common/octeontx2/otx2_dev.h           |   2 +-
 drivers/common/octeontx2/otx2_sec_idev.c      |   2 +-
 drivers/common/qat/qat_device.h               |   2 +-
 drivers/common/qat/qat_qp.c                   |   2 +-
 drivers/common/sfc_efx/sfc_efx.h              |   2 +-
 drivers/compress/mlx5/mlx5_compress.c         |   2 +-
 drivers/compress/octeontx/otx_zip.h           |   2 +-
 drivers/compress/qat/qat_comp.c               |   2 +-
 drivers/crypto/ccp/ccp_dev.h                  |   2 +-
 drivers/crypto/ccp/ccp_pci.h                  |   2 +-
 drivers/crypto/ccp/rte_ccp_pmd.c              |   2 +-
 drivers/crypto/cnxk/cn10k_cryptodev.c         |   2 +-
 drivers/crypto/cnxk/cn9k_cryptodev.c          |   2 +-
 drivers/crypto/mlx5/mlx5_crypto.c             |   2 +-
 drivers/crypto/nitrox/nitrox_device.h         |   2 +-
 drivers/crypto/octeontx/otx_cryptodev.c       |   2 +-
 drivers/crypto/octeontx/otx_cryptodev_ops.c   |   2 +-
 drivers/crypto/octeontx2/otx2_cryptodev.c     |   2 +-
 drivers/crypto/qat/qat_sym.c                  |   2 +-
 drivers/crypto/qat/qat_sym_pmd.c              |   2 +-
 drivers/crypto/virtio/virtio_cryptodev.c      |   2 +-
 drivers/crypto/virtio/virtio_pci.h            |   2 +-
 drivers/event/dlb2/pf/dlb2_main.h             |   2 +-
 drivers/event/dlb2/pf/dlb2_pf.c               |   2 +-
 drivers/event/octeontx/ssovf_probe.c          |   2 +-
 drivers/event/octeontx/timvf_probe.c          |   2 +-
 drivers/event/octeontx2/otx2_evdev.c          |   2 +-
 drivers/mempool/cnxk/cnxk_mempool.c           |   2 +-
 drivers/mempool/octeontx/octeontx_fpavf.c     |   2 +-
 drivers/mempool/octeontx2/otx2_mempool.c      |   2 +-
 drivers/mempool/octeontx2/otx2_mempool.h      |   2 +-
 drivers/mempool/octeontx2/otx2_mempool_irq.c  |   2 +-
 drivers/meson.build                           |   4 +
 drivers/net/ark/ark_ethdev.c                  |   2 +-
 drivers/net/avp/avp_ethdev.c                  |   2 +-
 drivers/net/bnx2x/bnx2x.h                     |   2 +-
 drivers/net/bnxt/bnxt.h                       |   2 +-
 drivers/net/bonding/rte_eth_bond_args.c       |   2 +-
 drivers/net/cxgbe/base/adapter.h              |   2 +-
 drivers/net/cxgbe/cxgbe_ethdev.c              |   2 +-
 drivers/net/e1000/em_ethdev.c                 |   2 +-
 drivers/net/e1000/em_rxtx.c                   |   2 +-
 drivers/net/e1000/igb_ethdev.c                |   2 +-
 drivers/net/e1000/igb_pf.c                    |   2 +-
 drivers/net/ena/ena_ethdev.h                  |   2 +-
 drivers/net/enic/base/vnic_dev.h              |   2 +-
 drivers/net/enic/enic_ethdev.c                |   2 +-
 drivers/net/enic/enic_main.c                  |   2 +-
 drivers/net/enic/enic_vf_representor.c        |   2 +-
 drivers/net/hinic/base/hinic_pmd_hwdev.c      |   2 +-
 drivers/net/hinic/base/hinic_pmd_hwif.c       |   2 +-
 drivers/net/hinic/base/hinic_pmd_nicio.c      |   2 +-
 drivers/net/hinic/hinic_pmd_ethdev.c          |   2 +-
 drivers/net/hns3/hns3_ethdev.c                |   2 +-
 drivers/net/hns3/hns3_rxtx.c                  |   2 +-
 drivers/net/i40e/i40e_ethdev.c                |   2 +-
 drivers/net/i40e/i40e_ethdev_vf.c             |   2 +-
 drivers/net/i40e/i40e_vf_representor.c        |   2 +-
 drivers/net/igc/igc_ethdev.c                  |   2 +-
 drivers/net/ionic/ionic.h                     |   2 +-
 drivers/net/ionic/ionic_ethdev.c              |   2 +-
 drivers/net/ipn3ke/ipn3ke_ethdev.c            |   2 +-
 drivers/net/ipn3ke/ipn3ke_representor.c       |   2 +-
 drivers/net/ipn3ke/ipn3ke_tm.c                |   2 +-
 drivers/net/ixgbe/ixgbe_ethdev.c              |   2 +-
 drivers/net/ixgbe/ixgbe_ethdev.h              |   2 +-
 drivers/net/mlx4/mlx4_ethdev.c                |   2 +-
 drivers/net/mlx5/linux/mlx5_ethdev_os.c       |   2 +-
 drivers/net/mlx5/linux/mlx5_os.c              |   2 +-
 drivers/net/mlx5/mlx5.c                       |   2 +-
 drivers/net/mlx5/mlx5_ethdev.c                |   2 +-
 drivers/net/mlx5/mlx5_txq.c                   |   2 +-
 drivers/net/netvsc/hn_vf.c                    |   2 +-
 drivers/net/octeontx/base/octeontx_pkivf.c    |   2 +-
 drivers/net/octeontx/base/octeontx_pkovf.c    |   2 +-
 drivers/net/octeontx2/otx2_ethdev_irq.c       |   2 +-
 drivers/net/qede/base/bcm_osal.h              |   2 +-
 drivers/net/sfc/sfc.h                         |   2 +-
 drivers/net/sfc/sfc_ethdev.c                  |   2 +-
 drivers/net/sfc/sfc_sriov.c                   |   2 +-
 drivers/net/thunderx/nicvf_ethdev.c           |   2 +-
 drivers/net/txgbe/txgbe_ethdev.h              |   2 +-
 drivers/net/txgbe/txgbe_flow.c                |   2 +-
 drivers/net/txgbe/txgbe_pf.c                  |   2 +-
 drivers/net/virtio/virtio_pci.h               |   2 +-
 drivers/net/virtio/virtio_pci_ethdev.c        |   2 +-
 drivers/net/vmxnet3/vmxnet3_ethdev.c          |   2 +-
 drivers/raw/cnxk_bphy/cnxk_bphy.c             |   2 +-
 drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c         |   2 +-
 drivers/raw/cnxk_bphy/cnxk_bphy_irq.c         |   2 +-
 drivers/raw/cnxk_bphy/cnxk_bphy_irq.h         |   2 +-
 drivers/raw/ifpga/ifpga_rawdev.c              |   2 +-
 drivers/raw/ifpga/rte_pmd_ifpga.c             |   2 +-
 drivers/raw/ioat/idxd_pci.c                   |   2 +-
 drivers/raw/ioat/ioat_rawdev.c                |   2 +-
 drivers/raw/ntb/ntb.c                         |   2 +-
 drivers/raw/ntb/ntb_hw_intel.c                |   2 +-
 drivers/raw/octeontx2_dma/otx2_dpi_rawdev.c   |   2 +-
 drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c     |   2 +-
 drivers/raw/octeontx2_ep/otx2_ep_rawdev.c     |   2 +-
 drivers/regex/mlx5/mlx5_regex.c               |   2 +-
 drivers/regex/mlx5/mlx5_regex_fastpath.c      |   2 +-
 drivers/vdpa/ifc/base/ifcvf_osdep.h           |   2 +-
 drivers/vdpa/ifc/ifcvf_vdpa.c                 |   2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |   2 +-
 lib/ethdev/ethdev_pci.h                       |   2 +-
 lib/eventdev/eventdev_pmd_pci.h               |   2 +-
 126 files changed, 546 insertions(+), 508 deletions(-)
 create mode 100644 drivers/bus/pci/pci_driver.h

diff --git a/app/test/virtual_pmd.c b/app/test/virtual_pmd.c
index 7036f401ed..555f2969ab 100644
--- a/app/test/virtual_pmd.c
+++ b/app/test/virtual_pmd.c
@@ -6,7 +6,7 @@
 #include <rte_ethdev.h>
 #include <ethdev_driver.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_malloc.h>
 #include <rte_memcpy.h>
 #include <rte_memory.h>
diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst
index 1c9abb74ec..8c24eb3235 100644
--- a/doc/guides/rel_notes/release_21_11.rst
+++ b/doc/guides/rel_notes/release_21_11.rst
@@ -112,6 +112,8 @@ ABI Changes
    Also, make sure to start the actual text at the margin.
    =======================================================
 
+* pci: Removed all ABIs defined in rte_bus_pci.h except the function
+  ``rte_pci_dump()``.
 
 Known Issues
 ------------
diff --git a/drivers/baseband/acc100/rte_acc100_pmd.c b/drivers/baseband/acc100/rte_acc100_pmd.c
index 68ba523ea9..72734784c9 100644
--- a/drivers/baseband/acc100/rte_acc100_pmd.c
+++ b/drivers/baseband/acc100/rte_acc100_pmd.c
@@ -14,7 +14,7 @@
 #include <rte_branch_prediction.h>
 #include <rte_hexdump.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #ifdef RTE_BBDEV_OFFLOAD_COST
 #include <rte_cycles.h>
 #endif
diff --git a/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c b/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c
index 6485cc824a..31cb7e5605 100644
--- a/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c
+++ b/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c
@@ -11,7 +11,7 @@
 #include <rte_mempool.h>
 #include <rte_errno.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_byteorder.h>
 #ifdef RTE_BBDEV_OFFLOAD_COST
 #include <rte_cycles.h>
diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c
index 350c4248eb..0dc1417dde 100644
--- a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c
+++ b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c
@@ -11,7 +11,7 @@
 #include <rte_mempool.h>
 #include <rte_errno.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_byteorder.h>
 #ifdef RTE_BBDEV_OFFLOAD_COST
 #include <rte_cycles.h>
diff --git a/drivers/bus/pci/bsd/pci.c b/drivers/bus/pci/bsd/pci.c
index d189bff311..b7d3a8df33 100644
--- a/drivers/bus/pci/bsd/pci.c
+++ b/drivers/bus/pci/bsd/pci.c
@@ -28,7 +28,6 @@
 #include <rte_interrupts.h>
 #include <rte_log.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
 #include <rte_common.h>
 #include <rte_launch.h>
 #include <rte_memory.h>
diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index 4d261b55ee..8f91e0233c 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -8,7 +8,6 @@
 #include <rte_log.h>
 #include <rte_bus.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
 #include <rte_malloc.h>
 #include <rte_devargs.h>
 #include <rte_memcpy.h>
diff --git a/drivers/bus/pci/linux/pci_uio.c b/drivers/bus/pci/linux/pci_uio.c
index 39ebeac2a0..de028e4874 100644
--- a/drivers/bus/pci/linux/pci_uio.c
+++ b/drivers/bus/pci/linux/pci_uio.c
@@ -19,7 +19,6 @@
 #include <rte_string_fns.h>
 #include <rte_log.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
 #include <rte_common.h>
 #include <rte_malloc.h>
 
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index a024269140..2ccd0fa101 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -13,7 +13,6 @@
 
 #include <rte_log.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
 #include <rte_eal_paging.h>
 #include <rte_malloc.h>
 #include <rte_vfio.h>
diff --git a/drivers/bus/pci/meson.build b/drivers/bus/pci/meson.build
index 81c7e94c00..33c09e2622 100644
--- a/drivers/bus/pci/meson.build
+++ b/drivers/bus/pci/meson.build
@@ -29,4 +29,8 @@ if is_windows
     includes += include_directories('windows')
 endif
 
+driver_sdk_headers += files(
+        'pci_driver.h',
+)
+
 deps += ['kvargs']
diff --git a/drivers/bus/pci/pci_common_uio.c b/drivers/bus/pci/pci_common_uio.c
index 318f9a1d55..00ef86c1dd 100644
--- a/drivers/bus/pci/pci_common_uio.c
+++ b/drivers/bus/pci/pci_common_uio.c
@@ -11,7 +11,6 @@
 
 #include <rte_eal.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
 #include <rte_tailq.h>
 #include <rte_log.h>
 #include <rte_malloc.h>
diff --git a/drivers/bus/pci/pci_driver.h b/drivers/bus/pci/pci_driver.h
new file mode 100644
index 0000000000..7a913d54c5
--- /dev/null
+++ b/drivers/bus/pci/pci_driver.h
@@ -0,0 +1,402 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2015 Intel Corporation.
+ * Copyright 2013-2014 6WIND S.A.
+ */
+
+#ifndef _PCI_DRIVER_H_
+#define _PCI_DRIVER_H_
+
+/**
+ * @file
+ * PCI device & driver interface
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <limits.h>
+#include <errno.h>
+#include <sys/queue.h>
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_debug.h>
+#include <rte_interrupts.h>
+#include <rte_dev.h>
+#include <rte_bus.h>
+#include <rte_pci.h>
+
+/** Pathname of PCI devices directory. */
+__rte_internal
+const char *rte_pci_get_sysfs_path(void);
+
+/* Forward declarations */
+struct rte_pci_device;
+struct rte_pci_driver;
+
+/** List of PCI devices */
+TAILQ_HEAD(rte_pci_device_list, rte_pci_device);
+/** List of PCI drivers */
+TAILQ_HEAD(rte_pci_driver_list, rte_pci_driver);
+
+/* PCI Bus iterators */
+#define FOREACH_DEVICE_ON_PCIBUS(p)	\
+		TAILQ_FOREACH(p, &(rte_pci_bus.device_list), next)
+
+#define FOREACH_DRIVER_ON_PCIBUS(p)	\
+		TAILQ_FOREACH(p, &(rte_pci_bus.driver_list), next)
+
+struct rte_devargs;
+
+enum rte_pci_kernel_driver {
+	RTE_PCI_KDRV_UNKNOWN = 0,  /* may be misc UIO or bifurcated driver */
+	RTE_PCI_KDRV_IGB_UIO,      /* igb_uio for Linux */
+	RTE_PCI_KDRV_VFIO,         /* VFIO for Linux */
+	RTE_PCI_KDRV_UIO_GENERIC,  /* uio_pci_generic for Linux */
+	RTE_PCI_KDRV_NIC_UIO,      /* nic_uio for FreeBSD */
+	RTE_PCI_KDRV_NONE,         /* no attached driver */
+	RTE_PCI_KDRV_NET_UIO,      /* NetUIO for Windows */
+};
+
+/**
+ * A structure describing a PCI device.
+ */
+struct rte_pci_device {
+	TAILQ_ENTRY(rte_pci_device) next;   /**< Next probed PCI device. */
+	struct rte_device device;           /**< Inherit core device */
+	struct rte_pci_addr addr;           /**< PCI location. */
+	struct rte_pci_id id;               /**< PCI ID. */
+	struct rte_mem_resource mem_resource[PCI_MAX_RESOURCE];
+					    /**< PCI Memory Resource */
+	struct rte_intr_handle intr_handle; /**< Interrupt handle */
+	struct rte_pci_driver *driver;      /**< PCI driver used in probing */
+	uint16_t max_vfs;                   /**< sriov enable if not zero */
+	enum rte_pci_kernel_driver kdrv;    /**< Kernel driver passthrough */
+	char name[PCI_PRI_STR_SIZE+1];      /**< PCI location (ASCII) */
+	struct rte_intr_handle vfio_req_intr_handle;
+				/**< Handler of VFIO request interrupt */
+};
+
+/**
+ * @internal
+ * Helper macro for drivers that need to convert to struct rte_pci_device.
+ */
+#define RTE_DEV_TO_PCI(ptr) container_of(ptr, struct rte_pci_device, device)
+
+#define RTE_DEV_TO_PCI_CONST(ptr) \
+	container_of(ptr, const struct rte_pci_device, device)
+
+#define RTE_ETH_DEV_TO_PCI(eth_dev)	RTE_DEV_TO_PCI((eth_dev)->device)
+
+#ifdef __cplusplus
+/** C++ macro used to help building up tables of device IDs */
+#define RTE_PCI_DEVICE(vend, dev) \
+	RTE_CLASS_ANY_ID,         \
+	(vend),                   \
+	(dev),                    \
+	RTE_PCI_ANY_ID,           \
+	RTE_PCI_ANY_ID
+#else
+/** Macro used to help building up tables of device IDs */
+#define RTE_PCI_DEVICE(vend, dev)          \
+	.class_id = RTE_CLASS_ANY_ID,      \
+	.vendor_id = (vend),               \
+	.device_id = (dev),                \
+	.subsystem_vendor_id = RTE_PCI_ANY_ID, \
+	.subsystem_device_id = RTE_PCI_ANY_ID
+#endif
+
+/**
+ * Initialisation function for the driver called during PCI probing.
+ */
+typedef int (rte_pci_probe_t)(struct rte_pci_driver *, struct rte_pci_device *);
+
+/**
+ * Uninitialisation function for the driver called during hotplugging.
+ */
+typedef int (rte_pci_remove_t)(struct rte_pci_device *);
+
+/**
+ * Driver-specific DMA mapping. After a successful call the device
+ * will be able to read/write from/to this segment.
+ *
+ * @param dev
+ *   Pointer to the PCI device.
+ * @param addr
+ *   Starting virtual address of memory to be mapped.
+ * @param iova
+ *   Starting IOVA address of memory to be mapped.
+ * @param len
+ *   Length of memory segment being mapped.
+ * @return
+ *   - 0 On success.
+ *   - Negative value and rte_errno is set otherwise.
+ */
+typedef int (pci_dma_map_t)(struct rte_pci_device *dev, void *addr,
+			    uint64_t iova, size_t len);
+
+/**
+ * Driver-specific DMA un-mapping. After a successful call the device
+ * will not be able to read/write from/to this segment.
+ *
+ * @param dev
+ *   Pointer to the PCI device.
+ * @param addr
+ *   Starting virtual address of memory to be unmapped.
+ * @param iova
+ *   Starting IOVA address of memory to be unmapped.
+ * @param len
+ *   Length of memory segment being unmapped.
+ * @return
+ *   - 0 On success.
+ *   - Negative value and rte_errno is set otherwise.
+ */
+typedef int (pci_dma_unmap_t)(struct rte_pci_device *dev, void *addr,
+			      uint64_t iova, size_t len);
+
+/**
+ * A structure describing a PCI driver.
+ */
+struct rte_pci_driver {
+	TAILQ_ENTRY(rte_pci_driver) next;  /**< Next in list. */
+	struct rte_driver driver;          /**< Inherit core driver. */
+	struct rte_pci_bus *bus;           /**< PCI bus reference. */
+	rte_pci_probe_t *probe;            /**< Device probe function. */
+	rte_pci_remove_t *remove;          /**< Device remove function. */
+	pci_dma_map_t *dma_map;		   /**< device dma map function. */
+	pci_dma_unmap_t *dma_unmap;	   /**< device dma unmap function. */
+	const struct rte_pci_id *id_table; /**< ID table, NULL terminated. */
+	uint32_t drv_flags;                /**< Flags RTE_PCI_DRV_*. */
+};
+
+/**
+ * Structure describing the PCI bus
+ */
+struct rte_pci_bus {
+	struct rte_bus bus;               /**< Inherit the generic class */
+	struct rte_pci_device_list device_list;  /**< List of PCI devices */
+	struct rte_pci_driver_list driver_list;  /**< List of PCI drivers */
+};
+
+/** Device needs PCI BAR mapping (done with either IGB_UIO or VFIO) */
+#define RTE_PCI_DRV_NEED_MAPPING 0x0001
+/** Device needs PCI BAR mapping with enabled write combining (wc) */
+#define RTE_PCI_DRV_WC_ACTIVATE 0x0002
+/** Device already probed can be probed again to check for new ports. */
+#define RTE_PCI_DRV_PROBE_AGAIN 0x0004
+/** Device driver supports link state interrupt */
+#define RTE_PCI_DRV_INTR_LSC	0x0008
+/** Device driver supports device removal interrupt */
+#define RTE_PCI_DRV_INTR_RMV 0x0010
+/** Device driver needs to keep mapped resources if unsupported dev detected */
+#define RTE_PCI_DRV_KEEP_MAPPED_RES 0x0020
+/** Device driver needs IOVA as VA and cannot work with IOVA as PA */
+#define RTE_PCI_DRV_NEED_IOVA_AS_VA 0x0040
+
+/**
+ * Map the PCI device resources in user space virtual memory address
+ *
+ * Note that driver should not call this function when flag
+ * RTE_PCI_DRV_NEED_MAPPING is set, as EAL will do that for
+ * you when it's on.
+ *
+ * @param dev
+ *   A pointer to a rte_pci_device structure describing the device
+ *   to use
+ *
+ * @return
+ *   0 on success, negative on error and positive if no driver
+ *   is found for the device.
+ */
+__rte_internal
+int rte_pci_map_device(struct rte_pci_device *dev);
+
+/**
+ * Unmap this device
+ *
+ * @param dev
+ *   A pointer to a rte_pci_device structure describing the device
+ *   to use
+ */
+__rte_internal
+void rte_pci_unmap_device(struct rte_pci_device *dev);
+
+/**
+ * Find device's extended PCI capability.
+ *
+ *  @param dev
+ *    A pointer to rte_pci_device structure.
+ *
+ *  @param cap
+ *    Extended capability to be found, which can be any from
+ *    RTE_PCI_EXT_CAP_ID_*, defined in librte_pci.
+ *
+ *  @return
+ *  > 0: The offset of the next matching extended capability structure
+ *       within the device's PCI configuration space.
+ *  < 0: An error in PCI config space read.
+ *  = 0: Device does not support it.
+ */
+__rte_internal
+off_t rte_pci_find_ext_capability(struct rte_pci_device *dev, uint32_t cap);
+
+/**
+ * Enables/Disables Bus Master for device's PCI command register.
+ *
+ *  @param dev
+ *    A pointer to rte_pci_device structure.
+ *  @param enable
+ *    Enable or disable Bus Master.
+ *
+ *  @return
+ *  0 on success, -1 on error in PCI config space read/write.
+ */
+__rte_internal
+int rte_pci_set_bus_master(struct rte_pci_device *dev, bool enable);
+
+/**
+ * Register a PCI driver.
+ *
+ * @param driver
+ *   A pointer to a rte_pci_driver structure describing the driver
+ *   to be registered.
+ */
+__rte_internal
+void rte_pci_register(struct rte_pci_driver *driver);
+
+/** Helper for PCI device registration from driver (eth, crypto) instance */
+#define RTE_PMD_REGISTER_PCI(nm, pci_drv) \
+RTE_INIT(pciinitfn_ ##nm) \
+{\
+	(pci_drv).driver.name = RTE_STR(nm);\
+	rte_pci_register(&pci_drv); \
+} \
+RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+/**
+ * Unregister a PCI driver.
+ *
+ * @param driver
+ *   A pointer to a rte_pci_driver structure describing the driver
+ *   to be unregistered.
+ */
+__rte_internal
+void rte_pci_unregister(struct rte_pci_driver *driver);
+
+/**
+ * Read PCI config space.
+ *
+ * @param device
+ *   A pointer to a rte_pci_device structure describing the device
+ *   to use
+ * @param buf
+ *   A data buffer where the bytes should be read into
+ * @param len
+ *   The length of the data buffer.
+ * @param offset
+ *   The offset into PCI config space
+ * @return
+ *  Number of bytes read on success, negative on error.
+ */
+__rte_internal
+int rte_pci_read_config(const struct rte_pci_device *device,
+		void *buf, size_t len, off_t offset);
+
+/**
+ * Write PCI config space.
+ *
+ * @param device
+ *   A pointer to a rte_pci_device structure describing the device
+ *   to use
+ * @param buf
+ *   A data buffer containing the bytes should be written
+ * @param len
+ *   The length of the data buffer.
+ * @param offset
+ *   The offset into PCI config space
+ */
+__rte_internal
+int rte_pci_write_config(const struct rte_pci_device *device,
+		const void *buf, size_t len, off_t offset);
+
+/**
+ * A structure used to access io resources for a pci device.
+ * rte_pci_ioport is arch, os, driver specific, and should not be used outside
+ * of pci ioport api.
+ */
+struct rte_pci_ioport {
+	struct rte_pci_device *dev;
+	uint64_t base;
+	uint64_t len; /* only filled for memory mapped ports */
+};
+
+/**
+ * Initialize a rte_pci_ioport object for a pci device io resource.
+ *
+ * This object is then used to gain access to those io resources (see below).
+ *
+ * @param dev
+ *   A pointer to a rte_pci_device structure describing the device
+ *   to use.
+ * @param bar
+ *   Index of the io pci resource we want to access.
+ * @param p
+ *   The rte_pci_ioport object to be initialized.
+ * @return
+ *  0 on success, negative on error.
+ */
+__rte_internal
+int rte_pci_ioport_map(struct rte_pci_device *dev, int bar,
+		struct rte_pci_ioport *p);
+
+/**
+ * Release any resources used in a rte_pci_ioport object.
+ *
+ * @param p
+ *   The rte_pci_ioport object to be uninitialized.
+ * @return
+ *  0 on success, negative on error.
+ */
+__rte_internal
+int rte_pci_ioport_unmap(struct rte_pci_ioport *p);
+
+/**
+ * Read from a io pci resource.
+ *
+ * @param p
+ *   The rte_pci_ioport object from which we want to read.
+ * @param data
+ *   A data buffer where the bytes should be read into
+ * @param len
+ *   The length of the data buffer.
+ * @param offset
+ *   The offset into the pci io resource.
+ */
+__rte_internal
+void rte_pci_ioport_read(struct rte_pci_ioport *p,
+		void *data, size_t len, off_t offset);
+
+/**
+ * Write to a io pci resource.
+ *
+ * @param p
+ *   The rte_pci_ioport object to which we want to write.
+ * @param data
+ *   A data buffer where the bytes should be read into
+ * @param len
+ *   The length of the data buffer.
+ * @param offset
+ *   The offset into the pci io resource.
+ */
+__rte_internal
+void rte_pci_ioport_write(struct rte_pci_ioport *p,
+		const void *data, size_t len, off_t offset);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _PCI_DRIVER_H_ */
diff --git a/drivers/bus/pci/pci_params.c b/drivers/bus/pci/pci_params.c
index 691b5ea018..3f4f94cc72 100644
--- a/drivers/bus/pci/pci_params.c
+++ b/drivers/bus/pci/pci_params.c
@@ -3,7 +3,6 @@
  */
 
 #include <rte_bus.h>
-#include <rte_bus_pci.h>
 #include <rte_dev.h>
 #include <rte_errno.h>
 #include <rte_kvargs.h>
diff --git a/drivers/bus/pci/private.h b/drivers/bus/pci/private.h
index 0fbef8e1d8..dd42baf516 100644
--- a/drivers/bus/pci/private.h
+++ b/drivers/bus/pci/private.h
@@ -8,10 +8,11 @@
 #include <stdbool.h>
 #include <stdio.h>
 
-#include <rte_bus_pci.h>
 #include <rte_os_shim.h>
 #include <rte_pci.h>
 
+#include "pci_driver.h"
+
 extern struct rte_pci_bus rte_pci_bus;
 
 struct rte_pci_driver;
diff --git a/drivers/bus/pci/rte_bus_pci.h b/drivers/bus/pci/rte_bus_pci.h
index 21d9dd4289..2fb35801f8 100644
--- a/drivers/bus/pci/rte_bus_pci.h
+++ b/drivers/bus/pci/rte_bus_pci.h
@@ -1,225 +1,17 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2015 Intel Corporation.
- * Copyright 2013-2014 6WIND S.A.
+ * Copyright(c) 2021 Intel Corporation.
  */
 
 #ifndef _RTE_BUS_PCI_H_
 #define _RTE_BUS_PCI_H_
 
-/**
- * @file
- * PCI device & driver interface
- */
-
 #ifdef __cplusplus
 extern "C" {
 #endif
 
-#include <stdio.h>
-#include <stdlib.h>
-#include <limits.h>
-#include <errno.h>
-#include <sys/queue.h>
 #include <stdint.h>
-#include <inttypes.h>
-
-#include <rte_debug.h>
-#include <rte_interrupts.h>
-#include <rte_dev.h>
-#include <rte_bus.h>
-#include <rte_pci.h>
-
-/** Pathname of PCI devices directory. */
-const char *rte_pci_get_sysfs_path(void);
-
-/* Forward declarations */
-struct rte_pci_device;
-struct rte_pci_driver;
-
-/** List of PCI devices */
-TAILQ_HEAD(rte_pci_device_list, rte_pci_device);
-/** List of PCI drivers */
-TAILQ_HEAD(rte_pci_driver_list, rte_pci_driver);
-
-/* PCI Bus iterators */
-#define FOREACH_DEVICE_ON_PCIBUS(p)	\
-		TAILQ_FOREACH(p, &(rte_pci_bus.device_list), next)
 
-#define FOREACH_DRIVER_ON_PCIBUS(p)	\
-		TAILQ_FOREACH(p, &(rte_pci_bus.driver_list), next)
-
-struct rte_devargs;
-
-enum rte_pci_kernel_driver {
-	RTE_PCI_KDRV_UNKNOWN = 0,  /* may be misc UIO or bifurcated driver */
-	RTE_PCI_KDRV_IGB_UIO,      /* igb_uio for Linux */
-	RTE_PCI_KDRV_VFIO,         /* VFIO for Linux */
-	RTE_PCI_KDRV_UIO_GENERIC,  /* uio_pci_generic for Linux */
-	RTE_PCI_KDRV_NIC_UIO,      /* nic_uio for FreeBSD */
-	RTE_PCI_KDRV_NONE,         /* no attached driver */
-	RTE_PCI_KDRV_NET_UIO,      /* NetUIO for Windows */
-};
-
-/**
- * A structure describing a PCI device.
- */
-struct rte_pci_device {
-	TAILQ_ENTRY(rte_pci_device) next;   /**< Next probed PCI device. */
-	struct rte_device device;           /**< Inherit core device */
-	struct rte_pci_addr addr;           /**< PCI location. */
-	struct rte_pci_id id;               /**< PCI ID. */
-	struct rte_mem_resource mem_resource[PCI_MAX_RESOURCE];
-					    /**< PCI Memory Resource */
-	struct rte_intr_handle intr_handle; /**< Interrupt handle */
-	struct rte_pci_driver *driver;      /**< PCI driver used in probing */
-	uint16_t max_vfs;                   /**< sriov enable if not zero */
-	enum rte_pci_kernel_driver kdrv;    /**< Kernel driver passthrough */
-	char name[PCI_PRI_STR_SIZE+1];      /**< PCI location (ASCII) */
-	struct rte_intr_handle vfio_req_intr_handle;
-				/**< Handler of VFIO request interrupt */
-};
-
-/**
- * @internal
- * Helper macro for drivers that need to convert to struct rte_pci_device.
- */
-#define RTE_DEV_TO_PCI(ptr) container_of(ptr, struct rte_pci_device, device)
-
-#define RTE_DEV_TO_PCI_CONST(ptr) \
-	container_of(ptr, const struct rte_pci_device, device)
-
-#define RTE_ETH_DEV_TO_PCI(eth_dev)	RTE_DEV_TO_PCI((eth_dev)->device)
-
-#ifdef __cplusplus
-/** C++ macro used to help building up tables of device IDs */
-#define RTE_PCI_DEVICE(vend, dev) \
-	RTE_CLASS_ANY_ID,         \
-	(vend),                   \
-	(dev),                    \
-	RTE_PCI_ANY_ID,           \
-	RTE_PCI_ANY_ID
-#else
-/** Macro used to help building up tables of device IDs */
-#define RTE_PCI_DEVICE(vend, dev)          \
-	.class_id = RTE_CLASS_ANY_ID,      \
-	.vendor_id = (vend),               \
-	.device_id = (dev),                \
-	.subsystem_vendor_id = RTE_PCI_ANY_ID, \
-	.subsystem_device_id = RTE_PCI_ANY_ID
-#endif
-
-/**
- * Initialisation function for the driver called during PCI probing.
- */
-typedef int (rte_pci_probe_t)(struct rte_pci_driver *, struct rte_pci_device *);
-
-/**
- * Uninitialisation function for the driver called during hotplugging.
- */
-typedef int (rte_pci_remove_t)(struct rte_pci_device *);
-
-/**
- * Driver-specific DMA mapping. After a successful call the device
- * will be able to read/write from/to this segment.
- *
- * @param dev
- *   Pointer to the PCI device.
- * @param addr
- *   Starting virtual address of memory to be mapped.
- * @param iova
- *   Starting IOVA address of memory to be mapped.
- * @param len
- *   Length of memory segment being mapped.
- * @return
- *   - 0 On success.
- *   - Negative value and rte_errno is set otherwise.
- */
-typedef int (pci_dma_map_t)(struct rte_pci_device *dev, void *addr,
-			    uint64_t iova, size_t len);
-
-/**
- * Driver-specific DMA un-mapping. After a successful call the device
- * will not be able to read/write from/to this segment.
- *
- * @param dev
- *   Pointer to the PCI device.
- * @param addr
- *   Starting virtual address of memory to be unmapped.
- * @param iova
- *   Starting IOVA address of memory to be unmapped.
- * @param len
- *   Length of memory segment being unmapped.
- * @return
- *   - 0 On success.
- *   - Negative value and rte_errno is set otherwise.
- */
-typedef int (pci_dma_unmap_t)(struct rte_pci_device *dev, void *addr,
-			      uint64_t iova, size_t len);
-
-/**
- * A structure describing a PCI driver.
- */
-struct rte_pci_driver {
-	TAILQ_ENTRY(rte_pci_driver) next;  /**< Next in list. */
-	struct rte_driver driver;          /**< Inherit core driver. */
-	struct rte_pci_bus *bus;           /**< PCI bus reference. */
-	rte_pci_probe_t *probe;            /**< Device probe function. */
-	rte_pci_remove_t *remove;          /**< Device remove function. */
-	pci_dma_map_t *dma_map;		   /**< device dma map function. */
-	pci_dma_unmap_t *dma_unmap;	   /**< device dma unmap function. */
-	const struct rte_pci_id *id_table; /**< ID table, NULL terminated. */
-	uint32_t drv_flags;                /**< Flags RTE_PCI_DRV_*. */
-};
-
-/**
- * Structure describing the PCI bus
- */
-struct rte_pci_bus {
-	struct rte_bus bus;               /**< Inherit the generic class */
-	struct rte_pci_device_list device_list;  /**< List of PCI devices */
-	struct rte_pci_driver_list driver_list;  /**< List of PCI drivers */
-};
-
-/** Device needs PCI BAR mapping (done with either IGB_UIO or VFIO) */
-#define RTE_PCI_DRV_NEED_MAPPING 0x0001
-/** Device needs PCI BAR mapping with enabled write combining (wc) */
-#define RTE_PCI_DRV_WC_ACTIVATE 0x0002
-/** Device already probed can be probed again to check for new ports. */
-#define RTE_PCI_DRV_PROBE_AGAIN 0x0004
-/** Device driver supports link state interrupt */
-#define RTE_PCI_DRV_INTR_LSC	0x0008
-/** Device driver supports device removal interrupt */
-#define RTE_PCI_DRV_INTR_RMV 0x0010
-/** Device driver needs to keep mapped resources if unsupported dev detected */
-#define RTE_PCI_DRV_KEEP_MAPPED_RES 0x0020
-/** Device driver needs IOVA as VA and cannot work with IOVA as PA */
-#define RTE_PCI_DRV_NEED_IOVA_AS_VA 0x0040
-
-/**
- * Map the PCI device resources in user space virtual memory address
- *
- * Note that driver should not call this function when flag
- * RTE_PCI_DRV_NEED_MAPPING is set, as EAL will do that for
- * you when it's on.
- *
- * @param dev
- *   A pointer to a rte_pci_device structure describing the device
- *   to use
- *
- * @return
- *   0 on success, negative on error and positive if no driver
- *   is found for the device.
- */
-int rte_pci_map_device(struct rte_pci_device *dev);
-
-/**
- * Unmap this device
- *
- * @param dev
- *   A pointer to a rte_pci_device structure describing the device
- *   to use
- */
-void rte_pci_unmap_device(struct rte_pci_device *dev);
+#include <rte_compat.h>
 
 /**
  * Dump the content of the PCI bus.
@@ -229,169 +21,6 @@ void rte_pci_unmap_device(struct rte_pci_device *dev);
  */
 void rte_pci_dump(FILE *f);
 
-/**
- * Find device's extended PCI capability.
- *
- *  @param dev
- *    A pointer to rte_pci_device structure.
- *
- *  @param cap
- *    Extended capability to be found, which can be any from
- *    RTE_PCI_EXT_CAP_ID_*, defined in librte_pci.
- *
- *  @return
- *  > 0: The offset of the next matching extended capability structure
- *       within the device's PCI configuration space.
- *  < 0: An error in PCI config space read.
- *  = 0: Device does not support it.
- */
-__rte_experimental
-off_t rte_pci_find_ext_capability(struct rte_pci_device *dev, uint32_t cap);
-
-/**
- * Enables/Disables Bus Master for device's PCI command register.
- *
- *  @param dev
- *    A pointer to rte_pci_device structure.
- *  @param enable
- *    Enable or disable Bus Master.
- *
- *  @return
- *  0 on success, -1 on error in PCI config space read/write.
- */
-__rte_experimental
-int rte_pci_set_bus_master(struct rte_pci_device *dev, bool enable);
-
-/**
- * Register a PCI driver.
- *
- * @param driver
- *   A pointer to a rte_pci_driver structure describing the driver
- *   to be registered.
- */
-void rte_pci_register(struct rte_pci_driver *driver);
-
-/** Helper for PCI device registration from driver (eth, crypto) instance */
-#define RTE_PMD_REGISTER_PCI(nm, pci_drv) \
-RTE_INIT(pciinitfn_ ##nm) \
-{\
-	(pci_drv).driver.name = RTE_STR(nm);\
-	rte_pci_register(&pci_drv); \
-} \
-RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
-
-/**
- * Unregister a PCI driver.
- *
- * @param driver
- *   A pointer to a rte_pci_driver structure describing the driver
- *   to be unregistered.
- */
-void rte_pci_unregister(struct rte_pci_driver *driver);
-
-/**
- * Read PCI config space.
- *
- * @param device
- *   A pointer to a rte_pci_device structure describing the device
- *   to use
- * @param buf
- *   A data buffer where the bytes should be read into
- * @param len
- *   The length of the data buffer.
- * @param offset
- *   The offset into PCI config space
- * @return
- *  Number of bytes read on success, negative on error.
- */
-int rte_pci_read_config(const struct rte_pci_device *device,
-		void *buf, size_t len, off_t offset);
-
-/**
- * Write PCI config space.
- *
- * @param device
- *   A pointer to a rte_pci_device structure describing the device
- *   to use
- * @param buf
- *   A data buffer containing the bytes should be written
- * @param len
- *   The length of the data buffer.
- * @param offset
- *   The offset into PCI config space
- */
-int rte_pci_write_config(const struct rte_pci_device *device,
-		const void *buf, size_t len, off_t offset);
-
-/**
- * A structure used to access io resources for a pci device.
- * rte_pci_ioport is arch, os, driver specific, and should not be used outside
- * of pci ioport api.
- */
-struct rte_pci_ioport {
-	struct rte_pci_device *dev;
-	uint64_t base;
-	uint64_t len; /* only filled for memory mapped ports */
-};
-
-/**
- * Initialize a rte_pci_ioport object for a pci device io resource.
- *
- * This object is then used to gain access to those io resources (see below).
- *
- * @param dev
- *   A pointer to a rte_pci_device structure describing the device
- *   to use.
- * @param bar
- *   Index of the io pci resource we want to access.
- * @param p
- *   The rte_pci_ioport object to be initialized.
- * @return
- *  0 on success, negative on error.
- */
-int rte_pci_ioport_map(struct rte_pci_device *dev, int bar,
-		struct rte_pci_ioport *p);
-
-/**
- * Release any resources used in a rte_pci_ioport object.
- *
- * @param p
- *   The rte_pci_ioport object to be uninitialized.
- * @return
- *  0 on success, negative on error.
- */
-int rte_pci_ioport_unmap(struct rte_pci_ioport *p);
-
-/**
- * Read from a io pci resource.
- *
- * @param p
- *   The rte_pci_ioport object from which we want to read.
- * @param data
- *   A data buffer where the bytes should be read into
- * @param len
- *   The length of the data buffer.
- * @param offset
- *   The offset into the pci io resource.
- */
-void rte_pci_ioport_read(struct rte_pci_ioport *p,
-		void *data, size_t len, off_t offset);
-
-/**
- * Write to a io pci resource.
- *
- * @param p
- *   The rte_pci_ioport object to which we want to write.
- * @param data
- *   A data buffer where the bytes should be read into
- * @param len
- *   The length of the data buffer.
- * @param offset
- *   The offset into the pci io resource.
- */
-void rte_pci_ioport_write(struct rte_pci_ioport *p,
-		const void *data, size_t len, off_t offset);
-
 /**
  * Read 4 bytes from PCI memory resource.
  *
diff --git a/drivers/bus/pci/version.map b/drivers/bus/pci/version.map
index 01ec836559..73cf881778 100644
--- a/drivers/bus/pci/version.map
+++ b/drivers/bus/pci/version.map
@@ -2,6 +2,22 @@ DPDK_22 {
 	global:
 
 	rte_pci_dump;
+
+	local: *;
+};
+
+EXPERIMENTAL {
+	global:
+
+	# added in 21.11
+	rte_pci_mem_rd32;
+	rte_pci_mem_wr32;
+};
+
+INTERNAL {
+	global:
+
+	rte_pci_find_ext_capability;
 	rte_pci_get_sysfs_path;
 	rte_pci_ioport_map;
 	rte_pci_ioport_read;
@@ -10,22 +26,8 @@ DPDK_22 {
 	rte_pci_map_device;
 	rte_pci_read_config;
 	rte_pci_register;
+	rte_pci_set_bus_master;
 	rte_pci_unmap_device;
 	rte_pci_unregister;
 	rte_pci_write_config;
-
-	local: *;
-};
-
-EXPERIMENTAL {
-	global:
-
-	rte_pci_find_ext_capability;
-
-	# added in 21.08
-	rte_pci_set_bus_master;
-
-	# added in 21.11
-	rte_pci_mem_rd32;
-	rte_pci_mem_wr32;
 };
diff --git a/drivers/common/cnxk/roc_platform.h b/drivers/common/cnxk/roc_platform.h
index 285b24b82d..fa8a6ef0af 100644
--- a/drivers/common/cnxk/roc_platform.h
+++ b/drivers/common/cnxk/roc_platform.h
@@ -7,7 +7,7 @@
 
 #include <rte_alarm.h>
 #include <rte_bitmap.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_byteorder.h>
 #include <rte_common.h>
 #include <rte_cycles.h>
diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c
index 9080bd3e87..65d795643b 100644
--- a/drivers/common/mlx5/linux/mlx5_common_verbs.c
+++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c
@@ -11,7 +11,7 @@
 #include <inttypes.h>
 
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_bus_auxiliary.h>
 
 #include "mlx5_common_utils.h"
diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c
index 8b38091d87..eaa9a0fff5 100644
--- a/drivers/common/mlx5/mlx5_common_pci.c
+++ b/drivers/common/mlx5/mlx5_common_pci.c
@@ -9,7 +9,7 @@
 #include <rte_errno.h>
 #include <rte_class.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "mlx5_common_log.h"
 #include "mlx5_common_private.h"
diff --git a/drivers/common/octeontx2/otx2_dev.h b/drivers/common/octeontx2/otx2_dev.h
index d5b2b0d9af..1f36f24cb0 100644
--- a/drivers/common/octeontx2/otx2_dev.h
+++ b/drivers/common/octeontx2/otx2_dev.h
@@ -5,7 +5,7 @@
 #ifndef _OTX2_DEV_H
 #define _OTX2_DEV_H
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "otx2_common.h"
 #include "otx2_irq.h"
diff --git a/drivers/common/octeontx2/otx2_sec_idev.c b/drivers/common/octeontx2/otx2_sec_idev.c
index 6e9643c383..a73c27c970 100644
--- a/drivers/common/octeontx2/otx2_sec_idev.c
+++ b/drivers/common/octeontx2/otx2_sec_idev.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_atomic.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ethdev.h>
 #include <rte_spinlock.h>
 
diff --git a/drivers/common/qat/qat_device.h b/drivers/common/qat/qat_device.h
index 228c057d1e..8d18365bae 100644
--- a/drivers/common/qat/qat_device.h
+++ b/drivers/common/qat/qat_device.h
@@ -4,7 +4,7 @@
 #ifndef _QAT_DEVICE_H_
 #define _QAT_DEVICE_H_
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "qat_common.h"
 #include "qat_logs.h"
diff --git a/drivers/common/qat/qat_qp.c b/drivers/common/qat/qat_qp.c
index 026ea5ee01..a964231d2c 100644
--- a/drivers/common/qat/qat_qp.c
+++ b/drivers/common/qat/qat_qp.c
@@ -8,7 +8,7 @@
 #include <rte_malloc.h>
 #include <rte_memzone.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_atomic.h>
 #include <rte_prefetch.h>
 
diff --git a/drivers/common/sfc_efx/sfc_efx.h b/drivers/common/sfc_efx/sfc_efx.h
index c16eca60f3..9829ed0e96 100644
--- a/drivers/common/sfc_efx/sfc_efx.h
+++ b/drivers/common/sfc_efx/sfc_efx.h
@@ -10,7 +10,7 @@
 #ifndef _SFC_EFX_H_
 #define _SFC_EFX_H_
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "efx.h"
 #include "efsys.h"
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 883e720ec1..e8112935c3 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -5,7 +5,7 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_spinlock.h>
 #include <rte_comp.h>
 #include <rte_compressdev.h>
diff --git a/drivers/compress/octeontx/otx_zip.h b/drivers/compress/octeontx/otx_zip.h
index e43f7f5c3e..56136ce734 100644
--- a/drivers/compress/octeontx/otx_zip.h
+++ b/drivers/compress/octeontx/otx_zip.h
@@ -7,7 +7,7 @@
 
 #include <unistd.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_comp.h>
 #include <rte_compressdev.h>
 #include <rte_compressdev_pmd.h>
diff --git a/drivers/compress/qat/qat_comp.c b/drivers/compress/qat/qat_comp.c
index 7ac25a3b4c..6fa82ad9a8 100644
--- a/drivers/compress/qat/qat_comp.c
+++ b/drivers/compress/qat/qat_comp.c
@@ -6,7 +6,7 @@
 #include <rte_mbuf.h>
 #include <rte_hexdump.h>
 #include <rte_comp.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_byteorder.h>
 #include <rte_memcpy.h>
 #include <rte_common.h>
diff --git a/drivers/crypto/ccp/ccp_dev.h b/drivers/crypto/ccp/ccp_dev.h
index ca5145c278..eb8cd94716 100644
--- a/drivers/crypto/ccp/ccp_dev.h
+++ b/drivers/crypto/ccp/ccp_dev.h
@@ -10,7 +10,7 @@
 #include <stdint.h>
 #include <string.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_atomic.h>
 #include <rte_byteorder.h>
 #include <rte_io.h>
diff --git a/drivers/crypto/ccp/ccp_pci.h b/drivers/crypto/ccp/ccp_pci.h
index 7ed3bac406..c8ba9f8789 100644
--- a/drivers/crypto/ccp/ccp_pci.h
+++ b/drivers/crypto/ccp/ccp_pci.h
@@ -7,7 +7,7 @@
 
 #include <stdint.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #define SYSFS_PCI_DEVICES "/sys/bus/pci/devices"
 #define PROC_MODULES "/proc/modules"
diff --git a/drivers/crypto/ccp/rte_ccp_pmd.c b/drivers/crypto/ccp/rte_ccp_pmd.c
index ab9416942e..8f5351511e 100644
--- a/drivers/crypto/ccp/rte_ccp_pmd.c
+++ b/drivers/crypto/ccp/rte_ccp_pmd.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_string_fns.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_bus_vdev.h>
 #include <rte_common.h>
 #include <rte_cryptodev.h>
diff --git a/drivers/crypto/cnxk/cn10k_cryptodev.c b/drivers/crypto/cnxk/cn10k_cryptodev.c
index db7b5aa7c6..ed96f98e51 100644
--- a/drivers/crypto/cnxk/cn10k_cryptodev.c
+++ b/drivers/crypto/cnxk/cn10k_cryptodev.c
@@ -2,7 +2,7 @@
  * Copyright(C) 2021 Marvell.
  */
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_crypto.h>
 #include <rte_cryptodev.h>
diff --git a/drivers/crypto/cnxk/cn9k_cryptodev.c b/drivers/crypto/cnxk/cn9k_cryptodev.c
index 9ff2383d98..017febefa9 100644
--- a/drivers/crypto/cnxk/cn9k_cryptodev.c
+++ b/drivers/crypto/cnxk/cn9k_cryptodev.c
@@ -2,7 +2,7 @@
  * Copyright(C) 2021 Marvell.
  */
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_crypto.h>
 #include <rte_cryptodev.h>
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index b3d5200ca3..6518ed0c19 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -6,7 +6,7 @@
 #include <rte_mempool.h>
 #include <rte_errno.h>
 #include <rte_log.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_memory.h>
 
 #include <mlx5_glue.h>
diff --git a/drivers/crypto/nitrox/nitrox_device.h b/drivers/crypto/nitrox/nitrox_device.h
index 6b8095f42b..91b15d9913 100644
--- a/drivers/crypto/nitrox/nitrox_device.h
+++ b/drivers/crypto/nitrox/nitrox_device.h
@@ -5,7 +5,7 @@
 #ifndef _NITROX_DEVICE_H_
 #define _NITROX_DEVICE_H_
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_cryptodev.h>
 
 struct nitrox_sym_device;
diff --git a/drivers/crypto/octeontx/otx_cryptodev.c b/drivers/crypto/octeontx/otx_cryptodev.c
index 3822c0d779..0b86757fc1 100644
--- a/drivers/crypto/octeontx/otx_cryptodev.c
+++ b/drivers/crypto/octeontx/otx_cryptodev.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2018 Cavium, Inc
  */
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_cryptodev.h>
 #include <rte_cryptodev_pmd.h>
diff --git a/drivers/crypto/octeontx/otx_cryptodev_ops.c b/drivers/crypto/octeontx/otx_cryptodev_ops.c
index eac6796cfb..212388da96 100644
--- a/drivers/crypto/octeontx/otx_cryptodev_ops.c
+++ b/drivers/crypto/octeontx/otx_cryptodev_ops.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_alarm.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_cryptodev.h>
 #include <rte_cryptodev_pmd.h>
 #include <rte_eventdev.h>
diff --git a/drivers/crypto/octeontx2/otx2_cryptodev.c b/drivers/crypto/octeontx2/otx2_cryptodev.c
index 75fb4f9a3b..d0451e25b4 100644
--- a/drivers/crypto/octeontx2/otx2_cryptodev.c
+++ b/drivers/crypto/octeontx2/otx2_cryptodev.c
@@ -2,7 +2,7 @@
  * Copyright (C) 2019 Marvell International Ltd.
  */
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_crypto.h>
 #include <rte_cryptodev.h>
diff --git a/drivers/crypto/qat/qat_sym.c b/drivers/crypto/qat/qat_sym.c
index 93b257522b..574e3d577d 100644
--- a/drivers/crypto/qat/qat_sym.c
+++ b/drivers/crypto/qat/qat_sym.c
@@ -7,7 +7,7 @@
 #include <rte_mempool.h>
 #include <rte_mbuf.h>
 #include <rte_crypto_sym.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_byteorder.h>
 
 #include "qat_sym.h"
diff --git a/drivers/crypto/qat/qat_sym_pmd.c b/drivers/crypto/qat/qat_sym_pmd.c
index 6868e5f001..76c58346d1 100644
--- a/drivers/crypto/qat/qat_sym_pmd.c
+++ b/drivers/crypto/qat/qat_sym_pmd.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2015-2018 Intel Corporation
  */
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_dev.h>
 #include <rte_malloc.h>
diff --git a/drivers/crypto/virtio/virtio_cryptodev.c b/drivers/crypto/virtio/virtio_cryptodev.c
index 4bae74a487..706fa59629 100644
--- a/drivers/crypto/virtio/virtio_cryptodev.c
+++ b/drivers/crypto/virtio/virtio_cryptodev.c
@@ -7,7 +7,7 @@
 #include <rte_common.h>
 #include <rte_errno.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_cryptodev.h>
 #include <rte_cryptodev_pmd.h>
 #include <rte_eal.h>
diff --git a/drivers/crypto/virtio/virtio_pci.h b/drivers/crypto/virtio/virtio_pci.h
index 0a7ea1bb64..889c263555 100644
--- a/drivers/crypto/virtio/virtio_pci.h
+++ b/drivers/crypto/virtio/virtio_pci.h
@@ -9,7 +9,7 @@
 
 #include <rte_eal_paging.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_cryptodev.h>
 
 #include "virtio_crypto.h"
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
index 9eeda482a3..40178e0dda 100644
--- a/drivers/event/dlb2/pf/dlb2_main.h
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -9,7 +9,7 @@
 #include <rte_log.h>
 #include <rte_spinlock.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_eal_paging.h>
 
 #include "base/dlb2_hw_types.h"
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index e9da89d650..8f8c742286 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -25,7 +25,7 @@
 #include <rte_cycles.h>
 #include <rte_io.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_eventdev.h>
 #include <eventdev_pmd.h>
 #include <eventdev_pmd_pci.h>
diff --git a/drivers/event/octeontx/ssovf_probe.c b/drivers/event/octeontx/ssovf_probe.c
index 4da7d1ae45..f45f005e33 100644
--- a/drivers/event/octeontx/ssovf_probe.c
+++ b/drivers/event/octeontx/ssovf_probe.c
@@ -7,7 +7,7 @@
 #include <rte_eal.h>
 #include <rte_io.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "octeontx_mbox.h"
 #include "ssovf_evdev.h"
diff --git a/drivers/event/octeontx/timvf_probe.c b/drivers/event/octeontx/timvf_probe.c
index 59bba31e8e..5a75494b12 100644
--- a/drivers/event/octeontx/timvf_probe.c
+++ b/drivers/event/octeontx/timvf_probe.c
@@ -5,7 +5,7 @@
 #include <rte_eal.h>
 #include <rte_io.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include <octeontx_mbox.h>
 
diff --git a/drivers/event/octeontx2/otx2_evdev.c b/drivers/event/octeontx2/otx2_evdev.c
index 38a6b651d9..5db1880455 100644
--- a/drivers/event/octeontx2/otx2_evdev.c
+++ b/drivers/event/octeontx2/otx2_evdev.c
@@ -4,7 +4,7 @@
 
 #include <inttypes.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_eal.h>
 #include <eventdev_pmd_pci.h>
diff --git a/drivers/mempool/cnxk/cnxk_mempool.c b/drivers/mempool/cnxk/cnxk_mempool.c
index dd4d74ca05..175bad355f 100644
--- a/drivers/mempool/cnxk/cnxk_mempool.c
+++ b/drivers/mempool/cnxk/cnxk_mempool.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_atomic.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_devargs.h>
 #include <rte_eal.h>
diff --git a/drivers/mempool/octeontx/octeontx_fpavf.c b/drivers/mempool/octeontx/octeontx_fpavf.c
index 94dc5cd815..a7a09f7872 100644
--- a/drivers/mempool/octeontx/octeontx_fpavf.c
+++ b/drivers/mempool/octeontx/octeontx_fpavf.c
@@ -13,7 +13,7 @@
 
 #include <rte_atomic.h>
 #include <rte_eal.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_errno.h>
 #include <rte_memory.h>
 #include <rte_malloc.h>
diff --git a/drivers/mempool/octeontx2/otx2_mempool.c b/drivers/mempool/octeontx2/otx2_mempool.c
index fb630fecf8..6c3969fcfb 100644
--- a/drivers/mempool/octeontx2/otx2_mempool.c
+++ b/drivers/mempool/octeontx2/otx2_mempool.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_atomic.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_eal.h>
 #include <rte_io.h>
diff --git a/drivers/mempool/octeontx2/otx2_mempool.h b/drivers/mempool/octeontx2/otx2_mempool.h
index 8aa548248d..b2ed9a56e8 100644
--- a/drivers/mempool/octeontx2/otx2_mempool.h
+++ b/drivers/mempool/octeontx2/otx2_mempool.h
@@ -6,7 +6,7 @@
 #define __OTX2_MEMPOOL_H__
 
 #include <rte_bitmap.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_devargs.h>
 #include <rte_mempool.h>
 
diff --git a/drivers/mempool/octeontx2/otx2_mempool_irq.c b/drivers/mempool/octeontx2/otx2_mempool_irq.c
index 5fa22b9612..c7c9b0d35d 100644
--- a/drivers/mempool/octeontx2/otx2_mempool_irq.c
+++ b/drivers/mempool/octeontx2/otx2_mempool_irq.c
@@ -5,7 +5,7 @@
 #include <inttypes.h>
 
 #include <rte_common.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "otx2_common.h"
 #include "otx2_irq.h"
diff --git a/drivers/meson.build b/drivers/meson.build
index d9e331ec85..44fe109892 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -85,6 +85,7 @@ foreach subpath:subdirs
         name = drv
         sources = []
         headers = []
+        driver_sdk_headers = [] # public headers included by drivers
         objs = []
         cflags = default_cflags
         includes = [include_directories(drv_path)]
@@ -150,6 +151,9 @@ foreach subpath:subdirs
         dpdk_extra_ldflags += pkgconfig_extra_libs
 
         install_headers(headers)
+        if get_option('enable_driver_sdk')
+            install_headers(driver_sdk_headers)
+        endif
 
         # generate pmdinfo sources by building a temporary
         # lib and then running pmdinfogen on the contents of
diff --git a/drivers/net/ark/ark_ethdev.c b/drivers/net/ark/ark_ethdev.c
index 377299b14c..4c564ce21e 100644
--- a/drivers/net/ark/ark_ethdev.c
+++ b/drivers/net/ark/ark_ethdev.c
@@ -6,7 +6,7 @@
 #include <sys/stat.h>
 #include <dlfcn.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <ethdev_pci.h>
 #include <rte_kvargs.h>
 
diff --git a/drivers/net/avp/avp_ethdev.c b/drivers/net/avp/avp_ethdev.c
index 623fa5e5ff..87e29519e6 100644
--- a/drivers/net/avp/avp_ethdev.c
+++ b/drivers/net/avp/avp_ethdev.c
@@ -16,7 +16,7 @@
 #include <rte_atomic.h>
 #include <rte_branch_prediction.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ether.h>
 #include <rte_common.h>
 #include <rte_cycles.h>
diff --git a/drivers/net/bnx2x/bnx2x.h b/drivers/net/bnx2x/bnx2x.h
index 80d19cbfd6..425acaa5ac 100644
--- a/drivers/net/bnx2x/bnx2x.h
+++ b/drivers/net/bnx2x/bnx2x.h
@@ -16,7 +16,7 @@
 
 #include <rte_byteorder.h>
 #include <rte_spinlock.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_io.h>
 
 #include "bnx2x_osal.h"
diff --git a/drivers/net/bnxt/bnxt.h b/drivers/net/bnxt/bnxt.h
index 494a1eff37..6a456e20e2 100644
--- a/drivers/net/bnxt/bnxt.h
+++ b/drivers/net/bnxt/bnxt.h
@@ -11,7 +11,7 @@
 #include <sys/queue.h>
 
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <ethdev_driver.h>
 #include <rte_memory.h>
 #include <rte_lcore.h>
diff --git a/drivers/net/bonding/rte_eth_bond_args.c b/drivers/net/bonding/rte_eth_bond_args.c
index 5406e1c934..24ff649ca7 100644
--- a/drivers/net/bonding/rte_eth_bond_args.c
+++ b/drivers/net/bonding/rte_eth_bond_args.c
@@ -4,7 +4,7 @@
 
 #include <rte_devargs.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_kvargs.h>
 
 #include "rte_eth_bond.h"
diff --git a/drivers/net/cxgbe/base/adapter.h b/drivers/net/cxgbe/base/adapter.h
index 01a2a9d147..80809d7cd6 100644
--- a/drivers/net/cxgbe/base/adapter.h
+++ b/drivers/net/cxgbe/base/adapter.h
@@ -8,7 +8,7 @@
 #ifndef __T4_ADAPTER_H__
 #define __T4_ADAPTER_H__
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_mbuf.h>
 #include <rte_io.h>
 #include <rte_rwlock.h>
diff --git a/drivers/net/cxgbe/cxgbe_ethdev.c b/drivers/net/cxgbe/cxgbe_ethdev.c
index 177eca3976..fb03d69d66 100644
--- a/drivers/net/cxgbe/cxgbe_ethdev.c
+++ b/drivers/net/cxgbe/cxgbe_ethdev.c
@@ -20,7 +20,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_branch_prediction.h>
 #include <rte_memory.h>
 #include <rte_tailq.h>
diff --git a/drivers/net/e1000/em_ethdev.c b/drivers/net/e1000/em_ethdev.c
index a0ca371b02..657de61aa4 100644
--- a/drivers/net/e1000/em_ethdev.c
+++ b/drivers/net/e1000/em_ethdev.c
@@ -13,7 +13,7 @@
 #include <rte_byteorder.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ether.h>
 #include <ethdev_driver.h>
 #include <ethdev_pci.h>
diff --git a/drivers/net/e1000/em_rxtx.c b/drivers/net/e1000/em_rxtx.c
index dfd8f2fd00..c75be0d1e5 100644
--- a/drivers/net/e1000/em_rxtx.c
+++ b/drivers/net/e1000/em_rxtx.c
@@ -18,7 +18,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_memory.h>
 #include <rte_memcpy.h>
 #include <rte_memzone.h>
diff --git a/drivers/net/e1000/igb_ethdev.c b/drivers/net/e1000/igb_ethdev.c
index 10ee0f3341..94181da21a 100644
--- a/drivers/net/e1000/igb_ethdev.c
+++ b/drivers/net/e1000/igb_ethdev.c
@@ -15,7 +15,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ether.h>
 #include <ethdev_driver.h>
 #include <ethdev_pci.h>
diff --git a/drivers/net/e1000/igb_pf.c b/drivers/net/e1000/igb_pf.c
index 2ce74dd5a9..294fb75dc3 100644
--- a/drivers/net/e1000/igb_pf.c
+++ b/drivers/net/e1000/igb_pf.c
@@ -10,7 +10,7 @@
 #include <stdarg.h>
 #include <inttypes.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_interrupts.h>
 #include <rte_log.h>
 #include <rte_debug.h>
diff --git a/drivers/net/ena/ena_ethdev.h b/drivers/net/ena/ena_ethdev.h
index 06ac8b06b5..a3d76383e0 100644
--- a/drivers/net/ena/ena_ethdev.h
+++ b/drivers/net/ena/ena_ethdev.h
@@ -12,7 +12,7 @@
 #include <ethdev_pci.h>
 #include <rte_cycles.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_timer.h>
 #include <rte_dev.h>
 #include <rte_net.h>
diff --git a/drivers/net/enic/base/vnic_dev.h b/drivers/net/enic/base/vnic_dev.h
index 4b9f75b65f..db9b2af3cd 100644
--- a/drivers/net/enic/base/vnic_dev.h
+++ b/drivers/net/enic/base/vnic_dev.h
@@ -9,7 +9,7 @@
 #include <stdbool.h>
 
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "enic_compat.h"
 #include "vnic_resource.h"
diff --git a/drivers/net/enic/enic_ethdev.c b/drivers/net/enic/enic_ethdev.c
index 8d5797523b..1202fcd4f9 100644
--- a/drivers/net/enic/enic_ethdev.c
+++ b/drivers/net/enic/enic_ethdev.c
@@ -8,7 +8,7 @@
 
 #include <rte_dev.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <ethdev_driver.h>
 #include <ethdev_pci.h>
 #include <rte_geneve.h>
diff --git a/drivers/net/enic/enic_main.c b/drivers/net/enic/enic_main.c
index 2affd380c6..f5c1131839 100644
--- a/drivers/net/enic/enic_main.c
+++ b/drivers/net/enic/enic_main.c
@@ -10,7 +10,7 @@
 #include <fcntl.h>
 
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_memzone.h>
 #include <rte_malloc.h>
 #include <rte_mbuf.h>
diff --git a/drivers/net/enic/enic_vf_representor.c b/drivers/net/enic/enic_vf_representor.c
index 79dd6e5640..f44f6c013e 100644
--- a/drivers/net/enic/enic_vf_representor.c
+++ b/drivers/net/enic/enic_vf_representor.c
@@ -5,7 +5,7 @@
 #include <stdint.h>
 #include <stdio.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_dev.h>
 #include <ethdev_driver.h>
diff --git a/drivers/net/hinic/base/hinic_pmd_hwdev.c b/drivers/net/hinic/base/hinic_pmd_hwdev.c
index cb9cf6efa2..49a6b523f5 100644
--- a/drivers/net/hinic/base/hinic_pmd_hwdev.c
+++ b/drivers/net/hinic/base/hinic_pmd_hwdev.c
@@ -3,7 +3,7 @@
  */
 
 #include<ethdev_driver.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_hash.h>
 #include <rte_jhash.h>
 
diff --git a/drivers/net/hinic/base/hinic_pmd_hwif.c b/drivers/net/hinic/base/hinic_pmd_hwif.c
index 26fa1e27d4..9b1b41ca95 100644
--- a/drivers/net/hinic/base/hinic_pmd_hwif.c
+++ b/drivers/net/hinic/base/hinic_pmd_hwif.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2017 Huawei Technologies Co., Ltd
  */
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "hinic_compat.h"
 #include "hinic_csr.h"
diff --git a/drivers/net/hinic/base/hinic_pmd_nicio.c b/drivers/net/hinic/base/hinic_pmd_nicio.c
index ad5db9f1de..234b433c90 100644
--- a/drivers/net/hinic/base/hinic_pmd_nicio.c
+++ b/drivers/net/hinic/base/hinic_pmd_nicio.c
@@ -1,7 +1,7 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2017 Huawei Technologies Co., Ltd
  */
-#include<rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "hinic_compat.h"
 #include "hinic_pmd_hwdev.h"
diff --git a/drivers/net/hinic/hinic_pmd_ethdev.c b/drivers/net/hinic/hinic_pmd_ethdev.c
index 1a72401546..d36dad0579 100644
--- a/drivers/net/hinic/hinic_pmd_ethdev.c
+++ b/drivers/net/hinic/hinic_pmd_ethdev.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <ethdev_pci.h>
 #include <rte_mbuf.h>
 #include <rte_malloc.h>
diff --git a/drivers/net/hns3/hns3_ethdev.c b/drivers/net/hns3/hns3_ethdev.c
index 7d37004972..d5d5bc8e0f 100644
--- a/drivers/net/hns3/hns3_ethdev.c
+++ b/drivers/net/hns3/hns3_ethdev.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_alarm.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <ethdev_pci.h>
 #include <rte_pci.h>
 #include <rte_kvargs.h>
diff --git a/drivers/net/hns3/hns3_rxtx.c b/drivers/net/hns3/hns3_rxtx.c
index 0f222b37f9..54cd15d40d 100644
--- a/drivers/net/hns3/hns3_rxtx.c
+++ b/drivers/net/hns3/hns3_rxtx.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2018-2021 HiSilicon Limited.
  */
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_cycles.h>
 #include <rte_geneve.h>
diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 7b230e2ed1..51fface85e 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -15,7 +15,7 @@
 #include <rte_eal.h>
 #include <rte_string_fns.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ether.h>
 #include <ethdev_driver.h>
 #include <ethdev_pci.h>
diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index 0cfe13b7b2..356ca6de0c 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -18,7 +18,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_atomic.h>
 #include <rte_branch_prediction.h>
 #include <rte_memory.h>
diff --git a/drivers/net/i40e/i40e_vf_representor.c b/drivers/net/i40e/i40e_vf_representor.c
index 0481b55381..650d8ad502 100644
--- a/drivers/net/i40e/i40e_vf_representor.c
+++ b/drivers/net/i40e/i40e_vf_representor.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2018 Intel Corporation.
  */
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ethdev.h>
 #include <rte_pci.h>
 #include <rte_malloc.h>
diff --git a/drivers/net/igc/igc_ethdev.c b/drivers/net/igc/igc_ethdev.c
index 224a095483..3d3d0d6f3c 100644
--- a/drivers/net/igc/igc_ethdev.c
+++ b/drivers/net/igc/igc_ethdev.c
@@ -7,7 +7,7 @@
 
 #include <rte_string_fns.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <ethdev_driver.h>
 #include <ethdev_pci.h>
 #include <rte_malloc.h>
diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index 49b90d1b7c..e946d92c29 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -8,7 +8,7 @@
 #include <stdint.h>
 #include <inttypes.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "ionic_dev.h"
 #include "ionic_if.h"
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index e620793966..e8bcccccb4 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ethdev.h>
 #include <ethdev_driver.h>
 #include <rte_malloc.h>
diff --git a/drivers/net/ipn3ke/ipn3ke_ethdev.c b/drivers/net/ipn3ke/ipn3ke_ethdev.c
index 964506c6db..2493df2bcd 100644
--- a/drivers/net/ipn3ke/ipn3ke_ethdev.c
+++ b/drivers/net/ipn3ke/ipn3ke_ethdev.c
@@ -4,7 +4,7 @@
 
 #include <stdint.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ethdev.h>
 #include <rte_pci.h>
 #include <rte_malloc.h>
diff --git a/drivers/net/ipn3ke/ipn3ke_representor.c b/drivers/net/ipn3ke/ipn3ke_representor.c
index 589d9fa587..f288bf09d8 100644
--- a/drivers/net/ipn3ke/ipn3ke_representor.c
+++ b/drivers/net/ipn3ke/ipn3ke_representor.c
@@ -5,7 +5,7 @@
 #include <stdint.h>
 #include <unistd.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ethdev.h>
 #include <rte_pci.h>
 #include <rte_malloc.h>
diff --git a/drivers/net/ipn3ke/ipn3ke_tm.c b/drivers/net/ipn3ke/ipn3ke_tm.c
index 6a9b98fd7f..532d232dbf 100644
--- a/drivers/net/ipn3ke/ipn3ke_tm.c
+++ b/drivers/net/ipn3ke/ipn3ke_tm.c
@@ -6,7 +6,7 @@
 #include <stdlib.h>
 #include <string.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ethdev.h>
 #include <rte_pci.h>
 #include <rte_malloc.h>
diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index b5371568b5..f873757a7b 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -20,7 +20,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_branch_prediction.h>
 #include <rte_memory.h>
 #include <rte_kvargs.h>
diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
index a0ce18ca24..7b0c1ad542 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.h
+++ b/drivers/net/ixgbe/ixgbe_ethdev.h
@@ -19,7 +19,7 @@
 #include <rte_time.h>
 #include <rte_hash.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_tm_driver.h>
 
 /* need update link, bit flag */
diff --git a/drivers/net/mlx4/mlx4_ethdev.c b/drivers/net/mlx4/mlx4_ethdev.c
index 783ff94dce..b86f65e32a 100644
--- a/drivers/net/mlx4/mlx4_ethdev.c
+++ b/drivers/net/mlx4/mlx4_ethdev.c
@@ -32,7 +32,7 @@
 #pragma GCC diagnostic error "-Wpedantic"
 #endif
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_errno.h>
 #include <ethdev_driver.h>
 #include <rte_ether.h>
diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
index f34133e2c6..2e0a40f0cc 100644
--- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
+++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
@@ -25,7 +25,7 @@
 #include <time.h>
 
 #include <ethdev_driver.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_mbuf.h>
 #include <rte_common.h>
 #include <rte_interrupts.h>
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 5f8766aa48..ba184470e1 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -19,7 +19,7 @@
 #include <ethdev_driver.h>
 #include <ethdev_pci.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_bus_auxiliary.h>
 #include <rte_common.h>
 #include <rte_kvargs.h>
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index f84e061fe7..bbcdec0f49 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -13,7 +13,7 @@
 #include <rte_malloc.h>
 #include <ethdev_driver.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_kvargs.h>
 #include <rte_rwlock.h>
diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c
index 82e2284d98..8888702fc8 100644
--- a/drivers/net/mlx5/mlx5_ethdev.c
+++ b/drivers/net/mlx5/mlx5_ethdev.c
@@ -11,7 +11,7 @@
 #include <errno.h>
 
 #include <ethdev_driver.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_mbuf.h>
 #include <rte_common.h>
 #include <rte_interrupts.h>
diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c
index eb4d34ca55..4d8c26f7bc 100644
--- a/drivers/net/mlx5/mlx5_txq.c
+++ b/drivers/net/mlx5/mlx5_txq.c
@@ -13,7 +13,7 @@
 #include <rte_mbuf.h>
 #include <rte_malloc.h>
 #include <ethdev_driver.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_eal_paging.h>
 
diff --git a/drivers/net/netvsc/hn_vf.c b/drivers/net/netvsc/hn_vf.c
index 75192e6319..1da4eacb6f 100644
--- a/drivers/net/netvsc/hn_vf.c
+++ b/drivers/net/netvsc/hn_vf.c
@@ -21,7 +21,7 @@
 #include <rte_memory.h>
 #include <rte_bus_vmbus.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_log.h>
 #include <rte_string_fns.h>
 #include <rte_alarm.h>
diff --git a/drivers/net/octeontx/base/octeontx_pkivf.c b/drivers/net/octeontx/base/octeontx_pkivf.c
index 0ddff54886..9ff5b78fd0 100644
--- a/drivers/net/octeontx/base/octeontx_pkivf.c
+++ b/drivers/net/octeontx/base/octeontx_pkivf.c
@@ -5,7 +5,7 @@
 #include <string.h>
 
 #include <rte_eal.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "../octeontx_logs.h"
 #include "octeontx_io.h"
diff --git a/drivers/net/octeontx/base/octeontx_pkovf.c b/drivers/net/octeontx/base/octeontx_pkovf.c
index bf28bc7992..a125acadce 100644
--- a/drivers/net/octeontx/base/octeontx_pkovf.c
+++ b/drivers/net/octeontx/base/octeontx_pkovf.c
@@ -10,7 +10,7 @@
 #include <rte_cycles.h>
 #include <rte_malloc.h>
 #include <rte_memory.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_spinlock.h>
 
 #include "../octeontx_logs.h"
diff --git a/drivers/net/octeontx2/otx2_ethdev_irq.c b/drivers/net/octeontx2/otx2_ethdev_irq.c
index b121488faf..40541c9280 100644
--- a/drivers/net/octeontx2/otx2_ethdev_irq.c
+++ b/drivers/net/octeontx2/otx2_ethdev_irq.c
@@ -4,7 +4,7 @@
 
 #include <inttypes.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_malloc.h>
 
 #include "otx2_ethdev.h"
diff --git a/drivers/net/qede/base/bcm_osal.h b/drivers/net/qede/base/bcm_osal.h
index c5b5399282..10ec1e2fb7 100644
--- a/drivers/net/qede/base/bcm_osal.h
+++ b/drivers/net/qede/base/bcm_osal.h
@@ -21,7 +21,7 @@
 #include <rte_ether.h>
 #include <rte_io.h>
 #include <rte_version.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 /* Forward declaration */
 struct ecore_dev;
diff --git a/drivers/net/sfc/sfc.h b/drivers/net/sfc/sfc.h
index 331e06bac6..f57da8fc96 100644
--- a/drivers/net/sfc/sfc.h
+++ b/drivers/net/sfc/sfc.h
@@ -13,7 +13,7 @@
 #include <stdbool.h>
 
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <ethdev_driver.h>
 #include <rte_kvargs.h>
 #include <rte_spinlock.h>
diff --git a/drivers/net/sfc/sfc_ethdev.c b/drivers/net/sfc/sfc_ethdev.c
index 2db0d000c3..7f8b052845 100644
--- a/drivers/net/sfc/sfc_ethdev.c
+++ b/drivers/net/sfc/sfc_ethdev.c
@@ -11,7 +11,7 @@
 #include <ethdev_driver.h>
 #include <ethdev_pci.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_errno.h>
 #include <rte_string_fns.h>
 #include <rte_ether.h>
diff --git a/drivers/net/sfc/sfc_sriov.c b/drivers/net/sfc/sfc_sriov.c
index baa0242433..48c64c8311 100644
--- a/drivers/net/sfc/sfc_sriov.c
+++ b/drivers/net/sfc/sfc_sriov.c
@@ -8,7 +8,7 @@
  */
 
 #include <rte_common.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "sfc.h"
 #include "sfc_log.h"
diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index fc1844ddfc..cf87a3861f 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -32,7 +32,7 @@
 #include <rte_malloc.h>
 #include <rte_random.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_tailq.h>
 #include <rte_devargs.h>
 #include <rte_kvargs.h>
diff --git a/drivers/net/txgbe/txgbe_ethdev.h b/drivers/net/txgbe/txgbe_ethdev.h
index 3021933965..faf0c58bd5 100644
--- a/drivers/net/txgbe/txgbe_ethdev.h
+++ b/drivers/net/txgbe/txgbe_ethdev.h
@@ -20,7 +20,7 @@
 #include <rte_ethdev_core.h>
 #include <rte_hash.h>
 #include <rte_hash_crc.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_tm_driver.h>
 
 /* need update link, bit flag */
diff --git a/drivers/net/txgbe/txgbe_flow.c b/drivers/net/txgbe/txgbe_flow.c
index eae400b141..f70a9309d9 100644
--- a/drivers/net/txgbe/txgbe_flow.c
+++ b/drivers/net/txgbe/txgbe_flow.c
@@ -4,7 +4,7 @@
  */
 
 #include <sys/queue.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_malloc.h>
 #include <rte_flow.h>
 #include <rte_flow_driver.h>
diff --git a/drivers/net/txgbe/txgbe_pf.c b/drivers/net/txgbe/txgbe_pf.c
index 494d779a3c..58159bc226 100644
--- a/drivers/net/txgbe/txgbe_pf.c
+++ b/drivers/net/txgbe/txgbe_pf.c
@@ -20,7 +20,7 @@
 #include <rte_memcpy.h>
 #include <rte_malloc.h>
 #include <rte_random.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "base/txgbe.h"
 #include "txgbe_ethdev.h"
diff --git a/drivers/net/virtio/virtio_pci.h b/drivers/net/virtio/virtio_pci.h
index 11e25a0142..a9016534aa 100644
--- a/drivers/net/virtio/virtio_pci.h
+++ b/drivers/net/virtio/virtio_pci.h
@@ -9,7 +9,7 @@
 #include <stdbool.h>
 
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <ethdev_driver.h>
 
 #include "virtio.h"
diff --git a/drivers/net/virtio/virtio_pci_ethdev.c b/drivers/net/virtio/virtio_pci_ethdev.c
index 4083853c48..13dbc0c419 100644
--- a/drivers/net/virtio/virtio_pci_ethdev.c
+++ b/drivers/net/virtio/virtio_pci_ethdev.c
@@ -11,7 +11,7 @@
 #include <ethdev_driver.h>
 #include <ethdev_pci.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_errno.h>
 
 #include <rte_memory.h>
diff --git a/drivers/net/vmxnet3/vmxnet3_ethdev.c b/drivers/net/vmxnet3/vmxnet3_ethdev.c
index 1a3291273a..ccbab9364e 100644
--- a/drivers/net/vmxnet3/vmxnet3_ethdev.c
+++ b/drivers/net/vmxnet3/vmxnet3_ethdev.c
@@ -19,7 +19,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_branch_prediction.h>
 #include <rte_memory.h>
 #include <rte_memzone.h>
diff --git a/drivers/raw/cnxk_bphy/cnxk_bphy.c b/drivers/raw/cnxk_bphy/cnxk_bphy.c
index 9cb3f8d332..61c60ed363 100644
--- a/drivers/raw/cnxk_bphy/cnxk_bphy.c
+++ b/drivers/raw/cnxk_bphy/cnxk_bphy.c
@@ -1,7 +1,7 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(C) 2021 Marvell.
  */
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_dev.h>
 #include <rte_eal.h>
diff --git a/drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c b/drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c
index ade45ab741..f9d7353a18 100644
--- a/drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c
+++ b/drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c
@@ -3,7 +3,7 @@
  */
 #include <string.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_rawdev.h>
 #include <rte_rawdev_pmd.h>
 
diff --git a/drivers/raw/cnxk_bphy/cnxk_bphy_irq.c b/drivers/raw/cnxk_bphy/cnxk_bphy_irq.c
index bbcc285a7a..6014992934 100644
--- a/drivers/raw/cnxk_bphy/cnxk_bphy_irq.c
+++ b/drivers/raw/cnxk_bphy/cnxk_bphy_irq.c
@@ -1,7 +1,7 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(C) 2021 Marvell.
  */
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_pci.h>
 #include <rte_rawdev.h>
 #include <rte_rawdev_pmd.h>
diff --git a/drivers/raw/cnxk_bphy/cnxk_bphy_irq.h b/drivers/raw/cnxk_bphy/cnxk_bphy_irq.h
index b55147b93e..7a623b48c8 100644
--- a/drivers/raw/cnxk_bphy/cnxk_bphy_irq.h
+++ b/drivers/raw/cnxk_bphy/cnxk_bphy_irq.h
@@ -5,7 +5,7 @@
 #ifndef _CNXK_BPHY_IRQ_
 #define _CNXK_BPHY_IRQ_
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_dev.h>
 
 #include <roc_api.h>
diff --git a/drivers/raw/ifpga/ifpga_rawdev.c b/drivers/raw/ifpga/ifpga_rawdev.c
index 76e6a8530b..eb0c6e271f 100644
--- a/drivers/raw/ifpga/ifpga_rawdev.c
+++ b/drivers/raw/ifpga/ifpga_rawdev.c
@@ -16,7 +16,7 @@
 #include <rte_devargs.h>
 #include <rte_memcpy.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_kvargs.h>
 #include <rte_alarm.h>
 #include <rte_interrupts.h>
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.c b/drivers/raw/ifpga/rte_pmd_ifpga.c
index 23146432c2..959aad414d 100644
--- a/drivers/raw/ifpga/rte_pmd_ifpga.c
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_rawdev.h>
 #include <rte_rawdev_pmd.h>
 #include "rte_pmd_ifpga.h"
diff --git a/drivers/raw/ioat/idxd_pci.c b/drivers/raw/ioat/idxd_pci.c
index 13515dbc6c..257e4c004a 100644
--- a/drivers/raw/ioat/idxd_pci.c
+++ b/drivers/raw/ioat/idxd_pci.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2020 Intel Corporation
  */
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_memzone.h>
 #include <rte_devargs.h>
 
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index 5396671d4f..f1d4220b69 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_cycles.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_memzone.h>
 #include <rte_string_fns.h>
 #include <rte_rawdev_pmd.h>
diff --git a/drivers/raw/ntb/ntb.c b/drivers/raw/ntb/ntb.c
index 78cfcd79f7..a74a42f8d2 100644
--- a/drivers/raw/ntb/ntb.c
+++ b/drivers/raw/ntb/ntb.c
@@ -13,7 +13,7 @@
 #include <rte_log.h>
 #include <rte_pci.h>
 #include <rte_mbuf.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_memzone.h>
 #include <rte_memcpy.h>
 #include <rte_rawdev.h>
diff --git a/drivers/raw/ntb/ntb_hw_intel.c b/drivers/raw/ntb/ntb_hw_intel.c
index a742e8fbb9..072c2dd7b9 100644
--- a/drivers/raw/ntb/ntb_hw_intel.c
+++ b/drivers/raw/ntb/ntb_hw_intel.c
@@ -8,7 +8,7 @@
 #include <rte_io.h>
 #include <rte_eal.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_rawdev.h>
 #include <rte_rawdev_pmd.h>
 
diff --git a/drivers/raw/octeontx2_dma/otx2_dpi_rawdev.c b/drivers/raw/octeontx2_dma/otx2_dpi_rawdev.c
index 8c01f25ec7..179e3e6dbb 100644
--- a/drivers/raw/octeontx2_dma/otx2_dpi_rawdev.c
+++ b/drivers/raw/octeontx2_dma/otx2_dpi_rawdev.c
@@ -6,7 +6,7 @@
 #include <unistd.h>
 
 #include <rte_bus.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_eal.h>
 #include <rte_lcore.h>
diff --git a/drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c b/drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c
index d04e957d82..214fd83169 100644
--- a/drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c
+++ b/drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c
@@ -8,7 +8,7 @@
 #include <fcntl.h>
 
 #include <rte_bus.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_eal.h>
 #include <rte_lcore.h>
 #include <rte_mempool.h>
diff --git a/drivers/raw/octeontx2_ep/otx2_ep_rawdev.c b/drivers/raw/octeontx2_ep/otx2_ep_rawdev.c
index b2ccdda83e..560a8bbf03 100644
--- a/drivers/raw/octeontx2_ep/otx2_ep_rawdev.c
+++ b/drivers/raw/octeontx2_ep/otx2_ep_rawdev.c
@@ -5,7 +5,7 @@
 #include <unistd.h>
 
 #include <rte_bus.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_eal.h>
 #include <rte_lcore.h>
 #include <rte_mempool.h>
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index f17b6df47f..0926aec5bc 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -9,7 +9,7 @@
 #include <rte_regexdev.h>
 #include <rte_regexdev_core.h>
 #include <rte_regexdev_driver.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include <mlx5_common.h>
 #include <mlx5_common_mr.h>
diff --git a/drivers/regex/mlx5/mlx5_regex_fastpath.c b/drivers/regex/mlx5/mlx5_regex_fastpath.c
index 786718af53..7693171106 100644
--- a/drivers/regex/mlx5/mlx5_regex_fastpath.c
+++ b/drivers/regex/mlx5/mlx5_regex_fastpath.c
@@ -10,7 +10,7 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_pci.h>
 #include <rte_regexdev_driver.h>
 #include <rte_mbuf.h>
diff --git a/drivers/vdpa/ifc/base/ifcvf_osdep.h b/drivers/vdpa/ifc/base/ifcvf_osdep.h
index 6aef25ea45..3f3eb46f3f 100644
--- a/drivers/vdpa/ifc/base/ifcvf_osdep.h
+++ b/drivers/vdpa/ifc/base/ifcvf_osdep.h
@@ -10,7 +10,7 @@
 
 #include <rte_cycles.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_log.h>
 #include <rte_io.h>
 
diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c
index 1dc813d0a3..968dea5055 100644
--- a/drivers/vdpa/ifc/ifcvf_vdpa.c
+++ b/drivers/vdpa/ifc/ifcvf_vdpa.c
@@ -14,7 +14,7 @@
 #include <rte_eal_paging.h>
 #include <rte_malloc.h>
 #include <rte_memory.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_vhost.h>
 #include <rte_vdpa.h>
 #include <rte_vdpa_dev.h>
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 6d17d7a6f3..81c7e738f5 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -12,7 +12,7 @@
 #include <rte_log.h>
 #include <rte_errno.h>
 #include <rte_string_fns.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
diff --git a/lib/ethdev/ethdev_pci.h b/lib/ethdev/ethdev_pci.h
index 8edca82ce8..76779faa6c 100644
--- a/lib/ethdev/ethdev_pci.h
+++ b/lib/ethdev/ethdev_pci.h
@@ -8,7 +8,7 @@
 
 #include <rte_malloc.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_config.h>
 #include <ethdev_driver.h>
 
diff --git a/lib/eventdev/eventdev_pmd_pci.h b/lib/eventdev/eventdev_pmd_pci.h
index d14ea634b8..260362aacb 100644
--- a/lib/eventdev/eventdev_pmd_pci.h
+++ b/lib/eventdev/eventdev_pmd_pci.h
@@ -24,7 +24,7 @@ extern "C" {
 #include <rte_eal.h>
 #include <rte_lcore.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "eventdev_pmd.h"
 
-- 
2.17.1

Re: [dpdk-dev] [PATCH 8/8] bus/pci: remove ABIs in PCI bus

[Kinsella, Ray]

On 10/09/2021 03:24, Chenbo Xia wrote:
> As announced in the deprecation note, most of ABIs in PCI bus are
> removed in this patch. Only the function rte_pci_dump is still ABI
> and experimental APIs are kept for future promotion.
> 
> This patch creates a new file named pci_driver.h and moves most of
> the content in original rte_bus_pci.h to it. After that, pci_driver.h
> is considered the interface for drivers and rte_bus_pci.h for
> applications. pci_driver.h is defined as driver_sdk_headers so that
> out-of-tree drivers can use it.
> 
> Then this patch replaces the including of rte_bus_pci.h with pci_driver.h
> in all related drivers.
> 
> Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
> ---
>  app/test/virtual_pmd.c                        |   2 +-
>  doc/guides/rel_notes/release_21_11.rst        |   2 +
>  drivers/baseband/acc100/rte_acc100_pmd.c      |   2 +-
>  .../fpga_5gnr_fec/rte_fpga_5gnr_fec.c         |   2 +-
>  drivers/baseband/fpga_lte_fec/fpga_lte_fec.c  |   2 +-
>  drivers/bus/pci/bsd/pci.c                     |   1 -
>  drivers/bus/pci/linux/pci.c                   |   1 -
>  drivers/bus/pci/linux/pci_uio.c               |   1 -
>  drivers/bus/pci/linux/pci_vfio.c              |   1 -
>  drivers/bus/pci/meson.build                   |   4 +
>  drivers/bus/pci/pci_common_uio.c              |   1 -
>  drivers/bus/pci/pci_driver.h                  | 402 ++++++++++++++++++
>  drivers/bus/pci/pci_params.c                  |   1 -
>  drivers/bus/pci/private.h                     |   3 +-
>  drivers/bus/pci/rte_bus_pci.h                 | 375 +---------------
>  drivers/bus/pci/version.map                   |  32 +-
>  drivers/common/cnxk/roc_platform.h            |   2 +-
>  drivers/common/mlx5/linux/mlx5_common_verbs.c |   2 +-
>  drivers/common/mlx5/mlx5_common_pci.c         |   2 +-
>  drivers/common/octeontx2/otx2_dev.h           |   2 +-
>  drivers/common/octeontx2/otx2_sec_idev.c      |   2 +-
>  drivers/common/qat/qat_device.h               |   2 +-
>  drivers/common/qat/qat_qp.c                   |   2 +-
>  drivers/common/sfc_efx/sfc_efx.h              |   2 +-
>  drivers/compress/mlx5/mlx5_compress.c         |   2 +-
>  drivers/compress/octeontx/otx_zip.h           |   2 +-
>  drivers/compress/qat/qat_comp.c               |   2 +-
>  drivers/crypto/ccp/ccp_dev.h                  |   2 +-
>  drivers/crypto/ccp/ccp_pci.h                  |   2 +-
>  drivers/crypto/ccp/rte_ccp_pmd.c              |   2 +-
>  drivers/crypto/cnxk/cn10k_cryptodev.c         |   2 +-
>  drivers/crypto/cnxk/cn9k_cryptodev.c          |   2 +-
>  drivers/crypto/mlx5/mlx5_crypto.c             |   2 +-
>  drivers/crypto/nitrox/nitrox_device.h         |   2 +-
>  drivers/crypto/octeontx/otx_cryptodev.c       |   2 +-
>  drivers/crypto/octeontx/otx_cryptodev_ops.c   |   2 +-
>  drivers/crypto/octeontx2/otx2_cryptodev.c     |   2 +-
>  drivers/crypto/qat/qat_sym.c                  |   2 +-
>  drivers/crypto/qat/qat_sym_pmd.c              |   2 +-
>  drivers/crypto/virtio/virtio_cryptodev.c      |   2 +-
>  drivers/crypto/virtio/virtio_pci.h            |   2 +-
>  drivers/event/dlb2/pf/dlb2_main.h             |   2 +-
>  drivers/event/dlb2/pf/dlb2_pf.c               |   2 +-
>  drivers/event/octeontx/ssovf_probe.c          |   2 +-
>  drivers/event/octeontx/timvf_probe.c          |   2 +-
>  drivers/event/octeontx2/otx2_evdev.c          |   2 +-
>  drivers/mempool/cnxk/cnxk_mempool.c           |   2 +-
>  drivers/mempool/octeontx/octeontx_fpavf.c     |   2 +-
>  drivers/mempool/octeontx2/otx2_mempool.c      |   2 +-
>  drivers/mempool/octeontx2/otx2_mempool.h      |   2 +-
>  drivers/mempool/octeontx2/otx2_mempool_irq.c  |   2 +-
>  drivers/meson.build                           |   4 +
>  drivers/net/ark/ark_ethdev.c                  |   2 +-
>  drivers/net/avp/avp_ethdev.c                  |   2 +-
>  drivers/net/bnx2x/bnx2x.h                     |   2 +-
>  drivers/net/bnxt/bnxt.h                       |   2 +-
>  drivers/net/bonding/rte_eth_bond_args.c       |   2 +-
>  drivers/net/cxgbe/base/adapter.h              |   2 +-
>  drivers/net/cxgbe/cxgbe_ethdev.c              |   2 +-
>  drivers/net/e1000/em_ethdev.c                 |   2 +-
>  drivers/net/e1000/em_rxtx.c                   |   2 +-
>  drivers/net/e1000/igb_ethdev.c                |   2 +-
>  drivers/net/e1000/igb_pf.c                    |   2 +-
>  drivers/net/ena/ena_ethdev.h                  |   2 +-
>  drivers/net/enic/base/vnic_dev.h              |   2 +-
>  drivers/net/enic/enic_ethdev.c                |   2 +-
>  drivers/net/enic/enic_main.c                  |   2 +-
>  drivers/net/enic/enic_vf_representor.c        |   2 +-
>  drivers/net/hinic/base/hinic_pmd_hwdev.c      |   2 +-
>  drivers/net/hinic/base/hinic_pmd_hwif.c       |   2 +-
>  drivers/net/hinic/base/hinic_pmd_nicio.c      |   2 +-
>  drivers/net/hinic/hinic_pmd_ethdev.c          |   2 +-
>  drivers/net/hns3/hns3_ethdev.c                |   2 +-
>  drivers/net/hns3/hns3_rxtx.c                  |   2 +-
>  drivers/net/i40e/i40e_ethdev.c                |   2 +-
>  drivers/net/i40e/i40e_ethdev_vf.c             |   2 +-
>  drivers/net/i40e/i40e_vf_representor.c        |   2 +-
>  drivers/net/igc/igc_ethdev.c                  |   2 +-
>  drivers/net/ionic/ionic.h                     |   2 +-
>  drivers/net/ionic/ionic_ethdev.c              |   2 +-
>  drivers/net/ipn3ke/ipn3ke_ethdev.c            |   2 +-
>  drivers/net/ipn3ke/ipn3ke_representor.c       |   2 +-
>  drivers/net/ipn3ke/ipn3ke_tm.c                |   2 +-
>  drivers/net/ixgbe/ixgbe_ethdev.c              |   2 +-
>  drivers/net/ixgbe/ixgbe_ethdev.h              |   2 +-
>  drivers/net/mlx4/mlx4_ethdev.c                |   2 +-
>  drivers/net/mlx5/linux/mlx5_ethdev_os.c       |   2 +-
>  drivers/net/mlx5/linux/mlx5_os.c              |   2 +-
>  drivers/net/mlx5/mlx5.c                       |   2 +-
>  drivers/net/mlx5/mlx5_ethdev.c                |   2 +-
>  drivers/net/mlx5/mlx5_txq.c                   |   2 +-
>  drivers/net/netvsc/hn_vf.c                    |   2 +-
>  drivers/net/octeontx/base/octeontx_pkivf.c    |   2 +-
>  drivers/net/octeontx/base/octeontx_pkovf.c    |   2 +-
>  drivers/net/octeontx2/otx2_ethdev_irq.c       |   2 +-
>  drivers/net/qede/base/bcm_osal.h              |   2 +-
>  drivers/net/sfc/sfc.h                         |   2 +-
>  drivers/net/sfc/sfc_ethdev.c                  |   2 +-
>  drivers/net/sfc/sfc_sriov.c                   |   2 +-
>  drivers/net/thunderx/nicvf_ethdev.c           |   2 +-
>  drivers/net/txgbe/txgbe_ethdev.h              |   2 +-
>  drivers/net/txgbe/txgbe_flow.c                |   2 +-
>  drivers/net/txgbe/txgbe_pf.c                  |   2 +-
>  drivers/net/virtio/virtio_pci.h               |   2 +-
>  drivers/net/virtio/virtio_pci_ethdev.c        |   2 +-
>  drivers/net/vmxnet3/vmxnet3_ethdev.c          |   2 +-
>  drivers/raw/cnxk_bphy/cnxk_bphy.c             |   2 +-
>  drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c         |   2 +-
>  drivers/raw/cnxk_bphy/cnxk_bphy_irq.c         |   2 +-
>  drivers/raw/cnxk_bphy/cnxk_bphy_irq.h         |   2 +-
>  drivers/raw/ifpga/ifpga_rawdev.c              |   2 +-
>  drivers/raw/ifpga/rte_pmd_ifpga.c             |   2 +-
>  drivers/raw/ioat/idxd_pci.c                   |   2 +-
>  drivers/raw/ioat/ioat_rawdev.c                |   2 +-
>  drivers/raw/ntb/ntb.c                         |   2 +-
>  drivers/raw/ntb/ntb_hw_intel.c                |   2 +-
>  drivers/raw/octeontx2_dma/otx2_dpi_rawdev.c   |   2 +-
>  drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c     |   2 +-
>  drivers/raw/octeontx2_ep/otx2_ep_rawdev.c     |   2 +-
>  drivers/regex/mlx5/mlx5_regex.c               |   2 +-
>  drivers/regex/mlx5/mlx5_regex_fastpath.c      |   2 +-
>  drivers/vdpa/ifc/base/ifcvf_osdep.h           |   2 +-
>  drivers/vdpa/ifc/ifcvf_vdpa.c                 |   2 +-
>  drivers/vdpa/mlx5/mlx5_vdpa.c                 |   2 +-
>  lib/ethdev/ethdev_pci.h                       |   2 +-
>  lib/eventdev/eventdev_pmd_pci.h               |   2 +-
>  126 files changed, 546 insertions(+), 508 deletions(-)
>  create mode 100644 drivers/bus/pci/pci_driver.h
> 

Acked-by: Ray Kinsella <mdr@ashroe.eu>

Re: [dpdk-dev] [PATCH 8/8] bus/pci: remove ABIs in PCI bus

[Xu, Rosen]

> -----Original Message-----
> From: Xia, Chenbo <chenbo.xia@intel.com>
> Sent: Friday, September 10, 2021 10:24
> To: dev@dpdk.org
> Cc: Chautru, Nicolas <nicolas.chautru@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Burakov, Anatoly <anatoly.burakov@intel.com>;
> Ray Kinsella <mdr@ashroe.eu>; Nithin Dabilpuram
> <ndabilpuram@marvell.com>; Kiran Kumar K <kirankumark@marvell.com>;
> Sunil Kumar Kori <skori@marvell.com>; Satha Rao
> <skoteshwar@marvell.com>; Matan Azrad <matan@nvidia.com>; Shahaf
> Shuler <shahafs@nvidia.com>; Viacheslav Ovsiienko
> <viacheslavo@nvidia.com>; Jerin Jacob <jerinj@marvell.com>; Anoob Joseph
> <anoobj@marvell.com>; Trahe, Fiona <fiona.trahe@intel.com>; Griffin, John
> <john.griffin@intel.com>; Jain, Deepak K <deepak.k.jain@intel.com>;
> Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>; Ashish Gupta
> <ashish.gupta@marvell.com>; Somalapuram Amaranath
> <asomalap@amd.com>; Ankur Dwivedi <adwivedi@marvell.com>; Tejasree
> Kondoj <ktejasree@marvell.com>; Nagadheeraj Rottela
> <rnagadheeraj@marvell.com>; Srikanth Jampala <jsrikanth@marvell.com>;
> Jay Zhou <jianjay.zhou@huawei.com>; McDaniel, Timothy
> <timothy.mcdaniel@intel.com>; Pavan Nikhilesh
> <pbhagavatula@marvell.com>; Ashwin Sekhar T K <asekhar@marvell.com>;
> Harman Kalra <hkalra@marvell.com>; Shepard Siegel
> <shepard.siegel@atomicrules.com>; Ed Czeck <ed.czeck@atomicrules.com>;
> John Miller <john.miller@atomicrules.com>; Steven Webster
> <steven.webster@windriver.com>; Peters, Matt
> <matt.peters@windriver.com>; Rasesh Mody <rmody@marvell.com>;
> Shahed Shaikh <shshaikh@marvell.com>; Ajit Khaparde
> <ajit.khaparde@broadcom.com>; Somnath Kotur
> <somnath.kotur@broadcom.com>; Chas Williams <chas3@att.com>; Min Hu
> (Connor) <humin29@huawei.com>; Rahul Lakkireddy
> <rahul.lakkireddy@chelsio.com>; Wang, Haiyue <haiyue.wang@intel.com>;
> Marcin Wojtas <mw@semihalf.com>; Michal Krawczyk <mk@semihalf.com>;
> Shai Brandes <shaibran@amazon.com>; Evgeny Schemeilin
> <evgenys@amazon.com>; Igor Chauskin <igorch@amazon.com>; Daley, John
> <johndale@cisco.com>; Hyong Youb Kim <hyonkim@cisco.com>; Ziyang
> Xuan <xuanziyang2@huawei.com>; Xiaoyun Wang
> <cloud.wangxiaoyun@huawei.com>; Guoyang Zhou
> <zhouguoyang@huawei.com>; Yisen Zhuang <yisen.zhuang@huawei.com>;
> Lijun Ou <oulijun@huawei.com>; Xing, Beilei <beilei.xing@intel.com>;
> Andrew Boyer <aboyer@pensando.io>; Xu, Rosen <rosen.xu@intel.com>;
> Stephen Hemminger <sthemmin@microsoft.com>; Long Li
> <longli@microsoft.com>; Devendra Singh Rawat
> <dsinghrawat@marvell.com>; Maciej Czekaj <mczekaj@marvell.com>;
> Jiawen Wu <jiawenwu@trustnetic.com>; Jian Wang
> <jianwang@trustnetic.com>; Maxime Coquelin
> <maxime.coquelin@redhat.com>; Yong Wang <yongwang@vmware.com>;
> Jakub Palider <jpalider@marvell.com>; Tomasz Duszynski
> <tduszynski@marvell.com>; Zhang, Tianfei <tianfei.zhang@intel.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; Li, Xiaoyun
> <xiaoyun.li@intel.com>; Wu, Jingjing <jingjing.wu@intel.com>; Radha Mohan
> Chintakuntla <radhac@marvell.com>; Veerasenareddy Burru
> <vburru@marvell.com>; Ori Kam <orika@nvidia.com>; Wang, Xiao W
> <xiao.w.wang@intel.com>; Thomas Monjalon <thomas@monjalon.net>
> Subject: [PATCH 8/8] bus/pci: remove ABIs in PCI bus
> 
> As announced in the deprecation note, most of ABIs in PCI bus are
> removed in this patch. Only the function rte_pci_dump is still ABI
> and experimental APIs are kept for future promotion.
> 
> This patch creates a new file named pci_driver.h and moves most of
> the content in original rte_bus_pci.h to it. After that, pci_driver.h
> is considered the interface for drivers and rte_bus_pci.h for
> applications. pci_driver.h is defined as driver_sdk_headers so that
> out-of-tree drivers can use it.
> 
> Then this patch replaces the including of rte_bus_pci.h with pci_driver.h
> in all related drivers.
> 
> Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
> ---
>  app/test/virtual_pmd.c                        |   2 +-
>  doc/guides/rel_notes/release_21_11.rst        |   2 +
>  drivers/baseband/acc100/rte_acc100_pmd.c      |   2 +-
>  .../fpga_5gnr_fec/rte_fpga_5gnr_fec.c         |   2 +-
>  drivers/baseband/fpga_lte_fec/fpga_lte_fec.c  |   2 +-
>  drivers/bus/pci/bsd/pci.c                     |   1 -
>  drivers/bus/pci/linux/pci.c                   |   1 -
>  drivers/bus/pci/linux/pci_uio.c               |   1 -
>  drivers/bus/pci/linux/pci_vfio.c              |   1 -
>  drivers/bus/pci/meson.build                   |   4 +
>  drivers/bus/pci/pci_common_uio.c              |   1 -
>  drivers/bus/pci/pci_driver.h                  | 402 ++++++++++++++++++
>  drivers/bus/pci/pci_params.c                  |   1 -
>  drivers/bus/pci/private.h                     |   3 +-
>  drivers/bus/pci/rte_bus_pci.h                 | 375 +---------------
>  drivers/bus/pci/version.map                   |  32 +-
>  drivers/common/cnxk/roc_platform.h            |   2 +-
>  drivers/common/mlx5/linux/mlx5_common_verbs.c |   2 +-
>  drivers/common/mlx5/mlx5_common_pci.c         |   2 +-
>  drivers/common/octeontx2/otx2_dev.h           |   2 +-
>  drivers/common/octeontx2/otx2_sec_idev.c      |   2 +-
>  drivers/common/qat/qat_device.h               |   2 +-
>  drivers/common/qat/qat_qp.c                   |   2 +-
>  drivers/common/sfc_efx/sfc_efx.h              |   2 +-
>  drivers/compress/mlx5/mlx5_compress.c         |   2 +-
>  drivers/compress/octeontx/otx_zip.h           |   2 +-
>  drivers/compress/qat/qat_comp.c               |   2 +-
>  drivers/crypto/ccp/ccp_dev.h                  |   2 +-
>  drivers/crypto/ccp/ccp_pci.h                  |   2 +-
>  drivers/crypto/ccp/rte_ccp_pmd.c              |   2 +-
>  drivers/crypto/cnxk/cn10k_cryptodev.c         |   2 +-
>  drivers/crypto/cnxk/cn9k_cryptodev.c          |   2 +-
>  drivers/crypto/mlx5/mlx5_crypto.c             |   2 +-
>  drivers/crypto/nitrox/nitrox_device.h         |   2 +-
>  drivers/crypto/octeontx/otx_cryptodev.c       |   2 +-
>  drivers/crypto/octeontx/otx_cryptodev_ops.c   |   2 +-
>  drivers/crypto/octeontx2/otx2_cryptodev.c     |   2 +-
>  drivers/crypto/qat/qat_sym.c                  |   2 +-
>  drivers/crypto/qat/qat_sym_pmd.c              |   2 +-
>  drivers/crypto/virtio/virtio_cryptodev.c      |   2 +-
>  drivers/crypto/virtio/virtio_pci.h            |   2 +-
>  drivers/event/dlb2/pf/dlb2_main.h             |   2 +-
>  drivers/event/dlb2/pf/dlb2_pf.c               |   2 +-
>  drivers/event/octeontx/ssovf_probe.c          |   2 +-
>  drivers/event/octeontx/timvf_probe.c          |   2 +-
>  drivers/event/octeontx2/otx2_evdev.c          |   2 +-
>  drivers/mempool/cnxk/cnxk_mempool.c           |   2 +-
>  drivers/mempool/octeontx/octeontx_fpavf.c     |   2 +-
>  drivers/mempool/octeontx2/otx2_mempool.c      |   2 +-
>  drivers/mempool/octeontx2/otx2_mempool.h      |   2 +-
>  drivers/mempool/octeontx2/otx2_mempool_irq.c  |   2 +-
>  drivers/meson.build                           |   4 +
>  drivers/net/ark/ark_ethdev.c                  |   2 +-
>  drivers/net/avp/avp_ethdev.c                  |   2 +-
>  drivers/net/bnx2x/bnx2x.h                     |   2 +-
>  drivers/net/bnxt/bnxt.h                       |   2 +-
>  drivers/net/bonding/rte_eth_bond_args.c       |   2 +-
>  drivers/net/cxgbe/base/adapter.h              |   2 +-
>  drivers/net/cxgbe/cxgbe_ethdev.c              |   2 +-
>  drivers/net/e1000/em_ethdev.c                 |   2 +-
>  drivers/net/e1000/em_rxtx.c                   |   2 +-
>  drivers/net/e1000/igb_ethdev.c                |   2 +-
>  drivers/net/e1000/igb_pf.c                    |   2 +-
>  drivers/net/ena/ena_ethdev.h                  |   2 +-
>  drivers/net/enic/base/vnic_dev.h              |   2 +-
>  drivers/net/enic/enic_ethdev.c                |   2 +-
>  drivers/net/enic/enic_main.c                  |   2 +-
>  drivers/net/enic/enic_vf_representor.c        |   2 +-
>  drivers/net/hinic/base/hinic_pmd_hwdev.c      |   2 +-
>  drivers/net/hinic/base/hinic_pmd_hwif.c       |   2 +-
>  drivers/net/hinic/base/hinic_pmd_nicio.c      |   2 +-
>  drivers/net/hinic/hinic_pmd_ethdev.c          |   2 +-
>  drivers/net/hns3/hns3_ethdev.c                |   2 +-
>  drivers/net/hns3/hns3_rxtx.c                  |   2 +-
>  drivers/net/i40e/i40e_ethdev.c                |   2 +-
>  drivers/net/i40e/i40e_ethdev_vf.c             |   2 +-
>  drivers/net/i40e/i40e_vf_representor.c        |   2 +-
>  drivers/net/igc/igc_ethdev.c                  |   2 +-
>  drivers/net/ionic/ionic.h                     |   2 +-
>  drivers/net/ionic/ionic_ethdev.c              |   2 +-
>  drivers/net/ipn3ke/ipn3ke_ethdev.c            |   2 +-
>  drivers/net/ipn3ke/ipn3ke_representor.c       |   2 +-
>  drivers/net/ipn3ke/ipn3ke_tm.c                |   2 +-
>  drivers/net/ixgbe/ixgbe_ethdev.c              |   2 +-
>  drivers/net/ixgbe/ixgbe_ethdev.h              |   2 +-
>  drivers/net/mlx4/mlx4_ethdev.c                |   2 +-
>  drivers/net/mlx5/linux/mlx5_ethdev_os.c       |   2 +-
>  drivers/net/mlx5/linux/mlx5_os.c              |   2 +-
>  drivers/net/mlx5/mlx5.c                       |   2 +-
>  drivers/net/mlx5/mlx5_ethdev.c                |   2 +-
>  drivers/net/mlx5/mlx5_txq.c                   |   2 +-
>  drivers/net/netvsc/hn_vf.c                    |   2 +-
>  drivers/net/octeontx/base/octeontx_pkivf.c    |   2 +-
>  drivers/net/octeontx/base/octeontx_pkovf.c    |   2 +-
>  drivers/net/octeontx2/otx2_ethdev_irq.c       |   2 +-
>  drivers/net/qede/base/bcm_osal.h              |   2 +-
>  drivers/net/sfc/sfc.h                         |   2 +-
>  drivers/net/sfc/sfc_ethdev.c                  |   2 +-
>  drivers/net/sfc/sfc_sriov.c                   |   2 +-
>  drivers/net/thunderx/nicvf_ethdev.c           |   2 +-
>  drivers/net/txgbe/txgbe_ethdev.h              |   2 +-
>  drivers/net/txgbe/txgbe_flow.c                |   2 +-
>  drivers/net/txgbe/txgbe_pf.c                  |   2 +-
>  drivers/net/virtio/virtio_pci.h               |   2 +-
>  drivers/net/virtio/virtio_pci_ethdev.c        |   2 +-
>  drivers/net/vmxnet3/vmxnet3_ethdev.c          |   2 +-
>  drivers/raw/cnxk_bphy/cnxk_bphy.c             |   2 +-
>  drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c         |   2 +-
>  drivers/raw/cnxk_bphy/cnxk_bphy_irq.c         |   2 +-
>  drivers/raw/cnxk_bphy/cnxk_bphy_irq.h         |   2 +-
>  drivers/raw/ifpga/ifpga_rawdev.c              |   2 +-
>  drivers/raw/ifpga/rte_pmd_ifpga.c             |   2 +-
>  drivers/raw/ioat/idxd_pci.c                   |   2 +-
>  drivers/raw/ioat/ioat_rawdev.c                |   2 +-
>  drivers/raw/ntb/ntb.c                         |   2 +-
>  drivers/raw/ntb/ntb_hw_intel.c                |   2 +-
>  drivers/raw/octeontx2_dma/otx2_dpi_rawdev.c   |   2 +-
>  drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c     |   2 +-
>  drivers/raw/octeontx2_ep/otx2_ep_rawdev.c     |   2 +-
>  drivers/regex/mlx5/mlx5_regex.c               |   2 +-
>  drivers/regex/mlx5/mlx5_regex_fastpath.c      |   2 +-
>  drivers/vdpa/ifc/base/ifcvf_osdep.h           |   2 +-
>  drivers/vdpa/ifc/ifcvf_vdpa.c                 |   2 +-
>  drivers/vdpa/mlx5/mlx5_vdpa.c                 |   2 +-
>  lib/ethdev/ethdev_pci.h                       |   2 +-
>  lib/eventdev/eventdev_pmd_pci.h               |   2 +-
>  126 files changed, 546 insertions(+), 508 deletions(-)
>  create mode 100644 drivers/bus/pci/pci_driver.h
> 

Acked-by: Rosen Xu <rosen.xu@intel.com>

[dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Chenbo Xia]

As announced in the deprecation notice, most ABIs in PCI bus will be removed.

As there exist some applications that want to access PCI memory resource,
two new APIs are defined in Patch 1 and corresponding changes are applied
to testpmd in Patch 2.

Patch 3-4 clean up the unnecessary usage of PCI bus header in examples.

Patch 5-6 clean up the unused PCI related structure in kni library and related
tests and examples.

Patch 7 finally removes most of ABIs in PCI bus.

---
v2: 
 - Add check on call of port_id_pci_reg_write (Xiaoyun)
 - Combine two clean-up patches in test and example, and backport (David)

Chenbo Xia (7):
  bus/pci: add new memory resource access APIs
  app/testpmd: use PCI memory resource access APIs
  examples/ethtool: use PCI library API to get PCI address
  examples/kni: remove unused PCI bus header
  kni: remove unused PCI info from test and example
  kni: replace unused variable definition with reserved bytes
  bus/pci: remove ABIs in PCI bus

 app/test-pmd/config.c                         |  50 +--
 app/test-pmd/testpmd.h                        |  54 +--
 app/test/test_kni.c                           |  57 ---
 app/test/virtual_pmd.c                        |   2 +-
 doc/guides/rel_notes/release_21_11.rst        |   8 +
 drivers/baseband/acc100/rte_acc100_pmd.c      |   2 +-
 .../fpga_5gnr_fec/rte_fpga_5gnr_fec.c         |   2 +-
 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c  |   2 +-
 drivers/bus/pci/bsd/pci.c                     |   1 -
 drivers/bus/pci/linux/pci.c                   |   1 -
 drivers/bus/pci/linux/pci_uio.c               |   1 -
 drivers/bus/pci/linux/pci_vfio.c              |   1 -
 drivers/bus/pci/meson.build                   |   4 +
 drivers/bus/pci/pci_common.c                  |  78 ++++
 drivers/bus/pci/pci_common_uio.c              |   1 -
 drivers/bus/pci/pci_driver.h                  | 402 ++++++++++++++++++
 drivers/bus/pci/pci_params.c                  |   1 -
 drivers/bus/pci/private.h                     |   3 +-
 drivers/bus/pci/rte_bus_pci.h                 | 387 ++---------------
 drivers/bus/pci/version.map                   |  28 +-
 drivers/common/cnxk/roc_platform.h            |   2 +-
 drivers/common/mlx5/linux/mlx5_common_verbs.c |   2 +-
 drivers/common/mlx5/mlx5_common_pci.c         |   2 +-
 drivers/common/octeontx2/otx2_dev.h           |   2 +-
 drivers/common/octeontx2/otx2_sec_idev.c      |   2 +-
 drivers/common/qat/qat_device.h               |   2 +-
 drivers/common/qat/qat_qp.c                   |   2 +-
 drivers/common/sfc_efx/sfc_efx.h              |   2 +-
 drivers/compress/mlx5/mlx5_compress.c         |   2 +-
 drivers/compress/octeontx/otx_zip.h           |   2 +-
 drivers/compress/qat/qat_comp.c               |   2 +-
 drivers/crypto/ccp/ccp_dev.h                  |   2 +-
 drivers/crypto/ccp/ccp_pci.h                  |   2 +-
 drivers/crypto/ccp/rte_ccp_pmd.c              |   2 +-
 drivers/crypto/cnxk/cn10k_cryptodev.c         |   2 +-
 drivers/crypto/cnxk/cn9k_cryptodev.c          |   2 +-
 drivers/crypto/mlx5/mlx5_crypto.c             |   2 +-
 drivers/crypto/nitrox/nitrox_device.h         |   2 +-
 drivers/crypto/octeontx/otx_cryptodev.c       |   2 +-
 drivers/crypto/octeontx/otx_cryptodev_ops.c   |   2 +-
 drivers/crypto/octeontx2/otx2_cryptodev.c     |   2 +-
 drivers/crypto/qat/qat_sym.c                  |   2 +-
 drivers/crypto/qat/qat_sym_pmd.c              |   2 +-
 drivers/crypto/virtio/virtio_cryptodev.c      |   2 +-
 drivers/crypto/virtio/virtio_pci.h            |   2 +-
 drivers/event/dlb2/pf/dlb2_main.h             |   2 +-
 drivers/event/dlb2/pf/dlb2_pf.c               |   2 +-
 drivers/event/octeontx/ssovf_probe.c          |   2 +-
 drivers/event/octeontx/timvf_probe.c          |   2 +-
 drivers/event/octeontx2/otx2_evdev.c          |   2 +-
 drivers/mempool/cnxk/cnxk_mempool.c           |   2 +-
 drivers/mempool/octeontx/octeontx_fpavf.c     |   2 +-
 drivers/mempool/octeontx2/otx2_mempool.c      |   2 +-
 drivers/mempool/octeontx2/otx2_mempool.h      |   2 +-
 drivers/mempool/octeontx2/otx2_mempool_irq.c  |   2 +-
 drivers/meson.build                           |   4 +
 drivers/net/ark/ark_ethdev.c                  |   2 +-
 drivers/net/avp/avp_ethdev.c                  |   2 +-
 drivers/net/bnx2x/bnx2x.h                     |   2 +-
 drivers/net/bnxt/bnxt.h                       |   2 +-
 drivers/net/bonding/rte_eth_bond_args.c       |   2 +-
 drivers/net/cxgbe/base/adapter.h              |   2 +-
 drivers/net/cxgbe/cxgbe_ethdev.c              |   2 +-
 drivers/net/e1000/em_ethdev.c                 |   2 +-
 drivers/net/e1000/em_rxtx.c                   |   2 +-
 drivers/net/e1000/igb_ethdev.c                |   2 +-
 drivers/net/e1000/igb_pf.c                    |   2 +-
 drivers/net/ena/ena_ethdev.h                  |   2 +-
 drivers/net/enic/base/vnic_dev.h              |   2 +-
 drivers/net/enic/enic_ethdev.c                |   2 +-
 drivers/net/enic/enic_main.c                  |   2 +-
 drivers/net/enic/enic_vf_representor.c        |   2 +-
 drivers/net/hinic/base/hinic_pmd_hwdev.c      |   2 +-
 drivers/net/hinic/base/hinic_pmd_hwif.c       |   2 +-
 drivers/net/hinic/base/hinic_pmd_nicio.c      |   2 +-
 drivers/net/hinic/hinic_pmd_ethdev.c          |   2 +-
 drivers/net/hns3/hns3_ethdev.c                |   2 +-
 drivers/net/hns3/hns3_rxtx.c                  |   2 +-
 drivers/net/i40e/i40e_ethdev.c                |   2 +-
 drivers/net/i40e/i40e_ethdev_vf.c             |   2 +-
 drivers/net/i40e/i40e_vf_representor.c        |   2 +-
 drivers/net/igc/igc_ethdev.c                  |   2 +-
 drivers/net/ionic/ionic.h                     |   2 +-
 drivers/net/ionic/ionic_ethdev.c              |   2 +-
 drivers/net/ipn3ke/ipn3ke_ethdev.c            |   2 +-
 drivers/net/ipn3ke/ipn3ke_representor.c       |   2 +-
 drivers/net/ipn3ke/ipn3ke_tm.c                |   2 +-
 drivers/net/ixgbe/ixgbe_ethdev.c              |   2 +-
 drivers/net/ixgbe/ixgbe_ethdev.h              |   2 +-
 drivers/net/mlx4/mlx4_ethdev.c                |   2 +-
 drivers/net/mlx5/linux/mlx5_ethdev_os.c       |   2 +-
 drivers/net/mlx5/linux/mlx5_os.c              |   2 +-
 drivers/net/mlx5/mlx5.c                       |   2 +-
 drivers/net/mlx5/mlx5_ethdev.c                |   2 +-
 drivers/net/mlx5/mlx5_txq.c                   |   2 +-
 drivers/net/netvsc/hn_vf.c                    |   2 +-
 drivers/net/octeontx/base/octeontx_pkivf.c    |   2 +-
 drivers/net/octeontx/base/octeontx_pkovf.c    |   2 +-
 drivers/net/octeontx2/otx2_ethdev_irq.c       |   2 +-
 drivers/net/qede/base/bcm_osal.h              |   2 +-
 drivers/net/sfc/sfc.h                         |   2 +-
 drivers/net/sfc/sfc_ethdev.c                  |   2 +-
 drivers/net/sfc/sfc_sriov.c                   |   2 +-
 drivers/net/thunderx/nicvf_ethdev.c           |   2 +-
 drivers/net/txgbe/txgbe_ethdev.h              |   2 +-
 drivers/net/txgbe/txgbe_flow.c                |   2 +-
 drivers/net/txgbe/txgbe_pf.c                  |   2 +-
 drivers/net/virtio/virtio_pci.h               |   2 +-
 drivers/net/virtio/virtio_pci_ethdev.c        |   2 +-
 drivers/net/vmxnet3/vmxnet3_ethdev.c          |   2 +-
 drivers/raw/cnxk_bphy/cnxk_bphy.c             |   2 +-
 drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c         |   2 +-
 drivers/raw/cnxk_bphy/cnxk_bphy_irq.c         |   2 +-
 drivers/raw/cnxk_bphy/cnxk_bphy_irq.h         |   2 +-
 drivers/raw/ifpga/ifpga_rawdev.c              |   2 +-
 drivers/raw/ifpga/rte_pmd_ifpga.c             |   2 +-
 drivers/raw/ioat/idxd_pci.c                   |   2 +-
 drivers/raw/ioat/ioat_rawdev.c                |   2 +-
 drivers/raw/ntb/ntb.c                         |   2 +-
 drivers/raw/ntb/ntb_hw_intel.c                |   2 +-
 drivers/raw/octeontx2_dma/otx2_dpi_rawdev.c   |   2 +-
 drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c     |   2 +-
 drivers/raw/octeontx2_ep/otx2_ep_rawdev.c     |   2 +-
 drivers/regex/mlx5/mlx5_regex.c               |   2 +-
 drivers/regex/mlx5/mlx5_regex_fastpath.c      |   2 +-
 drivers/vdpa/ifc/base/ifcvf_osdep.h           |   2 +-
 drivers/vdpa/ifc/ifcvf_vdpa.c                 |   2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |   2 +-
 examples/ethtool/lib/rte_ethtool.c            |  14 +-
 examples/ethtool/meson.build                  |   2 +-
 examples/ip_pipeline/kni.c                    |  16 -
 examples/kni/main.c                           |   1 -
 lib/ethdev/ethdev_pci.h                       |   2 +-
 lib/eventdev/eventdev_pmd_pci.h               |   2 +-
 lib/kni/rte_kni.h                             |   4 +-
 135 files changed, 711 insertions(+), 633 deletions(-)
 create mode 100644 drivers/bus/pci/pci_driver.h

-- 
2.17.1

[dpdk-dev] [PATCH v2 1/7] bus/pci: add new memory resource access APIs

[Chenbo Xia]

Some applications wants to access PCI memory resource. Currently
applications use struct rte_pci_device to access it. Since the
structure will be made internal later, this patch adds two APIs
for memory resource access.

Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
Acked-by: Ray Kinsella <mdr@ashroe.eu>
---
 doc/guides/rel_notes/release_21_11.rst |  6 ++
 drivers/bus/pci/pci_common.c           | 78 ++++++++++++++++++++++++++
 drivers/bus/pci/rte_bus_pci.h          | 36 ++++++++++++
 drivers/bus/pci/version.map            |  4 ++
 4 files changed, 124 insertions(+)

diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst
index 1d56fa9bf2..ce3f554e10 100644
--- a/doc/guides/rel_notes/release_21_11.rst
+++ b/doc/guides/rel_notes/release_21_11.rst
@@ -55,6 +55,12 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added new memory resource read/write APIs in PCI bus.**
+
+  Added new memory resource read/write APIs ``rte_pci_mem_rd32`` and
+  ``rte_pci_mem_wr32`` for applications to read/write PCI memory
+  resource.
+
 * **Enabled new devargs parser.**
 
   * Enabled devargs syntax
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index 3406e03b29..5bc7c8e2c7 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -25,6 +25,7 @@
 #include <rte_common.h>
 #include <rte_devargs.h>
 #include <rte_vfio.h>
+#include <rte_io.h>
 
 #include "private.h"
 
@@ -777,6 +778,83 @@ rte_pci_set_bus_master(struct rte_pci_device *dev, bool enable)
 	return 0;
 }
 
+static void *
+get_pci_mem_addr(const char *name, uint16_t idx, uint64_t offset)
+{
+	struct rte_pci_device *dev = NULL;
+	struct rte_pci_addr addr = {0};
+	struct rte_mem_resource *res = NULL;
+	bool found = false;
+
+	if (rte_pci_addr_parse(name, &addr)) {
+		RTE_LOG(ERR, EAL, "Wrong name format of PCI device (%s)", name);
+		return NULL;
+	}
+
+	FOREACH_DEVICE_ON_PCIBUS(dev) {
+		if (rte_pci_addr_cmp(&dev->addr, &addr)) {
+			continue;
+		} else {
+			found = true;
+			break;
+		}
+	}
+
+	if (!found) {
+		RTE_LOG(ERR, EAL, "Can not find the device (%s)", name);
+		return NULL;
+	}
+
+	res = &dev->mem_resource[idx];
+	if (idx >= PCI_MAX_RESOURCE || res->len == 0 || res->addr == NULL) {
+		RTE_LOG(ERR, EAL, "Invalid index of a mapped memory resourse");
+		return NULL;
+	}
+
+	if (offset + 4 > res->len) {
+		RTE_LOG(ERR, EAL, "Invalid offset of a memory resourse");
+		return NULL;
+	}
+
+	return (void *)((char *)res->addr + offset);
+}
+
+int
+rte_pci_mem_rd32(const char *name, uint16_t idx, uint32_t *data, uint64_t offset)
+{
+	void *reg_addr = NULL;
+
+	if (data == NULL) {
+		RTE_LOG(ERR, EAL, "NULL data buffer for PCI memory access");
+		return -EINVAL;
+	}
+
+	reg_addr = get_pci_mem_addr(name, idx, offset);
+	if (reg_addr == NULL)
+		return -EINVAL;
+
+	*data = rte_read32(reg_addr);
+	return 0;
+}
+
+int
+rte_pci_mem_wr32(const char *name, uint16_t idx, const uint32_t *data, uint64_t offset)
+{
+	void *reg_addr = NULL;
+
+	if (data == NULL) {
+		RTE_LOG(ERR, EAL, "NULL data buffer for PCI memory access");
+		return -EINVAL;
+	}
+
+	reg_addr = get_pci_mem_addr(name, idx, offset);
+	if (reg_addr == NULL)
+		return -EINVAL;
+
+	rte_write32(*data, reg_addr);
+	return 0;
+}
+
 struct rte_pci_bus rte_pci_bus = {
 	.bus = {
 		.scan = rte_pci_scan,
diff --git a/drivers/bus/pci/rte_bus_pci.h b/drivers/bus/pci/rte_bus_pci.h
index 583470e831..21d9dd4289 100644
--- a/drivers/bus/pci/rte_bus_pci.h
+++ b/drivers/bus/pci/rte_bus_pci.h
@@ -392,6 +392,42 @@ void rte_pci_ioport_read(struct rte_pci_ioport *p,
 void rte_pci_ioport_write(struct rte_pci_ioport *p,
 		const void *data, size_t len, off_t offset);
 
+/**
+ * Read 4 bytes from PCI memory resource.
+ *
+ * @param name
+ *   PCI device name (e.g., 0000:18:00.0).
+ * @param idx
+ *   Memory resource index.
+ * @param data
+ *   Data buffer where the bytes should be read into.
+ * @param offset
+ *   The offset into the PCI memory resource.
+ * @return
+ *  0 on success, negative value on error.
+ */
+__rte_experimental
+int
+rte_pci_mem_rd32(const char *name, uint16_t idx, uint32_t *data, uint64_t offset);
+
+/**
+ * Write 4 bytes to PCI memory resource.
+ *
+ * @param name
+ *   PCI device name (e.g., 0000:18:00.0).
+ * @param idx
+ *   Memory resource index.
+ * @param data
+ *   Buffer of data that should be written to PCI memory.
+ * @param offset
+ *   The offset into the PCI memory resource.
+ * @return
+ *  0 on success, negative value on error.
+ */
+__rte_experimental
+int
+rte_pci_mem_wr32(const char *name, uint16_t idx, const uint32_t *data, uint64_t offset);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/drivers/bus/pci/version.map b/drivers/bus/pci/version.map
index aa56439c2b..01ec836559 100644
--- a/drivers/bus/pci/version.map
+++ b/drivers/bus/pci/version.map
@@ -24,4 +24,8 @@ EXPERIMENTAL {
 
 	# added in 21.08
 	rte_pci_set_bus_master;
+
+	# added in 21.11
+	rte_pci_mem_rd32;
+	rte_pci_mem_wr32;
 };
-- 
2.17.1

[dpdk-dev] [PATCH v2 2/7] app/testpmd: use PCI memory resource access APIs

[Chenbo Xia]

Currently testpmd uses struct rte_pci_device to access PCI memory
resource. Since this structure will be internal later, this patch
replaces use of rte_pci_device with new PCI memory resource access
APIs to read/write BAR 0.

Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
---
 app/test-pmd/config.c  | 50 ++++++++++++++------------------------
 app/test-pmd/testpmd.h | 54 ++++++++++++++++++++++--------------------
 2 files changed, 46 insertions(+), 58 deletions(-)

diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index f5765b34f7..67be2f9ee7 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -948,10 +948,6 @@ vlan_id_is_invalid(uint16_t vlan_id)
 static int
 port_reg_off_is_invalid(portid_t port_id, uint32_t reg_off)
 {
-	const struct rte_pci_device *pci_dev;
-	const struct rte_bus *bus;
-	uint64_t pci_len;
-
 	if (reg_off & 0x3) {
 		fprintf(stderr,
 			"Port register offset 0x%X not aligned on a 4-byte boundary\n",
@@ -964,22 +960,6 @@ port_reg_off_is_invalid(portid_t port_id, uint32_t reg_off)
 		return 0;
 	}
 
-	bus = rte_bus_find_by_device(ports[port_id].dev_info.device);
-	if (bus && !strcmp(bus->name, "pci")) {
-		pci_dev = RTE_DEV_TO_PCI(ports[port_id].dev_info.device);
-	} else {
-		fprintf(stderr, "Not a PCI device\n");
-		return 1;
-	}
-
-	pci_len = pci_dev->mem_resource[0].len;
-	if (reg_off >= pci_len) {
-		fprintf(stderr,
-			"Port %d: register offset %u (0x%X) out of port PCI resource (length=%"PRIu64")\n",
-			port_id, (unsigned int)reg_off, (unsigned int)reg_off,
-			pci_len);
-		return 1;
-	}
 	return 0;
 }
 
@@ -1007,14 +987,14 @@ port_reg_bit_display(portid_t port_id, uint32_t reg_off, uint8_t bit_x)
 {
 	uint32_t reg_v;
 
-
 	if (port_id_is_invalid(port_id, ENABLED_WARN))
 		return;
 	if (port_reg_off_is_invalid(port_id, reg_off))
 		return;
 	if (reg_bit_pos_is_invalid(bit_x))
 		return;
-	reg_v = port_id_pci_reg_read(port_id, reg_off);
+	if (port_id_pci_reg_read(port_id, reg_off, &reg_v))
+		return;
 	display_port_and_reg_off(port_id, (unsigned)reg_off);
 	printf("bit %d=%d\n", bit_x, (int) ((reg_v & (1 << bit_x)) >> bit_x));
 }
@@ -1040,7 +1020,8 @@ port_reg_bit_field_display(portid_t port_id, uint32_t reg_off,
 	else
 		l_bit = bit1_pos, h_bit = bit2_pos;
 
-	reg_v = port_id_pci_reg_read(port_id, reg_off);
+	if (port_id_pci_reg_read(port_id, reg_off, &reg_v))
+		return;
 	reg_v >>= l_bit;
 	if (h_bit < 31)
 		reg_v &= ((1 << (h_bit - l_bit + 1)) - 1);
@@ -1058,7 +1039,8 @@ port_reg_display(portid_t port_id, uint32_t reg_off)
 		return;
 	if (port_reg_off_is_invalid(port_id, reg_off))
 		return;
-	reg_v = port_id_pci_reg_read(port_id, reg_off);
+	if (port_id_pci_reg_read(port_id, reg_off, &reg_v))
+		return;
 	display_port_reg_value(port_id, reg_off, reg_v);
 }
 
@@ -1079,13 +1061,15 @@ port_reg_bit_set(portid_t port_id, uint32_t reg_off, uint8_t bit_pos,
 			(int) bit_v);
 		return;
 	}
-	reg_v = port_id_pci_reg_read(port_id, reg_off);
+	if (port_id_pci_reg_read(port_id, reg_off, &reg_v))
+		return;
+
 	if (bit_v == 0)
 		reg_v &= ~(1 << bit_pos);
 	else
 		reg_v |= (1 << bit_pos);
-	port_id_pci_reg_write(port_id, reg_off, reg_v);
-	display_port_reg_value(port_id, reg_off, reg_v);
+	if (!port_id_pci_reg_write(port_id, reg_off, reg_v))
+		display_port_reg_value(port_id, reg_off, reg_v);
 }
 
 void
@@ -1121,11 +1105,13 @@ port_reg_bit_field_set(portid_t port_id, uint32_t reg_off,
 				(unsigned)max_v, (unsigned)max_v);
 		return;
 	}
-	reg_v = port_id_pci_reg_read(port_id, reg_off);
+	if (port_id_pci_reg_read(port_id, reg_off, &reg_v))
+		return;
+
 	reg_v &= ~(max_v << l_bit); /* Keep unchanged bits */
 	reg_v |= (value << l_bit); /* Set changed bits */
-	port_id_pci_reg_write(port_id, reg_off, reg_v);
-	display_port_reg_value(port_id, reg_off, reg_v);
+	if (!port_id_pci_reg_write(port_id, reg_off, reg_v))
+		display_port_reg_value(port_id, reg_off, reg_v);
 }
 
 void
@@ -1135,8 +1121,8 @@ port_reg_set(portid_t port_id, uint32_t reg_off, uint32_t reg_v)
 		return;
 	if (port_reg_off_is_invalid(port_id, reg_off))
 		return;
-	port_id_pci_reg_write(port_id, reg_off, reg_v);
-	display_port_reg_value(port_id, reg_off, reg_v);
+	if (!port_id_pci_reg_write(port_id, reg_off, reg_v))
+		display_port_reg_value(port_id, reg_off, reg_v);
 }
 
 void
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 5863b2f43f..0025ad2f1a 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -688,61 +688,63 @@ mbuf_pool_find(unsigned int sock_id, uint16_t idx)
 /**
  * Read/Write operations on a PCI register of a port.
  */
-static inline uint32_t
-port_pci_reg_read(struct rte_port *port, uint32_t reg_off)
+static inline int
+port_id_pci_reg_read(portid_t pt_id, uint32_t reg_off, uint32_t *reg_v)
 {
-	const struct rte_pci_device *pci_dev;
+	struct rte_port *port = &ports[(pt_id)];
+	char name[RTE_ETH_NAME_MAX_LEN];
 	const struct rte_bus *bus;
-	void *reg_addr;
-	uint32_t reg_v;
 
 	if (!port->dev_info.device) {
 		fprintf(stderr, "Invalid device\n");
-		return 0;
+		return -1;
 	}
 
 	bus = rte_bus_find_by_device(port->dev_info.device);
 	if (bus && !strcmp(bus->name, "pci")) {
-		pci_dev = RTE_DEV_TO_PCI(port->dev_info.device);
+		rte_eth_dev_get_name_by_port(pt_id, name);
 	} else {
 		fprintf(stderr, "Not a PCI device\n");
-		return 0;
+		return -1;
 	}
 
-	reg_addr = ((char *)pci_dev->mem_resource[0].addr + reg_off);
-	reg_v = *((volatile uint32_t *)reg_addr);
-	return rte_le_to_cpu_32(reg_v);
-}
+	if (rte_pci_mem_rd32(name, 0, reg_v, reg_off)) {
+		fprintf(stderr, "Failed to read register\n");
+		return -1;
+	}
 
-#define port_id_pci_reg_read(pt_id, reg_off) \
-	port_pci_reg_read(&ports[(pt_id)], (reg_off))
+	*reg_v = rte_le_to_cpu_32(*reg_v);
+	return 0;
+}
 
-static inline void
-port_pci_reg_write(struct rte_port *port, uint32_t reg_off, uint32_t reg_v)
+static inline int
+port_id_pci_reg_write(portid_t pt_id, uint32_t reg_off, uint32_t reg_v)
 {
-	const struct rte_pci_device *pci_dev;
+	struct rte_port *port = &ports[(pt_id)];
+	char name[RTE_ETH_NAME_MAX_LEN];
 	const struct rte_bus *bus;
-	void *reg_addr;
 
 	if (!port->dev_info.device) {
 		fprintf(stderr, "Invalid device\n");
-		return;
+		return -1;
 	}
 
 	bus = rte_bus_find_by_device(port->dev_info.device);
 	if (bus && !strcmp(bus->name, "pci")) {
-		pci_dev = RTE_DEV_TO_PCI(port->dev_info.device);
+		rte_eth_dev_get_name_by_port(pt_id, name);
 	} else {
 		fprintf(stderr, "Not a PCI device\n");
-		return;
+		return -1;
 	}
 
-	reg_addr = ((char *)pci_dev->mem_resource[0].addr + reg_off);
-	*((volatile uint32_t *)reg_addr) = rte_cpu_to_le_32(reg_v);
-}
+	reg_v = rte_cpu_to_le_32(reg_v);
+	if (rte_pci_mem_wr32(name, 0, &reg_v, reg_off)) {
+		fprintf(stderr, "Failed to write register\n");
+		return -1;
+	}
 
-#define port_id_pci_reg_write(pt_id, reg_off, reg_value) \
-	port_pci_reg_write(&ports[(pt_id)], (reg_off), (reg_value))
+	return 0;
+}
 
 static inline void
 get_start_cycles(uint64_t *start_tsc)
-- 
2.17.1

Re: [dpdk-dev] [PATCH v2 2/7] app/testpmd: use PCI memory resource access APIs

[Li, Xiaoyun]

> -----Original Message-----
> From: Xia, Chenbo <chenbo.xia@intel.com>
> Sent: Saturday, September 18, 2021 10:25
> To: dev@dpdk.org; david.marchand@redhat.com
> Cc: Li, Xiaoyun <xiaoyun.li@intel.com>
> Subject: [PATCH v2 2/7] app/testpmd: use PCI memory resource access APIs
> 
> Currently testpmd uses struct rte_pci_device to access PCI memory resource.
> Since this structure will be internal later, this patch replaces use of
> rte_pci_device with new PCI memory resource access APIs to read/write BAR 0.
> 
> Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
> ---
>  app/test-pmd/config.c  | 50 ++++++++++++++------------------------
>  app/test-pmd/testpmd.h | 54 ++++++++++++++++++++++--------------------
>  2 files changed, 46 insertions(+), 58 deletions(-)

Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>

[dpdk-dev] [PATCH v2 3/7] examples/ethtool: use PCI library API to get PCI address

[Chenbo Xia]

Currently ethtool example uses struct rte_pci_device to know PCI
address of a device. As this API will be removed later in PCI bus,
this patch uses PCI library API to get the PCI address.

Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
---
 examples/ethtool/lib/rte_ethtool.c | 14 +++++++++-----
 examples/ethtool/meson.build       |  2 +-
 2 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/examples/ethtool/lib/rte_ethtool.c b/examples/ethtool/lib/rte_ethtool.c
index 4132516307..89727c9f72 100644
--- a/examples/ethtool/lib/rte_ethtool.c
+++ b/examples/ethtool/lib/rte_ethtool.c
@@ -8,7 +8,7 @@
 #include <rte_version.h>
 #include <rte_ethdev.h>
 #include <rte_ether.h>
-#include <rte_bus_pci.h>
+#include <rte_pci.h>
 #ifdef RTE_NET_IXGBE
 #include <rte_pmd_ixgbe.h>
 #endif
@@ -23,8 +23,9 @@ rte_ethtool_get_drvinfo(uint16_t port_id, struct ethtool_drvinfo *drvinfo)
 {
 	struct rte_eth_dev_info dev_info;
 	struct rte_dev_reg_info reg_info;
-	const struct rte_pci_device *pci_dev;
 	const struct rte_bus *bus = NULL;
+	char name[RTE_ETH_NAME_MAX_LEN];
+	struct rte_pci_addr addr;
 	int n;
 	int ret;
 
@@ -56,11 +57,14 @@ rte_ethtool_get_drvinfo(uint16_t port_id, struct ethtool_drvinfo *drvinfo)
 	if (dev_info.device)
 		bus = rte_bus_find_by_device(dev_info.device);
 	if (bus && !strcmp(bus->name, "pci")) {
-		pci_dev = RTE_DEV_TO_PCI(dev_info.device);
+		rte_eth_dev_get_name_by_port(port_id, name);
+		if (rte_pci_addr_parse(name, &addr)) {
+			printf("Failed to parse pci address\n");
+			return -1;
+		}
 		snprintf(drvinfo->bus_info, sizeof(drvinfo->bus_info),
 			"%04x:%02x:%02x.%x",
-			pci_dev->addr.domain, pci_dev->addr.bus,
-			pci_dev->addr.devid, pci_dev->addr.function);
+			addr.domain, addr.bus, addr.devid, addr.function);
 	} else {
 		snprintf(drvinfo->bus_info, sizeof(drvinfo->bus_info), "N/A");
 	}
diff --git a/examples/ethtool/meson.build b/examples/ethtool/meson.build
index d7f63d48af..c2dbf8ae5a 100644
--- a/examples/ethtool/meson.build
+++ b/examples/ethtool/meson.build
@@ -18,7 +18,7 @@ sources = files(
 )
 includes = include_directories('lib', 'ethtool-app')
 
-deps += 'bus_pci'
+deps += 'pci'
 if dpdk_conf.has('RTE_NET_IXGBE')
     deps += 'net_ixgbe'
 endif
-- 
2.17.1

[dpdk-dev] [PATCH v2 4/7] examples/kni: remove unused PCI bus header

[Chenbo Xia]

The header rte_bus_pci.h is included in kni example but nothing
in it is used. So remove it.

Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
 examples/kni/main.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/examples/kni/main.c b/examples/kni/main.c
index beabb3c848..6dc335c0b5 100644
--- a/examples/kni/main.c
+++ b/examples/kni/main.c
@@ -31,7 +31,6 @@
 #include <rte_lcore.h>
 #include <rte_branch_prediction.h>
 #include <rte_interrupts.h>
-#include <rte_bus_pci.h>
 #include <rte_debug.h>
 #include <rte_ether.h>
 #include <rte_ethdev.h>
-- 
2.17.1

[dpdk-dev] [PATCH v2 5/7] kni: remove unused PCI info from test and example

[Chenbo Xia]

PCI device id and address in structure rte_kni_conf are never used
in the test, example and kni library. So remove the related code.

Fixes: ea6b39b5b847 ("kni: remove ethtool support")
Cc: stable@dpdk.org

Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
 app/test/test_kni.c        | 57 --------------------------------------
 examples/ip_pipeline/kni.c | 16 -----------
 2 files changed, 73 deletions(-)

diff --git a/app/test/test_kni.c b/app/test/test_kni.c
index 96733554b6..aa9a316c50 100644
--- a/app/test/test_kni.c
+++ b/app/test/test_kni.c
@@ -25,7 +25,6 @@ test_kni(void)
 #include <rte_string_fns.h>
 #include <rte_mempool.h>
 #include <rte_ethdev.h>
-#include <rte_bus_pci.h>
 #include <rte_cycles.h>
 #include <rte_kni.h>
 
@@ -424,32 +423,14 @@ test_kni_processing(uint16_t port_id, struct rte_mempool *mp)
 	unsigned i;
 	struct rte_kni *kni;
 	struct rte_kni_conf conf;
-	struct rte_eth_dev_info info;
 	struct rte_kni_ops ops;
-	const struct rte_pci_device *pci_dev;
-	const struct rte_bus *bus = NULL;
 
 	if (!mp)
 		return -1;
 
 	memset(&conf, 0, sizeof(conf));
-	memset(&info, 0, sizeof(info));
 	memset(&ops, 0, sizeof(ops));
 
-	ret = rte_eth_dev_info_get(port_id, &info);
-	if (ret != 0) {
-		printf("Error during getting device (port %u) info: %s\n",
-				port_id, strerror(-ret));
-		return -1;
-	}
-
-	if (info.device)
-		bus = rte_bus_find_by_device(info.device);
-	if (bus && !strcmp(bus->name, "pci")) {
-		pci_dev = RTE_DEV_TO_PCI(info.device);
-		conf.addr = pci_dev->addr;
-		conf.id = pci_dev->id;
-	}
 	snprintf(conf.name, sizeof(conf.name), TEST_KNI_PORT);
 
 	/* core id 1 configured for kernel thread */
@@ -543,10 +524,7 @@ test_kni(void)
 	struct rte_kni *kni;
 	struct rte_mempool *mp;
 	struct rte_kni_conf conf;
-	struct rte_eth_dev_info info;
 	struct rte_kni_ops ops;
-	const struct rte_pci_device *pci_dev;
-	const struct rte_bus *bus;
 	FILE *fd;
 	DIR *dir;
 	char buf[16];
@@ -634,26 +612,9 @@ test_kni(void)
 	fclose(fd);
 
 	/* test of allocating KNI with NULL mempool pointer */
-	memset(&info, 0, sizeof(info));
 	memset(&conf, 0, sizeof(conf));
 	memset(&ops, 0, sizeof(ops));
 
-	ret = rte_eth_dev_info_get(port_id, &info);
-	if (ret != 0) {
-		printf("Error during getting device (port %u) info: %s\n",
-				port_id, strerror(-ret));
-		return -1;
-	}
-
-	if (info.device)
-		bus = rte_bus_find_by_device(info.device);
-	else
-		bus = NULL;
-	if (bus && !strcmp(bus->name, "pci")) {
-		pci_dev = RTE_DEV_TO_PCI(info.device);
-		conf.addr = pci_dev->addr;
-		conf.id = pci_dev->id;
-	}
 	conf.group_id = port_id;
 	conf.mbuf_size = MAX_PACKET_SZ;
 
@@ -678,26 +639,8 @@ test_kni(void)
 
 	/* test of allocating KNI without a name */
 	memset(&conf, 0, sizeof(conf));
-	memset(&info, 0, sizeof(info));
 	memset(&ops, 0, sizeof(ops));
 
-	ret = rte_eth_dev_info_get(port_id, &info);
-	if (ret != 0) {
-		printf("Error during getting device (port %u) info: %s\n",
-				port_id, strerror(-ret));
-		ret = -1;
-		goto fail;
-	}
-
-	if (info.device)
-		bus = rte_bus_find_by_device(info.device);
-	else
-		bus = NULL;
-	if (bus && !strcmp(bus->name, "pci")) {
-		pci_dev = RTE_DEV_TO_PCI(info.device);
-		conf.addr = pci_dev->addr;
-		conf.id = pci_dev->id;
-	}
 	conf.group_id = port_id;
 	conf.mbuf_size = MAX_PACKET_SZ;
 
diff --git a/examples/ip_pipeline/kni.c b/examples/ip_pipeline/kni.c
index a2d3331cb0..fccecc3dc6 100644
--- a/examples/ip_pipeline/kni.c
+++ b/examples/ip_pipeline/kni.c
@@ -6,7 +6,6 @@
 #include <string.h>
 
 #include <rte_ethdev.h>
-#include <rte_bus_pci.h>
 #include <rte_string_fns.h>
 
 #include "kni.h"
@@ -100,16 +99,12 @@ kni_change_mtu(uint16_t port_id, unsigned int new_mtu)
 struct kni *
 kni_create(const char *name, struct kni_params *params)
 {
-	struct rte_eth_dev_info dev_info;
 	struct rte_kni_conf kni_conf;
 	struct rte_kni_ops kni_ops;
 	struct kni *kni;
 	struct mempool *mempool;
 	struct link *link;
 	struct rte_kni *k;
-	const struct rte_pci_device *pci_dev;
-	const struct rte_bus *bus = NULL;
-	int ret;
 
 	/* Check input params */
 	if ((name == NULL) ||
@@ -124,23 +119,12 @@ kni_create(const char *name, struct kni_params *params)
 		return NULL;
 
 	/* Resource create */
-	ret = rte_eth_dev_info_get(link->port_id, &dev_info);
-	if (ret != 0)
-		return NULL;
-
 	memset(&kni_conf, 0, sizeof(kni_conf));
 	strlcpy(kni_conf.name, name, RTE_KNI_NAMESIZE);
 	kni_conf.force_bind = params->force_bind;
 	kni_conf.core_id = params->thread_id;
 	kni_conf.group_id = link->port_id;
 	kni_conf.mbuf_size = mempool->buffer_size;
-	if (dev_info.device)
-		bus = rte_bus_find_by_device(dev_info.device);
-	if (bus && !strcmp(bus->name, "pci")) {
-		pci_dev = RTE_DEV_TO_PCI(dev_info.device);
-		kni_conf.addr = pci_dev->addr;
-		kni_conf.id = pci_dev->id;
-	}
 
 	memset(&kni_ops, 0, sizeof(kni_ops));
 	kni_ops.port_id = link->port_id;
-- 
2.17.1

[dpdk-dev] [PATCH v2 6/7] kni: replace unused variable definition with reserved bytes

[Chenbo Xia]

PCI ID and address in structure rte_kni_conf are never used. And in
order not to break ABI, replace these variables with reserved bytes.

Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
---
 lib/kni/rte_kni.h | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/lib/kni/rte_kni.h b/lib/kni/rte_kni.h
index b0eaf46104..2281abbf6a 100644
--- a/lib/kni/rte_kni.h
+++ b/lib/kni/rte_kni.h
@@ -17,7 +17,6 @@
  * and burst transmit packets to KNI interfaces.
  */
 
-#include <rte_pci.h>
 #include <rte_memory.h>
 #include <rte_mempool.h>
 #include <rte_ether.h>
@@ -66,8 +65,7 @@ struct rte_kni_conf {
 	uint32_t core_id;   /* Core ID to bind kernel thread on */
 	uint16_t group_id;  /* Group ID */
 	unsigned mbuf_size; /* mbuf size */
-	struct rte_pci_addr addr; /* depreciated */
-	struct rte_pci_id id; /* depreciated */
+	uint8_t rsvd[20];
 
 	__extension__
 	uint8_t force_bind : 1; /* Flag to bind kernel thread */
-- 
2.17.1

[dpdk-dev] [PATCH v2 7/7] bus/pci: remove ABIs in PCI bus

[Chenbo Xia]

As announced in the deprecation note, most of ABIs in PCI bus are
removed in this patch. Only the function rte_pci_dump is still ABI
and experimental APIs are kept for future promotion.

This patch creates a new file named pci_driver.h and moves most of
the content in original rte_bus_pci.h to it. After that, pci_driver.h
is considered the interface for drivers and rte_bus_pci.h for
applications. pci_driver.h is defined as driver_sdk_headers so that
out-of-tree drivers can use it.

Then this patch replaces the including of rte_bus_pci.h with pci_driver.h
in all related drivers.

Signed-off-by: Chenbo Xia <chenbo.xia@intel.com>
Acked-by: Ray Kinsella <mdr@ashroe.eu>
Acked-by: Rosen Xu <rosen.xu@intel.com>
---
 app/test/virtual_pmd.c                        |   2 +-
 doc/guides/rel_notes/release_21_11.rst        |   2 +
 drivers/baseband/acc100/rte_acc100_pmd.c      |   2 +-
 .../fpga_5gnr_fec/rte_fpga_5gnr_fec.c         |   2 +-
 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c  |   2 +-
 drivers/bus/pci/bsd/pci.c                     |   1 -
 drivers/bus/pci/linux/pci.c                   |   1 -
 drivers/bus/pci/linux/pci_uio.c               |   1 -
 drivers/bus/pci/linux/pci_vfio.c              |   1 -
 drivers/bus/pci/meson.build                   |   4 +
 drivers/bus/pci/pci_common_uio.c              |   1 -
 drivers/bus/pci/pci_driver.h                  | 402 ++++++++++++++++++
 drivers/bus/pci/pci_params.c                  |   1 -
 drivers/bus/pci/private.h                     |   3 +-
 drivers/bus/pci/rte_bus_pci.h                 | 375 +---------------
 drivers/bus/pci/version.map                   |  32 +-
 drivers/common/cnxk/roc_platform.h            |   2 +-
 drivers/common/mlx5/linux/mlx5_common_verbs.c |   2 +-
 drivers/common/mlx5/mlx5_common_pci.c         |   2 +-
 drivers/common/octeontx2/otx2_dev.h           |   2 +-
 drivers/common/octeontx2/otx2_sec_idev.c      |   2 +-
 drivers/common/qat/qat_device.h               |   2 +-
 drivers/common/qat/qat_qp.c                   |   2 +-
 drivers/common/sfc_efx/sfc_efx.h              |   2 +-
 drivers/compress/mlx5/mlx5_compress.c         |   2 +-
 drivers/compress/octeontx/otx_zip.h           |   2 +-
 drivers/compress/qat/qat_comp.c               |   2 +-
 drivers/crypto/ccp/ccp_dev.h                  |   2 +-
 drivers/crypto/ccp/ccp_pci.h                  |   2 +-
 drivers/crypto/ccp/rte_ccp_pmd.c              |   2 +-
 drivers/crypto/cnxk/cn10k_cryptodev.c         |   2 +-
 drivers/crypto/cnxk/cn9k_cryptodev.c          |   2 +-
 drivers/crypto/mlx5/mlx5_crypto.c             |   2 +-
 drivers/crypto/nitrox/nitrox_device.h         |   2 +-
 drivers/crypto/octeontx/otx_cryptodev.c       |   2 +-
 drivers/crypto/octeontx/otx_cryptodev_ops.c   |   2 +-
 drivers/crypto/octeontx2/otx2_cryptodev.c     |   2 +-
 drivers/crypto/qat/qat_sym.c                  |   2 +-
 drivers/crypto/qat/qat_sym_pmd.c              |   2 +-
 drivers/crypto/virtio/virtio_cryptodev.c      |   2 +-
 drivers/crypto/virtio/virtio_pci.h            |   2 +-
 drivers/event/dlb2/pf/dlb2_main.h             |   2 +-
 drivers/event/dlb2/pf/dlb2_pf.c               |   2 +-
 drivers/event/octeontx/ssovf_probe.c          |   2 +-
 drivers/event/octeontx/timvf_probe.c          |   2 +-
 drivers/event/octeontx2/otx2_evdev.c          |   2 +-
 drivers/mempool/cnxk/cnxk_mempool.c           |   2 +-
 drivers/mempool/octeontx/octeontx_fpavf.c     |   2 +-
 drivers/mempool/octeontx2/otx2_mempool.c      |   2 +-
 drivers/mempool/octeontx2/otx2_mempool.h      |   2 +-
 drivers/mempool/octeontx2/otx2_mempool_irq.c  |   2 +-
 drivers/meson.build                           |   4 +
 drivers/net/ark/ark_ethdev.c                  |   2 +-
 drivers/net/avp/avp_ethdev.c                  |   2 +-
 drivers/net/bnx2x/bnx2x.h                     |   2 +-
 drivers/net/bnxt/bnxt.h                       |   2 +-
 drivers/net/bonding/rte_eth_bond_args.c       |   2 +-
 drivers/net/cxgbe/base/adapter.h              |   2 +-
 drivers/net/cxgbe/cxgbe_ethdev.c              |   2 +-
 drivers/net/e1000/em_ethdev.c                 |   2 +-
 drivers/net/e1000/em_rxtx.c                   |   2 +-
 drivers/net/e1000/igb_ethdev.c                |   2 +-
 drivers/net/e1000/igb_pf.c                    |   2 +-
 drivers/net/ena/ena_ethdev.h                  |   2 +-
 drivers/net/enic/base/vnic_dev.h              |   2 +-
 drivers/net/enic/enic_ethdev.c                |   2 +-
 drivers/net/enic/enic_main.c                  |   2 +-
 drivers/net/enic/enic_vf_representor.c        |   2 +-
 drivers/net/hinic/base/hinic_pmd_hwdev.c      |   2 +-
 drivers/net/hinic/base/hinic_pmd_hwif.c       |   2 +-
 drivers/net/hinic/base/hinic_pmd_nicio.c      |   2 +-
 drivers/net/hinic/hinic_pmd_ethdev.c          |   2 +-
 drivers/net/hns3/hns3_ethdev.c                |   2 +-
 drivers/net/hns3/hns3_rxtx.c                  |   2 +-
 drivers/net/i40e/i40e_ethdev.c                |   2 +-
 drivers/net/i40e/i40e_ethdev_vf.c             |   2 +-
 drivers/net/i40e/i40e_vf_representor.c        |   2 +-
 drivers/net/igc/igc_ethdev.c                  |   2 +-
 drivers/net/ionic/ionic.h                     |   2 +-
 drivers/net/ionic/ionic_ethdev.c              |   2 +-
 drivers/net/ipn3ke/ipn3ke_ethdev.c            |   2 +-
 drivers/net/ipn3ke/ipn3ke_representor.c       |   2 +-
 drivers/net/ipn3ke/ipn3ke_tm.c                |   2 +-
 drivers/net/ixgbe/ixgbe_ethdev.c              |   2 +-
 drivers/net/ixgbe/ixgbe_ethdev.h              |   2 +-
 drivers/net/mlx4/mlx4_ethdev.c                |   2 +-
 drivers/net/mlx5/linux/mlx5_ethdev_os.c       |   2 +-
 drivers/net/mlx5/linux/mlx5_os.c              |   2 +-
 drivers/net/mlx5/mlx5.c                       |   2 +-
 drivers/net/mlx5/mlx5_ethdev.c                |   2 +-
 drivers/net/mlx5/mlx5_txq.c                   |   2 +-
 drivers/net/netvsc/hn_vf.c                    |   2 +-
 drivers/net/octeontx/base/octeontx_pkivf.c    |   2 +-
 drivers/net/octeontx/base/octeontx_pkovf.c    |   2 +-
 drivers/net/octeontx2/otx2_ethdev_irq.c       |   2 +-
 drivers/net/qede/base/bcm_osal.h              |   2 +-
 drivers/net/sfc/sfc.h                         |   2 +-
 drivers/net/sfc/sfc_ethdev.c                  |   2 +-
 drivers/net/sfc/sfc_sriov.c                   |   2 +-
 drivers/net/thunderx/nicvf_ethdev.c           |   2 +-
 drivers/net/txgbe/txgbe_ethdev.h              |   2 +-
 drivers/net/txgbe/txgbe_flow.c                |   2 +-
 drivers/net/txgbe/txgbe_pf.c                  |   2 +-
 drivers/net/virtio/virtio_pci.h               |   2 +-
 drivers/net/virtio/virtio_pci_ethdev.c        |   2 +-
 drivers/net/vmxnet3/vmxnet3_ethdev.c          |   2 +-
 drivers/raw/cnxk_bphy/cnxk_bphy.c             |   2 +-
 drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c         |   2 +-
 drivers/raw/cnxk_bphy/cnxk_bphy_irq.c         |   2 +-
 drivers/raw/cnxk_bphy/cnxk_bphy_irq.h         |   2 +-
 drivers/raw/ifpga/ifpga_rawdev.c              |   2 +-
 drivers/raw/ifpga/rte_pmd_ifpga.c             |   2 +-
 drivers/raw/ioat/idxd_pci.c                   |   2 +-
 drivers/raw/ioat/ioat_rawdev.c                |   2 +-
 drivers/raw/ntb/ntb.c                         |   2 +-
 drivers/raw/ntb/ntb_hw_intel.c                |   2 +-
 drivers/raw/octeontx2_dma/otx2_dpi_rawdev.c   |   2 +-
 drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c     |   2 +-
 drivers/raw/octeontx2_ep/otx2_ep_rawdev.c     |   2 +-
 drivers/regex/mlx5/mlx5_regex.c               |   2 +-
 drivers/regex/mlx5/mlx5_regex_fastpath.c      |   2 +-
 drivers/vdpa/ifc/base/ifcvf_osdep.h           |   2 +-
 drivers/vdpa/ifc/ifcvf_vdpa.c                 |   2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |   2 +-
 lib/ethdev/ethdev_pci.h                       |   2 +-
 lib/eventdev/eventdev_pmd_pci.h               |   2 +-
 126 files changed, 546 insertions(+), 508 deletions(-)
 create mode 100644 drivers/bus/pci/pci_driver.h

diff --git a/app/test/virtual_pmd.c b/app/test/virtual_pmd.c
index 7036f401ed..555f2969ab 100644
--- a/app/test/virtual_pmd.c
+++ b/app/test/virtual_pmd.c
@@ -6,7 +6,7 @@
 #include <rte_ethdev.h>
 #include <ethdev_driver.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_malloc.h>
 #include <rte_memcpy.h>
 #include <rte_memory.h>
diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst
index ce3f554e10..864069f9c2 100644
--- a/doc/guides/rel_notes/release_21_11.rst
+++ b/doc/guides/rel_notes/release_21_11.rst
@@ -153,6 +153,8 @@ ABI Changes
    Also, make sure to start the actual text at the margin.
    =======================================================
 
+* pci: Removed all ABIs defined in rte_bus_pci.h except the function
+  ``rte_pci_dump()``.
 
 Known Issues
 ------------
diff --git a/drivers/baseband/acc100/rte_acc100_pmd.c b/drivers/baseband/acc100/rte_acc100_pmd.c
index 68ba523ea9..72734784c9 100644
--- a/drivers/baseband/acc100/rte_acc100_pmd.c
+++ b/drivers/baseband/acc100/rte_acc100_pmd.c
@@ -14,7 +14,7 @@
 #include <rte_branch_prediction.h>
 #include <rte_hexdump.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #ifdef RTE_BBDEV_OFFLOAD_COST
 #include <rte_cycles.h>
 #endif
diff --git a/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c b/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c
index 6485cc824a..31cb7e5605 100644
--- a/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c
+++ b/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c
@@ -11,7 +11,7 @@
 #include <rte_mempool.h>
 #include <rte_errno.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_byteorder.h>
 #ifdef RTE_BBDEV_OFFLOAD_COST
 #include <rte_cycles.h>
diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c
index 350c4248eb..0dc1417dde 100644
--- a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c
+++ b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c
@@ -11,7 +11,7 @@
 #include <rte_mempool.h>
 #include <rte_errno.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_byteorder.h>
 #ifdef RTE_BBDEV_OFFLOAD_COST
 #include <rte_cycles.h>
diff --git a/drivers/bus/pci/bsd/pci.c b/drivers/bus/pci/bsd/pci.c
index d189bff311..b7d3a8df33 100644
--- a/drivers/bus/pci/bsd/pci.c
+++ b/drivers/bus/pci/bsd/pci.c
@@ -28,7 +28,6 @@
 #include <rte_interrupts.h>
 #include <rte_log.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
 #include <rte_common.h>
 #include <rte_launch.h>
 #include <rte_memory.h>
diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index 4d261b55ee..8f91e0233c 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -8,7 +8,6 @@
 #include <rte_log.h>
 #include <rte_bus.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
 #include <rte_malloc.h>
 #include <rte_devargs.h>
 #include <rte_memcpy.h>
diff --git a/drivers/bus/pci/linux/pci_uio.c b/drivers/bus/pci/linux/pci_uio.c
index 39ebeac2a0..de028e4874 100644
--- a/drivers/bus/pci/linux/pci_uio.c
+++ b/drivers/bus/pci/linux/pci_uio.c
@@ -19,7 +19,6 @@
 #include <rte_string_fns.h>
 #include <rte_log.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
 #include <rte_common.h>
 #include <rte_malloc.h>
 
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index a024269140..2ccd0fa101 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -13,7 +13,6 @@
 
 #include <rte_log.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
 #include <rte_eal_paging.h>
 #include <rte_malloc.h>
 #include <rte_vfio.h>
diff --git a/drivers/bus/pci/meson.build b/drivers/bus/pci/meson.build
index 81c7e94c00..33c09e2622 100644
--- a/drivers/bus/pci/meson.build
+++ b/drivers/bus/pci/meson.build
@@ -29,4 +29,8 @@ if is_windows
     includes += include_directories('windows')
 endif
 
+driver_sdk_headers += files(
+        'pci_driver.h',
+)
+
 deps += ['kvargs']
diff --git a/drivers/bus/pci/pci_common_uio.c b/drivers/bus/pci/pci_common_uio.c
index 318f9a1d55..00ef86c1dd 100644
--- a/drivers/bus/pci/pci_common_uio.c
+++ b/drivers/bus/pci/pci_common_uio.c
@@ -11,7 +11,6 @@
 
 #include <rte_eal.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
 #include <rte_tailq.h>
 #include <rte_log.h>
 #include <rte_malloc.h>
diff --git a/drivers/bus/pci/pci_driver.h b/drivers/bus/pci/pci_driver.h
new file mode 100644
index 0000000000..7a913d54c5
--- /dev/null
+++ b/drivers/bus/pci/pci_driver.h
@@ -0,0 +1,402 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2015 Intel Corporation.
+ * Copyright 2013-2014 6WIND S.A.
+ */
+
+#ifndef _PCI_DRIVER_H_
+#define _PCI_DRIVER_H_
+
+/**
+ * @file
+ * PCI device & driver interface
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <limits.h>
+#include <errno.h>
+#include <sys/queue.h>
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_debug.h>
+#include <rte_interrupts.h>
+#include <rte_dev.h>
+#include <rte_bus.h>
+#include <rte_pci.h>
+
+/** Pathname of PCI devices directory. */
+__rte_internal
+const char *rte_pci_get_sysfs_path(void);
+
+/* Forward declarations */
+struct rte_pci_device;
+struct rte_pci_driver;
+
+/** List of PCI devices */
+TAILQ_HEAD(rte_pci_device_list, rte_pci_device);
+/** List of PCI drivers */
+TAILQ_HEAD(rte_pci_driver_list, rte_pci_driver);
+
+/* PCI Bus iterators */
+#define FOREACH_DEVICE_ON_PCIBUS(p)	\
+		TAILQ_FOREACH(p, &(rte_pci_bus.device_list), next)
+
+#define FOREACH_DRIVER_ON_PCIBUS(p)	\
+		TAILQ_FOREACH(p, &(rte_pci_bus.driver_list), next)
+
+struct rte_devargs;
+
+enum rte_pci_kernel_driver {
+	RTE_PCI_KDRV_UNKNOWN = 0,  /* may be misc UIO or bifurcated driver */
+	RTE_PCI_KDRV_IGB_UIO,      /* igb_uio for Linux */
+	RTE_PCI_KDRV_VFIO,         /* VFIO for Linux */
+	RTE_PCI_KDRV_UIO_GENERIC,  /* uio_pci_generic for Linux */
+	RTE_PCI_KDRV_NIC_UIO,      /* nic_uio for FreeBSD */
+	RTE_PCI_KDRV_NONE,         /* no attached driver */
+	RTE_PCI_KDRV_NET_UIO,      /* NetUIO for Windows */
+};
+
+/**
+ * A structure describing a PCI device.
+ */
+struct rte_pci_device {
+	TAILQ_ENTRY(rte_pci_device) next;   /**< Next probed PCI device. */
+	struct rte_device device;           /**< Inherit core device */
+	struct rte_pci_addr addr;           /**< PCI location. */
+	struct rte_pci_id id;               /**< PCI ID. */
+	struct rte_mem_resource mem_resource[PCI_MAX_RESOURCE];
+					    /**< PCI Memory Resource */
+	struct rte_intr_handle intr_handle; /**< Interrupt handle */
+	struct rte_pci_driver *driver;      /**< PCI driver used in probing */
+	uint16_t max_vfs;                   /**< sriov enable if not zero */
+	enum rte_pci_kernel_driver kdrv;    /**< Kernel driver passthrough */
+	char name[PCI_PRI_STR_SIZE+1];      /**< PCI location (ASCII) */
+	struct rte_intr_handle vfio_req_intr_handle;
+				/**< Handler of VFIO request interrupt */
+};
+
+/**
+ * @internal
+ * Helper macro for drivers that need to convert to struct rte_pci_device.
+ */
+#define RTE_DEV_TO_PCI(ptr) container_of(ptr, struct rte_pci_device, device)
+
+#define RTE_DEV_TO_PCI_CONST(ptr) \
+	container_of(ptr, const struct rte_pci_device, device)
+
+#define RTE_ETH_DEV_TO_PCI(eth_dev)	RTE_DEV_TO_PCI((eth_dev)->device)
+
+#ifdef __cplusplus
+/** C++ macro used to help building up tables of device IDs */
+#define RTE_PCI_DEVICE(vend, dev) \
+	RTE_CLASS_ANY_ID,         \
+	(vend),                   \
+	(dev),                    \
+	RTE_PCI_ANY_ID,           \
+	RTE_PCI_ANY_ID
+#else
+/** Macro used to help building up tables of device IDs */
+#define RTE_PCI_DEVICE(vend, dev)          \
+	.class_id = RTE_CLASS_ANY_ID,      \
+	.vendor_id = (vend),               \
+	.device_id = (dev),                \
+	.subsystem_vendor_id = RTE_PCI_ANY_ID, \
+	.subsystem_device_id = RTE_PCI_ANY_ID
+#endif
+
+/**
+ * Initialisation function for the driver called during PCI probing.
+ */
+typedef int (rte_pci_probe_t)(struct rte_pci_driver *, struct rte_pci_device *);
+
+/**
+ * Uninitialisation function for the driver called during hotplugging.
+ */
+typedef int (rte_pci_remove_t)(struct rte_pci_device *);
+
+/**
+ * Driver-specific DMA mapping. After a successful call the device
+ * will be able to read/write from/to this segment.
+ *
+ * @param dev
+ *   Pointer to the PCI device.
+ * @param addr
+ *   Starting virtual address of memory to be mapped.
+ * @param iova
+ *   Starting IOVA address of memory to be mapped.
+ * @param len
+ *   Length of memory segment being mapped.
+ * @return
+ *   - 0 On success.
+ *   - Negative value and rte_errno is set otherwise.
+ */
+typedef int (pci_dma_map_t)(struct rte_pci_device *dev, void *addr,
+			    uint64_t iova, size_t len);
+
+/**
+ * Driver-specific DMA un-mapping. After a successful call the device
+ * will not be able to read/write from/to this segment.
+ *
+ * @param dev
+ *   Pointer to the PCI device.
+ * @param addr
+ *   Starting virtual address of memory to be unmapped.
+ * @param iova
+ *   Starting IOVA address of memory to be unmapped.
+ * @param len
+ *   Length of memory segment being unmapped.
+ * @return
+ *   - 0 On success.
+ *   - Negative value and rte_errno is set otherwise.
+ */
+typedef int (pci_dma_unmap_t)(struct rte_pci_device *dev, void *addr,
+			      uint64_t iova, size_t len);
+
+/**
+ * A structure describing a PCI driver.
+ */
+struct rte_pci_driver {
+	TAILQ_ENTRY(rte_pci_driver) next;  /**< Next in list. */
+	struct rte_driver driver;          /**< Inherit core driver. */
+	struct rte_pci_bus *bus;           /**< PCI bus reference. */
+	rte_pci_probe_t *probe;            /**< Device probe function. */
+	rte_pci_remove_t *remove;          /**< Device remove function. */
+	pci_dma_map_t *dma_map;		   /**< device dma map function. */
+	pci_dma_unmap_t *dma_unmap;	   /**< device dma unmap function. */
+	const struct rte_pci_id *id_table; /**< ID table, NULL terminated. */
+	uint32_t drv_flags;                /**< Flags RTE_PCI_DRV_*. */
+};
+
+/**
+ * Structure describing the PCI bus
+ */
+struct rte_pci_bus {
+	struct rte_bus bus;               /**< Inherit the generic class */
+	struct rte_pci_device_list device_list;  /**< List of PCI devices */
+	struct rte_pci_driver_list driver_list;  /**< List of PCI drivers */
+};
+
+/** Device needs PCI BAR mapping (done with either IGB_UIO or VFIO) */
+#define RTE_PCI_DRV_NEED_MAPPING 0x0001
+/** Device needs PCI BAR mapping with enabled write combining (wc) */
+#define RTE_PCI_DRV_WC_ACTIVATE 0x0002
+/** Device already probed can be probed again to check for new ports. */
+#define RTE_PCI_DRV_PROBE_AGAIN 0x0004
+/** Device driver supports link state interrupt */
+#define RTE_PCI_DRV_INTR_LSC	0x0008
+/** Device driver supports device removal interrupt */
+#define RTE_PCI_DRV_INTR_RMV 0x0010
+/** Device driver needs to keep mapped resources if unsupported dev detected */
+#define RTE_PCI_DRV_KEEP_MAPPED_RES 0x0020
+/** Device driver needs IOVA as VA and cannot work with IOVA as PA */
+#define RTE_PCI_DRV_NEED_IOVA_AS_VA 0x0040
+
+/**
+ * Map the PCI device resources in user space virtual memory address
+ *
+ * Note that driver should not call this function when flag
+ * RTE_PCI_DRV_NEED_MAPPING is set, as EAL will do that for
+ * you when it's on.
+ *
+ * @param dev
+ *   A pointer to a rte_pci_device structure describing the device
+ *   to use
+ *
+ * @return
+ *   0 on success, negative on error and positive if no driver
+ *   is found for the device.
+ */
+__rte_internal
+int rte_pci_map_device(struct rte_pci_device *dev);
+
+/**
+ * Unmap this device
+ *
+ * @param dev
+ *   A pointer to a rte_pci_device structure describing the device
+ *   to use
+ */
+__rte_internal
+void rte_pci_unmap_device(struct rte_pci_device *dev);
+
+/**
+ * Find device's extended PCI capability.
+ *
+ *  @param dev
+ *    A pointer to rte_pci_device structure.
+ *
+ *  @param cap
+ *    Extended capability to be found, which can be any from
+ *    RTE_PCI_EXT_CAP_ID_*, defined in librte_pci.
+ *
+ *  @return
+ *  > 0: The offset of the next matching extended capability structure
+ *       within the device's PCI configuration space.
+ *  < 0: An error in PCI config space read.
+ *  = 0: Device does not support it.
+ */
+__rte_internal
+off_t rte_pci_find_ext_capability(struct rte_pci_device *dev, uint32_t cap);
+
+/**
+ * Enables/Disables Bus Master for device's PCI command register.
+ *
+ *  @param dev
+ *    A pointer to rte_pci_device structure.
+ *  @param enable
+ *    Enable or disable Bus Master.
+ *
+ *  @return
+ *  0 on success, -1 on error in PCI config space read/write.
+ */
+__rte_internal
+int rte_pci_set_bus_master(struct rte_pci_device *dev, bool enable);
+
+/**
+ * Register a PCI driver.
+ *
+ * @param driver
+ *   A pointer to a rte_pci_driver structure describing the driver
+ *   to be registered.
+ */
+__rte_internal
+void rte_pci_register(struct rte_pci_driver *driver);
+
+/** Helper for PCI device registration from driver (eth, crypto) instance */
+#define RTE_PMD_REGISTER_PCI(nm, pci_drv) \
+RTE_INIT(pciinitfn_ ##nm) \
+{\
+	(pci_drv).driver.name = RTE_STR(nm);\
+	rte_pci_register(&pci_drv); \
+} \
+RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+/**
+ * Unregister a PCI driver.
+ *
+ * @param driver
+ *   A pointer to a rte_pci_driver structure describing the driver
+ *   to be unregistered.
+ */
+__rte_internal
+void rte_pci_unregister(struct rte_pci_driver *driver);
+
+/**
+ * Read PCI config space.
+ *
+ * @param device
+ *   A pointer to a rte_pci_device structure describing the device
+ *   to use
+ * @param buf
+ *   A data buffer where the bytes should be read into
+ * @param len
+ *   The length of the data buffer.
+ * @param offset
+ *   The offset into PCI config space
+ * @return
+ *  Number of bytes read on success, negative on error.
+ */
+__rte_internal
+int rte_pci_read_config(const struct rte_pci_device *device,
+		void *buf, size_t len, off_t offset);
+
+/**
+ * Write PCI config space.
+ *
+ * @param device
+ *   A pointer to a rte_pci_device structure describing the device
+ *   to use
+ * @param buf
+ *   A data buffer containing the bytes should be written
+ * @param len
+ *   The length of the data buffer.
+ * @param offset
+ *   The offset into PCI config space
+ */
+__rte_internal
+int rte_pci_write_config(const struct rte_pci_device *device,
+		const void *buf, size_t len, off_t offset);
+
+/**
+ * A structure used to access io resources for a pci device.
+ * rte_pci_ioport is arch, os, driver specific, and should not be used outside
+ * of pci ioport api.
+ */
+struct rte_pci_ioport {
+	struct rte_pci_device *dev;
+	uint64_t base;
+	uint64_t len; /* only filled for memory mapped ports */
+};
+
+/**
+ * Initialize a rte_pci_ioport object for a pci device io resource.
+ *
+ * This object is then used to gain access to those io resources (see below).
+ *
+ * @param dev
+ *   A pointer to a rte_pci_device structure describing the device
+ *   to use.
+ * @param bar
+ *   Index of the io pci resource we want to access.
+ * @param p
+ *   The rte_pci_ioport object to be initialized.
+ * @return
+ *  0 on success, negative on error.
+ */
+__rte_internal
+int rte_pci_ioport_map(struct rte_pci_device *dev, int bar,
+		struct rte_pci_ioport *p);
+
+/**
+ * Release any resources used in a rte_pci_ioport object.
+ *
+ * @param p
+ *   The rte_pci_ioport object to be uninitialized.
+ * @return
+ *  0 on success, negative on error.
+ */
+__rte_internal
+int rte_pci_ioport_unmap(struct rte_pci_ioport *p);
+
+/**
+ * Read from a io pci resource.
+ *
+ * @param p
+ *   The rte_pci_ioport object from which we want to read.
+ * @param data
+ *   A data buffer where the bytes should be read into
+ * @param len
+ *   The length of the data buffer.
+ * @param offset
+ *   The offset into the pci io resource.
+ */
+__rte_internal
+void rte_pci_ioport_read(struct rte_pci_ioport *p,
+		void *data, size_t len, off_t offset);
+
+/**
+ * Write to a io pci resource.
+ *
+ * @param p
+ *   The rte_pci_ioport object to which we want to write.
+ * @param data
+ *   A data buffer where the bytes should be read into
+ * @param len
+ *   The length of the data buffer.
+ * @param offset
+ *   The offset into the pci io resource.
+ */
+__rte_internal
+void rte_pci_ioport_write(struct rte_pci_ioport *p,
+		const void *data, size_t len, off_t offset);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _PCI_DRIVER_H_ */
diff --git a/drivers/bus/pci/pci_params.c b/drivers/bus/pci/pci_params.c
index 691b5ea018..3f4f94cc72 100644
--- a/drivers/bus/pci/pci_params.c
+++ b/drivers/bus/pci/pci_params.c
@@ -3,7 +3,6 @@
  */
 
 #include <rte_bus.h>
-#include <rte_bus_pci.h>
 #include <rte_dev.h>
 #include <rte_errno.h>
 #include <rte_kvargs.h>
diff --git a/drivers/bus/pci/private.h b/drivers/bus/pci/private.h
index 0fbef8e1d8..dd42baf516 100644
--- a/drivers/bus/pci/private.h
+++ b/drivers/bus/pci/private.h
@@ -8,10 +8,11 @@
 #include <stdbool.h>
 #include <stdio.h>
 
-#include <rte_bus_pci.h>
 #include <rte_os_shim.h>
 #include <rte_pci.h>
 
+#include "pci_driver.h"
+
 extern struct rte_pci_bus rte_pci_bus;
 
 struct rte_pci_driver;
diff --git a/drivers/bus/pci/rte_bus_pci.h b/drivers/bus/pci/rte_bus_pci.h
index 21d9dd4289..2fb35801f8 100644
--- a/drivers/bus/pci/rte_bus_pci.h
+++ b/drivers/bus/pci/rte_bus_pci.h
@@ -1,225 +1,17 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2015 Intel Corporation.
- * Copyright 2013-2014 6WIND S.A.
+ * Copyright(c) 2021 Intel Corporation.
  */
 
 #ifndef _RTE_BUS_PCI_H_
 #define _RTE_BUS_PCI_H_
 
-/**
- * @file
- * PCI device & driver interface
- */
-
 #ifdef __cplusplus
 extern "C" {
 #endif
 
-#include <stdio.h>
-#include <stdlib.h>
-#include <limits.h>
-#include <errno.h>
-#include <sys/queue.h>
 #include <stdint.h>
-#include <inttypes.h>
-
-#include <rte_debug.h>
-#include <rte_interrupts.h>
-#include <rte_dev.h>
-#include <rte_bus.h>
-#include <rte_pci.h>
-
-/** Pathname of PCI devices directory. */
-const char *rte_pci_get_sysfs_path(void);
-
-/* Forward declarations */
-struct rte_pci_device;
-struct rte_pci_driver;
-
-/** List of PCI devices */
-TAILQ_HEAD(rte_pci_device_list, rte_pci_device);
-/** List of PCI drivers */
-TAILQ_HEAD(rte_pci_driver_list, rte_pci_driver);
-
-/* PCI Bus iterators */
-#define FOREACH_DEVICE_ON_PCIBUS(p)	\
-		TAILQ_FOREACH(p, &(rte_pci_bus.device_list), next)
 
-#define FOREACH_DRIVER_ON_PCIBUS(p)	\
-		TAILQ_FOREACH(p, &(rte_pci_bus.driver_list), next)
-
-struct rte_devargs;
-
-enum rte_pci_kernel_driver {
-	RTE_PCI_KDRV_UNKNOWN = 0,  /* may be misc UIO or bifurcated driver */
-	RTE_PCI_KDRV_IGB_UIO,      /* igb_uio for Linux */
-	RTE_PCI_KDRV_VFIO,         /* VFIO for Linux */
-	RTE_PCI_KDRV_UIO_GENERIC,  /* uio_pci_generic for Linux */
-	RTE_PCI_KDRV_NIC_UIO,      /* nic_uio for FreeBSD */
-	RTE_PCI_KDRV_NONE,         /* no attached driver */
-	RTE_PCI_KDRV_NET_UIO,      /* NetUIO for Windows */
-};
-
-/**
- * A structure describing a PCI device.
- */
-struct rte_pci_device {
-	TAILQ_ENTRY(rte_pci_device) next;   /**< Next probed PCI device. */
-	struct rte_device device;           /**< Inherit core device */
-	struct rte_pci_addr addr;           /**< PCI location. */
-	struct rte_pci_id id;               /**< PCI ID. */
-	struct rte_mem_resource mem_resource[PCI_MAX_RESOURCE];
-					    /**< PCI Memory Resource */
-	struct rte_intr_handle intr_handle; /**< Interrupt handle */
-	struct rte_pci_driver *driver;      /**< PCI driver used in probing */
-	uint16_t max_vfs;                   /**< sriov enable if not zero */
-	enum rte_pci_kernel_driver kdrv;    /**< Kernel driver passthrough */
-	char name[PCI_PRI_STR_SIZE+1];      /**< PCI location (ASCII) */
-	struct rte_intr_handle vfio_req_intr_handle;
-				/**< Handler of VFIO request interrupt */
-};
-
-/**
- * @internal
- * Helper macro for drivers that need to convert to struct rte_pci_device.
- */
-#define RTE_DEV_TO_PCI(ptr) container_of(ptr, struct rte_pci_device, device)
-
-#define RTE_DEV_TO_PCI_CONST(ptr) \
-	container_of(ptr, const struct rte_pci_device, device)
-
-#define RTE_ETH_DEV_TO_PCI(eth_dev)	RTE_DEV_TO_PCI((eth_dev)->device)
-
-#ifdef __cplusplus
-/** C++ macro used to help building up tables of device IDs */
-#define RTE_PCI_DEVICE(vend, dev) \
-	RTE_CLASS_ANY_ID,         \
-	(vend),                   \
-	(dev),                    \
-	RTE_PCI_ANY_ID,           \
-	RTE_PCI_ANY_ID
-#else
-/** Macro used to help building up tables of device IDs */
-#define RTE_PCI_DEVICE(vend, dev)          \
-	.class_id = RTE_CLASS_ANY_ID,      \
-	.vendor_id = (vend),               \
-	.device_id = (dev),                \
-	.subsystem_vendor_id = RTE_PCI_ANY_ID, \
-	.subsystem_device_id = RTE_PCI_ANY_ID
-#endif
-
-/**
- * Initialisation function for the driver called during PCI probing.
- */
-typedef int (rte_pci_probe_t)(struct rte_pci_driver *, struct rte_pci_device *);
-
-/**
- * Uninitialisation function for the driver called during hotplugging.
- */
-typedef int (rte_pci_remove_t)(struct rte_pci_device *);
-
-/**
- * Driver-specific DMA mapping. After a successful call the device
- * will be able to read/write from/to this segment.
- *
- * @param dev
- *   Pointer to the PCI device.
- * @param addr
- *   Starting virtual address of memory to be mapped.
- * @param iova
- *   Starting IOVA address of memory to be mapped.
- * @param len
- *   Length of memory segment being mapped.
- * @return
- *   - 0 On success.
- *   - Negative value and rte_errno is set otherwise.
- */
-typedef int (pci_dma_map_t)(struct rte_pci_device *dev, void *addr,
-			    uint64_t iova, size_t len);
-
-/**
- * Driver-specific DMA un-mapping. After a successful call the device
- * will not be able to read/write from/to this segment.
- *
- * @param dev
- *   Pointer to the PCI device.
- * @param addr
- *   Starting virtual address of memory to be unmapped.
- * @param iova
- *   Starting IOVA address of memory to be unmapped.
- * @param len
- *   Length of memory segment being unmapped.
- * @return
- *   - 0 On success.
- *   - Negative value and rte_errno is set otherwise.
- */
-typedef int (pci_dma_unmap_t)(struct rte_pci_device *dev, void *addr,
-			      uint64_t iova, size_t len);
-
-/**
- * A structure describing a PCI driver.
- */
-struct rte_pci_driver {
-	TAILQ_ENTRY(rte_pci_driver) next;  /**< Next in list. */
-	struct rte_driver driver;          /**< Inherit core driver. */
-	struct rte_pci_bus *bus;           /**< PCI bus reference. */
-	rte_pci_probe_t *probe;            /**< Device probe function. */
-	rte_pci_remove_t *remove;          /**< Device remove function. */
-	pci_dma_map_t *dma_map;		   /**< device dma map function. */
-	pci_dma_unmap_t *dma_unmap;	   /**< device dma unmap function. */
-	const struct rte_pci_id *id_table; /**< ID table, NULL terminated. */
-	uint32_t drv_flags;                /**< Flags RTE_PCI_DRV_*. */
-};
-
-/**
- * Structure describing the PCI bus
- */
-struct rte_pci_bus {
-	struct rte_bus bus;               /**< Inherit the generic class */
-	struct rte_pci_device_list device_list;  /**< List of PCI devices */
-	struct rte_pci_driver_list driver_list;  /**< List of PCI drivers */
-};
-
-/** Device needs PCI BAR mapping (done with either IGB_UIO or VFIO) */
-#define RTE_PCI_DRV_NEED_MAPPING 0x0001
-/** Device needs PCI BAR mapping with enabled write combining (wc) */
-#define RTE_PCI_DRV_WC_ACTIVATE 0x0002
-/** Device already probed can be probed again to check for new ports. */
-#define RTE_PCI_DRV_PROBE_AGAIN 0x0004
-/** Device driver supports link state interrupt */
-#define RTE_PCI_DRV_INTR_LSC	0x0008
-/** Device driver supports device removal interrupt */
-#define RTE_PCI_DRV_INTR_RMV 0x0010
-/** Device driver needs to keep mapped resources if unsupported dev detected */
-#define RTE_PCI_DRV_KEEP_MAPPED_RES 0x0020
-/** Device driver needs IOVA as VA and cannot work with IOVA as PA */
-#define RTE_PCI_DRV_NEED_IOVA_AS_VA 0x0040
-
-/**
- * Map the PCI device resources in user space virtual memory address
- *
- * Note that driver should not call this function when flag
- * RTE_PCI_DRV_NEED_MAPPING is set, as EAL will do that for
- * you when it's on.
- *
- * @param dev
- *   A pointer to a rte_pci_device structure describing the device
- *   to use
- *
- * @return
- *   0 on success, negative on error and positive if no driver
- *   is found for the device.
- */
-int rte_pci_map_device(struct rte_pci_device *dev);
-
-/**
- * Unmap this device
- *
- * @param dev
- *   A pointer to a rte_pci_device structure describing the device
- *   to use
- */
-void rte_pci_unmap_device(struct rte_pci_device *dev);
+#include <rte_compat.h>
 
 /**
  * Dump the content of the PCI bus.
@@ -229,169 +21,6 @@ void rte_pci_unmap_device(struct rte_pci_device *dev);
  */
 void rte_pci_dump(FILE *f);
 
-/**
- * Find device's extended PCI capability.
- *
- *  @param dev
- *    A pointer to rte_pci_device structure.
- *
- *  @param cap
- *    Extended capability to be found, which can be any from
- *    RTE_PCI_EXT_CAP_ID_*, defined in librte_pci.
- *
- *  @return
- *  > 0: The offset of the next matching extended capability structure
- *       within the device's PCI configuration space.
- *  < 0: An error in PCI config space read.
- *  = 0: Device does not support it.
- */
-__rte_experimental
-off_t rte_pci_find_ext_capability(struct rte_pci_device *dev, uint32_t cap);
-
-/**
- * Enables/Disables Bus Master for device's PCI command register.
- *
- *  @param dev
- *    A pointer to rte_pci_device structure.
- *  @param enable
- *    Enable or disable Bus Master.
- *
- *  @return
- *  0 on success, -1 on error in PCI config space read/write.
- */
-__rte_experimental
-int rte_pci_set_bus_master(struct rte_pci_device *dev, bool enable);
-
-/**
- * Register a PCI driver.
- *
- * @param driver
- *   A pointer to a rte_pci_driver structure describing the driver
- *   to be registered.
- */
-void rte_pci_register(struct rte_pci_driver *driver);
-
-/** Helper for PCI device registration from driver (eth, crypto) instance */
-#define RTE_PMD_REGISTER_PCI(nm, pci_drv) \
-RTE_INIT(pciinitfn_ ##nm) \
-{\
-	(pci_drv).driver.name = RTE_STR(nm);\
-	rte_pci_register(&pci_drv); \
-} \
-RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
-
-/**
- * Unregister a PCI driver.
- *
- * @param driver
- *   A pointer to a rte_pci_driver structure describing the driver
- *   to be unregistered.
- */
-void rte_pci_unregister(struct rte_pci_driver *driver);
-
-/**
- * Read PCI config space.
- *
- * @param device
- *   A pointer to a rte_pci_device structure describing the device
- *   to use
- * @param buf
- *   A data buffer where the bytes should be read into
- * @param len
- *   The length of the data buffer.
- * @param offset
- *   The offset into PCI config space
- * @return
- *  Number of bytes read on success, negative on error.
- */
-int rte_pci_read_config(const struct rte_pci_device *device,
-		void *buf, size_t len, off_t offset);
-
-/**
- * Write PCI config space.
- *
- * @param device
- *   A pointer to a rte_pci_device structure describing the device
- *   to use
- * @param buf
- *   A data buffer containing the bytes should be written
- * @param len
- *   The length of the data buffer.
- * @param offset
- *   The offset into PCI config space
- */
-int rte_pci_write_config(const struct rte_pci_device *device,
-		const void *buf, size_t len, off_t offset);
-
-/**
- * A structure used to access io resources for a pci device.
- * rte_pci_ioport is arch, os, driver specific, and should not be used outside
- * of pci ioport api.
- */
-struct rte_pci_ioport {
-	struct rte_pci_device *dev;
-	uint64_t base;
-	uint64_t len; /* only filled for memory mapped ports */
-};
-
-/**
- * Initialize a rte_pci_ioport object for a pci device io resource.
- *
- * This object is then used to gain access to those io resources (see below).
- *
- * @param dev
- *   A pointer to a rte_pci_device structure describing the device
- *   to use.
- * @param bar
- *   Index of the io pci resource we want to access.
- * @param p
- *   The rte_pci_ioport object to be initialized.
- * @return
- *  0 on success, negative on error.
- */
-int rte_pci_ioport_map(struct rte_pci_device *dev, int bar,
-		struct rte_pci_ioport *p);
-
-/**
- * Release any resources used in a rte_pci_ioport object.
- *
- * @param p
- *   The rte_pci_ioport object to be uninitialized.
- * @return
- *  0 on success, negative on error.
- */
-int rte_pci_ioport_unmap(struct rte_pci_ioport *p);
-
-/**
- * Read from a io pci resource.
- *
- * @param p
- *   The rte_pci_ioport object from which we want to read.
- * @param data
- *   A data buffer where the bytes should be read into
- * @param len
- *   The length of the data buffer.
- * @param offset
- *   The offset into the pci io resource.
- */
-void rte_pci_ioport_read(struct rte_pci_ioport *p,
-		void *data, size_t len, off_t offset);
-
-/**
- * Write to a io pci resource.
- *
- * @param p
- *   The rte_pci_ioport object to which we want to write.
- * @param data
- *   A data buffer where the bytes should be read into
- * @param len
- *   The length of the data buffer.
- * @param offset
- *   The offset into the pci io resource.
- */
-void rte_pci_ioport_write(struct rte_pci_ioport *p,
-		const void *data, size_t len, off_t offset);
-
 /**
  * Read 4 bytes from PCI memory resource.
  *
diff --git a/drivers/bus/pci/version.map b/drivers/bus/pci/version.map
index 01ec836559..73cf881778 100644
--- a/drivers/bus/pci/version.map
+++ b/drivers/bus/pci/version.map
@@ -2,6 +2,22 @@ DPDK_22 {
 	global:
 
 	rte_pci_dump;
+
+	local: *;
+};
+
+EXPERIMENTAL {
+	global:
+
+	# added in 21.11
+	rte_pci_mem_rd32;
+	rte_pci_mem_wr32;
+};
+
+INTERNAL {
+	global:
+
+	rte_pci_find_ext_capability;
 	rte_pci_get_sysfs_path;
 	rte_pci_ioport_map;
 	rte_pci_ioport_read;
@@ -10,22 +26,8 @@ DPDK_22 {
 	rte_pci_map_device;
 	rte_pci_read_config;
 	rte_pci_register;
+	rte_pci_set_bus_master;
 	rte_pci_unmap_device;
 	rte_pci_unregister;
 	rte_pci_write_config;
-
-	local: *;
-};
-
-EXPERIMENTAL {
-	global:
-
-	rte_pci_find_ext_capability;
-
-	# added in 21.08
-	rte_pci_set_bus_master;
-
-	# added in 21.11
-	rte_pci_mem_rd32;
-	rte_pci_mem_wr32;
 };
diff --git a/drivers/common/cnxk/roc_platform.h b/drivers/common/cnxk/roc_platform.h
index 285b24b82d..fa8a6ef0af 100644
--- a/drivers/common/cnxk/roc_platform.h
+++ b/drivers/common/cnxk/roc_platform.h
@@ -7,7 +7,7 @@
 
 #include <rte_alarm.h>
 #include <rte_bitmap.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_byteorder.h>
 #include <rte_common.h>
 #include <rte_cycles.h>
diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c
index 9080bd3e87..65d795643b 100644
--- a/drivers/common/mlx5/linux/mlx5_common_verbs.c
+++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c
@@ -11,7 +11,7 @@
 #include <inttypes.h>
 
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_bus_auxiliary.h>
 
 #include "mlx5_common_utils.h"
diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c
index 8b38091d87..eaa9a0fff5 100644
--- a/drivers/common/mlx5/mlx5_common_pci.c
+++ b/drivers/common/mlx5/mlx5_common_pci.c
@@ -9,7 +9,7 @@
 #include <rte_errno.h>
 #include <rte_class.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "mlx5_common_log.h"
 #include "mlx5_common_private.h"
diff --git a/drivers/common/octeontx2/otx2_dev.h b/drivers/common/octeontx2/otx2_dev.h
index d5b2b0d9af..1f36f24cb0 100644
--- a/drivers/common/octeontx2/otx2_dev.h
+++ b/drivers/common/octeontx2/otx2_dev.h
@@ -5,7 +5,7 @@
 #ifndef _OTX2_DEV_H
 #define _OTX2_DEV_H
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "otx2_common.h"
 #include "otx2_irq.h"
diff --git a/drivers/common/octeontx2/otx2_sec_idev.c b/drivers/common/octeontx2/otx2_sec_idev.c
index 6e9643c383..a73c27c970 100644
--- a/drivers/common/octeontx2/otx2_sec_idev.c
+++ b/drivers/common/octeontx2/otx2_sec_idev.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_atomic.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ethdev.h>
 #include <rte_spinlock.h>
 
diff --git a/drivers/common/qat/qat_device.h b/drivers/common/qat/qat_device.h
index 228c057d1e..8d18365bae 100644
--- a/drivers/common/qat/qat_device.h
+++ b/drivers/common/qat/qat_device.h
@@ -4,7 +4,7 @@
 #ifndef _QAT_DEVICE_H_
 #define _QAT_DEVICE_H_
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "qat_common.h"
 #include "qat_logs.h"
diff --git a/drivers/common/qat/qat_qp.c b/drivers/common/qat/qat_qp.c
index 026ea5ee01..a964231d2c 100644
--- a/drivers/common/qat/qat_qp.c
+++ b/drivers/common/qat/qat_qp.c
@@ -8,7 +8,7 @@
 #include <rte_malloc.h>
 #include <rte_memzone.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_atomic.h>
 #include <rte_prefetch.h>
 
diff --git a/drivers/common/sfc_efx/sfc_efx.h b/drivers/common/sfc_efx/sfc_efx.h
index c16eca60f3..9829ed0e96 100644
--- a/drivers/common/sfc_efx/sfc_efx.h
+++ b/drivers/common/sfc_efx/sfc_efx.h
@@ -10,7 +10,7 @@
 #ifndef _SFC_EFX_H_
 #define _SFC_EFX_H_
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "efx.h"
 #include "efsys.h"
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index c5e0a83a8c..46b888c127 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -5,7 +5,7 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_spinlock.h>
 #include <rte_comp.h>
 #include <rte_compressdev.h>
diff --git a/drivers/compress/octeontx/otx_zip.h b/drivers/compress/octeontx/otx_zip.h
index e43f7f5c3e..56136ce734 100644
--- a/drivers/compress/octeontx/otx_zip.h
+++ b/drivers/compress/octeontx/otx_zip.h
@@ -7,7 +7,7 @@
 
 #include <unistd.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_comp.h>
 #include <rte_compressdev.h>
 #include <rte_compressdev_pmd.h>
diff --git a/drivers/compress/qat/qat_comp.c b/drivers/compress/qat/qat_comp.c
index 7ac25a3b4c..6fa82ad9a8 100644
--- a/drivers/compress/qat/qat_comp.c
+++ b/drivers/compress/qat/qat_comp.c
@@ -6,7 +6,7 @@
 #include <rte_mbuf.h>
 #include <rte_hexdump.h>
 #include <rte_comp.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_byteorder.h>
 #include <rte_memcpy.h>
 #include <rte_common.h>
diff --git a/drivers/crypto/ccp/ccp_dev.h b/drivers/crypto/ccp/ccp_dev.h
index ca5145c278..eb8cd94716 100644
--- a/drivers/crypto/ccp/ccp_dev.h
+++ b/drivers/crypto/ccp/ccp_dev.h
@@ -10,7 +10,7 @@
 #include <stdint.h>
 #include <string.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_atomic.h>
 #include <rte_byteorder.h>
 #include <rte_io.h>
diff --git a/drivers/crypto/ccp/ccp_pci.h b/drivers/crypto/ccp/ccp_pci.h
index 7ed3bac406..c8ba9f8789 100644
--- a/drivers/crypto/ccp/ccp_pci.h
+++ b/drivers/crypto/ccp/ccp_pci.h
@@ -7,7 +7,7 @@
 
 #include <stdint.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #define SYSFS_PCI_DEVICES "/sys/bus/pci/devices"
 #define PROC_MODULES "/proc/modules"
diff --git a/drivers/crypto/ccp/rte_ccp_pmd.c b/drivers/crypto/ccp/rte_ccp_pmd.c
index a54d81de46..1dd8edceba 100644
--- a/drivers/crypto/ccp/rte_ccp_pmd.c
+++ b/drivers/crypto/ccp/rte_ccp_pmd.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_string_fns.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_bus_vdev.h>
 #include <rte_common.h>
 #include <rte_cryptodev.h>
diff --git a/drivers/crypto/cnxk/cn10k_cryptodev.c b/drivers/crypto/cnxk/cn10k_cryptodev.c
index 012eb0c051..969442bb11 100644
--- a/drivers/crypto/cnxk/cn10k_cryptodev.c
+++ b/drivers/crypto/cnxk/cn10k_cryptodev.c
@@ -2,7 +2,7 @@
  * Copyright(C) 2021 Marvell.
  */
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_crypto.h>
 #include <rte_cryptodev.h>
diff --git a/drivers/crypto/cnxk/cn9k_cryptodev.c b/drivers/crypto/cnxk/cn9k_cryptodev.c
index 6b8cb01a12..2dcd04cb2e 100644
--- a/drivers/crypto/cnxk/cn9k_cryptodev.c
+++ b/drivers/crypto/cnxk/cn9k_cryptodev.c
@@ -2,7 +2,7 @@
  * Copyright(C) 2021 Marvell.
  */
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_crypto.h>
 #include <rte_cryptodev.h>
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index e01be15ade..7957a2242f 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -6,7 +6,7 @@
 #include <rte_mempool.h>
 #include <rte_errno.h>
 #include <rte_log.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_memory.h>
 
 #include <mlx5_glue.h>
diff --git a/drivers/crypto/nitrox/nitrox_device.h b/drivers/crypto/nitrox/nitrox_device.h
index 6b8095f42b..91b15d9913 100644
--- a/drivers/crypto/nitrox/nitrox_device.h
+++ b/drivers/crypto/nitrox/nitrox_device.h
@@ -5,7 +5,7 @@
 #ifndef _NITROX_DEVICE_H_
 #define _NITROX_DEVICE_H_
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_cryptodev.h>
 
 struct nitrox_sym_device;
diff --git a/drivers/crypto/octeontx/otx_cryptodev.c b/drivers/crypto/octeontx/otx_cryptodev.c
index c294f86d79..ca92e56a0d 100644
--- a/drivers/crypto/octeontx/otx_cryptodev.c
+++ b/drivers/crypto/octeontx/otx_cryptodev.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2018 Cavium, Inc
  */
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_cryptodev.h>
 #include <cryptodev_pmd.h>
diff --git a/drivers/crypto/octeontx/otx_cryptodev_ops.c b/drivers/crypto/octeontx/otx_cryptodev_ops.c
index 9b5bde53f8..e7558079db 100644
--- a/drivers/crypto/octeontx/otx_cryptodev_ops.c
+++ b/drivers/crypto/octeontx/otx_cryptodev_ops.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_alarm.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_cryptodev.h>
 #include <cryptodev_pmd.h>
 #include <rte_eventdev.h>
diff --git a/drivers/crypto/octeontx2/otx2_cryptodev.c b/drivers/crypto/octeontx2/otx2_cryptodev.c
index 85b1f00263..1ea9b54f41 100644
--- a/drivers/crypto/octeontx2/otx2_cryptodev.c
+++ b/drivers/crypto/octeontx2/otx2_cryptodev.c
@@ -2,7 +2,7 @@
  * Copyright (C) 2019 Marvell International Ltd.
  */
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_crypto.h>
 #include <rte_cryptodev.h>
diff --git a/drivers/crypto/qat/qat_sym.c b/drivers/crypto/qat/qat_sym.c
index 93b257522b..574e3d577d 100644
--- a/drivers/crypto/qat/qat_sym.c
+++ b/drivers/crypto/qat/qat_sym.c
@@ -7,7 +7,7 @@
 #include <rte_mempool.h>
 #include <rte_mbuf.h>
 #include <rte_crypto_sym.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_byteorder.h>
 
 #include "qat_sym.h"
diff --git a/drivers/crypto/qat/qat_sym_pmd.c b/drivers/crypto/qat/qat_sym_pmd.c
index efda921c05..50acd5d189 100644
--- a/drivers/crypto/qat/qat_sym_pmd.c
+++ b/drivers/crypto/qat/qat_sym_pmd.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2015-2018 Intel Corporation
  */
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_dev.h>
 #include <rte_malloc.h>
diff --git a/drivers/crypto/virtio/virtio_cryptodev.c b/drivers/crypto/virtio/virtio_cryptodev.c
index 8faa39df4a..7929c71249 100644
--- a/drivers/crypto/virtio/virtio_cryptodev.c
+++ b/drivers/crypto/virtio/virtio_cryptodev.c
@@ -7,7 +7,7 @@
 #include <rte_common.h>
 #include <rte_errno.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_cryptodev.h>
 #include <cryptodev_pmd.h>
 #include <rte_eal.h>
diff --git a/drivers/crypto/virtio/virtio_pci.h b/drivers/crypto/virtio/virtio_pci.h
index 0a7ea1bb64..889c263555 100644
--- a/drivers/crypto/virtio/virtio_pci.h
+++ b/drivers/crypto/virtio/virtio_pci.h
@@ -9,7 +9,7 @@
 
 #include <rte_eal_paging.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_cryptodev.h>
 
 #include "virtio_crypto.h"
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
index 9eeda482a3..40178e0dda 100644
--- a/drivers/event/dlb2/pf/dlb2_main.h
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -9,7 +9,7 @@
 #include <rte_log.h>
 #include <rte_spinlock.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_eal_paging.h>
 
 #include "base/dlb2_hw_types.h"
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index e9da89d650..8f8c742286 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -25,7 +25,7 @@
 #include <rte_cycles.h>
 #include <rte_io.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_eventdev.h>
 #include <eventdev_pmd.h>
 #include <eventdev_pmd_pci.h>
diff --git a/drivers/event/octeontx/ssovf_probe.c b/drivers/event/octeontx/ssovf_probe.c
index 4da7d1ae45..f45f005e33 100644
--- a/drivers/event/octeontx/ssovf_probe.c
+++ b/drivers/event/octeontx/ssovf_probe.c
@@ -7,7 +7,7 @@
 #include <rte_eal.h>
 #include <rte_io.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "octeontx_mbox.h"
 #include "ssovf_evdev.h"
diff --git a/drivers/event/octeontx/timvf_probe.c b/drivers/event/octeontx/timvf_probe.c
index 59bba31e8e..5a75494b12 100644
--- a/drivers/event/octeontx/timvf_probe.c
+++ b/drivers/event/octeontx/timvf_probe.c
@@ -5,7 +5,7 @@
 #include <rte_eal.h>
 #include <rte_io.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include <octeontx_mbox.h>
 
diff --git a/drivers/event/octeontx2/otx2_evdev.c b/drivers/event/octeontx2/otx2_evdev.c
index 38a6b651d9..5db1880455 100644
--- a/drivers/event/octeontx2/otx2_evdev.c
+++ b/drivers/event/octeontx2/otx2_evdev.c
@@ -4,7 +4,7 @@
 
 #include <inttypes.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_eal.h>
 #include <eventdev_pmd_pci.h>
diff --git a/drivers/mempool/cnxk/cnxk_mempool.c b/drivers/mempool/cnxk/cnxk_mempool.c
index dd4d74ca05..175bad355f 100644
--- a/drivers/mempool/cnxk/cnxk_mempool.c
+++ b/drivers/mempool/cnxk/cnxk_mempool.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_atomic.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_devargs.h>
 #include <rte_eal.h>
diff --git a/drivers/mempool/octeontx/octeontx_fpavf.c b/drivers/mempool/octeontx/octeontx_fpavf.c
index 94dc5cd815..a7a09f7872 100644
--- a/drivers/mempool/octeontx/octeontx_fpavf.c
+++ b/drivers/mempool/octeontx/octeontx_fpavf.c
@@ -13,7 +13,7 @@
 
 #include <rte_atomic.h>
 #include <rte_eal.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_errno.h>
 #include <rte_memory.h>
 #include <rte_malloc.h>
diff --git a/drivers/mempool/octeontx2/otx2_mempool.c b/drivers/mempool/octeontx2/otx2_mempool.c
index fb630fecf8..6c3969fcfb 100644
--- a/drivers/mempool/octeontx2/otx2_mempool.c
+++ b/drivers/mempool/octeontx2/otx2_mempool.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_atomic.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_eal.h>
 #include <rte_io.h>
diff --git a/drivers/mempool/octeontx2/otx2_mempool.h b/drivers/mempool/octeontx2/otx2_mempool.h
index 8aa548248d..b2ed9a56e8 100644
--- a/drivers/mempool/octeontx2/otx2_mempool.h
+++ b/drivers/mempool/octeontx2/otx2_mempool.h
@@ -6,7 +6,7 @@
 #define __OTX2_MEMPOOL_H__
 
 #include <rte_bitmap.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_devargs.h>
 #include <rte_mempool.h>
 
diff --git a/drivers/mempool/octeontx2/otx2_mempool_irq.c b/drivers/mempool/octeontx2/otx2_mempool_irq.c
index 5fa22b9612..c7c9b0d35d 100644
--- a/drivers/mempool/octeontx2/otx2_mempool_irq.c
+++ b/drivers/mempool/octeontx2/otx2_mempool_irq.c
@@ -5,7 +5,7 @@
 #include <inttypes.h>
 
 #include <rte_common.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "otx2_common.h"
 #include "otx2_irq.h"
diff --git a/drivers/meson.build b/drivers/meson.build
index 3d08540581..0cb82e01ca 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -85,6 +85,7 @@ foreach subpath:subdirs
         name = drv
         sources = []
         headers = []
+        driver_sdk_headers = [] # public headers included by drivers
         objs = []
         cflags = default_cflags
         includes = [include_directories(drv_path)]
@@ -153,6 +154,9 @@ foreach subpath:subdirs
         dpdk_extra_ldflags += pkgconfig_extra_libs
 
         install_headers(headers)
+        if get_option('enable_driver_sdk')
+            install_headers(driver_sdk_headers)
+        endif
 
         # generate pmdinfo sources by building a temporary
         # lib and then running pmdinfogen on the contents of
diff --git a/drivers/net/ark/ark_ethdev.c b/drivers/net/ark/ark_ethdev.c
index 377299b14c..4c564ce21e 100644
--- a/drivers/net/ark/ark_ethdev.c
+++ b/drivers/net/ark/ark_ethdev.c
@@ -6,7 +6,7 @@
 #include <sys/stat.h>
 #include <dlfcn.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <ethdev_pci.h>
 #include <rte_kvargs.h>
 
diff --git a/drivers/net/avp/avp_ethdev.c b/drivers/net/avp/avp_ethdev.c
index 623fa5e5ff..87e29519e6 100644
--- a/drivers/net/avp/avp_ethdev.c
+++ b/drivers/net/avp/avp_ethdev.c
@@ -16,7 +16,7 @@
 #include <rte_atomic.h>
 #include <rte_branch_prediction.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ether.h>
 #include <rte_common.h>
 #include <rte_cycles.h>
diff --git a/drivers/net/bnx2x/bnx2x.h b/drivers/net/bnx2x/bnx2x.h
index 80d19cbfd6..425acaa5ac 100644
--- a/drivers/net/bnx2x/bnx2x.h
+++ b/drivers/net/bnx2x/bnx2x.h
@@ -16,7 +16,7 @@
 
 #include <rte_byteorder.h>
 #include <rte_spinlock.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_io.h>
 
 #include "bnx2x_osal.h"
diff --git a/drivers/net/bnxt/bnxt.h b/drivers/net/bnxt/bnxt.h
index a64b138bc3..381593cbfd 100644
--- a/drivers/net/bnxt/bnxt.h
+++ b/drivers/net/bnxt/bnxt.h
@@ -11,7 +11,7 @@
 #include <sys/queue.h>
 
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <ethdev_driver.h>
 #include <rte_memory.h>
 #include <rte_lcore.h>
diff --git a/drivers/net/bonding/rte_eth_bond_args.c b/drivers/net/bonding/rte_eth_bond_args.c
index 5406e1c934..24ff649ca7 100644
--- a/drivers/net/bonding/rte_eth_bond_args.c
+++ b/drivers/net/bonding/rte_eth_bond_args.c
@@ -4,7 +4,7 @@
 
 #include <rte_devargs.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_kvargs.h>
 
 #include "rte_eth_bond.h"
diff --git a/drivers/net/cxgbe/base/adapter.h b/drivers/net/cxgbe/base/adapter.h
index 01a2a9d147..80809d7cd6 100644
--- a/drivers/net/cxgbe/base/adapter.h
+++ b/drivers/net/cxgbe/base/adapter.h
@@ -8,7 +8,7 @@
 #ifndef __T4_ADAPTER_H__
 #define __T4_ADAPTER_H__
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_mbuf.h>
 #include <rte_io.h>
 #include <rte_rwlock.h>
diff --git a/drivers/net/cxgbe/cxgbe_ethdev.c b/drivers/net/cxgbe/cxgbe_ethdev.c
index 177eca3976..fb03d69d66 100644
--- a/drivers/net/cxgbe/cxgbe_ethdev.c
+++ b/drivers/net/cxgbe/cxgbe_ethdev.c
@@ -20,7 +20,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_branch_prediction.h>
 #include <rte_memory.h>
 #include <rte_tailq.h>
diff --git a/drivers/net/e1000/em_ethdev.c b/drivers/net/e1000/em_ethdev.c
index a0ca371b02..657de61aa4 100644
--- a/drivers/net/e1000/em_ethdev.c
+++ b/drivers/net/e1000/em_ethdev.c
@@ -13,7 +13,7 @@
 #include <rte_byteorder.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ether.h>
 #include <ethdev_driver.h>
 #include <ethdev_pci.h>
diff --git a/drivers/net/e1000/em_rxtx.c b/drivers/net/e1000/em_rxtx.c
index dfd8f2fd00..c75be0d1e5 100644
--- a/drivers/net/e1000/em_rxtx.c
+++ b/drivers/net/e1000/em_rxtx.c
@@ -18,7 +18,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_memory.h>
 #include <rte_memcpy.h>
 #include <rte_memzone.h>
diff --git a/drivers/net/e1000/igb_ethdev.c b/drivers/net/e1000/igb_ethdev.c
index d80fad01e3..4a598db052 100644
--- a/drivers/net/e1000/igb_ethdev.c
+++ b/drivers/net/e1000/igb_ethdev.c
@@ -15,7 +15,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ether.h>
 #include <ethdev_driver.h>
 #include <ethdev_pci.h>
diff --git a/drivers/net/e1000/igb_pf.c b/drivers/net/e1000/igb_pf.c
index 2ce74dd5a9..294fb75dc3 100644
--- a/drivers/net/e1000/igb_pf.c
+++ b/drivers/net/e1000/igb_pf.c
@@ -10,7 +10,7 @@
 #include <stdarg.h>
 #include <inttypes.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_interrupts.h>
 #include <rte_log.h>
 #include <rte_debug.h>
diff --git a/drivers/net/ena/ena_ethdev.h b/drivers/net/ena/ena_ethdev.h
index 06ac8b06b5..a3d76383e0 100644
--- a/drivers/net/ena/ena_ethdev.h
+++ b/drivers/net/ena/ena_ethdev.h
@@ -12,7 +12,7 @@
 #include <ethdev_pci.h>
 #include <rte_cycles.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_timer.h>
 #include <rte_dev.h>
 #include <rte_net.h>
diff --git a/drivers/net/enic/base/vnic_dev.h b/drivers/net/enic/base/vnic_dev.h
index 4b9f75b65f..db9b2af3cd 100644
--- a/drivers/net/enic/base/vnic_dev.h
+++ b/drivers/net/enic/base/vnic_dev.h
@@ -9,7 +9,7 @@
 #include <stdbool.h>
 
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "enic_compat.h"
 #include "vnic_resource.h"
diff --git a/drivers/net/enic/enic_ethdev.c b/drivers/net/enic/enic_ethdev.c
index 8d5797523b..1202fcd4f9 100644
--- a/drivers/net/enic/enic_ethdev.c
+++ b/drivers/net/enic/enic_ethdev.c
@@ -8,7 +8,7 @@
 
 #include <rte_dev.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <ethdev_driver.h>
 #include <ethdev_pci.h>
 #include <rte_geneve.h>
diff --git a/drivers/net/enic/enic_main.c b/drivers/net/enic/enic_main.c
index 2affd380c6..f5c1131839 100644
--- a/drivers/net/enic/enic_main.c
+++ b/drivers/net/enic/enic_main.c
@@ -10,7 +10,7 @@
 #include <fcntl.h>
 
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_memzone.h>
 #include <rte_malloc.h>
 #include <rte_mbuf.h>
diff --git a/drivers/net/enic/enic_vf_representor.c b/drivers/net/enic/enic_vf_representor.c
index 79dd6e5640..f44f6c013e 100644
--- a/drivers/net/enic/enic_vf_representor.c
+++ b/drivers/net/enic/enic_vf_representor.c
@@ -5,7 +5,7 @@
 #include <stdint.h>
 #include <stdio.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_dev.h>
 #include <ethdev_driver.h>
diff --git a/drivers/net/hinic/base/hinic_pmd_hwdev.c b/drivers/net/hinic/base/hinic_pmd_hwdev.c
index cb9cf6efa2..49a6b523f5 100644
--- a/drivers/net/hinic/base/hinic_pmd_hwdev.c
+++ b/drivers/net/hinic/base/hinic_pmd_hwdev.c
@@ -3,7 +3,7 @@
  */
 
 #include<ethdev_driver.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_hash.h>
 #include <rte_jhash.h>
 
diff --git a/drivers/net/hinic/base/hinic_pmd_hwif.c b/drivers/net/hinic/base/hinic_pmd_hwif.c
index 26fa1e27d4..9b1b41ca95 100644
--- a/drivers/net/hinic/base/hinic_pmd_hwif.c
+++ b/drivers/net/hinic/base/hinic_pmd_hwif.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2017 Huawei Technologies Co., Ltd
  */
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "hinic_compat.h"
 #include "hinic_csr.h"
diff --git a/drivers/net/hinic/base/hinic_pmd_nicio.c b/drivers/net/hinic/base/hinic_pmd_nicio.c
index ad5db9f1de..234b433c90 100644
--- a/drivers/net/hinic/base/hinic_pmd_nicio.c
+++ b/drivers/net/hinic/base/hinic_pmd_nicio.c
@@ -1,7 +1,7 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2017 Huawei Technologies Co., Ltd
  */
-#include<rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "hinic_compat.h"
 #include "hinic_pmd_hwdev.h"
diff --git a/drivers/net/hinic/hinic_pmd_ethdev.c b/drivers/net/hinic/hinic_pmd_ethdev.c
index c01e2ec1d4..9820e8aa0b 100644
--- a/drivers/net/hinic/hinic_pmd_ethdev.c
+++ b/drivers/net/hinic/hinic_pmd_ethdev.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <ethdev_pci.h>
 #include <rte_mbuf.h>
 #include <rte_malloc.h>
diff --git a/drivers/net/hns3/hns3_ethdev.c b/drivers/net/hns3/hns3_ethdev.c
index 7d37004972..d5d5bc8e0f 100644
--- a/drivers/net/hns3/hns3_ethdev.c
+++ b/drivers/net/hns3/hns3_ethdev.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_alarm.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <ethdev_pci.h>
 #include <rte_pci.h>
 #include <rte_kvargs.h>
diff --git a/drivers/net/hns3/hns3_rxtx.c b/drivers/net/hns3/hns3_rxtx.c
index 481872e395..9a8a1289c6 100644
--- a/drivers/net/hns3/hns3_rxtx.c
+++ b/drivers/net/hns3/hns3_rxtx.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2018-2021 HiSilicon Limited.
  */
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_cycles.h>
 #include <rte_geneve.h>
diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 7b230e2ed1..51fface85e 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -15,7 +15,7 @@
 #include <rte_eal.h>
 #include <rte_string_fns.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ether.h>
 #include <ethdev_driver.h>
 #include <ethdev_pci.h>
diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index e8dd6d1dab..0ddff5e343 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -18,7 +18,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_atomic.h>
 #include <rte_branch_prediction.h>
 #include <rte_memory.h>
diff --git a/drivers/net/i40e/i40e_vf_representor.c b/drivers/net/i40e/i40e_vf_representor.c
index 0481b55381..650d8ad502 100644
--- a/drivers/net/i40e/i40e_vf_representor.c
+++ b/drivers/net/i40e/i40e_vf_representor.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2018 Intel Corporation.
  */
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ethdev.h>
 #include <rte_pci.h>
 #include <rte_malloc.h>
diff --git a/drivers/net/igc/igc_ethdev.c b/drivers/net/igc/igc_ethdev.c
index 224a095483..3d3d0d6f3c 100644
--- a/drivers/net/igc/igc_ethdev.c
+++ b/drivers/net/igc/igc_ethdev.c
@@ -7,7 +7,7 @@
 
 #include <rte_string_fns.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <ethdev_driver.h>
 #include <ethdev_pci.h>
 #include <rte_malloc.h>
diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index 49b90d1b7c..e946d92c29 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -8,7 +8,7 @@
 #include <stdint.h>
 #include <inttypes.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "ionic_dev.h"
 #include "ionic_if.h"
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index e620793966..e8bcccccb4 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ethdev.h>
 #include <ethdev_driver.h>
 #include <rte_malloc.h>
diff --git a/drivers/net/ipn3ke/ipn3ke_ethdev.c b/drivers/net/ipn3ke/ipn3ke_ethdev.c
index 964506c6db..2493df2bcd 100644
--- a/drivers/net/ipn3ke/ipn3ke_ethdev.c
+++ b/drivers/net/ipn3ke/ipn3ke_ethdev.c
@@ -4,7 +4,7 @@
 
 #include <stdint.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ethdev.h>
 #include <rte_pci.h>
 #include <rte_malloc.h>
diff --git a/drivers/net/ipn3ke/ipn3ke_representor.c b/drivers/net/ipn3ke/ipn3ke_representor.c
index 589d9fa587..f288bf09d8 100644
--- a/drivers/net/ipn3ke/ipn3ke_representor.c
+++ b/drivers/net/ipn3ke/ipn3ke_representor.c
@@ -5,7 +5,7 @@
 #include <stdint.h>
 #include <unistd.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ethdev.h>
 #include <rte_pci.h>
 #include <rte_malloc.h>
diff --git a/drivers/net/ipn3ke/ipn3ke_tm.c b/drivers/net/ipn3ke/ipn3ke_tm.c
index 6a9b98fd7f..532d232dbf 100644
--- a/drivers/net/ipn3ke/ipn3ke_tm.c
+++ b/drivers/net/ipn3ke/ipn3ke_tm.c
@@ -6,7 +6,7 @@
 #include <stdlib.h>
 #include <string.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_ethdev.h>
 #include <rte_pci.h>
 #include <rte_malloc.h>
diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 47693c0c47..ab277bc0dd 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -20,7 +20,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_branch_prediction.h>
 #include <rte_memory.h>
 #include <rte_kvargs.h>
diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
index a0ce18ca24..7b0c1ad542 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.h
+++ b/drivers/net/ixgbe/ixgbe_ethdev.h
@@ -19,7 +19,7 @@
 #include <rte_time.h>
 #include <rte_hash.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_tm_driver.h>
 
 /* need update link, bit flag */
diff --git a/drivers/net/mlx4/mlx4_ethdev.c b/drivers/net/mlx4/mlx4_ethdev.c
index 783ff94dce..b86f65e32a 100644
--- a/drivers/net/mlx4/mlx4_ethdev.c
+++ b/drivers/net/mlx4/mlx4_ethdev.c
@@ -32,7 +32,7 @@
 #pragma GCC diagnostic error "-Wpedantic"
 #endif
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_errno.h>
 #include <ethdev_driver.h>
 #include <rte_ether.h>
diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
index f34133e2c6..2e0a40f0cc 100644
--- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
+++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
@@ -25,7 +25,7 @@
 #include <time.h>
 
 #include <ethdev_driver.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_mbuf.h>
 #include <rte_common.h>
 #include <rte_interrupts.h>
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 470b16cb9a..e263818139 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -19,7 +19,7 @@
 #include <ethdev_driver.h>
 #include <ethdev_pci.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_bus_auxiliary.h>
 #include <rte_common.h>
 #include <rte_kvargs.h>
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index f84e061fe7..bbcdec0f49 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -13,7 +13,7 @@
 #include <rte_malloc.h>
 #include <ethdev_driver.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_kvargs.h>
 #include <rte_rwlock.h>
diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c
index 82e2284d98..8888702fc8 100644
--- a/drivers/net/mlx5/mlx5_ethdev.c
+++ b/drivers/net/mlx5/mlx5_ethdev.c
@@ -11,7 +11,7 @@
 #include <errno.h>
 
 #include <ethdev_driver.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_mbuf.h>
 #include <rte_common.h>
 #include <rte_interrupts.h>
diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c
index eb4d34ca55..4d8c26f7bc 100644
--- a/drivers/net/mlx5/mlx5_txq.c
+++ b/drivers/net/mlx5/mlx5_txq.c
@@ -13,7 +13,7 @@
 #include <rte_mbuf.h>
 #include <rte_malloc.h>
 #include <ethdev_driver.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_eal_paging.h>
 
diff --git a/drivers/net/netvsc/hn_vf.c b/drivers/net/netvsc/hn_vf.c
index 75192e6319..1da4eacb6f 100644
--- a/drivers/net/netvsc/hn_vf.c
+++ b/drivers/net/netvsc/hn_vf.c
@@ -21,7 +21,7 @@
 #include <rte_memory.h>
 #include <rte_bus_vmbus.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_log.h>
 #include <rte_string_fns.h>
 #include <rte_alarm.h>
diff --git a/drivers/net/octeontx/base/octeontx_pkivf.c b/drivers/net/octeontx/base/octeontx_pkivf.c
index 0ddff54886..9ff5b78fd0 100644
--- a/drivers/net/octeontx/base/octeontx_pkivf.c
+++ b/drivers/net/octeontx/base/octeontx_pkivf.c
@@ -5,7 +5,7 @@
 #include <string.h>
 
 #include <rte_eal.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "../octeontx_logs.h"
 #include "octeontx_io.h"
diff --git a/drivers/net/octeontx/base/octeontx_pkovf.c b/drivers/net/octeontx/base/octeontx_pkovf.c
index bf28bc7992..a125acadce 100644
--- a/drivers/net/octeontx/base/octeontx_pkovf.c
+++ b/drivers/net/octeontx/base/octeontx_pkovf.c
@@ -10,7 +10,7 @@
 #include <rte_cycles.h>
 #include <rte_malloc.h>
 #include <rte_memory.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_spinlock.h>
 
 #include "../octeontx_logs.h"
diff --git a/drivers/net/octeontx2/otx2_ethdev_irq.c b/drivers/net/octeontx2/otx2_ethdev_irq.c
index b121488faf..40541c9280 100644
--- a/drivers/net/octeontx2/otx2_ethdev_irq.c
+++ b/drivers/net/octeontx2/otx2_ethdev_irq.c
@@ -4,7 +4,7 @@
 
 #include <inttypes.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_malloc.h>
 
 #include "otx2_ethdev.h"
diff --git a/drivers/net/qede/base/bcm_osal.h b/drivers/net/qede/base/bcm_osal.h
index c5b5399282..10ec1e2fb7 100644
--- a/drivers/net/qede/base/bcm_osal.h
+++ b/drivers/net/qede/base/bcm_osal.h
@@ -21,7 +21,7 @@
 #include <rte_ether.h>
 #include <rte_io.h>
 #include <rte_version.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 /* Forward declaration */
 struct ecore_dev;
diff --git a/drivers/net/sfc/sfc.h b/drivers/net/sfc/sfc.h
index 331e06bac6..f57da8fc96 100644
--- a/drivers/net/sfc/sfc.h
+++ b/drivers/net/sfc/sfc.h
@@ -13,7 +13,7 @@
 #include <stdbool.h>
 
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <ethdev_driver.h>
 #include <rte_kvargs.h>
 #include <rte_spinlock.h>
diff --git a/drivers/net/sfc/sfc_ethdev.c b/drivers/net/sfc/sfc_ethdev.c
index 2db0d000c3..7f8b052845 100644
--- a/drivers/net/sfc/sfc_ethdev.c
+++ b/drivers/net/sfc/sfc_ethdev.c
@@ -11,7 +11,7 @@
 #include <ethdev_driver.h>
 #include <ethdev_pci.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_errno.h>
 #include <rte_string_fns.h>
 #include <rte_ether.h>
diff --git a/drivers/net/sfc/sfc_sriov.c b/drivers/net/sfc/sfc_sriov.c
index baa0242433..48c64c8311 100644
--- a/drivers/net/sfc/sfc_sriov.c
+++ b/drivers/net/sfc/sfc_sriov.c
@@ -8,7 +8,7 @@
  */
 
 #include <rte_common.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "sfc.h"
 #include "sfc_log.h"
diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index 561a98fc81..929ce86535 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -32,7 +32,7 @@
 #include <rte_malloc.h>
 #include <rte_random.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_tailq.h>
 #include <rte_devargs.h>
 #include <rte_kvargs.h>
diff --git a/drivers/net/txgbe/txgbe_ethdev.h b/drivers/net/txgbe/txgbe_ethdev.h
index 3021933965..faf0c58bd5 100644
--- a/drivers/net/txgbe/txgbe_ethdev.h
+++ b/drivers/net/txgbe/txgbe_ethdev.h
@@ -20,7 +20,7 @@
 #include <rte_ethdev_core.h>
 #include <rte_hash.h>
 #include <rte_hash_crc.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_tm_driver.h>
 
 /* need update link, bit flag */
diff --git a/drivers/net/txgbe/txgbe_flow.c b/drivers/net/txgbe/txgbe_flow.c
index eae400b141..f70a9309d9 100644
--- a/drivers/net/txgbe/txgbe_flow.c
+++ b/drivers/net/txgbe/txgbe_flow.c
@@ -4,7 +4,7 @@
  */
 
 #include <sys/queue.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_malloc.h>
 #include <rte_flow.h>
 #include <rte_flow_driver.h>
diff --git a/drivers/net/txgbe/txgbe_pf.c b/drivers/net/txgbe/txgbe_pf.c
index 494d779a3c..58159bc226 100644
--- a/drivers/net/txgbe/txgbe_pf.c
+++ b/drivers/net/txgbe/txgbe_pf.c
@@ -20,7 +20,7 @@
 #include <rte_memcpy.h>
 #include <rte_malloc.h>
 #include <rte_random.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "base/txgbe.h"
 #include "txgbe_ethdev.h"
diff --git a/drivers/net/virtio/virtio_pci.h b/drivers/net/virtio/virtio_pci.h
index 11e25a0142..a9016534aa 100644
--- a/drivers/net/virtio/virtio_pci.h
+++ b/drivers/net/virtio/virtio_pci.h
@@ -9,7 +9,7 @@
 #include <stdbool.h>
 
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <ethdev_driver.h>
 
 #include "virtio.h"
diff --git a/drivers/net/virtio/virtio_pci_ethdev.c b/drivers/net/virtio/virtio_pci_ethdev.c
index 4083853c48..13dbc0c419 100644
--- a/drivers/net/virtio/virtio_pci_ethdev.c
+++ b/drivers/net/virtio/virtio_pci_ethdev.c
@@ -11,7 +11,7 @@
 #include <ethdev_driver.h>
 #include <ethdev_pci.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_errno.h>
 
 #include <rte_memory.h>
diff --git a/drivers/net/vmxnet3/vmxnet3_ethdev.c b/drivers/net/vmxnet3/vmxnet3_ethdev.c
index 2f40ae907d..e73f63e8ce 100644
--- a/drivers/net/vmxnet3/vmxnet3_ethdev.c
+++ b/drivers/net/vmxnet3/vmxnet3_ethdev.c
@@ -19,7 +19,7 @@
 #include <rte_log.h>
 #include <rte_debug.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_branch_prediction.h>
 #include <rte_memory.h>
 #include <rte_memzone.h>
diff --git a/drivers/raw/cnxk_bphy/cnxk_bphy.c b/drivers/raw/cnxk_bphy/cnxk_bphy.c
index 9cb3f8d332..61c60ed363 100644
--- a/drivers/raw/cnxk_bphy/cnxk_bphy.c
+++ b/drivers/raw/cnxk_bphy/cnxk_bphy.c
@@ -1,7 +1,7 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(C) 2021 Marvell.
  */
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_dev.h>
 #include <rte_eal.h>
diff --git a/drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c b/drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c
index ade45ab741..f9d7353a18 100644
--- a/drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c
+++ b/drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c
@@ -3,7 +3,7 @@
  */
 #include <string.h>
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_rawdev.h>
 #include <rte_rawdev_pmd.h>
 
diff --git a/drivers/raw/cnxk_bphy/cnxk_bphy_irq.c b/drivers/raw/cnxk_bphy/cnxk_bphy_irq.c
index bbcc285a7a..6014992934 100644
--- a/drivers/raw/cnxk_bphy/cnxk_bphy_irq.c
+++ b/drivers/raw/cnxk_bphy/cnxk_bphy_irq.c
@@ -1,7 +1,7 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(C) 2021 Marvell.
  */
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_pci.h>
 #include <rte_rawdev.h>
 #include <rte_rawdev_pmd.h>
diff --git a/drivers/raw/cnxk_bphy/cnxk_bphy_irq.h b/drivers/raw/cnxk_bphy/cnxk_bphy_irq.h
index b55147b93e..7a623b48c8 100644
--- a/drivers/raw/cnxk_bphy/cnxk_bphy_irq.h
+++ b/drivers/raw/cnxk_bphy/cnxk_bphy_irq.h
@@ -5,7 +5,7 @@
 #ifndef _CNXK_BPHY_IRQ_
 #define _CNXK_BPHY_IRQ_
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_dev.h>
 
 #include <roc_api.h>
diff --git a/drivers/raw/ifpga/ifpga_rawdev.c b/drivers/raw/ifpga/ifpga_rawdev.c
index 76e6a8530b..eb0c6e271f 100644
--- a/drivers/raw/ifpga/ifpga_rawdev.c
+++ b/drivers/raw/ifpga/ifpga_rawdev.c
@@ -16,7 +16,7 @@
 #include <rte_devargs.h>
 #include <rte_memcpy.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_kvargs.h>
 #include <rte_alarm.h>
 #include <rte_interrupts.h>
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.c b/drivers/raw/ifpga/rte_pmd_ifpga.c
index 23146432c2..959aad414d 100644
--- a/drivers/raw/ifpga/rte_pmd_ifpga.c
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_rawdev.h>
 #include <rte_rawdev_pmd.h>
 #include "rte_pmd_ifpga.h"
diff --git a/drivers/raw/ioat/idxd_pci.c b/drivers/raw/ioat/idxd_pci.c
index 13515dbc6c..257e4c004a 100644
--- a/drivers/raw/ioat/idxd_pci.c
+++ b/drivers/raw/ioat/idxd_pci.c
@@ -2,7 +2,7 @@
  * Copyright(c) 2020 Intel Corporation
  */
 
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_memzone.h>
 #include <rte_devargs.h>
 
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index 5396671d4f..f1d4220b69 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -3,7 +3,7 @@
  */
 
 #include <rte_cycles.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_memzone.h>
 #include <rte_string_fns.h>
 #include <rte_rawdev_pmd.h>
diff --git a/drivers/raw/ntb/ntb.c b/drivers/raw/ntb/ntb.c
index 78cfcd79f7..a74a42f8d2 100644
--- a/drivers/raw/ntb/ntb.c
+++ b/drivers/raw/ntb/ntb.c
@@ -13,7 +13,7 @@
 #include <rte_log.h>
 #include <rte_pci.h>
 #include <rte_mbuf.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_memzone.h>
 #include <rte_memcpy.h>
 #include <rte_rawdev.h>
diff --git a/drivers/raw/ntb/ntb_hw_intel.c b/drivers/raw/ntb/ntb_hw_intel.c
index a742e8fbb9..072c2dd7b9 100644
--- a/drivers/raw/ntb/ntb_hw_intel.c
+++ b/drivers/raw/ntb/ntb_hw_intel.c
@@ -8,7 +8,7 @@
 #include <rte_io.h>
 #include <rte_eal.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_rawdev.h>
 #include <rte_rawdev_pmd.h>
 
diff --git a/drivers/raw/octeontx2_dma/otx2_dpi_rawdev.c b/drivers/raw/octeontx2_dma/otx2_dpi_rawdev.c
index 8c01f25ec7..179e3e6dbb 100644
--- a/drivers/raw/octeontx2_dma/otx2_dpi_rawdev.c
+++ b/drivers/raw/octeontx2_dma/otx2_dpi_rawdev.c
@@ -6,7 +6,7 @@
 #include <unistd.h>
 
 #include <rte_bus.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_common.h>
 #include <rte_eal.h>
 #include <rte_lcore.h>
diff --git a/drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c b/drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c
index d04e957d82..214fd83169 100644
--- a/drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c
+++ b/drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c
@@ -8,7 +8,7 @@
 #include <fcntl.h>
 
 #include <rte_bus.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_eal.h>
 #include <rte_lcore.h>
 #include <rte_mempool.h>
diff --git a/drivers/raw/octeontx2_ep/otx2_ep_rawdev.c b/drivers/raw/octeontx2_ep/otx2_ep_rawdev.c
index b2ccdda83e..560a8bbf03 100644
--- a/drivers/raw/octeontx2_ep/otx2_ep_rawdev.c
+++ b/drivers/raw/octeontx2_ep/otx2_ep_rawdev.c
@@ -5,7 +5,7 @@
 #include <unistd.h>
 
 #include <rte_bus.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_eal.h>
 #include <rte_lcore.h>
 #include <rte_mempool.h>
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index f17b6df47f..0926aec5bc 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -9,7 +9,7 @@
 #include <rte_regexdev.h>
 #include <rte_regexdev_core.h>
 #include <rte_regexdev_driver.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include <mlx5_common.h>
 #include <mlx5_common_mr.h>
diff --git a/drivers/regex/mlx5/mlx5_regex_fastpath.c b/drivers/regex/mlx5/mlx5_regex_fastpath.c
index c79445ce7d..1ba12cd576 100644
--- a/drivers/regex/mlx5/mlx5_regex_fastpath.c
+++ b/drivers/regex/mlx5/mlx5_regex_fastpath.c
@@ -10,7 +10,7 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_pci.h>
 #include <rte_regexdev_driver.h>
 #include <rte_mbuf.h>
diff --git a/drivers/vdpa/ifc/base/ifcvf_osdep.h b/drivers/vdpa/ifc/base/ifcvf_osdep.h
index 6aef25ea45..3f3eb46f3f 100644
--- a/drivers/vdpa/ifc/base/ifcvf_osdep.h
+++ b/drivers/vdpa/ifc/base/ifcvf_osdep.h
@@ -10,7 +10,7 @@
 
 #include <rte_cycles.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_log.h>
 #include <rte_io.h>
 
diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c
index 1dc813d0a3..968dea5055 100644
--- a/drivers/vdpa/ifc/ifcvf_vdpa.c
+++ b/drivers/vdpa/ifc/ifcvf_vdpa.c
@@ -14,7 +14,7 @@
 #include <rte_eal_paging.h>
 #include <rte_malloc.h>
 #include <rte_memory.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_vhost.h>
 #include <rte_vdpa.h>
 #include <rte_vdpa_dev.h>
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 6d17d7a6f3..81c7e738f5 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -12,7 +12,7 @@
 #include <rte_log.h>
 #include <rte_errno.h>
 #include <rte_string_fns.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
diff --git a/lib/ethdev/ethdev_pci.h b/lib/ethdev/ethdev_pci.h
index 8edca82ce8..76779faa6c 100644
--- a/lib/ethdev/ethdev_pci.h
+++ b/lib/ethdev/ethdev_pci.h
@@ -8,7 +8,7 @@
 
 #include <rte_malloc.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 #include <rte_config.h>
 #include <ethdev_driver.h>
 
diff --git a/lib/eventdev/eventdev_pmd_pci.h b/lib/eventdev/eventdev_pmd_pci.h
index d14ea634b8..260362aacb 100644
--- a/lib/eventdev/eventdev_pmd_pci.h
+++ b/lib/eventdev/eventdev_pmd_pci.h
@@ -24,7 +24,7 @@ extern "C" {
 #include <rte_eal.h>
 #include <rte_lcore.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
+#include <pci_driver.h>
 
 #include "eventdev_pmd.h"
 
-- 
2.17.1

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Xia, Chenbo]

Gentle ping for comments..

@David, could you help me understand what is the compile error in Fedora 31?
DPDK_compile_spdk failure is expected as the header name for SPDK is changed,
I am not sure if it's the same error...

Thanks,
Chenbo

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Chenbo Xia
> Sent: Saturday, September 18, 2021 10:25 AM
> To: dev@dpdk.org; david.marchand@redhat.com
> Subject: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs
> 
> As announced in the deprecation notice, most ABIs in PCI bus will be removed.
> 
> As there exist some applications that want to access PCI memory resource,
> two new APIs are defined in Patch 1 and corresponding changes are applied
> to testpmd in Patch 2.
> 
> Patch 3-4 clean up the unnecessary usage of PCI bus header in examples.
> 
> Patch 5-6 clean up the unused PCI related structure in kni library and related
> tests and examples.
> 
> Patch 7 finally removes most of ABIs in PCI bus.
> 
> ---
> v2:
>  - Add check on call of port_id_pci_reg_write (Xiaoyun)
>  - Combine two clean-up patches in test and example, and backport (David)
> 
> Chenbo Xia (7):
>   bus/pci: add new memory resource access APIs
>   app/testpmd: use PCI memory resource access APIs
>   examples/ethtool: use PCI library API to get PCI address
>   examples/kni: remove unused PCI bus header
>   kni: remove unused PCI info from test and example
>   kni: replace unused variable definition with reserved bytes
>   bus/pci: remove ABIs in PCI bus
> 
>  app/test-pmd/config.c                         |  50 +--
>  app/test-pmd/testpmd.h                        |  54 +--
>  app/test/test_kni.c                           |  57 ---
>  app/test/virtual_pmd.c                        |   2 +-
>  doc/guides/rel_notes/release_21_11.rst        |   8 +
>  drivers/baseband/acc100/rte_acc100_pmd.c      |   2 +-
>  .../fpga_5gnr_fec/rte_fpga_5gnr_fec.c         |   2 +-
>  drivers/baseband/fpga_lte_fec/fpga_lte_fec.c  |   2 +-
>  drivers/bus/pci/bsd/pci.c                     |   1 -
>  drivers/bus/pci/linux/pci.c                   |   1 -
>  drivers/bus/pci/linux/pci_uio.c               |   1 -
>  drivers/bus/pci/linux/pci_vfio.c              |   1 -
>  drivers/bus/pci/meson.build                   |   4 +
>  drivers/bus/pci/pci_common.c                  |  78 ++++
>  drivers/bus/pci/pci_common_uio.c              |   1 -
>  drivers/bus/pci/pci_driver.h                  | 402 ++++++++++++++++++
>  drivers/bus/pci/pci_params.c                  |   1 -
>  drivers/bus/pci/private.h                     |   3 +-
>  drivers/bus/pci/rte_bus_pci.h                 | 387 ++---------------
>  drivers/bus/pci/version.map                   |  28 +-
>  drivers/common/cnxk/roc_platform.h            |   2 +-
>  drivers/common/mlx5/linux/mlx5_common_verbs.c |   2 +-
>  drivers/common/mlx5/mlx5_common_pci.c         |   2 +-
>  drivers/common/octeontx2/otx2_dev.h           |   2 +-
>  drivers/common/octeontx2/otx2_sec_idev.c      |   2 +-
>  drivers/common/qat/qat_device.h               |   2 +-
>  drivers/common/qat/qat_qp.c                   |   2 +-
>  drivers/common/sfc_efx/sfc_efx.h              |   2 +-
>  drivers/compress/mlx5/mlx5_compress.c         |   2 +-
>  drivers/compress/octeontx/otx_zip.h           |   2 +-
>  drivers/compress/qat/qat_comp.c               |   2 +-
>  drivers/crypto/ccp/ccp_dev.h                  |   2 +-
>  drivers/crypto/ccp/ccp_pci.h                  |   2 +-
>  drivers/crypto/ccp/rte_ccp_pmd.c              |   2 +-
>  drivers/crypto/cnxk/cn10k_cryptodev.c         |   2 +-
>  drivers/crypto/cnxk/cn9k_cryptodev.c          |   2 +-
>  drivers/crypto/mlx5/mlx5_crypto.c             |   2 +-
>  drivers/crypto/nitrox/nitrox_device.h         |   2 +-
>  drivers/crypto/octeontx/otx_cryptodev.c       |   2 +-
>  drivers/crypto/octeontx/otx_cryptodev_ops.c   |   2 +-
>  drivers/crypto/octeontx2/otx2_cryptodev.c     |   2 +-
>  drivers/crypto/qat/qat_sym.c                  |   2 +-
>  drivers/crypto/qat/qat_sym_pmd.c              |   2 +-
>  drivers/crypto/virtio/virtio_cryptodev.c      |   2 +-
>  drivers/crypto/virtio/virtio_pci.h            |   2 +-
>  drivers/event/dlb2/pf/dlb2_main.h             |   2 +-
>  drivers/event/dlb2/pf/dlb2_pf.c               |   2 +-
>  drivers/event/octeontx/ssovf_probe.c          |   2 +-
>  drivers/event/octeontx/timvf_probe.c          |   2 +-
>  drivers/event/octeontx2/otx2_evdev.c          |   2 +-
>  drivers/mempool/cnxk/cnxk_mempool.c           |   2 +-
>  drivers/mempool/octeontx/octeontx_fpavf.c     |   2 +-
>  drivers/mempool/octeontx2/otx2_mempool.c      |   2 +-
>  drivers/mempool/octeontx2/otx2_mempool.h      |   2 +-
>  drivers/mempool/octeontx2/otx2_mempool_irq.c  |   2 +-
>  drivers/meson.build                           |   4 +
>  drivers/net/ark/ark_ethdev.c                  |   2 +-
>  drivers/net/avp/avp_ethdev.c                  |   2 +-
>  drivers/net/bnx2x/bnx2x.h                     |   2 +-
>  drivers/net/bnxt/bnxt.h                       |   2 +-
>  drivers/net/bonding/rte_eth_bond_args.c       |   2 +-
>  drivers/net/cxgbe/base/adapter.h              |   2 +-
>  drivers/net/cxgbe/cxgbe_ethdev.c              |   2 +-
>  drivers/net/e1000/em_ethdev.c                 |   2 +-
>  drivers/net/e1000/em_rxtx.c                   |   2 +-
>  drivers/net/e1000/igb_ethdev.c                |   2 +-
>  drivers/net/e1000/igb_pf.c                    |   2 +-
>  drivers/net/ena/ena_ethdev.h                  |   2 +-
>  drivers/net/enic/base/vnic_dev.h              |   2 +-
>  drivers/net/enic/enic_ethdev.c                |   2 +-
>  drivers/net/enic/enic_main.c                  |   2 +-
>  drivers/net/enic/enic_vf_representor.c        |   2 +-
>  drivers/net/hinic/base/hinic_pmd_hwdev.c      |   2 +-
>  drivers/net/hinic/base/hinic_pmd_hwif.c       |   2 +-
>  drivers/net/hinic/base/hinic_pmd_nicio.c      |   2 +-
>  drivers/net/hinic/hinic_pmd_ethdev.c          |   2 +-
>  drivers/net/hns3/hns3_ethdev.c                |   2 +-
>  drivers/net/hns3/hns3_rxtx.c                  |   2 +-
>  drivers/net/i40e/i40e_ethdev.c                |   2 +-
>  drivers/net/i40e/i40e_ethdev_vf.c             |   2 +-
>  drivers/net/i40e/i40e_vf_representor.c        |   2 +-
>  drivers/net/igc/igc_ethdev.c                  |   2 +-
>  drivers/net/ionic/ionic.h                     |   2 +-
>  drivers/net/ionic/ionic_ethdev.c              |   2 +-
>  drivers/net/ipn3ke/ipn3ke_ethdev.c            |   2 +-
>  drivers/net/ipn3ke/ipn3ke_representor.c       |   2 +-
>  drivers/net/ipn3ke/ipn3ke_tm.c                |   2 +-
>  drivers/net/ixgbe/ixgbe_ethdev.c              |   2 +-
>  drivers/net/ixgbe/ixgbe_ethdev.h              |   2 +-
>  drivers/net/mlx4/mlx4_ethdev.c                |   2 +-
>  drivers/net/mlx5/linux/mlx5_ethdev_os.c       |   2 +-
>  drivers/net/mlx5/linux/mlx5_os.c              |   2 +-
>  drivers/net/mlx5/mlx5.c                       |   2 +-
>  drivers/net/mlx5/mlx5_ethdev.c                |   2 +-
>  drivers/net/mlx5/mlx5_txq.c                   |   2 +-
>  drivers/net/netvsc/hn_vf.c                    |   2 +-
>  drivers/net/octeontx/base/octeontx_pkivf.c    |   2 +-
>  drivers/net/octeontx/base/octeontx_pkovf.c    |   2 +-
>  drivers/net/octeontx2/otx2_ethdev_irq.c       |   2 +-
>  drivers/net/qede/base/bcm_osal.h              |   2 +-
>  drivers/net/sfc/sfc.h                         |   2 +-
>  drivers/net/sfc/sfc_ethdev.c                  |   2 +-
>  drivers/net/sfc/sfc_sriov.c                   |   2 +-
>  drivers/net/thunderx/nicvf_ethdev.c           |   2 +-
>  drivers/net/txgbe/txgbe_ethdev.h              |   2 +-
>  drivers/net/txgbe/txgbe_flow.c                |   2 +-
>  drivers/net/txgbe/txgbe_pf.c                  |   2 +-
>  drivers/net/virtio/virtio_pci.h               |   2 +-
>  drivers/net/virtio/virtio_pci_ethdev.c        |   2 +-
>  drivers/net/vmxnet3/vmxnet3_ethdev.c          |   2 +-
>  drivers/raw/cnxk_bphy/cnxk_bphy.c             |   2 +-
>  drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c         |   2 +-
>  drivers/raw/cnxk_bphy/cnxk_bphy_irq.c         |   2 +-
>  drivers/raw/cnxk_bphy/cnxk_bphy_irq.h         |   2 +-
>  drivers/raw/ifpga/ifpga_rawdev.c              |   2 +-
>  drivers/raw/ifpga/rte_pmd_ifpga.c             |   2 +-
>  drivers/raw/ioat/idxd_pci.c                   |   2 +-
>  drivers/raw/ioat/ioat_rawdev.c                |   2 +-
>  drivers/raw/ntb/ntb.c                         |   2 +-
>  drivers/raw/ntb/ntb_hw_intel.c                |   2 +-
>  drivers/raw/octeontx2_dma/otx2_dpi_rawdev.c   |   2 +-
>  drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c     |   2 +-
>  drivers/raw/octeontx2_ep/otx2_ep_rawdev.c     |   2 +-
>  drivers/regex/mlx5/mlx5_regex.c               |   2 +-
>  drivers/regex/mlx5/mlx5_regex_fastpath.c      |   2 +-
>  drivers/vdpa/ifc/base/ifcvf_osdep.h           |   2 +-
>  drivers/vdpa/ifc/ifcvf_vdpa.c                 |   2 +-
>  drivers/vdpa/mlx5/mlx5_vdpa.c                 |   2 +-
>  examples/ethtool/lib/rte_ethtool.c            |  14 +-
>  examples/ethtool/meson.build                  |   2 +-
>  examples/ip_pipeline/kni.c                    |  16 -
>  examples/kni/main.c                           |   1 -
>  lib/ethdev/ethdev_pci.h                       |   2 +-
>  lib/eventdev/eventdev_pmd_pci.h               |   2 +-
>  lib/kni/rte_kni.h                             |   4 +-
>  135 files changed, 711 insertions(+), 633 deletions(-)
>  create mode 100644 drivers/bus/pci/pci_driver.h
> 
> --
> 2.17.1

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[David Marchand]

Hello Chenbo,

On Wed, Sep 29, 2021 at 9:38 AM Xia, Chenbo <chenbo.xia@intel.com> wrote:
>
> Gentle ping for comments..

Sorry, I'll try to look at it.

>
> @David, could you help me understand what is the compile error in Fedora 31?
> DPDK_compile_spdk failure is expected as the header name for SPDK is changed,
> I am not sure if it's the same error...

The error log is odd (no compilation "backtrace").
You'll need to test spdk manually I guess.


-- 
David Marchand

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[David Marchand]

On Thu, Sep 30, 2021 at 10:45 AM David Marchand
<david.marchand@redhat.com> wrote:
> On Wed, Sep 29, 2021 at 9:38 AM Xia, Chenbo <chenbo.xia@intel.com> wrote:
> > @David, could you help me understand what is the compile error in Fedora 31?
> > DPDK_compile_spdk failure is expected as the header name for SPDK is changed,
> > I am not sure if it's the same error...
>
> The error log is odd (no compilation "backtrace").
> You'll need to test spdk manually I guess.

Tried your series with SPDK (w/o and w/ enable_driver_sdk).
I think the same, and the error is likely due to the file rename.

$ make
  CC lib/env_dpdk/env.o
In file included from env.c:39:0:
env_internal.h:64:25: error: field ‘driver’ has incomplete type
  struct rte_pci_driver  driver;
                         ^
env_internal.h:75:59: warning: ‘struct rte_pci_device’ declared inside
parameter list [enabled by default]
 int pci_device_init(struct rte_pci_driver *driver, struct
rte_pci_device *device);
                                                           ^
env_internal.h:75:59: warning: its scope is only this definition or
declaration, which is probably not what you want [enabled by default]
env_internal.h:76:28: warning: ‘struct rte_pci_device’ declared inside
parameter list [enabled by default]
 int pci_device_fini(struct rte_pci_device *device);
                            ^
env_internal.h:89:38: warning: ‘struct rte_pci_device’ declared inside
parameter list [enabled by default]
 void vtophys_pci_device_added(struct rte_pci_device *pci_device);
                                      ^
env_internal.h:96:40: warning: ‘struct rte_pci_device’ declared inside
parameter list [enabled by default]
 void vtophys_pci_device_removed(struct rte_pci_device *pci_device);
                                        ^
make[2]: *** [env.o] Error 1
make[1]: *** [env_dpdk] Error 2
make: *** [lib] Error 2



So basically, SPDK needs some updates since it has its own pci drivers.
I copied some SPDK folks for info.

*Disclaimer* I only checked it links fine against my 21.11 dpdk env,
and did not test the other cases:

diff --git a/dpdkbuild/Makefile b/dpdkbuild/Makefile
index d51b1a6e5..0e666735d 100644
--- a/dpdkbuild/Makefile
+++ b/dpdkbuild/Makefile
@@ -166,6 +166,7 @@ all: $(SPDK_ROOT_DIR)/dpdk/build-tmp
 $(SPDK_ROOT_DIR)/dpdk/build-tmp: $(SPDK_ROOT_DIR)/mk/cc.mk
$(SPDK_ROOT_DIR)/include/spdk/config.h
        $(Q)rm -rf $(SPDK_ROOT_DIR)/dpdk/build $(SPDK_ROOT_DIR)/dpdk/build-tmp
        $(Q)cd "$(SPDK_ROOT_DIR)/dpdk"; CC="$(SUB_CC)" meson
--prefix="$(MESON_PREFIX)" --libdir lib -Dc_args="$(DPDK_CFLAGS)"
-Dc_link_args="$(DPDK_LDFLAGS)" $(DPDK_OPTS)
-Ddisable_drivers="$(shell echo $(DPDK_DISABLED_DRVERS) | sed -E "s/
+/,/g")" build-tmp
+       $(Q)! meson configure build-tmp | grep -qw enable_driver_sdk
|| meson configure build-tmp -Denable_driver_sdk=true
        $(Q)sed $(SED_INPLACE_FLAG) 's/#define RTE_EAL_PMD_PATH
.*/#define RTE_EAL_PMD_PATH ""/g'
$(SPDK_ROOT_DIR)/dpdk/build-tmp/rte_build_config.h
        $(Q) \
        # TODO Meson build adds libbsd dependency when it's available.
This means any app will be \
diff --git a/lib/env_dpdk/env.mk b/lib/env_dpdk/env.mk
index cc7db8aab..e24c6942f 100644bits with an embedded dpdk
--- a/lib/env_dpdk/env.mk
+++ b/lib/env_dpdk/env.mk
@@ -172,6 +172,12 @@ DPDK_PRIVATE_LINKER_ARGS += -lnuma
 endif
 endif

+ifneq (,$(wildcard $(DPDK_INC_DIR)/rte_build_config.h))
+ifneq (,$(shell grep -e "define RTE_HAS_LIBARCHIVE 1"
$(DPDK_INC_DIR)/rte_build_config.h))
+DPDK_PRIVATE_LINKER_ARGS += -larchive
+endif
+endif
+
 ifeq ($(OS),Linux)
 DPDK_PRIVATE_LINKER_ARGS += -ldl
 endif
diff --git a/lib/env_dpdk/env_internal.h b/lib/env_dpdk/env_internal.h
index 2303f432c..24b377545 100644
--- a/lib/env_dpdk/env_internal.h
+++ b/lib/env_dpdk/env_internal.h
@@ -43,13 +43,18 @@
 #include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_pci.h>
-#include <rte_bus_pci.h>
 #include <rte_dev.h>

 #if RTE_VERSION < RTE_VERSION_NUM(19, 11, 0, 0)
 #error RTE_VERSION is too old! Minimum 19.11 is required.
 #endif

+#if RTE_VERSION < RTE_VERSION_NUM(21, 11, 0, 0)
+#include <rte_bus_pci.h>
+#else
+#include <pci_driver.h>
+#endif
+
 /* x86-64 and ARM userspace virtual addresses use only the low 48 bits [0..47],
  * which is enough to cover 256 TB.
  */



-- 
David Marchand

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Harris, James R]

Adding Changpeng Liu from SPDK side.

On 10/4/21, 6:48 AM, "David Marchand" <david.marchand@redhat.com> wrote:

    On Thu, Sep 30, 2021 at 10:45 AM David Marchand
    <david.marchand@redhat.com> wrote:
    > On Wed, Sep 29, 2021 at 9:38 AM Xia, Chenbo <chenbo.xia@intel.com> wrote:
    > > @David, could you help me understand what is the compile error in Fedora 31?
    > > DPDK_compile_spdk failure is expected as the header name for SPDK is changed,
    > > I am not sure if it's the same error...
    >
    > The error log is odd (no compilation "backtrace").
    > You'll need to test spdk manually I guess.

    Tried your series with SPDK (w/o and w/ enable_driver_sdk).
    I think the same, and the error is likely due to the file rename.

    $ make
      CC lib/env_dpdk/env.o
    In file included from env.c:39:0:
    env_internal.h:64:25: error: field ‘driver’ has incomplete type
      struct rte_pci_driver  driver;
                             ^
    env_internal.h:75:59: warning: ‘struct rte_pci_device’ declared inside
    parameter list [enabled by default]
     int pci_device_init(struct rte_pci_driver *driver, struct
    rte_pci_device *device);
                                                               ^
    env_internal.h:75:59: warning: its scope is only this definition or
    declaration, which is probably not what you want [enabled by default]
    env_internal.h:76:28: warning: ‘struct rte_pci_device’ declared inside
    parameter list [enabled by default]
     int pci_device_fini(struct rte_pci_device *device);
                                ^
    env_internal.h:89:38: warning: ‘struct rte_pci_device’ declared inside
    parameter list [enabled by default]
     void vtophys_pci_device_added(struct rte_pci_device *pci_device);
                                          ^
    env_internal.h:96:40: warning: ‘struct rte_pci_device’ declared inside
    parameter list [enabled by default]
     void vtophys_pci_device_removed(struct rte_pci_device *pci_device);
                                            ^
    make[2]: *** [env.o] Error 1
    make[1]: *** [env_dpdk] Error 2
    make: *** [lib] Error 2



    So basically, SPDK needs some updates since it has its own pci drivers.
    I copied some SPDK folks for info.

    *Disclaimer* I only checked it links fine against my 21.11 dpdk env,
    and did not test the other cases:

    diff --git a/dpdkbuild/Makefile b/dpdkbuild/Makefile
    index d51b1a6e5..0e666735d 100644
    --- a/dpdkbuild/Makefile
    +++ b/dpdkbuild/Makefile
    @@ -166,6 +166,7 @@ all: $(SPDK_ROOT_DIR)/dpdk/build-tmp
     $(SPDK_ROOT_DIR)/dpdk/build-tmp: $(SPDK_ROOT_DIR)/mk/cc.mk
    $(SPDK_ROOT_DIR)/include/spdk/config.h
            $(Q)rm -rf $(SPDK_ROOT_DIR)/dpdk/build $(SPDK_ROOT_DIR)/dpdk/build-tmp
            $(Q)cd "$(SPDK_ROOT_DIR)/dpdk"; CC="$(SUB_CC)" meson
    --prefix="$(MESON_PREFIX)" --libdir lib -Dc_args="$(DPDK_CFLAGS)"
    -Dc_link_args="$(DPDK_LDFLAGS)" $(DPDK_OPTS)
    -Ddisable_drivers="$(shell echo $(DPDK_DISABLED_DRVERS) | sed -E "s/
    +/,/g")" build-tmp
    +       $(Q)! meson configure build-tmp | grep -qw enable_driver_sdk
    || meson configure build-tmp -Denable_driver_sdk=true
            $(Q)sed $(SED_INPLACE_FLAG) 's/#define RTE_EAL_PMD_PATH
    .*/#define RTE_EAL_PMD_PATH ""/g'
    $(SPDK_ROOT_DIR)/dpdk/build-tmp/rte_build_config.h
            $(Q) \
            # TODO Meson build adds libbsd dependency when it's available.
    This means any app will be \
    diff --git a/lib/env_dpdk/env.mk b/lib/env_dpdk/env.mk
    index cc7db8aab..e24c6942f 100644bits with an embedded dpdk
    --- a/lib/env_dpdk/env.mk
    +++ b/lib/env_dpdk/env.mk
    @@ -172,6 +172,12 @@ DPDK_PRIVATE_LINKER_ARGS += -lnuma
     endif
     endif

    +ifneq (,$(wildcard $(DPDK_INC_DIR)/rte_build_config.h))
    +ifneq (,$(shell grep -e "define RTE_HAS_LIBARCHIVE 1"
    $(DPDK_INC_DIR)/rte_build_config.h))
    +DPDK_PRIVATE_LINKER_ARGS += -larchive
    +endif
    +endif
    +
     ifeq ($(OS),Linux)
     DPDK_PRIVATE_LINKER_ARGS += -ldl
     endif
    diff --git a/lib/env_dpdk/env_internal.h b/lib/env_dpdk/env_internal.h
    index 2303f432c..24b377545 100644
    --- a/lib/env_dpdk/env_internal.h
    +++ b/lib/env_dpdk/env_internal.h
    @@ -43,13 +43,18 @@
     #include <rte_eal.h>
     #include <rte_bus.h>
     #include <rte_pci.h>
    -#include <rte_bus_pci.h>
     #include <rte_dev.h>

     #if RTE_VERSION < RTE_VERSION_NUM(19, 11, 0, 0)
     #error RTE_VERSION is too old! Minimum 19.11 is required.
     #endif

    +#if RTE_VERSION < RTE_VERSION_NUM(21, 11, 0, 0)
    +#include <rte_bus_pci.h>
    +#else
    +#include <pci_driver.h>
    +#endif
    +
     /* x86-64 and ARM userspace virtual addresses use only the low 48 bits [0..47],
      * which is enough to cover 256 TB.
      */



    -- 
    David Marchand

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Xia, Chenbo]

Thanks David for helping check this and including SPDK folks!

Hi Changpeng,

Although we have synced about this during last release's deprecation notice,
I’d like to summarize two points for SPDK to change if this patchset applied.

1. The pci bus header for drivers will only be exposed if meson option
'enable_driver_sdk' is added, so SPDK need this DPDK meson option to build.

2. As some functions in pci bus is needed for apps and the rest for drivers,
the header for driver is renamed to pci_driver.h (header for app is rte_bus_pci.h).
So SPDK drivers will need pci_driver.h instead of rte_bus_pci.h starting from DPDK
21.11. David showed some tests he did below.

Could you help check above two updates are fine to SPDK?

Thanks,
Chenbo

> -----Original Message-----
> From: Harris, James R <james.r.harris@intel.com>
> Sent: Monday, October 4, 2021 11:56 PM
> To: David Marchand <david.marchand@redhat.com>; Xia, Chenbo
> <chenbo.xia@intel.com>; Liu, Changpeng <changpeng.liu@intel.com>
> Cc: dev@dpdk.org; ci@dpdk.org; Aaron Conole <aconole@redhat.com>; dpdklab
> <dpdklab@iol.unh.edu>; Zawadzki, Tomasz <tomasz.zawadzki@intel.com>;
> alexeymar@mellanox.com
> Subject: Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs
> 
> Adding Changpeng Liu from SPDK side.
> 
> On 10/4/21, 6:48 AM, "David Marchand" <david.marchand@redhat.com> wrote:
> 
>     On Thu, Sep 30, 2021 at 10:45 AM David Marchand
>     <david.marchand@redhat.com> wrote:
>     > On Wed, Sep 29, 2021 at 9:38 AM Xia, Chenbo <chenbo.xia@intel.com>
> wrote:
>     > > @David, could you help me understand what is the compile error in
> Fedora 31?
>     > > DPDK_compile_spdk failure is expected as the header name for SPDK
> is changed,
>     > > I am not sure if it's the same error...
>     >
>     > The error log is odd (no compilation "backtrace").
>     > You'll need to test spdk manually I guess.
> 
>     Tried your series with SPDK (w/o and w/ enable_driver_sdk).
>     I think the same, and the error is likely due to the file rename.
> 
>     $ make
>       CC lib/env_dpdk/env.o
>     In file included from env.c:39:0:
>     env_internal.h:64:25: error: field ‘driver’ has incomplete type
>       struct rte_pci_driver  driver;
>                              ^
>     env_internal.h:75:59: warning: ‘struct rte_pci_device’ declared inside
>     parameter list [enabled by default]
>      int pci_device_init(struct rte_pci_driver *driver, struct
>     rte_pci_device *device);
>                                                                ^
>     env_internal.h:75:59: warning: its scope is only this definition or
>     declaration, which is probably not what you want [enabled by default]
>     env_internal.h:76:28: warning: ‘struct rte_pci_device’ declared inside
>     parameter list [enabled by default]
>      int pci_device_fini(struct rte_pci_device *device);
>                                 ^
>     env_internal.h:89:38: warning: ‘struct rte_pci_device’ declared inside
>     parameter list [enabled by default]
>      void vtophys_pci_device_added(struct rte_pci_device *pci_device);
>                                           ^
>     env_internal.h:96:40: warning: ‘struct rte_pci_device’ declared inside
>     parameter list [enabled by default]
>      void vtophys_pci_device_removed(struct rte_pci_device *pci_device);
>                                             ^
>     make[2]: *** [env.o] Error 1
>     make[1]: *** [env_dpdk] Error 2
>     make: *** [lib] Error 2
> 
> 
> 
>     So basically, SPDK needs some updates since it has its own pci drivers.
>     I copied some SPDK folks for info.
> 
>     *Disclaimer* I only checked it links fine against my 21.11 dpdk env,
>     and did not test the other cases:
> 
>     diff --git a/dpdkbuild/Makefile b/dpdkbuild/Makefile
>     index d51b1a6e5..0e666735d 100644
>     --- a/dpdkbuild/Makefile
>     +++ b/dpdkbuild/Makefile
>     @@ -166,6 +166,7 @@ all: $(SPDK_ROOT_DIR)/dpdk/build-tmp
>      $(SPDK_ROOT_DIR)/dpdk/build-tmp: $(SPDK_ROOT_DIR)/mk/cc.mk
>     $(SPDK_ROOT_DIR)/include/spdk/config.h
>             $(Q)rm -rf $(SPDK_ROOT_DIR)/dpdk/build
> $(SPDK_ROOT_DIR)/dpdk/build-tmp
>             $(Q)cd "$(SPDK_ROOT_DIR)/dpdk"; CC="$(SUB_CC)" meson
>     --prefix="$(MESON_PREFIX)" --libdir lib -Dc_args="$(DPDK_CFLAGS)"
>     -Dc_link_args="$(DPDK_LDFLAGS)" $(DPDK_OPTS)
>     -Ddisable_drivers="$(shell echo $(DPDK_DISABLED_DRVERS) | sed -E "s/
>     +/,/g")" build-tmp
>     +       $(Q)! meson configure build-tmp | grep -qw enable_driver_sdk
>     || meson configure build-tmp -Denable_driver_sdk=true
>             $(Q)sed $(SED_INPLACE_FLAG) 's/#define RTE_EAL_PMD_PATH
>     .*/#define RTE_EAL_PMD_PATH ""/g'
>     $(SPDK_ROOT_DIR)/dpdk/build-tmp/rte_build_config.h
>             $(Q) \
>             # TODO Meson build adds libbsd dependency when it's available.
>     This means any app will be \
>     diff --git a/lib/env_dpdk/env.mk b/lib/env_dpdk/env.mk
>     index cc7db8aab..e24c6942f 100644bits with an embedded dpdk
>     --- a/lib/env_dpdk/env.mk
>     +++ b/lib/env_dpdk/env.mk
>     @@ -172,6 +172,12 @@ DPDK_PRIVATE_LINKER_ARGS += -lnuma
>      endif
>      endif
> 
>     +ifneq (,$(wildcard $(DPDK_INC_DIR)/rte_build_config.h))
>     +ifneq (,$(shell grep -e "define RTE_HAS_LIBARCHIVE 1"
>     $(DPDK_INC_DIR)/rte_build_config.h))
>     +DPDK_PRIVATE_LINKER_ARGS += -larchive
>     +endif
>     +endif
>     +
>      ifeq ($(OS),Linux)
>      DPDK_PRIVATE_LINKER_ARGS += -ldl
>      endif
>     diff --git a/lib/env_dpdk/env_internal.h b/lib/env_dpdk/env_internal.h
>     index 2303f432c..24b377545 100644
>     --- a/lib/env_dpdk/env_internal.h
>     +++ b/lib/env_dpdk/env_internal.h
>     @@ -43,13 +43,18 @@
>      #include <rte_eal.h>
>      #include <rte_bus.h>
>      #include <rte_pci.h>
>     -#include <rte_bus_pci.h>
>      #include <rte_dev.h>
> 
>      #if RTE_VERSION < RTE_VERSION_NUM(19, 11, 0, 0)
>      #error RTE_VERSION is too old! Minimum 19.11 is required.
>      #endif
> 
>     +#if RTE_VERSION < RTE_VERSION_NUM(21, 11, 0, 0)
>     +#include <rte_bus_pci.h>
>     +#else
>     +#include <pci_driver.h>
>     +#endif
>     +
>      /* x86-64 and ARM userspace virtual addresses use only the low 48
> bits [0..47],
>       * which is enough to cover 256 TB.
>       */
> 
> 
> 
>     --
>     David Marchand
> 

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Liu, Changpeng]

I tried the above DPDK patches, and got the following errors:

pci.c:115:7: error: call to ‘rte_pci_read_config’ declared with attribute error: Symbol is not public ABI
  115 |  rc = rte_pci_read_config(dev->dev_handle, value, len, offset);
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pci.c: In function ‘cfg_write_rte’:
pci.c:125:7: error: call to ‘rte_pci_write_config’ declared with attribute error: Symbol is not public ABI
  125 |  rc = rte_pci_write_config(dev->dev_handle, value, len, offset);
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pci.c: In function ‘register_rte_driver’:
pci.c:375:2: error: call to ‘rte_pci_register’ declared with attribute error: Symbol is not public ABI
  375 |  rte_pci_register(&driver->driver); 

We may use the new added API to replace rte_pci_write_config and rte_pci_read_config, but SPDK
do require rte_pci_register().


> -----Original Message-----
> From: Xia, Chenbo <chenbo.xia@intel.com>
> Sent: Wednesday, October 6, 2021 12:26 PM
> To: Harris, James R <james.r.harris@intel.com>; David Marchand
> <david.marchand@redhat.com>; Liu, Changpeng <changpeng.liu@intel.com>
> Cc: dev@dpdk.org; ci@dpdk.org; Aaron Conole <aconole@redhat.com>; dpdklab
> <dpdklab@iol.unh.edu>; Zawadzki, Tomasz <tomasz.zawadzki@intel.com>;
> alexeymar@mellanox.com
> Subject: RE: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs
> 
> Thanks David for helping check this and including SPDK folks!
> 
> Hi Changpeng,
> 
> Although we have synced about this during last release's deprecation notice,
> I’d like to summarize two points for SPDK to change if this patchset applied.
> 
> 1. The pci bus header for drivers will only be exposed if meson option
> 'enable_driver_sdk' is added, so SPDK need this DPDK meson option to build.
> 
> 2. As some functions in pci bus is needed for apps and the rest for drivers,
> the header for driver is renamed to pci_driver.h (header for app is rte_bus_pci.h).
> So SPDK drivers will need pci_driver.h instead of rte_bus_pci.h starting from
> DPDK
> 21.11. David showed some tests he did below.
> 
> Could you help check above two updates are fine to SPDK?
> 
> Thanks,
> Chenbo
> 
> > -----Original Message-----
> > From: Harris, James R <james.r.harris@intel.com>
> > Sent: Monday, October 4, 2021 11:56 PM
> > To: David Marchand <david.marchand@redhat.com>; Xia, Chenbo
> > <chenbo.xia@intel.com>; Liu, Changpeng <changpeng.liu@intel.com>
> > Cc: dev@dpdk.org; ci@dpdk.org; Aaron Conole <aconole@redhat.com>;
> dpdklab
> > <dpdklab@iol.unh.edu>; Zawadzki, Tomasz <tomasz.zawadzki@intel.com>;
> > alexeymar@mellanox.com
> > Subject: Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs
> >
> > Adding Changpeng Liu from SPDK side.
> >
> > On 10/4/21, 6:48 AM, "David Marchand" <david.marchand@redhat.com>
> wrote:
> >
> >     On Thu, Sep 30, 2021 at 10:45 AM David Marchand
> >     <david.marchand@redhat.com> wrote:
> >     > On Wed, Sep 29, 2021 at 9:38 AM Xia, Chenbo <chenbo.xia@intel.com>
> > wrote:
> >     > > @David, could you help me understand what is the compile error in
> > Fedora 31?
> >     > > DPDK_compile_spdk failure is expected as the header name for SPDK
> > is changed,
> >     > > I am not sure if it's the same error...
> >     >
> >     > The error log is odd (no compilation "backtrace").
> >     > You'll need to test spdk manually I guess.
> >
> >     Tried your series with SPDK (w/o and w/ enable_driver_sdk).
> >     I think the same, and the error is likely due to the file rename.
> >
> >     $ make
> >       CC lib/env_dpdk/env.o
> >     In file included from env.c:39:0:
> >     env_internal.h:64:25: error: field ‘driver’ has incomplete type
> >       struct rte_pci_driver  driver;
> >                              ^
> >     env_internal.h:75:59: warning: ‘struct rte_pci_device’ declared inside
> >     parameter list [enabled by default]
> >      int pci_device_init(struct rte_pci_driver *driver, struct
> >     rte_pci_device *device);
> >                                                                ^
> >     env_internal.h:75:59: warning: its scope is only this definition or
> >     declaration, which is probably not what you want [enabled by default]
> >     env_internal.h:76:28: warning: ‘struct rte_pci_device’ declared inside
> >     parameter list [enabled by default]
> >      int pci_device_fini(struct rte_pci_device *device);
> >                                 ^
> >     env_internal.h:89:38: warning: ‘struct rte_pci_device’ declared inside
> >     parameter list [enabled by default]
> >      void vtophys_pci_device_added(struct rte_pci_device *pci_device);
> >                                           ^
> >     env_internal.h:96:40: warning: ‘struct rte_pci_device’ declared inside
> >     parameter list [enabled by default]
> >      void vtophys_pci_device_removed(struct rte_pci_device *pci_device);
> >                                             ^
> >     make[2]: *** [env.o] Error 1
> >     make[1]: *** [env_dpdk] Error 2
> >     make: *** [lib] Error 2
> >
> >
> >
> >     So basically, SPDK needs some updates since it has its own pci drivers.
> >     I copied some SPDK folks for info.
> >
> >     *Disclaimer* I only checked it links fine against my 21.11 dpdk env,
> >     and did not test the other cases:
> >
> >     diff --git a/dpdkbuild/Makefile b/dpdkbuild/Makefile
> >     index d51b1a6e5..0e666735d 100644
> >     --- a/dpdkbuild/Makefile
> >     +++ b/dpdkbuild/Makefile
> >     @@ -166,6 +166,7 @@ all: $(SPDK_ROOT_DIR)/dpdk/build-tmp
> >      $(SPDK_ROOT_DIR)/dpdk/build-tmp: $(SPDK_ROOT_DIR)/mk/cc.mk
> >     $(SPDK_ROOT_DIR)/include/spdk/config.h
> >             $(Q)rm -rf $(SPDK_ROOT_DIR)/dpdk/build
> > $(SPDK_ROOT_DIR)/dpdk/build-tmp
> >             $(Q)cd "$(SPDK_ROOT_DIR)/dpdk"; CC="$(SUB_CC)" meson
> >     --prefix="$(MESON_PREFIX)" --libdir lib -Dc_args="$(DPDK_CFLAGS)"
> >     -Dc_link_args="$(DPDK_LDFLAGS)" $(DPDK_OPTS)
> >     -Ddisable_drivers="$(shell echo $(DPDK_DISABLED_DRVERS) | sed -E "s/
> >     +/,/g")" build-tmp
> >     +       $(Q)! meson configure build-tmp | grep -qw enable_driver_sdk
> >     || meson configure build-tmp -Denable_driver_sdk=true
> >             $(Q)sed $(SED_INPLACE_FLAG) 's/#define RTE_EAL_PMD_PATH
> >     .*/#define RTE_EAL_PMD_PATH ""/g'
> >     $(SPDK_ROOT_DIR)/dpdk/build-tmp/rte_build_config.h
> >             $(Q) \
> >             # TODO Meson build adds libbsd dependency when it's available.
> >     This means any app will be \
> >     diff --git a/lib/env_dpdk/env.mk b/lib/env_dpdk/env.mk
> >     index cc7db8aab..e24c6942f 100644bits with an embedded dpdk
> >     --- a/lib/env_dpdk/env.mk
> >     +++ b/lib/env_dpdk/env.mk
> >     @@ -172,6 +172,12 @@ DPDK_PRIVATE_LINKER_ARGS += -lnuma
> >      endif
> >      endif
> >
> >     +ifneq (,$(wildcard $(DPDK_INC_DIR)/rte_build_config.h))
> >     +ifneq (,$(shell grep -e "define RTE_HAS_LIBARCHIVE 1"
> >     $(DPDK_INC_DIR)/rte_build_config.h))
> >     +DPDK_PRIVATE_LINKER_ARGS += -larchive
> >     +endif
> >     +endif
> >     +
> >      ifeq ($(OS),Linux)
> >      DPDK_PRIVATE_LINKER_ARGS += -ldl
> >      endif
> >     diff --git a/lib/env_dpdk/env_internal.h b/lib/env_dpdk/env_internal.h
> >     index 2303f432c..24b377545 100644
> >     --- a/lib/env_dpdk/env_internal.h
> >     +++ b/lib/env_dpdk/env_internal.h
> >     @@ -43,13 +43,18 @@
> >      #include <rte_eal.h>
> >      #include <rte_bus.h>
> >      #include <rte_pci.h>
> >     -#include <rte_bus_pci.h>
> >      #include <rte_dev.h>
> >
> >      #if RTE_VERSION < RTE_VERSION_NUM(19, 11, 0, 0)
> >      #error RTE_VERSION is too old! Minimum 19.11 is required.
> >      #endif
> >
> >     +#if RTE_VERSION < RTE_VERSION_NUM(21, 11, 0, 0)
> >     +#include <rte_bus_pci.h>
> >     +#else
> >     +#include <pci_driver.h>
> >     +#endif
> >     +
> >      /* x86-64 and ARM userspace virtual addresses use only the low 48
> > bits [0..47],
> >       * which is enough to cover 256 TB.
> >       */
> >
> >
> >
> >     --
> >     David Marchand
> >

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[David Marchand]

Hello,

On Fri, Oct 8, 2021 at 8:15 AM Liu, Changpeng <changpeng.liu@intel.com> wrote:
>
> I tried the above DPDK patches, and got the following errors:
>
> pci.c:115:7: error: call to ‘rte_pci_read_config’ declared with attribute error: Symbol is not public ABI
>   115 |  rc = rte_pci_read_config(dev->dev_handle, value, len, offset);
>       |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> pci.c: In function ‘cfg_write_rte’:
> pci.c:125:7: error: call to ‘rte_pci_write_config’ declared with attribute error: Symbol is not public ABI
>   125 |  rc = rte_pci_write_config(dev->dev_handle, value, len, offset);
>       |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> pci.c: In function ‘register_rte_driver’:
> pci.c:375:2: error: call to ‘rte_pci_register’ declared with attribute error: Symbol is not public ABI
>   375 |  rte_pci_register(&driver->driver);

I should have got this warning... but compilation passed fine for me.
Happy you tested it.

>
> We may use the new added API to replace rte_pci_write_config and rte_pci_read_config, but SPDK
> do require rte_pci_register().

Since SPDK has a PCI driver, you'll need to compile code that calls
those PCI driver internal API with ALLOW_INTERNAL_API defined.
You can probably add a #define ALLOW_INTERNAL_API first thing (it's
important to have it defined before including any dpdk header) in
pci.c

Another option, is to add it to lib/env_dpdk/env.mk:ENV_CFLAGS =
$(DPDK_INC) -DALLOW_EXPERIMENTAL_API.

Can someone from SPDK take over this and sync with Chenbo?


Thanks.

-- 
David Marchand

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Liu, Changpeng]

Thanks, I have worked with Chenbo to address this issue before.  After enable the `ALLOW_INTERNAL_API` option, it works now with SPDK.

Another issue raised by Jim Harris is that for distro packaged DPDK, since this option isn't enabled by default, this will not allow SPDK
to use the distro packaged DPDK after this release.

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Friday, October 8, 2021 3:08 PM
> To: Liu, Changpeng <changpeng.liu@intel.com>
> Cc: Xia, Chenbo <chenbo.xia@intel.com>; Harris, James R
> <james.r.harris@intel.com>; dev@dpdk.org; ci@dpdk.org; Aaron Conole
> <aconole@redhat.com>; dpdklab <dpdklab@iol.unh.edu>; Zawadzki, Tomasz
> <tomasz.zawadzki@intel.com>; alexeymar@mellanox.com
> Subject: Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs
> 
> Hello,
> 
> On Fri, Oct 8, 2021 at 8:15 AM Liu, Changpeng <changpeng.liu@intel.com> wrote:
> >
> > I tried the above DPDK patches, and got the following errors:
> >
> > pci.c:115:7: error: call to ‘rte_pci_read_config’ declared with attribute error:
> Symbol is not public ABI
> >   115 |  rc = rte_pci_read_config(dev->dev_handle, value, len, offset);
> >       |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > pci.c: In function ‘cfg_write_rte’:
> > pci.c:125:7: error: call to ‘rte_pci_write_config’ declared with attribute error:
> Symbol is not public ABI
> >   125 |  rc = rte_pci_write_config(dev->dev_handle, value, len, offset);
> >       |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > pci.c: In function ‘register_rte_driver’:
> > pci.c:375:2: error: call to ‘rte_pci_register’ declared with attribute error:
> Symbol is not public ABI
> >   375 |  rte_pci_register(&driver->driver);
> 
> I should have got this warning... but compilation passed fine for me.
> Happy you tested it.
> 
> >
> > We may use the new added API to replace rte_pci_write_config and
> rte_pci_read_config, but SPDK
> > do require rte_pci_register().
> 
> Since SPDK has a PCI driver, you'll need to compile code that calls
> those PCI driver internal API with ALLOW_INTERNAL_API defined.
> You can probably add a #define ALLOW_INTERNAL_API first thing (it's
> important to have it defined before including any dpdk header) in
> pci.c
> 
> Another option, is to add it to lib/env_dpdk/env.mk:ENV_CFLAGS =
> $(DPDK_INC) -DALLOW_EXPERIMENTAL_API.
> 
> Can someone from SPDK take over this and sync with Chenbo?
> 
> 
> Thanks.
> 
> --
> David Marchand

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Xia, Chenbo]

Hi David & Changpeng,

> -----Original Message-----
> From: Liu, Changpeng <changpeng.liu@intel.com>
> Sent: Friday, October 8, 2021 3:45 PM
> To: David Marchand <david.marchand@redhat.com>; Harris, James R
> <james.r.harris@intel.com>
> Cc: Xia, Chenbo <chenbo.xia@intel.com>; dev@dpdk.org; ci@dpdk.org; Aaron
> Conole <aconole@redhat.com>; dpdklab <dpdklab@iol.unh.edu>; Zawadzki, Tomasz
> <tomasz.zawadzki@intel.com>; alexeymar@mellanox.com
> Subject: RE: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs
> 
> Thanks, I have worked with Chenbo to address this issue before.  After enable
> the `ALLOW_INTERNAL_API` option, it works now with SPDK.
> 
> Another issue raised by Jim Harris is that for distro packaged DPDK, since
> this option isn't enabled by default, this will not allow SPDK
> to use the distro packaged DPDK after this release.

I think for this problem, we have two options: enable driver sdk by default or
let OSV configure the option when building distros. I'm fine with either option.

@David, What do you think?

Thanks,
Chenbo

> 
> > -----Original Message-----
> > From: David Marchand <david.marchand@redhat.com>
> > Sent: Friday, October 8, 2021 3:08 PM
> > To: Liu, Changpeng <changpeng.liu@intel.com>
> > Cc: Xia, Chenbo <chenbo.xia@intel.com>; Harris, James R
> > <james.r.harris@intel.com>; dev@dpdk.org; ci@dpdk.org; Aaron Conole
> > <aconole@redhat.com>; dpdklab <dpdklab@iol.unh.edu>; Zawadzki, Tomasz
> > <tomasz.zawadzki@intel.com>; alexeymar@mellanox.com
> > Subject: Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs
> >
> > Hello,
> >
> > On Fri, Oct 8, 2021 at 8:15 AM Liu, Changpeng <changpeng.liu@intel.com>
> wrote:
> > >
> > > I tried the above DPDK patches, and got the following errors:
> > >
> > > pci.c:115:7: error: call to ‘rte_pci_read_config’ declared with attribute
> error:
> > Symbol is not public ABI
> > >   115 |  rc = rte_pci_read_config(dev->dev_handle, value, len, offset);
> > >       |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > pci.c: In function ‘cfg_write_rte’:
> > > pci.c:125:7: error: call to ‘rte_pci_write_config’ declared with attribute
> error:
> > Symbol is not public ABI
> > >   125 |  rc = rte_pci_write_config(dev->dev_handle, value, len, offset);
> > >       |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > pci.c: In function ‘register_rte_driver’:
> > > pci.c:375:2: error: call to ‘rte_pci_register’ declared with attribute
> error:
> > Symbol is not public ABI
> > >   375 |  rte_pci_register(&driver->driver);
> >
> > I should have got this warning... but compilation passed fine for me.
> > Happy you tested it.
> >
> > >
> > > We may use the new added API to replace rte_pci_write_config and
> > rte_pci_read_config, but SPDK
> > > do require rte_pci_register().
> >
> > Since SPDK has a PCI driver, you'll need to compile code that calls
> > those PCI driver internal API with ALLOW_INTERNAL_API defined.
> > You can probably add a #define ALLOW_INTERNAL_API first thing (it's
> > important to have it defined before including any dpdk header) in
> > pci.c
> >
> > Another option, is to add it to lib/env_dpdk/env.mk:ENV_CFLAGS =
> > $(DPDK_INC) -DALLOW_EXPERIMENTAL_API.
> >
> > Can someone from SPDK take over this and sync with Chenbo?
> >
> >
> > Thanks.
> >
> > --
> > David Marchand

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Thomas Monjalon]

11/10/2021 08:58, Xia, Chenbo:
> From: Liu, Changpeng <changpeng.liu@intel.com>
> > Another issue raised by Jim Harris is that for distro packaged DPDK, since
> > this option isn't enabled by default, this will not allow SPDK
> > to use the distro packaged DPDK after this release.
> 
> I think for this problem, we have two options: enable driver sdk by default or
> let OSV configure the option when building distros. I'm fine with either option.

The meson option enable_driver_sdk is described as "Install headers to build drivers."
Standard development packages should provide headers to build an application.
This option is for projects extending DPDK drivers out of the tree.
The preferred option is to develop drivers inside DPDK.

If a project needs the special option enable_driver_sdk,
1/ it is not following the recommended approach,
2/ it has to manage the burden of driver compatibility with DPDK,
3/ it can compile DPDK itself.

So I think we neither need to make it a default, nor force distros to enable it.
Am I missing something?

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Harris, James R]

On 10/11/21, 5:55 AM, "Thomas Monjalon" <thomas@monjalon.net> wrote:

    11/10/2021 08:58, Xia, Chenbo:
    > From: Liu, Changpeng <changpeng.liu@intel.com>
    > > Another issue raised by Jim Harris is that for distro packaged DPDK, since
    > > this option isn't enabled by default, this will not allow SPDK
    > > to use the distro packaged DPDK after this release.
    > 
    > I think for this problem, we have two options: enable driver sdk by default or
    > let OSV configure the option when building distros. I'm fine with either option.

    The meson option enable_driver_sdk is described as "Install headers to build drivers."
    Standard development packages should provide headers to build an application.
    This option is for projects extending DPDK drivers out of the tree.
    The preferred option is to develop drivers inside DPDK.

    If a project needs the special option enable_driver_sdk,
    1/ it is not following the recommended approach,
    2/ it has to manage the burden of driver compatibility with DPDK,
    3/ it can compile DPDK itself.

    So I think we neither need to make it a default, nor force distros to enable it.
    Am I missing something?

Hi Thomas,

This preference to develop PCI drivers inside of DPDK seems to be a very recent preference.  enable_driver_sdk was just added in DPDK 21.05, and for building out-of-tree ethdev drivers. But DPDK has always enabled building out-of-tree PCI drivers with its default build configuration - SPDK has relied on these APIs since its inception.

We have always viewed DPDK as being a very useful toolkit for building userspace drivers (especially storage drivers!) that aren't part of DPDK itself.  We hope that continues to be the case.

All of that being said, SPDK already compiles DPDK itself as the default configuration. We maintain a DPDK fork for patches that have not yet hit DPDK upstream. If this gets merged we can document that users building DPDK themselves must set enable_driver_sdk. We can also document to our users that SPDK may not build against distro DPDK packages, once distros pick up these changes.

-Jim

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Thomas Monjalon]

12/10/2021 02:35, Harris, James R:
> On 10/11/21, 5:55 AM, "Thomas Monjalon" <thomas@monjalon.net> wrote:
> 
>     11/10/2021 08:58, Xia, Chenbo:
>     > From: Liu, Changpeng <changpeng.liu@intel.com>
>     > > Another issue raised by Jim Harris is that for distro packaged DPDK, since
>     > > this option isn't enabled by default, this will not allow SPDK
>     > > to use the distro packaged DPDK after this release.
>     > 
>     > I think for this problem, we have two options: enable driver sdk by default or
>     > let OSV configure the option when building distros. I'm fine with either option.
> 
>     The meson option enable_driver_sdk is described as "Install headers to build drivers."
>     Standard development packages should provide headers to build an application.
>     This option is for projects extending DPDK drivers out of the tree.
>     The preferred option is to develop drivers inside DPDK.
> 
>     If a project needs the special option enable_driver_sdk,
>     1/ it is not following the recommended approach,
>     2/ it has to manage the burden of driver compatibility with DPDK,
>     3/ it can compile DPDK itself.
> 
>     So I think we neither need to make it a default, nor force distros to enable it.
>     Am I missing something?
> 
> Hi Thomas,
> 
> This preference to develop PCI drivers inside of DPDK seems to be a very recent preference.  enable_driver_sdk was just added in DPDK 21.05, and for building out-of-tree ethdev drivers. But DPDK has always enabled building out-of-tree PCI drivers with its default build configuration - SPDK has relied on these APIs since its inception.

Yes DPDK allows out-of-tree drivers, but it has never been recommended.
We have introduced enable_driver_sdk option recently to keep allowing out-of-tree drivers.

> We have always viewed DPDK as being a very useful toolkit for building userspace drivers (especially storage drivers!) that aren't part of DPDK itself.  We hope that continues to be the case.

Yes, there is no plan to stop that, but also no plan to make it easier.

> All of that being said, SPDK already compiles DPDK itself as the default configuration. We maintain a DPDK fork for patches that have not yet hit DPDK upstream. If this gets merged we can document that users building DPDK themselves must set enable_driver_sdk. We can also document to our users that SPDK may not build against distro DPDK packages, once distros pick up these changes.

Yes I think that's the right thing to do.

Note: I don't remember the reason to keep your drivers out of DPDK?

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Walker, Benjamin]

> From: dev <dev-bounces@dpdk.org> On Behalf Of Thomas Monjalon
> 
> 12/10/2021 02:35, Harris, James R:
> > On 10/11/21, 5:55 AM, "Thomas Monjalon" <thomas@monjalon.net> wrote:
> >
> >     The meson option enable_driver_sdk is described as "Install headers to build
> drivers."
> >     Standard development packages should provide headers to build an
> application.
> >     This option is for projects extending DPDK drivers out of the tree.
> >     The preferred option is to develop drivers inside DPDK.
> >
> >     If a project needs the special option enable_driver_sdk,
> >     1/ it is not following the recommended approach,
> >     2/ it has to manage the burden of driver compatibility with DPDK,
> >     3/ it can compile DPDK itself.
> >
> >     So I think we neither need to make it a default, nor force distros to enable it.
> >     Am I missing something?
> >
> > Hi Thomas,
> >
> > This preference to develop PCI drivers inside of DPDK seems to be a very
> recent preference.  enable_driver_sdk was just added in DPDK 21.05, and for
> building out-of-tree ethdev drivers. But DPDK has always enabled building out-
> of-tree PCI drivers with its default build configuration - SPDK has relied on these
> APIs since its inception.
> 
> Yes DPDK allows out-of-tree drivers, but it has never been recommended.

For networking drivers, maybe. But certainly years and years ago when SPDK was started no one recommended putting an nvme driver into DPDK.

> We have introduced enable_driver_sdk option recently to keep allowing out-of-
> tree drivers.
> 
> > We have always viewed DPDK as being a very useful toolkit for building
> userspace drivers (especially storage drivers!) that aren't part of DPDK itself.  We
> hope that continues to be the case.
> 
> Yes, there is no plan to stop that, but also no plan to make it easier.

To be clear, this change actively makes it harder. DPDK has changed the longstanding status quo.

> 
> > All of that being said, SPDK already compiles DPDK itself as the default
> configuration. We maintain a DPDK fork for patches that have not yet hit DPDK
> upstream. If this gets merged we can document that users building DPDK
> themselves must set enable_driver_sdk. We can also document to our users that
> SPDK may not build against distro DPDK packages, once distros pick up these
> changes.
> 
> Yes I think that's the right thing to do.
> 

This means that a distro-packaged SPDK cannot exist, because it cannot use a distro-packaged DPDK as a dependency. While using a distro-packaged SPDK is not the common case (people just build it themselves), my personal view is that we need to be able to support this and this change from DPDK is unacceptable.

> Note: I don't remember the reason to keep your drivers out of DPDK?

SPDK uses DPDK as a framework for writing user space drivers only - scanning the PCI bus, allocating DMA-safe memory, etc. This functionality is hidden behind an abstraction layer that can be reimplemented by our users to remove the DPDK dependency entirely, and real production users have elected to do this. The reasons they do this are varied, but the shortest way to say it is that DPDK is a framework that requires their application to buy-in across the board, whereas SPDK is a set of libraries that integrates into their existing application more easily.

SPDK simply uses DPDK as the default implementation for this functionality. We cannot port our drivers into DPDK or it would break this use case. 

Thanks,
Ben

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Thomas Monjalon]

12/10/2021 18:59, Walker, Benjamin:
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Thomas Monjalon
> > 12/10/2021 02:35, Harris, James R:
> > > On 10/11/21, 5:55 AM, "Thomas Monjalon" <thomas@monjalon.net> wrote:
> > >
> > >     The meson option enable_driver_sdk is described as "Install headers to build
> > drivers."
> > >     Standard development packages should provide headers to build an
> > application.
> > >     This option is for projects extending DPDK drivers out of the tree.
> > >     The preferred option is to develop drivers inside DPDK.
> > >
> > >     If a project needs the special option enable_driver_sdk,
> > >     1/ it is not following the recommended approach,
> > >     2/ it has to manage the burden of driver compatibility with DPDK,
> > >     3/ it can compile DPDK itself.
> > >
> > >     So I think we neither need to make it a default, nor force distros to enable it.
> > >     Am I missing something?
> > >
> > > Hi Thomas,
> > >
> > > This preference to develop PCI drivers inside of DPDK seems to be a very
> > recent preference.  enable_driver_sdk was just added in DPDK 21.05, and for
> > building out-of-tree ethdev drivers. But DPDK has always enabled building out-
> > of-tree PCI drivers with its default build configuration - SPDK has relied on these
> > APIs since its inception.
> > 
> > Yes DPDK allows out-of-tree drivers, but it has never been recommended.
> 
> For networking drivers, maybe. But certainly years and years ago when SPDK was started no one recommended putting an nvme driver into DPDK.

No one from SPDK project proposed such thing.
I asked several times in person why that,
and even the DPDK techboard asked for such a merge:
https://mails.dpdk.org/archives/dev/2018-December/120706.html
The reply:
http://inbox.dpdk.org/dev/20181217141030.bhe5pwlqnzb3w3i7@platinum/
Even older question in 2015:
http://inbox.dpdk.org/dev/6421280.5XkMhqyP4M@xps13/


> > We have introduced enable_driver_sdk option recently to keep allowing out-of-
> > tree drivers.
> > 
> > > We have always viewed DPDK as being a very useful toolkit for building
> > userspace drivers (especially storage drivers!) that aren't part of DPDK itself.  We
> > hope that continues to be the case.
> > 
> > Yes, there is no plan to stop that, but also no plan to make it easier.
> 
> To be clear, this change actively makes it harder. DPDK has changed the longstanding status quo.

Yes it requires a compilation option.


> > > All of that being said, SPDK already compiles DPDK itself as the default
> > configuration. We maintain a DPDK fork for patches that have not yet hit DPDK
> > upstream. If this gets merged we can document that users building DPDK
> > themselves must set enable_driver_sdk. We can also document to our users that
> > SPDK may not build against distro DPDK packages, once distros pick up these
> > changes.
> > 
> > Yes I think that's the right thing to do.
> 
> This means that a distro-packaged SPDK cannot exist, because it cannot use a distro-packaged DPDK as a dependency.

I don't think so.
Once SPDK is packaged, what do you need from DPDK?
I think you need only .so files for some libs like EAL and PCI,
so that's available in the DPDK package, right?

> While using a distro-packaged SPDK is not the common case (people just build it themselves),
> my personal view is that we need to be able to support this and this change from DPDK is unacceptable.

I agree you should be able to package SPDK.


> > Note: I don't remember the reason to keep your drivers out of DPDK?
> 
> SPDK uses DPDK as a framework for writing user space drivers only - scanning the PCI bus, allocating DMA-safe memory, etc. This functionality is hidden behind an abstraction layer that can be reimplemented by our users to remove the DPDK dependency entirely, and real production users have elected to do this. The reasons they do this are varied, but the shortest way to say it is that DPDK is a framework that requires their application to buy-in across the board, whereas SPDK is a set of libraries that integrates into their existing application more easily.
> 
> SPDK simply uses DPDK as the default implementation for this functionality. We cannot port our drivers into DPDK or it would break this use case.
> 
> Thanks,
> Ben

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Walker, Benjamin]

> From: Thomas Monjalon <thomas@monjalon.net>
> 12/10/2021 18:59, Walker, Benjamin:
> > For networking drivers, maybe. But certainly years and years ago when SPDK
> was started no one recommended putting an nvme driver into DPDK.
> 
> No one from SPDK project proposed such thing.
> I asked several times in person why that, and even the DPDK techboard asked for
> such a merge:
> https://mails.dpdk.org/archives/dev/2018-December/120706.html
> The reply:
> http://inbox.dpdk.org/dev/20181217141030.bhe5pwlqnzb3w3i7@platinum/
> Even older question in 2015:
> http://inbox.dpdk.org/dev/6421280.5XkMhqyP4M@xps13/
> 

For my part in these discussions, it was always about merging the governance of the projects rather than the code. I don't think a merger even occurred to anyone I spoke with during that - certainly it didn't to me. SPDK is huge and beyond its use of EAL/PCI doesn't share much in common with the rest of DPDK (SPDK uses lightweight green threading, all virtual addresses, etc.). Anyway, as I pointed out one of our key use cases for several users is the ability to replace DPDK entirely, so merging isn't an option.

> > This means that a distro-packaged SPDK cannot exist, because it cannot use a
> distro-packaged DPDK as a dependency.
> 
> I don't think so.
> Once SPDK is packaged, what do you need from DPDK?
> I think you need only .so files for some libs like EAL and PCI, so that's available in
> the DPDK package, right?
> 

So is DPDK committed to maintaining the existing ABI, such that the necessary symbols are still exported even when enable_driver_sdk is off? This option will, into the foreseeable future, only impact the installation of those header files? If that's the case, we can just copy the header file into SPDK, as could anyone else that wants to continue using DPDK to implement out of tree drivers. Can you clarify if something like this scheme would be considered a supported use of DPDK?

Thanks,
Ben

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Thomas Monjalon]

12/10/2021 21:26, Walker, Benjamin:
> > From: Thomas Monjalon <thomas@monjalon.net>
> > 12/10/2021 18:59, Walker, Benjamin:
> > > For networking drivers, maybe. But certainly years and years ago when SPDK
> > was started no one recommended putting an nvme driver into DPDK.
> > 
> > No one from SPDK project proposed such thing.
> > I asked several times in person why that, and even the DPDK techboard asked for
> > such a merge:
> > https://mails.dpdk.org/archives/dev/2018-December/120706.html
> > The reply:
> > http://inbox.dpdk.org/dev/20181217141030.bhe5pwlqnzb3w3i7@platinum/
> > Even older question in 2015:
> > http://inbox.dpdk.org/dev/6421280.5XkMhqyP4M@xps13/
> > 
> 
> For my part in these discussions, it was always about merging the governance of the projects rather than the code. I don't think a merger even occurred to anyone I spoke with during that - certainly it didn't to me. SPDK is huge and beyond its use of EAL/PCI doesn't share much in common with the rest of DPDK (SPDK uses lightweight green threading, all virtual addresses, etc.). Anyway, as I pointed out one of our key use cases for several users is the ability to replace DPDK entirely, so merging isn't an option.

OK I understand, that's clear.
I would be interesting to know if the NVMe drivers could be split in two parts:
one part in DPDK, and the other part in SPDK for the non-DPDK case.
I ask because it may ease things for DPDK integration in SPDK.
There is probably a cost for the SPDK project, so it could be interesting
to compare pros and cons, if possible at all.


> > > This means that a distro-packaged SPDK cannot exist, because it cannot use a
> > distro-packaged DPDK as a dependency.
> > 
> > I don't think so.
> > Once SPDK is packaged, what do you need from DPDK?
> > I think you need only .so files for some libs like EAL and PCI, so that's available in
> > the DPDK package, right?
> > 
> 
> So is DPDK committed to maintaining the existing ABI,
> such that the necessary symbols are still exported
> even when enable_driver_sdk is off?

Symbols required by drivers are necessarily exported.
Do you think I am missing something?
Do you need EAL internal functions?
We should check which functions are called by SPDK,
because there is a trend to export less functions if not needed.

> This option will, into the foreseeable future,
> only impact the installation of those header files?

I don't see what else it could impact.

> If that's the case, we can just copy the header file into SPDK,
> as could anyone else that wants to continue using DPDK
> to implement out of tree drivers.
> Can you clarify if something like this scheme would be considered a supported use of DPDK?

DPDK can be used by anybody as far as the (permissive) license is respected.
I consider copying files as a source of sync issues, but you are free.

In order to be perfectly clear, all the changes done
around this option enable_driver_sdk share the goal of tidying stuff
in DPDK so that ABI becomes better manageable.
I think that nobody want to annoy the SPDK project.
I understand that the changes effectively add troubles, and I am sorry
about that. If SPDK and other projects can manage with this change, good.
If there is a real blocker, we should discuss what are the options.

Thanks for your understanding

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Walker, Benjamin]

> From: Thomas Monjalon <thomas@monjalon.net>
> 
> In order to be perfectly clear, all the changes done around this option
> enable_driver_sdk share the goal of tidying stuff in DPDK so that ABI becomes
> better manageable.
> I think that nobody want to annoy the SPDK project.
> I understand that the changes effectively add troubles, and I am sorry about
> that. If SPDK and other projects can manage with this change, good.
> If there is a real blocker, we should discuss what are the options.
> 
> Thanks for your understanding

I completely understand the desire to make the ABI manageable. If I were in your shoes, I'd be doing the same exact thing. What I don't currently understand is the motivation behind this enable_driver_sdk option. My guess is that it's one of two things.

\1 ABI manageability: You say that's the purpose above, and that was my initial assumption. But wouldn't that necessarily mean, over time, no longer considering the symbols that were defined by the header files as part of the stable ABI? If you still consider these symbols as part of the ABI in shared library builds, then the enable_driver_sdk option does absolutely nothing to improve the ABI situation, so why bother to have it at all? We can't have packaged SPDK relying on symbols in a packaged DPDK that are not part of the official ABI.
\2 Not supporting out-of-tree drivers: Another option is that you just don't want people writing out of tree drivers. You can't just drop it outright because people already do it, but you'd like to not support it for shared library builds at least.

So I'd like to really understand which of these two motivated the enable_driver_sdk option . Maybe it's not even one of the two above. If it is #1, then I think maybe we can work with DPDK to define a very small set of out-of-tree driver APIs/ABIs that need to continue to exist in the shared libraries by default. I do think SPDK needs only a very small number. If it's #2, then that's the entire SPDK use case and I'd ask you to reconsider the direction.

Thanks,
Ben

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Thomas Monjalon]

13/10/2021 19:56, Walker, Benjamin:
> > From: Thomas Monjalon <thomas@monjalon.net>
> > 
> > In order to be perfectly clear, all the changes done around this option
> > enable_driver_sdk share the goal of tidying stuff in DPDK so that ABI becomes
> > better manageable.
> > I think that nobody want to annoy the SPDK project.
> > I understand that the changes effectively add troubles, and I am sorry about
> > that. If SPDK and other projects can manage with this change, good.
> > If there is a real blocker, we should discuss what are the options.
> > 
> > Thanks for your understanding
> 
> I completely understand the desire to make the ABI manageable. If I were in your shoes, I'd be doing the same exact thing. What I don't currently understand is the motivation behind this enable_driver_sdk option. My guess is that it's one of two things.
> 
> \1 ABI manageability: You say that's the purpose above, and that was my initial assumption. But wouldn't that necessarily mean, over time, no longer considering the symbols that were defined by the header files as part of the stable ABI?

Absolutely. The idea is that we don't guarantee ABI for the drivers.

> If you still consider these symbols as part of the ABI in shared library builds, then the enable_driver_sdk option does absolutely nothing to improve the ABI situation, so why bother to have it at all? We can't have packaged SPDK relying on symbols in a packaged DPDK that are not part of the official ABI.

> \2 Not supporting out-of-tree drivers: Another option is that you just don't want people writing out of tree drivers.

We don't want complications due to support of out-of-tree drivers,
but we don't want to forbid them.

> You can't just drop it outright because people already do it,
> but you'd like to not support it for shared library builds at least.

I didn't think about it in these terms.
But saying we don't offer compatibility for shared library drivers
is not too far of "no support" indeed.

> So I'd like to really understand which of these two motivated the enable_driver_sdk option . Maybe it's not even one of the two above. If it is #1, then I think maybe we can work with DPDK to define a very small set of out-of-tree driver APIs/ABIs that need to continue to exist in the shared libraries by default. I do think SPDK needs only a very small number. If it's #2, then that's the entire SPDK use case and I'd ask you to reconsider the direction.

Yes I think we need to agree on functions to keep as-is for compatibility.
Waiting for your input please.

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Walker, Benjamin]

> From: Thomas Monjalon <thomas@monjalon.net>
 
> Yes I think we need to agree on functions to keep as-is for compatibility.
> Waiting for your input please.

We've added a task to our backlog to propose a stable ABI for out of tree drivers here. It's not as simple as just keeping a couple of the existing functions - we're currently manipulating structures directly. We'll need to work a bit to design the simplest possible set of functions that we need.

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Thomas Monjalon]

14/10/2021 00:48, Walker, Benjamin:
> > From: Thomas Monjalon <thomas@monjalon.net>
>  
> > Yes I think we need to agree on functions to keep as-is for compatibility.
> > Waiting for your input please.
> 
> We've added a task to our backlog to propose a stable ABI for out of tree drivers here. It's not as simple as just keeping a couple of the existing functions - we're currently manipulating structures directly. We'll need to work a bit to design the simplest possible set of functions that we need.

OK thanks

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Thomas Monjalon]

9 months have passed. Do you have any news from SPDK side?


14/10/2021 08:41, Thomas Monjalon:
> 14/10/2021 00:48, Walker, Benjamin:
> > > From: Thomas Monjalon <thomas@monjalon.net>
> >  
> > > Yes I think we need to agree on functions to keep as-is for compatibility.
> > > Waiting for your input please.
> > 
> > We've added a task to our backlog to propose a stable ABI for out of tree drivers here. It's not as simple as just keeping a couple of the existing functions - we're currently manipulating structures directly. We'll need to work a bit to design the simplest possible set of functions that we need.
> 
> OK thanks

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Xia, Chenbo]

Hi Thomas,

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Thursday, October 14, 2021 3:00 AM
> To: Harris, James R <james.r.harris@intel.com>; Walker, Benjamin
> <benjamin.walker@intel.com>
> Cc: Liu, Changpeng <changpeng.liu@intel.com>; Xia, Chenbo
> <chenbo.xia@intel.com>; David Marchand <david.marchand@redhat.com>;
> dev@dpdk.org; Aaron Conole <aconole@redhat.com>; Zawadzki, Tomasz
> <tomasz.zawadzki@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs
> 
> 13/10/2021 19:56, Walker, Benjamin:
> > > From: Thomas Monjalon <thomas@monjalon.net>
> > >
> > > In order to be perfectly clear, all the changes done around this option
> > > enable_driver_sdk share the goal of tidying stuff in DPDK so that ABI
> becomes
> > > better manageable.
> > > I think that nobody want to annoy the SPDK project.
> > > I understand that the changes effectively add troubles, and I am sorry
> about
> > > that. If SPDK and other projects can manage with this change, good.
> > > If there is a real blocker, we should discuss what are the options.
> > >
> > > Thanks for your understanding
> >
> > I completely understand the desire to make the ABI manageable. If I were in
> your shoes, I'd be doing the same exact thing. What I don't currently
> understand is the motivation behind this enable_driver_sdk option. My guess is
> that it's one of two things.
> >
> > \1 ABI manageability: You say that's the purpose above, and that was my
> initial assumption. But wouldn't that necessarily mean, over time, no longer
> considering the symbols that were defined by the header files as part of the
> stable ABI?
> 
> Absolutely. The idea is that we don't guarantee ABI for the drivers.
> 
> > If you still consider these symbols as part of the ABI in shared library
> builds, then the enable_driver_sdk option does absolutely nothing to improve
> the ABI situation, so why bother to have it at all? We can't have packaged
> SPDK relying on symbols in a packaged DPDK that are not part of the official
> ABI.
> 
> > \2 Not supporting out-of-tree drivers: Another option is that you just don't
> want people writing out of tree drivers.
> 
> We don't want complications due to support of out-of-tree drivers,
> but we don't want to forbid them.
> 
> > You can't just drop it outright because people already do it,
> > but you'd like to not support it for shared library builds at least.
> 
> I didn't think about it in these terms.
> But saying we don't offer compatibility for shared library drivers
> is not too far of "no support" indeed.
> 
> > So I'd like to really understand which of these two motivated the
> enable_driver_sdk option . Maybe it's not even one of the two above. If it is
> #1, then I think maybe we can work with DPDK to define a very small set of
> out-of-tree driver APIs/ABIs that need to continue to exist in the shared
> libraries by default. I do think SPDK needs only a very small number. If it's
> #2, then that's the entire SPDK use case and I'd ask you to reconsider the
> direction.
> 
> Yes I think we need to agree on functions to keep as-is for compatibility.
> Waiting for your input please.

So, do you mean currently DPDK doesn't guarantee ABI for drivers but could have
driver ABI in the future?

Thanks,
Chenbo

> 

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Thomas Monjalon]

14/10/2021 04:21, Xia, Chenbo:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 13/10/2021 19:56, Walker, Benjamin:
> > > > From: Thomas Monjalon <thomas@monjalon.net>
> > > >
> > > > In order to be perfectly clear, all the changes done around this option
> > > > enable_driver_sdk share the goal of tidying stuff in DPDK so that ABI
> > becomes
> > > > better manageable.
> > > > I think that nobody want to annoy the SPDK project.
> > > > I understand that the changes effectively add troubles, and I am sorry
> > about
> > > > that. If SPDK and other projects can manage with this change, good.
> > > > If there is a real blocker, we should discuss what are the options.
> > > >
> > > > Thanks for your understanding
> > >
> > > I completely understand the desire to make the ABI manageable. If I were in
> > your shoes, I'd be doing the same exact thing. What I don't currently
> > understand is the motivation behind this enable_driver_sdk option. My guess is
> > that it's one of two things.
> > >
> > > \1 ABI manageability: You say that's the purpose above, and that was my
> > initial assumption. But wouldn't that necessarily mean, over time, no longer
> > considering the symbols that were defined by the header files as part of the
> > stable ABI?
> > 
> > Absolutely. The idea is that we don't guarantee ABI for the drivers.
> > 
> > > If you still consider these symbols as part of the ABI in shared library
> > builds, then the enable_driver_sdk option does absolutely nothing to improve
> > the ABI situation, so why bother to have it at all? We can't have packaged
> > SPDK relying on symbols in a packaged DPDK that are not part of the official
> > ABI.
> > 
> > > \2 Not supporting out-of-tree drivers: Another option is that you just don't
> > want people writing out of tree drivers.
> > 
> > We don't want complications due to support of out-of-tree drivers,
> > but we don't want to forbid them.
> > 
> > > You can't just drop it outright because people already do it,
> > > but you'd like to not support it for shared library builds at least.
> > 
> > I didn't think about it in these terms.
> > But saying we don't offer compatibility for shared library drivers
> > is not too far of "no support" indeed.
> > 
> > > So I'd like to really understand which of these two motivated the
> > enable_driver_sdk option . Maybe it's not even one of the two above. If it is
> > #1, then I think maybe we can work with DPDK to define a very small set of
> > out-of-tree driver APIs/ABIs that need to continue to exist in the shared
> > libraries by default. I do think SPDK needs only a very small number. If it's
> > #2, then that's the entire SPDK use case and I'd ask you to reconsider the
> > direction.
> > 
> > Yes I think we need to agree on functions to keep as-is for compatibility.
> > Waiting for your input please.
> 
> So, do you mean currently DPDK doesn't guarantee ABI for drivers

Yes

> but could have driver ABI in the future?

I don't think so, not general compatibility,
but we can think about a way to avoid breaking SPDK specifically,
which has less requirements.

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Xia, Chenbo]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Thursday, October 14, 2021 2:42 PM
> To: Harris, James R <james.r.harris@intel.com>; Walker, Benjamin
> <benjamin.walker@intel.com>; Xia, Chenbo <chenbo.xia@intel.com>
> Cc: Liu, Changpeng <changpeng.liu@intel.com>; David Marchand
> <david.marchand@redhat.com>; dev@dpdk.org; Aaron Conole <aconole@redhat.com>;
> Zawadzki, Tomasz <tomasz.zawadzki@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs
> 
> 14/10/2021 04:21, Xia, Chenbo:
> > From: Thomas Monjalon <thomas@monjalon.net>
> > > 13/10/2021 19:56, Walker, Benjamin:
> > > > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > >
> > > > > In order to be perfectly clear, all the changes done around this
> option
> > > > > enable_driver_sdk share the goal of tidying stuff in DPDK so that ABI
> > > becomes
> > > > > better manageable.
> > > > > I think that nobody want to annoy the SPDK project.
> > > > > I understand that the changes effectively add troubles, and I am sorry
> > > about
> > > > > that. If SPDK and other projects can manage with this change, good.
> > > > > If there is a real blocker, we should discuss what are the options.
> > > > >
> > > > > Thanks for your understanding
> > > >
> > > > I completely understand the desire to make the ABI manageable. If I were
> in
> > > your shoes, I'd be doing the same exact thing. What I don't currently
> > > understand is the motivation behind this enable_driver_sdk option. My
> guess is
> > > that it's one of two things.
> > > >
> > > > \1 ABI manageability: You say that's the purpose above, and that was my
> > > initial assumption. But wouldn't that necessarily mean, over time, no
> longer
> > > considering the symbols that were defined by the header files as part of
> the
> > > stable ABI?
> > >
> > > Absolutely. The idea is that we don't guarantee ABI for the drivers.
> > >
> > > > If you still consider these symbols as part of the ABI in shared library
> > > builds, then the enable_driver_sdk option does absolutely nothing to
> improve
> > > the ABI situation, so why bother to have it at all? We can't have packaged
> > > SPDK relying on symbols in a packaged DPDK that are not part of the
> official
> > > ABI.
> > >
> > > > \2 Not supporting out-of-tree drivers: Another option is that you just
> don't
> > > want people writing out of tree drivers.
> > >
> > > We don't want complications due to support of out-of-tree drivers,
> > > but we don't want to forbid them.
> > >
> > > > You can't just drop it outright because people already do it,
> > > > but you'd like to not support it for shared library builds at least.
> > >
> > > I didn't think about it in these terms.
> > > But saying we don't offer compatibility for shared library drivers
> > > is not too far of "no support" indeed.
> > >
> > > > So I'd like to really understand which of these two motivated the
> > > enable_driver_sdk option . Maybe it's not even one of the two above. If it
> is
> > > #1, then I think maybe we can work with DPDK to define a very small set of
> > > out-of-tree driver APIs/ABIs that need to continue to exist in the shared
> > > libraries by default. I do think SPDK needs only a very small number. If
> it's
> > > #2, then that's the entire SPDK use case and I'd ask you to reconsider the
> > > direction.
> > >
> > > Yes I think we need to agree on functions to keep as-is for compatibility.
> > > Waiting for your input please.
> >
> > So, do you mean currently DPDK doesn't guarantee ABI for drivers
> 
> Yes
> 
> > but could have driver ABI in the future?
> 
> I don't think so, not general compatibility,
> but we can think about a way to avoid breaking SPDK specifically,
> which has less requirements.

So the problem here is exposing some APIs to SPDK directly? Without the 'enable_driver_sdk'
option, I don't see a solution of both exposed and not-ABI. Any idea in your mind?

Thanks,
Chenbo

> 
> 

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Thomas Monjalon]

14/10/2021 09:00, Xia, Chenbo:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 14/10/2021 04:21, Xia, Chenbo:
> > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > Yes I think we need to agree on functions to keep as-is for compatibility.
> > > > Waiting for your input please.
> > >
> > > So, do you mean currently DPDK doesn't guarantee ABI for drivers
> > 
> > Yes
> > 
> > > but could have driver ABI in the future?
> > 
> > I don't think so, not general compatibility,
> > but we can think about a way to avoid breaking SPDK specifically,
> > which has less requirements.
> 
> So the problem here is exposing some APIs to SPDK directly? Without the 'enable_driver_sdk'
> option, I don't see a solution of both exposed and not-ABI. Any idea in your mind?

No the idea is to keep using enable_driver_sdk.
But so far, there is no compatibility guarantee for driver SDK.
The discussion is about which basic compatibility requirement is needed for SPDK.

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Xia, Chenbo]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Thursday, October 14, 2021 3:08 PM
> To: Harris, James R <james.r.harris@intel.com>; Walker, Benjamin
> <benjamin.walker@intel.com>; Xia, Chenbo <chenbo.xia@intel.com>
> Cc: Liu, Changpeng <changpeng.liu@intel.com>; David Marchand
> <david.marchand@redhat.com>; dev@dpdk.org; Aaron Conole <aconole@redhat.com>;
> Zawadzki, Tomasz <tomasz.zawadzki@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs
> 
> 14/10/2021 09:00, Xia, Chenbo:
> > From: Thomas Monjalon <thomas@monjalon.net>
> > > 14/10/2021 04:21, Xia, Chenbo:
> > > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > > Yes I think we need to agree on functions to keep as-is for
> compatibility.
> > > > > Waiting for your input please.
> > > >
> > > > So, do you mean currently DPDK doesn't guarantee ABI for drivers
> > >
> > > Yes
> > >
> > > > but could have driver ABI in the future?
> > >
> > > I don't think so, not general compatibility,
> > > but we can think about a way to avoid breaking SPDK specifically,
> > > which has less requirements.
> >
> > So the problem here is exposing some APIs to SPDK directly? Without the
> 'enable_driver_sdk'
> > option, I don't see a solution of both exposed and not-ABI. Any idea in your
> mind?
> 
> No the idea is to keep using enable_driver_sdk.
> But so far, there is no compatibility guarantee for driver SDK.
> The discussion is about which basic compatibility requirement is needed for
> SPDK.

Sorry for not understanding your point quickly, but what's the difference of
'general compatibility' and 'basic compatibility'? Because in my mind, one
struct or function should either be ABI-compatible or not. Could you help explain
it a bit?

Thanks,
Chenbo

> 
> 

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Thomas Monjalon]

14/10/2021 10:07, Xia, Chenbo:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 14/10/2021 09:00, Xia, Chenbo:
> > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > 14/10/2021 04:21, Xia, Chenbo:
> > > > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > > > Yes I think we need to agree on functions to keep as-is for
> > compatibility.
> > > > > > Waiting for your input please.
> > > > >
> > > > > So, do you mean currently DPDK doesn't guarantee ABI for drivers
> > > >
> > > > Yes
> > > >
> > > > > but could have driver ABI in the future?
> > > >
> > > > I don't think so, not general compatibility,
> > > > but we can think about a way to avoid breaking SPDK specifically,
> > > > which has less requirements.
> > >
> > > So the problem here is exposing some APIs to SPDK directly? Without the
> > 'enable_driver_sdk'
> > > option, I don't see a solution of both exposed and not-ABI. Any idea in your
> > mind?
> > 
> > No the idea is to keep using enable_driver_sdk.
> > But so far, there is no compatibility guarantee for driver SDK.
> > The discussion is about which basic compatibility requirement is needed for
> > SPDK.
> 
> Sorry for not understanding your point quickly, but what's the difference of
> 'general compatibility' and 'basic compatibility'? Because in my mind, one
> struct or function should either be ABI-compatible or not. Could you help explain
> it a bit?

I wonder whether we could have a guarantee for a subset of structs and functions.
Anyway, this is just opening the discussion to collect some inputs first.
Then we'll have to check what is possible and get a techboard approval.

Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs

[Xia, Chenbo]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Thursday, October 14, 2021 4:26 PM
> To: Harris, James R <james.r.harris@intel.com>; Walker, Benjamin
> <benjamin.walker@intel.com>; Xia, Chenbo <chenbo.xia@intel.com>
> Cc: Liu, Changpeng <changpeng.liu@intel.com>; David Marchand
> <david.marchand@redhat.com>; dev@dpdk.org; Aaron Conole <aconole@redhat.com>;
> Zawadzki, Tomasz <tomasz.zawadzki@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v2 0/7] Removal of PCI bus ABIs
> 
> 14/10/2021 10:07, Xia, Chenbo:
> > From: Thomas Monjalon <thomas@monjalon.net>
> > > 14/10/2021 09:00, Xia, Chenbo:
> > > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > > 14/10/2021 04:21, Xia, Chenbo:
> > > > > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > > > > Yes I think we need to agree on functions to keep as-is for
> > > compatibility.
> > > > > > > Waiting for your input please.
> > > > > >
> > > > > > So, do you mean currently DPDK doesn't guarantee ABI for drivers
> > > > >
> > > > > Yes
> > > > >
> > > > > > but could have driver ABI in the future?
> > > > >
> > > > > I don't think so, not general compatibility,
> > > > > but we can think about a way to avoid breaking SPDK specifically,
> > > > > which has less requirements.
> > > >
> > > > So the problem here is exposing some APIs to SPDK directly? Without the
> > > 'enable_driver_sdk'
> > > > option, I don't see a solution of both exposed and not-ABI. Any idea in
> your
> > > mind?
> > >
> > > No the idea is to keep using enable_driver_sdk.
> > > But so far, there is no compatibility guarantee for driver SDK.
> > > The discussion is about which basic compatibility requirement is needed
> for
> > > SPDK.
> >
> > Sorry for not understanding your point quickly, but what's the difference of
> > 'general compatibility' and 'basic compatibility'? Because in my mind, one
> > struct or function should either be ABI-compatible or not. Could you help
> explain
> > it a bit?
> 
> I wonder whether we could have a guarantee for a subset of structs and
> functions.
> Anyway, this is just opening the discussion to collect some inputs first.
> Then we'll have to check what is possible and get a techboard approval.
> 

After going through related code in SPDK, I think we can add some new functions and keep
some macros in the exposed header (i.e., rte_bus_pci.h) for SPDK to register pci driver
and get needed info.

Most structs/marocs will be hided and SPDK can use the new proposed APIs and small set
of macros/structs to build. In this way, the problem of SPDK building with DPDK distros
and ABI issue can both be solved. APIs like struct rte_pci_device and struct rte_pci_driver
can be hided to minimize pci bus ABI.

Thomas & SPDK folks, please share your opinions of above.

Thanks,
Chenbo