[dpdk-dev] [PATCH v13 0/4] raw/ifpga: add extra OPAE APIs

[dpdk-dev] [PATCH v13 0/4] raw/ifpga: add extra OPAE APIs

[Wei Huang]

Cyborg is part of OpenStack, it needs some OPAE APIs to manage
devices with Intel FPGA. The first three patches implement extra
APIs to meet Cyborg requirement. The last patch add an example
to show how to use these APIs.

Main changes from v12:
- implement ifpga APIs in rte_pmd_ifpga.c

Wei Huang (4):
  raw/ifpga: add fpga rsu APIs
  raw/ifpga: add APIs to get fpga information
  raw/ifpga: add miscellaneous APIs
  examples/ifpga: add example for ifpga APIs

 MAINTAINERS                                |    3 +
 doc/api/doxy-api-index.md                  |    3 +-
 doc/guides/sample_app_ug/ifpga.rst         |  387 +++++
 doc/guides/sample_app_ug/index.rst         |    1 +
 drivers/raw/ifpga/base/ifpga_api.c         |   34 +
 drivers/raw/ifpga/base/ifpga_defines.h     |    1 +
 drivers/raw/ifpga/base/ifpga_feature_dev.c |   21 +
 drivers/raw/ifpga/base/ifpga_feature_dev.h |    1 +
 drivers/raw/ifpga/base/ifpga_fme.c         |   36 +-
 drivers/raw/ifpga/base/ifpga_fme_rsu.c     |  428 +++++
 drivers/raw/ifpga/base/ifpga_hw.h          |    1 +
 drivers/raw/ifpga/base/ifpga_sec_mgr.c     |  639 ++++++++
 drivers/raw/ifpga/base/ifpga_sec_mgr.h     |   93 ++
 drivers/raw/ifpga/base/meson.build         |    2 +
 drivers/raw/ifpga/base/opae_hw_api.c       |   77 +
 drivers/raw/ifpga/base/opae_hw_api.h       |   12 +
 drivers/raw/ifpga/base/opae_ifpga_hw_api.h |    1 +
 drivers/raw/ifpga/base/opae_intel_max10.c  |   48 +
 drivers/raw/ifpga/base/opae_intel_max10.h  |   44 +
 drivers/raw/ifpga/ifpga_rawdev.c           |   30 +
 drivers/raw/ifpga/ifpga_rawdev.h           |    7 +-
 drivers/raw/ifpga/meson.build              |    4 +-
 drivers/raw/ifpga/rte_pmd_ifpga.c          |  429 +++++
 drivers/raw/ifpga/rte_pmd_ifpga.h          |  308 ++++
 drivers/raw/ifpga/version.map              |   18 +
 examples/ifpga/Makefile                    |   43 +
 examples/ifpga/commands.c                  | 1294 ++++++++++++++++
 examples/ifpga/commands.h                  |   16 +
 examples/ifpga/main.c                      |   38 +
 examples/ifpga/meson.build                 |   20 +
 examples/ifpga/opae_api.c                  | 1632 ++++++++++++++++++++
 examples/ifpga/opae_api.h                  |  244 +++
 examples/meson.build                       |    2 +-
 33 files changed, 5911 insertions(+), 6 deletions(-)
 create mode 100644 doc/guides/sample_app_ug/ifpga.rst
 create mode 100644 drivers/raw/ifpga/base/ifpga_fme_rsu.c
 create mode 100644 drivers/raw/ifpga/base/ifpga_sec_mgr.c
 create mode 100644 drivers/raw/ifpga/base/ifpga_sec_mgr.h
 create mode 100644 drivers/raw/ifpga/rte_pmd_ifpga.c
 create mode 100644 drivers/raw/ifpga/rte_pmd_ifpga.h
 create mode 100644 examples/ifpga/Makefile
 create mode 100644 examples/ifpga/commands.c
 create mode 100644 examples/ifpga/commands.h
 create mode 100644 examples/ifpga/main.c
 create mode 100644 examples/ifpga/meson.build
 create mode 100644 examples/ifpga/opae_api.c
 create mode 100644 examples/ifpga/opae_api.h

-- 
2.29.2

[dpdk-dev] [PATCH v13 1/4] raw/ifpga: add fpga rsu APIs

[Wei Huang]

RSU (Remote System Update) depends on secure manager which may be
different on various implementations, so a new secure manager device
is implemented for adapting such difference.
There are five APIs added:
1. rte_pmd_ifpga_get_dev_id() get raw device ID of ifpga device from
PCI address like 'Domain:Bus:Dev.Func'.
2. rte_pmd_ifpga_update_flash() update flash with specific image file.
3. rte_pmd_ifpga_stop_update() abort flash update process.
4. rte_pmd_ifpga_reboot_try() check current ifpga status and change it
to reboot status if it is idle.
5. rte_pmd_ifpga_reload() trigger full reconfiguration of ifpga device.

Signed-off-by: Wei Huang <wei.huang@intel.com>
Acked-by: Tianfei Zhang <tianfei.zhang@intel.com>
Acked-by: Rosen Xu <rosen.xu@intel.com>
---
v2: fix coding style issue in ifpga_fme_rsu.c and ifpga_sec_mgr.c
---
v3: fix compilation issues in ifpga_fme_rsu.c
---
v4: fix compilation issues in opae_intel_max10.c
---
v5: implement APIs in rte_pmd_ifpga.c
---
 doc/api/doxy-api-index.md                 |   3 +-
 drivers/raw/ifpga/base/ifpga_api.c        |  26 +
 drivers/raw/ifpga/base/ifpga_fme.c        |   8 +
 drivers/raw/ifpga/base/ifpga_fme_rsu.c    | 428 +++++++++++++++
 drivers/raw/ifpga/base/ifpga_hw.h         |   1 +
 drivers/raw/ifpga/base/ifpga_sec_mgr.c    | 639 ++++++++++++++++++++++
 drivers/raw/ifpga/base/ifpga_sec_mgr.h    |  93 ++++
 drivers/raw/ifpga/base/meson.build        |   2 +
 drivers/raw/ifpga/base/opae_hw_api.c      |  59 ++
 drivers/raw/ifpga/base/opae_hw_api.h      |  10 +
 drivers/raw/ifpga/base/opae_intel_max10.c |  48 ++
 drivers/raw/ifpga/base/opae_intel_max10.h |  44 ++
 drivers/raw/ifpga/meson.build             |   4 +-
 drivers/raw/ifpga/rte_pmd_ifpga.c         | 163 ++++++
 drivers/raw/ifpga/rte_pmd_ifpga.h         | 132 +++++
 drivers/raw/ifpga/version.map             |  11 +
 16 files changed, 1669 insertions(+), 2 deletions(-)
 create mode 100644 drivers/raw/ifpga/base/ifpga_fme_rsu.c
 create mode 100644 drivers/raw/ifpga/base/ifpga_sec_mgr.c
 create mode 100644 drivers/raw/ifpga/base/ifpga_sec_mgr.h
 create mode 100644 drivers/raw/ifpga/rte_pmd_ifpga.c
 create mode 100644 drivers/raw/ifpga/rte_pmd_ifpga.h

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 748514e243..8a48af1042 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -56,7 +56,8 @@ The public API headers are grouped by topics:
   [dpaa2_qdma]         (@ref rte_pmd_dpaa2_qdma.h),
   [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h),
   [dlb]                (@ref rte_pmd_dlb.h),
-  [dlb2]               (@ref rte_pmd_dlb2.h)
+  [dlb2]               (@ref rte_pmd_dlb2.h),
+  [ifpga]              (@ref rte_pmd_ifpga.h)
 
 - **memory**:
   [memseg]             (@ref rte_memory.h),
diff --git a/drivers/raw/ifpga/base/ifpga_api.c b/drivers/raw/ifpga/base/ifpga_api.c
index 1ff57fa188..1aedf150bc 100644
--- a/drivers/raw/ifpga/base/ifpga_api.c
+++ b/drivers/raw/ifpga/base/ifpga_api.c
@@ -5,6 +5,7 @@
 #include "ifpga_api.h"
 #include "ifpga_enumerate.h"
 #include "ifpga_feature_dev.h"
+#include "ifpga_sec_mgr.h"
 
 #include "opae_hw_api.h"
 
@@ -228,11 +229,36 @@ static int ifpga_mgr_get_board_info(struct opae_manager *mgr,
 	return 0;
 }
 
+static int ifpga_mgr_update_flash(struct opae_manager *mgr, const char *image,
+	u64 *status)
+{
+	struct ifpga_fme_hw *fme = mgr->data;
+
+	return fpga_update_flash(fme, image, status);
+}
+
+static int ifpga_mgr_stop_flash_update(struct opae_manager *mgr, int force)
+{
+	struct ifpga_fme_hw *fme = mgr->data;
+
+	return fpga_stop_flash_update(fme, force);
+}
+
+static int ifpga_mgr_reload(struct opae_manager *mgr, int type, int page)
+{
+	struct ifpga_fme_hw *fme = mgr->data;
+
+	return fpga_reload(fme, type, page);
+}
+
 struct opae_manager_ops ifpga_mgr_ops = {
 	.flash = ifpga_mgr_flash,
 	.get_eth_group_region_info = ifpga_mgr_get_eth_group_region_info,
 	.get_sensor_value = ifpga_mgr_get_sensor_value,
 	.get_board_info = ifpga_mgr_get_board_info,
+	.update_flash = ifpga_mgr_update_flash,
+	.stop_flash_update = ifpga_mgr_stop_flash_update,
+	.reload = ifpga_mgr_reload,
 };
 
 static int ifpga_mgr_read_mac_rom(struct opae_manager *mgr, int offset,
diff --git a/drivers/raw/ifpga/base/ifpga_fme.c b/drivers/raw/ifpga/base/ifpga_fme.c
index f29ff3159b..34fd9a818e 100644
--- a/drivers/raw/ifpga/base/ifpga_fme.c
+++ b/drivers/raw/ifpga/base/ifpga_fme.c
@@ -7,6 +7,7 @@
 #include "opae_intel_max10.h"
 #include "opae_i2c.h"
 #include "opae_at24_eeprom.h"
+#include "ifpga_sec_mgr.h"
 
 #define PWR_THRESHOLD_MAX       0x7F
 
@@ -1152,6 +1153,12 @@ static int fme_nios_spi_init(struct ifpga_feature *feature)
 	if (spi_self_checking(max10))
 		goto spi_fail;
 
+	ret = init_sec_mgr(fme);
+	if (ret) {
+		dev_err(fme, "security manager init fail\n");
+		goto spi_fail;
+	}
+
 	return ret;
 
 spi_fail:
@@ -1165,6 +1172,7 @@ static void fme_nios_spi_uinit(struct ifpga_feature *feature)
 {
 	struct ifpga_fme_hw *fme = (struct ifpga_fme_hw *)feature->parent;
 
+	release_sec_mgr(fme);
 	if (fme->max10_dev)
 		intel_max10_device_remove(fme->max10_dev);
 }
diff --git a/drivers/raw/ifpga/base/ifpga_fme_rsu.c b/drivers/raw/ifpga/base/ifpga_fme_rsu.c
new file mode 100644
index 0000000000..28198abd78
--- /dev/null
+++ b/drivers/raw/ifpga/base/ifpga_fme_rsu.c
@@ -0,0 +1,428 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <fcntl.h>
+#include <signal.h>
+#include <unistd.h>
+#include "ifpga_sec_mgr.h"
+
+static struct ifpga_sec_mgr *sec_mgr;
+
+static void set_rsu_control(struct ifpga_sec_mgr *smgr, uint32_t ctrl)
+{
+	if (smgr && smgr->rsu_control)
+		*smgr->rsu_control = ctrl;
+}
+
+static uint32_t get_rsu_control(struct ifpga_sec_mgr *smgr)
+{
+	if (smgr && smgr->rsu_control)
+		return *smgr->rsu_control;
+	return 0;
+}
+
+static void set_rsu_status(struct ifpga_sec_mgr *smgr, uint32_t status,
+	uint32_t progress)
+{
+	if (smgr && smgr->rsu_status)
+		*smgr->rsu_status = IFPGA_RSU_STATUS(status, progress);
+}
+
+static void get_rsu_status(struct ifpga_sec_mgr *smgr, uint32_t *status,
+	uint32_t *progress)
+{
+	if (smgr && smgr->rsu_status) {
+		if (status)
+			*status = IFPGA_RSU_GET_STAT(*smgr->rsu_status);
+		if (progress)
+			*progress = IFPGA_RSU_GET_PROG(*smgr->rsu_status);
+	}
+}
+
+static void sig_handler(int sig, siginfo_t *info, void *data)
+{
+	(void)(info);
+	(void)(data);
+
+	switch (sig) {
+	case SIGINT:
+		if (sec_mgr) {
+			dev_info(sec_mgr, "Interrupt secure flash update"
+				" by keyboard\n");
+			set_rsu_control(sec_mgr, IFPGA_RSU_ABORT);
+		}
+		break;
+	default:
+		break;
+	}
+}
+
+static void log_time(time_t t, const char *msg)
+{
+	uint32_t h = 0;
+	uint32_t m = 0;
+	uint32_t s = 0;
+
+	if (t < 60) {
+		s = (uint32_t)t;
+	} else if (t < 3600) {
+		s = (uint32_t)(t % 60);
+		m = (uint32_t)(t / 60);
+	} else {
+		s = (uint32_t)(t % 60);
+		m = (uint32_t)((t % 3600) / 60);
+		h = (uint32_t)(t / 3600);
+	}
+	printf("%s - %02u:%02u:%02u\n", msg, h, m, s);
+}
+
+static int start_flash_update(struct ifpga_sec_mgr *smgr)
+{
+	if (!smgr)
+		return -ENODEV;
+
+	if (!smgr->ops || !smgr->ops->prepare)
+		return -EINVAL;
+
+	return smgr->ops->prepare(smgr);
+}
+
+static int write_flash_image(struct ifpga_sec_mgr *smgr, const char *image,
+	uint32_t offset)
+{
+	void *buf = NULL;
+	int retry = 0;
+	uint32_t length = 0;
+	uint32_t to_transfer = 0;
+	uint32_t one_percent = 0;
+	uint32_t prog = 0;
+	uint32_t old_prog = -1;
+	ssize_t read_size = 0;
+	int fd = -1;
+	int ret = 0;
+
+	if (!smgr)
+		return -ENODEV;
+
+	if (!smgr->ops || !smgr->ops->write_blk)
+		return -EINVAL;
+
+	fd = open(image, O_RDONLY);
+	if (fd < 0) {
+		dev_err(smgr,
+			"Failed to open \'%s\' for RD [e:%s]\n",
+			image, strerror(errno));
+		return -EIO;
+	}
+
+	buf = malloc(IFPGA_RSU_DATA_BLK_SIZE);
+	if (!buf) {
+		dev_err(smgr, "Failed to allocate memory for flash update\n");
+		close(fd);
+		return -ENOMEM;
+	}
+
+	length = smgr->rsu_length;
+	one_percent = length / 100;
+	do {
+		to_transfer = (length > IFPGA_RSU_DATA_BLK_SIZE) ?
+			IFPGA_RSU_DATA_BLK_SIZE : length;
+		lseek(fd, offset, SEEK_SET);
+		read_size = read(fd, buf, to_transfer);
+		if (read_size < 0) {
+			dev_err(smgr, "Failed to read from \'%s\' [e:%s]\n",
+				image, strerror(errno));
+			ret = -EIO;
+			goto end;
+		}
+		if ((uint32_t)read_size != to_transfer) {
+			dev_err(smgr,
+				"Read length %zd is not expected [e:%u]\n",
+				read_size, to_transfer);
+			ret = -EIO;
+			goto end;
+		}
+
+		retry = 0;
+		do {
+			if (get_rsu_control(smgr) == IFPGA_RSU_ABORT) {
+				ret = -EAGAIN;
+				goto end;
+			}
+			ret = smgr->ops->write_blk(smgr, buf, offset,
+				to_transfer);
+			if (ret == 0)
+				break;
+			sleep(1);
+		} while (++retry <= IFPGA_RSU_WRITE_RETRY);
+		if (retry > IFPGA_RSU_WRITE_RETRY) {
+			dev_err(smgr, "Failed to write to staging area 0x%x\n",
+				offset);
+			ret = -EAGAIN;
+			goto end;
+		}
+
+		length -= to_transfer;
+		offset += to_transfer;
+		prog = offset / one_percent;
+		if (prog != old_prog) {
+			printf("\r%d%%", prog);
+			fflush(stdout);
+			set_rsu_status(smgr, IFPGA_RSU_READY, prog);
+			old_prog = prog;
+		}
+	} while (length > 0);
+	set_rsu_status(smgr, IFPGA_RSU_READY, 100);
+	printf("\n");
+
+end:
+	free(buf);
+	close(fd);
+	return ret;
+}
+
+static int apply_flash_update(struct ifpga_sec_mgr *smgr)
+{
+	uint32_t one_percent = 0;
+	uint32_t one_percent_time = 0;
+	uint32_t prog = 0;
+	uint32_t old_prog = -1;
+	uint32_t copy_time = 0;
+	int ret = 0;
+
+	if (!smgr)
+		return -ENODEV;
+
+	if (!smgr->ops || !smgr->ops->write_done || !smgr->ops->check_complete)
+		return -EINVAL;
+
+	if (smgr->ops->write_done(smgr) < 0) {
+		dev_err(smgr, "Failed to apply flash update\n");
+		return -EAGAIN;
+	}
+
+	one_percent = (smgr->rsu_length + 99) / 100;
+	if (smgr->copy_speed == 0)   /* avoid zero divide fault */
+		smgr->copy_speed = 1;
+	one_percent_time = (one_percent + smgr->copy_speed - 1) /
+		smgr->copy_speed;
+	if (one_percent_time == 0)   /* avoid zero divide fault */
+		one_percent_time = 1;
+
+	do {
+		ret = smgr->ops->check_complete(smgr);
+		if (ret != -EAGAIN)
+			break;
+		sleep(1);
+		copy_time += 1;
+		prog = copy_time / one_percent_time;
+		if (prog >= 100)
+			prog = 99;
+		if (prog != old_prog) {
+			printf("\r%d%%", prog);
+			fflush(stdout);
+			set_rsu_status(smgr, IFPGA_RSU_COPYING, prog);
+			old_prog = prog;
+		}
+	} while (true);
+
+	if (ret < 0) {
+		printf("\n");
+		dev_err(smgr, "Failed to complete secure flash update\n");
+	} else {
+		printf("\r100%%\n");
+		set_rsu_status(smgr, IFPGA_RSU_COPYING, 100);
+	}
+
+	return ret;
+}
+
+static int secure_update_cancel(struct ifpga_sec_mgr *smgr)
+{
+	if (!smgr)
+		return -ENODEV;
+
+	if (!smgr->ops || !smgr->ops->cancel)
+		return -EINVAL;
+
+	return smgr->ops->cancel(smgr);
+}
+
+static int secure_update_status(struct ifpga_sec_mgr *smgr, uint64_t *status)
+{
+	if (!smgr)
+		return -ENODEV;
+
+	if (!smgr->ops || !smgr->ops->get_hw_errinfo)
+		return -EINVAL;
+
+	if (status)
+		*status = smgr->ops->get_hw_errinfo(smgr);
+
+	return 0;
+}
+
+int fpga_update_flash(struct ifpga_fme_hw *fme, const char *image,
+	uint64_t *status)
+{
+	struct ifpga_hw *hw = NULL;
+	struct ifpga_sec_mgr *smgr = NULL;
+	uint32_t rsu_stat = 0;
+	int fd = -1;
+	struct sigaction old_sigint_action;
+	struct sigaction sa;
+	time_t start;
+	int ret = 0;
+
+	if (!fme || !image || !status) {
+		dev_err(fme, "Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	hw = (struct ifpga_hw *)fme->parent;
+	if (!hw) {
+		dev_err(fme, "Parent of FME not found\n");
+		return -ENODEV;
+	}
+
+	smgr = (struct ifpga_sec_mgr *)fme->sec_mgr;
+	if (!smgr || !smgr->max10_dev) {
+		dev_err(smgr, "Security manager not initialized\n");
+		return -ENODEV;
+	}
+
+	opae_adapter_lock(hw->adapter, -1);
+	get_rsu_status(smgr, &rsu_stat, NULL);
+	if (rsu_stat != IFPGA_RSU_IDLE) {
+		opae_adapter_unlock(hw->adapter);
+		if (rsu_stat == IFPGA_RSU_REBOOT)
+			dev_info(smgr, "Reboot is in progress\n");
+		else
+			dev_info(smgr, "Update is in progress\n");
+		return -EAGAIN;
+	}
+	set_rsu_control(smgr, 0);
+	set_rsu_status(smgr, IFPGA_RSU_PREPARE, 0);
+	opae_adapter_unlock(hw->adapter);
+
+	fd = open(image, O_RDONLY);
+	if (fd < 0) {
+		dev_err(smgr,
+			"Failed to open \'%s\' for RD [e:%s]\n",
+			image, strerror(errno));
+		return -EIO;
+	}
+	smgr->rsu_length = lseek(fd, 0, SEEK_END);
+	close(fd);
+
+	if (smgr->max10_dev->staging_area_size < smgr->rsu_length) {
+		dev_err(dev, "Size of staging area is small than image length "
+			"[%u<%u]\n", smgr->max10_dev->staging_area_size,
+			smgr->rsu_length);
+		return -EINVAL;
+	}
+
+	printf("Updating from file \'%s\' with size %u\n",
+		image, smgr->rsu_length);
+
+	sec_mgr = smgr;
+	memset(&sa, 0, sizeof(struct sigaction));
+	sa.sa_flags = SA_SIGINFO | SA_RESETHAND;
+	sa.sa_sigaction = sig_handler;
+	ret = sigaction(SIGINT, &sa, &old_sigint_action);
+	if (ret < 0) {
+		dev_warn(dev, "Failed to register signal handler"
+			" [e:%d]\n", ret);
+		sec_mgr = NULL;
+	}
+
+	start = time(NULL);
+	log_time(time(NULL) - start, "Starting secure flash update");
+	ret = start_flash_update(smgr);
+	if (ret < 0)
+		goto end;
+
+	set_rsu_status(smgr, IFPGA_RSU_READY, 0);
+	log_time(time(NULL) - start, "Writing to staging area");
+	ret = write_flash_image(smgr, image, 0);
+	if (ret < 0)
+		goto end;
+
+	set_rsu_status(smgr, IFPGA_RSU_COPYING, 0);
+	log_time(time(NULL) - start, "Applying secure flash update");
+	ret = apply_flash_update(smgr);
+
+end:
+	if (sec_mgr) {
+		sec_mgr = NULL;
+		if (sigaction(SIGINT, &old_sigint_action, NULL) < 0)
+			dev_err(smgr, "Failed to unregister signal handler\n");
+	}
+
+	secure_update_status(smgr, status);
+	if (ret < 0) {
+		log_time(time(NULL) - start, "Secure flash update ERROR");
+		if (ret == -EAGAIN)
+			secure_update_cancel(smgr);
+	} else {
+		log_time(time(NULL) - start, "Secure flash update OK");
+	}
+	set_rsu_status(smgr, IFPGA_RSU_IDLE, 0);
+
+	return ret;
+}
+
+int fpga_stop_flash_update(struct ifpga_fme_hw *fme, int force)
+{
+	struct ifpga_sec_mgr *smgr = NULL;
+	uint32_t status = 0;
+	int retry = IFPGA_RSU_CANCEL_RETRY;
+	int ret = 0;
+
+	if (!fme) {
+		dev_err(fme, "Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+	smgr = (struct ifpga_sec_mgr *)fme->sec_mgr;
+
+	get_rsu_status(smgr, &status, NULL);
+	if (status != IFPGA_RSU_IDLE) {
+		dev_info(smgr, "Cancel secure flash update\n");
+		set_rsu_control(smgr, IFPGA_RSU_ABORT);
+	}
+
+	if (force) {
+		sleep(2);
+		do {
+			get_rsu_status(smgr, &status, NULL);
+			if (status == IFPGA_RSU_IDLE)
+				break;
+			if (secure_update_cancel(smgr) == 0)
+				set_rsu_status(smgr, IFPGA_RSU_IDLE, 0);
+			sleep(1);
+		} while (--retry > 0);
+		if (retry <= 0) {
+			dev_err(smgr, "Failed to stop flash update\n");
+			ret = -EAGAIN;
+		}
+	}
+
+	return ret;
+}
+
+int fpga_reload(struct ifpga_fme_hw *fme, int type, int page)
+{
+	struct ifpga_sec_mgr *smgr = NULL;
+
+	if (!fme) {
+		dev_err(fme, "Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+	smgr = (struct ifpga_sec_mgr *)fme->sec_mgr;
+
+	if (!smgr || !smgr->ops || !smgr->ops->reload)
+		return -EINVAL;
+
+	return smgr->ops->reload(smgr, type, page);
+}
diff --git a/drivers/raw/ifpga/base/ifpga_hw.h b/drivers/raw/ifpga/base/ifpga_hw.h
index 7c3307fe77..ed5edc6016 100644
--- a/drivers/raw/ifpga/base/ifpga_hw.h
+++ b/drivers/raw/ifpga/base/ifpga_hw.h
@@ -91,6 +91,7 @@ struct ifpga_fme_hw {
 	struct opae_board_info board_info;
 	int nums_eth_dev;
 	unsigned int nums_acc_region;
+	void *sec_mgr;
 };
 
 enum ifpga_port_state {
diff --git a/drivers/raw/ifpga/base/ifpga_sec_mgr.c b/drivers/raw/ifpga/base/ifpga_sec_mgr.c
new file mode 100644
index 0000000000..4cf1db3049
--- /dev/null
+++ b/drivers/raw/ifpga/base/ifpga_sec_mgr.c
@@ -0,0 +1,639 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <fcntl.h>
+#include <signal.h>
+#include <unistd.h>
+#include "ifpga_sec_mgr.h"
+
+
+static const char * const rsu_prog[] = {"IDLE", "PREPARING", "SLEEPING",
+	"READY", "AUTHENTICATING", "COPYING", "CANCELLATION", "PROGRAMMING_KEY",
+	"DONE", "PKVL_DONE"};
+static const char * const rsu_statl[] = {"NORMAL", "TIMEOUT", "AUTH_FAIL",
+	"COPY_FAIL", "FATAL", "PKVL_REJECT", "NON_INCR", "ERASE_FAIL",
+	"WEAROUT"};
+static const char * const rsu_stath[] = {"NIOS_OK", "USER_OK", "FACTORY_OK",
+	"USER_FAIL", "FACTORY_FAIL", "NIOS_FLASH_ERR", "FPGA_FLASH_ERR"};
+
+static const char *rsu_progress_name(uint32_t prog)
+{
+	if (prog > SEC_PROGRESS_PKVL_PROM_DONE)
+		return "UNKNOWN";
+	else
+		return rsu_prog[prog];
+}
+
+static const char *rsu_status_name(uint32_t stat)
+{
+	if (stat >= SEC_STATUS_NIOS_OK) {
+		if (stat > SEC_STATUS_FPGA_FLASH_ERR)
+			return "UNKNOWN";
+		else
+			return rsu_stath[stat-SEC_STATUS_NIOS_OK];
+	} else {
+		if (stat > SEC_STATUS_WEAROUT)
+			return "UNKNOWN";
+		else
+			return rsu_statl[stat];
+	}
+}
+
+static bool secure_start_done(uint32_t doorbell)
+{
+	return (SEC_STATUS_G(doorbell) == SEC_STATUS_ERASE_FAIL ||
+		SEC_STATUS_G(doorbell) == SEC_STATUS_WEAROUT ||
+		(SEC_PROGRESS_G(doorbell) != SEC_PROGRESS_IDLE &&
+		SEC_PROGRESS_G(doorbell) != SEC_PROGRESS_RSU_DONE));
+}
+
+static bool secure_prog_ready(uint32_t doorbell)
+{
+	return (SEC_PROGRESS_G(doorbell) != SEC_PROGRESS_READY);
+}
+
+static int poll_timeout(struct intel_max10_device *dev, uint32_t offset,
+	bool (*cond)(uint32_t), uint32_t interval_ms, uint32_t timeout_ms)
+{
+	uint32_t val = 0;
+	int ret = 0;
+
+	for (;;) {
+		ret = max10_sys_read(dev, offset, &val);
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to read max10 register 0x%x [e:%d]\n",
+				offset, ret);
+			break;
+		}
+
+		if (cond(val)) {
+			dev_debug(dev,
+				"Read 0x%08x from max10 register 0x%x "
+				"[poll success]\n", val, offset);
+			ret = 0;
+			break;
+		}
+		if (timeout_ms > interval_ms)
+			timeout_ms -= interval_ms;
+		else
+			timeout_ms = 0;
+		if (timeout_ms == 0) {
+			dev_debug(dev,
+				"Read 0x%08x from max10 register 0x%x "
+				"[poll timeout]\n", val, offset);
+			ret = -ETIMEDOUT;
+			break;
+		}
+		msleep(interval_ms);
+	}
+
+	return ret;
+}
+
+static int n3000_secure_update_start(struct intel_max10_device *dev)
+{
+	uint32_t doorbell = 0;
+	uint32_t prog = 0;
+	uint32_t status = 0;
+	int ret = 0;
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	prog = SEC_PROGRESS_G(doorbell);
+	if ((prog != SEC_PROGRESS_IDLE) && (prog != SEC_PROGRESS_RSU_DONE)) {
+		dev_debug(dev, "Current RSU progress is %s\n",
+			rsu_progress_name(prog));
+		return -EBUSY;
+	}
+
+	ret = max10_sys_update_bits(dev, MAX10_DOORBELL,
+		RSU_REQUEST | HOST_STATUS, RSU_REQUEST);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to updt max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	ret = poll_timeout(dev, MAX10_DOORBELL, secure_start_done,
+		IFPGA_SEC_START_INTERVAL_MS, IFPGA_SEC_START_TIMEOUT_MS);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to poll max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	status = SEC_STATUS_G(doorbell);
+	if (status == SEC_STATUS_WEAROUT)
+		return -EAGAIN;
+
+	if (status == SEC_STATUS_ERASE_FAIL)
+		return -EIO;
+
+	return 0;
+}
+
+static int n3000_cancel(struct ifpga_sec_mgr *smgr)
+{
+	struct intel_max10_device *dev = NULL;
+	uint32_t doorbell = 0;
+	uint32_t prog = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+	dev = (struct intel_max10_device *)smgr->max10_dev;
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	prog = SEC_PROGRESS_G(doorbell);
+	if (prog == SEC_PROGRESS_IDLE)
+		return 0;
+	if (prog != SEC_PROGRESS_READY)
+		return -EBUSY;
+
+	return max10_sys_update_bits(dev, MAX10_DOORBELL, HOST_STATUS,
+		HOST_STATUS_S(HOST_STATUS_ABORT_RSU));
+}
+
+static int n3000_prepare(struct ifpga_sec_mgr *smgr)
+{
+	struct intel_max10_device *dev = NULL;
+	int retry = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+	dev = (struct intel_max10_device *)smgr->max10_dev;
+
+	ret = n3000_secure_update_start(dev);
+	if (ret == -EBUSY)
+		n3000_cancel(smgr);
+
+	while (ret) {
+		if (++retry > IFPGA_RSU_START_RETRY)
+			break;
+		msleep(1000);
+		ret = n3000_secure_update_start(dev);
+	}
+	if (retry > IFPGA_RSU_START_RETRY) {
+		dev_err(dev, "Failed to start secure flash update\n");
+		ret = -EAGAIN;
+	}
+
+	return ret;
+}
+
+static int n3000_bulk_write(struct intel_max10_device *dev, uint32_t addr,
+	char *buf, uint32_t len)
+{
+	uint32_t i = 0;
+	uint32_t n = 0;
+	uint32_t v = 0;
+	uint32_t p = 0;
+	int ret = 0;
+
+	if (len & 0x3) {
+		dev_err(dev,
+			"Length of data block is not 4 bytes aligned [e:%u]\n",
+			len);
+		return -EINVAL;
+	}
+
+	n = len >> 2;
+	for (i = 0; i < n; i++) {
+		p = i << 2;
+		v = *(uint32_t *)(buf + p);
+		ret = max10_reg_write(dev, addr + p, v);
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to write to staging area 0x%08x [e:%d]\n",
+				addr + p, ret);
+			return ret;
+		}
+		usleep(1);
+	}
+
+	return 0;
+}
+
+static int n3000_write_blk(struct ifpga_sec_mgr *smgr, char *buf,
+	uint32_t offset, uint32_t len)
+{
+	struct intel_max10_device *dev = NULL;
+	uint32_t doorbell = 0;
+	uint32_t prog = 0;
+	uint32_t m = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+	dev = (struct intel_max10_device *)smgr->max10_dev;
+
+	if (offset + len > dev->staging_area_size) {
+		dev_err(dev,
+			"Write position would be out of staging area [e:%u]\n",
+			dev->staging_area_size);
+		return -ENOMEM;
+	}
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	prog = SEC_PROGRESS_G(doorbell);
+	if (prog == SEC_PROGRESS_PREPARE)
+		return -EAGAIN;
+	else if (prog != SEC_PROGRESS_READY)
+		return -EBUSY;
+
+	m = len & 0x3;
+	if (m != 0)
+		len += 4 - m;   /* make length to 4 bytes align */
+
+	return n3000_bulk_write(dev, dev->staging_area_base + offset, buf, len);
+}
+
+static int n3000_write_done(struct ifpga_sec_mgr *smgr)
+{
+	struct intel_max10_device *dev = NULL;
+	uint32_t doorbell = 0;
+	uint32_t prog = 0;
+	uint32_t status = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+	dev = (struct intel_max10_device *)smgr->max10_dev;
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	prog = SEC_PROGRESS_G(doorbell);
+	if (prog != SEC_PROGRESS_READY)
+		return -EBUSY;
+
+	ret = max10_sys_update_bits(dev, MAX10_DOORBELL, HOST_STATUS,
+		HOST_STATUS_S(HOST_STATUS_WRITE_DONE));
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to update max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	ret = poll_timeout(dev, MAX10_DOORBELL, secure_prog_ready,
+		IFPGA_NIOS_HANDSHAKE_INTERVAL_MS,
+		IFPGA_NIOS_HANDSHAKE_TIMEOUT_MS);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to poll max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	status = SEC_STATUS_G(doorbell);
+	switch (status) {
+	case SEC_STATUS_NORMAL:
+	case SEC_STATUS_NIOS_OK:
+	case SEC_STATUS_USER_OK:
+	case SEC_STATUS_FACTORY_OK:
+		ret = 0;
+		break;
+	default:
+		ret = -EIO;
+		break;
+	}
+
+	return ret;
+}
+
+static int n3000_check_complete(struct ifpga_sec_mgr *smgr)
+{
+	struct intel_max10_device *dev = NULL;
+	uint32_t doorbell = 0;
+	uint32_t status = 0;
+	uint32_t prog = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+	dev = (struct intel_max10_device *)smgr->max10_dev;
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	status = SEC_STATUS_G(doorbell);
+	switch (status) {
+	case SEC_STATUS_NORMAL:
+	case SEC_STATUS_NIOS_OK:
+	case SEC_STATUS_USER_OK:
+	case SEC_STATUS_FACTORY_OK:
+	case SEC_STATUS_WEAROUT:
+		break;
+	default:
+		return -EIO;
+	}
+
+	prog = SEC_PROGRESS_G(doorbell);
+	switch (prog) {
+	case SEC_PROGRESS_IDLE:
+	case SEC_PROGRESS_RSU_DONE:
+		return 0;
+	case SEC_PROGRESS_AUTHENTICATING:
+	case SEC_PROGRESS_COPYING:
+	case SEC_PROGRESS_UPDATE_CANCEL:
+	case SEC_PROGRESS_PROGRAM_KEY_HASH:
+		return -EAGAIN;
+	case SEC_PROGRESS_PREPARE:
+	case SEC_PROGRESS_READY:
+		return -EBUSY;
+	default:
+		return -EIO;
+	}
+
+	return 0;
+}
+
+static int n3000_reload_fpga(struct intel_max10_device *dev, int page)
+{
+	int ret = 0;
+
+	dev_info(dev, "Reload FPGA\n");
+
+	if (!dev || ((page != 0) && (page != 1))) {
+		dev_err(dev, "Input parameter of %s is invalid\n", __func__);
+		ret = -EINVAL;
+		goto end;
+	}
+
+	if (dev->flags & MAX10_FLAGS_SECURE) {
+		ret = max10_sys_update_bits(dev, FPGA_RECONF_REG,
+			SFPGA_RP_LOAD, 0);
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to update max10 reconfig register [e:%d]\n",
+				ret);
+			goto end;
+		}
+		ret = max10_sys_update_bits(dev, FPGA_RECONF_REG,
+			SFPGA_RP_LOAD | SFPGA_RECONF_PAGE,
+			SFPGA_RP_LOAD | SFPGA_PAGE(page));
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to update max10 reconfig register [e:%d]\n",
+				ret);
+			goto end;
+		}
+	} else {
+		ret = max10_sys_update_bits(dev, RSU_REG, FPGA_RP_LOAD, 0);
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to update max10 rsu register [e:%d]\n",
+				ret);
+			goto end;
+		}
+		ret = max10_sys_update_bits(dev, RSU_REG,
+			FPGA_RP_LOAD | FPGA_RECONF_PAGE,
+			FPGA_RP_LOAD | FPGA_PAGE(page));
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to update max10 rsu register [e:%d]\n",
+				ret);
+			goto end;
+		}
+	}
+
+	ret = max10_sys_update_bits(dev, FPGA_RECONF_REG, COUNTDOWN_START, 0);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to update max10 reconfig register [e:%d]\n",
+			ret);
+		goto end;
+	}
+
+	ret = max10_sys_update_bits(dev, FPGA_RECONF_REG, COUNTDOWN_START,
+		COUNTDOWN_START);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to update max10 reconfig register [e:%d]\n",
+			ret);
+	}
+end:
+	if (ret < 0)
+		dev_err(dev, "Failed to reload FPGA\n");
+
+	return ret;
+}
+
+static int n3000_reload_bmc(struct intel_max10_device *dev, int page)
+{
+	uint32_t val = 0;
+	int ret = 0;
+
+	dev_info(dev, "Reload BMC\n");
+
+	if (!dev || ((page != 0) && (page != 1))) {
+		dev_err(dev, "Input parameter of %s is invalid\n", __func__);
+		ret = -EINVAL;
+		goto end;
+	}
+
+	if (dev->flags & MAX10_FLAGS_SECURE) {
+		ret = max10_sys_update_bits(dev, MAX10_DOORBELL,
+			CONFIG_SEL | REBOOT_REQ,
+			CONFIG_SEL_S(page) | REBOOT_REQ);
+	} else {
+		val = (page == 0) ? 0x1 : 0x3;
+		ret = max10_reg_write(dev, IFPGA_DUAL_CFG_CTRL1, val);
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to write to dual config1 register [e:%d]\n",
+				ret);
+			goto end;
+		}
+
+		ret = max10_reg_write(dev, IFPGA_DUAL_CFG_CTRL0, 0x1);
+		if (ret < 0) {
+			if (ret == -EIO) {
+				ret = 0;
+				goto end;
+			}
+			dev_err(dev,
+				"Failed to write to dual config0 register [e:%d]\n",
+				ret);
+		}
+	}
+
+end:
+	if (ret < 0)
+		dev_err(dev, "Failed to reload BMC\n");
+
+	return ret;
+}
+
+static int n3000_reload(struct ifpga_sec_mgr *smgr, int type, int page)
+{
+	int psel = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+
+	if (type == IFPGA_BOOT_TYPE_FPGA) {
+		psel = (page == IFPGA_BOOT_PAGE_FACTORY ? 0 : 1);
+		ret = n3000_reload_fpga(smgr->max10_dev, psel);
+	} else if (type == IFPGA_BOOT_TYPE_BMC) {
+		psel = (page == IFPGA_BOOT_PAGE_FACTORY ? 1 : 0);
+		ret = n3000_reload_bmc(smgr->max10_dev, psel);
+	} else {
+		ret = -EINVAL;
+	}
+
+	return ret;
+}
+
+static uint64_t n3000_get_hw_errinfo(struct ifpga_sec_mgr *smgr)
+{
+	struct intel_max10_device *dev = NULL;
+	uint32_t doorbell = 0;
+	uint32_t stat = 0;
+	uint32_t prog = 0;
+	uint32_t auth_result = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+	dev = (struct intel_max10_device *)smgr->max10_dev;
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev, "Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return -1;
+	}
+	stat = SEC_STATUS_G(doorbell);
+	prog = SEC_PROGRESS_G(doorbell);
+	dev_debug(dev, "Current RSU status is %s, progress is %s\n",
+		rsu_status_name(stat), rsu_progress_name(prog));
+
+	ret = max10_sys_read(dev, MAX10_AUTH_RESULT, &auth_result);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read authenticate result register [e:%d]\n",
+			ret);
+		return -1;
+	}
+
+	return (uint64_t)doorbell << 32 | (uint64_t)auth_result;
+}
+
+static const struct ifpga_sec_ops n3000_sec_ops = {
+	.prepare = n3000_prepare,
+	.write_blk = n3000_write_blk,
+	.write_done = n3000_write_done,
+	.check_complete = n3000_check_complete,
+	.reload = n3000_reload,
+	.cancel = n3000_cancel,
+	.cleanup = NULL,
+	.get_hw_errinfo = n3000_get_hw_errinfo,
+};
+
+int init_sec_mgr(struct ifpga_fme_hw *fme)
+{
+	struct ifpga_hw *hw = NULL;
+	opae_share_data *sd = NULL;
+	struct ifpga_sec_mgr *smgr = NULL;
+
+	if (!fme || !fme->max10_dev)
+		return -ENODEV;
+
+	smgr = (struct ifpga_sec_mgr *)malloc(sizeof(*smgr));
+	if (!smgr) {
+		dev_err(NULL, "Failed to allocate memory for security manager\n");
+		return -ENOMEM;
+	}
+	fme->sec_mgr = smgr;
+
+	hw = (struct ifpga_hw *)fme->parent;
+	if (hw && hw->adapter && hw->adapter->shm.ptr) {
+		sd = (opae_share_data *)hw->adapter->shm.ptr;
+		smgr->rsu_control = &sd->rsu_ctrl;
+		smgr->rsu_status = &sd->rsu_stat;
+	} else {
+		smgr->rsu_control = NULL;
+		smgr->rsu_status = NULL;
+	}
+
+	if ((hw->pci_data->device_id == IFPGA_N3000_DID) &&
+		(hw->pci_data->vendor_id == IFPGA_N3000_VID)) {
+		smgr->ops = &n3000_sec_ops;
+		smgr->copy_speed = IFPGA_N3000_COPY_SPEED;
+	} else {
+		dev_err(NULL, "No operation for security manager\n");
+		smgr->ops = NULL;
+	}
+
+	smgr->fme = fme;
+	smgr->max10_dev = fme->max10_dev;
+
+	return 0;
+}
+
+void release_sec_mgr(struct ifpga_fme_hw *fme)
+{
+	struct ifpga_sec_mgr *smgr = NULL;
+
+	if (fme) {
+		smgr = (struct ifpga_sec_mgr *)fme->sec_mgr;
+		if (smgr) {
+			fme->sec_mgr = NULL;
+			free(smgr);
+		}
+	}
+}
diff --git a/drivers/raw/ifpga/base/ifpga_sec_mgr.h b/drivers/raw/ifpga/base/ifpga_sec_mgr.h
new file mode 100644
index 0000000000..fbeba561f4
--- /dev/null
+++ b/drivers/raw/ifpga/base/ifpga_sec_mgr.h
@@ -0,0 +1,93 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _IFPGA_FME_RSU_H_
+#define _IFPGA_FME_RSU_H_
+
+
+#include "ifpga_hw.h"
+
+#define IFPGA_N3000_VID     0x8086
+#define IFPGA_N3000_DID     0x0b30
+
+#define IFPGA_BOOT_TYPE_FPGA     0
+#define IFPGA_BOOT_TYPE_BMC      1
+
+#define IFPGA_BOOT_PAGE_FACTORY  0
+#define IFPGA_BOOT_PAGE_USER     1
+
+#define IFPGA_RSU_DATA_BLK_SIZE  32768
+#define IFPGA_RSU_START_RETRY    120
+#define IFPGA_RSU_WRITE_RETRY    10
+#define IFPGA_RSU_CANCEL_RETRY   30
+
+#define IFPGA_N3000_COPY_SPEED   42700
+
+/* status */
+#define IFPGA_RSU_IDLE       0
+#define IFPGA_RSU_PREPARE    1
+#define IFPGA_RSU_READY      2
+#define IFPGA_RSU_COPYING    3
+#define IFPGA_RSU_REBOOT     4
+
+#define IFPGA_RSU_GET_STAT(v)  (((v) >> 16) & 0xffff)
+#define IFPGA_RSU_GET_PROG(v)  ((v) & 0xffff)
+#define IFPGA_RSU_STATUS(s, p) ((((s) << 16) & 0xffff0000) | ((p) & 0xffff))
+
+/* control */
+#define IFPGA_RSU_ABORT      1
+
+#define IFPGA_DUAL_CFG_CTRL0     0x200020
+#define IFPGA_DUAL_CFG_CTRL1     0x200024
+
+#define IFPGA_SEC_START_INTERVAL_MS       100
+#define IFPGA_SEC_START_TIMEOUT_MS        20000
+#define IFPGA_NIOS_HANDSHAKE_INTERVAL_MS  100
+#define IFPGA_NIOS_HANDSHAKE_TIMEOUT_MS   5000
+
+#define IFPGA_RSU_ERR_HW_ERROR		-1
+#define IFPGA_RSU_ERR_TIMEOUT		-2
+#define IFPGA_RSU_ERR_CANCELED		-3
+#define IFPGA_RSU_ERR_BUSY			-4
+#define IFPGA_RSU_ERR_INVALID_SIZE	-5
+#define IFPGA_RSU_ERR_RW_ERROR		-6
+#define IFPGA_RSU_ERR_WEAROUT		-7
+#define IFPGA_RSU_ERR_FILE_READ		-8
+
+struct ifpga_sec_mgr;
+
+struct ifpga_sec_ops {
+	int (*prepare)(struct ifpga_sec_mgr *smgr);
+	int (*write_blk)(struct ifpga_sec_mgr *smgr, char *buf, uint32_t offset,
+		uint32_t size);
+	int (*write_done)(struct ifpga_sec_mgr *smgr);
+	int (*check_complete)(struct ifpga_sec_mgr *smgr);
+	int (*reload)(struct ifpga_sec_mgr *smgr, int type, int page);
+	int (*cancel)(struct ifpga_sec_mgr *smgr);
+	void (*cleanup)(struct ifpga_sec_mgr *smgr);
+	u64 (*get_hw_errinfo)(struct ifpga_sec_mgr *smgr);
+};
+
+struct ifpga_sec_mgr {
+	struct ifpga_fme_hw *fme;
+	struct intel_max10_device *max10_dev;
+	unsigned int rsu_length;
+	/* number of bytes that copied from staging area to working area
+	 * in one second, which is calculated by experiment
+	 */
+	unsigned int copy_speed;
+	unsigned int *rsu_control;
+	unsigned int *rsu_status;
+	const struct ifpga_sec_ops *ops;
+};
+
+int init_sec_mgr(struct ifpga_fme_hw *fme);
+void release_sec_mgr(struct ifpga_fme_hw *fme);
+int fpga_update_flash(struct ifpga_fme_hw *fme, const char *image,
+	uint64_t *status);
+int fpga_stop_flash_update(struct ifpga_fme_hw *fme, int force);
+int fpga_reload(struct ifpga_fme_hw *fme, int type, int page);
+
+
+#endif /* _IFPGA_FME_RSU_H_ */
diff --git a/drivers/raw/ifpga/base/meson.build b/drivers/raw/ifpga/base/meson.build
index da2d6e33ca..3549afafa1 100644
--- a/drivers/raw/ifpga/base/meson.build
+++ b/drivers/raw/ifpga/base/meson.build
@@ -12,6 +12,8 @@ sources = [
 	'ifpga_port.c',
 	'ifpga_port_error.c',
 	'ifpga_fme_pr.c',
+	'ifpga_fme_rsu.c',
+	'ifpga_sec_mgr.c',
 	'opae_hw_api.c',
 	'opae_ifpga_hw_api.c',
 	'opae_debug.c',
diff --git a/drivers/raw/ifpga/base/opae_hw_api.c b/drivers/raw/ifpga/base/opae_hw_api.c
index d5cd5fe608..86ad88f720 100644
--- a/drivers/raw/ifpga/base/opae_hw_api.c
+++ b/drivers/raw/ifpga/base/opae_hw_api.c
@@ -470,6 +470,8 @@ static void opae_adapter_shm_init(struct opae_adapter *adapter)
 	opae_mutex_init(&sd->i2c_mutex);
 	sd->ref_cnt = 0;
 	sd->dtb_size = SHM_BLK_SIZE;
+	sd->rsu_ctrl = 0;
+	sd->rsu_stat = 0;
 }
 
 static void *opae_adapter_shm_alloc(struct opae_adapter *adapter)
@@ -964,3 +966,60 @@ opae_mgr_get_board_info(struct opae_manager *mgr,
 
 	return -ENOENT;
 }
+
+/**
+ * opae_mgr_update_flash -  update image in flash.
+ * @mgr: targeted manager
+ * @image: name of image file
+ * @status: status of update
+ *
+ * Return: 0 on success, otherwise error code.
+ */
+int opae_mgr_update_flash(struct opae_manager *mgr, const char *image,
+	uint64_t *status)
+{
+	if (!mgr)
+		return -EINVAL;
+
+	if (mgr->ops && mgr->ops->update_flash)
+		return mgr->ops->update_flash(mgr, image, status);
+
+	return -ENOENT;
+}
+
+/**
+ * opae_stop_flash_update -  stop flash update.
+ * @mgr: targeted manager
+ * @force: make sure the update process is stopped
+ *
+ * Return: 0 on success, otherwise error code.
+ */
+int opae_mgr_stop_flash_update(struct opae_manager *mgr, int force)
+{
+	if (!mgr)
+		return -EINVAL;
+
+	if (mgr->ops && mgr->ops->stop_flash_update)
+		return mgr->ops->stop_flash_update(mgr, force);
+
+	return -ENOENT;
+}
+
+/**
+ * opae_mgr_reload -  reload FPGA.
+ * @mgr: targeted manager
+ * @type: FPGA type
+ * @page: reload from which page
+ *
+ * Return: 0 on success, otherwise error code.
+ */
+int opae_mgr_reload(struct opae_manager *mgr, int type, int page)
+{
+	if (!mgr)
+		return -EINVAL;
+
+	if (mgr->ops && mgr->ops->reload)
+		return mgr->ops->reload(mgr, type, page);
+
+	return -ENOENT;
+}
diff --git a/drivers/raw/ifpga/base/opae_hw_api.h b/drivers/raw/ifpga/base/opae_hw_api.h
index e99ee4564c..91d26d9b5b 100644
--- a/drivers/raw/ifpga/base/opae_hw_api.h
+++ b/drivers/raw/ifpga/base/opae_hw_api.h
@@ -55,6 +55,10 @@ struct opae_manager_ops {
 			unsigned int *value);
 	int (*get_board_info)(struct opae_manager *mgr,
 			struct opae_board_info **info);
+	int (*update_flash)(struct opae_manager *mgr, const char *image,
+			u64 *status);
+	int (*stop_flash_update)(struct opae_manager *mgr, int force);
+	int (*reload)(struct opae_manager *mgr, int type, int page);
 };
 
 /* networking management ops in FME */
@@ -276,6 +280,8 @@ typedef struct {
 			pthread_mutex_t i2c_mutex;
 			u32 ref_cnt;    /* reference count of shared memory */
 			u32 dtb_size;   /* actual length of DTB data in byte */
+			u32 rsu_ctrl;   /* used to control rsu */
+			u32 rsu_stat;   /* used to report status for rsu */
 		};
 	};
 	u8 dtb[SHM_BLK_SIZE];   /* DTB data */
@@ -354,4 +360,8 @@ int opae_manager_eth_group_read_reg(struct opae_manager *mgr, u8 group_id,
 		u8 type, u8 index, u16 addr, u32 *data);
 int opae_mgr_get_board_info(struct opae_manager *mgr,
 		struct opae_board_info **info);
+int opae_mgr_update_flash(struct opae_manager *mgr, const char *image,
+		uint64_t *status);
+int opae_mgr_stop_flash_update(struct opae_manager *mgr, int force);
+int opae_mgr_reload(struct opae_manager *mgr, int type, int page);
 #endif /* _OPAE_HW_API_H_*/
diff --git a/drivers/raw/ifpga/base/opae_intel_max10.c b/drivers/raw/ifpga/base/opae_intel_max10.c
index 1a526ea549..443e248fb3 100644
--- a/drivers/raw/ifpga/base/opae_intel_max10.c
+++ b/drivers/raw/ifpga/base/opae_intel_max10.c
@@ -51,6 +51,22 @@ int max10_sys_write(struct intel_max10_device *dev,
 	return max10_reg_write(dev, dev->base + offset, val);
 }
 
+int max10_sys_update_bits(struct intel_max10_device *dev, unsigned int offset,
+					unsigned int msk, unsigned int val)
+{
+	int ret = 0;
+	unsigned int temp = 0;
+
+	ret = max10_sys_read(dev, offset, &temp);
+	if (ret < 0)
+		return ret;
+
+	temp &= ~msk;
+	temp |= val & msk;
+
+	return max10_sys_write(dev, offset, temp);
+}
+
 static struct max10_compatible_id max10_id_table[] = {
 	{.compatible = MAX10_PAC,},
 	{.compatible = MAX10_PAC_N3000,},
@@ -557,6 +573,36 @@ static int check_max10_version(struct intel_max10_device *dev)
 	return -ENODEV;
 }
 
+static int max10_staging_area_init(struct intel_max10_device *dev)
+{
+	char *fdt_root = dev->fdt_root;
+	int ret, offset = 0;
+	u64 start, size;
+
+	if (!fdt_root) {
+		dev_debug(dev,
+			"skip staging area init as not find Device Tree\n");
+		return -ENODEV;
+	}
+
+	dev->staging_area_size = 0;
+
+	fdt_for_each_subnode(offset, fdt_root, 0) {
+		if (fdt_node_check_compatible(fdt_root, offset,
+					      "ifpga-sec-mgr,staging-area"))
+			continue;
+
+		ret = fdt_get_reg(fdt_root, offset, 0, &start, &size);
+		if (!ret) {
+			dev->staging_area_base = start;
+			dev->staging_area_size = size;
+		}
+		return ret;
+	}
+
+	return -ENODEV;
+}
+
 static int
 max10_secure_hw_init(struct intel_max10_device *dev)
 {
@@ -581,6 +627,8 @@ max10_secure_hw_init(struct intel_max10_device *dev)
 
 	max10_sensor_init(dev, sysmgr_offset);
 
+	max10_staging_area_init(dev);
+
 	return 0;
 }
 
diff --git a/drivers/raw/ifpga/base/opae_intel_max10.h b/drivers/raw/ifpga/base/opae_intel_max10.h
index 123cdc48b9..670683f017 100644
--- a/drivers/raw/ifpga/base/opae_intel_max10.h
+++ b/drivers/raw/ifpga/base/opae_intel_max10.h
@@ -38,6 +38,8 @@ struct intel_max10_device {
 	unsigned int base; /* max10 base address */
 	u16 bus;
 	struct opae_sensor_list opae_sensor_list;
+	u32 staging_area_base;
+	u32 staging_area_size;
 };
 
 /* retimer speed */
@@ -98,6 +100,7 @@ struct opae_retimer_status {
 #define   MAX10_MAC_COUNT	GENMASK(23, 16)
 #define RSU_REG			0x2c
 #define   FPGA_RECONF_PAGE	GENMASK(2, 0)
+#define   FPGA_PAGE(p)		((p) & 0x1)
 #define   FPGA_RP_LOAD		BIT(3)
 #define   NIOS2_PRERESET	BIT(4)
 #define   NIOS2_HANG		BIT(5)
@@ -106,6 +109,9 @@ struct opae_retimer_status {
 #define   NIOS2_I2C2_POLL_STOP	BIT(13)
 #define   PKVL_EEPROM_LOAD	BIT(31)
 #define FPGA_RECONF_REG		0x30
+#define   SFPGA_RECONF_PAGE	GENMASK(22, 20)
+#define   SFPGA_PAGE(p)		(((p) & 0x1) << 20)
+#define   SFPGA_RP_LOAD		BIT(23)
 #define MAX10_TEST_REG		0x3c
 #define   COUNTDOWN_START	BIT(18)
 #define MAX10_BUILD_VER		0x68
@@ -118,8 +124,44 @@ struct opae_retimer_status {
 #define MAX10_DOORBELL		0x400
 #define   RSU_REQUEST		BIT(0)
 #define   SEC_PROGRESS		GENMASK(7, 4)
+#define   SEC_PROGRESS_G(v)	(((v) >> 4) & 0xf)
+#define   SEC_PROGRESS_IDLE				0x0
+#define   SEC_PROGRESS_PREPARE			0x1
+#define   SEC_PROGRESS_SLEEP			0x2
+#define   SEC_PROGRESS_READY			0x3
+#define   SEC_PROGRESS_AUTHENTICATING	0x4
+#define   SEC_PROGRESS_COPYING			0x5
+#define   SEC_PROGRESS_UPDATE_CANCEL	0x6
+#define   SEC_PROGRESS_PROGRAM_KEY_HASH	0x7
+#define   SEC_PROGRESS_RSU_DONE			0x8
+#define   SEC_PROGRESS_PKVL_PROM_DONE	0x9
 #define   HOST_STATUS		GENMASK(11, 8)
+#define   HOST_STATUS_S(v)	(((v) << 8) & 0xf00)
+#define   HOST_STATUS_IDLE			0x0
+#define   HOST_STATUS_WRITE_DONE	0x1
+#define   HOST_STATUS_ABORT_RSU		0x2
 #define   SEC_STATUS		GENMASK(23, 16)
+#define   SEC_STATUS_G(v)	(((v) >> 16) & 0xff)
+#define   SEC_STATUS_NORMAL			0x0
+#define   SEC_STATUS_TIMEOUT		0x1
+#define   SEC_STATUS_AUTH_FAIL		0x2
+#define   SEC_STATUS_COPY_FAIL		0x3
+#define   SEC_STATUS_FATAL			0x4
+#define   SEC_STATUS_PKVL_REJECT	0x5
+#define   SEC_STATUS_NON_INC		0x6
+#define   SEC_STATUS_ERASE_FAIL		0x7
+#define   SEC_STATUS_WEAROUT		0x8
+#define   SEC_STATUS_NIOS_OK		0x80
+#define   SEC_STATUS_USER_OK		0x81
+#define   SEC_STATUS_FACTORY_OK		0x82
+#define   SEC_STATUS_USER_FAIL		0x83
+#define   SEC_STATUS_FACTORY_FAIL	0x84
+#define   SEC_STATUS_NIOS_FLASH_ERR	0x85
+#define   SEC_STATUS_FPGA_FLASH_ERR	0x86
+#define   CONFIG_SEL		BIT(28)
+#define   CONFIG_SEL_S(v)	(((v) & 0x1) << 28)
+#define   REBOOT_REQ		BIT(29)
+#define MAX10_AUTH_RESULT	0x404
 
 /* PKVL related registers, in system register region */
 #define PKVL_POLLING_CTRL		0x80
@@ -149,6 +191,8 @@ int max10_sys_read(struct intel_max10_device *dev,
 	unsigned int offset, unsigned int *val);
 int max10_sys_write(struct intel_max10_device *dev,
 	unsigned int offset, unsigned int val);
+int max10_sys_update_bits(struct intel_max10_device *dev,
+	unsigned int offset, unsigned int msk, unsigned int val);
 struct intel_max10_device *
 intel_max10_device_probe(struct altera_spi_device *spi,
 		int chipselect);
diff --git a/drivers/raw/ifpga/meson.build b/drivers/raw/ifpga/meson.build
index 027ff80562..60ea59ae28 100644
--- a/drivers/raw/ifpga/meson.build
+++ b/drivers/raw/ifpga/meson.build
@@ -13,8 +13,10 @@ objs = [base_objs]
 deps += ['ethdev', 'rawdev', 'pci', 'bus_pci', 'kvargs',
 	'bus_vdev', 'bus_ifpga', 'net', 'net_i40e', 'net_ipn3ke']
 
-sources = files('ifpga_rawdev.c')
+sources = files('ifpga_rawdev.c', 'rte_pmd_ifpga.c')
 
 includes += include_directories('base')
 includes += include_directories('../../net/ipn3ke')
 includes += include_directories('../../net/i40e')
+
+headers = files('rte_pmd_ifpga.h')
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.c b/drivers/raw/ifpga/rte_pmd_ifpga.c
new file mode 100644
index 0000000000..af6f175e98
--- /dev/null
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.c
@@ -0,0 +1,163 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 Intel Corporation
+ */
+
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+#include <rte_rawdev.h>
+#include <rte_rawdev_pmd.h>
+#include "rte_pmd_ifpga.h"
+#include "ifpga_rawdev.h"
+#include "base/ifpga_sec_mgr.h"
+
+
+int
+rte_pmd_ifpga_get_dev_id(const char *pci_addr, uint16_t *dev_id)
+{
+	struct rte_pci_addr addr;
+	struct rte_rawdev *rdev = NULL;
+	char rdev_name[RTE_RAWDEV_NAME_MAX_LEN] = {0};
+
+	if (!pci_addr || !dev_id) {
+		IFPGA_RAWDEV_PMD_ERR("Input parameter is invalid.");
+		return -EINVAL;
+	}
+
+	if (strnlen(pci_addr, PCI_PRI_STR_SIZE) == PCI_PRI_STR_SIZE) {
+		IFPGA_RAWDEV_PMD_ERR("PCI address is too long.");
+		return -EINVAL;
+	}
+
+	if (rte_pci_addr_parse(pci_addr, &addr)) {
+		IFPGA_RAWDEV_PMD_ERR("PCI address %s is invalid.", pci_addr);
+		return -EINVAL;
+	}
+
+	snprintf(rdev_name, RTE_RAWDEV_NAME_MAX_LEN, "IFPGA:%02x:%02x.%x",
+		addr.bus, addr.devid, addr.function);
+	rdev = rte_rawdev_pmd_get_named_dev(rdev_name);
+	if (!rdev) {
+		IFPGA_RAWDEV_PMD_DEBUG("%s is not probed by ifpga driver.",
+			pci_addr);
+		return -ENODEV;
+	}
+	*dev_id = rdev->dev_id;
+
+	return 0;
+}
+
+static struct rte_rawdev *
+get_rte_rawdev(uint16_t dev_id)
+{
+	struct rte_rawdev *dev = NULL;
+
+	if (dev_id >= RTE_RAWDEV_MAX_DEVS)
+		return NULL;
+
+	dev = &rte_rawdevs[dev_id];
+	if (dev->attached == RTE_RAWDEV_ATTACHED)
+		return dev;
+
+	return NULL;
+}
+
+static struct opae_adapter *
+get_opae_adapter(uint16_t dev_id)
+{
+	struct rte_rawdev *dev = NULL;
+	struct opae_adapter *adapter = NULL;
+
+	dev = get_rte_rawdev(dev_id);
+	if (!dev) {
+		IFPGA_RAWDEV_PMD_ERR("Device ID %u is invalid.", dev_id);
+		return NULL;
+	}
+
+	adapter = ifpga_rawdev_get_priv(dev);
+	if (!adapter) {
+		IFPGA_RAWDEV_PMD_ERR("Adapter is not registered.");
+		return NULL;
+	}
+
+	return adapter;
+}
+
+static opae_share_data *
+get_share_data(struct opae_adapter *adapter)
+{
+	opae_share_data *sd = NULL;
+
+	if (!adapter)
+		return NULL;
+
+	sd = (opae_share_data *)adapter->shm.ptr;
+	if (!sd) {
+		IFPGA_RAWDEV_PMD_ERR("Share data is not initialized.");
+		return NULL;
+	}
+
+	return sd;
+}
+
+int
+rte_pmd_ifpga_update_flash(uint16_t dev_id, const char *image,
+	uint64_t *status)
+{
+	struct opae_adapter *adapter = NULL;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	return opae_mgr_update_flash(adapter->mgr, image, status);
+}
+
+int
+rte_pmd_ifpga_stop_update(uint16_t dev_id, int force)
+{
+	struct opae_adapter *adapter = NULL;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	return opae_mgr_stop_flash_update(adapter->mgr, force);
+}
+
+int
+rte_pmd_ifpga_reboot_try(uint16_t dev_id)
+{
+	struct opae_adapter *adapter = NULL;
+	opae_share_data *sd = NULL;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	sd = get_share_data(adapter);
+	if (!sd)
+		return -ENOMEM;
+
+	opae_adapter_lock(adapter, -1);
+	if (IFPGA_RSU_GET_STAT(sd->rsu_stat) != IFPGA_RSU_IDLE) {
+		opae_adapter_unlock(adapter);
+		IFPGA_RAWDEV_PMD_WARN("Update or reboot is in progress.");
+		return -EBUSY;
+	}
+	sd->rsu_stat = IFPGA_RSU_STATUS(IFPGA_RSU_REBOOT, 0);
+	opae_adapter_unlock(adapter);
+
+	return 0;
+}
+
+int
+rte_pmd_ifpga_reload(uint16_t dev_id, int type, int page)
+{
+	struct opae_adapter *adapter = NULL;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	return opae_mgr_reload(adapter->mgr, type, page);
+}
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.h b/drivers/raw/ifpga/rte_pmd_ifpga.h
new file mode 100644
index 0000000000..023a011116
--- /dev/null
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.h
@@ -0,0 +1,132 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 Intel Corporation
+ */
+
+#ifndef _RTE_PMD_IFPGA_H_
+#define _RTE_PMD_IFPGA_H_
+
+/**
+ * @file rte_pmd_ifpga.h
+ *
+ * ifpga PMD specific functions.
+ *
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Get raw device ID from PCI address string like 'Domain:Bus:Dev.Func'
+ *
+ * @param pci_addr
+ *    The PCI address of specified Intel FPGA device.
+ * @param dev_id
+ *    The buffer to output device ID.
+ * @return
+ *   - (0) if successful.
+ *   - (-EINVAL) if bad parameter.
+ *   - (-ENODEV) if FPGA is not probed by ifpga driver.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_get_dev_id(const char *pci_addr, uint16_t *dev_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Update image flash of specified Intel FPGA device
+ *
+ * @param dev_id
+ *   The raw device ID of specified Intel FPGA device.
+ * @param image
+ *   The image file name string.
+ * @param status
+ *   The detailed update status for debug.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-EINVAL) if bad parameter or staging area is not initialized.
+ *   - (-EBUSY) if FPGA is updating or rebooting.
+ *   - (-EIO) if failed to open image file.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_update_flash(uint16_t dev_id, const char *image,
+	uint64_t *status);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Stop flash update of specified Intel FPGA device
+ *
+ * @param dev_id
+ *   The raw device ID of specified Intel FPGA device.
+ * @param force
+ *   Abort the update process by writing register if set non-zero.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-EINVAL) if bad parameter.
+ *   - (-EAGAIN) if failed with force.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_stop_update(uint16_t dev_id, int force);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Check current Intel FPGA status and change it to reboot status if it is idle
+ *
+ * @param dev_id
+ *    The raw device ID of specified Intel FPGA device.
+ * @return
+ *   - (0) if FPGA is ready to reboot.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-ENOMEM) if share data is not initialized.
+ *   - (-EBUSY) if FPGA is updating or rebooting.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_reboot_try(uint16_t dev_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Trigger full reconfiguration of specified Intel FPGA device
+ *
+ * @param dev_id
+ *    The raw device ID of specified Intel FPGA device.
+ * @param type
+ *    Select reconfiguration type.
+ *    0 - reconfigure FPGA only.
+ *    1 - reboot the whole card including FPGA.
+ * @param page
+ *    Select image from which flash partition.
+ *    0 - factory partition.
+ *    1 - user partition.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-EINVAL) if bad parameter.
+ *   - (-EBUSY) if failed to access BMC register.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_reload(uint16_t dev_id, int type, int page);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMD_IFPGA_H_ */
diff --git a/drivers/raw/ifpga/version.map b/drivers/raw/ifpga/version.map
index 4a76d1d52d..16584f7fe4 100644
--- a/drivers/raw/ifpga/version.map
+++ b/drivers/raw/ifpga/version.map
@@ -1,3 +1,14 @@
 DPDK_21 {
 	local: *;
 };
+
+EXPERIMENTAL {
+	global:
+
+	# added in 21.05
+	rte_pmd_ifpga_get_dev_id;
+	rte_pmd_ifpga_update_flash;
+	rte_pmd_ifpga_stop_update;
+	rte_pmd_ifpga_reboot_try;
+	rte_pmd_ifpga_reload;
+};
-- 
2.29.2

[dpdk-dev] [PATCH v13 2/4] raw/ifpga: add APIs to get fpga information

[Wei Huang]

There are some information data can be got from FPGA, they are
implemented in below APIs:
1. rte_pmd_ifpga_get_property() get properties of FPGA (include BMC).
2. rte_pmd_ifpga_get_phy_info() get information of PHY connect to FPGA.
3. rte_pmd_ifpga_get_rsu_status() get status of rsu process.

Signed-off-by: Wei Huang <wei.huang@intel.com>
Acked-by: Tianfei Zhang <tianfei.zhang@intel.com>
Acked-by: Rosen Xu <rosen.xu@intel.com>
---
v2: implement APIs in rte_pmd_ifpga.c
---
 drivers/raw/ifpga/base/ifpga_api.c         |   8 +
 drivers/raw/ifpga/base/ifpga_defines.h     |   1 +
 drivers/raw/ifpga/base/ifpga_feature_dev.c |  21 ++
 drivers/raw/ifpga/base/ifpga_feature_dev.h |   1 +
 drivers/raw/ifpga/base/ifpga_fme.c         |  28 ++-
 drivers/raw/ifpga/base/opae_hw_api.c       |  18 ++
 drivers/raw/ifpga/base/opae_hw_api.h       |   2 +
 drivers/raw/ifpga/base/opae_ifpga_hw_api.h |   1 +
 drivers/raw/ifpga/rte_pmd_ifpga.c          | 221 +++++++++++++++++++++
 drivers/raw/ifpga/rte_pmd_ifpga.h          | 110 ++++++++++
 drivers/raw/ifpga/version.map              |   3 +
 11 files changed, 412 insertions(+), 2 deletions(-)

diff --git a/drivers/raw/ifpga/base/ifpga_api.c b/drivers/raw/ifpga/base/ifpga_api.c
index 1aedf150bc..4610ef101e 100644
--- a/drivers/raw/ifpga/base/ifpga_api.c
+++ b/drivers/raw/ifpga/base/ifpga_api.c
@@ -229,6 +229,13 @@ static int ifpga_mgr_get_board_info(struct opae_manager *mgr,
 	return 0;
 }
 
+static int ifpga_mgr_get_uuid(struct opae_manager *mgr, struct uuid *uuid)
+{
+	struct ifpga_fme_hw *fme = mgr->data;
+
+	return fpga_get_pr_uuid(fme, uuid);
+}
+
 static int ifpga_mgr_update_flash(struct opae_manager *mgr, const char *image,
 	u64 *status)
 {
@@ -256,6 +263,7 @@ struct opae_manager_ops ifpga_mgr_ops = {
 	.get_eth_group_region_info = ifpga_mgr_get_eth_group_region_info,
 	.get_sensor_value = ifpga_mgr_get_sensor_value,
 	.get_board_info = ifpga_mgr_get_board_info,
+	.get_uuid = ifpga_mgr_get_uuid,
 	.update_flash = ifpga_mgr_update_flash,
 	.stop_flash_update = ifpga_mgr_stop_flash_update,
 	.reload = ifpga_mgr_reload,
diff --git a/drivers/raw/ifpga/base/ifpga_defines.h b/drivers/raw/ifpga/base/ifpga_defines.h
index 9f0147d1ed..dca1518a83 100644
--- a/drivers/raw/ifpga/base/ifpga_defines.h
+++ b/drivers/raw/ifpga/base/ifpga_defines.h
@@ -1727,6 +1727,7 @@ struct opae_board_info {
 	u8 seu;
 	u8 ptp;
 
+	u32 boot_page;
 	u32 max10_version;
 	u32 nios_fw_version;
 	u32 nums_of_retimer;
diff --git a/drivers/raw/ifpga/base/ifpga_feature_dev.c b/drivers/raw/ifpga/base/ifpga_feature_dev.c
index 0f852a75ad..08135137ad 100644
--- a/drivers/raw/ifpga/base/ifpga_feature_dev.c
+++ b/drivers/raw/ifpga/base/ifpga_feature_dev.c
@@ -87,6 +87,27 @@ int fpga_get_afu_uuid(struct ifpga_port_hw *port, struct uuid *uuid)
 	return 0;
 }
 
+int fpga_get_pr_uuid(struct ifpga_fme_hw *fme, struct uuid *uuid)
+{
+	struct feature_fme_pr *fme_pr;
+	u64 guidl, guidh;
+
+	if (!fme || !uuid)
+		return -EINVAL;
+
+	fme_pr = get_fme_feature_ioaddr_by_index(fme, FME_FEATURE_ID_PR_MGMT);
+
+	spinlock_lock(&fme->lock);
+	guidl = readq(&fme_pr->fme_pr_intfc_id_l);
+	guidh = readq(&fme_pr->fme_pr_intfc_id_h);
+	spinlock_unlock(&fme->lock);
+
+	opae_memcpy(uuid->b, &guidl, sizeof(u64));
+	opae_memcpy(uuid->b + 8, &guidh, sizeof(u64));
+
+	return 0;
+}
+
 /* Mask / Unmask Port Errors by the Error Mask register. */
 void port_err_mask(struct ifpga_port_hw *port, bool mask)
 {
diff --git a/drivers/raw/ifpga/base/ifpga_feature_dev.h b/drivers/raw/ifpga/base/ifpga_feature_dev.h
index 2b1309b44a..b355d22b0e 100644
--- a/drivers/raw/ifpga/base/ifpga_feature_dev.h
+++ b/drivers/raw/ifpga/base/ifpga_feature_dev.h
@@ -103,6 +103,7 @@ is_port_feature_present(struct ifpga_port_hw *port, int index)
 }
 
 int fpga_get_afu_uuid(struct ifpga_port_hw *port, struct uuid *uuid);
+int fpga_get_pr_uuid(struct ifpga_fme_hw *fme, struct uuid *uuid);
 
 int __fpga_port_disable(struct ifpga_port_hw *port);
 void __fpga_port_enable(struct ifpga_port_hw *port);
diff --git a/drivers/raw/ifpga/base/ifpga_fme.c b/drivers/raw/ifpga/base/ifpga_fme.c
index 34fd9a818e..43c7b9c3dc 100644
--- a/drivers/raw/ifpga/base/ifpga_fme.c
+++ b/drivers/raw/ifpga/base/ifpga_fme.c
@@ -101,6 +101,24 @@ static int fme_hdr_get_ports_num(struct ifpga_fme_hw *fme, u64 *ports_num)
 	return 0;
 }
 
+static int fme_hdr_get_port_type(struct ifpga_fme_hw *fme, u64 *port_type)
+{
+	struct feature_fme_header *fme_hdr
+		= get_fme_feature_ioaddr_by_index(fme, FME_FEATURE_ID_HEADER);
+	struct feature_fme_port pt;
+	u32 port = (u32)((*port_type >> 32) & 0xffffffff);
+
+	pt.csr = readq(&fme_hdr->port[port]);
+	if (!pt.port_implemented)
+		return -ENODEV;
+	if (pt.afu_access_control)
+		*port_type |= 0x1;
+	else
+		*port_type &= ~0x1;
+
+	return 0;
+}
+
 static int fme_hdr_get_cache_size(struct ifpga_fme_hw *fme, u64 *cache_size)
 {
 	struct feature_fme_header *fme_hdr
@@ -179,6 +197,8 @@ fme_hdr_get_prop(struct ifpga_feature *feature, struct feature_prop *prop)
 		return fme_hdr_get_bitstream_id(fme, &prop->data);
 	case FME_HDR_PROP_BITSTREAM_METADATA:
 		return fme_hdr_get_bitstream_metadata(fme, &prop->data);
+	case FME_HDR_PROP_PORT_TYPE:
+		return fme_hdr_get_port_type(fme, &prop->data);
 	}
 
 	return -ENOENT;
@@ -891,13 +911,17 @@ static int fme_get_board_interface(struct ifpga_fme_hw *fme)
 			fme->board_info.nums_of_fvl,
 			fme->board_info.ports_per_fvl);
 
+	if (max10_sys_read(fme->max10_dev, FPGA_PAGE_INFO, &val))
+		return -EINVAL;
+	fme->board_info.boot_page = val & 0x7;
+
 	if (max10_sys_read(fme->max10_dev, MAX10_BUILD_VER, &val))
 		return -EINVAL;
-	fme->board_info.max10_version = val & 0xffffff;
+	fme->board_info.max10_version = val;
 
 	if (max10_sys_read(fme->max10_dev, NIOS2_FW_VERSION, &val))
 		return -EINVAL;
-	fme->board_info.nios_fw_version = val & 0xffffff;
+	fme->board_info.nios_fw_version = val;
 
 	dev_info(fme, "max10 version 0x%x, nios fw version 0x%x\n",
 		fme->board_info.max10_version,
diff --git a/drivers/raw/ifpga/base/opae_hw_api.c b/drivers/raw/ifpga/base/opae_hw_api.c
index 86ad88f720..11c9887c7f 100644
--- a/drivers/raw/ifpga/base/opae_hw_api.c
+++ b/drivers/raw/ifpga/base/opae_hw_api.c
@@ -967,6 +967,24 @@ opae_mgr_get_board_info(struct opae_manager *mgr,
 	return -ENOENT;
 }
 
+/**
+ * opae_mgr_get_uuid -  get manager's UUID.
+ * @mgr: targeted manager
+ * @uuid: a pointer to UUID
+ *
+ * Return: 0 on success, otherwise error code.
+ */
+int opae_mgr_get_uuid(struct opae_manager *mgr, struct uuid *uuid)
+{
+	if (!mgr || !uuid)
+		return -EINVAL;
+
+	if (mgr->ops && mgr->ops->get_uuid)
+		return mgr->ops->get_uuid(mgr, uuid);
+
+	return -ENOENT;
+}
+
 /**
  * opae_mgr_update_flash -  update image in flash.
  * @mgr: targeted manager
diff --git a/drivers/raw/ifpga/base/opae_hw_api.h b/drivers/raw/ifpga/base/opae_hw_api.h
index 91d26d9b5b..7e04b56471 100644
--- a/drivers/raw/ifpga/base/opae_hw_api.h
+++ b/drivers/raw/ifpga/base/opae_hw_api.h
@@ -55,6 +55,7 @@ struct opae_manager_ops {
 			unsigned int *value);
 	int (*get_board_info)(struct opae_manager *mgr,
 			struct opae_board_info **info);
+	int (*get_uuid)(struct opae_manager *mgr, struct uuid *uuid);
 	int (*update_flash)(struct opae_manager *mgr, const char *image,
 			u64 *status);
 	int (*stop_flash_update)(struct opae_manager *mgr, int force);
@@ -360,6 +361,7 @@ int opae_manager_eth_group_read_reg(struct opae_manager *mgr, u8 group_id,
 		u8 type, u8 index, u16 addr, u32 *data);
 int opae_mgr_get_board_info(struct opae_manager *mgr,
 		struct opae_board_info **info);
+int opae_mgr_get_uuid(struct opae_manager *mgr, struct uuid *uuid);
 int opae_mgr_update_flash(struct opae_manager *mgr, const char *image,
 		uint64_t *status);
 int opae_mgr_stop_flash_update(struct opae_manager *mgr, int force);
diff --git a/drivers/raw/ifpga/base/opae_ifpga_hw_api.h b/drivers/raw/ifpga/base/opae_ifpga_hw_api.h
index bab33862ee..ffdbebf704 100644
--- a/drivers/raw/ifpga/base/opae_ifpga_hw_api.h
+++ b/drivers/raw/ifpga/base/opae_ifpga_hw_api.h
@@ -61,6 +61,7 @@ struct feature_prop {
 #define FME_HDR_PROP_SOCKET_ID		0x5	/* RDONLY */
 #define FME_HDR_PROP_BITSTREAM_ID		0x6	/* RDONLY */
 #define FME_HDR_PROP_BITSTREAM_METADATA	0x7	/* RDONLY */
+#define FME_HDR_PROP_PORT_TYPE		0x8	/* RDWR */
 
 /* FME error reporting feature's properties */
 /* FME error reporting properties format */
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.c b/drivers/raw/ifpga/rte_pmd_ifpga.c
index af6f175e98..8e04e22d5f 100644
--- a/drivers/raw/ifpga/rte_pmd_ifpga.c
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.c
@@ -8,6 +8,7 @@
 #include <rte_rawdev_pmd.h>
 #include "rte_pmd_ifpga.h"
 #include "ifpga_rawdev.h"
+#include "base/ifpga_api.h"
 #include "base/ifpga_sec_mgr.h"
 
 
@@ -99,6 +100,226 @@ get_share_data(struct opae_adapter *adapter)
 	return sd;
 }
 
+int
+rte_pmd_ifpga_get_rsu_status(uint16_t dev_id, uint32_t *stat, uint32_t *prog)
+{
+	struct opae_adapter *adapter = NULL;
+	opae_share_data *sd = NULL;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	sd = get_share_data(adapter);
+	if (!sd)
+		return -ENOMEM;
+
+	if (stat)
+		*stat = IFPGA_RSU_GET_STAT(sd->rsu_stat);
+	if (prog)
+		*prog = IFPGA_RSU_GET_PROG(sd->rsu_stat);
+
+	return 0;
+}
+
+static int
+ifpga_is_rebooting(struct opae_adapter *adapter)
+{
+	opae_share_data *sd = NULL;
+
+	sd = get_share_data(adapter);
+	if (!sd)
+		return 1;
+
+	if (IFPGA_RSU_GET_STAT(sd->rsu_stat) == IFPGA_RSU_REBOOT) {
+		IFPGA_RAWDEV_PMD_WARN("Reboot is in progress.");
+		return 1;
+	}
+
+	return 0;
+}
+
+static int
+get_common_property(struct opae_adapter *adapter,
+	rte_pmd_ifpga_common_prop *prop)
+{
+	struct ifpga_fme_hw *fme = NULL;
+	struct opae_board_info *info = NULL;
+	struct feature_prop fp;
+	struct uuid pr_id;
+	int ret = 0;
+
+	if (!adapter || !prop)
+		return -EINVAL;
+
+	if (!adapter->mgr || !adapter->mgr->data) {
+		IFPGA_RAWDEV_PMD_ERR("Manager is not registered.");
+		return -ENODEV;
+	}
+
+	fme = adapter->mgr->data;
+	fp.feature_id = FME_FEATURE_ID_HEADER;
+	fp.prop_id = FME_HDR_PROP_PORTS_NUM;
+	ret = ifpga_get_prop(fme->parent, FEATURE_FIU_ID_FME, 0, &fp);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get port number.");
+		return ret;
+	}
+	prop->num_ports = fp.data;
+
+	fp.prop_id = FME_HDR_PROP_BITSTREAM_ID;
+	ret = ifpga_get_prop(fme->parent, FEATURE_FIU_ID_FME, 0, &fp);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get bitstream ID.");
+		return ret;
+	}
+	prop->bitstream_id = fp.data;
+
+	fp.prop_id = FME_HDR_PROP_BITSTREAM_METADATA;
+	ret = ifpga_get_prop(fme->parent, FEATURE_FIU_ID_FME, 0, &fp);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get bitstream metadata.");
+		return ret;
+	}
+	prop->bitstream_metadata = fp.data;
+
+	ret = opae_mgr_get_uuid(adapter->mgr, &pr_id);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get PR ID.");
+		return ret;
+	}
+	memcpy(prop->pr_id.b, pr_id.b, sizeof(rte_pmd_ifpga_uuid));
+
+	ret = opae_mgr_get_board_info(adapter->mgr, &info);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get board info.");
+		return ret;
+	}
+	prop->boot_page = info->boot_page;
+	prop->bmc_version = info->max10_version;
+	prop->bmc_nios_version = info->nios_fw_version;
+
+	return 0;
+}
+
+static int
+get_port_property(struct opae_adapter *adapter, uint16_t port,
+	rte_pmd_ifpga_port_prop *prop)
+{
+	struct ifpga_fme_hw *fme = NULL;
+	struct feature_prop fp;
+	struct opae_accelerator *acc = NULL;
+	struct uuid afu_id;
+	int ret = 0;
+
+	if (!adapter || !prop)
+		return -EINVAL;
+
+	if (!adapter->mgr || !adapter->mgr->data) {
+		IFPGA_RAWDEV_PMD_ERR("Manager is not registered.");
+		return -ENODEV;
+	}
+
+	fme = adapter->mgr->data;
+	fp.feature_id = FME_FEATURE_ID_HEADER;
+	fp.prop_id = FME_HDR_PROP_PORT_TYPE;
+	fp.data = port;
+	fp.data <<= 32;
+	ret = ifpga_get_prop(fme->parent, FEATURE_FIU_ID_FME, 0, &fp);
+	if (ret)
+		return ret;
+	prop->type = fp.data & 0xffffffff;
+
+	if (prop->type == 0) {
+		acc = opae_adapter_get_acc(adapter, port);
+		ret = opae_acc_get_uuid(acc, &afu_id);
+		if (ret) {
+			IFPGA_RAWDEV_PMD_ERR("Failed to get port%u AFU ID.",
+				port);
+			return ret;
+		}
+		memcpy(prop->afu_id.b, afu_id.b, sizeof(rte_pmd_ifpga_uuid));
+	}
+
+	return 0;
+}
+
+int
+rte_pmd_ifpga_get_property(uint16_t dev_id, rte_pmd_ifpga_prop *prop)
+{
+	struct opae_adapter *adapter = NULL;
+	uint32_t i = 0;
+	int ret = 0;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	opae_adapter_lock(adapter, -1);
+	if (ifpga_is_rebooting(adapter)) {
+		ret = -EBUSY;
+		goto unlock_dev;
+	}
+
+	ret = get_common_property(adapter, &prop->common);
+	if (ret) {
+		ret = -EIO;
+		goto unlock_dev;
+	}
+
+	for (i = 0; i < prop->common.num_ports; i++) {
+		ret = get_port_property(adapter, i, &prop->port[i]);
+		if (ret) {
+			ret = -EIO;
+			break;
+		}
+	}
+
+unlock_dev:
+	opae_adapter_unlock(adapter);
+	return ret;
+}
+
+int
+rte_pmd_ifpga_get_phy_info(uint16_t dev_id, rte_pmd_ifpga_phy_info *info)
+{
+	struct opae_adapter *adapter = NULL;
+	struct opae_retimer_info rtm_info;
+	struct opae_retimer_status rtm_status;
+	int ret = 0;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	opae_adapter_lock(adapter, -1);
+	if (ifpga_is_rebooting(adapter)) {
+		ret = -EBUSY;
+		goto unlock_dev;
+	}
+
+	ret = opae_manager_get_retimer_info(adapter->mgr, &rtm_info);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get retimer info.");
+		ret = -EIO;
+		goto unlock_dev;
+	}
+	info->num_retimers = rtm_info.nums_retimer;
+
+	ret = opae_manager_get_retimer_status(adapter->mgr, &rtm_status);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get retimer status.");
+		ret = -EIO;
+		goto unlock_dev;
+	}
+	info->link_speed = rtm_status.speed;
+	info->link_status = rtm_status.line_link_bitmap;
+
+unlock_dev:
+	opae_adapter_unlock(adapter);
+	return ret;
+}
+
 int
 rte_pmd_ifpga_update_flash(uint16_t dev_id, const char *image,
 	uint64_t *status)
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.h b/drivers/raw/ifpga/rte_pmd_ifpga.h
index 023a011116..633f6e9d1b 100644
--- a/drivers/raw/ifpga/rte_pmd_ifpga.h
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.h
@@ -18,6 +18,55 @@
 extern "C" {
 #endif
 
+#include <stdint.h>
+
+#define IFPGA_MAX_PORT_NUM   4
+
+/**
+ * UUID data structure.
+ */
+typedef struct {
+	uint8_t b[16];
+} rte_pmd_ifpga_uuid;
+
+/**
+ * FME property data structure.
+ */
+typedef struct {
+	uint32_t num_ports;
+	uint32_t boot_page;
+	uint64_t bitstream_id;
+	uint64_t bitstream_metadata;
+	rte_pmd_ifpga_uuid pr_id;
+	uint32_t bmc_version;
+	uint32_t bmc_nios_version;
+} rte_pmd_ifpga_common_prop;
+
+/**
+ * port property data structure.
+ */
+typedef struct {
+	rte_pmd_ifpga_uuid afu_id;
+	uint32_t type;   /* AFU memory access control type */
+} rte_pmd_ifpga_port_prop;
+
+/**
+ * FPGA property data structure.
+ */
+typedef struct {
+	rte_pmd_ifpga_common_prop  common;
+	rte_pmd_ifpga_port_prop    port[IFPGA_MAX_PORT_NUM];
+} rte_pmd_ifpga_prop;
+
+/**
+ * PHY information data structure.
+ */
+typedef struct {
+	uint32_t num_retimers;
+	uint32_t link_speed;
+	uint32_t link_status;
+} rte_pmd_ifpga_phy_info;
+
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
@@ -37,6 +86,67 @@ __rte_experimental
 int
 rte_pmd_ifpga_get_dev_id(const char *pci_addr, uint16_t *dev_id);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Get current RSU status of the specified Intel FPGA device
+ *
+ * @param dev_id
+ *    The raw device ID of specified Intel FPGA device.
+ * @param stat
+ *    The buffer to output RSU status.
+ * @param prog
+ *    The buffer to output RSU progress.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-ENOMEM) if share data is not initialized.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_get_rsu_status(uint16_t dev_id, uint32_t *stat, uint32_t *prog);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Get FPGA property of specified Intel FPGA device
+ *
+ * @param dev_id
+ *    The raw device ID of specified Intel FPGA device.
+ * @param prop
+ *    The data pointer of FPGA property buffer.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-EBUSY) if FPGA is rebooting.
+ *   - (-EIO) if failed to access hardware.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_get_property(uint16_t dev_id, rte_pmd_ifpga_prop *prop);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Get PHY information of specified Intel FPGA device
+ *
+ * @param dev_id
+ *    The raw device ID of specified Intel FPGA device.
+ * @param info
+ *    The data pointer of PHY information buffer.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-EBUSY) if FPGA is rebooting.
+ *   - (-EIO) if failed to access hardware.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_get_phy_info(uint16_t dev_id, rte_pmd_ifpga_phy_info *info);
+
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
diff --git a/drivers/raw/ifpga/version.map b/drivers/raw/ifpga/version.map
index 16584f7fe4..ca6f7f5810 100644
--- a/drivers/raw/ifpga/version.map
+++ b/drivers/raw/ifpga/version.map
@@ -7,6 +7,9 @@ EXPERIMENTAL {
 
 	# added in 21.05
 	rte_pmd_ifpga_get_dev_id;
+	rte_pmd_ifpga_get_rsu_status;
+	rte_pmd_ifpga_get_property;
+	rte_pmd_ifpga_get_phy_info;
 	rte_pmd_ifpga_update_flash;
 	rte_pmd_ifpga_stop_update;
 	rte_pmd_ifpga_reboot_try;
-- 
2.29.2

[dpdk-dev] [PATCH v13 3/4] raw/ifpga: add miscellaneous APIs

[Wei Huang]

Below miscellaneous APIs are used to implement OPAE application.
1. rte_pmd_ifpga_get_pci_bus() get PCI bus ifpga driver registered.
2. rte_pmd_ifpga_partial_reconfigure() do partial reconfiguration.
3. rte_pmd_ifpga_cleanup() free software resources allocated by driver.
4. rte_pmd_ifpga_set_rsu_status() set status of rsu process.

Signed-off-by: Wei Huang <wei.huang@intel.com>
Acked-by: Tianfei Zhang <tianfei.zhang@intel.com>
Acked-by: Rosen Xu <rosen.xu@intel.com>
---
v2: implement APIs in rte_pmd_ifpga.c
---
 drivers/raw/ifpga/ifpga_rawdev.c  | 30 ++++++++++++++
 drivers/raw/ifpga/ifpga_rawdev.h  |  7 +++-
 drivers/raw/ifpga/rte_pmd_ifpga.c | 45 +++++++++++++++++++++
 drivers/raw/ifpga/rte_pmd_ifpga.h | 66 +++++++++++++++++++++++++++++++
 drivers/raw/ifpga/version.map     |  4 ++
 5 files changed, 151 insertions(+), 1 deletion(-)

diff --git a/drivers/raw/ifpga/ifpga_rawdev.c b/drivers/raw/ifpga/ifpga_rawdev.c
index 27129b133e..05d79bfcc2 100644
--- a/drivers/raw/ifpga/ifpga_rawdev.c
+++ b/drivers/raw/ifpga/ifpga_rawdev.c
@@ -1737,3 +1737,33 @@ RTE_PMD_REGISTER_PARAM_STRING(ifpga_rawdev_cfg,
 	"ifpga=<string> "
 	"port=<int> "
 	"afu_bts=<path>");
+
+struct rte_pci_bus *ifpga_get_pci_bus(void)
+{
+	return rte_ifpga_rawdev_pmd.bus;
+}
+
+int ifpga_rawdev_partial_reconfigure(struct rte_rawdev *dev, int port,
+	const char *file)
+{
+	if (!dev) {
+		IFPGA_RAWDEV_PMD_ERR("Input parameter is invalid");
+		return -EINVAL;
+	}
+
+	return rte_fpga_do_pr(dev, port, file);
+}
+
+void ifpga_rawdev_cleanup(void)
+{
+	struct ifpga_rawdev *dev;
+	unsigned int i;
+
+	for (i = 0; i < IFPGA_RAWDEV_NUM; i++) {
+		dev = &ifpga_rawdevices[i];
+		if (dev->rawdev) {
+			rte_rawdev_pmd_release(dev->rawdev);
+			dev->rawdev = NULL;
+		}
+	}
+}
diff --git a/drivers/raw/ifpga/ifpga_rawdev.h b/drivers/raw/ifpga/ifpga_rawdev.h
index 7754beb02b..9bbe9a4278 100644
--- a/drivers/raw/ifpga/ifpga_rawdev.h
+++ b/drivers/raw/ifpga/ifpga_rawdev.h
@@ -43,7 +43,7 @@ enum ifpga_rawdev_device_state {
 static inline struct opae_adapter *
 ifpga_rawdev_get_priv(const struct rte_rawdev *rawdev)
 {
-	return rawdev->dev_private;
+	return (struct opae_adapter *)rawdev->dev_private;
 }
 
 #define IFPGA_RAWDEV_MSIX_IRQ_NUM 7
@@ -76,4 +76,9 @@ int
 ifpga_unregister_msix_irq(enum ifpga_irq_type type,
 		int vec_start, rte_intr_callback_fn handler, void *arg);
 
+struct rte_pci_bus *ifpga_get_pci_bus(void);
+int ifpga_rawdev_partial_reconfigure(struct rte_rawdev *dev, int port,
+	const char *file);
+void ifpga_rawdev_cleanup(void);
+
 #endif /* _IFPGA_RAWDEV_H_ */
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.c b/drivers/raw/ifpga/rte_pmd_ifpga.c
index 8e04e22d5f..6e23a2581a 100644
--- a/drivers/raw/ifpga/rte_pmd_ifpga.c
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.c
@@ -122,6 +122,25 @@ rte_pmd_ifpga_get_rsu_status(uint16_t dev_id, uint32_t *stat, uint32_t *prog)
 	return 0;
 }
 
+int
+rte_pmd_ifpga_set_rsu_status(uint16_t dev_id, uint32_t stat, uint32_t prog)
+{
+	struct opae_adapter *adapter = NULL;
+	opae_share_data *sd = NULL;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	sd = get_share_data(adapter);
+	if (!sd)
+		return -ENOMEM;
+
+	sd->rsu_stat = IFPGA_RSU_STATUS(stat, prog);
+
+	return 0;
+}
+
 static int
 ifpga_is_rebooting(struct opae_adapter *adapter)
 {
@@ -382,3 +401,29 @@ rte_pmd_ifpga_reload(uint16_t dev_id, int type, int page)
 
 	return opae_mgr_reload(adapter->mgr, type, page);
 }
+
+const struct rte_pci_bus *
+rte_pmd_ifpga_get_pci_bus(void)
+{
+	return ifpga_get_pci_bus();
+}
+
+int
+rte_pmd_ifpga_partial_reconfigure(uint16_t dev_id, int port, const char *file)
+{
+	struct rte_rawdev *dev = NULL;
+
+	dev = get_rte_rawdev(dev_id);
+	if (!dev) {
+		IFPGA_RAWDEV_PMD_ERR("Device ID %u is invalid.", dev_id);
+		return -EINVAL;
+	}
+
+	return ifpga_rawdev_partial_reconfigure(dev, port, file);
+}
+
+void
+rte_pmd_ifpga_cleanup(void)
+{
+	ifpga_rawdev_cleanup();
+}
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.h b/drivers/raw/ifpga/rte_pmd_ifpga.h
index 633f6e9d1b..47d66ba655 100644
--- a/drivers/raw/ifpga/rte_pmd_ifpga.h
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.h
@@ -107,6 +107,27 @@ __rte_experimental
 int
 rte_pmd_ifpga_get_rsu_status(uint16_t dev_id, uint32_t *stat, uint32_t *prog);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Set current RSU status of the specified Intel FPGA device
+ *
+ * @param dev_id
+ *    The raw device ID of specified Intel FPGA device.
+ * @param stat
+ *    The RSU status value to set.
+ * @param prog
+ *    The RSU progress value to set.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-ENOMEM) if share data is not initialized.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_set_rsu_status(uint16_t dev_id, uint32_t stat, uint32_t prog);
+
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
@@ -235,6 +256,51 @@ __rte_experimental
 int
 rte_pmd_ifpga_reload(uint16_t dev_id, int type, int page);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Get PCI bus the Intel FPGA driver register to
+ *
+ * @return
+ *   - (valid pointer) if successful.
+ *   - (NULL) if the Intel FPGA driver is not registered to any PCI bus.
+ */
+__rte_experimental
+const struct rte_pci_bus *
+rte_pmd_ifpga_get_pci_bus(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Perform PR (partial reconfiguration) on specified Intel FPGA device
+ *
+ * @param dev_id
+ *   The raw device ID of specified Intel FPGA device.
+ * @param port
+ *   The port index of the partial reconfiguration area.
+ * @param file
+ *   The GBS (Green BitStream) image file name string.
+ * @return
+ *   - (0) if successful.
+ *   - (-EINVAL) if bad parameter or operation failed.
+ *   - (-ENOMEM) if failed to allocate memory.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_partial_reconfigure(uint16_t dev_id, int port, const char *file);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Free software resources allocated by Intel FPGA PMD
+ */
+__rte_experimental
+void
+rte_pmd_ifpga_cleanup(void);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/drivers/raw/ifpga/version.map b/drivers/raw/ifpga/version.map
index ca6f7f5810..995c419a9b 100644
--- a/drivers/raw/ifpga/version.map
+++ b/drivers/raw/ifpga/version.map
@@ -8,10 +8,14 @@ EXPERIMENTAL {
 	# added in 21.05
 	rte_pmd_ifpga_get_dev_id;
 	rte_pmd_ifpga_get_rsu_status;
+	rte_pmd_ifpga_set_rsu_status;
 	rte_pmd_ifpga_get_property;
 	rte_pmd_ifpga_get_phy_info;
 	rte_pmd_ifpga_update_flash;
 	rte_pmd_ifpga_stop_update;
 	rte_pmd_ifpga_reboot_try;
 	rte_pmd_ifpga_reload;
+	rte_pmd_ifpga_get_pci_bus;
+	rte_pmd_ifpga_partial_reconfigure;
+	rte_pmd_ifpga_cleanup;
 };
-- 
2.29.2

[dpdk-dev] [PATCH v13 4/4] examples/ifpga: add example for ifpga APIs

[Wei Huang]

This example application shows how to use ifpga APIs.
A set of OPAE APIs are implemented based on these ifpga APIs.
You can test each OPAE API by running corresponding command.
A guide is also added to show how to run the example.

Signed-off-by: Wei Huang <wei.huang@intel.com>
Acked-by: Tianfei Zhang <tianfei.zhang@intel.com>
Acked-by: Rosen Xu <rosen.xu@intel.com>
---
v2: fix coding style issue in commands.c
---
v3: add guide for running example
---
v4: fix compilation issue of ifpga.rst
---
v5: add ifpga.rst into sample_app_ug/index.rst
---
v6: implement OPAE APIs in example instead of in driver
---
 MAINTAINERS                        |    3 +
 doc/guides/sample_app_ug/ifpga.rst |  387 +++++++
 doc/guides/sample_app_ug/index.rst |    1 +
 examples/ifpga/Makefile            |   43 +
 examples/ifpga/commands.c          | 1294 ++++++++++++++++++++++
 examples/ifpga/commands.h          |   16 +
 examples/ifpga/main.c              |   38 +
 examples/ifpga/meson.build         |   20 +
 examples/ifpga/opae_api.c          | 1632 ++++++++++++++++++++++++++++
 examples/ifpga/opae_api.h          |  244 +++++
 examples/meson.build               |    2 +-
 11 files changed, 3679 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/sample_app_ug/ifpga.rst
 create mode 100644 examples/ifpga/Makefile
 create mode 100644 examples/ifpga/commands.c
 create mode 100644 examples/ifpga/commands.h
 create mode 100644 examples/ifpga/main.c
 create mode 100644 examples/ifpga/meson.build
 create mode 100644 examples/ifpga/opae_api.c
 create mode 100644 examples/ifpga/opae_api.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 1a12916f56..48c2ea44d4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1236,9 +1236,12 @@ Rawdev Drivers
 Intel FPGA
 M: Rosen Xu <rosen.xu@intel.com>
 M: Tianfei zhang <tianfei.zhang@intel.com>
+M: Wei Huang <wei.huang@intel.com>
 T: git://dpdk.org/next/dpdk-next-net-intel
 F: drivers/raw/ifpga/
 F: doc/guides/rawdevs/ifpga.rst
+F: examples/ifpga/
+F: doc/guides/sample_app_ug/ifpga.rst
 
 IOAT Rawdev
 M: Bruce Richardson <bruce.richardson@intel.com>
diff --git a/doc/guides/sample_app_ug/ifpga.rst b/doc/guides/sample_app_ug/ifpga.rst
new file mode 100644
index 0000000000..bc05153418
--- /dev/null
+++ b/doc/guides/sample_app_ug/ifpga.rst
@@ -0,0 +1,387 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020-2021 Intel Corporation.
+
+Intel FPGA Sample Application
+=============================
+
+The Intel FPGA sample application is an example of how to use rte_pmd_ifpga API
+to manage Intel FPGA.
+
+Overview
+--------
+
+The Intel FPGA sample application is a simple application that demonstrates
+the use of the APIs provided by ifpga driver in the DPDK.
+A set of OPAE APIs are implemented in this example with 'opae' prefix. These
+APIs use PCI address ('Domain:Bus:Dev.Func') as an unique ID of FPGA,
+application based on OPAE can manage specified FPGA easily with them.
+OPAE (Open Programmable Acceleration Engine) is a software framework for
+managing and accessing programmable accelerators (FPGAs). For more information
+about OPAE, please refer to https://opae.github.io/2.0.0/index.html .
+This application is a readline-like interface that can be used to manage
+Intel FPGA, in a Linux* application environment.
+
+Compiling the Application
+-------------------------
+
+To compile the sample application see :doc:`compiling`
+
+The application is located in the ``ifpga`` sub-directory.
+
+Running the Application
+-----------------------
+
+To run the application in linux environment, issue the following command:
+
+.. code-block:: console
+
+    $ ./<build_dir>/examples/dpdk-ifpga --proc-type=auto
+
+Refer to the *DPDK Getting Started Guide* for general information on running
+applications and the Environment Abstraction Layer (EAL) options.
+
+Commands Description
+--------------------
+
+The following sections provide some explanation of the commands.
+
+help command
+~~~~~~~~~~~~
+
+The application has on-line help for the commands that are available at runtime.
+
+.. code-block:: console
+
+   opae> help
+    get_api_version                        get OPAE API version
+    get_proc_type                          get DPDK process type
+    get_image_info <FILE>                  get information of image file
+    get_status <BDF>                       get current status & progress of FPGA
+    get_property <BDF> <0|1|2|4|8>         get property of FPGA
+    get_phy_info <BDF>                     get information of PHY
+    get_parent <BDF>                       get parent PCI device of FPGA
+    get_child <BDF>                        get child PCI device of FPGA
+    get_pf1 <BDF>                          get physical function 1 device of FPGA
+    set_log_level <0-4>                    set logging level
+    set_log_file <FILE>                    set logging file
+    set_status <BDF> <0-4> <0-100>         set current status & progress of FPGA
+    enumerate <VID> <DID>                  enumerate specified FPGA
+    bind <BDF> <DRIVER>                    bind FPGA with kernel driver
+    unbind <BDF>                           unbind FPGA from kernel driver
+    probe <BDF>                            probe FPGA with IFPGA driver
+    remove <BDF>                           remove FPGA from IFPGA driver
+    flash <BDF> <FILE>                     update flash of FPGA
+    pr <BDF> <PORT> <FILE>                 partial reconfigure FPGA
+    reboot <BDF> <fpga|bmc> <0-1>          reboot FPGA or MAX10
+    cancel <BDF>                           cancel flash update
+    pci_read <BDF> <0-1024>                read PCI configuration space
+    pci_write <BDF> <0-1024> <NUM>         write PCI configuration space
+    quit                                   exit DPDK application
+    help                                   show commands list
+
+get_api_version command
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Show OPAE API version which is same to the version of DPDK.
+
+.. code-block:: console
+
+   opae> get_api_version
+   21.2.0
+
+set_log_level command
+~~~~~~~~~~~~~~~~~~~~~
+
+Set logging level of OPAE API. Logging level is defined as below.
+0 - CRITICAL
+1 - ERROR
+2 - WARNING
+3 - INFORMATION
+4 - DEBUG
+
+.. code-block:: console
+
+   opae> set_log_level 4
+   OPAE-API: Current log level is DEBUG
+   Successful
+   opae> set_log_level 6
+   OPAE-API: Current log level is DEBUG
+   Failed
+
+set_log_file command
+~~~~~~~~~~~~~~~~~~~~
+
+Set logging file of OPAE API.
+
+.. code-block:: console
+
+   opae> set_log_file /tmp/ifpga.log
+   Successful
+
+get_proc_type command
+~~~~~~~~~~~~~~~~~~~~~
+
+Show the process type of DPDK. If you start multiple instances of the
+application, the process type of the first one is 'Primary', the others
+are 'Secondary'.
+
+.. code-block:: console
+
+   opae> get_proc_type
+   Primary
+
+get_image_info command
+~~~~~~~~~~~~~~~~~~~~~~
+
+Display information of FPGA image file.
+
+.. code-block:: console
+
+   opae> get_image_info /home/wei/a10.bin
+   Type:           FPGA_BBS
+   Action:         UPDATE
+   Total length:   58720256
+   Payload offset: 1024
+   Payload length: 58719232
+   opae> get_image_info /home/wei/data.bin
+   OPAE-ERR: Image '/home/wei/data.bin' can not be recognized
+   Invalid image file
+
+enumerate command
+~~~~~~~~~~~~~~~~~
+
+Display PCI address of FPGA with specified vendor ID and device ID. ID value can
+be set to 0xffff for arbitrary ID.
+
+.. code-block:: console
+
+   opae> enumerate 0x8086 0x0b30
+   0000:24:00.0
+
+get_property command
+~~~~~~~~~~~~~~~~~~~~
+
+Display property information of specified FPGA. Property type is defined as below.
+0 - All properties
+1 - PCI property
+2 - FME property
+4 - port property
+8 - BMC property
+PCI property is always available, other properties can only be displayed after
+ifpga driver is probed to the FPGA.
+
+.. code-block:: console
+
+   opae> get_property 24:00.0 0
+   PCI:
+    PCIe s:b:d.f     : 0000:24:00.0
+    kernel driver    : vfio-pci
+   FME:
+    platform         : Vista Creek
+    DCP version      : DCP 1.2
+    phase            : Beta
+    interface        : 2x2x25G
+    build version    : 0.0.2
+    ports num        : 1
+    boot page        : user
+    pr interface id  : a5d72a3c-c8b0-4939-912c-f715e5dc10ca
+   PORT0:
+    access type      : PF
+    accelerator id   : 8892c23e-2eed-4b44-8bb6-5c88606e07df
+   BMC:
+    MAX10 version    : D.2.0.5
+    NIOS FW version  : D.2.0.12
+
+get_phy_info command
+~~~~~~~~~~~~~~~~~~~~
+
+Display information and status of PHY connects to the specified FPGA.
+
+.. code-block:: console
+
+   opae> get_phy_info 24:00.0
+    retimers num     : 2
+    link speed       : 25G
+    link status      : 00
+
+get_parent command
+~~~~~~~~~~~~~~~~~~
+
+Display PCI address of upstream device connects to the specified FPGA.
+
+.. code-block:: console
+
+   opae> get_parent 24:00.0
+   0000:22:09.0
+
+get_child command
+~~~~~~~~~~~~~~~~~
+
+Display PCI address of downstream device connects to the specified FPGA.
+
+.. code-block:: console
+
+   opae> get_child 24:00.0
+   No child
+   opae> get_child 22:09.0
+   0000:24:00.0
+
+get_pf1 command
+~~~~~~~~~~~~~~~
+
+Display PCI address of PF1 (physical function 1) of specified FPGA.
+
+.. code-block:: console
+
+   opae> get_pf1 24:00.0
+   0000:26:00.0
+   0000:26:00.1
+
+get_status command
+~~~~~~~~~~~~~~~~~~
+
+Display current RSU status of specified FPGA.
+
+.. code-block:: console
+
+   opae> get_status 24:00.0
+   Status:   IDLE
+   Progress: 0%
+
+set_status command
+~~~~~~~~~~~~~~~~~~
+
+Set current RSU status of specified FPGA. This command is mainly used for debug
+purpose. Status value is defined as below.
+0 - IDLE
+1 - PREPARE
+2 - PROGRAM
+3 - COPY
+4 - REBOOT
+
+.. code-block:: console
+
+   opae> set_status 24:00.0 2 35
+   Successful
+   opae> get_status 24:00.0
+   Status:   PROGRAM
+   Progress: 35%
+
+unbind command
+~~~~~~~~~~~~~~
+
+Unbind kernel driver from specified FPGA.
+
+.. code-block:: console
+
+   opae> unbind 24:00.0
+   OPAE-ERR: 0000:24:00.0 is probed, remove it first
+   Failed
+   opae> remove 24:00.0
+   Successful
+   opae> unbind 24:00.0
+   Successful
+
+bind command
+~~~~~~~~~~~~
+
+Bind specified kernel driver to specified FPGA.
+
+.. code-block:: console
+
+   opae> bind 24:00.0 vfio-pci
+   Successful
+
+probe command
+~~~~~~~~~~~~~
+
+Probe specified FPGA with DPDK PMD driver.
+
+.. code-block:: console
+
+   opae> probe 24:00.0
+   Successful
+
+remove command
+~~~~~~~~~~~~~~
+
+Remove specified FPGA from DPDK PMD driver. It's a reverse operation to probe
+command.
+
+.. code-block:: console
+
+   opae> remove 24:00.0
+   Successful
+
+flash command
+~~~~~~~~~~~~~
+
+Update image in flash of specified FPGA.
+
+.. code-block:: console
+
+   opae> flash 24:00.0 /home/wei/a10.bin
+   Successful
+
+pr command
+~~~~~~~~~~
+
+Do partial reconfiguration of specified FPGA.
+
+.. code-block:: console
+
+   opae> pr 24:00.0 0 /home/wei/nlb0.gbs
+   Successful
+
+reboot command
+~~~~~~~~~~~~~~
+
+Reboot specified FPGA. Reboot type and page is defined as below.
+fpga - reboot FPGA only
+bmc - reboot whole card with FPGA
+0 - factory page
+1 - user page
+
+.. code-block:: console
+
+   opae> reboot 24:00.0 fpga 1
+   Successful
+
+cancel command
+~~~~~~~~~~~~~~
+
+Cancel flash programming of specified FPGA.
+
+.. code-block:: console
+
+   opae> cancel 24:00.0
+   Successful
+
+pci_read command
+~~~~~~~~~~~~~~~~
+
+Read PCI configuration space of specified FPGA.
+
+.. code-block:: console
+
+   opae> pci_read 24:00.0 0
+   0x0b308086
+
+pci_write command
+~~~~~~~~~~~~~~~~~
+
+Write PCI configuration space of specified FPGA.
+
+.. code-block:: console
+
+   opae> pci_write 24:00.0 4 0x100406
+   Successful
+
+quit command
+~~~~~~~~~~~~
+
+Exit this sample application.
+
+.. code-block:: console
+
+   opae> quit
diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
index e8db83d3a7..beb94ff3e7 100644
--- a/doc/guides/sample_app_ug/index.rst
+++ b/doc/guides/sample_app_ug/index.rst
@@ -23,6 +23,7 @@ Sample Applications User Guides
     kernel_nic_interface
     keep_alive
     ioat
+    ifpga
     l2_forward_crypto
     l2_forward_job_stats
     l2_forward_real_virtual
diff --git a/examples/ifpga/Makefile b/examples/ifpga/Makefile
new file mode 100644
index 0000000000..7d76479a36
--- /dev/null
+++ b/examples/ifpga/Makefile
@@ -0,0 +1,43 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020-2021 Intel Corporation
+
+# binary name
+APP = ifpga
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c opae_api.c
+
+# Build using pkg-config variables if possible
+ifneq ($(shell pkg-config --exists libdpdk && echo 0),0)
+$(error "no installation of DPDK found")
+endif
+
+all: shared
+.PHONY: shared static
+shared: build/$(APP)-shared
+	ln -sf $(APP)-shared build/$(APP)
+static: build/$(APP)-static
+	ln -sf $(APP)-static build/$(APP)
+
+PKGCONF ?= pkg-config
+
+PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk) -lrte_bus_pci -lrte_raw_ifpga
+LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
+
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
+
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
+
+build:
+	@mkdir -p $@
+
+.PHONY: clean
+clean:
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	test -d build && rmdir -p build || true
diff --git a/examples/ifpga/commands.c b/examples/ifpga/commands.c
new file mode 100644
index 0000000000..ee58382571
--- /dev/null
+++ b/examples/ifpga/commands.c
@@ -0,0 +1,1294 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020-2021 Intel Corporation.
+ * All rights reserved.
+ */
+
+#include <stdio.h>
+#include <stdint.h>
+#include <string.h>
+#include <stdlib.h>
+#include <stdarg.h>
+#include <errno.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+
+#include "commands.h"
+
+static int parse_pciaddr(const char *bdf, opae_pci_device *id)
+{
+	size_t len = 0;
+	unsigned int domain = 0;
+	unsigned int bus = 0;
+	unsigned int devid = 0;
+	unsigned int function = 0;
+
+	if (!bdf || !id)
+		return -EINVAL;
+
+	len = strlen(bdf);
+	if ((len < 5) || (len > 12))
+		return -EINVAL;
+
+	len = sscanf(bdf, "%x:%x:%x.%d", &domain, &bus, &devid, &function);
+	if (len == 4) {
+		snprintf(id->bdf, sizeof(id->bdf), "%04x:%02x:%02x.%d",
+			domain, bus, devid, function);
+	} else {
+		len = sscanf(bdf, "%x:%x.%d", &bus, &devid, &function);
+		if (len == 3) {
+			snprintf(id->bdf, sizeof(id->bdf), "%04x:%02x:%02x.%d",
+				0, bus, devid, function);
+		} else {
+			return -EINVAL;
+		}
+	}
+	return 0;
+}
+
+static void uuid_to_str(opae_uuid *id, uuid_str *str)
+{
+	uint8_t *b = NULL;
+	char *p = NULL;
+	int i, j;
+
+	if (!id || !str)
+		return;
+
+	b = &id->b[15];
+	p = str->s;
+	for (i = 0; i < 4; i++, b--, p += 2)
+		sprintf(p, "%02x", *b);
+	sprintf(p++, "-");
+	for (i = 0; i < 3; i++) {
+		for (j = 0; j < 2; j++, b--, p += 2)
+			sprintf(p, "%02x", *b);
+		sprintf(p++, "-");
+	}
+	for (i = 0; i < 6; i++, b--, p += 2)
+		sprintf(p, "%02x", *b);
+}
+
+/* *** GET API VERSION *** */
+struct cmd_version_result {
+	cmdline_fixed_string_t cmd;
+};
+
+static void cmd_version_parsed(__rte_unused void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	opae_api_version ver;
+	opae_get_api_version(&ver);
+	cmdline_printf(cl, "%d.%d.%d\n", ver.major, ver.minor, ver.micro);
+}
+
+cmdline_parse_token_string_t cmd_version_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_version_result, cmd, "get_api_version");
+
+cmdline_parse_inst_t cmd_get_api_version = {
+	.f = cmd_version_parsed,
+	.data = NULL,
+	.help_str = "get OPAE API version",
+	.tokens = {
+		(void *)&cmd_version_cmd,
+		NULL,
+	},
+};
+
+/* *** GET PROC TYPE *** */
+struct cmd_proc_type_result {
+	cmdline_fixed_string_t cmd;
+};
+
+static void cmd_proc_type_parsed(__rte_unused void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	int type = opae_get_proc_type();
+
+	if (type == 0)
+		cmdline_printf(cl, "Primary\n");
+	else if (type == 1)
+		cmdline_printf(cl, "Secondary\n");
+	else
+		cmdline_printf(cl, "Unknown\n");
+}
+
+cmdline_parse_token_string_t cmd_proc_type_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_proc_type_result, cmd, "get_proc_type");
+
+cmdline_parse_inst_t cmd_get_proc_type = {
+	.f = cmd_proc_type_parsed,
+	.data = NULL,
+	.help_str = "get DPDK process type",
+	.tokens = {
+		(void *)&cmd_proc_type_cmd,
+		NULL,
+	},
+};
+
+/* *** GET IMAGE INFO *** */
+struct cmd_image_info_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t path;
+};
+
+static void cmd_image_info_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_image_info_result *res = parsed_result;
+	opae_img_info info;
+
+	if (opae_get_image_info(res->path, &info) == 0) {
+		cmdline_printf(cl, "%-16s", "Type:");
+		if (info.type == OPAE_IMG_TYPE_BBS)
+			cmdline_printf(cl, "FPGA_BBS\n");
+		else if (info.type == OPAE_IMG_TYPE_BMC)
+			cmdline_printf(cl, "BMC\n");
+		else if (info.type == OPAE_IMG_TYPE_GBS)
+			cmdline_printf(cl, "FGPA_GBS\n");
+		else
+			cmdline_printf(cl, "Unknown\n");
+		cmdline_printf(cl, "%-16s", "Action:");
+		if (info.subtype == OPAE_IMG_SUBTYPE_UPDATE)
+			cmdline_printf(cl, "UPDATE\n");
+		else if (info.subtype == OPAE_IMG_SUBTYPE_CANCELLATION)
+			cmdline_printf(cl, "CANCELLATION\n");
+		else if (info.subtype == OPAE_IMG_SUBTYPE_ROOT_KEY_HASH_256)
+			cmdline_printf(cl, "ROOT_HASH_256\n");
+		else if (info.subtype == OPAE_IMG_SUBTYPE_ROOT_KEY_HASH_384)
+			cmdline_printf(cl, "ROOT_HASH_384\n");
+		else
+			cmdline_printf(cl, "Unknown\n");
+		cmdline_printf(cl, "%-16s%u\n", "Total length:",
+			info.total_len);
+		cmdline_printf(cl, "%-16s%u\n", "Payload offset:",
+			info.payload_offset);
+		cmdline_printf(cl, "%-16s%u\n", "Payload length:",
+			info.payload_len);
+	} else {
+		cmdline_printf(cl, "Invalid image file\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_image_info_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_image_info_result, cmd,
+		"get_image_info");
+cmdline_parse_token_string_t cmd_image_info_path =
+	TOKEN_STRING_INITIALIZER(struct cmd_image_info_result, path, NULL);
+
+cmdline_parse_inst_t cmd_get_image_info = {
+	.f = cmd_image_info_parsed,
+	.data = NULL,
+	.help_str = "get information of image file",
+	.tokens = {
+		(void *)&cmd_image_info_cmd,
+		(void *)&cmd_image_info_path,
+		NULL,
+	},
+};
+
+/* *** GET STATUS *** */
+struct cmd_get_status_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_get_status_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_get_status_result *res = parsed_result;
+	opae_pci_device id;
+	uint32_t stat, prog;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_load_rsu_status(&id, &stat, &prog) == 0) {
+		cmdline_printf(cl, "%-10s", "Status:");
+		if (stat == 0)
+			cmdline_printf(cl, "IDLE\n");
+		else if (stat == 1)
+			cmdline_printf(cl, "PREPARE\n");
+		else if (stat == 2)
+			cmdline_printf(cl, "PROGRAM\n");
+		else if (stat == 3)
+			cmdline_printf(cl, "COPY\n");
+		else if (stat == 4)
+			cmdline_printf(cl, "REBOOT\n");
+		else
+			cmdline_printf(cl, "unknown\n");
+		cmdline_printf(cl, "%-10s%u%%\n", "Progress:", prog);
+	} else {
+		cmdline_printf(cl, "Failed\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_get_status_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_get_status_result, cmd, "get_status");
+cmdline_parse_token_string_t cmd_get_status_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_get_status_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_get_status = {
+	.f = cmd_get_status_parsed,
+	.data = NULL,
+	.help_str = "get current status & progress of FPGA",
+	.tokens = {
+		(void *)&cmd_get_status_cmd,
+		(void *)&cmd_get_status_bdf,
+		NULL,
+	},
+};
+
+/* *** GET PROPERTY *** */
+struct cmd_property_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	int32_t type;
+};
+
+static void cmd_property_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_property_result *res = parsed_result;
+	opae_pci_device id;
+	opae_fpga_property prop;
+	uuid_str str;
+	uint32_t port = 0;
+
+	switch (res->type) {
+	case 0:
+	case 1:
+	case 2:
+	case 4:
+	case 8:
+		break;
+	default:
+		cmdline_printf(cl, "%d is invalid type of property\n",
+			res->type);
+		return;
+	}
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_get_property(&id, &prop, res->type) == 0) {
+		if ((res->type == 0) || (res->type == 1)) {
+			cmdline_printf(cl, "%s:\n", "PCI");
+			cmdline_printf(cl, " %-16s : %s\n",
+				"PCIe s:b:d.f", prop.pci.pci_addr);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"kernel driver", prop.pci.drv_name);
+		}
+		if ((res->type == 0) || (res->type == 2)) {
+			cmdline_printf(cl, "%s:\n", "FME");
+			cmdline_printf(cl, " %-16s : %s\n",
+				"platform", prop.fme.platform_name);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"DCP version", prop.fme.dcp_version);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"phase", prop.fme.release_name);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"interface", prop.fme.interface_type);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"build version", prop.fme.build_version);
+			cmdline_printf(cl, " %-16s : %u\n",
+				"ports num", prop.fme.num_ports);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"boot page", prop.fme.boot_page ? "user" : "factory");
+			uuid_to_str(&prop.fme.pr_id, &str);
+			cmdline_printf(cl, " %-16s : %s\n", "pr interface id",
+				str.s);
+		}
+		if ((res->type == 0) || (res->type == 4)) {
+			for (port = 0; port < prop.fme.num_ports; port++) {
+				cmdline_printf(cl, "%s%d:\n", "PORT", port);
+				cmdline_printf(cl, " %-16s : %s\n",
+					"access type",
+					prop.port[port].type ? "VF" : "PF");
+				uuid_to_str(&prop.port[port].afu_id, &str);
+				cmdline_printf(cl, " %-16s : %s\n",
+					"accelerator id", str.s);
+			}
+		}
+		if ((res->type == 0) || (res->type == 8)) {
+			cmdline_printf(cl, "%s:\n", "BMC");
+			cmdline_printf(cl, " %-16s : %s\n",
+				"MAX10 version", prop.bmc.bmc_version);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"NIOS FW version", prop.bmc.fw_version);
+		}
+	} else {
+		cmdline_printf(cl, "Failed\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_property_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_property_result, cmd, "get_property");
+cmdline_parse_token_string_t cmd_property_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_property_result, bdf, NULL);
+cmdline_parse_token_num_t cmd_property_type =
+	TOKEN_NUM_INITIALIZER(struct cmd_property_result, type, RTE_INT32);
+
+cmdline_parse_inst_t cmd_get_property = {
+	.f = cmd_property_parsed,
+	.data = NULL,
+	.help_str = "get property of FPGA",
+	.tokens = {
+		(void *)&cmd_property_cmd,
+		(void *)&cmd_property_bdf,
+		(void *)&cmd_property_type,
+		NULL,
+	},
+};
+
+/* *** GET PHY INFO *** */
+struct cmd_phy_info_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_phy_info_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_phy_info_result *res = parsed_result;
+	opae_pci_device id;
+	opae_phy_info info;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_get_phy_info(&id, &info) == 0) {
+		cmdline_printf(cl, " %-16s : %u\n",
+			"retimers num", info.num_retimers);
+		cmdline_printf(cl, " %-16s : %uG\n",
+			"link speed", info.link_speed);
+		cmdline_printf(cl, " %-16s : %02xh\n",
+			"link status", info.link_status);
+	} else {
+		cmdline_printf(cl, "Failed\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_phy_info_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_phy_info_result, cmd, "get_phy_info");
+cmdline_parse_token_string_t cmd_phy_info_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_phy_info_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_phy_info = {
+	.f = cmd_phy_info_parsed,
+	.data = NULL,
+	.help_str = "get information of PHY",
+	.tokens = {
+		(void *)&cmd_phy_info_cmd,
+		(void *)&cmd_phy_info_bdf,
+		NULL,
+	},
+};
+
+/* *** GET PARENT *** */
+struct cmd_parent_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_parent_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_parent_result *res = parsed_result;
+	opae_pci_device id;
+	opae_pci_device parent;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_get_parent(&id, &parent) > 0)
+		cmdline_printf(cl, "%s\n", parent.bdf);
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_parent_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_parent_result, cmd, "get_parent");
+cmdline_parse_token_string_t cmd_parent_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_parent_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_get_parent = {
+	.f = cmd_parent_parsed,
+	.data = NULL,
+	.help_str = "get parent PCI device of FPGA",
+	.tokens = {
+		(void *)&cmd_parent_cmd,
+		(void *)&cmd_parent_bdf,
+		NULL,
+	},
+};
+
+/* *** GET CHILD *** */
+struct cmd_child_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_child_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_child_result *res = parsed_result;
+	opae_pci_device id;
+	pcidev_id child;
+	int i, count = 0;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	count = opae_get_child(&id, NULL, 0);
+	if (count > 0) {
+		child = (pcidev_id)malloc(sizeof(opae_pci_device) * count);
+		if (child) {
+			opae_get_child(&id, child, count);
+			for (i = 0; i < count; i++)
+				cmdline_printf(cl, "%s\n", child[i].bdf);
+			free(child);
+		} else {
+			cmdline_printf(cl, "No memory\n");
+		}
+	} else if (count == 0) {
+		cmdline_printf(cl, "No child\n");
+	} else {
+		cmdline_printf(cl, "Failed\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_child_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_child_result, cmd, "get_child");
+cmdline_parse_token_string_t cmd_child_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_child_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_get_child = {
+	.f = cmd_child_parsed,
+	.data = NULL,
+	.help_str = "get child PCI device of FPGA",
+	.tokens = {
+		(void *)&cmd_child_cmd,
+		(void *)&cmd_child_bdf,
+		NULL,
+	},
+};
+
+/* *** GET PF1 *** */
+struct cmd_pf1_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_pf1_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_pf1_result *res = parsed_result;
+	opae_pci_device id;
+	pcidev_id peer;
+	int i, count = 0;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	count = opae_get_pf1(&id, NULL, 0);
+	if (count > 0) {
+		peer = (pcidev_id)malloc(sizeof(opae_pci_device) * count);
+		if (peer) {
+			opae_get_pf1(&id, peer, count);
+			for (i = 0; i < count; i++)
+				cmdline_printf(cl, "%s\n", peer[i].bdf);
+			free(peer);
+		} else {
+			cmdline_printf(cl, "No memory\n");
+		}
+	} else if (count == 0) {
+		cmdline_printf(cl, "No PF1\n");
+	} else {
+		cmdline_printf(cl, "Failed\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_pf1_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_pf1_result, cmd, "get_pf1");
+cmdline_parse_token_string_t cmd_pf1_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_pf1_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_get_pf1 = {
+	.f = cmd_pf1_parsed,
+	.data = NULL,
+	.help_str = "get physical function 1 device of FPGA",
+	.tokens = {
+		(void *)&cmd_pf1_cmd,
+		(void *)&cmd_pf1_bdf,
+		NULL,
+	},
+};
+
+/* *** SET LOG LEVEL *** */
+struct cmd_log_level_result {
+	cmdline_fixed_string_t cmd;
+	int32_t level;
+};
+
+static void cmd_log_level_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_log_level_result *res = parsed_result;
+	if (opae_set_log_level(res->level) == res->level)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_log_level_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_log_level_result, cmd, "set_log_level");
+cmdline_parse_token_num_t cmd_log_level_level =
+	TOKEN_NUM_INITIALIZER(struct cmd_log_level_result, level, RTE_INT32);
+
+cmdline_parse_inst_t cmd_set_log_level = {
+	.f = cmd_log_level_parsed,
+	.data = NULL,
+	.help_str = "set logging level",
+	.tokens = {
+		(void *)&cmd_log_level_cmd,
+		(void *)&cmd_log_level_level,
+		NULL,
+	},
+};
+
+/* *** SET LOG FILE *** */
+struct cmd_log_file_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t path;
+};
+
+static void cmd_log_file_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_log_file_result *res = parsed_result;
+	if (opae_set_log_file(res->path, 1) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_log_file_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_log_file_result, cmd, "set_log_file");
+cmdline_parse_token_string_t cmd_log_file_path =
+	TOKEN_STRING_INITIALIZER(struct cmd_log_file_result, path, NULL);
+
+cmdline_parse_inst_t cmd_set_log_file = {
+	.f = cmd_log_file_parsed,
+	.data = NULL,
+	.help_str = "set logging file",
+	.tokens = {
+		(void *)&cmd_log_file_cmd,
+		(void *)&cmd_log_file_path,
+		NULL,
+	},
+};
+
+/* *** SET STATUS *** */
+struct cmd_set_status_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	uint32_t stat;
+	uint32_t prog;
+};
+
+static void cmd_set_status_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_set_status_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+	if ((res->stat > 4) || (res->prog > 100)) {
+		cmdline_printf(cl, "%u,%u is invalid status\n", res->stat,
+			res->prog);
+		return;
+	}
+
+	if (opae_store_rsu_status(&id, res->stat, res->prog) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_set_status_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_set_status_result, cmd, "set_status");
+cmdline_parse_token_string_t cmd_set_status_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_set_status_result, bdf, NULL);
+cmdline_parse_token_num_t cmd_set_status_stat =
+	TOKEN_NUM_INITIALIZER(struct cmd_set_status_result, stat, RTE_UINT32);
+cmdline_parse_token_num_t cmd_set_status_prog =
+	TOKEN_NUM_INITIALIZER(struct cmd_set_status_result, prog, RTE_UINT32);
+
+cmdline_parse_inst_t cmd_set_status = {
+	.f = cmd_set_status_parsed,
+	.data = NULL,
+	.help_str = "set current status & progress of FPGA",
+	.tokens = {
+		(void *)&cmd_set_status_cmd,
+		(void *)&cmd_set_status_bdf,
+		(void *)&cmd_set_status_stat,
+		(void *)&cmd_set_status_prog,
+		NULL,
+	},
+};
+
+/* *** ENUMERATE *** */
+struct cmd_enumerate_result {
+	cmdline_fixed_string_t cmd;
+	uint32_t vid;
+	uint32_t did;
+};
+
+static void cmd_enumerate_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_enumerate_result *res = parsed_result;
+	opae_pci_id filter;
+	opae_pci_device *id;
+	int i, count = 0;
+
+	filter.vendor_id = res->vid;
+	filter.device_id = res->did;
+	filter.class_id = BIT_SET_32;
+	filter.subsystem_vendor_id = BIT_SET_16;
+	filter.subsystem_device_id = BIT_SET_16;
+
+	count = opae_enumerate(&filter, NULL, 0);
+	if (count > 0) {
+		id = (opae_pci_device *)malloc(sizeof(opae_pci_device) * count);
+		if (id) {
+			opae_enumerate(&filter, id, count);
+			for (i = 0; i < count; i++)
+				cmdline_printf(cl, "%s\n", id[i].bdf);
+			free(id);
+		} else {
+			cmdline_printf(cl, "No memory\n");
+		}
+	} else if (count == 0) {
+		cmdline_printf(cl, "Not found\n");
+	} else {
+		cmdline_printf(cl, "Failed\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_enumerate_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_enumerate_result, cmd, "enumerate");
+cmdline_parse_token_num_t cmd_enumerate_vid =
+	TOKEN_NUM_INITIALIZER(struct cmd_enumerate_result, vid, RTE_UINT32);
+cmdline_parse_token_num_t cmd_enumerate_did =
+	TOKEN_NUM_INITIALIZER(struct cmd_enumerate_result, did, RTE_UINT32);
+
+cmdline_parse_inst_t cmd_enumerate = {
+	.f = cmd_enumerate_parsed,
+	.data = NULL,
+	.help_str = "enumerate specified FPGA",
+	.tokens = {
+		(void *)&cmd_enumerate_cmd,
+		(void *)&cmd_enumerate_vid,
+		(void *)&cmd_enumerate_did,
+		NULL,
+	},
+};
+
+/* *** BIND *** */
+struct cmd_bind_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	cmdline_fixed_string_t drv;
+};
+
+static void cmd_bind_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_bind_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_bind_driver(&id, res->drv) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_bind_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_bind_result, cmd, "bind");
+cmdline_parse_token_string_t cmd_bind_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_bind_result, bdf, NULL);
+cmdline_parse_token_string_t cmd_bind_drv =
+	TOKEN_STRING_INITIALIZER(struct cmd_bind_result, drv, NULL);
+
+cmdline_parse_inst_t cmd_bind = {
+	.f = cmd_bind_parsed,
+	.data = NULL,
+	.help_str = "bind FPGA with kernel driver",
+	.tokens = {
+		(void *)&cmd_bind_cmd,
+		(void *)&cmd_bind_bdf,
+		(void *)&cmd_bind_drv,
+		NULL,
+	},
+};
+
+/* *** UNBIND *** */
+struct cmd_unbind_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_unbind_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_unbind_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_unbind_driver(&id) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_unbind_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_bind_result, cmd, "unbind");
+cmdline_parse_token_string_t cmd_unbind_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_bind_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_unbind = {
+	.f = cmd_unbind_parsed,
+	.data = NULL,
+	.help_str = "unbind FPGA from kernel driver",
+	.tokens = {
+		(void *)&cmd_unbind_cmd,
+		(void *)&cmd_unbind_bdf,
+		NULL,
+	},
+};
+
+/* *** PROBE *** */
+struct cmd_probe_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_probe_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_probe_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_probe_device(&id) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_probe_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_probe_result, cmd, "probe");
+cmdline_parse_token_string_t cmd_probe_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_probe_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_probe = {
+	.f = cmd_probe_parsed,
+	.data = NULL,
+	.help_str = "probe FPGA with IFPGA driver",
+	.tokens = {
+		(void *)&cmd_probe_cmd,
+		(void *)&cmd_probe_bdf,
+		NULL,
+	},
+};
+
+/* *** REMOVE *** */
+struct cmd_remove_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_remove_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_remove_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_remove_device(&id) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_remove_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_remove_result, cmd, "remove");
+cmdline_parse_token_string_t cmd_remove_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_remove_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_remove = {
+	.f = cmd_remove_parsed,
+	.data = NULL,
+	.help_str = "remove FPGA from IFPGA driver",
+	.tokens = {
+		(void *)&cmd_remove_cmd,
+		(void *)&cmd_remove_bdf,
+		NULL,
+	},
+};
+
+/* *** FLASH *** */
+struct cmd_flash_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	cmdline_fixed_string_t path;
+};
+
+static void cmd_flash_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_flash_result *res = parsed_result;
+	opae_pci_device id;
+	uint64_t stat = 0;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_update_flash(&id, res->path, &stat))
+		cmdline_printf(cl, "Error: 0x%lx\n", stat);
+}
+
+cmdline_parse_token_string_t cmd_flash_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_flash_result, cmd, "flash");
+cmdline_parse_token_string_t cmd_flash_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_flash_result, bdf, NULL);
+cmdline_parse_token_string_t cmd_flash_path =
+	TOKEN_STRING_INITIALIZER(struct cmd_flash_result, path, NULL);
+
+cmdline_parse_inst_t cmd_flash = {
+	.f = cmd_flash_parsed,
+	.data = NULL,
+	.help_str = "update flash of FPGA",
+	.tokens = {
+		(void *)&cmd_flash_cmd,
+		(void *)&cmd_flash_bdf,
+		(void *)&cmd_flash_path,
+		NULL,
+	},
+};
+
+/* *** PR *** */
+struct cmd_pr_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	int32_t port;
+	cmdline_fixed_string_t path;
+};
+
+static void cmd_pr_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_pr_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_partial_reconfigure(&id, res->port, res->path) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_pr_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_pr_result, cmd, "pr");
+cmdline_parse_token_string_t cmd_pr_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_pr_result, bdf, NULL);
+cmdline_parse_token_num_t cmd_pr_port =
+	TOKEN_NUM_INITIALIZER(struct cmd_pr_result, port, RTE_INT32);
+cmdline_parse_token_string_t cmd_pr_path =
+	TOKEN_STRING_INITIALIZER(struct cmd_pr_result, path, NULL);
+
+cmdline_parse_inst_t cmd_pr = {
+	.f = cmd_pr_parsed,
+	.data = NULL,
+	.help_str = "partial reconfigure FPGA",
+	.tokens = {
+		(void *)&cmd_pr_cmd,
+		(void *)&cmd_pr_bdf,
+		(void *)&cmd_pr_port,
+		(void *)&cmd_pr_path,
+		NULL,
+	},
+};
+
+/* *** REBOOT *** */
+struct cmd_reboot_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	cmdline_fixed_string_t type;
+	int32_t page;
+};
+
+static void cmd_reboot_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_reboot_result *res = parsed_result;
+	opae_pci_device id;
+	int type = 0;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (!strcmp(res->type, "fpga")) {
+		type = 0;
+	} else if (!strcmp(res->type, "bmc")) {
+		type = 1;
+	} else {
+		cmdline_printf(cl, "%s is invalid reboot type\n", res->type);
+		return;
+	}
+
+	if (opae_reboot_device(&id, type, res->page) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_reboot_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_reboot_result, cmd, "reboot");
+cmdline_parse_token_string_t cmd_reboot_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_reboot_result, bdf, NULL);
+cmdline_parse_token_string_t cmd_reboot_type =
+	TOKEN_STRING_INITIALIZER(struct cmd_reboot_result, type, NULL);
+cmdline_parse_token_num_t cmd_reboot_page =
+	TOKEN_NUM_INITIALIZER(struct cmd_reboot_result, page, RTE_INT32);
+
+cmdline_parse_inst_t cmd_reboot = {
+	.f = cmd_reboot_parsed,
+	.data = NULL,
+	.help_str = "reboot FPGA or MAX10",
+	.tokens = {
+		(void *)&cmd_reboot_cmd,
+		(void *)&cmd_reboot_bdf,
+		(void *)&cmd_reboot_type,
+		(void *)&cmd_reboot_page,
+		NULL,
+	},
+};
+
+/* *** CANCEL *** */
+struct cmd_cancel_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_cancel_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_cancel_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_cancel_flash_update(&id, 0) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_cancel_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_cancel_result, cmd, "cancel");
+cmdline_parse_token_string_t cmd_cancel_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_cancel_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_cancel = {
+	.f = cmd_cancel_parsed,
+	.data = NULL,
+	.help_str = "cancel flash update",
+	.tokens = {
+		(void *)&cmd_cancel_cmd,
+		(void *)&cmd_cancel_bdf,
+		NULL,
+	},
+};
+
+/* *** PCI READ *** */
+struct cmd_pci_read_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	uint32_t offset;
+};
+
+static void cmd_pci_read_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_pci_read_result *res = parsed_result;
+	opae_pci_device id;
+	uint32_t offset = 0;
+	uint32_t value = 0;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (res->offset & 0x3) {
+		offset = res->offset & ~3;
+		cmdline_printf(cl, "align offset to 0x%x\n", offset);
+	} else {
+		offset = res->offset;
+	}
+
+	if (opae_read_pci_cfg(&id, offset, &value) == 0)
+		cmdline_printf(cl, "0x%08x\n", value);
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_pci_read_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_pci_read_result, cmd, "pci_read");
+cmdline_parse_token_string_t cmd_pci_read_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_pci_read_result, bdf, NULL);
+cmdline_parse_token_num_t cmd_pci_read_offset =
+	TOKEN_NUM_INITIALIZER(struct cmd_pci_read_result, offset, RTE_UINT32);
+
+cmdline_parse_inst_t cmd_pci_read = {
+	.f = cmd_pci_read_parsed,
+	.data = NULL,
+	.help_str = "read PCI configuration space",
+	.tokens = {
+		(void *)&cmd_pci_read_cmd,
+		(void *)&cmd_pci_read_bdf,
+		(void *)&cmd_pci_read_offset,
+		NULL,
+	},
+};
+
+/* *** PCI WRITE *** */
+struct cmd_pci_write_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	uint32_t offset;
+	uint32_t value;
+};
+
+static void cmd_pci_write_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_pci_write_result *res = parsed_result;
+	opae_pci_device id;
+	uint32_t offset = 0;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (res->offset & 0x3) {
+		offset = res->offset & ~3;
+		cmdline_printf(cl, "align offset to 0x%x\n", offset);
+	} else {
+		offset = res->offset;
+	}
+
+	if (opae_write_pci_cfg(&id, offset, res->value) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_pci_write_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_pci_write_result, cmd, "pci_write");
+cmdline_parse_token_string_t cmd_pci_write_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_pci_write_result, bdf, NULL);
+cmdline_parse_token_num_t cmd_pci_write_offset =
+	TOKEN_NUM_INITIALIZER(struct cmd_pci_write_result, offset, RTE_UINT32);
+cmdline_parse_token_num_t cmd_pci_write_value =
+	TOKEN_NUM_INITIALIZER(struct cmd_pci_write_result, value, RTE_UINT32);
+
+cmdline_parse_inst_t cmd_pci_write = {
+	.f = cmd_pci_write_parsed,
+	.data = NULL,
+	.help_str = "write PCI configuration space",
+	.tokens = {
+		(void *)&cmd_pci_write_cmd,
+		(void *)&cmd_pci_write_bdf,
+		(void *)&cmd_pci_write_offset,
+		(void *)&cmd_pci_write_value,
+		NULL,
+	},
+};
+
+/* *** QUIT *** */
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__rte_unused void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,
+	.data = NULL,
+	.help_str = "exit DPDK application",
+	.tokens = {
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/* *** HELP *** */
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__rte_unused void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	cmdline_printf(cl,
+		" get_api_version               \t\t"
+			"get OPAE API version\n"
+		" get_proc_type                 \t\t"
+			"get DPDK process type\n"
+		" get_image_info <FILE>         \t\t"
+			"get information of image file\n"
+		" get_status <BDF>              \t\t"
+			"get current status & progress of FPGA\n"
+		" get_property <BDF> <0|1|2|4|8>\t\t"
+			"get property of FPGA\n"
+		" get_phy_info <BDF>            \t\t"
+			"get information of PHY\n"
+		" get_parent <BDF>              \t\t"
+			"get parent PCI device of FPGA\n"
+		" get_child <BDF>               \t\t"
+			"get child PCI device of FPGA\n"
+		" get_pf1 <BDF>                 \t\t"
+			"get physical function 1 device of FPGA\n"
+		" set_log_level <0-4>           \t\t"
+			"set logging level\n"
+		" set_log_file <FILE>           \t\t"
+			"set logging file\n"
+		" set_status <BDF> <0-4> <0-100>\t\t"
+			"set current status & progress of FPGA\n"
+		" enumerate <VID> <DID>         \t\t"
+			"enumerate specified FPGA\n"
+		" bind <BDF> <DRIVER>           \t\t"
+			"bind FPGA with kernel driver\n"
+		" unbind <BDF>                  \t\t"
+			"unbind FPGA from kernel driver\n"
+		" probe <BDF>                   \t\t"
+			"probe FPGA with IFPGA driver\n"
+		" remove <BDF>                  \t\t"
+			"remove FPGA from IFPGA driver\n"
+		" flash <BDF> <FILE>            \t\t"
+			"update flash of FPGA\n"
+		" pr <BDF> <PORT> <FILE>        \t\t"
+			"partial reconfigure FPGA\n"
+		" reboot <BDF> <fpga|bmc> <0-1> \t\t"
+			"reboot FPGA or MAX10\n"
+		" cancel <BDF>                  \t\t"
+			"cancel flash update\n"
+		" pci_read <BDF> <0-1024>       \t\t"
+			"read PCI configuration space\n"
+		" pci_write <BDF> <0-1024> <NUM>\t\t"
+			"write PCI configuration space\n"
+		" quit                          \t\t"
+			"exit DPDK application\n"
+		" help                          \t\t"
+			"show commands list\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,
+	.data = NULL,
+	.help_str = "show commands list",
+	.tokens = {
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/****** CONTEXT (list of commands) */
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_get_image_info,
+	(cmdline_parse_inst_t *)&cmd_get_api_version,
+	(cmdline_parse_inst_t *)&cmd_get_proc_type,
+	(cmdline_parse_inst_t *)&cmd_get_status,
+	(cmdline_parse_inst_t *)&cmd_get_property,
+	(cmdline_parse_inst_t *)&cmd_phy_info,
+	(cmdline_parse_inst_t *)&cmd_get_parent,
+	(cmdline_parse_inst_t *)&cmd_get_child,
+	(cmdline_parse_inst_t *)&cmd_get_pf1,
+	(cmdline_parse_inst_t *)&cmd_set_log_level,
+	(cmdline_parse_inst_t *)&cmd_set_log_file,
+	(cmdline_parse_inst_t *)&cmd_set_status,
+	(cmdline_parse_inst_t *)&cmd_enumerate,
+	(cmdline_parse_inst_t *)&cmd_bind,
+	(cmdline_parse_inst_t *)&cmd_unbind,
+	(cmdline_parse_inst_t *)&cmd_probe,
+	(cmdline_parse_inst_t *)&cmd_remove,
+	(cmdline_parse_inst_t *)&cmd_flash,
+	(cmdline_parse_inst_t *)&cmd_pr,
+	(cmdline_parse_inst_t *)&cmd_reboot,
+	(cmdline_parse_inst_t *)&cmd_cancel,
+	(cmdline_parse_inst_t *)&cmd_pci_read,
+	(cmdline_parse_inst_t *)&cmd_pci_write,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_help,
+	NULL,
+};
diff --git a/examples/ifpga/commands.h b/examples/ifpga/commands.h
new file mode 100644
index 0000000000..06fe9a68b2
--- /dev/null
+++ b/examples/ifpga/commands.h
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020-2021 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+#include "opae_api.h"
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+typedef struct {
+	char s[38];
+} uuid_str;
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/ifpga/main.c b/examples/ifpga/main.c
new file mode 100644
index 0000000000..e9380d581a
--- /dev/null
+++ b/examples/ifpga/main.c
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020-2021 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+
+int main(int argc, char **argv)
+{
+	struct cmdline *cl;
+	int ret;
+
+	ret = opae_init_eal(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+	cl = cmdline_stdin_new(main_ctx, "opae> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	opae_cleanup_eal();
+	cmdline_stdin_exit(cl);
+	return 0;
+}
diff --git a/examples/ifpga/meson.build b/examples/ifpga/meson.build
new file mode 100644
index 0000000000..b88a37a7a5
--- /dev/null
+++ b/examples/ifpga/meson.build
@@ -0,0 +1,20 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020-2021 Intel Corporation
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+# require the raw_ifpga library
+build = dpdk_conf.has('RTE_RAW_IFPGA')
+if not build
+	subdir_done()
+endif
+
+deps += 'raw_ifpga'
+allow_experimental_apis = true
+sources = files(
+	'main.c', 'commands.c', 'opae_api.c'
+)
+cflags += '-fPIC'
diff --git a/examples/ifpga/opae_api.c b/examples/ifpga/opae_api.c
new file mode 100644
index 0000000000..888734f7f6
--- /dev/null
+++ b/examples/ifpga/opae_api.c
@@ -0,0 +1,1632 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <stdlib.h>
+#include <string.h>
+#include <dirent.h>
+#include <fcntl.h>
+#include <glob.h>
+#include <unistd.h>
+#include <sys/stat.h>
+#include <rte_eal.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+#include <rte_rawdev_pmd.h>
+#include "rte_pmd_ifpga.h"
+#include "opae_api.h"
+
+
+int opae_log_level;
+FILE *opae_log_file;
+
+static opae_api_version api_ver = {21, 5, 0};
+static int eal_inited;
+static uint32_t dev_aer[2] = {0};
+
+static const char * const log_level_name[] = {"CRITICAL", "ERROR",
+	"WARNING", "INFORMATION", "DEBUG"};
+static const char * const proc_type_name[] = {"NON-DPDK", "PRIMARY",
+	"SECONDARY"};
+static const char * const platform_name[] = {"Vista Creek", "Rush Creek",
+	"Darby Creek", "Lightning Creek"};
+static const char * const release_name[] = {"Pre-Alpha", "Alpha", "Beta", "PV"};
+static const char * const interface_type[] = {"8x10G", "4x25G", "2x1x25G",
+	"4x25G+2x25G", "2x2x25G", "2x1x25Gx2FVL", "1x2x25G"};
+static const char * const kdrv[] = {OPAE_KDRV_UNKNOWN, OPAE_KDRV_IGB_UIO,
+	OPAE_KDRV_VFIO_PCI, OPAE_KDRV_UIO_PCI};
+
+void opae_get_api_version(opae_api_version *version)
+{
+	if (version)
+		memcpy(version, &api_ver, sizeof(opae_api_version));
+	opae_log_info("API version is %u.%u.%u\n",
+		api_ver.major, api_ver.minor, api_ver.micro);
+}
+
+int opae_set_log_level(int level)
+{
+	if ((level >= OPAE_LOG_API) && (level <= OPAE_LOG_DEBUG))
+		opae_log_level = level;
+	opae_log_api("Current log level is %s\n",
+		log_level_name[opae_log_level]);
+	return opae_log_level;
+}
+
+int opae_set_log_file(char *path, int clean)
+{
+	FILE *f = NULL;
+	time_t start;
+	struct tm *lt = NULL;
+
+	if (path) {
+		if (clean)
+			f = fopen(path, "w+");
+		else
+			f = fopen(path, "a+");
+
+		if (f) {
+			if (opae_log_file) {
+				fclose(opae_log_file);
+				opae_log_file = NULL;
+			}
+			time(&start);
+			lt = localtime(&start);
+			if (lt)
+				fprintf(f, "================%d-%02d-%02d "
+					"%02d:%02d:%02d================\n",
+					1900 + lt->tm_year, 1 + lt->tm_mon,
+					lt->tm_mday,
+					lt->tm_hour, lt->tm_min, lt->tm_sec);
+			fflush(f);
+			opae_log_file = f;
+		} else {
+			opae_log_err("failed to open log file \'%s\'\n", path);
+			return -1;
+		}
+	} else {
+		if (opae_log_file) {
+			fclose(opae_log_file);
+			opae_log_file = NULL;
+		}
+	}
+
+	return 0;
+}
+
+int opae_get_image_info(const char *image, opae_img_info *info)
+{
+	int fd = -1;
+	off_t file_size = 0;
+	opae_img_hdr hdr;
+	ssize_t read_size = 0;
+	int ret = 0;
+
+	if (!image || !info) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	fd = open(image, O_RDONLY);
+	if (fd < 0) {
+		opae_log_err("Failed to open \'%s\' for RD [e:%s]\n",
+			image, strerror(errno));
+		return -EIO;
+	}
+
+	file_size = lseek(fd, 0, SEEK_END);
+	if (file_size < (OPAE_IMG_HDR_SIZE + OPAE_IMG_PL_MIN_SIZE)) {
+		opae_log_err("Size of \'%s\' is less than expected [e:%u]\n",
+			image, OPAE_IMG_HDR_SIZE + OPAE_IMG_PL_MIN_SIZE);
+		ret = -EINVAL;
+		goto close_fd;
+	}
+
+	/* read image header */
+	lseek(fd, 0, SEEK_SET);
+	read_size = read(fd, (void *)&hdr, sizeof(opae_img_hdr));
+	if (read_size < 0) {
+		opae_log_err("Failed to read from \'%s\' [e:%s]\n",
+			image, strerror(errno));
+		ret = -EIO;
+		goto close_fd;
+	}
+	if ((size_t)read_size != sizeof(opae_img_hdr)) {
+		opae_log_err("Read length %zd is not expected [e:%zu]\n",
+			read_size, sizeof(opae_img_hdr));
+		ret = -EIO;
+		goto close_fd;
+	}
+
+	info->total_len = file_size;
+	/* check signed image header */
+	if (hdr.magic == OPAE_IMG_BLK0_MAGIC) {
+		info->type = OPAE_IMG_TYPE(hdr.payload_type);
+		info->subtype = OPAE_IMG_SUBTYPE(hdr.payload_type);
+		info->payload_offset = OPAE_IMG_HDR_SIZE;
+		info->payload_len = hdr.payload_len;
+	} else {
+		opae_log_err("Image \'%s\' can not be recognized\n", image);
+		ret = -EINVAL;
+	}
+close_fd:
+	close(fd);
+	return ret;
+}
+
+static int write_file(char *path, char *buf, int size)
+{
+	int fd = -1;
+	ssize_t n = 0;
+
+	if (!path || !buf || (size <= 0))
+		return -EINVAL;
+
+	fd = open(path, O_WRONLY);
+	if (fd < 0) {
+		opae_log_err("Failed to open \'%s\' for WR [e:%s]\n",
+			path, strerror(errno));
+		return -EIO;
+	}
+	opae_log_dbg("Write \"%s\" to \'%s\'\n", buf, path);
+	n = write(fd, buf, size);
+	if (n < size)  {
+		opae_log_err("Failed to write to \'%s\' [e:%s]\n",
+			path, strerror(errno));
+		close(fd);
+		return -EIO;
+	}
+	close(fd);
+
+	return 0;
+}
+
+static int read_file(char *path, char *buf, int size)
+{
+	int fd = -1;
+	ssize_t n = 0;
+
+	if (!path || !buf || (size <= 0))
+		return -EINVAL;
+
+	fd = open(path, O_RDONLY);
+	if (fd < 0) {
+		opae_log_err("Failed to open \'%s\' for RD [e:%s]\n",
+			path, strerror(errno));
+		return -EIO;
+	}
+	n = read(fd, buf, size);
+	if (n < 0)  {
+		opae_log_err("Failed to read from \'%s\' [e:%s]\n",
+			path, strerror(errno));
+		close(fd);
+		return -EIO;
+	}
+	close(fd);
+
+	if (n > 0)
+		buf[n-1] = 0;
+
+	opae_log_dbg("Read \"%s\" from \'%s\'\n", buf, path);
+	return 0;
+}
+
+int opae_get_proc_type(void)
+{
+	int type = -1;
+
+	if (eal_inited) {
+		if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+			type = 0;
+		else
+			type = 1;
+	}
+	opae_log_info("Current process type is %s\n", proc_type_name[type+1]);
+
+	return type;
+}
+
+static bool check_eal(int inited)
+{
+	if (!eal_inited) {
+		if (inited) {
+			opae_log_warn("EAL is not initialized\n");
+			return 0;
+		}
+	} else {
+		if (!inited) {
+			opae_log_warn("EAL is already initialized\n");
+			return 0;
+		}
+	}
+
+	return 1;
+}
+
+int opae_init_eal(int argc, char **argv)
+{
+	int ret = 0;
+
+	if (!check_eal(0))
+		return ret;
+
+	opae_log_level = OPAE_LOG_ERR;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0) {
+		if (rte_errno == EALREADY) {
+			eal_inited = 1;
+			return 0;
+		}
+		opae_log_err("Cannot initialize EAL [e:%d]\n", ret);
+		if (rte_eal_cleanup())
+			opae_log_warn("EAL could not release all resources\n");
+	} else {
+		eal_inited = 1;
+		opae_log_info("Initialize EAL done\n");
+	}
+
+	return ret;
+}
+
+int opae_cleanup_eal(void)
+{
+	int ret = 0;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	rte_pmd_ifpga_cleanup();
+
+	ret = rte_eal_cleanup();
+	if (ret)
+		opae_log_err("Failed to cleanup EAL [e:%d]\n", ret);
+
+	if (opae_log_file) {
+		fclose(opae_log_file);
+		opae_log_file = NULL;
+	}
+
+	return ret;
+}
+
+static int compare_pci_id(opae_pci_id *id, opae_pci_id *expected_id)
+{
+	if ((expected_id->class_id != BIT_SET_32) &&
+		(expected_id->class_id != id->class_id))
+		return -1;
+	if ((expected_id->vendor_id != BIT_SET_16) &&
+		(expected_id->vendor_id != id->vendor_id))
+		return -1;
+	if ((expected_id->device_id != BIT_SET_16) &&
+		(expected_id->device_id != id->device_id))
+		return -1;
+	if ((expected_id->subsystem_vendor_id != BIT_SET_16) &&
+		(expected_id->subsystem_vendor_id != id->subsystem_vendor_id))
+		return -1;
+	if ((expected_id->subsystem_device_id != BIT_SET_16) &&
+		(expected_id->subsystem_device_id != id->subsystem_device_id))
+		return -1;
+
+	return 0;
+}
+
+static int parse_sysfs_value(char *node, uint32_t *val)
+{
+	char buf[16];
+	char *end = NULL;
+	int ret = 0;
+
+	ret = read_file(node, buf, sizeof(buf));
+	if (ret < 0)
+		return ret;
+
+	*val = (uint32_t)strtoul(buf, &end, 0);
+	return 0;
+}
+
+static int get_pci_id(const char *dev_path, opae_pci_id *id)
+{
+	char path[PATH_MAX] = {0};
+	uint32_t tmp;
+
+	if (!dev_path || !id)
+		return -EINVAL;
+
+	snprintf(path, sizeof(path), "%s/vendor", dev_path);
+	if (parse_sysfs_value(path, &tmp) < 0)
+		return -ENODEV;
+	id->vendor_id = (uint16_t)tmp;
+
+	snprintf(path, sizeof(path), "%s/device", dev_path);
+	if (parse_sysfs_value(path, &tmp) < 0)
+		return -ENODEV;
+	id->device_id = (uint16_t)tmp;
+
+	snprintf(path, sizeof(path), "%s/subsystem_vendor", dev_path);
+	if (parse_sysfs_value(path, &tmp) < 0)
+		return -ENODEV;
+	id->subsystem_vendor_id = (uint16_t)tmp;
+
+	snprintf(path, sizeof(path), "%s/subsystem_device", dev_path);
+	if (parse_sysfs_value(path, &tmp) < 0)
+		return -ENODEV;
+	id->subsystem_device_id = (uint16_t)tmp;
+
+	snprintf(path, sizeof(path), "%s/class", dev_path);
+	if (parse_sysfs_value(path, &tmp) < 0)
+		return -ENODEV;
+	id->class_id = (uint32_t)tmp & RTE_CLASS_ANY_ID;
+
+	return 0;
+}
+
+static int extract_path(char *in, int ridx, char *out, uint32_t size)
+{
+	char src[PATH_MAX] = {0};
+	char *p = NULL;
+	int ret = 0;
+
+	if (!in || (strlen(in) > PATH_MAX) || (ridx < 0) || !out)
+		return -EINVAL;
+
+	strncpy(src, in, sizeof(src));
+	*out = 0;
+
+	while (1) {
+		p = strrchr(src, '/');
+		if (p) {
+			*p++ = 0;
+			if (*p) {
+				if (ridx-- <= 0) {
+					if (size > strlen(p)) {
+						strncpy(out, p, size);
+						ret = strlen(p);
+					}
+					break;
+				}
+			}
+		} else {
+			break;
+		}
+	}
+
+	return ret;
+}
+
+int opae_enumerate(opae_pci_id *filter, pcidev_id list, int size)
+{
+	DIR *dir = NULL;
+	struct dirent *dirent = NULL;
+	char path[PATH_MAX] = {0};
+	opae_pci_id id;
+	int n = 0;
+
+	if (!filter || (size < 0) || (!list && (size > 0))) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	dir = opendir(rte_pci_get_sysfs_path());
+	if (!dir) {
+		opae_log_err("Failed to open \'%s\'\n",
+			rte_pci_get_sysfs_path());
+		return -EINVAL;
+	}
+	while ((dirent = readdir(dir))) {
+		if (!strcmp(dirent->d_name, "."))
+			continue;
+		if (!strcmp(dirent->d_name, ".."))
+			continue;
+
+		snprintf(path, PATH_MAX, "%s/%s", rte_pci_get_sysfs_path(),
+			dirent->d_name);
+		if (get_pci_id(path, &id) < 0)
+			continue;
+		if (compare_pci_id(&id, filter) < 0)
+			continue;
+
+		if (n++ < size) {
+			snprintf(list->bdf, sizeof(list->bdf), "%s",
+				dirent->d_name);
+			list++;
+		}
+	}
+	closedir(dir);
+
+	return n;
+}
+
+static int get_driver(pcidev_id id, char *drv_name, uint32_t size)
+{
+	char path[PATH_MAX] = {0};
+	char link[PATH_MAX] = {0};
+	int ret = 0;
+
+	if (!id || !drv_name) {
+		ret = -EINVAL;
+		goto end;
+	}
+	size--;   /* reserve one byte for the end of string */
+
+	snprintf(path, PATH_MAX, "%s/%s/driver",
+		rte_pci_get_sysfs_path(), id->bdf);
+	ret = readlink(path, link, PATH_MAX);
+	if (ret >= PATH_MAX) {
+		opae_log_err("Link path too long [%d]\n", ret);
+		ret = -ENAMETOOLONG;
+		goto end;
+	}
+	if (ret > 0) {
+		ret = extract_path(link, 0, drv_name, size);
+	} else {
+		*drv_name = 0;
+		opae_log_info("No link path for \'%s\'\n", path);
+		ret = 0;
+	}
+
+end:
+	if (ret < 0)
+		opae_log_err("Failed to get driver of %s\n", id->bdf);
+
+	return ret;
+}
+
+static int get_pci_addr(const char *bdf, opae_pci_addr *addr)
+{
+	struct rte_pci_addr pci_addr;
+	int ret = 0;
+
+	if (!bdf || !addr)
+		return -EINVAL;
+
+	ret = rte_pci_addr_parse(bdf, &pci_addr);
+	if (ret == 0) {
+		addr->domain = pci_addr.domain;
+		addr->bus = pci_addr.bus;
+		addr->devid = pci_addr.devid;
+		addr->function = pci_addr.function;
+	}
+
+	return ret;
+}
+
+static int check_pcidev_id(pcidev_id id)
+{
+	if (!id) {
+		opae_log_err("ID is NULL\n");
+		return -1;
+	}
+
+	if (strnlen(id->bdf, PCI_PRI_STR_SIZE) == PCI_PRI_STR_SIZE) {
+		opae_log_err("PCI address is too long\n");
+		return -2;
+	}
+
+	return 0;
+}
+
+static int get_rawdev_id(pcidev_id id, uint16_t *dev_id, int log)
+{
+	if (check_pcidev_id(id))
+		return -1;
+
+	if (rte_pmd_ifpga_get_dev_id(id->bdf, dev_id)) {
+		if (log)
+			opae_log_warn("%s is not probed by ifpga driver\n",
+				id->bdf);
+		return -1;
+	}
+
+	return 0;
+}
+
+static struct rte_pci_device *get_rte_pcidev(pcidev_id id, int log)
+{
+	const struct rte_pci_bus *pci_bus = NULL;
+	struct rte_pci_device *pci_dev = NULL;
+	struct rte_pci_addr addr;
+
+	if (check_pcidev_id(id))
+		return NULL;
+
+	if (rte_pci_addr_parse(id->bdf, &addr)) {
+		opae_log_err("PCI address %s is invalid\n", id->bdf);
+		return NULL;
+	}
+
+	pci_bus = rte_pmd_ifpga_get_pci_bus();
+	if (pci_bus) {
+		TAILQ_FOREACH(pci_dev, &pci_bus->device_list, next) {
+			if (!rte_pci_addr_cmp(&pci_dev->addr, &addr))
+				return pci_dev;
+		}
+	}
+
+	if (log)
+		opae_log_err("No rte_pci_device for %s\n", id->bdf);
+
+	return NULL;
+}
+
+int opae_load_rsu_status(pcidev_id id, uint32_t *status, uint32_t *progress)
+{
+	uint16_t dev_id = 0;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	return rte_pmd_ifpga_get_rsu_status(dev_id, status, progress);
+}
+
+int opae_store_rsu_status(pcidev_id id, uint32_t status, uint32_t progress)
+{
+	uint16_t dev_id = 0;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	return rte_pmd_ifpga_set_rsu_status(dev_id, status, progress);
+}
+
+static int get_pci_property(pcidev_id id, opae_pci_property *prop)
+{
+	char path[PATH_MAX] = {0};
+	int ret = 0;
+
+	if (!id || !prop)
+		return -EINVAL;
+
+	snprintf(path, PATH_MAX, "%s/%s", rte_pci_get_sysfs_path(), id->bdf);
+
+	ret = get_pci_id(path, &prop->id);
+	if (ret < 0)
+		return ret;
+
+	ret = get_pci_addr(id->bdf, &prop->addr);
+	if (ret < 0)
+		return ret;
+
+	snprintf(prop->pci_addr, OPAE_NAME_SIZE, "%s", id->bdf);
+	get_driver(id, prop->drv_name, sizeof(prop->drv_name));
+
+	return 0;
+}
+
+static int get_fme_property(rte_pmd_ifpga_common_prop *common,
+	opae_fme_property *prop)
+{
+	opae_bitstream_id bbs_id;
+
+	if (!common || !prop)
+		return -EINVAL;
+
+	prop->boot_page = common->boot_page;
+	prop->num_ports = common->num_ports;
+	prop->bitstream_id = common->bitstream_id;
+	prop->bitstream_metadata = common->bitstream_metadata;
+	memcpy(prop->pr_id.b, common->pr_id.b, sizeof(opae_uuid));
+
+	bbs_id.id = prop->bitstream_id;
+	if (bbs_id.major < sizeof(platform_name) / sizeof(char *)) {
+		snprintf(prop->platform_name,
+			sizeof(prop->platform_name), "%s",
+			platform_name[bbs_id.major]);
+	} else {
+		snprintf(prop->platform_name,
+			sizeof(prop->platform_name), "unknown");
+	}
+
+	snprintf(prop->dcp_version, sizeof(prop->dcp_version),
+		"DCP 1.%u", bbs_id.minor);
+
+	if (bbs_id.patch < sizeof(release_name)/sizeof(char *)) {
+		snprintf(prop->release_name, sizeof(prop->release_name),
+			"%s", release_name[bbs_id.patch]);
+	} else {
+		snprintf(prop->release_name, sizeof(prop->release_name),
+			"unknown");
+	}
+
+	if (bbs_id.major == 0) {  /* Vista Creek */
+		if (bbs_id.interface <
+			sizeof(interface_type) / sizeof(char *)) {
+			snprintf(prop->interface_type,
+				sizeof(prop->interface_type), "%s",
+				interface_type[bbs_id.interface]);
+		} else {
+			snprintf(prop->interface_type,
+				sizeof(prop->interface_type), "unknown");
+		}
+	} else {
+		snprintf(prop->interface_type,
+			sizeof(prop->interface_type), "unknown");
+	}
+
+	snprintf(prop->build_version, sizeof(prop->build_version),
+		"%u.%u.%u", bbs_id.build_major, bbs_id.build_minor,
+		bbs_id.build_patch);
+
+	return 0;
+}
+
+static int get_port_property(rte_pmd_ifpga_port_prop *port,
+	opae_port_property *prop)
+{
+	if (!port || !prop)
+		return -EINVAL;
+
+	memcpy(prop->afu_id.b, port->afu_id.b, sizeof(opae_uuid));
+	prop->type = port->type;
+
+	return 0;
+}
+
+static int get_bmc_property(rte_pmd_ifpga_common_prop *common,
+	opae_bmc_property *prop)
+{
+	opae_bmc_version ver;
+
+	if (!common || !prop)
+		return -EINVAL;
+
+	ver.version = common->bmc_version;
+	snprintf(prop->bmc_version, sizeof(prop->bmc_version), "%c.%u.%u.%u",
+		ver.board, ver.major, ver.minor, ver.micro);
+
+	ver.version = common->bmc_nios_version;
+	snprintf(prop->fw_version, sizeof(prop->fw_version), "%c.%u.%u.%u",
+		ver.board, ver.major, ver.minor, ver.micro);
+
+	return 0;
+}
+
+int opae_get_property(pcidev_id id, opae_fpga_property *prop, int type)
+{
+	uint16_t dev_id = 0;
+	uint32_t i = 0;
+	rte_pmd_ifpga_prop fpga_prop;
+	int ret = 0;
+
+	if (!prop) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	if (type == 0)
+		type = OPAE_PROP_ALL;
+
+	memset(prop, 0, sizeof(opae_fpga_property));
+
+	/* PCI properties */
+	if (type & OPAE_PROP_PCI) {
+		if (get_pci_property(id, &prop->pci) < 0) {
+			opae_log_err("Failed to get PCI property\n");
+			return -EAGAIN;
+		}
+	}
+
+	if (type == OPAE_PROP_PCI)
+		return 0;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	ret = rte_pmd_ifpga_get_property(dev_id, &fpga_prop);
+	if (ret) {
+		opae_log_err("Failed to get FPGA property [e:%d]\n", ret);
+		return -EAGAIN;
+	}
+
+	/* FME properties */
+	if (type & (OPAE_PROP_FME | OPAE_PROP_PORT))
+		get_fme_property(&fpga_prop.common, &prop->fme);
+
+	/* PORT properties */
+	if (type & OPAE_PROP_PORT) {
+		for (i = 0; i < prop->fme.num_ports; i++) {
+			prop->port[i].index = i;
+			get_port_property(&fpga_prop.port[i], &prop->port[i]);
+		}
+	}
+
+	/* BMC properties */
+	if (type & OPAE_PROP_BMC)
+		get_bmc_property(&fpga_prop.common, &prop->bmc);
+
+	return 0;
+}
+
+int opae_get_phy_info(pcidev_id id, opae_phy_info *info)
+{
+	uint16_t dev_id = 0;
+	rte_pmd_ifpga_phy_info phy_info;
+	int ret = 0;
+
+	if (!info)
+		return -EINVAL;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	ret = rte_pmd_ifpga_get_phy_info(dev_id, &phy_info);
+	if (ret) {
+		opae_log_err("Failed to get PHY information [e:%d]\n", ret);
+		return -EAGAIN;
+	}
+
+	info->num_retimers = phy_info.num_retimers;
+	info->link_speed = phy_info.link_speed;
+	info->link_status = phy_info.link_status;
+
+	return 0;
+}
+
+static int update_driver(pcidev_id id, char *drv_name)
+{
+	struct rte_pci_device *pci_dev = NULL;
+	char name[OPAE_NAME_SIZE] = {0};
+	int ret = 0;
+
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	if (drv_name) {
+		if (strlen(drv_name) >= OPAE_NAME_SIZE) {
+			opae_log_err("Driver name \'%s\' too long\n",
+				drv_name);
+			return -EINVAL;
+		}
+		strncpy(name, drv_name, sizeof(name));
+	} else {
+		ret = get_driver(id, name, sizeof(name));
+		if (ret < 0)
+			return ret;
+	}
+
+	pci_dev = get_rte_pcidev(id, 0);
+	if (pci_dev) {
+		if (strlen(name) == 0) {
+			pci_dev->kdrv = RTE_PCI_KDRV_NONE;
+		} else {
+			if (!strcmp(name, OPAE_KDRV_VFIO_PCI))
+				pci_dev->kdrv = RTE_PCI_KDRV_VFIO;
+			else if (!strcmp(name, OPAE_KDRV_IGB_UIO))
+				pci_dev->kdrv = RTE_PCI_KDRV_IGB_UIO;
+			else if (!strcmp(name, OPAE_KDRV_UIO_PCI))
+				pci_dev->kdrv = RTE_PCI_KDRV_UIO_GENERIC;
+			else
+				pci_dev->kdrv = RTE_PCI_KDRV_UNKNOWN;
+		}
+	}
+
+	return 0;
+}
+
+int opae_unbind_driver(pcidev_id id)
+{
+	uint16_t dev_id = 0;
+	char path[PATH_MAX] = {0};
+	char drv_name[OPAE_NAME_SIZE] = {0};
+	char null[] = {0};
+	int ret = 0;
+
+	if (!get_rawdev_id(id, &dev_id, 0)) {
+		opae_log_err("%s is probed, remove it first\n", id->bdf);
+		return -EBUSY;
+	}
+
+	ret = get_driver(id, drv_name, sizeof(drv_name));
+	if (ret < 0)
+		return ret;
+
+	if (strlen(drv_name) > 0) {
+		snprintf(path, PATH_MAX, "/sys/bus/pci/drivers/%s/unbind",
+			drv_name);
+		ret = write_file(path, id->bdf, strlen(id->bdf) + 1);
+		if (ret == 0)
+			ret = update_driver(id, null);
+	}
+
+	return ret;
+}
+
+static int check_driver(const char *drv_name)
+{
+	char path[PATH_MAX] = {0};
+	struct stat buf;
+
+	if (!drv_name)
+		return -EINVAL;
+
+	if (strlen(drv_name) > 0) {
+		snprintf(path, PATH_MAX, "/sys/bus/pci/drivers/%s", drv_name);
+		if ((stat(path, &buf) < 0) || ((buf.st_mode & S_IFDIR) == 0)) {
+			opae_log_warn("Driver %s is not installed\n",
+				drv_name);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+int opae_bind_driver(pcidev_id id, char *drv_name)
+{
+	char path[PATH_MAX] = {0};
+	char name[OPAE_NAME_SIZE] = {0};
+	char null[] = {0};
+	int ret = 0;
+
+	ret = check_driver(drv_name);
+	if (ret < 0)
+		return ret;
+
+	ret = get_driver(id, name, sizeof(name));
+	if (ret < 0)
+		return ret;
+
+	if (!strcmp(drv_name, name))   /* driver not change */
+		return 0;
+
+	ret = opae_unbind_driver(id);
+	if (ret < 0)
+		return ret;
+
+	if (strlen(drv_name) > 0) {
+		/* bind driver */
+		snprintf(path, PATH_MAX, "%s/%s/driver_override",
+			rte_pci_get_sysfs_path(), id->bdf);
+		ret = write_file(path, drv_name, strlen(drv_name) + 1);
+		if (ret < 0)
+			goto update_drv;
+
+		snprintf(path, PATH_MAX, "/sys/bus/pci/drivers/%s/bind",
+			drv_name);
+		ret = write_file(path, id->bdf, strlen(id->bdf) + 1);
+		if (ret < 0)
+			goto update_drv;
+
+		snprintf(path, PATH_MAX, "%s/%s/driver_override",
+			rte_pci_get_sysfs_path(), id->bdf);
+		ret = write_file(path, null, 1);
+		if (ret < 0)
+			goto update_drv;
+	}
+
+update_drv:
+	ret = update_driver(id, NULL);
+	if (ret < 0)
+		opae_log_err("Failed to update driver information of %s\n",
+			id->bdf);
+
+	return 0;
+}
+
+int opae_probe_device(pcidev_id id)
+{
+	const struct rte_pci_bus *pci_bus = NULL;
+	struct rte_pci_device *pci_dev = NULL;
+
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	/* make sure device is added in rte_pci_bus devices list */
+	pci_bus = rte_pmd_ifpga_get_pci_bus();
+	if (pci_bus && pci_bus->bus.scan)
+		pci_bus->bus.scan();
+
+	pci_dev = get_rte_pcidev(id, 1);
+	if (!pci_dev)
+		return -ENODEV;
+
+	if (pci_dev->kdrv != RTE_PCI_KDRV_VFIO) {
+		opae_log_err("vfio-pci driver is not bound to %s\n", id->bdf);
+		return -EINVAL;
+	}
+
+	if (!pci_bus || !pci_bus->bus.plug)
+		return -ENODEV;
+
+	return pci_bus->bus.plug(&pci_dev->device);
+}
+
+int opae_remove_device(pcidev_id id)
+{
+	struct rte_pci_device *pci_dev = NULL;
+	struct rte_pci_driver *pci_drv = NULL;
+	int ret = 0;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	pci_dev = get_rte_pcidev(id, 0);
+	if (pci_dev && pci_dev->driver) {
+		pci_drv = pci_dev->driver;
+		ret = pci_drv->remove(pci_dev);
+		if (ret < 0) {
+			opae_log_err("Failed to remove %s [e:%d]\n",
+				id->bdf, ret);
+			return ret;
+		}
+		pci_dev->driver = NULL;
+		pci_dev->device.driver = NULL;
+		if (pci_drv->drv_flags & RTE_PCI_DRV_NEED_MAPPING)
+			rte_pci_unmap_device(pci_dev);
+	}
+
+	return ret;
+}
+
+static int is_pac(pcidev_id id)
+{
+	char path[PATH_MAX] = {0};
+	opae_pci_id pci_id;
+
+	if (check_pcidev_id(id))
+		return 0;
+
+	snprintf(path, PATH_MAX, "%s/%s", rte_pci_get_sysfs_path(), id->bdf);
+	if (get_pci_id(path, &pci_id) < 0)
+		return 0;
+
+	if ((pci_id.vendor_id == 0x8086) && (pci_id.device_id == 0x0b30))
+		return 1;
+
+	return 0;
+}
+
+int opae_get_parent(pcidev_id id, pcidev_id parent)
+{
+	char path[PATH_MAX] = {0};
+	char link[PATH_MAX] = {0};
+	int ret = 0;
+
+	if (!id || !parent) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		ret = -EINVAL;
+		goto end;
+	}
+
+	snprintf(path, PATH_MAX, "%s/%s", rte_pci_get_sysfs_path(), id->bdf);
+	ret = readlink(path, link, PATH_MAX);
+	if (ret >= PATH_MAX) {
+		opae_log_err("Length of link path exceeds %u\n", PATH_MAX);
+		ret = -ENAMETOOLONG;
+		goto end;
+	}
+
+	if (ret > 0) {
+		ret = extract_path(link, 1, parent->bdf, sizeof(parent->bdf));
+		if (!strncmp(parent->bdf, "pci", 3)) {
+			parent->bdf[0] = 0;
+			ret = -ENODEV;
+		}
+	} else {
+		parent->bdf[0] = 0;
+		if (ret == 0)
+			opae_log_err("Length of link path is 0\n");
+		else
+			opae_log_err("No link path for \'%s\'\n", path);
+	}
+end:
+	if (ret <= 0)
+		opae_log_err("%s has no parent\n", id->bdf);
+
+	return ret;
+}
+
+int opae_get_child(pcidev_id id, pcidev_id child, int size)
+{
+	glob_t pglob = {.gl_pathc = 0, .gl_pathv = NULL};
+	char path[PATH_MAX] = {0};
+	int i, count = 0;
+	int len = 0;
+	int ret = 0;
+
+	if (!id || (size < 0)) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	snprintf(path, PATH_MAX, "%s/%s/*:*:*.?", rte_pci_get_sysfs_path(),
+		id->bdf);
+	ret = glob(path, 0, NULL, &pglob);
+	if (ret == 0) {
+		if (child && (size > 0)) {
+			for (i = 0; i < (int)pglob.gl_pathc; i++) {
+				len = extract_path(pglob.gl_pathv[i], 0,
+					child->bdf, sizeof(child->bdf));
+				if (len <= 0) {
+					child->bdf[0] = 0;
+					continue;
+				}
+				if (++count >= size)
+					break;
+				child++;
+			}
+		} else {
+			count = (int)pglob.gl_pathc;
+		}
+		globfree(&pglob);
+	} else {
+		if (pglob.gl_pathv)
+			globfree(&pglob);
+	}
+
+	return count;
+}
+
+int opae_get_pf1(pcidev_id id, pcidev_id peer, int size)
+{
+	opae_pci_device parent;
+	opae_pci_device child[4];
+	int n = 0;
+	int ret = 0;
+
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	if (!is_pac(id)) {
+		opae_log_info("%s has no peer function\n", id->bdf);
+		return -EINVAL;
+	}
+
+	ret = opae_get_parent(id, &parent);
+	if (ret < 0)
+		return -ENODEV;
+	ret = opae_get_parent(&parent, &parent);
+	if (ret < 0)
+		return -ENODEV;
+
+	n = opae_get_child(&parent, child,
+		sizeof(child) / sizeof(opae_pci_device));
+	/* there should have four downstream ports of PCI switch on board */
+	if (n == 4) {
+		n = opae_get_child(&child[3], peer, size);
+	} else {
+		peer->bdf[0] = 0;
+		opae_log_dbg("%s has %d child(s)\n", parent.bdf, n);
+		n = 0;
+	}
+
+	return n;
+}
+
+int opae_update_flash(pcidev_id id, const char *image, uint64_t *status)
+{
+	opae_img_info info;
+	uint16_t dev_id = 0;
+	int ret = 0;
+
+	ret = opae_get_image_info(image, &info);
+	if (ret < 0) {
+		opae_log_err("Failed to get image information [e:%d]\n", ret);
+		return -EINVAL;
+	}
+
+	if ((info.type != OPAE_IMG_TYPE_BBS) &&
+		(info.type != OPAE_IMG_TYPE_BMC)) {
+		opae_log_err("Image is not supported [t:%u]\n", info.type);
+		return -EOPNOTSUPP;
+	}
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	return rte_pmd_ifpga_update_flash(dev_id, image, status);
+}
+
+int opae_cancel_flash_update(pcidev_id id, int force)
+{
+	uint16_t dev_id = 0;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	return rte_pmd_ifpga_stop_update(dev_id, force);
+}
+
+static int find_pci_ecap(int fd, unsigned int cap)
+{
+	uint32_t header = 0;
+	int ttl = (RTE_PCI_CFG_SPACE_EXP_SIZE - RTE_PCI_CFG_SPACE_SIZE) / 8;
+	int pos = RTE_PCI_CFG_SPACE_SIZE;
+	int ret = 0;
+
+	ret = pread(fd, &header, sizeof(header), pos);
+	if (ret < 0) {
+		opae_log_err("Failed to read from PCI configuration space [e:%s]\n",
+			strerror(errno));
+		return ret;
+	}
+	opae_log_dbg("Read 0x%08x from PCI configuration space 0x%x\n",
+		header, pos);
+
+	if (header == 0) {
+		opae_log_err("Capability is empty\n");
+		return 0;
+	}
+
+	while (ttl-- > 0) {
+		if ((RTE_PCI_EXT_CAP_ID(header) == cap) && (pos != 0))
+			return pos;
+
+		pos = RTE_PCI_EXT_CAP_NEXT(header);
+		if (pos < RTE_PCI_CFG_SPACE_SIZE) {
+			opae_log_err("Position of capability is invalid"
+						 "[e:%d]\n", pos);
+			break;
+		}
+		ret = pread(fd, &header, sizeof(header), pos);
+		if (ret < 0) {
+			opae_log_err("Failed to read from PCI config space [e:%s]\n",
+				strerror(errno));
+			return ret;
+		}
+		opae_log_dbg("Read 0x%08x from PCI configuration space 0x%x\n",
+			header, pos);
+	}
+
+	return 0;
+}
+
+static int set_aer(pcidev_id id, uint32_t v1, uint32_t v2, int record)
+{
+	char path[PATH_MAX] = {0};
+	uint32_t val = 0;
+	int fd = -1;
+	int pos = 0;
+	int ret = 0;
+
+	if (!id)
+		return -EINVAL;
+
+	snprintf(path, PATH_MAX, "%s/%s/config",
+		rte_pci_get_sysfs_path(), id->bdf);
+	fd = open(path, O_RDWR);
+	if (fd < 0) {
+		opae_log_err("Failed to open \'%s\' for RDWR [e:%s]\n",
+			path, strerror(errno));
+		return -EIO;
+	}
+
+	pos = find_pci_ecap(fd, RTE_PCI_EXT_CAP_ID_ERR);
+	if (pos <= 0) {
+		opae_log_warn("AER capability is not present\n");
+		ret = -ENXIO;
+		goto close_fd;
+	}
+
+	if (record) {
+		ret = pread(fd, &val, sizeof(val), pos + 0x08);
+		if (ret < 0) {
+			opae_log_err("Failed to read from PCI config space [e:%s]\n",
+				strerror(errno));
+			goto close_fd;
+		}
+		opae_log_dbg("Read 0x%08x from PCI configuration space 0x%x\n",
+			val, pos + 0x08);
+		dev_aer[0] = val;
+
+		ret = pread(fd, &val, sizeof(val), pos + 0x14);
+		if (ret < 0) {
+			opae_log_err("Failed to read from PCI config space [e:%s]\n",
+				strerror(errno));
+			goto close_fd;
+		}
+		opae_log_dbg("Read 0x%08x from PCI configuration space 0x%x\n",
+			val, pos + 0x14);
+		dev_aer[1] = val;
+	}
+
+	opae_log_dbg("Write 0x%08x to PCI configuration space 0x%x\n",
+		v1, pos + 0x08);
+	ret = pwrite(fd, &v1, sizeof(v1), pos + 0x08);
+	if (ret < 0) {
+		opae_log_err("Failed to write to PCI config space 0x%x [e:%s]\n",
+			pos + 0x08, strerror(errno));
+		goto close_fd;
+	}
+
+	opae_log_dbg("Write 0x%08x to PCI configuration space 0x%x\n",
+		v2, pos + 0x14);
+	ret = pwrite(fd, &v2, sizeof(v2), pos + 0x14);
+	if (ret < 0) {
+		opae_log_err("Failed to write to PCI config space 0x%x [e:%s]\n",
+			pos + 0x14, strerror(errno));
+	}
+
+close_fd:
+	close(fd);
+	return ret < 0 ? ret : 0;
+}
+
+static int enable_aer(pcidev_id id)
+{
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	opae_log_info("Enable AER of %s\n", id->bdf);
+
+	return set_aer(id, dev_aer[0], dev_aer[1], 0);
+}
+
+static int disable_aer(pcidev_id id)
+{
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	opae_log_info("Disable AER of %s\n", id->bdf);
+
+	return set_aer(id, 0xffffffff, 0xffffffff, 1);
+}
+
+static int remove_tree(pcidev_id id)
+{
+	int i, n = 0;
+	pcidev_id child;
+	int ret = 0;
+
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	n = opae_get_child(id, NULL, 0);
+	if (n > 0) {
+		child = (pcidev_id)rte_zmalloc(NULL,
+			sizeof(opae_pci_device) * n, 0);
+		if (!child) {
+			opae_log_err("Failed to malloc for children of %s\n",
+				id->bdf);
+			ret = -ENOMEM;
+			goto end;
+		}
+
+		opae_get_child(id, child, n);
+		for (i = 0; i < n; i++)
+			remove_tree(&child[i]);
+		rte_free(child);
+	}
+
+end:
+	opae_remove_device(id);
+	return ret;
+}
+
+static int remove_device(pcidev_id id)
+{
+	char path[PATH_MAX] = {0};
+	char one[] = {'1', 0};
+	int ret = 0;
+
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	opae_log_info("Remove %s from system\n", id->bdf);
+
+	snprintf(path, PATH_MAX, "%s/%s/remove",
+		rte_pci_get_sysfs_path(), id->bdf);
+	ret = write_file(path, one, strlen(one));
+	if (ret < 0) {
+		opae_log_err("Failed to remove %s from system\n", id->bdf);
+		return ret;
+	}
+
+	remove_tree(id);
+
+	return 0;
+}
+
+static int scan_device(pcidev_id parent, pcidev_id id)
+{
+	char path[PATH_MAX] = {0};
+	char bus[8] = {0};
+	char one[] = {'1', 0};
+	char pwr[16] = {0};
+	char pwr_on[] = {'o', 'n', 0};
+	int pwr_on_failed = 0;
+	int ret = 0;
+
+	if (!parent) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+	opae_log_info("Rescan devices under %s\n", parent->bdf);
+
+	if (id) {   /* scan specified bus under parent device */
+		snprintf(path, PATH_MAX, "%s/%s/power/control",
+			rte_pci_get_sysfs_path(), parent->bdf);
+		ret = read_file(path, pwr, sizeof(pwr));
+		if (ret < 0)
+			return ret;
+
+		if (strcmp(pwr, "on")) {
+			ret = write_file(path, pwr_on, strlen(pwr_on));
+			if (ret < 0)
+				pwr_on_failed = 1;
+			else
+				sleep(1);
+		}
+
+		snprintf(bus, sizeof(bus), "%s", id->bdf);
+		snprintf(path, PATH_MAX, "%s/%s/pci_bus/%s/rescan",
+			rte_pci_get_sysfs_path(), parent->bdf, bus);
+		ret = write_file(path, one, strlen(one));
+		if (ret < 0)
+			return ret;
+
+		if (pwr_on_failed) {   /* workaround for power on failed */
+			ret = write_file(path, one, strlen(one));
+			if (ret < 0)
+				return ret;
+		}
+
+		if (strcmp(pwr, "on")) {
+			snprintf(path, PATH_MAX, "%s/%s/power/control",
+				rte_pci_get_sysfs_path(), parent->bdf);
+			ret = write_file(path, pwr, strlen(pwr));
+		}
+	} else {   /* scan all buses under parent device */
+		snprintf(path, PATH_MAX, "%s/%s/rescan",
+			rte_pci_get_sysfs_path(), parent->bdf);
+		ret = write_file(path, one, strlen(one));
+	}
+
+	return ret;
+}
+
+int opae_reboot_device(pcidev_id id, int type, int page)
+{
+	uint16_t dev_id = 0;
+	opae_pci_device fpga;    /* FPGA after reboot */
+	opae_pci_device parent;
+	opae_pci_device peer[2];   /* physical function 1 of FPGA */
+	opae_pci_device peer_parent;
+	opae_pci_device ups;   /* upstream port device */
+	opae_pci_device root;  /* port connected to PAC */
+	pcidev_id peer_primary = NULL;
+	char drv_name[OPAE_NAME_SIZE] = {0};
+	int n = 0;
+	int i = 0;
+	int ret = 0;
+
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	if (!is_pac(id)) {
+		opae_log_err("%s can not be rebooted\n", id->bdf);
+		return -EINVAL;
+	}
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	ret = opae_get_parent(id, &parent);
+	if (ret < 0)
+		return -ENODEV;
+	ret = opae_get_parent(&parent, &ups);
+	if (ret < 0)
+		return -ENODEV;
+	ret = opae_get_parent(&ups, &root);
+	if (ret < 0)
+		return -ENODEV;
+
+	n = opae_get_pf1(id, peer, sizeof(peer) / sizeof(opae_pci_device));
+	if (n <= 0) {
+		opae_log_err("PF1 of %s is not found\n", id->bdf);
+	} else {
+		peer_primary = &peer[0];
+		ret = opae_get_parent(peer_primary, &peer_parent);
+		if (ret < 0)
+			return -ENODEV;
+	}
+
+	get_driver(id, drv_name, sizeof(drv_name));  /* save original driver */
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (rte_pmd_ifpga_reboot_try(dev_id)) {
+		opae_log_warn("Update or reboot is in progress\n");
+		return -EAGAIN;
+	}
+
+	if (type == 0) {
+		/* disable AER */
+		ret = disable_aer(&parent);
+		if (ret < 0) {
+			opae_log_err("Failed to disable AER of %s\n",
+				parent.bdf);
+			goto reboot_end;
+		}
+		ret = disable_aer(&peer_parent);
+		if (ret < 0) {
+			opae_log_err("Failed to disable AER of %s\n",
+				peer_parent.bdf);
+			goto reboot_end;
+		}
+
+		/* trigger reconfiguration */
+		ret = rte_pmd_ifpga_reload(dev_id, type, page);
+		if (ret == 0) {
+			ret = remove_device(id);
+			for (i = 0; i < n; i++)
+				ret += remove_device(&peer[i]);
+			if (ret == 0) {
+				opae_log_info("Wait 10 seconds for FPGA reloading\n");
+				sleep(10);
+				ret = scan_device(&parent, id);
+				if (ret < 0)
+					opae_log_err("Failed to rescan %s\n",
+						id->bdf);
+				if (peer_primary) {
+					ret = scan_device(&peer_parent,
+						peer_primary);
+					if (ret < 0) {
+						opae_log_err("Failed to rescan %s\n",
+							peer_primary->bdf);
+					}
+				}
+			}
+		}
+
+		/* restore AER */
+		if (enable_aer(&parent) < 0) {
+			opae_log_err("Failed to enable AER of %s\n",
+				parent.bdf);
+		}
+		if (enable_aer(&peer_parent) < 0) {
+			opae_log_err("Failed to enable AER of %s\n",
+				peer_parent.bdf);
+		}
+	} else if (type == 1) {
+		/* disable AER */
+		ret = disable_aer(&root);
+		if (ret < 0) {
+			opae_log_err("Failed to disable AER of %s\n", root.bdf);
+			goto reboot_end;
+		}
+
+		/* trigger reconfiguration */
+		ret = rte_pmd_ifpga_reload(dev_id, type, page);
+		if (ret == 0) {
+			ret += remove_device(&ups);
+			if (ret == 0) {
+				opae_log_info("Wait 10 seconds for BMC reloading\n");
+				sleep(10);
+				ret = scan_device(&root, &ups);
+				if (ret < 0)
+					opae_log_err("Failed to rescan %s\n",
+						ups.bdf);
+			}
+		}
+
+		/* restore AER */
+		if (enable_aer(&root) < 0)
+			opae_log_err("Failed to enable AER of %s\n", root.bdf);
+	} else {
+		opae_log_err("Type of reboot is not supported [t:%d]\n", type);
+		ret = -EINVAL;
+		goto reboot_end;
+	}
+
+	/* update id if bdf changed after reboot */
+	if (opae_get_child(&parent, &fpga, 1) == 1) {
+		if (strcmp(id->bdf, fpga.bdf))
+			id = &fpga;
+	}
+
+	ret = opae_bind_driver(id, drv_name);
+	if (ret < 0)
+		opae_log_err("Failed to bind original driver of %s\n", id->bdf);
+
+	ret = opae_probe_device(id);
+	if (ret < 0)
+		opae_log_err("Failed to probe %s [e:%d]\n", id->bdf, ret);
+
+reboot_end:
+	rte_pmd_ifpga_set_rsu_status(dev_id, 0, 0);
+	return ret;
+}
+
+int opae_partial_reconfigure(pcidev_id id, int port, const char *gbs)
+{
+	uint16_t dev_id = 0;
+
+	if (!id || !gbs) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	return rte_pmd_ifpga_partial_reconfigure(dev_id, port, gbs);
+}
+
+int opae_read_pci_cfg(pcidev_id id, uint32_t address, uint32_t *value)
+{
+	char path[PATH_MAX] = {0};
+	int fd = -1;
+	int ret = 0;
+
+	if (!id || !value) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	snprintf(path, PATH_MAX, "%s/%s/config", rte_pci_get_sysfs_path(),
+		id->bdf);
+	fd = open(path, O_RDONLY);
+	if (fd < 0) {
+		opae_log_dbg("Failed to open \'%s\' for RDONLY [e:%s]\n",
+			path, strerror(errno));
+		return -EIO;
+	}
+
+	ret = pread(fd, value, 4, address);
+	if (ret < 0) {
+		opae_log_err("Failed to read from PCI device %s [e:%s]\n",
+			id->bdf, strerror(errno));
+		close(fd);
+		return ret;
+	}
+
+	opae_log_dbg("CONFIG+0x%08x -> 0x%08x\n", address, *value);
+	close(fd);
+	return 0;
+}
+
+int opae_write_pci_cfg(pcidev_id id, uint32_t address, uint32_t value)
+{
+	char path[PATH_MAX] = {0};
+	int fd = -1;
+	int ret = 0;
+
+	if (!id) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	snprintf(path, PATH_MAX, "%s/%s/config", rte_pci_get_sysfs_path(),
+		id->bdf);
+	fd = open(path, O_WRONLY);
+	if (fd < 0) {
+		opae_log_dbg("Failed to open \'%s\' for WRONLY [e:%s]\n",
+			path, strerror(errno));
+		return -EIO;
+	}
+
+	ret = pwrite(fd, &value, 4, address);
+	if (ret < 0) {
+		opae_log_err("Failed to write to PCI device %s [e:%s]\n",
+			id->bdf, strerror(errno));
+		close(fd);
+		return ret;
+	}
+
+	opae_log_dbg("CONFIG+0x%08x <- 0x%08x\n", address, value);
+	close(fd);
+	return 0;
+}
diff --git a/examples/ifpga/opae_api.h b/examples/ifpga/opae_api.h
new file mode 100644
index 0000000000..3b6bc01fbc
--- /dev/null
+++ b/examples/ifpga/opae_api.h
@@ -0,0 +1,244 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _OPAE_API_H
+#define _OPAE_API_H
+
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+extern int opae_log_level;
+extern FILE *opae_log_file;
+
+#define OPAE_LOG_API      0  /**< Critical conditions.              */
+#define OPAE_LOG_ERR      1  /**< Error conditions.                 */
+#define OPAE_LOG_WARN     2  /**< Warning conditions.               */
+#define OPAE_LOG_INFO     3  /**< Informational.                    */
+#define OPAE_LOG_DEBUG    4  /**< Debug-level messages.             */
+
+#define opae_log(type, fmt, args...)             \
+do {                                             \
+	if (opae_log_level >= OPAE_LOG_##type) {     \
+		printf(fmt, ##args);                     \
+		if (opae_log_file) {                     \
+			fprintf(opae_log_file, fmt, ##args); \
+			fflush(opae_log_file);               \
+		}                                        \
+	}                                            \
+} while (0)
+
+#define opae_log_api(fmt, args...)  opae_log(API, "OPAE-API: "fmt, ##args)
+#define opae_log_err(fmt, args...)  opae_log(ERR, "OPAE-ERR: "fmt, ##args)
+#define opae_log_dbg(fmt, args...)  opae_log(DEBUG, "OPAE-DBG: "fmt, ##args)
+#define opae_log_warn(fmt, args...) opae_log(WARN, "OPAE-WARN: "fmt, ##args)
+#define opae_log_info(fmt, args...) opae_log(INFO, "OPAE-INFO: "fmt, ##args)
+
+#define EAL_INIT_FUNCTION    "init"
+#define EAL_DEFAULT_OPTIONS  "--proc-type auto"
+
+#define OPAE_KDRV_UNKNOWN           "unknown"
+#define OPAE_KDRV_VFIO_PCI          "vfio-pci"
+#define OPAE_KDRV_IGB_UIO           "igb_uio"
+#define OPAE_KDRV_UIO_PCI           "uio_pci_generic"
+#define OPAE_KDRV_INTEL_FPGA_PCI    "intel-fpga-pci"
+
+typedef struct {
+	uint32_t major;
+	uint32_t minor;
+	uint32_t micro;
+} opae_api_version;
+
+#define OPAE_NAME_SIZE  32
+
+typedef struct {
+	char bdf[OPAE_NAME_SIZE];   /* segment:bus:device.function */
+} opae_pci_device;
+
+typedef opae_pci_device *pcidev_id;
+
+typedef struct {
+	uint32_t class_id;            /**< Class ID or RTE_CLASS_ANY_ID. */
+	uint16_t vendor_id;           /**< Vendor ID or PCI_ANY_ID. */
+	uint16_t device_id;           /**< Device ID or PCI_ANY_ID. */
+	uint16_t subsystem_vendor_id; /**< Subsystem vendor ID or PCI_ANY_ID. */
+	uint16_t subsystem_device_id; /**< Subsystem device ID or PCI_ANY_ID. */
+} opae_pci_id;
+
+typedef struct {
+	uint32_t domain;              /**< Device domain */
+	uint8_t bus;                  /**< Device bus */
+	uint8_t devid;                /**< Device ID */
+	uint8_t function;             /**< Device function. */
+} opae_pci_addr;
+
+typedef struct {
+	char pci_addr[OPAE_NAME_SIZE];  /* segment:bus:device.function */
+	char drv_name[OPAE_NAME_SIZE];  /* vfio-pci, intel-fpga-pci, etc. */
+	opae_pci_id id;
+	opae_pci_addr addr;
+} opae_pci_property;
+
+#define BIT_SET_8   0xFF
+#define BIT_SET_16  0xFFFF
+#define BIT_SET_32  0xFFFFFFFF
+
+typedef struct {
+	uint8_t b[16];
+} opae_uuid;
+
+typedef struct {
+	uint32_t boot_page;
+	uint32_t num_ports;
+	uint64_t bitstream_id;
+	uint64_t bitstream_metadata;
+	opae_uuid pr_id;
+	char platform_name[OPAE_NAME_SIZE];
+	char dcp_version[OPAE_NAME_SIZE];
+	char release_name[OPAE_NAME_SIZE];
+	char interface_type[OPAE_NAME_SIZE];
+	char build_version[OPAE_NAME_SIZE];
+} opae_fme_property;
+
+typedef struct {
+	opae_uuid afu_id;
+	uint32_t type;   /* AFU memory access control type */
+	uint32_t index;  /* PORT index */
+} opae_port_property;
+
+typedef struct {
+	char bmc_version[OPAE_NAME_SIZE];
+	char fw_version[OPAE_NAME_SIZE];
+} opae_bmc_property;
+
+typedef struct {
+	uint32_t num_retimers;
+	uint32_t link_speed;
+	uint32_t link_status;  /* each bit corresponding to one link status */
+} opae_phy_info;
+
+typedef struct {
+	union {
+		uint64_t id;
+		struct {
+			uint8_t build_patch;
+			uint8_t build_minor;
+			uint8_t build_major;
+			uint8_t fvl_bypass:1;
+			uint8_t mac_lightweight:1;
+			uint8_t disagregate:1;
+			uint8_t lightweiht:1;
+			uint8_t seu:1;
+			uint8_t ptp:1;
+			uint8_t reserve:2;
+			uint16_t interface:4;
+			uint16_t afu_revision:12;
+			uint16_t patch:4;
+			uint16_t minor:4;
+			uint16_t major:4;
+			uint16_t reserved:4;
+		};
+	};
+} opae_bitstream_id;
+
+typedef struct {
+	union {
+		uint32_t version;
+		struct {
+			uint8_t micro;
+			uint8_t minor;
+			uint8_t major;
+			uint8_t board;
+		};
+	};
+} opae_bmc_version;
+
+#define OPAE_MAX_PORT_NUM   4
+
+#define OPAE_PROP_PCI   0x01
+#define OPAE_PROP_FME   0x02
+#define OPAE_PROP_PORT  0x04
+#define OPAE_PROP_BMC   0x08
+#define OPAE_PROP_ALL   \
+	(OPAE_PROP_PCI | OPAE_PROP_FME | OPAE_PROP_PORT | OPAE_PROP_BMC)
+
+typedef struct {
+	opae_pci_property pci;
+	opae_fme_property fme;
+	opae_port_property port[OPAE_MAX_PORT_NUM];
+	opae_bmc_property bmc;
+} opae_fpga_property;
+
+typedef struct {
+	uint64_t guid_h;
+	uint64_t guid_l;
+	uint32_t metadata_len;
+} gbs_header;
+
+#define OPAE_IMG_TYPE_BBS       0
+#define OPAE_IMG_TYPE_BMC       1
+#define OPAE_IMG_TYPE_GBS       2
+#define OPAE_IMG_TYPE(t)        ((t) & 0xff)
+
+#define OPAE_IMG_SUBTYPE_UPDATE             0
+#define OPAE_IMG_SUBTYPE_CANCELLATION       1
+#define OPAE_IMG_SUBTYPE_ROOT_KEY_HASH_256  2
+#define OPAE_IMG_SUBTYPE_ROOT_KEY_HASH_384  3
+#define OPAE_IMG_SUBTYPE(t)     (((t) >> 8) & 0xff)
+
+#define OPAE_IMG_BLK0_SIZE      128
+#define OPAE_IMG_BLK0_MAGIC     0xb6eafd19
+#define OPAE_IMG_BLK1_SIZE      896
+#define OPAE_IMG_HDR_SIZE   (OPAE_IMG_BLK0_SIZE + OPAE_IMG_BLK1_SIZE)
+#define OPAE_IMG_PL_MIN_SIZE    128
+
+typedef struct {
+	uint32_t magic;
+	uint32_t payload_len;
+	uint32_t payload_type;
+} opae_img_hdr;
+
+typedef struct {
+	int type;
+	int subtype;
+	uint32_t total_len;
+	uint32_t payload_offset;
+	uint32_t payload_len;
+} opae_img_info;
+
+void opae_get_api_version(opae_api_version *version);
+int opae_set_log_level(int level);
+int opae_set_log_file(char *path, int clean);
+int opae_get_proc_type(void);
+int opae_get_parent(pcidev_id id, pcidev_id parent);
+int opae_get_child(pcidev_id id, pcidev_id child, int size);
+int opae_get_pf1(pcidev_id id, pcidev_id peer, int size);
+int opae_init_eal(int argc, char **argv);
+int opae_cleanup_eal(void);
+int opae_enumerate(opae_pci_id *filter, pcidev_id list, int size);
+int opae_probe_device(pcidev_id id);
+int opae_remove_device(pcidev_id id);
+int opae_unbind_driver(pcidev_id id);
+int opae_bind_driver(pcidev_id id, char *drv_name);
+int opae_get_property(pcidev_id id, opae_fpga_property *prop, int type);
+int opae_get_phy_info(pcidev_id id, opae_phy_info *info);
+int opae_partial_reconfigure(pcidev_id id, int port, const char *gbs);
+int opae_get_image_info(const char *image, opae_img_info *info);
+int opae_cancel_flash_update(pcidev_id id, int force);
+int opae_update_flash(pcidev_id id, const char *image, uint64_t *status);
+int opae_reboot_device(pcidev_id id, int type, int page);
+int opae_store_rsu_status(pcidev_id id, uint32_t status, uint32_t progress);
+int opae_load_rsu_status(pcidev_id id, uint32_t *status, uint32_t *progress);
+int opae_read_pci_cfg(pcidev_id id, uint32_t address, uint32_t *value);
+int opae_write_pci_cfg(pcidev_id id, uint32_t address, uint32_t value);
+
+#ifdef __cplusplus
+}
+#endif
+
+
+#endif  /* _OPAE_API_H */
diff --git a/examples/meson.build b/examples/meson.build
index b9ab24223f..b029715277 100644
--- a/examples/meson.build
+++ b/examples/meson.build
@@ -16,7 +16,7 @@ all_examples = [
 	'eventdev_pipeline',
 	'fips_validation', 'flow_classify',
 	'flow_filtering', 'helloworld',
-	'ioat',
+	'ioat', 'ifpga',
 	'ip_fragmentation', 'ip_pipeline',
 	'ip_reassembly', 'ipsec-secgw',
 	'ipv4_multicast', 'kni',
-- 
2.29.2

Re: [dpdk-dev] [dpdk-stable] [PATCH v13 0/4] raw/ifpga: add extra OPAE APIs

[Thomas Monjalon]

10/02/2021 02:48, Wei Huang:
> Cyborg is part of OpenStack, it needs some OPAE APIs to manage
> devices with Intel FPGA. The first three patches implement extra
> APIs to meet Cyborg requirement. The last patch add an example
> to show how to use these APIs.
> 
> Main changes from v12:
> - implement ifpga APIs in rte_pmd_ifpga.c
> 
> Wei Huang (4):
>   raw/ifpga: add fpga rsu APIs
>   raw/ifpga: add APIs to get fpga information
>   raw/ifpga: add miscellaneous APIs
>   examples/ifpga: add example for ifpga APIs

Not sure we want to add an example for a driver-specific API.

PS: please use --in-reply-to to keep all versions in the same thread.

Re: [dpdk-dev] [dpdk-stable] [PATCH v13 0/4] raw/ifpga: add extra OPAE APIs

[Huang, Wei]

-----Original Message-----
From: Thomas Monjalon <thomas@monjalon.net> 
Sent: Wednesday, February 10, 2021 17:38
To: Huang, Wei <wei.huang@intel.com>
Cc: dev@dpdk.org; Xu, Rosen <rosen.xu@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>; stable@dpdk.org; Zhang, Tianfei <tianfei.zhang@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>
Subject: Re: [dpdk-stable] [PATCH v13 0/4] raw/ifpga: add extra OPAE APIs

10/02/2021 02:48, Wei Huang:
> Cyborg is part of OpenStack, it needs some OPAE APIs to manage devices 
> with Intel FPGA. The first three patches implement extra APIs to meet 
> Cyborg requirement. The last patch add an example to show how to use 
> these APIs.
> 
> Main changes from v12:
> - implement ifpga APIs in rte_pmd_ifpga.c
> 
> Wei Huang (4):
>   raw/ifpga: add fpga rsu APIs
>   raw/ifpga: add APIs to get fpga information
>   raw/ifpga: add miscellaneous APIs
>   examples/ifpga: add example for ifpga APIs

Not sure we want to add an example for a driver-specific API.
Thomas, this example has two purposes, one is for how to use ifpga API, the other is to create static library for Cyborg application.
PS: please use --in-reply-to to keep all versions in the same thread.

Re: [dpdk-dev] [dpdk-stable] [PATCH v13 0/4] raw/ifpga: add extra OPAE APIs

[Thomas Monjalon]

22/02/2021 02:59, Huang, Wei:
> 
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net> 
> Sent: Wednesday, February 10, 2021 17:38
> To: Huang, Wei <wei.huang@intel.com>
> Cc: dev@dpdk.org; Xu, Rosen <rosen.xu@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>; stable@dpdk.org; Zhang, Tianfei <tianfei.zhang@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>
> Subject: Re: [dpdk-stable] [PATCH v13 0/4] raw/ifpga: add extra OPAE APIs
> 
> 10/02/2021 02:48, Wei Huang:
> > Cyborg is part of OpenStack, it needs some OPAE APIs to manage devices 
> > with Intel FPGA. The first three patches implement extra APIs to meet 
> > Cyborg requirement. The last patch add an example to show how to use 
> > these APIs.
> > 
> > Main changes from v12:
> > - implement ifpga APIs in rte_pmd_ifpga.c
> > 
> > Wei Huang (4):
> >   raw/ifpga: add fpga rsu APIs
> >   raw/ifpga: add APIs to get fpga information
> >   raw/ifpga: add miscellaneous APIs
> >   examples/ifpga: add example for ifpga APIs
> 
> Not sure we want to add an example for a driver-specific API.
> Thomas, this example has two purposes, one is for how to use ifpga API, the other is to create static library for Cyborg application.
> PS: please use --in-reply-to to keep all versions in the same thread.

It seems you are missing my and your reply together
without distinction.
Please take care your email client is well configured
to prefix lines with >

About the example, I still think it is a no-go,
but it should be a techboard decision.

[dpdk-dev] [PATCH v14 0/4] raw/ifpga: add extra OPAE APIs

[Wei Huang]

Cyborg is part of OpenStack, it needs some OPAE APIs to manage
devices with Intel FPGA. The first three patches implement extra
APIs to meet Cyborg requirement. The last patch add an example
to show how to use these APIs.

Main changes from v13:
- fix compilation issue of unused const variable 'kdrv'

Wei Huang (4):
  raw/ifpga: add fpga rsu APIs
  raw/ifpga: add APIs to get fpga information
  raw/ifpga: add miscellaneous APIs
  examples/ifpga: add example for ifpga APIs

 MAINTAINERS                                |    3 +
 doc/api/doxy-api-index.md                  |    3 +-
 doc/guides/sample_app_ug/ifpga.rst         |  387 +++++
 doc/guides/sample_app_ug/index.rst         |    1 +
 drivers/raw/ifpga/base/ifpga_api.c         |   34 +
 drivers/raw/ifpga/base/ifpga_defines.h     |    1 +
 drivers/raw/ifpga/base/ifpga_feature_dev.c |   21 +
 drivers/raw/ifpga/base/ifpga_feature_dev.h |    1 +
 drivers/raw/ifpga/base/ifpga_fme.c         |   36 +-
 drivers/raw/ifpga/base/ifpga_fme_rsu.c     |  428 +++++
 drivers/raw/ifpga/base/ifpga_hw.h          |    1 +
 drivers/raw/ifpga/base/ifpga_sec_mgr.c     |  639 ++++++++
 drivers/raw/ifpga/base/ifpga_sec_mgr.h     |   93 ++
 drivers/raw/ifpga/base/meson.build         |    2 +
 drivers/raw/ifpga/base/opae_hw_api.c       |   77 +
 drivers/raw/ifpga/base/opae_hw_api.h       |   12 +
 drivers/raw/ifpga/base/opae_ifpga_hw_api.h |    1 +
 drivers/raw/ifpga/base/opae_intel_max10.c  |   48 +
 drivers/raw/ifpga/base/opae_intel_max10.h  |   44 +
 drivers/raw/ifpga/ifpga_rawdev.c           |   30 +
 drivers/raw/ifpga/ifpga_rawdev.h           |    7 +-
 drivers/raw/ifpga/meson.build              |    4 +-
 drivers/raw/ifpga/rte_pmd_ifpga.c          |  429 ++++++
 drivers/raw/ifpga/rte_pmd_ifpga.h          |  308 ++++
 drivers/raw/ifpga/version.map              |   18 +
 examples/ifpga/Makefile                    |   43 +
 examples/ifpga/commands.c                  | 1294 ++++++++++++++++
 examples/ifpga/commands.h                  |   16 +
 examples/ifpga/main.c                      |   38 +
 examples/ifpga/meson.build                 |   20 +
 examples/ifpga/opae_api.c                  | 1630 ++++++++++++++++++++
 examples/ifpga/opae_api.h                  |  244 +++
 examples/meson.build                       |    2 +-
 33 files changed, 5909 insertions(+), 6 deletions(-)
 create mode 100644 doc/guides/sample_app_ug/ifpga.rst
 create mode 100644 drivers/raw/ifpga/base/ifpga_fme_rsu.c
 create mode 100644 drivers/raw/ifpga/base/ifpga_sec_mgr.c
 create mode 100644 drivers/raw/ifpga/base/ifpga_sec_mgr.h
 create mode 100644 drivers/raw/ifpga/rte_pmd_ifpga.c
 create mode 100644 drivers/raw/ifpga/rte_pmd_ifpga.h
 create mode 100644 examples/ifpga/Makefile
 create mode 100644 examples/ifpga/commands.c
 create mode 100644 examples/ifpga/commands.h
 create mode 100644 examples/ifpga/main.c
 create mode 100644 examples/ifpga/meson.build
 create mode 100644 examples/ifpga/opae_api.c
 create mode 100644 examples/ifpga/opae_api.h

-- 
2.29.2

[dpdk-dev] [PATCH v14 1/4] raw/ifpga: add fpga rsu APIs

[Wei Huang]

RSU (Remote System Update) depends on secure manager which may be
different on various implementations, so a new secure manager device
is implemented for adapting such difference.
There are five APIs added:
1. rte_pmd_ifpga_get_dev_id() get raw device ID of ifpga device from
PCI address like 'Domain:Bus:Dev.Func'.
2. rte_pmd_ifpga_update_flash() update flash with specific image file.
3. rte_pmd_ifpga_stop_update() abort flash update process.
4. rte_pmd_ifpga_reboot_try() check current ifpga status and change it
to reboot status if it is idle.
5. rte_pmd_ifpga_reload() trigger full reconfiguration of ifpga device.

Signed-off-by: Wei Huang <wei.huang@intel.com>
Acked-by: Tianfei Zhang <tianfei.zhang@intel.com>
Acked-by: Rosen Xu <rosen.xu@intel.com>
---
v2: fix coding style issue in ifpga_fme_rsu.c and ifpga_sec_mgr.c
---
v3: fix compilation issues in ifpga_fme_rsu.c
---
v4: fix compilation issues in opae_intel_max10.c
---
v5: implement APIs in rte_pmd_ifpga.c
---
 doc/api/doxy-api-index.md                 |   3 +-
 drivers/raw/ifpga/base/ifpga_api.c        |  26 +
 drivers/raw/ifpga/base/ifpga_fme.c        |   8 +
 drivers/raw/ifpga/base/ifpga_fme_rsu.c    | 428 +++++++++++++++
 drivers/raw/ifpga/base/ifpga_hw.h         |   1 +
 drivers/raw/ifpga/base/ifpga_sec_mgr.c    | 639 ++++++++++++++++++++++
 drivers/raw/ifpga/base/ifpga_sec_mgr.h    |  93 ++++
 drivers/raw/ifpga/base/meson.build        |   2 +
 drivers/raw/ifpga/base/opae_hw_api.c      |  59 ++
 drivers/raw/ifpga/base/opae_hw_api.h      |  10 +
 drivers/raw/ifpga/base/opae_intel_max10.c |  48 ++
 drivers/raw/ifpga/base/opae_intel_max10.h |  44 ++
 drivers/raw/ifpga/meson.build             |   4 +-
 drivers/raw/ifpga/rte_pmd_ifpga.c         | 163 ++++++
 drivers/raw/ifpga/rte_pmd_ifpga.h         | 132 +++++
 drivers/raw/ifpga/version.map             |  11 +
 16 files changed, 1669 insertions(+), 2 deletions(-)
 create mode 100644 drivers/raw/ifpga/base/ifpga_fme_rsu.c
 create mode 100644 drivers/raw/ifpga/base/ifpga_sec_mgr.c
 create mode 100644 drivers/raw/ifpga/base/ifpga_sec_mgr.h
 create mode 100644 drivers/raw/ifpga/rte_pmd_ifpga.c
 create mode 100644 drivers/raw/ifpga/rte_pmd_ifpga.h

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 748514e243..8a48af1042 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -56,7 +56,8 @@ The public API headers are grouped by topics:
   [dpaa2_qdma]         (@ref rte_pmd_dpaa2_qdma.h),
   [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h),
   [dlb]                (@ref rte_pmd_dlb.h),
-  [dlb2]               (@ref rte_pmd_dlb2.h)
+  [dlb2]               (@ref rte_pmd_dlb2.h),
+  [ifpga]              (@ref rte_pmd_ifpga.h)
 
 - **memory**:
   [memseg]             (@ref rte_memory.h),
diff --git a/drivers/raw/ifpga/base/ifpga_api.c b/drivers/raw/ifpga/base/ifpga_api.c
index 1ff57fa188..1aedf150bc 100644
--- a/drivers/raw/ifpga/base/ifpga_api.c
+++ b/drivers/raw/ifpga/base/ifpga_api.c
@@ -5,6 +5,7 @@
 #include "ifpga_api.h"
 #include "ifpga_enumerate.h"
 #include "ifpga_feature_dev.h"
+#include "ifpga_sec_mgr.h"
 
 #include "opae_hw_api.h"
 
@@ -228,11 +229,36 @@ static int ifpga_mgr_get_board_info(struct opae_manager *mgr,
 	return 0;
 }
 
+static int ifpga_mgr_update_flash(struct opae_manager *mgr, const char *image,
+	u64 *status)
+{
+	struct ifpga_fme_hw *fme = mgr->data;
+
+	return fpga_update_flash(fme, image, status);
+}
+
+static int ifpga_mgr_stop_flash_update(struct opae_manager *mgr, int force)
+{
+	struct ifpga_fme_hw *fme = mgr->data;
+
+	return fpga_stop_flash_update(fme, force);
+}
+
+static int ifpga_mgr_reload(struct opae_manager *mgr, int type, int page)
+{
+	struct ifpga_fme_hw *fme = mgr->data;
+
+	return fpga_reload(fme, type, page);
+}
+
 struct opae_manager_ops ifpga_mgr_ops = {
 	.flash = ifpga_mgr_flash,
 	.get_eth_group_region_info = ifpga_mgr_get_eth_group_region_info,
 	.get_sensor_value = ifpga_mgr_get_sensor_value,
 	.get_board_info = ifpga_mgr_get_board_info,
+	.update_flash = ifpga_mgr_update_flash,
+	.stop_flash_update = ifpga_mgr_stop_flash_update,
+	.reload = ifpga_mgr_reload,
 };
 
 static int ifpga_mgr_read_mac_rom(struct opae_manager *mgr, int offset,
diff --git a/drivers/raw/ifpga/base/ifpga_fme.c b/drivers/raw/ifpga/base/ifpga_fme.c
index f29ff3159b..34fd9a818e 100644
--- a/drivers/raw/ifpga/base/ifpga_fme.c
+++ b/drivers/raw/ifpga/base/ifpga_fme.c
@@ -7,6 +7,7 @@
 #include "opae_intel_max10.h"
 #include "opae_i2c.h"
 #include "opae_at24_eeprom.h"
+#include "ifpga_sec_mgr.h"
 
 #define PWR_THRESHOLD_MAX       0x7F
 
@@ -1152,6 +1153,12 @@ static int fme_nios_spi_init(struct ifpga_feature *feature)
 	if (spi_self_checking(max10))
 		goto spi_fail;
 
+	ret = init_sec_mgr(fme);
+	if (ret) {
+		dev_err(fme, "security manager init fail\n");
+		goto spi_fail;
+	}
+
 	return ret;
 
 spi_fail:
@@ -1165,6 +1172,7 @@ static void fme_nios_spi_uinit(struct ifpga_feature *feature)
 {
 	struct ifpga_fme_hw *fme = (struct ifpga_fme_hw *)feature->parent;
 
+	release_sec_mgr(fme);
 	if (fme->max10_dev)
 		intel_max10_device_remove(fme->max10_dev);
 }
diff --git a/drivers/raw/ifpga/base/ifpga_fme_rsu.c b/drivers/raw/ifpga/base/ifpga_fme_rsu.c
new file mode 100644
index 0000000000..28198abd78
--- /dev/null
+++ b/drivers/raw/ifpga/base/ifpga_fme_rsu.c
@@ -0,0 +1,428 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <fcntl.h>
+#include <signal.h>
+#include <unistd.h>
+#include "ifpga_sec_mgr.h"
+
+static struct ifpga_sec_mgr *sec_mgr;
+
+static void set_rsu_control(struct ifpga_sec_mgr *smgr, uint32_t ctrl)
+{
+	if (smgr && smgr->rsu_control)
+		*smgr->rsu_control = ctrl;
+}
+
+static uint32_t get_rsu_control(struct ifpga_sec_mgr *smgr)
+{
+	if (smgr && smgr->rsu_control)
+		return *smgr->rsu_control;
+	return 0;
+}
+
+static void set_rsu_status(struct ifpga_sec_mgr *smgr, uint32_t status,
+	uint32_t progress)
+{
+	if (smgr && smgr->rsu_status)
+		*smgr->rsu_status = IFPGA_RSU_STATUS(status, progress);
+}
+
+static void get_rsu_status(struct ifpga_sec_mgr *smgr, uint32_t *status,
+	uint32_t *progress)
+{
+	if (smgr && smgr->rsu_status) {
+		if (status)
+			*status = IFPGA_RSU_GET_STAT(*smgr->rsu_status);
+		if (progress)
+			*progress = IFPGA_RSU_GET_PROG(*smgr->rsu_status);
+	}
+}
+
+static void sig_handler(int sig, siginfo_t *info, void *data)
+{
+	(void)(info);
+	(void)(data);
+
+	switch (sig) {
+	case SIGINT:
+		if (sec_mgr) {
+			dev_info(sec_mgr, "Interrupt secure flash update"
+				" by keyboard\n");
+			set_rsu_control(sec_mgr, IFPGA_RSU_ABORT);
+		}
+		break;
+	default:
+		break;
+	}
+}
+
+static void log_time(time_t t, const char *msg)
+{
+	uint32_t h = 0;
+	uint32_t m = 0;
+	uint32_t s = 0;
+
+	if (t < 60) {
+		s = (uint32_t)t;
+	} else if (t < 3600) {
+		s = (uint32_t)(t % 60);
+		m = (uint32_t)(t / 60);
+	} else {
+		s = (uint32_t)(t % 60);
+		m = (uint32_t)((t % 3600) / 60);
+		h = (uint32_t)(t / 3600);
+	}
+	printf("%s - %02u:%02u:%02u\n", msg, h, m, s);
+}
+
+static int start_flash_update(struct ifpga_sec_mgr *smgr)
+{
+	if (!smgr)
+		return -ENODEV;
+
+	if (!smgr->ops || !smgr->ops->prepare)
+		return -EINVAL;
+
+	return smgr->ops->prepare(smgr);
+}
+
+static int write_flash_image(struct ifpga_sec_mgr *smgr, const char *image,
+	uint32_t offset)
+{
+	void *buf = NULL;
+	int retry = 0;
+	uint32_t length = 0;
+	uint32_t to_transfer = 0;
+	uint32_t one_percent = 0;
+	uint32_t prog = 0;
+	uint32_t old_prog = -1;
+	ssize_t read_size = 0;
+	int fd = -1;
+	int ret = 0;
+
+	if (!smgr)
+		return -ENODEV;
+
+	if (!smgr->ops || !smgr->ops->write_blk)
+		return -EINVAL;
+
+	fd = open(image, O_RDONLY);
+	if (fd < 0) {
+		dev_err(smgr,
+			"Failed to open \'%s\' for RD [e:%s]\n",
+			image, strerror(errno));
+		return -EIO;
+	}
+
+	buf = malloc(IFPGA_RSU_DATA_BLK_SIZE);
+	if (!buf) {
+		dev_err(smgr, "Failed to allocate memory for flash update\n");
+		close(fd);
+		return -ENOMEM;
+	}
+
+	length = smgr->rsu_length;
+	one_percent = length / 100;
+	do {
+		to_transfer = (length > IFPGA_RSU_DATA_BLK_SIZE) ?
+			IFPGA_RSU_DATA_BLK_SIZE : length;
+		lseek(fd, offset, SEEK_SET);
+		read_size = read(fd, buf, to_transfer);
+		if (read_size < 0) {
+			dev_err(smgr, "Failed to read from \'%s\' [e:%s]\n",
+				image, strerror(errno));
+			ret = -EIO;
+			goto end;
+		}
+		if ((uint32_t)read_size != to_transfer) {
+			dev_err(smgr,
+				"Read length %zd is not expected [e:%u]\n",
+				read_size, to_transfer);
+			ret = -EIO;
+			goto end;
+		}
+
+		retry = 0;
+		do {
+			if (get_rsu_control(smgr) == IFPGA_RSU_ABORT) {
+				ret = -EAGAIN;
+				goto end;
+			}
+			ret = smgr->ops->write_blk(smgr, buf, offset,
+				to_transfer);
+			if (ret == 0)
+				break;
+			sleep(1);
+		} while (++retry <= IFPGA_RSU_WRITE_RETRY);
+		if (retry > IFPGA_RSU_WRITE_RETRY) {
+			dev_err(smgr, "Failed to write to staging area 0x%x\n",
+				offset);
+			ret = -EAGAIN;
+			goto end;
+		}
+
+		length -= to_transfer;
+		offset += to_transfer;
+		prog = offset / one_percent;
+		if (prog != old_prog) {
+			printf("\r%d%%", prog);
+			fflush(stdout);
+			set_rsu_status(smgr, IFPGA_RSU_READY, prog);
+			old_prog = prog;
+		}
+	} while (length > 0);
+	set_rsu_status(smgr, IFPGA_RSU_READY, 100);
+	printf("\n");
+
+end:
+	free(buf);
+	close(fd);
+	return ret;
+}
+
+static int apply_flash_update(struct ifpga_sec_mgr *smgr)
+{
+	uint32_t one_percent = 0;
+	uint32_t one_percent_time = 0;
+	uint32_t prog = 0;
+	uint32_t old_prog = -1;
+	uint32_t copy_time = 0;
+	int ret = 0;
+
+	if (!smgr)
+		return -ENODEV;
+
+	if (!smgr->ops || !smgr->ops->write_done || !smgr->ops->check_complete)
+		return -EINVAL;
+
+	if (smgr->ops->write_done(smgr) < 0) {
+		dev_err(smgr, "Failed to apply flash update\n");
+		return -EAGAIN;
+	}
+
+	one_percent = (smgr->rsu_length + 99) / 100;
+	if (smgr->copy_speed == 0)   /* avoid zero divide fault */
+		smgr->copy_speed = 1;
+	one_percent_time = (one_percent + smgr->copy_speed - 1) /
+		smgr->copy_speed;
+	if (one_percent_time == 0)   /* avoid zero divide fault */
+		one_percent_time = 1;
+
+	do {
+		ret = smgr->ops->check_complete(smgr);
+		if (ret != -EAGAIN)
+			break;
+		sleep(1);
+		copy_time += 1;
+		prog = copy_time / one_percent_time;
+		if (prog >= 100)
+			prog = 99;
+		if (prog != old_prog) {
+			printf("\r%d%%", prog);
+			fflush(stdout);
+			set_rsu_status(smgr, IFPGA_RSU_COPYING, prog);
+			old_prog = prog;
+		}
+	} while (true);
+
+	if (ret < 0) {
+		printf("\n");
+		dev_err(smgr, "Failed to complete secure flash update\n");
+	} else {
+		printf("\r100%%\n");
+		set_rsu_status(smgr, IFPGA_RSU_COPYING, 100);
+	}
+
+	return ret;
+}
+
+static int secure_update_cancel(struct ifpga_sec_mgr *smgr)
+{
+	if (!smgr)
+		return -ENODEV;
+
+	if (!smgr->ops || !smgr->ops->cancel)
+		return -EINVAL;
+
+	return smgr->ops->cancel(smgr);
+}
+
+static int secure_update_status(struct ifpga_sec_mgr *smgr, uint64_t *status)
+{
+	if (!smgr)
+		return -ENODEV;
+
+	if (!smgr->ops || !smgr->ops->get_hw_errinfo)
+		return -EINVAL;
+
+	if (status)
+		*status = smgr->ops->get_hw_errinfo(smgr);
+
+	return 0;
+}
+
+int fpga_update_flash(struct ifpga_fme_hw *fme, const char *image,
+	uint64_t *status)
+{
+	struct ifpga_hw *hw = NULL;
+	struct ifpga_sec_mgr *smgr = NULL;
+	uint32_t rsu_stat = 0;
+	int fd = -1;
+	struct sigaction old_sigint_action;
+	struct sigaction sa;
+	time_t start;
+	int ret = 0;
+
+	if (!fme || !image || !status) {
+		dev_err(fme, "Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	hw = (struct ifpga_hw *)fme->parent;
+	if (!hw) {
+		dev_err(fme, "Parent of FME not found\n");
+		return -ENODEV;
+	}
+
+	smgr = (struct ifpga_sec_mgr *)fme->sec_mgr;
+	if (!smgr || !smgr->max10_dev) {
+		dev_err(smgr, "Security manager not initialized\n");
+		return -ENODEV;
+	}
+
+	opae_adapter_lock(hw->adapter, -1);
+	get_rsu_status(smgr, &rsu_stat, NULL);
+	if (rsu_stat != IFPGA_RSU_IDLE) {
+		opae_adapter_unlock(hw->adapter);
+		if (rsu_stat == IFPGA_RSU_REBOOT)
+			dev_info(smgr, "Reboot is in progress\n");
+		else
+			dev_info(smgr, "Update is in progress\n");
+		return -EAGAIN;
+	}
+	set_rsu_control(smgr, 0);
+	set_rsu_status(smgr, IFPGA_RSU_PREPARE, 0);
+	opae_adapter_unlock(hw->adapter);
+
+	fd = open(image, O_RDONLY);
+	if (fd < 0) {
+		dev_err(smgr,
+			"Failed to open \'%s\' for RD [e:%s]\n",
+			image, strerror(errno));
+		return -EIO;
+	}
+	smgr->rsu_length = lseek(fd, 0, SEEK_END);
+	close(fd);
+
+	if (smgr->max10_dev->staging_area_size < smgr->rsu_length) {
+		dev_err(dev, "Size of staging area is small than image length "
+			"[%u<%u]\n", smgr->max10_dev->staging_area_size,
+			smgr->rsu_length);
+		return -EINVAL;
+	}
+
+	printf("Updating from file \'%s\' with size %u\n",
+		image, smgr->rsu_length);
+
+	sec_mgr = smgr;
+	memset(&sa, 0, sizeof(struct sigaction));
+	sa.sa_flags = SA_SIGINFO | SA_RESETHAND;
+	sa.sa_sigaction = sig_handler;
+	ret = sigaction(SIGINT, &sa, &old_sigint_action);
+	if (ret < 0) {
+		dev_warn(dev, "Failed to register signal handler"
+			" [e:%d]\n", ret);
+		sec_mgr = NULL;
+	}
+
+	start = time(NULL);
+	log_time(time(NULL) - start, "Starting secure flash update");
+	ret = start_flash_update(smgr);
+	if (ret < 0)
+		goto end;
+
+	set_rsu_status(smgr, IFPGA_RSU_READY, 0);
+	log_time(time(NULL) - start, "Writing to staging area");
+	ret = write_flash_image(smgr, image, 0);
+	if (ret < 0)
+		goto end;
+
+	set_rsu_status(smgr, IFPGA_RSU_COPYING, 0);
+	log_time(time(NULL) - start, "Applying secure flash update");
+	ret = apply_flash_update(smgr);
+
+end:
+	if (sec_mgr) {
+		sec_mgr = NULL;
+		if (sigaction(SIGINT, &old_sigint_action, NULL) < 0)
+			dev_err(smgr, "Failed to unregister signal handler\n");
+	}
+
+	secure_update_status(smgr, status);
+	if (ret < 0) {
+		log_time(time(NULL) - start, "Secure flash update ERROR");
+		if (ret == -EAGAIN)
+			secure_update_cancel(smgr);
+	} else {
+		log_time(time(NULL) - start, "Secure flash update OK");
+	}
+	set_rsu_status(smgr, IFPGA_RSU_IDLE, 0);
+
+	return ret;
+}
+
+int fpga_stop_flash_update(struct ifpga_fme_hw *fme, int force)
+{
+	struct ifpga_sec_mgr *smgr = NULL;
+	uint32_t status = 0;
+	int retry = IFPGA_RSU_CANCEL_RETRY;
+	int ret = 0;
+
+	if (!fme) {
+		dev_err(fme, "Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+	smgr = (struct ifpga_sec_mgr *)fme->sec_mgr;
+
+	get_rsu_status(smgr, &status, NULL);
+	if (status != IFPGA_RSU_IDLE) {
+		dev_info(smgr, "Cancel secure flash update\n");
+		set_rsu_control(smgr, IFPGA_RSU_ABORT);
+	}
+
+	if (force) {
+		sleep(2);
+		do {
+			get_rsu_status(smgr, &status, NULL);
+			if (status == IFPGA_RSU_IDLE)
+				break;
+			if (secure_update_cancel(smgr) == 0)
+				set_rsu_status(smgr, IFPGA_RSU_IDLE, 0);
+			sleep(1);
+		} while (--retry > 0);
+		if (retry <= 0) {
+			dev_err(smgr, "Failed to stop flash update\n");
+			ret = -EAGAIN;
+		}
+	}
+
+	return ret;
+}
+
+int fpga_reload(struct ifpga_fme_hw *fme, int type, int page)
+{
+	struct ifpga_sec_mgr *smgr = NULL;
+
+	if (!fme) {
+		dev_err(fme, "Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+	smgr = (struct ifpga_sec_mgr *)fme->sec_mgr;
+
+	if (!smgr || !smgr->ops || !smgr->ops->reload)
+		return -EINVAL;
+
+	return smgr->ops->reload(smgr, type, page);
+}
diff --git a/drivers/raw/ifpga/base/ifpga_hw.h b/drivers/raw/ifpga/base/ifpga_hw.h
index 7c3307fe77..ed5edc6016 100644
--- a/drivers/raw/ifpga/base/ifpga_hw.h
+++ b/drivers/raw/ifpga/base/ifpga_hw.h
@@ -91,6 +91,7 @@ struct ifpga_fme_hw {
 	struct opae_board_info board_info;
 	int nums_eth_dev;
 	unsigned int nums_acc_region;
+	void *sec_mgr;
 };
 
 enum ifpga_port_state {
diff --git a/drivers/raw/ifpga/base/ifpga_sec_mgr.c b/drivers/raw/ifpga/base/ifpga_sec_mgr.c
new file mode 100644
index 0000000000..4cf1db3049
--- /dev/null
+++ b/drivers/raw/ifpga/base/ifpga_sec_mgr.c
@@ -0,0 +1,639 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <fcntl.h>
+#include <signal.h>
+#include <unistd.h>
+#include "ifpga_sec_mgr.h"
+
+
+static const char * const rsu_prog[] = {"IDLE", "PREPARING", "SLEEPING",
+	"READY", "AUTHENTICATING", "COPYING", "CANCELLATION", "PROGRAMMING_KEY",
+	"DONE", "PKVL_DONE"};
+static const char * const rsu_statl[] = {"NORMAL", "TIMEOUT", "AUTH_FAIL",
+	"COPY_FAIL", "FATAL", "PKVL_REJECT", "NON_INCR", "ERASE_FAIL",
+	"WEAROUT"};
+static const char * const rsu_stath[] = {"NIOS_OK", "USER_OK", "FACTORY_OK",
+	"USER_FAIL", "FACTORY_FAIL", "NIOS_FLASH_ERR", "FPGA_FLASH_ERR"};
+
+static const char *rsu_progress_name(uint32_t prog)
+{
+	if (prog > SEC_PROGRESS_PKVL_PROM_DONE)
+		return "UNKNOWN";
+	else
+		return rsu_prog[prog];
+}
+
+static const char *rsu_status_name(uint32_t stat)
+{
+	if (stat >= SEC_STATUS_NIOS_OK) {
+		if (stat > SEC_STATUS_FPGA_FLASH_ERR)
+			return "UNKNOWN";
+		else
+			return rsu_stath[stat-SEC_STATUS_NIOS_OK];
+	} else {
+		if (stat > SEC_STATUS_WEAROUT)
+			return "UNKNOWN";
+		else
+			return rsu_statl[stat];
+	}
+}
+
+static bool secure_start_done(uint32_t doorbell)
+{
+	return (SEC_STATUS_G(doorbell) == SEC_STATUS_ERASE_FAIL ||
+		SEC_STATUS_G(doorbell) == SEC_STATUS_WEAROUT ||
+		(SEC_PROGRESS_G(doorbell) != SEC_PROGRESS_IDLE &&
+		SEC_PROGRESS_G(doorbell) != SEC_PROGRESS_RSU_DONE));
+}
+
+static bool secure_prog_ready(uint32_t doorbell)
+{
+	return (SEC_PROGRESS_G(doorbell) != SEC_PROGRESS_READY);
+}
+
+static int poll_timeout(struct intel_max10_device *dev, uint32_t offset,
+	bool (*cond)(uint32_t), uint32_t interval_ms, uint32_t timeout_ms)
+{
+	uint32_t val = 0;
+	int ret = 0;
+
+	for (;;) {
+		ret = max10_sys_read(dev, offset, &val);
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to read max10 register 0x%x [e:%d]\n",
+				offset, ret);
+			break;
+		}
+
+		if (cond(val)) {
+			dev_debug(dev,
+				"Read 0x%08x from max10 register 0x%x "
+				"[poll success]\n", val, offset);
+			ret = 0;
+			break;
+		}
+		if (timeout_ms > interval_ms)
+			timeout_ms -= interval_ms;
+		else
+			timeout_ms = 0;
+		if (timeout_ms == 0) {
+			dev_debug(dev,
+				"Read 0x%08x from max10 register 0x%x "
+				"[poll timeout]\n", val, offset);
+			ret = -ETIMEDOUT;
+			break;
+		}
+		msleep(interval_ms);
+	}
+
+	return ret;
+}
+
+static int n3000_secure_update_start(struct intel_max10_device *dev)
+{
+	uint32_t doorbell = 0;
+	uint32_t prog = 0;
+	uint32_t status = 0;
+	int ret = 0;
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	prog = SEC_PROGRESS_G(doorbell);
+	if ((prog != SEC_PROGRESS_IDLE) && (prog != SEC_PROGRESS_RSU_DONE)) {
+		dev_debug(dev, "Current RSU progress is %s\n",
+			rsu_progress_name(prog));
+		return -EBUSY;
+	}
+
+	ret = max10_sys_update_bits(dev, MAX10_DOORBELL,
+		RSU_REQUEST | HOST_STATUS, RSU_REQUEST);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to updt max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	ret = poll_timeout(dev, MAX10_DOORBELL, secure_start_done,
+		IFPGA_SEC_START_INTERVAL_MS, IFPGA_SEC_START_TIMEOUT_MS);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to poll max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	status = SEC_STATUS_G(doorbell);
+	if (status == SEC_STATUS_WEAROUT)
+		return -EAGAIN;
+
+	if (status == SEC_STATUS_ERASE_FAIL)
+		return -EIO;
+
+	return 0;
+}
+
+static int n3000_cancel(struct ifpga_sec_mgr *smgr)
+{
+	struct intel_max10_device *dev = NULL;
+	uint32_t doorbell = 0;
+	uint32_t prog = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+	dev = (struct intel_max10_device *)smgr->max10_dev;
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	prog = SEC_PROGRESS_G(doorbell);
+	if (prog == SEC_PROGRESS_IDLE)
+		return 0;
+	if (prog != SEC_PROGRESS_READY)
+		return -EBUSY;
+
+	return max10_sys_update_bits(dev, MAX10_DOORBELL, HOST_STATUS,
+		HOST_STATUS_S(HOST_STATUS_ABORT_RSU));
+}
+
+static int n3000_prepare(struct ifpga_sec_mgr *smgr)
+{
+	struct intel_max10_device *dev = NULL;
+	int retry = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+	dev = (struct intel_max10_device *)smgr->max10_dev;
+
+	ret = n3000_secure_update_start(dev);
+	if (ret == -EBUSY)
+		n3000_cancel(smgr);
+
+	while (ret) {
+		if (++retry > IFPGA_RSU_START_RETRY)
+			break;
+		msleep(1000);
+		ret = n3000_secure_update_start(dev);
+	}
+	if (retry > IFPGA_RSU_START_RETRY) {
+		dev_err(dev, "Failed to start secure flash update\n");
+		ret = -EAGAIN;
+	}
+
+	return ret;
+}
+
+static int n3000_bulk_write(struct intel_max10_device *dev, uint32_t addr,
+	char *buf, uint32_t len)
+{
+	uint32_t i = 0;
+	uint32_t n = 0;
+	uint32_t v = 0;
+	uint32_t p = 0;
+	int ret = 0;
+
+	if (len & 0x3) {
+		dev_err(dev,
+			"Length of data block is not 4 bytes aligned [e:%u]\n",
+			len);
+		return -EINVAL;
+	}
+
+	n = len >> 2;
+	for (i = 0; i < n; i++) {
+		p = i << 2;
+		v = *(uint32_t *)(buf + p);
+		ret = max10_reg_write(dev, addr + p, v);
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to write to staging area 0x%08x [e:%d]\n",
+				addr + p, ret);
+			return ret;
+		}
+		usleep(1);
+	}
+
+	return 0;
+}
+
+static int n3000_write_blk(struct ifpga_sec_mgr *smgr, char *buf,
+	uint32_t offset, uint32_t len)
+{
+	struct intel_max10_device *dev = NULL;
+	uint32_t doorbell = 0;
+	uint32_t prog = 0;
+	uint32_t m = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+	dev = (struct intel_max10_device *)smgr->max10_dev;
+
+	if (offset + len > dev->staging_area_size) {
+		dev_err(dev,
+			"Write position would be out of staging area [e:%u]\n",
+			dev->staging_area_size);
+		return -ENOMEM;
+	}
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	prog = SEC_PROGRESS_G(doorbell);
+	if (prog == SEC_PROGRESS_PREPARE)
+		return -EAGAIN;
+	else if (prog != SEC_PROGRESS_READY)
+		return -EBUSY;
+
+	m = len & 0x3;
+	if (m != 0)
+		len += 4 - m;   /* make length to 4 bytes align */
+
+	return n3000_bulk_write(dev, dev->staging_area_base + offset, buf, len);
+}
+
+static int n3000_write_done(struct ifpga_sec_mgr *smgr)
+{
+	struct intel_max10_device *dev = NULL;
+	uint32_t doorbell = 0;
+	uint32_t prog = 0;
+	uint32_t status = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+	dev = (struct intel_max10_device *)smgr->max10_dev;
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	prog = SEC_PROGRESS_G(doorbell);
+	if (prog != SEC_PROGRESS_READY)
+		return -EBUSY;
+
+	ret = max10_sys_update_bits(dev, MAX10_DOORBELL, HOST_STATUS,
+		HOST_STATUS_S(HOST_STATUS_WRITE_DONE));
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to update max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	ret = poll_timeout(dev, MAX10_DOORBELL, secure_prog_ready,
+		IFPGA_NIOS_HANDSHAKE_INTERVAL_MS,
+		IFPGA_NIOS_HANDSHAKE_TIMEOUT_MS);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to poll max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	status = SEC_STATUS_G(doorbell);
+	switch (status) {
+	case SEC_STATUS_NORMAL:
+	case SEC_STATUS_NIOS_OK:
+	case SEC_STATUS_USER_OK:
+	case SEC_STATUS_FACTORY_OK:
+		ret = 0;
+		break;
+	default:
+		ret = -EIO;
+		break;
+	}
+
+	return ret;
+}
+
+static int n3000_check_complete(struct ifpga_sec_mgr *smgr)
+{
+	struct intel_max10_device *dev = NULL;
+	uint32_t doorbell = 0;
+	uint32_t status = 0;
+	uint32_t prog = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+	dev = (struct intel_max10_device *)smgr->max10_dev;
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	status = SEC_STATUS_G(doorbell);
+	switch (status) {
+	case SEC_STATUS_NORMAL:
+	case SEC_STATUS_NIOS_OK:
+	case SEC_STATUS_USER_OK:
+	case SEC_STATUS_FACTORY_OK:
+	case SEC_STATUS_WEAROUT:
+		break;
+	default:
+		return -EIO;
+	}
+
+	prog = SEC_PROGRESS_G(doorbell);
+	switch (prog) {
+	case SEC_PROGRESS_IDLE:
+	case SEC_PROGRESS_RSU_DONE:
+		return 0;
+	case SEC_PROGRESS_AUTHENTICATING:
+	case SEC_PROGRESS_COPYING:
+	case SEC_PROGRESS_UPDATE_CANCEL:
+	case SEC_PROGRESS_PROGRAM_KEY_HASH:
+		return -EAGAIN;
+	case SEC_PROGRESS_PREPARE:
+	case SEC_PROGRESS_READY:
+		return -EBUSY;
+	default:
+		return -EIO;
+	}
+
+	return 0;
+}
+
+static int n3000_reload_fpga(struct intel_max10_device *dev, int page)
+{
+	int ret = 0;
+
+	dev_info(dev, "Reload FPGA\n");
+
+	if (!dev || ((page != 0) && (page != 1))) {
+		dev_err(dev, "Input parameter of %s is invalid\n", __func__);
+		ret = -EINVAL;
+		goto end;
+	}
+
+	if (dev->flags & MAX10_FLAGS_SECURE) {
+		ret = max10_sys_update_bits(dev, FPGA_RECONF_REG,
+			SFPGA_RP_LOAD, 0);
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to update max10 reconfig register [e:%d]\n",
+				ret);
+			goto end;
+		}
+		ret = max10_sys_update_bits(dev, FPGA_RECONF_REG,
+			SFPGA_RP_LOAD | SFPGA_RECONF_PAGE,
+			SFPGA_RP_LOAD | SFPGA_PAGE(page));
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to update max10 reconfig register [e:%d]\n",
+				ret);
+			goto end;
+		}
+	} else {
+		ret = max10_sys_update_bits(dev, RSU_REG, FPGA_RP_LOAD, 0);
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to update max10 rsu register [e:%d]\n",
+				ret);
+			goto end;
+		}
+		ret = max10_sys_update_bits(dev, RSU_REG,
+			FPGA_RP_LOAD | FPGA_RECONF_PAGE,
+			FPGA_RP_LOAD | FPGA_PAGE(page));
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to update max10 rsu register [e:%d]\n",
+				ret);
+			goto end;
+		}
+	}
+
+	ret = max10_sys_update_bits(dev, FPGA_RECONF_REG, COUNTDOWN_START, 0);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to update max10 reconfig register [e:%d]\n",
+			ret);
+		goto end;
+	}
+
+	ret = max10_sys_update_bits(dev, FPGA_RECONF_REG, COUNTDOWN_START,
+		COUNTDOWN_START);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to update max10 reconfig register [e:%d]\n",
+			ret);
+	}
+end:
+	if (ret < 0)
+		dev_err(dev, "Failed to reload FPGA\n");
+
+	return ret;
+}
+
+static int n3000_reload_bmc(struct intel_max10_device *dev, int page)
+{
+	uint32_t val = 0;
+	int ret = 0;
+
+	dev_info(dev, "Reload BMC\n");
+
+	if (!dev || ((page != 0) && (page != 1))) {
+		dev_err(dev, "Input parameter of %s is invalid\n", __func__);
+		ret = -EINVAL;
+		goto end;
+	}
+
+	if (dev->flags & MAX10_FLAGS_SECURE) {
+		ret = max10_sys_update_bits(dev, MAX10_DOORBELL,
+			CONFIG_SEL | REBOOT_REQ,
+			CONFIG_SEL_S(page) | REBOOT_REQ);
+	} else {
+		val = (page == 0) ? 0x1 : 0x3;
+		ret = max10_reg_write(dev, IFPGA_DUAL_CFG_CTRL1, val);
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to write to dual config1 register [e:%d]\n",
+				ret);
+			goto end;
+		}
+
+		ret = max10_reg_write(dev, IFPGA_DUAL_CFG_CTRL0, 0x1);
+		if (ret < 0) {
+			if (ret == -EIO) {
+				ret = 0;
+				goto end;
+			}
+			dev_err(dev,
+				"Failed to write to dual config0 register [e:%d]\n",
+				ret);
+		}
+	}
+
+end:
+	if (ret < 0)
+		dev_err(dev, "Failed to reload BMC\n");
+
+	return ret;
+}
+
+static int n3000_reload(struct ifpga_sec_mgr *smgr, int type, int page)
+{
+	int psel = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+
+	if (type == IFPGA_BOOT_TYPE_FPGA) {
+		psel = (page == IFPGA_BOOT_PAGE_FACTORY ? 0 : 1);
+		ret = n3000_reload_fpga(smgr->max10_dev, psel);
+	} else if (type == IFPGA_BOOT_TYPE_BMC) {
+		psel = (page == IFPGA_BOOT_PAGE_FACTORY ? 1 : 0);
+		ret = n3000_reload_bmc(smgr->max10_dev, psel);
+	} else {
+		ret = -EINVAL;
+	}
+
+	return ret;
+}
+
+static uint64_t n3000_get_hw_errinfo(struct ifpga_sec_mgr *smgr)
+{
+	struct intel_max10_device *dev = NULL;
+	uint32_t doorbell = 0;
+	uint32_t stat = 0;
+	uint32_t prog = 0;
+	uint32_t auth_result = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+	dev = (struct intel_max10_device *)smgr->max10_dev;
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev, "Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return -1;
+	}
+	stat = SEC_STATUS_G(doorbell);
+	prog = SEC_PROGRESS_G(doorbell);
+	dev_debug(dev, "Current RSU status is %s, progress is %s\n",
+		rsu_status_name(stat), rsu_progress_name(prog));
+
+	ret = max10_sys_read(dev, MAX10_AUTH_RESULT, &auth_result);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read authenticate result register [e:%d]\n",
+			ret);
+		return -1;
+	}
+
+	return (uint64_t)doorbell << 32 | (uint64_t)auth_result;
+}
+
+static const struct ifpga_sec_ops n3000_sec_ops = {
+	.prepare = n3000_prepare,
+	.write_blk = n3000_write_blk,
+	.write_done = n3000_write_done,
+	.check_complete = n3000_check_complete,
+	.reload = n3000_reload,
+	.cancel = n3000_cancel,
+	.cleanup = NULL,
+	.get_hw_errinfo = n3000_get_hw_errinfo,
+};
+
+int init_sec_mgr(struct ifpga_fme_hw *fme)
+{
+	struct ifpga_hw *hw = NULL;
+	opae_share_data *sd = NULL;
+	struct ifpga_sec_mgr *smgr = NULL;
+
+	if (!fme || !fme->max10_dev)
+		return -ENODEV;
+
+	smgr = (struct ifpga_sec_mgr *)malloc(sizeof(*smgr));
+	if (!smgr) {
+		dev_err(NULL, "Failed to allocate memory for security manager\n");
+		return -ENOMEM;
+	}
+	fme->sec_mgr = smgr;
+
+	hw = (struct ifpga_hw *)fme->parent;
+	if (hw && hw->adapter && hw->adapter->shm.ptr) {
+		sd = (opae_share_data *)hw->adapter->shm.ptr;
+		smgr->rsu_control = &sd->rsu_ctrl;
+		smgr->rsu_status = &sd->rsu_stat;
+	} else {
+		smgr->rsu_control = NULL;
+		smgr->rsu_status = NULL;
+	}
+
+	if ((hw->pci_data->device_id == IFPGA_N3000_DID) &&
+		(hw->pci_data->vendor_id == IFPGA_N3000_VID)) {
+		smgr->ops = &n3000_sec_ops;
+		smgr->copy_speed = IFPGA_N3000_COPY_SPEED;
+	} else {
+		dev_err(NULL, "No operation for security manager\n");
+		smgr->ops = NULL;
+	}
+
+	smgr->fme = fme;
+	smgr->max10_dev = fme->max10_dev;
+
+	return 0;
+}
+
+void release_sec_mgr(struct ifpga_fme_hw *fme)
+{
+	struct ifpga_sec_mgr *smgr = NULL;
+
+	if (fme) {
+		smgr = (struct ifpga_sec_mgr *)fme->sec_mgr;
+		if (smgr) {
+			fme->sec_mgr = NULL;
+			free(smgr);
+		}
+	}
+}
diff --git a/drivers/raw/ifpga/base/ifpga_sec_mgr.h b/drivers/raw/ifpga/base/ifpga_sec_mgr.h
new file mode 100644
index 0000000000..fbeba561f4
--- /dev/null
+++ b/drivers/raw/ifpga/base/ifpga_sec_mgr.h
@@ -0,0 +1,93 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _IFPGA_FME_RSU_H_
+#define _IFPGA_FME_RSU_H_
+
+
+#include "ifpga_hw.h"
+
+#define IFPGA_N3000_VID     0x8086
+#define IFPGA_N3000_DID     0x0b30
+
+#define IFPGA_BOOT_TYPE_FPGA     0
+#define IFPGA_BOOT_TYPE_BMC      1
+
+#define IFPGA_BOOT_PAGE_FACTORY  0
+#define IFPGA_BOOT_PAGE_USER     1
+
+#define IFPGA_RSU_DATA_BLK_SIZE  32768
+#define IFPGA_RSU_START_RETRY    120
+#define IFPGA_RSU_WRITE_RETRY    10
+#define IFPGA_RSU_CANCEL_RETRY   30
+
+#define IFPGA_N3000_COPY_SPEED   42700
+
+/* status */
+#define IFPGA_RSU_IDLE       0
+#define IFPGA_RSU_PREPARE    1
+#define IFPGA_RSU_READY      2
+#define IFPGA_RSU_COPYING    3
+#define IFPGA_RSU_REBOOT     4
+
+#define IFPGA_RSU_GET_STAT(v)  (((v) >> 16) & 0xffff)
+#define IFPGA_RSU_GET_PROG(v)  ((v) & 0xffff)
+#define IFPGA_RSU_STATUS(s, p) ((((s) << 16) & 0xffff0000) | ((p) & 0xffff))
+
+/* control */
+#define IFPGA_RSU_ABORT      1
+
+#define IFPGA_DUAL_CFG_CTRL0     0x200020
+#define IFPGA_DUAL_CFG_CTRL1     0x200024
+
+#define IFPGA_SEC_START_INTERVAL_MS       100
+#define IFPGA_SEC_START_TIMEOUT_MS        20000
+#define IFPGA_NIOS_HANDSHAKE_INTERVAL_MS  100
+#define IFPGA_NIOS_HANDSHAKE_TIMEOUT_MS   5000
+
+#define IFPGA_RSU_ERR_HW_ERROR		-1
+#define IFPGA_RSU_ERR_TIMEOUT		-2
+#define IFPGA_RSU_ERR_CANCELED		-3
+#define IFPGA_RSU_ERR_BUSY			-4
+#define IFPGA_RSU_ERR_INVALID_SIZE	-5
+#define IFPGA_RSU_ERR_RW_ERROR		-6
+#define IFPGA_RSU_ERR_WEAROUT		-7
+#define IFPGA_RSU_ERR_FILE_READ		-8
+
+struct ifpga_sec_mgr;
+
+struct ifpga_sec_ops {
+	int (*prepare)(struct ifpga_sec_mgr *smgr);
+	int (*write_blk)(struct ifpga_sec_mgr *smgr, char *buf, uint32_t offset,
+		uint32_t size);
+	int (*write_done)(struct ifpga_sec_mgr *smgr);
+	int (*check_complete)(struct ifpga_sec_mgr *smgr);
+	int (*reload)(struct ifpga_sec_mgr *smgr, int type, int page);
+	int (*cancel)(struct ifpga_sec_mgr *smgr);
+	void (*cleanup)(struct ifpga_sec_mgr *smgr);
+	u64 (*get_hw_errinfo)(struct ifpga_sec_mgr *smgr);
+};
+
+struct ifpga_sec_mgr {
+	struct ifpga_fme_hw *fme;
+	struct intel_max10_device *max10_dev;
+	unsigned int rsu_length;
+	/* number of bytes that copied from staging area to working area
+	 * in one second, which is calculated by experiment
+	 */
+	unsigned int copy_speed;
+	unsigned int *rsu_control;
+	unsigned int *rsu_status;
+	const struct ifpga_sec_ops *ops;
+};
+
+int init_sec_mgr(struct ifpga_fme_hw *fme);
+void release_sec_mgr(struct ifpga_fme_hw *fme);
+int fpga_update_flash(struct ifpga_fme_hw *fme, const char *image,
+	uint64_t *status);
+int fpga_stop_flash_update(struct ifpga_fme_hw *fme, int force);
+int fpga_reload(struct ifpga_fme_hw *fme, int type, int page);
+
+
+#endif /* _IFPGA_FME_RSU_H_ */
diff --git a/drivers/raw/ifpga/base/meson.build b/drivers/raw/ifpga/base/meson.build
index da2d6e33ca..3549afafa1 100644
--- a/drivers/raw/ifpga/base/meson.build
+++ b/drivers/raw/ifpga/base/meson.build
@@ -12,6 +12,8 @@ sources = [
 	'ifpga_port.c',
 	'ifpga_port_error.c',
 	'ifpga_fme_pr.c',
+	'ifpga_fme_rsu.c',
+	'ifpga_sec_mgr.c',
 	'opae_hw_api.c',
 	'opae_ifpga_hw_api.c',
 	'opae_debug.c',
diff --git a/drivers/raw/ifpga/base/opae_hw_api.c b/drivers/raw/ifpga/base/opae_hw_api.c
index d5cd5fe608..86ad88f720 100644
--- a/drivers/raw/ifpga/base/opae_hw_api.c
+++ b/drivers/raw/ifpga/base/opae_hw_api.c
@@ -470,6 +470,8 @@ static void opae_adapter_shm_init(struct opae_adapter *adapter)
 	opae_mutex_init(&sd->i2c_mutex);
 	sd->ref_cnt = 0;
 	sd->dtb_size = SHM_BLK_SIZE;
+	sd->rsu_ctrl = 0;
+	sd->rsu_stat = 0;
 }
 
 static void *opae_adapter_shm_alloc(struct opae_adapter *adapter)
@@ -964,3 +966,60 @@ opae_mgr_get_board_info(struct opae_manager *mgr,
 
 	return -ENOENT;
 }
+
+/**
+ * opae_mgr_update_flash -  update image in flash.
+ * @mgr: targeted manager
+ * @image: name of image file
+ * @status: status of update
+ *
+ * Return: 0 on success, otherwise error code.
+ */
+int opae_mgr_update_flash(struct opae_manager *mgr, const char *image,
+	uint64_t *status)
+{
+	if (!mgr)
+		return -EINVAL;
+
+	if (mgr->ops && mgr->ops->update_flash)
+		return mgr->ops->update_flash(mgr, image, status);
+
+	return -ENOENT;
+}
+
+/**
+ * opae_stop_flash_update -  stop flash update.
+ * @mgr: targeted manager
+ * @force: make sure the update process is stopped
+ *
+ * Return: 0 on success, otherwise error code.
+ */
+int opae_mgr_stop_flash_update(struct opae_manager *mgr, int force)
+{
+	if (!mgr)
+		return -EINVAL;
+
+	if (mgr->ops && mgr->ops->stop_flash_update)
+		return mgr->ops->stop_flash_update(mgr, force);
+
+	return -ENOENT;
+}
+
+/**
+ * opae_mgr_reload -  reload FPGA.
+ * @mgr: targeted manager
+ * @type: FPGA type
+ * @page: reload from which page
+ *
+ * Return: 0 on success, otherwise error code.
+ */
+int opae_mgr_reload(struct opae_manager *mgr, int type, int page)
+{
+	if (!mgr)
+		return -EINVAL;
+
+	if (mgr->ops && mgr->ops->reload)
+		return mgr->ops->reload(mgr, type, page);
+
+	return -ENOENT;
+}
diff --git a/drivers/raw/ifpga/base/opae_hw_api.h b/drivers/raw/ifpga/base/opae_hw_api.h
index e99ee4564c..91d26d9b5b 100644
--- a/drivers/raw/ifpga/base/opae_hw_api.h
+++ b/drivers/raw/ifpga/base/opae_hw_api.h
@@ -55,6 +55,10 @@ struct opae_manager_ops {
 			unsigned int *value);
 	int (*get_board_info)(struct opae_manager *mgr,
 			struct opae_board_info **info);
+	int (*update_flash)(struct opae_manager *mgr, const char *image,
+			u64 *status);
+	int (*stop_flash_update)(struct opae_manager *mgr, int force);
+	int (*reload)(struct opae_manager *mgr, int type, int page);
 };
 
 /* networking management ops in FME */
@@ -276,6 +280,8 @@ typedef struct {
 			pthread_mutex_t i2c_mutex;
 			u32 ref_cnt;    /* reference count of shared memory */
 			u32 dtb_size;   /* actual length of DTB data in byte */
+			u32 rsu_ctrl;   /* used to control rsu */
+			u32 rsu_stat;   /* used to report status for rsu */
 		};
 	};
 	u8 dtb[SHM_BLK_SIZE];   /* DTB data */
@@ -354,4 +360,8 @@ int opae_manager_eth_group_read_reg(struct opae_manager *mgr, u8 group_id,
 		u8 type, u8 index, u16 addr, u32 *data);
 int opae_mgr_get_board_info(struct opae_manager *mgr,
 		struct opae_board_info **info);
+int opae_mgr_update_flash(struct opae_manager *mgr, const char *image,
+		uint64_t *status);
+int opae_mgr_stop_flash_update(struct opae_manager *mgr, int force);
+int opae_mgr_reload(struct opae_manager *mgr, int type, int page);
 #endif /* _OPAE_HW_API_H_*/
diff --git a/drivers/raw/ifpga/base/opae_intel_max10.c b/drivers/raw/ifpga/base/opae_intel_max10.c
index 1a526ea549..443e248fb3 100644
--- a/drivers/raw/ifpga/base/opae_intel_max10.c
+++ b/drivers/raw/ifpga/base/opae_intel_max10.c
@@ -51,6 +51,22 @@ int max10_sys_write(struct intel_max10_device *dev,
 	return max10_reg_write(dev, dev->base + offset, val);
 }
 
+int max10_sys_update_bits(struct intel_max10_device *dev, unsigned int offset,
+					unsigned int msk, unsigned int val)
+{
+	int ret = 0;
+	unsigned int temp = 0;
+
+	ret = max10_sys_read(dev, offset, &temp);
+	if (ret < 0)
+		return ret;
+
+	temp &= ~msk;
+	temp |= val & msk;
+
+	return max10_sys_write(dev, offset, temp);
+}
+
 static struct max10_compatible_id max10_id_table[] = {
 	{.compatible = MAX10_PAC,},
 	{.compatible = MAX10_PAC_N3000,},
@@ -557,6 +573,36 @@ static int check_max10_version(struct intel_max10_device *dev)
 	return -ENODEV;
 }
 
+static int max10_staging_area_init(struct intel_max10_device *dev)
+{
+	char *fdt_root = dev->fdt_root;
+	int ret, offset = 0;
+	u64 start, size;
+
+	if (!fdt_root) {
+		dev_debug(dev,
+			"skip staging area init as not find Device Tree\n");
+		return -ENODEV;
+	}
+
+	dev->staging_area_size = 0;
+
+	fdt_for_each_subnode(offset, fdt_root, 0) {
+		if (fdt_node_check_compatible(fdt_root, offset,
+					      "ifpga-sec-mgr,staging-area"))
+			continue;
+
+		ret = fdt_get_reg(fdt_root, offset, 0, &start, &size);
+		if (!ret) {
+			dev->staging_area_base = start;
+			dev->staging_area_size = size;
+		}
+		return ret;
+	}
+
+	return -ENODEV;
+}
+
 static int
 max10_secure_hw_init(struct intel_max10_device *dev)
 {
@@ -581,6 +627,8 @@ max10_secure_hw_init(struct intel_max10_device *dev)
 
 	max10_sensor_init(dev, sysmgr_offset);
 
+	max10_staging_area_init(dev);
+
 	return 0;
 }
 
diff --git a/drivers/raw/ifpga/base/opae_intel_max10.h b/drivers/raw/ifpga/base/opae_intel_max10.h
index 123cdc48b9..670683f017 100644
--- a/drivers/raw/ifpga/base/opae_intel_max10.h
+++ b/drivers/raw/ifpga/base/opae_intel_max10.h
@@ -38,6 +38,8 @@ struct intel_max10_device {
 	unsigned int base; /* max10 base address */
 	u16 bus;
 	struct opae_sensor_list opae_sensor_list;
+	u32 staging_area_base;
+	u32 staging_area_size;
 };
 
 /* retimer speed */
@@ -98,6 +100,7 @@ struct opae_retimer_status {
 #define   MAX10_MAC_COUNT	GENMASK(23, 16)
 #define RSU_REG			0x2c
 #define   FPGA_RECONF_PAGE	GENMASK(2, 0)
+#define   FPGA_PAGE(p)		((p) & 0x1)
 #define   FPGA_RP_LOAD		BIT(3)
 #define   NIOS2_PRERESET	BIT(4)
 #define   NIOS2_HANG		BIT(5)
@@ -106,6 +109,9 @@ struct opae_retimer_status {
 #define   NIOS2_I2C2_POLL_STOP	BIT(13)
 #define   PKVL_EEPROM_LOAD	BIT(31)
 #define FPGA_RECONF_REG		0x30
+#define   SFPGA_RECONF_PAGE	GENMASK(22, 20)
+#define   SFPGA_PAGE(p)		(((p) & 0x1) << 20)
+#define   SFPGA_RP_LOAD		BIT(23)
 #define MAX10_TEST_REG		0x3c
 #define   COUNTDOWN_START	BIT(18)
 #define MAX10_BUILD_VER		0x68
@@ -118,8 +124,44 @@ struct opae_retimer_status {
 #define MAX10_DOORBELL		0x400
 #define   RSU_REQUEST		BIT(0)
 #define   SEC_PROGRESS		GENMASK(7, 4)
+#define   SEC_PROGRESS_G(v)	(((v) >> 4) & 0xf)
+#define   SEC_PROGRESS_IDLE				0x0
+#define   SEC_PROGRESS_PREPARE			0x1
+#define   SEC_PROGRESS_SLEEP			0x2
+#define   SEC_PROGRESS_READY			0x3
+#define   SEC_PROGRESS_AUTHENTICATING	0x4
+#define   SEC_PROGRESS_COPYING			0x5
+#define   SEC_PROGRESS_UPDATE_CANCEL	0x6
+#define   SEC_PROGRESS_PROGRAM_KEY_HASH	0x7
+#define   SEC_PROGRESS_RSU_DONE			0x8
+#define   SEC_PROGRESS_PKVL_PROM_DONE	0x9
 #define   HOST_STATUS		GENMASK(11, 8)
+#define   HOST_STATUS_S(v)	(((v) << 8) & 0xf00)
+#define   HOST_STATUS_IDLE			0x0
+#define   HOST_STATUS_WRITE_DONE	0x1
+#define   HOST_STATUS_ABORT_RSU		0x2
 #define   SEC_STATUS		GENMASK(23, 16)
+#define   SEC_STATUS_G(v)	(((v) >> 16) & 0xff)
+#define   SEC_STATUS_NORMAL			0x0
+#define   SEC_STATUS_TIMEOUT		0x1
+#define   SEC_STATUS_AUTH_FAIL		0x2
+#define   SEC_STATUS_COPY_FAIL		0x3
+#define   SEC_STATUS_FATAL			0x4
+#define   SEC_STATUS_PKVL_REJECT	0x5
+#define   SEC_STATUS_NON_INC		0x6
+#define   SEC_STATUS_ERASE_FAIL		0x7
+#define   SEC_STATUS_WEAROUT		0x8
+#define   SEC_STATUS_NIOS_OK		0x80
+#define   SEC_STATUS_USER_OK		0x81
+#define   SEC_STATUS_FACTORY_OK		0x82
+#define   SEC_STATUS_USER_FAIL		0x83
+#define   SEC_STATUS_FACTORY_FAIL	0x84
+#define   SEC_STATUS_NIOS_FLASH_ERR	0x85
+#define   SEC_STATUS_FPGA_FLASH_ERR	0x86
+#define   CONFIG_SEL		BIT(28)
+#define   CONFIG_SEL_S(v)	(((v) & 0x1) << 28)
+#define   REBOOT_REQ		BIT(29)
+#define MAX10_AUTH_RESULT	0x404
 
 /* PKVL related registers, in system register region */
 #define PKVL_POLLING_CTRL		0x80
@@ -149,6 +191,8 @@ int max10_sys_read(struct intel_max10_device *dev,
 	unsigned int offset, unsigned int *val);
 int max10_sys_write(struct intel_max10_device *dev,
 	unsigned int offset, unsigned int val);
+int max10_sys_update_bits(struct intel_max10_device *dev,
+	unsigned int offset, unsigned int msk, unsigned int val);
 struct intel_max10_device *
 intel_max10_device_probe(struct altera_spi_device *spi,
 		int chipselect);
diff --git a/drivers/raw/ifpga/meson.build b/drivers/raw/ifpga/meson.build
index 027ff80562..60ea59ae28 100644
--- a/drivers/raw/ifpga/meson.build
+++ b/drivers/raw/ifpga/meson.build
@@ -13,8 +13,10 @@ objs = [base_objs]
 deps += ['ethdev', 'rawdev', 'pci', 'bus_pci', 'kvargs',
 	'bus_vdev', 'bus_ifpga', 'net', 'net_i40e', 'net_ipn3ke']
 
-sources = files('ifpga_rawdev.c')
+sources = files('ifpga_rawdev.c', 'rte_pmd_ifpga.c')
 
 includes += include_directories('base')
 includes += include_directories('../../net/ipn3ke')
 includes += include_directories('../../net/i40e')
+
+headers = files('rte_pmd_ifpga.h')
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.c b/drivers/raw/ifpga/rte_pmd_ifpga.c
new file mode 100644
index 0000000000..af6f175e98
--- /dev/null
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.c
@@ -0,0 +1,163 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 Intel Corporation
+ */
+
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+#include <rte_rawdev.h>
+#include <rte_rawdev_pmd.h>
+#include "rte_pmd_ifpga.h"
+#include "ifpga_rawdev.h"
+#include "base/ifpga_sec_mgr.h"
+
+
+int
+rte_pmd_ifpga_get_dev_id(const char *pci_addr, uint16_t *dev_id)
+{
+	struct rte_pci_addr addr;
+	struct rte_rawdev *rdev = NULL;
+	char rdev_name[RTE_RAWDEV_NAME_MAX_LEN] = {0};
+
+	if (!pci_addr || !dev_id) {
+		IFPGA_RAWDEV_PMD_ERR("Input parameter is invalid.");
+		return -EINVAL;
+	}
+
+	if (strnlen(pci_addr, PCI_PRI_STR_SIZE) == PCI_PRI_STR_SIZE) {
+		IFPGA_RAWDEV_PMD_ERR("PCI address is too long.");
+		return -EINVAL;
+	}
+
+	if (rte_pci_addr_parse(pci_addr, &addr)) {
+		IFPGA_RAWDEV_PMD_ERR("PCI address %s is invalid.", pci_addr);
+		return -EINVAL;
+	}
+
+	snprintf(rdev_name, RTE_RAWDEV_NAME_MAX_LEN, "IFPGA:%02x:%02x.%x",
+		addr.bus, addr.devid, addr.function);
+	rdev = rte_rawdev_pmd_get_named_dev(rdev_name);
+	if (!rdev) {
+		IFPGA_RAWDEV_PMD_DEBUG("%s is not probed by ifpga driver.",
+			pci_addr);
+		return -ENODEV;
+	}
+	*dev_id = rdev->dev_id;
+
+	return 0;
+}
+
+static struct rte_rawdev *
+get_rte_rawdev(uint16_t dev_id)
+{
+	struct rte_rawdev *dev = NULL;
+
+	if (dev_id >= RTE_RAWDEV_MAX_DEVS)
+		return NULL;
+
+	dev = &rte_rawdevs[dev_id];
+	if (dev->attached == RTE_RAWDEV_ATTACHED)
+		return dev;
+
+	return NULL;
+}
+
+static struct opae_adapter *
+get_opae_adapter(uint16_t dev_id)
+{
+	struct rte_rawdev *dev = NULL;
+	struct opae_adapter *adapter = NULL;
+
+	dev = get_rte_rawdev(dev_id);
+	if (!dev) {
+		IFPGA_RAWDEV_PMD_ERR("Device ID %u is invalid.", dev_id);
+		return NULL;
+	}
+
+	adapter = ifpga_rawdev_get_priv(dev);
+	if (!adapter) {
+		IFPGA_RAWDEV_PMD_ERR("Adapter is not registered.");
+		return NULL;
+	}
+
+	return adapter;
+}
+
+static opae_share_data *
+get_share_data(struct opae_adapter *adapter)
+{
+	opae_share_data *sd = NULL;
+
+	if (!adapter)
+		return NULL;
+
+	sd = (opae_share_data *)adapter->shm.ptr;
+	if (!sd) {
+		IFPGA_RAWDEV_PMD_ERR("Share data is not initialized.");
+		return NULL;
+	}
+
+	return sd;
+}
+
+int
+rte_pmd_ifpga_update_flash(uint16_t dev_id, const char *image,
+	uint64_t *status)
+{
+	struct opae_adapter *adapter = NULL;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	return opae_mgr_update_flash(adapter->mgr, image, status);
+}
+
+int
+rte_pmd_ifpga_stop_update(uint16_t dev_id, int force)
+{
+	struct opae_adapter *adapter = NULL;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	return opae_mgr_stop_flash_update(adapter->mgr, force);
+}
+
+int
+rte_pmd_ifpga_reboot_try(uint16_t dev_id)
+{
+	struct opae_adapter *adapter = NULL;
+	opae_share_data *sd = NULL;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	sd = get_share_data(adapter);
+	if (!sd)
+		return -ENOMEM;
+
+	opae_adapter_lock(adapter, -1);
+	if (IFPGA_RSU_GET_STAT(sd->rsu_stat) != IFPGA_RSU_IDLE) {
+		opae_adapter_unlock(adapter);
+		IFPGA_RAWDEV_PMD_WARN("Update or reboot is in progress.");
+		return -EBUSY;
+	}
+	sd->rsu_stat = IFPGA_RSU_STATUS(IFPGA_RSU_REBOOT, 0);
+	opae_adapter_unlock(adapter);
+
+	return 0;
+}
+
+int
+rte_pmd_ifpga_reload(uint16_t dev_id, int type, int page)
+{
+	struct opae_adapter *adapter = NULL;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	return opae_mgr_reload(adapter->mgr, type, page);
+}
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.h b/drivers/raw/ifpga/rte_pmd_ifpga.h
new file mode 100644
index 0000000000..023a011116
--- /dev/null
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.h
@@ -0,0 +1,132 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 Intel Corporation
+ */
+
+#ifndef _RTE_PMD_IFPGA_H_
+#define _RTE_PMD_IFPGA_H_
+
+/**
+ * @file rte_pmd_ifpga.h
+ *
+ * ifpga PMD specific functions.
+ *
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Get raw device ID from PCI address string like 'Domain:Bus:Dev.Func'
+ *
+ * @param pci_addr
+ *    The PCI address of specified Intel FPGA device.
+ * @param dev_id
+ *    The buffer to output device ID.
+ * @return
+ *   - (0) if successful.
+ *   - (-EINVAL) if bad parameter.
+ *   - (-ENODEV) if FPGA is not probed by ifpga driver.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_get_dev_id(const char *pci_addr, uint16_t *dev_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Update image flash of specified Intel FPGA device
+ *
+ * @param dev_id
+ *   The raw device ID of specified Intel FPGA device.
+ * @param image
+ *   The image file name string.
+ * @param status
+ *   The detailed update status for debug.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-EINVAL) if bad parameter or staging area is not initialized.
+ *   - (-EBUSY) if FPGA is updating or rebooting.
+ *   - (-EIO) if failed to open image file.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_update_flash(uint16_t dev_id, const char *image,
+	uint64_t *status);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Stop flash update of specified Intel FPGA device
+ *
+ * @param dev_id
+ *   The raw device ID of specified Intel FPGA device.
+ * @param force
+ *   Abort the update process by writing register if set non-zero.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-EINVAL) if bad parameter.
+ *   - (-EAGAIN) if failed with force.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_stop_update(uint16_t dev_id, int force);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Check current Intel FPGA status and change it to reboot status if it is idle
+ *
+ * @param dev_id
+ *    The raw device ID of specified Intel FPGA device.
+ * @return
+ *   - (0) if FPGA is ready to reboot.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-ENOMEM) if share data is not initialized.
+ *   - (-EBUSY) if FPGA is updating or rebooting.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_reboot_try(uint16_t dev_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Trigger full reconfiguration of specified Intel FPGA device
+ *
+ * @param dev_id
+ *    The raw device ID of specified Intel FPGA device.
+ * @param type
+ *    Select reconfiguration type.
+ *    0 - reconfigure FPGA only.
+ *    1 - reboot the whole card including FPGA.
+ * @param page
+ *    Select image from which flash partition.
+ *    0 - factory partition.
+ *    1 - user partition.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-EINVAL) if bad parameter.
+ *   - (-EBUSY) if failed to access BMC register.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_reload(uint16_t dev_id, int type, int page);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMD_IFPGA_H_ */
diff --git a/drivers/raw/ifpga/version.map b/drivers/raw/ifpga/version.map
index 4a76d1d52d..16584f7fe4 100644
--- a/drivers/raw/ifpga/version.map
+++ b/drivers/raw/ifpga/version.map
@@ -1,3 +1,14 @@
 DPDK_21 {
 	local: *;
 };
+
+EXPERIMENTAL {
+	global:
+
+	# added in 21.05
+	rte_pmd_ifpga_get_dev_id;
+	rte_pmd_ifpga_update_flash;
+	rte_pmd_ifpga_stop_update;
+	rte_pmd_ifpga_reboot_try;
+	rte_pmd_ifpga_reload;
+};
-- 
2.29.2

[dpdk-dev] [PATCH v14 2/4] raw/ifpga: add APIs to get fpga information

[Wei Huang]

There are some information data can be got from FPGA, they are
implemented in below APIs:
1. rte_pmd_ifpga_get_property() get properties of FPGA (include BMC).
2. rte_pmd_ifpga_get_phy_info() get information of PHY connect to FPGA.
3. rte_pmd_ifpga_get_rsu_status() get status of rsu process.

Signed-off-by: Wei Huang <wei.huang@intel.com>
Acked-by: Tianfei Zhang <tianfei.zhang@intel.com>
Acked-by: Rosen Xu <rosen.xu@intel.com>
---
v2: implement APIs in rte_pmd_ifpga.c
---
 drivers/raw/ifpga/base/ifpga_api.c         |   8 +
 drivers/raw/ifpga/base/ifpga_defines.h     |   1 +
 drivers/raw/ifpga/base/ifpga_feature_dev.c |  21 ++
 drivers/raw/ifpga/base/ifpga_feature_dev.h |   1 +
 drivers/raw/ifpga/base/ifpga_fme.c         |  28 ++-
 drivers/raw/ifpga/base/opae_hw_api.c       |  18 ++
 drivers/raw/ifpga/base/opae_hw_api.h       |   2 +
 drivers/raw/ifpga/base/opae_ifpga_hw_api.h |   1 +
 drivers/raw/ifpga/rte_pmd_ifpga.c          | 221 +++++++++++++++++++++
 drivers/raw/ifpga/rte_pmd_ifpga.h          | 110 ++++++++++
 drivers/raw/ifpga/version.map              |   3 +
 11 files changed, 412 insertions(+), 2 deletions(-)

diff --git a/drivers/raw/ifpga/base/ifpga_api.c b/drivers/raw/ifpga/base/ifpga_api.c
index 1aedf150bc..4610ef101e 100644
--- a/drivers/raw/ifpga/base/ifpga_api.c
+++ b/drivers/raw/ifpga/base/ifpga_api.c
@@ -229,6 +229,13 @@ static int ifpga_mgr_get_board_info(struct opae_manager *mgr,
 	return 0;
 }
 
+static int ifpga_mgr_get_uuid(struct opae_manager *mgr, struct uuid *uuid)
+{
+	struct ifpga_fme_hw *fme = mgr->data;
+
+	return fpga_get_pr_uuid(fme, uuid);
+}
+
 static int ifpga_mgr_update_flash(struct opae_manager *mgr, const char *image,
 	u64 *status)
 {
@@ -256,6 +263,7 @@ struct opae_manager_ops ifpga_mgr_ops = {
 	.get_eth_group_region_info = ifpga_mgr_get_eth_group_region_info,
 	.get_sensor_value = ifpga_mgr_get_sensor_value,
 	.get_board_info = ifpga_mgr_get_board_info,
+	.get_uuid = ifpga_mgr_get_uuid,
 	.update_flash = ifpga_mgr_update_flash,
 	.stop_flash_update = ifpga_mgr_stop_flash_update,
 	.reload = ifpga_mgr_reload,
diff --git a/drivers/raw/ifpga/base/ifpga_defines.h b/drivers/raw/ifpga/base/ifpga_defines.h
index 9f0147d1ed..dca1518a83 100644
--- a/drivers/raw/ifpga/base/ifpga_defines.h
+++ b/drivers/raw/ifpga/base/ifpga_defines.h
@@ -1727,6 +1727,7 @@ struct opae_board_info {
 	u8 seu;
 	u8 ptp;
 
+	u32 boot_page;
 	u32 max10_version;
 	u32 nios_fw_version;
 	u32 nums_of_retimer;
diff --git a/drivers/raw/ifpga/base/ifpga_feature_dev.c b/drivers/raw/ifpga/base/ifpga_feature_dev.c
index 0f852a75ad..08135137ad 100644
--- a/drivers/raw/ifpga/base/ifpga_feature_dev.c
+++ b/drivers/raw/ifpga/base/ifpga_feature_dev.c
@@ -87,6 +87,27 @@ int fpga_get_afu_uuid(struct ifpga_port_hw *port, struct uuid *uuid)
 	return 0;
 }
 
+int fpga_get_pr_uuid(struct ifpga_fme_hw *fme, struct uuid *uuid)
+{
+	struct feature_fme_pr *fme_pr;
+	u64 guidl, guidh;
+
+	if (!fme || !uuid)
+		return -EINVAL;
+
+	fme_pr = get_fme_feature_ioaddr_by_index(fme, FME_FEATURE_ID_PR_MGMT);
+
+	spinlock_lock(&fme->lock);
+	guidl = readq(&fme_pr->fme_pr_intfc_id_l);
+	guidh = readq(&fme_pr->fme_pr_intfc_id_h);
+	spinlock_unlock(&fme->lock);
+
+	opae_memcpy(uuid->b, &guidl, sizeof(u64));
+	opae_memcpy(uuid->b + 8, &guidh, sizeof(u64));
+
+	return 0;
+}
+
 /* Mask / Unmask Port Errors by the Error Mask register. */
 void port_err_mask(struct ifpga_port_hw *port, bool mask)
 {
diff --git a/drivers/raw/ifpga/base/ifpga_feature_dev.h b/drivers/raw/ifpga/base/ifpga_feature_dev.h
index 2b1309b44a..b355d22b0e 100644
--- a/drivers/raw/ifpga/base/ifpga_feature_dev.h
+++ b/drivers/raw/ifpga/base/ifpga_feature_dev.h
@@ -103,6 +103,7 @@ is_port_feature_present(struct ifpga_port_hw *port, int index)
 }
 
 int fpga_get_afu_uuid(struct ifpga_port_hw *port, struct uuid *uuid);
+int fpga_get_pr_uuid(struct ifpga_fme_hw *fme, struct uuid *uuid);
 
 int __fpga_port_disable(struct ifpga_port_hw *port);
 void __fpga_port_enable(struct ifpga_port_hw *port);
diff --git a/drivers/raw/ifpga/base/ifpga_fme.c b/drivers/raw/ifpga/base/ifpga_fme.c
index 34fd9a818e..43c7b9c3dc 100644
--- a/drivers/raw/ifpga/base/ifpga_fme.c
+++ b/drivers/raw/ifpga/base/ifpga_fme.c
@@ -101,6 +101,24 @@ static int fme_hdr_get_ports_num(struct ifpga_fme_hw *fme, u64 *ports_num)
 	return 0;
 }
 
+static int fme_hdr_get_port_type(struct ifpga_fme_hw *fme, u64 *port_type)
+{
+	struct feature_fme_header *fme_hdr
+		= get_fme_feature_ioaddr_by_index(fme, FME_FEATURE_ID_HEADER);
+	struct feature_fme_port pt;
+	u32 port = (u32)((*port_type >> 32) & 0xffffffff);
+
+	pt.csr = readq(&fme_hdr->port[port]);
+	if (!pt.port_implemented)
+		return -ENODEV;
+	if (pt.afu_access_control)
+		*port_type |= 0x1;
+	else
+		*port_type &= ~0x1;
+
+	return 0;
+}
+
 static int fme_hdr_get_cache_size(struct ifpga_fme_hw *fme, u64 *cache_size)
 {
 	struct feature_fme_header *fme_hdr
@@ -179,6 +197,8 @@ fme_hdr_get_prop(struct ifpga_feature *feature, struct feature_prop *prop)
 		return fme_hdr_get_bitstream_id(fme, &prop->data);
 	case FME_HDR_PROP_BITSTREAM_METADATA:
 		return fme_hdr_get_bitstream_metadata(fme, &prop->data);
+	case FME_HDR_PROP_PORT_TYPE:
+		return fme_hdr_get_port_type(fme, &prop->data);
 	}
 
 	return -ENOENT;
@@ -891,13 +911,17 @@ static int fme_get_board_interface(struct ifpga_fme_hw *fme)
 			fme->board_info.nums_of_fvl,
 			fme->board_info.ports_per_fvl);
 
+	if (max10_sys_read(fme->max10_dev, FPGA_PAGE_INFO, &val))
+		return -EINVAL;
+	fme->board_info.boot_page = val & 0x7;
+
 	if (max10_sys_read(fme->max10_dev, MAX10_BUILD_VER, &val))
 		return -EINVAL;
-	fme->board_info.max10_version = val & 0xffffff;
+	fme->board_info.max10_version = val;
 
 	if (max10_sys_read(fme->max10_dev, NIOS2_FW_VERSION, &val))
 		return -EINVAL;
-	fme->board_info.nios_fw_version = val & 0xffffff;
+	fme->board_info.nios_fw_version = val;
 
 	dev_info(fme, "max10 version 0x%x, nios fw version 0x%x\n",
 		fme->board_info.max10_version,
diff --git a/drivers/raw/ifpga/base/opae_hw_api.c b/drivers/raw/ifpga/base/opae_hw_api.c
index 86ad88f720..11c9887c7f 100644
--- a/drivers/raw/ifpga/base/opae_hw_api.c
+++ b/drivers/raw/ifpga/base/opae_hw_api.c
@@ -967,6 +967,24 @@ opae_mgr_get_board_info(struct opae_manager *mgr,
 	return -ENOENT;
 }
 
+/**
+ * opae_mgr_get_uuid -  get manager's UUID.
+ * @mgr: targeted manager
+ * @uuid: a pointer to UUID
+ *
+ * Return: 0 on success, otherwise error code.
+ */
+int opae_mgr_get_uuid(struct opae_manager *mgr, struct uuid *uuid)
+{
+	if (!mgr || !uuid)
+		return -EINVAL;
+
+	if (mgr->ops && mgr->ops->get_uuid)
+		return mgr->ops->get_uuid(mgr, uuid);
+
+	return -ENOENT;
+}
+
 /**
  * opae_mgr_update_flash -  update image in flash.
  * @mgr: targeted manager
diff --git a/drivers/raw/ifpga/base/opae_hw_api.h b/drivers/raw/ifpga/base/opae_hw_api.h
index 91d26d9b5b..7e04b56471 100644
--- a/drivers/raw/ifpga/base/opae_hw_api.h
+++ b/drivers/raw/ifpga/base/opae_hw_api.h
@@ -55,6 +55,7 @@ struct opae_manager_ops {
 			unsigned int *value);
 	int (*get_board_info)(struct opae_manager *mgr,
 			struct opae_board_info **info);
+	int (*get_uuid)(struct opae_manager *mgr, struct uuid *uuid);
 	int (*update_flash)(struct opae_manager *mgr, const char *image,
 			u64 *status);
 	int (*stop_flash_update)(struct opae_manager *mgr, int force);
@@ -360,6 +361,7 @@ int opae_manager_eth_group_read_reg(struct opae_manager *mgr, u8 group_id,
 		u8 type, u8 index, u16 addr, u32 *data);
 int opae_mgr_get_board_info(struct opae_manager *mgr,
 		struct opae_board_info **info);
+int opae_mgr_get_uuid(struct opae_manager *mgr, struct uuid *uuid);
 int opae_mgr_update_flash(struct opae_manager *mgr, const char *image,
 		uint64_t *status);
 int opae_mgr_stop_flash_update(struct opae_manager *mgr, int force);
diff --git a/drivers/raw/ifpga/base/opae_ifpga_hw_api.h b/drivers/raw/ifpga/base/opae_ifpga_hw_api.h
index bab33862ee..ffdbebf704 100644
--- a/drivers/raw/ifpga/base/opae_ifpga_hw_api.h
+++ b/drivers/raw/ifpga/base/opae_ifpga_hw_api.h
@@ -61,6 +61,7 @@ struct feature_prop {
 #define FME_HDR_PROP_SOCKET_ID		0x5	/* RDONLY */
 #define FME_HDR_PROP_BITSTREAM_ID		0x6	/* RDONLY */
 #define FME_HDR_PROP_BITSTREAM_METADATA	0x7	/* RDONLY */
+#define FME_HDR_PROP_PORT_TYPE		0x8	/* RDWR */
 
 /* FME error reporting feature's properties */
 /* FME error reporting properties format */
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.c b/drivers/raw/ifpga/rte_pmd_ifpga.c
index af6f175e98..8e04e22d5f 100644
--- a/drivers/raw/ifpga/rte_pmd_ifpga.c
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.c
@@ -8,6 +8,7 @@
 #include <rte_rawdev_pmd.h>
 #include "rte_pmd_ifpga.h"
 #include "ifpga_rawdev.h"
+#include "base/ifpga_api.h"
 #include "base/ifpga_sec_mgr.h"
 
 
@@ -99,6 +100,226 @@ get_share_data(struct opae_adapter *adapter)
 	return sd;
 }
 
+int
+rte_pmd_ifpga_get_rsu_status(uint16_t dev_id, uint32_t *stat, uint32_t *prog)
+{
+	struct opae_adapter *adapter = NULL;
+	opae_share_data *sd = NULL;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	sd = get_share_data(adapter);
+	if (!sd)
+		return -ENOMEM;
+
+	if (stat)
+		*stat = IFPGA_RSU_GET_STAT(sd->rsu_stat);
+	if (prog)
+		*prog = IFPGA_RSU_GET_PROG(sd->rsu_stat);
+
+	return 0;
+}
+
+static int
+ifpga_is_rebooting(struct opae_adapter *adapter)
+{
+	opae_share_data *sd = NULL;
+
+	sd = get_share_data(adapter);
+	if (!sd)
+		return 1;
+
+	if (IFPGA_RSU_GET_STAT(sd->rsu_stat) == IFPGA_RSU_REBOOT) {
+		IFPGA_RAWDEV_PMD_WARN("Reboot is in progress.");
+		return 1;
+	}
+
+	return 0;
+}
+
+static int
+get_common_property(struct opae_adapter *adapter,
+	rte_pmd_ifpga_common_prop *prop)
+{
+	struct ifpga_fme_hw *fme = NULL;
+	struct opae_board_info *info = NULL;
+	struct feature_prop fp;
+	struct uuid pr_id;
+	int ret = 0;
+
+	if (!adapter || !prop)
+		return -EINVAL;
+
+	if (!adapter->mgr || !adapter->mgr->data) {
+		IFPGA_RAWDEV_PMD_ERR("Manager is not registered.");
+		return -ENODEV;
+	}
+
+	fme = adapter->mgr->data;
+	fp.feature_id = FME_FEATURE_ID_HEADER;
+	fp.prop_id = FME_HDR_PROP_PORTS_NUM;
+	ret = ifpga_get_prop(fme->parent, FEATURE_FIU_ID_FME, 0, &fp);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get port number.");
+		return ret;
+	}
+	prop->num_ports = fp.data;
+
+	fp.prop_id = FME_HDR_PROP_BITSTREAM_ID;
+	ret = ifpga_get_prop(fme->parent, FEATURE_FIU_ID_FME, 0, &fp);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get bitstream ID.");
+		return ret;
+	}
+	prop->bitstream_id = fp.data;
+
+	fp.prop_id = FME_HDR_PROP_BITSTREAM_METADATA;
+	ret = ifpga_get_prop(fme->parent, FEATURE_FIU_ID_FME, 0, &fp);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get bitstream metadata.");
+		return ret;
+	}
+	prop->bitstream_metadata = fp.data;
+
+	ret = opae_mgr_get_uuid(adapter->mgr, &pr_id);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get PR ID.");
+		return ret;
+	}
+	memcpy(prop->pr_id.b, pr_id.b, sizeof(rte_pmd_ifpga_uuid));
+
+	ret = opae_mgr_get_board_info(adapter->mgr, &info);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get board info.");
+		return ret;
+	}
+	prop->boot_page = info->boot_page;
+	prop->bmc_version = info->max10_version;
+	prop->bmc_nios_version = info->nios_fw_version;
+
+	return 0;
+}
+
+static int
+get_port_property(struct opae_adapter *adapter, uint16_t port,
+	rte_pmd_ifpga_port_prop *prop)
+{
+	struct ifpga_fme_hw *fme = NULL;
+	struct feature_prop fp;
+	struct opae_accelerator *acc = NULL;
+	struct uuid afu_id;
+	int ret = 0;
+
+	if (!adapter || !prop)
+		return -EINVAL;
+
+	if (!adapter->mgr || !adapter->mgr->data) {
+		IFPGA_RAWDEV_PMD_ERR("Manager is not registered.");
+		return -ENODEV;
+	}
+
+	fme = adapter->mgr->data;
+	fp.feature_id = FME_FEATURE_ID_HEADER;
+	fp.prop_id = FME_HDR_PROP_PORT_TYPE;
+	fp.data = port;
+	fp.data <<= 32;
+	ret = ifpga_get_prop(fme->parent, FEATURE_FIU_ID_FME, 0, &fp);
+	if (ret)
+		return ret;
+	prop->type = fp.data & 0xffffffff;
+
+	if (prop->type == 0) {
+		acc = opae_adapter_get_acc(adapter, port);
+		ret = opae_acc_get_uuid(acc, &afu_id);
+		if (ret) {
+			IFPGA_RAWDEV_PMD_ERR("Failed to get port%u AFU ID.",
+				port);
+			return ret;
+		}
+		memcpy(prop->afu_id.b, afu_id.b, sizeof(rte_pmd_ifpga_uuid));
+	}
+
+	return 0;
+}
+
+int
+rte_pmd_ifpga_get_property(uint16_t dev_id, rte_pmd_ifpga_prop *prop)
+{
+	struct opae_adapter *adapter = NULL;
+	uint32_t i = 0;
+	int ret = 0;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	opae_adapter_lock(adapter, -1);
+	if (ifpga_is_rebooting(adapter)) {
+		ret = -EBUSY;
+		goto unlock_dev;
+	}
+
+	ret = get_common_property(adapter, &prop->common);
+	if (ret) {
+		ret = -EIO;
+		goto unlock_dev;
+	}
+
+	for (i = 0; i < prop->common.num_ports; i++) {
+		ret = get_port_property(adapter, i, &prop->port[i]);
+		if (ret) {
+			ret = -EIO;
+			break;
+		}
+	}
+
+unlock_dev:
+	opae_adapter_unlock(adapter);
+	return ret;
+}
+
+int
+rte_pmd_ifpga_get_phy_info(uint16_t dev_id, rte_pmd_ifpga_phy_info *info)
+{
+	struct opae_adapter *adapter = NULL;
+	struct opae_retimer_info rtm_info;
+	struct opae_retimer_status rtm_status;
+	int ret = 0;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	opae_adapter_lock(adapter, -1);
+	if (ifpga_is_rebooting(adapter)) {
+		ret = -EBUSY;
+		goto unlock_dev;
+	}
+
+	ret = opae_manager_get_retimer_info(adapter->mgr, &rtm_info);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get retimer info.");
+		ret = -EIO;
+		goto unlock_dev;
+	}
+	info->num_retimers = rtm_info.nums_retimer;
+
+	ret = opae_manager_get_retimer_status(adapter->mgr, &rtm_status);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get retimer status.");
+		ret = -EIO;
+		goto unlock_dev;
+	}
+	info->link_speed = rtm_status.speed;
+	info->link_status = rtm_status.line_link_bitmap;
+
+unlock_dev:
+	opae_adapter_unlock(adapter);
+	return ret;
+}
+
 int
 rte_pmd_ifpga_update_flash(uint16_t dev_id, const char *image,
 	uint64_t *status)
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.h b/drivers/raw/ifpga/rte_pmd_ifpga.h
index 023a011116..633f6e9d1b 100644
--- a/drivers/raw/ifpga/rte_pmd_ifpga.h
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.h
@@ -18,6 +18,55 @@
 extern "C" {
 #endif
 
+#include <stdint.h>
+
+#define IFPGA_MAX_PORT_NUM   4
+
+/**
+ * UUID data structure.
+ */
+typedef struct {
+	uint8_t b[16];
+} rte_pmd_ifpga_uuid;
+
+/**
+ * FME property data structure.
+ */
+typedef struct {
+	uint32_t num_ports;
+	uint32_t boot_page;
+	uint64_t bitstream_id;
+	uint64_t bitstream_metadata;
+	rte_pmd_ifpga_uuid pr_id;
+	uint32_t bmc_version;
+	uint32_t bmc_nios_version;
+} rte_pmd_ifpga_common_prop;
+
+/**
+ * port property data structure.
+ */
+typedef struct {
+	rte_pmd_ifpga_uuid afu_id;
+	uint32_t type;   /* AFU memory access control type */
+} rte_pmd_ifpga_port_prop;
+
+/**
+ * FPGA property data structure.
+ */
+typedef struct {
+	rte_pmd_ifpga_common_prop  common;
+	rte_pmd_ifpga_port_prop    port[IFPGA_MAX_PORT_NUM];
+} rte_pmd_ifpga_prop;
+
+/**
+ * PHY information data structure.
+ */
+typedef struct {
+	uint32_t num_retimers;
+	uint32_t link_speed;
+	uint32_t link_status;
+} rte_pmd_ifpga_phy_info;
+
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
@@ -37,6 +86,67 @@ __rte_experimental
 int
 rte_pmd_ifpga_get_dev_id(const char *pci_addr, uint16_t *dev_id);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Get current RSU status of the specified Intel FPGA device
+ *
+ * @param dev_id
+ *    The raw device ID of specified Intel FPGA device.
+ * @param stat
+ *    The buffer to output RSU status.
+ * @param prog
+ *    The buffer to output RSU progress.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-ENOMEM) if share data is not initialized.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_get_rsu_status(uint16_t dev_id, uint32_t *stat, uint32_t *prog);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Get FPGA property of specified Intel FPGA device
+ *
+ * @param dev_id
+ *    The raw device ID of specified Intel FPGA device.
+ * @param prop
+ *    The data pointer of FPGA property buffer.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-EBUSY) if FPGA is rebooting.
+ *   - (-EIO) if failed to access hardware.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_get_property(uint16_t dev_id, rte_pmd_ifpga_prop *prop);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Get PHY information of specified Intel FPGA device
+ *
+ * @param dev_id
+ *    The raw device ID of specified Intel FPGA device.
+ * @param info
+ *    The data pointer of PHY information buffer.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-EBUSY) if FPGA is rebooting.
+ *   - (-EIO) if failed to access hardware.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_get_phy_info(uint16_t dev_id, rte_pmd_ifpga_phy_info *info);
+
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
diff --git a/drivers/raw/ifpga/version.map b/drivers/raw/ifpga/version.map
index 16584f7fe4..ca6f7f5810 100644
--- a/drivers/raw/ifpga/version.map
+++ b/drivers/raw/ifpga/version.map
@@ -7,6 +7,9 @@ EXPERIMENTAL {
 
 	# added in 21.05
 	rte_pmd_ifpga_get_dev_id;
+	rte_pmd_ifpga_get_rsu_status;
+	rte_pmd_ifpga_get_property;
+	rte_pmd_ifpga_get_phy_info;
 	rte_pmd_ifpga_update_flash;
 	rte_pmd_ifpga_stop_update;
 	rte_pmd_ifpga_reboot_try;
-- 
2.29.2

[dpdk-dev] [PATCH v14 3/4] raw/ifpga: add miscellaneous APIs

[Wei Huang]

Below miscellaneous APIs are used to implement OPAE application.
1. rte_pmd_ifpga_get_pci_bus() get PCI bus ifpga driver registered.
2. rte_pmd_ifpga_partial_reconfigure() do partial reconfiguration.
3. rte_pmd_ifpga_cleanup() free software resources allocated by driver.
4. rte_pmd_ifpga_set_rsu_status() set status of rsu process.

Signed-off-by: Wei Huang <wei.huang@intel.com>
Acked-by: Tianfei Zhang <tianfei.zhang@intel.com>
Acked-by: Rosen Xu <rosen.xu@intel.com>
---
v2: implement APIs in rte_pmd_ifpga.c
---
 drivers/raw/ifpga/ifpga_rawdev.c  | 30 ++++++++++++++
 drivers/raw/ifpga/ifpga_rawdev.h  |  7 +++-
 drivers/raw/ifpga/rte_pmd_ifpga.c | 45 +++++++++++++++++++++
 drivers/raw/ifpga/rte_pmd_ifpga.h | 66 +++++++++++++++++++++++++++++++
 drivers/raw/ifpga/version.map     |  4 ++
 5 files changed, 151 insertions(+), 1 deletion(-)

diff --git a/drivers/raw/ifpga/ifpga_rawdev.c b/drivers/raw/ifpga/ifpga_rawdev.c
index 27129b133e..05d79bfcc2 100644
--- a/drivers/raw/ifpga/ifpga_rawdev.c
+++ b/drivers/raw/ifpga/ifpga_rawdev.c
@@ -1737,3 +1737,33 @@ RTE_PMD_REGISTER_PARAM_STRING(ifpga_rawdev_cfg,
 	"ifpga=<string> "
 	"port=<int> "
 	"afu_bts=<path>");
+
+struct rte_pci_bus *ifpga_get_pci_bus(void)
+{
+	return rte_ifpga_rawdev_pmd.bus;
+}
+
+int ifpga_rawdev_partial_reconfigure(struct rte_rawdev *dev, int port,
+	const char *file)
+{
+	if (!dev) {
+		IFPGA_RAWDEV_PMD_ERR("Input parameter is invalid");
+		return -EINVAL;
+	}
+
+	return rte_fpga_do_pr(dev, port, file);
+}
+
+void ifpga_rawdev_cleanup(void)
+{
+	struct ifpga_rawdev *dev;
+	unsigned int i;
+
+	for (i = 0; i < IFPGA_RAWDEV_NUM; i++) {
+		dev = &ifpga_rawdevices[i];
+		if (dev->rawdev) {
+			rte_rawdev_pmd_release(dev->rawdev);
+			dev->rawdev = NULL;
+		}
+	}
+}
diff --git a/drivers/raw/ifpga/ifpga_rawdev.h b/drivers/raw/ifpga/ifpga_rawdev.h
index 7754beb02b..9bbe9a4278 100644
--- a/drivers/raw/ifpga/ifpga_rawdev.h
+++ b/drivers/raw/ifpga/ifpga_rawdev.h
@@ -43,7 +43,7 @@ enum ifpga_rawdev_device_state {
 static inline struct opae_adapter *
 ifpga_rawdev_get_priv(const struct rte_rawdev *rawdev)
 {
-	return rawdev->dev_private;
+	return (struct opae_adapter *)rawdev->dev_private;
 }
 
 #define IFPGA_RAWDEV_MSIX_IRQ_NUM 7
@@ -76,4 +76,9 @@ int
 ifpga_unregister_msix_irq(enum ifpga_irq_type type,
 		int vec_start, rte_intr_callback_fn handler, void *arg);
 
+struct rte_pci_bus *ifpga_get_pci_bus(void);
+int ifpga_rawdev_partial_reconfigure(struct rte_rawdev *dev, int port,
+	const char *file);
+void ifpga_rawdev_cleanup(void);
+
 #endif /* _IFPGA_RAWDEV_H_ */
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.c b/drivers/raw/ifpga/rte_pmd_ifpga.c
index 8e04e22d5f..6e23a2581a 100644
--- a/drivers/raw/ifpga/rte_pmd_ifpga.c
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.c
@@ -122,6 +122,25 @@ rte_pmd_ifpga_get_rsu_status(uint16_t dev_id, uint32_t *stat, uint32_t *prog)
 	return 0;
 }
 
+int
+rte_pmd_ifpga_set_rsu_status(uint16_t dev_id, uint32_t stat, uint32_t prog)
+{
+	struct opae_adapter *adapter = NULL;
+	opae_share_data *sd = NULL;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	sd = get_share_data(adapter);
+	if (!sd)
+		return -ENOMEM;
+
+	sd->rsu_stat = IFPGA_RSU_STATUS(stat, prog);
+
+	return 0;
+}
+
 static int
 ifpga_is_rebooting(struct opae_adapter *adapter)
 {
@@ -382,3 +401,29 @@ rte_pmd_ifpga_reload(uint16_t dev_id, int type, int page)
 
 	return opae_mgr_reload(adapter->mgr, type, page);
 }
+
+const struct rte_pci_bus *
+rte_pmd_ifpga_get_pci_bus(void)
+{
+	return ifpga_get_pci_bus();
+}
+
+int
+rte_pmd_ifpga_partial_reconfigure(uint16_t dev_id, int port, const char *file)
+{
+	struct rte_rawdev *dev = NULL;
+
+	dev = get_rte_rawdev(dev_id);
+	if (!dev) {
+		IFPGA_RAWDEV_PMD_ERR("Device ID %u is invalid.", dev_id);
+		return -EINVAL;
+	}
+
+	return ifpga_rawdev_partial_reconfigure(dev, port, file);
+}
+
+void
+rte_pmd_ifpga_cleanup(void)
+{
+	ifpga_rawdev_cleanup();
+}
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.h b/drivers/raw/ifpga/rte_pmd_ifpga.h
index 633f6e9d1b..47d66ba655 100644
--- a/drivers/raw/ifpga/rte_pmd_ifpga.h
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.h
@@ -107,6 +107,27 @@ __rte_experimental
 int
 rte_pmd_ifpga_get_rsu_status(uint16_t dev_id, uint32_t *stat, uint32_t *prog);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Set current RSU status of the specified Intel FPGA device
+ *
+ * @param dev_id
+ *    The raw device ID of specified Intel FPGA device.
+ * @param stat
+ *    The RSU status value to set.
+ * @param prog
+ *    The RSU progress value to set.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-ENOMEM) if share data is not initialized.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_set_rsu_status(uint16_t dev_id, uint32_t stat, uint32_t prog);
+
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
@@ -235,6 +256,51 @@ __rte_experimental
 int
 rte_pmd_ifpga_reload(uint16_t dev_id, int type, int page);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Get PCI bus the Intel FPGA driver register to
+ *
+ * @return
+ *   - (valid pointer) if successful.
+ *   - (NULL) if the Intel FPGA driver is not registered to any PCI bus.
+ */
+__rte_experimental
+const struct rte_pci_bus *
+rte_pmd_ifpga_get_pci_bus(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Perform PR (partial reconfiguration) on specified Intel FPGA device
+ *
+ * @param dev_id
+ *   The raw device ID of specified Intel FPGA device.
+ * @param port
+ *   The port index of the partial reconfiguration area.
+ * @param file
+ *   The GBS (Green BitStream) image file name string.
+ * @return
+ *   - (0) if successful.
+ *   - (-EINVAL) if bad parameter or operation failed.
+ *   - (-ENOMEM) if failed to allocate memory.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_partial_reconfigure(uint16_t dev_id, int port, const char *file);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Free software resources allocated by Intel FPGA PMD
+ */
+__rte_experimental
+void
+rte_pmd_ifpga_cleanup(void);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/drivers/raw/ifpga/version.map b/drivers/raw/ifpga/version.map
index ca6f7f5810..995c419a9b 100644
--- a/drivers/raw/ifpga/version.map
+++ b/drivers/raw/ifpga/version.map
@@ -8,10 +8,14 @@ EXPERIMENTAL {
 	# added in 21.05
 	rte_pmd_ifpga_get_dev_id;
 	rte_pmd_ifpga_get_rsu_status;
+	rte_pmd_ifpga_set_rsu_status;
 	rte_pmd_ifpga_get_property;
 	rte_pmd_ifpga_get_phy_info;
 	rte_pmd_ifpga_update_flash;
 	rte_pmd_ifpga_stop_update;
 	rte_pmd_ifpga_reboot_try;
 	rte_pmd_ifpga_reload;
+	rte_pmd_ifpga_get_pci_bus;
+	rte_pmd_ifpga_partial_reconfigure;
+	rte_pmd_ifpga_cleanup;
 };
-- 
2.29.2

[dpdk-dev] [PATCH v14 4/4] examples/ifpga: add example for ifpga APIs

[Wei Huang]

This example application shows how to use ifpga APIs.
A set of OPAE APIs are implemented based on these ifpga APIs.
You can test each OPAE API by running corresponding command.
A guide is also added to show how to run the example.

Signed-off-by: Wei Huang <wei.huang@intel.com>
Acked-by: Tianfei Zhang <tianfei.zhang@intel.com>
Acked-by: Rosen Xu <rosen.xu@intel.com>
---
v2: fix coding style issue in commands.c
---
v3: add guide for running example
---
v4: fix compilation issue of ifpga.rst
---
v5: add ifpga.rst into sample_app_ug/index.rst
---
v6: implement OPAE APIs in example instead of in driver
---
v7: fix compilation issue of unused const variable 'kdrv'
---
 MAINTAINERS                        |    3 +
 doc/guides/sample_app_ug/ifpga.rst |  387 +++++++
 doc/guides/sample_app_ug/index.rst |    1 +
 examples/ifpga/Makefile            |   43 +
 examples/ifpga/commands.c          | 1294 ++++++++++++++++++++++
 examples/ifpga/commands.h          |   16 +
 examples/ifpga/main.c              |   38 +
 examples/ifpga/meson.build         |   20 +
 examples/ifpga/opae_api.c          | 1630 ++++++++++++++++++++++++++++
 examples/ifpga/opae_api.h          |  244 +++++
 examples/meson.build               |    2 +-
 11 files changed, 3677 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/sample_app_ug/ifpga.rst
 create mode 100644 examples/ifpga/Makefile
 create mode 100644 examples/ifpga/commands.c
 create mode 100644 examples/ifpga/commands.h
 create mode 100644 examples/ifpga/main.c
 create mode 100644 examples/ifpga/meson.build
 create mode 100644 examples/ifpga/opae_api.c
 create mode 100644 examples/ifpga/opae_api.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 1a12916f56..48c2ea44d4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1236,9 +1236,12 @@ Rawdev Drivers
 Intel FPGA
 M: Rosen Xu <rosen.xu@intel.com>
 M: Tianfei zhang <tianfei.zhang@intel.com>
+M: Wei Huang <wei.huang@intel.com>
 T: git://dpdk.org/next/dpdk-next-net-intel
 F: drivers/raw/ifpga/
 F: doc/guides/rawdevs/ifpga.rst
+F: examples/ifpga/
+F: doc/guides/sample_app_ug/ifpga.rst
 
 IOAT Rawdev
 M: Bruce Richardson <bruce.richardson@intel.com>
diff --git a/doc/guides/sample_app_ug/ifpga.rst b/doc/guides/sample_app_ug/ifpga.rst
new file mode 100644
index 0000000000..bc05153418
--- /dev/null
+++ b/doc/guides/sample_app_ug/ifpga.rst
@@ -0,0 +1,387 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020-2021 Intel Corporation.
+
+Intel FPGA Sample Application
+=============================
+
+The Intel FPGA sample application is an example of how to use rte_pmd_ifpga API
+to manage Intel FPGA.
+
+Overview
+--------
+
+The Intel FPGA sample application is a simple application that demonstrates
+the use of the APIs provided by ifpga driver in the DPDK.
+A set of OPAE APIs are implemented in this example with 'opae' prefix. These
+APIs use PCI address ('Domain:Bus:Dev.Func') as an unique ID of FPGA,
+application based on OPAE can manage specified FPGA easily with them.
+OPAE (Open Programmable Acceleration Engine) is a software framework for
+managing and accessing programmable accelerators (FPGAs). For more information
+about OPAE, please refer to https://opae.github.io/2.0.0/index.html .
+This application is a readline-like interface that can be used to manage
+Intel FPGA, in a Linux* application environment.
+
+Compiling the Application
+-------------------------
+
+To compile the sample application see :doc:`compiling`
+
+The application is located in the ``ifpga`` sub-directory.
+
+Running the Application
+-----------------------
+
+To run the application in linux environment, issue the following command:
+
+.. code-block:: console
+
+    $ ./<build_dir>/examples/dpdk-ifpga --proc-type=auto
+
+Refer to the *DPDK Getting Started Guide* for general information on running
+applications and the Environment Abstraction Layer (EAL) options.
+
+Commands Description
+--------------------
+
+The following sections provide some explanation of the commands.
+
+help command
+~~~~~~~~~~~~
+
+The application has on-line help for the commands that are available at runtime.
+
+.. code-block:: console
+
+   opae> help
+    get_api_version                        get OPAE API version
+    get_proc_type                          get DPDK process type
+    get_image_info <FILE>                  get information of image file
+    get_status <BDF>                       get current status & progress of FPGA
+    get_property <BDF> <0|1|2|4|8>         get property of FPGA
+    get_phy_info <BDF>                     get information of PHY
+    get_parent <BDF>                       get parent PCI device of FPGA
+    get_child <BDF>                        get child PCI device of FPGA
+    get_pf1 <BDF>                          get physical function 1 device of FPGA
+    set_log_level <0-4>                    set logging level
+    set_log_file <FILE>                    set logging file
+    set_status <BDF> <0-4> <0-100>         set current status & progress of FPGA
+    enumerate <VID> <DID>                  enumerate specified FPGA
+    bind <BDF> <DRIVER>                    bind FPGA with kernel driver
+    unbind <BDF>                           unbind FPGA from kernel driver
+    probe <BDF>                            probe FPGA with IFPGA driver
+    remove <BDF>                           remove FPGA from IFPGA driver
+    flash <BDF> <FILE>                     update flash of FPGA
+    pr <BDF> <PORT> <FILE>                 partial reconfigure FPGA
+    reboot <BDF> <fpga|bmc> <0-1>          reboot FPGA or MAX10
+    cancel <BDF>                           cancel flash update
+    pci_read <BDF> <0-1024>                read PCI configuration space
+    pci_write <BDF> <0-1024> <NUM>         write PCI configuration space
+    quit                                   exit DPDK application
+    help                                   show commands list
+
+get_api_version command
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Show OPAE API version which is same to the version of DPDK.
+
+.. code-block:: console
+
+   opae> get_api_version
+   21.2.0
+
+set_log_level command
+~~~~~~~~~~~~~~~~~~~~~
+
+Set logging level of OPAE API. Logging level is defined as below.
+0 - CRITICAL
+1 - ERROR
+2 - WARNING
+3 - INFORMATION
+4 - DEBUG
+
+.. code-block:: console
+
+   opae> set_log_level 4
+   OPAE-API: Current log level is DEBUG
+   Successful
+   opae> set_log_level 6
+   OPAE-API: Current log level is DEBUG
+   Failed
+
+set_log_file command
+~~~~~~~~~~~~~~~~~~~~
+
+Set logging file of OPAE API.
+
+.. code-block:: console
+
+   opae> set_log_file /tmp/ifpga.log
+   Successful
+
+get_proc_type command
+~~~~~~~~~~~~~~~~~~~~~
+
+Show the process type of DPDK. If you start multiple instances of the
+application, the process type of the first one is 'Primary', the others
+are 'Secondary'.
+
+.. code-block:: console
+
+   opae> get_proc_type
+   Primary
+
+get_image_info command
+~~~~~~~~~~~~~~~~~~~~~~
+
+Display information of FPGA image file.
+
+.. code-block:: console
+
+   opae> get_image_info /home/wei/a10.bin
+   Type:           FPGA_BBS
+   Action:         UPDATE
+   Total length:   58720256
+   Payload offset: 1024
+   Payload length: 58719232
+   opae> get_image_info /home/wei/data.bin
+   OPAE-ERR: Image '/home/wei/data.bin' can not be recognized
+   Invalid image file
+
+enumerate command
+~~~~~~~~~~~~~~~~~
+
+Display PCI address of FPGA with specified vendor ID and device ID. ID value can
+be set to 0xffff for arbitrary ID.
+
+.. code-block:: console
+
+   opae> enumerate 0x8086 0x0b30
+   0000:24:00.0
+
+get_property command
+~~~~~~~~~~~~~~~~~~~~
+
+Display property information of specified FPGA. Property type is defined as below.
+0 - All properties
+1 - PCI property
+2 - FME property
+4 - port property
+8 - BMC property
+PCI property is always available, other properties can only be displayed after
+ifpga driver is probed to the FPGA.
+
+.. code-block:: console
+
+   opae> get_property 24:00.0 0
+   PCI:
+    PCIe s:b:d.f     : 0000:24:00.0
+    kernel driver    : vfio-pci
+   FME:
+    platform         : Vista Creek
+    DCP version      : DCP 1.2
+    phase            : Beta
+    interface        : 2x2x25G
+    build version    : 0.0.2
+    ports num        : 1
+    boot page        : user
+    pr interface id  : a5d72a3c-c8b0-4939-912c-f715e5dc10ca
+   PORT0:
+    access type      : PF
+    accelerator id   : 8892c23e-2eed-4b44-8bb6-5c88606e07df
+   BMC:
+    MAX10 version    : D.2.0.5
+    NIOS FW version  : D.2.0.12
+
+get_phy_info command
+~~~~~~~~~~~~~~~~~~~~
+
+Display information and status of PHY connects to the specified FPGA.
+
+.. code-block:: console
+
+   opae> get_phy_info 24:00.0
+    retimers num     : 2
+    link speed       : 25G
+    link status      : 00
+
+get_parent command
+~~~~~~~~~~~~~~~~~~
+
+Display PCI address of upstream device connects to the specified FPGA.
+
+.. code-block:: console
+
+   opae> get_parent 24:00.0
+   0000:22:09.0
+
+get_child command
+~~~~~~~~~~~~~~~~~
+
+Display PCI address of downstream device connects to the specified FPGA.
+
+.. code-block:: console
+
+   opae> get_child 24:00.0
+   No child
+   opae> get_child 22:09.0
+   0000:24:00.0
+
+get_pf1 command
+~~~~~~~~~~~~~~~
+
+Display PCI address of PF1 (physical function 1) of specified FPGA.
+
+.. code-block:: console
+
+   opae> get_pf1 24:00.0
+   0000:26:00.0
+   0000:26:00.1
+
+get_status command
+~~~~~~~~~~~~~~~~~~
+
+Display current RSU status of specified FPGA.
+
+.. code-block:: console
+
+   opae> get_status 24:00.0
+   Status:   IDLE
+   Progress: 0%
+
+set_status command
+~~~~~~~~~~~~~~~~~~
+
+Set current RSU status of specified FPGA. This command is mainly used for debug
+purpose. Status value is defined as below.
+0 - IDLE
+1 - PREPARE
+2 - PROGRAM
+3 - COPY
+4 - REBOOT
+
+.. code-block:: console
+
+   opae> set_status 24:00.0 2 35
+   Successful
+   opae> get_status 24:00.0
+   Status:   PROGRAM
+   Progress: 35%
+
+unbind command
+~~~~~~~~~~~~~~
+
+Unbind kernel driver from specified FPGA.
+
+.. code-block:: console
+
+   opae> unbind 24:00.0
+   OPAE-ERR: 0000:24:00.0 is probed, remove it first
+   Failed
+   opae> remove 24:00.0
+   Successful
+   opae> unbind 24:00.0
+   Successful
+
+bind command
+~~~~~~~~~~~~
+
+Bind specified kernel driver to specified FPGA.
+
+.. code-block:: console
+
+   opae> bind 24:00.0 vfio-pci
+   Successful
+
+probe command
+~~~~~~~~~~~~~
+
+Probe specified FPGA with DPDK PMD driver.
+
+.. code-block:: console
+
+   opae> probe 24:00.0
+   Successful
+
+remove command
+~~~~~~~~~~~~~~
+
+Remove specified FPGA from DPDK PMD driver. It's a reverse operation to probe
+command.
+
+.. code-block:: console
+
+   opae> remove 24:00.0
+   Successful
+
+flash command
+~~~~~~~~~~~~~
+
+Update image in flash of specified FPGA.
+
+.. code-block:: console
+
+   opae> flash 24:00.0 /home/wei/a10.bin
+   Successful
+
+pr command
+~~~~~~~~~~
+
+Do partial reconfiguration of specified FPGA.
+
+.. code-block:: console
+
+   opae> pr 24:00.0 0 /home/wei/nlb0.gbs
+   Successful
+
+reboot command
+~~~~~~~~~~~~~~
+
+Reboot specified FPGA. Reboot type and page is defined as below.
+fpga - reboot FPGA only
+bmc - reboot whole card with FPGA
+0 - factory page
+1 - user page
+
+.. code-block:: console
+
+   opae> reboot 24:00.0 fpga 1
+   Successful
+
+cancel command
+~~~~~~~~~~~~~~
+
+Cancel flash programming of specified FPGA.
+
+.. code-block:: console
+
+   opae> cancel 24:00.0
+   Successful
+
+pci_read command
+~~~~~~~~~~~~~~~~
+
+Read PCI configuration space of specified FPGA.
+
+.. code-block:: console
+
+   opae> pci_read 24:00.0 0
+   0x0b308086
+
+pci_write command
+~~~~~~~~~~~~~~~~~
+
+Write PCI configuration space of specified FPGA.
+
+.. code-block:: console
+
+   opae> pci_write 24:00.0 4 0x100406
+   Successful
+
+quit command
+~~~~~~~~~~~~
+
+Exit this sample application.
+
+.. code-block:: console
+
+   opae> quit
diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
index e8db83d3a7..beb94ff3e7 100644
--- a/doc/guides/sample_app_ug/index.rst
+++ b/doc/guides/sample_app_ug/index.rst
@@ -23,6 +23,7 @@ Sample Applications User Guides
     kernel_nic_interface
     keep_alive
     ioat
+    ifpga
     l2_forward_crypto
     l2_forward_job_stats
     l2_forward_real_virtual
diff --git a/examples/ifpga/Makefile b/examples/ifpga/Makefile
new file mode 100644
index 0000000000..7d76479a36
--- /dev/null
+++ b/examples/ifpga/Makefile
@@ -0,0 +1,43 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020-2021 Intel Corporation
+
+# binary name
+APP = ifpga
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c opae_api.c
+
+# Build using pkg-config variables if possible
+ifneq ($(shell pkg-config --exists libdpdk && echo 0),0)
+$(error "no installation of DPDK found")
+endif
+
+all: shared
+.PHONY: shared static
+shared: build/$(APP)-shared
+	ln -sf $(APP)-shared build/$(APP)
+static: build/$(APP)-static
+	ln -sf $(APP)-static build/$(APP)
+
+PKGCONF ?= pkg-config
+
+PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk) -lrte_bus_pci -lrte_raw_ifpga
+LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
+
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
+
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
+
+build:
+	@mkdir -p $@
+
+.PHONY: clean
+clean:
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	test -d build && rmdir -p build || true
diff --git a/examples/ifpga/commands.c b/examples/ifpga/commands.c
new file mode 100644
index 0000000000..ee58382571
--- /dev/null
+++ b/examples/ifpga/commands.c
@@ -0,0 +1,1294 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020-2021 Intel Corporation.
+ * All rights reserved.
+ */
+
+#include <stdio.h>
+#include <stdint.h>
+#include <string.h>
+#include <stdlib.h>
+#include <stdarg.h>
+#include <errno.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+
+#include "commands.h"
+
+static int parse_pciaddr(const char *bdf, opae_pci_device *id)
+{
+	size_t len = 0;
+	unsigned int domain = 0;
+	unsigned int bus = 0;
+	unsigned int devid = 0;
+	unsigned int function = 0;
+
+	if (!bdf || !id)
+		return -EINVAL;
+
+	len = strlen(bdf);
+	if ((len < 5) || (len > 12))
+		return -EINVAL;
+
+	len = sscanf(bdf, "%x:%x:%x.%d", &domain, &bus, &devid, &function);
+	if (len == 4) {
+		snprintf(id->bdf, sizeof(id->bdf), "%04x:%02x:%02x.%d",
+			domain, bus, devid, function);
+	} else {
+		len = sscanf(bdf, "%x:%x.%d", &bus, &devid, &function);
+		if (len == 3) {
+			snprintf(id->bdf, sizeof(id->bdf), "%04x:%02x:%02x.%d",
+				0, bus, devid, function);
+		} else {
+			return -EINVAL;
+		}
+	}
+	return 0;
+}
+
+static void uuid_to_str(opae_uuid *id, uuid_str *str)
+{
+	uint8_t *b = NULL;
+	char *p = NULL;
+	int i, j;
+
+	if (!id || !str)
+		return;
+
+	b = &id->b[15];
+	p = str->s;
+	for (i = 0; i < 4; i++, b--, p += 2)
+		sprintf(p, "%02x", *b);
+	sprintf(p++, "-");
+	for (i = 0; i < 3; i++) {
+		for (j = 0; j < 2; j++, b--, p += 2)
+			sprintf(p, "%02x", *b);
+		sprintf(p++, "-");
+	}
+	for (i = 0; i < 6; i++, b--, p += 2)
+		sprintf(p, "%02x", *b);
+}
+
+/* *** GET API VERSION *** */
+struct cmd_version_result {
+	cmdline_fixed_string_t cmd;
+};
+
+static void cmd_version_parsed(__rte_unused void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	opae_api_version ver;
+	opae_get_api_version(&ver);
+	cmdline_printf(cl, "%d.%d.%d\n", ver.major, ver.minor, ver.micro);
+}
+
+cmdline_parse_token_string_t cmd_version_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_version_result, cmd, "get_api_version");
+
+cmdline_parse_inst_t cmd_get_api_version = {
+	.f = cmd_version_parsed,
+	.data = NULL,
+	.help_str = "get OPAE API version",
+	.tokens = {
+		(void *)&cmd_version_cmd,
+		NULL,
+	},
+};
+
+/* *** GET PROC TYPE *** */
+struct cmd_proc_type_result {
+	cmdline_fixed_string_t cmd;
+};
+
+static void cmd_proc_type_parsed(__rte_unused void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	int type = opae_get_proc_type();
+
+	if (type == 0)
+		cmdline_printf(cl, "Primary\n");
+	else if (type == 1)
+		cmdline_printf(cl, "Secondary\n");
+	else
+		cmdline_printf(cl, "Unknown\n");
+}
+
+cmdline_parse_token_string_t cmd_proc_type_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_proc_type_result, cmd, "get_proc_type");
+
+cmdline_parse_inst_t cmd_get_proc_type = {
+	.f = cmd_proc_type_parsed,
+	.data = NULL,
+	.help_str = "get DPDK process type",
+	.tokens = {
+		(void *)&cmd_proc_type_cmd,
+		NULL,
+	},
+};
+
+/* *** GET IMAGE INFO *** */
+struct cmd_image_info_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t path;
+};
+
+static void cmd_image_info_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_image_info_result *res = parsed_result;
+	opae_img_info info;
+
+	if (opae_get_image_info(res->path, &info) == 0) {
+		cmdline_printf(cl, "%-16s", "Type:");
+		if (info.type == OPAE_IMG_TYPE_BBS)
+			cmdline_printf(cl, "FPGA_BBS\n");
+		else if (info.type == OPAE_IMG_TYPE_BMC)
+			cmdline_printf(cl, "BMC\n");
+		else if (info.type == OPAE_IMG_TYPE_GBS)
+			cmdline_printf(cl, "FGPA_GBS\n");
+		else
+			cmdline_printf(cl, "Unknown\n");
+		cmdline_printf(cl, "%-16s", "Action:");
+		if (info.subtype == OPAE_IMG_SUBTYPE_UPDATE)
+			cmdline_printf(cl, "UPDATE\n");
+		else if (info.subtype == OPAE_IMG_SUBTYPE_CANCELLATION)
+			cmdline_printf(cl, "CANCELLATION\n");
+		else if (info.subtype == OPAE_IMG_SUBTYPE_ROOT_KEY_HASH_256)
+			cmdline_printf(cl, "ROOT_HASH_256\n");
+		else if (info.subtype == OPAE_IMG_SUBTYPE_ROOT_KEY_HASH_384)
+			cmdline_printf(cl, "ROOT_HASH_384\n");
+		else
+			cmdline_printf(cl, "Unknown\n");
+		cmdline_printf(cl, "%-16s%u\n", "Total length:",
+			info.total_len);
+		cmdline_printf(cl, "%-16s%u\n", "Payload offset:",
+			info.payload_offset);
+		cmdline_printf(cl, "%-16s%u\n", "Payload length:",
+			info.payload_len);
+	} else {
+		cmdline_printf(cl, "Invalid image file\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_image_info_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_image_info_result, cmd,
+		"get_image_info");
+cmdline_parse_token_string_t cmd_image_info_path =
+	TOKEN_STRING_INITIALIZER(struct cmd_image_info_result, path, NULL);
+
+cmdline_parse_inst_t cmd_get_image_info = {
+	.f = cmd_image_info_parsed,
+	.data = NULL,
+	.help_str = "get information of image file",
+	.tokens = {
+		(void *)&cmd_image_info_cmd,
+		(void *)&cmd_image_info_path,
+		NULL,
+	},
+};
+
+/* *** GET STATUS *** */
+struct cmd_get_status_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_get_status_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_get_status_result *res = parsed_result;
+	opae_pci_device id;
+	uint32_t stat, prog;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_load_rsu_status(&id, &stat, &prog) == 0) {
+		cmdline_printf(cl, "%-10s", "Status:");
+		if (stat == 0)
+			cmdline_printf(cl, "IDLE\n");
+		else if (stat == 1)
+			cmdline_printf(cl, "PREPARE\n");
+		else if (stat == 2)
+			cmdline_printf(cl, "PROGRAM\n");
+		else if (stat == 3)
+			cmdline_printf(cl, "COPY\n");
+		else if (stat == 4)
+			cmdline_printf(cl, "REBOOT\n");
+		else
+			cmdline_printf(cl, "unknown\n");
+		cmdline_printf(cl, "%-10s%u%%\n", "Progress:", prog);
+	} else {
+		cmdline_printf(cl, "Failed\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_get_status_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_get_status_result, cmd, "get_status");
+cmdline_parse_token_string_t cmd_get_status_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_get_status_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_get_status = {
+	.f = cmd_get_status_parsed,
+	.data = NULL,
+	.help_str = "get current status & progress of FPGA",
+	.tokens = {
+		(void *)&cmd_get_status_cmd,
+		(void *)&cmd_get_status_bdf,
+		NULL,
+	},
+};
+
+/* *** GET PROPERTY *** */
+struct cmd_property_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	int32_t type;
+};
+
+static void cmd_property_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_property_result *res = parsed_result;
+	opae_pci_device id;
+	opae_fpga_property prop;
+	uuid_str str;
+	uint32_t port = 0;
+
+	switch (res->type) {
+	case 0:
+	case 1:
+	case 2:
+	case 4:
+	case 8:
+		break;
+	default:
+		cmdline_printf(cl, "%d is invalid type of property\n",
+			res->type);
+		return;
+	}
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_get_property(&id, &prop, res->type) == 0) {
+		if ((res->type == 0) || (res->type == 1)) {
+			cmdline_printf(cl, "%s:\n", "PCI");
+			cmdline_printf(cl, " %-16s : %s\n",
+				"PCIe s:b:d.f", prop.pci.pci_addr);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"kernel driver", prop.pci.drv_name);
+		}
+		if ((res->type == 0) || (res->type == 2)) {
+			cmdline_printf(cl, "%s:\n", "FME");
+			cmdline_printf(cl, " %-16s : %s\n",
+				"platform", prop.fme.platform_name);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"DCP version", prop.fme.dcp_version);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"phase", prop.fme.release_name);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"interface", prop.fme.interface_type);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"build version", prop.fme.build_version);
+			cmdline_printf(cl, " %-16s : %u\n",
+				"ports num", prop.fme.num_ports);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"boot page", prop.fme.boot_page ? "user" : "factory");
+			uuid_to_str(&prop.fme.pr_id, &str);
+			cmdline_printf(cl, " %-16s : %s\n", "pr interface id",
+				str.s);
+		}
+		if ((res->type == 0) || (res->type == 4)) {
+			for (port = 0; port < prop.fme.num_ports; port++) {
+				cmdline_printf(cl, "%s%d:\n", "PORT", port);
+				cmdline_printf(cl, " %-16s : %s\n",
+					"access type",
+					prop.port[port].type ? "VF" : "PF");
+				uuid_to_str(&prop.port[port].afu_id, &str);
+				cmdline_printf(cl, " %-16s : %s\n",
+					"accelerator id", str.s);
+			}
+		}
+		if ((res->type == 0) || (res->type == 8)) {
+			cmdline_printf(cl, "%s:\n", "BMC");
+			cmdline_printf(cl, " %-16s : %s\n",
+				"MAX10 version", prop.bmc.bmc_version);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"NIOS FW version", prop.bmc.fw_version);
+		}
+	} else {
+		cmdline_printf(cl, "Failed\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_property_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_property_result, cmd, "get_property");
+cmdline_parse_token_string_t cmd_property_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_property_result, bdf, NULL);
+cmdline_parse_token_num_t cmd_property_type =
+	TOKEN_NUM_INITIALIZER(struct cmd_property_result, type, RTE_INT32);
+
+cmdline_parse_inst_t cmd_get_property = {
+	.f = cmd_property_parsed,
+	.data = NULL,
+	.help_str = "get property of FPGA",
+	.tokens = {
+		(void *)&cmd_property_cmd,
+		(void *)&cmd_property_bdf,
+		(void *)&cmd_property_type,
+		NULL,
+	},
+};
+
+/* *** GET PHY INFO *** */
+struct cmd_phy_info_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_phy_info_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_phy_info_result *res = parsed_result;
+	opae_pci_device id;
+	opae_phy_info info;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_get_phy_info(&id, &info) == 0) {
+		cmdline_printf(cl, " %-16s : %u\n",
+			"retimers num", info.num_retimers);
+		cmdline_printf(cl, " %-16s : %uG\n",
+			"link speed", info.link_speed);
+		cmdline_printf(cl, " %-16s : %02xh\n",
+			"link status", info.link_status);
+	} else {
+		cmdline_printf(cl, "Failed\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_phy_info_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_phy_info_result, cmd, "get_phy_info");
+cmdline_parse_token_string_t cmd_phy_info_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_phy_info_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_phy_info = {
+	.f = cmd_phy_info_parsed,
+	.data = NULL,
+	.help_str = "get information of PHY",
+	.tokens = {
+		(void *)&cmd_phy_info_cmd,
+		(void *)&cmd_phy_info_bdf,
+		NULL,
+	},
+};
+
+/* *** GET PARENT *** */
+struct cmd_parent_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_parent_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_parent_result *res = parsed_result;
+	opae_pci_device id;
+	opae_pci_device parent;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_get_parent(&id, &parent) > 0)
+		cmdline_printf(cl, "%s\n", parent.bdf);
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_parent_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_parent_result, cmd, "get_parent");
+cmdline_parse_token_string_t cmd_parent_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_parent_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_get_parent = {
+	.f = cmd_parent_parsed,
+	.data = NULL,
+	.help_str = "get parent PCI device of FPGA",
+	.tokens = {
+		(void *)&cmd_parent_cmd,
+		(void *)&cmd_parent_bdf,
+		NULL,
+	},
+};
+
+/* *** GET CHILD *** */
+struct cmd_child_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_child_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_child_result *res = parsed_result;
+	opae_pci_device id;
+	pcidev_id child;
+	int i, count = 0;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	count = opae_get_child(&id, NULL, 0);
+	if (count > 0) {
+		child = (pcidev_id)malloc(sizeof(opae_pci_device) * count);
+		if (child) {
+			opae_get_child(&id, child, count);
+			for (i = 0; i < count; i++)
+				cmdline_printf(cl, "%s\n", child[i].bdf);
+			free(child);
+		} else {
+			cmdline_printf(cl, "No memory\n");
+		}
+	} else if (count == 0) {
+		cmdline_printf(cl, "No child\n");
+	} else {
+		cmdline_printf(cl, "Failed\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_child_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_child_result, cmd, "get_child");
+cmdline_parse_token_string_t cmd_child_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_child_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_get_child = {
+	.f = cmd_child_parsed,
+	.data = NULL,
+	.help_str = "get child PCI device of FPGA",
+	.tokens = {
+		(void *)&cmd_child_cmd,
+		(void *)&cmd_child_bdf,
+		NULL,
+	},
+};
+
+/* *** GET PF1 *** */
+struct cmd_pf1_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_pf1_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_pf1_result *res = parsed_result;
+	opae_pci_device id;
+	pcidev_id peer;
+	int i, count = 0;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	count = opae_get_pf1(&id, NULL, 0);
+	if (count > 0) {
+		peer = (pcidev_id)malloc(sizeof(opae_pci_device) * count);
+		if (peer) {
+			opae_get_pf1(&id, peer, count);
+			for (i = 0; i < count; i++)
+				cmdline_printf(cl, "%s\n", peer[i].bdf);
+			free(peer);
+		} else {
+			cmdline_printf(cl, "No memory\n");
+		}
+	} else if (count == 0) {
+		cmdline_printf(cl, "No PF1\n");
+	} else {
+		cmdline_printf(cl, "Failed\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_pf1_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_pf1_result, cmd, "get_pf1");
+cmdline_parse_token_string_t cmd_pf1_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_pf1_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_get_pf1 = {
+	.f = cmd_pf1_parsed,
+	.data = NULL,
+	.help_str = "get physical function 1 device of FPGA",
+	.tokens = {
+		(void *)&cmd_pf1_cmd,
+		(void *)&cmd_pf1_bdf,
+		NULL,
+	},
+};
+
+/* *** SET LOG LEVEL *** */
+struct cmd_log_level_result {
+	cmdline_fixed_string_t cmd;
+	int32_t level;
+};
+
+static void cmd_log_level_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_log_level_result *res = parsed_result;
+	if (opae_set_log_level(res->level) == res->level)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_log_level_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_log_level_result, cmd, "set_log_level");
+cmdline_parse_token_num_t cmd_log_level_level =
+	TOKEN_NUM_INITIALIZER(struct cmd_log_level_result, level, RTE_INT32);
+
+cmdline_parse_inst_t cmd_set_log_level = {
+	.f = cmd_log_level_parsed,
+	.data = NULL,
+	.help_str = "set logging level",
+	.tokens = {
+		(void *)&cmd_log_level_cmd,
+		(void *)&cmd_log_level_level,
+		NULL,
+	},
+};
+
+/* *** SET LOG FILE *** */
+struct cmd_log_file_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t path;
+};
+
+static void cmd_log_file_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_log_file_result *res = parsed_result;
+	if (opae_set_log_file(res->path, 1) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_log_file_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_log_file_result, cmd, "set_log_file");
+cmdline_parse_token_string_t cmd_log_file_path =
+	TOKEN_STRING_INITIALIZER(struct cmd_log_file_result, path, NULL);
+
+cmdline_parse_inst_t cmd_set_log_file = {
+	.f = cmd_log_file_parsed,
+	.data = NULL,
+	.help_str = "set logging file",
+	.tokens = {
+		(void *)&cmd_log_file_cmd,
+		(void *)&cmd_log_file_path,
+		NULL,
+	},
+};
+
+/* *** SET STATUS *** */
+struct cmd_set_status_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	uint32_t stat;
+	uint32_t prog;
+};
+
+static void cmd_set_status_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_set_status_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+	if ((res->stat > 4) || (res->prog > 100)) {
+		cmdline_printf(cl, "%u,%u is invalid status\n", res->stat,
+			res->prog);
+		return;
+	}
+
+	if (opae_store_rsu_status(&id, res->stat, res->prog) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_set_status_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_set_status_result, cmd, "set_status");
+cmdline_parse_token_string_t cmd_set_status_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_set_status_result, bdf, NULL);
+cmdline_parse_token_num_t cmd_set_status_stat =
+	TOKEN_NUM_INITIALIZER(struct cmd_set_status_result, stat, RTE_UINT32);
+cmdline_parse_token_num_t cmd_set_status_prog =
+	TOKEN_NUM_INITIALIZER(struct cmd_set_status_result, prog, RTE_UINT32);
+
+cmdline_parse_inst_t cmd_set_status = {
+	.f = cmd_set_status_parsed,
+	.data = NULL,
+	.help_str = "set current status & progress of FPGA",
+	.tokens = {
+		(void *)&cmd_set_status_cmd,
+		(void *)&cmd_set_status_bdf,
+		(void *)&cmd_set_status_stat,
+		(void *)&cmd_set_status_prog,
+		NULL,
+	},
+};
+
+/* *** ENUMERATE *** */
+struct cmd_enumerate_result {
+	cmdline_fixed_string_t cmd;
+	uint32_t vid;
+	uint32_t did;
+};
+
+static void cmd_enumerate_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_enumerate_result *res = parsed_result;
+	opae_pci_id filter;
+	opae_pci_device *id;
+	int i, count = 0;
+
+	filter.vendor_id = res->vid;
+	filter.device_id = res->did;
+	filter.class_id = BIT_SET_32;
+	filter.subsystem_vendor_id = BIT_SET_16;
+	filter.subsystem_device_id = BIT_SET_16;
+
+	count = opae_enumerate(&filter, NULL, 0);
+	if (count > 0) {
+		id = (opae_pci_device *)malloc(sizeof(opae_pci_device) * count);
+		if (id) {
+			opae_enumerate(&filter, id, count);
+			for (i = 0; i < count; i++)
+				cmdline_printf(cl, "%s\n", id[i].bdf);
+			free(id);
+		} else {
+			cmdline_printf(cl, "No memory\n");
+		}
+	} else if (count == 0) {
+		cmdline_printf(cl, "Not found\n");
+	} else {
+		cmdline_printf(cl, "Failed\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_enumerate_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_enumerate_result, cmd, "enumerate");
+cmdline_parse_token_num_t cmd_enumerate_vid =
+	TOKEN_NUM_INITIALIZER(struct cmd_enumerate_result, vid, RTE_UINT32);
+cmdline_parse_token_num_t cmd_enumerate_did =
+	TOKEN_NUM_INITIALIZER(struct cmd_enumerate_result, did, RTE_UINT32);
+
+cmdline_parse_inst_t cmd_enumerate = {
+	.f = cmd_enumerate_parsed,
+	.data = NULL,
+	.help_str = "enumerate specified FPGA",
+	.tokens = {
+		(void *)&cmd_enumerate_cmd,
+		(void *)&cmd_enumerate_vid,
+		(void *)&cmd_enumerate_did,
+		NULL,
+	},
+};
+
+/* *** BIND *** */
+struct cmd_bind_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	cmdline_fixed_string_t drv;
+};
+
+static void cmd_bind_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_bind_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_bind_driver(&id, res->drv) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_bind_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_bind_result, cmd, "bind");
+cmdline_parse_token_string_t cmd_bind_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_bind_result, bdf, NULL);
+cmdline_parse_token_string_t cmd_bind_drv =
+	TOKEN_STRING_INITIALIZER(struct cmd_bind_result, drv, NULL);
+
+cmdline_parse_inst_t cmd_bind = {
+	.f = cmd_bind_parsed,
+	.data = NULL,
+	.help_str = "bind FPGA with kernel driver",
+	.tokens = {
+		(void *)&cmd_bind_cmd,
+		(void *)&cmd_bind_bdf,
+		(void *)&cmd_bind_drv,
+		NULL,
+	},
+};
+
+/* *** UNBIND *** */
+struct cmd_unbind_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_unbind_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_unbind_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_unbind_driver(&id) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_unbind_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_bind_result, cmd, "unbind");
+cmdline_parse_token_string_t cmd_unbind_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_bind_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_unbind = {
+	.f = cmd_unbind_parsed,
+	.data = NULL,
+	.help_str = "unbind FPGA from kernel driver",
+	.tokens = {
+		(void *)&cmd_unbind_cmd,
+		(void *)&cmd_unbind_bdf,
+		NULL,
+	},
+};
+
+/* *** PROBE *** */
+struct cmd_probe_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_probe_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_probe_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_probe_device(&id) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_probe_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_probe_result, cmd, "probe");
+cmdline_parse_token_string_t cmd_probe_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_probe_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_probe = {
+	.f = cmd_probe_parsed,
+	.data = NULL,
+	.help_str = "probe FPGA with IFPGA driver",
+	.tokens = {
+		(void *)&cmd_probe_cmd,
+		(void *)&cmd_probe_bdf,
+		NULL,
+	},
+};
+
+/* *** REMOVE *** */
+struct cmd_remove_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_remove_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_remove_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_remove_device(&id) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_remove_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_remove_result, cmd, "remove");
+cmdline_parse_token_string_t cmd_remove_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_remove_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_remove = {
+	.f = cmd_remove_parsed,
+	.data = NULL,
+	.help_str = "remove FPGA from IFPGA driver",
+	.tokens = {
+		(void *)&cmd_remove_cmd,
+		(void *)&cmd_remove_bdf,
+		NULL,
+	},
+};
+
+/* *** FLASH *** */
+struct cmd_flash_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	cmdline_fixed_string_t path;
+};
+
+static void cmd_flash_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_flash_result *res = parsed_result;
+	opae_pci_device id;
+	uint64_t stat = 0;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_update_flash(&id, res->path, &stat))
+		cmdline_printf(cl, "Error: 0x%lx\n", stat);
+}
+
+cmdline_parse_token_string_t cmd_flash_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_flash_result, cmd, "flash");
+cmdline_parse_token_string_t cmd_flash_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_flash_result, bdf, NULL);
+cmdline_parse_token_string_t cmd_flash_path =
+	TOKEN_STRING_INITIALIZER(struct cmd_flash_result, path, NULL);
+
+cmdline_parse_inst_t cmd_flash = {
+	.f = cmd_flash_parsed,
+	.data = NULL,
+	.help_str = "update flash of FPGA",
+	.tokens = {
+		(void *)&cmd_flash_cmd,
+		(void *)&cmd_flash_bdf,
+		(void *)&cmd_flash_path,
+		NULL,
+	},
+};
+
+/* *** PR *** */
+struct cmd_pr_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	int32_t port;
+	cmdline_fixed_string_t path;
+};
+
+static void cmd_pr_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_pr_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_partial_reconfigure(&id, res->port, res->path) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_pr_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_pr_result, cmd, "pr");
+cmdline_parse_token_string_t cmd_pr_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_pr_result, bdf, NULL);
+cmdline_parse_token_num_t cmd_pr_port =
+	TOKEN_NUM_INITIALIZER(struct cmd_pr_result, port, RTE_INT32);
+cmdline_parse_token_string_t cmd_pr_path =
+	TOKEN_STRING_INITIALIZER(struct cmd_pr_result, path, NULL);
+
+cmdline_parse_inst_t cmd_pr = {
+	.f = cmd_pr_parsed,
+	.data = NULL,
+	.help_str = "partial reconfigure FPGA",
+	.tokens = {
+		(void *)&cmd_pr_cmd,
+		(void *)&cmd_pr_bdf,
+		(void *)&cmd_pr_port,
+		(void *)&cmd_pr_path,
+		NULL,
+	},
+};
+
+/* *** REBOOT *** */
+struct cmd_reboot_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	cmdline_fixed_string_t type;
+	int32_t page;
+};
+
+static void cmd_reboot_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_reboot_result *res = parsed_result;
+	opae_pci_device id;
+	int type = 0;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (!strcmp(res->type, "fpga")) {
+		type = 0;
+	} else if (!strcmp(res->type, "bmc")) {
+		type = 1;
+	} else {
+		cmdline_printf(cl, "%s is invalid reboot type\n", res->type);
+		return;
+	}
+
+	if (opae_reboot_device(&id, type, res->page) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_reboot_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_reboot_result, cmd, "reboot");
+cmdline_parse_token_string_t cmd_reboot_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_reboot_result, bdf, NULL);
+cmdline_parse_token_string_t cmd_reboot_type =
+	TOKEN_STRING_INITIALIZER(struct cmd_reboot_result, type, NULL);
+cmdline_parse_token_num_t cmd_reboot_page =
+	TOKEN_NUM_INITIALIZER(struct cmd_reboot_result, page, RTE_INT32);
+
+cmdline_parse_inst_t cmd_reboot = {
+	.f = cmd_reboot_parsed,
+	.data = NULL,
+	.help_str = "reboot FPGA or MAX10",
+	.tokens = {
+		(void *)&cmd_reboot_cmd,
+		(void *)&cmd_reboot_bdf,
+		(void *)&cmd_reboot_type,
+		(void *)&cmd_reboot_page,
+		NULL,
+	},
+};
+
+/* *** CANCEL *** */
+struct cmd_cancel_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_cancel_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_cancel_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_cancel_flash_update(&id, 0) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_cancel_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_cancel_result, cmd, "cancel");
+cmdline_parse_token_string_t cmd_cancel_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_cancel_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_cancel = {
+	.f = cmd_cancel_parsed,
+	.data = NULL,
+	.help_str = "cancel flash update",
+	.tokens = {
+		(void *)&cmd_cancel_cmd,
+		(void *)&cmd_cancel_bdf,
+		NULL,
+	},
+};
+
+/* *** PCI READ *** */
+struct cmd_pci_read_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	uint32_t offset;
+};
+
+static void cmd_pci_read_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_pci_read_result *res = parsed_result;
+	opae_pci_device id;
+	uint32_t offset = 0;
+	uint32_t value = 0;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (res->offset & 0x3) {
+		offset = res->offset & ~3;
+		cmdline_printf(cl, "align offset to 0x%x\n", offset);
+	} else {
+		offset = res->offset;
+	}
+
+	if (opae_read_pci_cfg(&id, offset, &value) == 0)
+		cmdline_printf(cl, "0x%08x\n", value);
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_pci_read_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_pci_read_result, cmd, "pci_read");
+cmdline_parse_token_string_t cmd_pci_read_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_pci_read_result, bdf, NULL);
+cmdline_parse_token_num_t cmd_pci_read_offset =
+	TOKEN_NUM_INITIALIZER(struct cmd_pci_read_result, offset, RTE_UINT32);
+
+cmdline_parse_inst_t cmd_pci_read = {
+	.f = cmd_pci_read_parsed,
+	.data = NULL,
+	.help_str = "read PCI configuration space",
+	.tokens = {
+		(void *)&cmd_pci_read_cmd,
+		(void *)&cmd_pci_read_bdf,
+		(void *)&cmd_pci_read_offset,
+		NULL,
+	},
+};
+
+/* *** PCI WRITE *** */
+struct cmd_pci_write_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	uint32_t offset;
+	uint32_t value;
+};
+
+static void cmd_pci_write_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_pci_write_result *res = parsed_result;
+	opae_pci_device id;
+	uint32_t offset = 0;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (res->offset & 0x3) {
+		offset = res->offset & ~3;
+		cmdline_printf(cl, "align offset to 0x%x\n", offset);
+	} else {
+		offset = res->offset;
+	}
+
+	if (opae_write_pci_cfg(&id, offset, res->value) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_pci_write_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_pci_write_result, cmd, "pci_write");
+cmdline_parse_token_string_t cmd_pci_write_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_pci_write_result, bdf, NULL);
+cmdline_parse_token_num_t cmd_pci_write_offset =
+	TOKEN_NUM_INITIALIZER(struct cmd_pci_write_result, offset, RTE_UINT32);
+cmdline_parse_token_num_t cmd_pci_write_value =
+	TOKEN_NUM_INITIALIZER(struct cmd_pci_write_result, value, RTE_UINT32);
+
+cmdline_parse_inst_t cmd_pci_write = {
+	.f = cmd_pci_write_parsed,
+	.data = NULL,
+	.help_str = "write PCI configuration space",
+	.tokens = {
+		(void *)&cmd_pci_write_cmd,
+		(void *)&cmd_pci_write_bdf,
+		(void *)&cmd_pci_write_offset,
+		(void *)&cmd_pci_write_value,
+		NULL,
+	},
+};
+
+/* *** QUIT *** */
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__rte_unused void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,
+	.data = NULL,
+	.help_str = "exit DPDK application",
+	.tokens = {
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/* *** HELP *** */
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__rte_unused void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	cmdline_printf(cl,
+		" get_api_version               \t\t"
+			"get OPAE API version\n"
+		" get_proc_type                 \t\t"
+			"get DPDK process type\n"
+		" get_image_info <FILE>         \t\t"
+			"get information of image file\n"
+		" get_status <BDF>              \t\t"
+			"get current status & progress of FPGA\n"
+		" get_property <BDF> <0|1|2|4|8>\t\t"
+			"get property of FPGA\n"
+		" get_phy_info <BDF>            \t\t"
+			"get information of PHY\n"
+		" get_parent <BDF>              \t\t"
+			"get parent PCI device of FPGA\n"
+		" get_child <BDF>               \t\t"
+			"get child PCI device of FPGA\n"
+		" get_pf1 <BDF>                 \t\t"
+			"get physical function 1 device of FPGA\n"
+		" set_log_level <0-4>           \t\t"
+			"set logging level\n"
+		" set_log_file <FILE>           \t\t"
+			"set logging file\n"
+		" set_status <BDF> <0-4> <0-100>\t\t"
+			"set current status & progress of FPGA\n"
+		" enumerate <VID> <DID>         \t\t"
+			"enumerate specified FPGA\n"
+		" bind <BDF> <DRIVER>           \t\t"
+			"bind FPGA with kernel driver\n"
+		" unbind <BDF>                  \t\t"
+			"unbind FPGA from kernel driver\n"
+		" probe <BDF>                   \t\t"
+			"probe FPGA with IFPGA driver\n"
+		" remove <BDF>                  \t\t"
+			"remove FPGA from IFPGA driver\n"
+		" flash <BDF> <FILE>            \t\t"
+			"update flash of FPGA\n"
+		" pr <BDF> <PORT> <FILE>        \t\t"
+			"partial reconfigure FPGA\n"
+		" reboot <BDF> <fpga|bmc> <0-1> \t\t"
+			"reboot FPGA or MAX10\n"
+		" cancel <BDF>                  \t\t"
+			"cancel flash update\n"
+		" pci_read <BDF> <0-1024>       \t\t"
+			"read PCI configuration space\n"
+		" pci_write <BDF> <0-1024> <NUM>\t\t"
+			"write PCI configuration space\n"
+		" quit                          \t\t"
+			"exit DPDK application\n"
+		" help                          \t\t"
+			"show commands list\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,
+	.data = NULL,
+	.help_str = "show commands list",
+	.tokens = {
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/****** CONTEXT (list of commands) */
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_get_image_info,
+	(cmdline_parse_inst_t *)&cmd_get_api_version,
+	(cmdline_parse_inst_t *)&cmd_get_proc_type,
+	(cmdline_parse_inst_t *)&cmd_get_status,
+	(cmdline_parse_inst_t *)&cmd_get_property,
+	(cmdline_parse_inst_t *)&cmd_phy_info,
+	(cmdline_parse_inst_t *)&cmd_get_parent,
+	(cmdline_parse_inst_t *)&cmd_get_child,
+	(cmdline_parse_inst_t *)&cmd_get_pf1,
+	(cmdline_parse_inst_t *)&cmd_set_log_level,
+	(cmdline_parse_inst_t *)&cmd_set_log_file,
+	(cmdline_parse_inst_t *)&cmd_set_status,
+	(cmdline_parse_inst_t *)&cmd_enumerate,
+	(cmdline_parse_inst_t *)&cmd_bind,
+	(cmdline_parse_inst_t *)&cmd_unbind,
+	(cmdline_parse_inst_t *)&cmd_probe,
+	(cmdline_parse_inst_t *)&cmd_remove,
+	(cmdline_parse_inst_t *)&cmd_flash,
+	(cmdline_parse_inst_t *)&cmd_pr,
+	(cmdline_parse_inst_t *)&cmd_reboot,
+	(cmdline_parse_inst_t *)&cmd_cancel,
+	(cmdline_parse_inst_t *)&cmd_pci_read,
+	(cmdline_parse_inst_t *)&cmd_pci_write,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_help,
+	NULL,
+};
diff --git a/examples/ifpga/commands.h b/examples/ifpga/commands.h
new file mode 100644
index 0000000000..06fe9a68b2
--- /dev/null
+++ b/examples/ifpga/commands.h
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020-2021 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+#include "opae_api.h"
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+typedef struct {
+	char s[38];
+} uuid_str;
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/ifpga/main.c b/examples/ifpga/main.c
new file mode 100644
index 0000000000..e9380d581a
--- /dev/null
+++ b/examples/ifpga/main.c
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020-2021 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+
+int main(int argc, char **argv)
+{
+	struct cmdline *cl;
+	int ret;
+
+	ret = opae_init_eal(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+	cl = cmdline_stdin_new(main_ctx, "opae> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	opae_cleanup_eal();
+	cmdline_stdin_exit(cl);
+	return 0;
+}
diff --git a/examples/ifpga/meson.build b/examples/ifpga/meson.build
new file mode 100644
index 0000000000..b88a37a7a5
--- /dev/null
+++ b/examples/ifpga/meson.build
@@ -0,0 +1,20 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020-2021 Intel Corporation
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+# require the raw_ifpga library
+build = dpdk_conf.has('RTE_RAW_IFPGA')
+if not build
+	subdir_done()
+endif
+
+deps += 'raw_ifpga'
+allow_experimental_apis = true
+sources = files(
+	'main.c', 'commands.c', 'opae_api.c'
+)
+cflags += '-fPIC'
diff --git a/examples/ifpga/opae_api.c b/examples/ifpga/opae_api.c
new file mode 100644
index 0000000000..7c27a7a278
--- /dev/null
+++ b/examples/ifpga/opae_api.c
@@ -0,0 +1,1630 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <stdlib.h>
+#include <string.h>
+#include <dirent.h>
+#include <fcntl.h>
+#include <glob.h>
+#include <unistd.h>
+#include <sys/stat.h>
+#include <rte_eal.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+#include <rte_rawdev_pmd.h>
+#include "rte_pmd_ifpga.h"
+#include "opae_api.h"
+
+
+int opae_log_level;
+FILE *opae_log_file;
+
+static opae_api_version api_ver = {21, 5, 0};
+static int eal_inited;
+static uint32_t dev_aer[2] = {0};
+
+static const char * const log_level_name[] = {"CRITICAL", "ERROR",
+	"WARNING", "INFORMATION", "DEBUG"};
+static const char * const proc_type_name[] = {"NON-DPDK", "PRIMARY",
+	"SECONDARY"};
+static const char * const platform_name[] = {"Vista Creek", "Rush Creek",
+	"Darby Creek", "Lightning Creek"};
+static const char * const release_name[] = {"Pre-Alpha", "Alpha", "Beta", "PV"};
+static const char * const interface_type[] = {"8x10G", "4x25G", "2x1x25G",
+	"4x25G+2x25G", "2x2x25G", "2x1x25Gx2FVL", "1x2x25G"};
+
+void opae_get_api_version(opae_api_version *version)
+{
+	if (version)
+		memcpy(version, &api_ver, sizeof(opae_api_version));
+	opae_log_info("API version is %u.%u.%u\n",
+		api_ver.major, api_ver.minor, api_ver.micro);
+}
+
+int opae_set_log_level(int level)
+{
+	if ((level >= OPAE_LOG_API) && (level <= OPAE_LOG_DEBUG))
+		opae_log_level = level;
+	opae_log_api("Current log level is %s\n",
+		log_level_name[opae_log_level]);
+	return opae_log_level;
+}
+
+int opae_set_log_file(char *path, int clean)
+{
+	FILE *f = NULL;
+	time_t start;
+	struct tm *lt = NULL;
+
+	if (path) {
+		if (clean)
+			f = fopen(path, "w+");
+		else
+			f = fopen(path, "a+");
+
+		if (f) {
+			if (opae_log_file) {
+				fclose(opae_log_file);
+				opae_log_file = NULL;
+			}
+			time(&start);
+			lt = localtime(&start);
+			if (lt)
+				fprintf(f, "================%d-%02d-%02d "
+					"%02d:%02d:%02d================\n",
+					1900 + lt->tm_year, 1 + lt->tm_mon,
+					lt->tm_mday,
+					lt->tm_hour, lt->tm_min, lt->tm_sec);
+			fflush(f);
+			opae_log_file = f;
+		} else {
+			opae_log_err("failed to open log file \'%s\'\n", path);
+			return -1;
+		}
+	} else {
+		if (opae_log_file) {
+			fclose(opae_log_file);
+			opae_log_file = NULL;
+		}
+	}
+
+	return 0;
+}
+
+int opae_get_image_info(const char *image, opae_img_info *info)
+{
+	int fd = -1;
+	off_t file_size = 0;
+	opae_img_hdr hdr;
+	ssize_t read_size = 0;
+	int ret = 0;
+
+	if (!image || !info) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	fd = open(image, O_RDONLY);
+	if (fd < 0) {
+		opae_log_err("Failed to open \'%s\' for RD [e:%s]\n",
+			image, strerror(errno));
+		return -EIO;
+	}
+
+	file_size = lseek(fd, 0, SEEK_END);
+	if (file_size < (OPAE_IMG_HDR_SIZE + OPAE_IMG_PL_MIN_SIZE)) {
+		opae_log_err("Size of \'%s\' is less than expected [e:%u]\n",
+			image, OPAE_IMG_HDR_SIZE + OPAE_IMG_PL_MIN_SIZE);
+		ret = -EINVAL;
+		goto close_fd;
+	}
+
+	/* read image header */
+	lseek(fd, 0, SEEK_SET);
+	read_size = read(fd, (void *)&hdr, sizeof(opae_img_hdr));
+	if (read_size < 0) {
+		opae_log_err("Failed to read from \'%s\' [e:%s]\n",
+			image, strerror(errno));
+		ret = -EIO;
+		goto close_fd;
+	}
+	if ((size_t)read_size != sizeof(opae_img_hdr)) {
+		opae_log_err("Read length %zd is not expected [e:%zu]\n",
+			read_size, sizeof(opae_img_hdr));
+		ret = -EIO;
+		goto close_fd;
+	}
+
+	info->total_len = file_size;
+	/* check signed image header */
+	if (hdr.magic == OPAE_IMG_BLK0_MAGIC) {
+		info->type = OPAE_IMG_TYPE(hdr.payload_type);
+		info->subtype = OPAE_IMG_SUBTYPE(hdr.payload_type);
+		info->payload_offset = OPAE_IMG_HDR_SIZE;
+		info->payload_len = hdr.payload_len;
+	} else {
+		opae_log_err("Image \'%s\' can not be recognized\n", image);
+		ret = -EINVAL;
+	}
+close_fd:
+	close(fd);
+	return ret;
+}
+
+static int write_file(char *path, char *buf, int size)
+{
+	int fd = -1;
+	ssize_t n = 0;
+
+	if (!path || !buf || (size <= 0))
+		return -EINVAL;
+
+	fd = open(path, O_WRONLY);
+	if (fd < 0) {
+		opae_log_err("Failed to open \'%s\' for WR [e:%s]\n",
+			path, strerror(errno));
+		return -EIO;
+	}
+	opae_log_dbg("Write \"%s\" to \'%s\'\n", buf, path);
+	n = write(fd, buf, size);
+	if (n < size)  {
+		opae_log_err("Failed to write to \'%s\' [e:%s]\n",
+			path, strerror(errno));
+		close(fd);
+		return -EIO;
+	}
+	close(fd);
+
+	return 0;
+}
+
+static int read_file(char *path, char *buf, int size)
+{
+	int fd = -1;
+	ssize_t n = 0;
+
+	if (!path || !buf || (size <= 0))
+		return -EINVAL;
+
+	fd = open(path, O_RDONLY);
+	if (fd < 0) {
+		opae_log_err("Failed to open \'%s\' for RD [e:%s]\n",
+			path, strerror(errno));
+		return -EIO;
+	}
+	n = read(fd, buf, size);
+	if (n < 0)  {
+		opae_log_err("Failed to read from \'%s\' [e:%s]\n",
+			path, strerror(errno));
+		close(fd);
+		return -EIO;
+	}
+	close(fd);
+
+	if (n > 0)
+		buf[n-1] = 0;
+
+	opae_log_dbg("Read \"%s\" from \'%s\'\n", buf, path);
+	return 0;
+}
+
+int opae_get_proc_type(void)
+{
+	int type = -1;
+
+	if (eal_inited) {
+		if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+			type = 0;
+		else
+			type = 1;
+	}
+	opae_log_info("Current process type is %s\n", proc_type_name[type+1]);
+
+	return type;
+}
+
+static bool check_eal(int inited)
+{
+	if (!eal_inited) {
+		if (inited) {
+			opae_log_warn("EAL is not initialized\n");
+			return 0;
+		}
+	} else {
+		if (!inited) {
+			opae_log_warn("EAL is already initialized\n");
+			return 0;
+		}
+	}
+
+	return 1;
+}
+
+int opae_init_eal(int argc, char **argv)
+{
+	int ret = 0;
+
+	if (!check_eal(0))
+		return ret;
+
+	opae_log_level = OPAE_LOG_ERR;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0) {
+		if (rte_errno == EALREADY) {
+			eal_inited = 1;
+			return 0;
+		}
+		opae_log_err("Cannot initialize EAL [e:%d]\n", ret);
+		if (rte_eal_cleanup())
+			opae_log_warn("EAL could not release all resources\n");
+	} else {
+		eal_inited = 1;
+		opae_log_info("Initialize EAL done\n");
+	}
+
+	return ret;
+}
+
+int opae_cleanup_eal(void)
+{
+	int ret = 0;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	rte_pmd_ifpga_cleanup();
+
+	ret = rte_eal_cleanup();
+	if (ret)
+		opae_log_err("Failed to cleanup EAL [e:%d]\n", ret);
+
+	if (opae_log_file) {
+		fclose(opae_log_file);
+		opae_log_file = NULL;
+	}
+
+	return ret;
+}
+
+static int compare_pci_id(opae_pci_id *id, opae_pci_id *expected_id)
+{
+	if ((expected_id->class_id != BIT_SET_32) &&
+		(expected_id->class_id != id->class_id))
+		return -1;
+	if ((expected_id->vendor_id != BIT_SET_16) &&
+		(expected_id->vendor_id != id->vendor_id))
+		return -1;
+	if ((expected_id->device_id != BIT_SET_16) &&
+		(expected_id->device_id != id->device_id))
+		return -1;
+	if ((expected_id->subsystem_vendor_id != BIT_SET_16) &&
+		(expected_id->subsystem_vendor_id != id->subsystem_vendor_id))
+		return -1;
+	if ((expected_id->subsystem_device_id != BIT_SET_16) &&
+		(expected_id->subsystem_device_id != id->subsystem_device_id))
+		return -1;
+
+	return 0;
+}
+
+static int parse_sysfs_value(char *node, uint32_t *val)
+{
+	char buf[16];
+	char *end = NULL;
+	int ret = 0;
+
+	ret = read_file(node, buf, sizeof(buf));
+	if (ret < 0)
+		return ret;
+
+	*val = (uint32_t)strtoul(buf, &end, 0);
+	return 0;
+}
+
+static int get_pci_id(const char *dev_path, opae_pci_id *id)
+{
+	char path[PATH_MAX] = {0};
+	uint32_t tmp;
+
+	if (!dev_path || !id)
+		return -EINVAL;
+
+	snprintf(path, sizeof(path), "%s/vendor", dev_path);
+	if (parse_sysfs_value(path, &tmp) < 0)
+		return -ENODEV;
+	id->vendor_id = (uint16_t)tmp;
+
+	snprintf(path, sizeof(path), "%s/device", dev_path);
+	if (parse_sysfs_value(path, &tmp) < 0)
+		return -ENODEV;
+	id->device_id = (uint16_t)tmp;
+
+	snprintf(path, sizeof(path), "%s/subsystem_vendor", dev_path);
+	if (parse_sysfs_value(path, &tmp) < 0)
+		return -ENODEV;
+	id->subsystem_vendor_id = (uint16_t)tmp;
+
+	snprintf(path, sizeof(path), "%s/subsystem_device", dev_path);
+	if (parse_sysfs_value(path, &tmp) < 0)
+		return -ENODEV;
+	id->subsystem_device_id = (uint16_t)tmp;
+
+	snprintf(path, sizeof(path), "%s/class", dev_path);
+	if (parse_sysfs_value(path, &tmp) < 0)
+		return -ENODEV;
+	id->class_id = (uint32_t)tmp & RTE_CLASS_ANY_ID;
+
+	return 0;
+}
+
+static int extract_path(char *in, int ridx, char *out, uint32_t size)
+{
+	char src[PATH_MAX] = {0};
+	char *p = NULL;
+	int ret = 0;
+
+	if (!in || (strlen(in) > PATH_MAX) || (ridx < 0) || !out)
+		return -EINVAL;
+
+	strncpy(src, in, sizeof(src));
+	*out = 0;
+
+	while (1) {
+		p = strrchr(src, '/');
+		if (p) {
+			*p++ = 0;
+			if (*p) {
+				if (ridx-- <= 0) {
+					if (size > strlen(p)) {
+						strncpy(out, p, size);
+						ret = strlen(p);
+					}
+					break;
+				}
+			}
+		} else {
+			break;
+		}
+	}
+
+	return ret;
+}
+
+int opae_enumerate(opae_pci_id *filter, pcidev_id list, int size)
+{
+	DIR *dir = NULL;
+	struct dirent *dirent = NULL;
+	char path[PATH_MAX] = {0};
+	opae_pci_id id;
+	int n = 0;
+
+	if (!filter || (size < 0) || (!list && (size > 0))) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	dir = opendir(rte_pci_get_sysfs_path());
+	if (!dir) {
+		opae_log_err("Failed to open \'%s\'\n",
+			rte_pci_get_sysfs_path());
+		return -EINVAL;
+	}
+	while ((dirent = readdir(dir))) {
+		if (!strcmp(dirent->d_name, "."))
+			continue;
+		if (!strcmp(dirent->d_name, ".."))
+			continue;
+
+		snprintf(path, PATH_MAX, "%s/%s", rte_pci_get_sysfs_path(),
+			dirent->d_name);
+		if (get_pci_id(path, &id) < 0)
+			continue;
+		if (compare_pci_id(&id, filter) < 0)
+			continue;
+
+		if (n++ < size) {
+			snprintf(list->bdf, sizeof(list->bdf), "%s",
+				dirent->d_name);
+			list++;
+		}
+	}
+	closedir(dir);
+
+	return n;
+}
+
+static int get_driver(pcidev_id id, char *drv_name, uint32_t size)
+{
+	char path[PATH_MAX] = {0};
+	char link[PATH_MAX] = {0};
+	int ret = 0;
+
+	if (!id || !drv_name) {
+		ret = -EINVAL;
+		goto end;
+	}
+	size--;   /* reserve one byte for the end of string */
+
+	snprintf(path, PATH_MAX, "%s/%s/driver",
+		rte_pci_get_sysfs_path(), id->bdf);
+	ret = readlink(path, link, PATH_MAX);
+	if (ret >= PATH_MAX) {
+		opae_log_err("Link path too long [%d]\n", ret);
+		ret = -ENAMETOOLONG;
+		goto end;
+	}
+	if (ret > 0) {
+		ret = extract_path(link, 0, drv_name, size);
+	} else {
+		*drv_name = 0;
+		opae_log_info("No link path for \'%s\'\n", path);
+		ret = 0;
+	}
+
+end:
+	if (ret < 0)
+		opae_log_err("Failed to get driver of %s\n", id->bdf);
+
+	return ret;
+}
+
+static int get_pci_addr(const char *bdf, opae_pci_addr *addr)
+{
+	struct rte_pci_addr pci_addr;
+	int ret = 0;
+
+	if (!bdf || !addr)
+		return -EINVAL;
+
+	ret = rte_pci_addr_parse(bdf, &pci_addr);
+	if (ret == 0) {
+		addr->domain = pci_addr.domain;
+		addr->bus = pci_addr.bus;
+		addr->devid = pci_addr.devid;
+		addr->function = pci_addr.function;
+	}
+
+	return ret;
+}
+
+static int check_pcidev_id(pcidev_id id)
+{
+	if (!id) {
+		opae_log_err("ID is NULL\n");
+		return -1;
+	}
+
+	if (strnlen(id->bdf, PCI_PRI_STR_SIZE) == PCI_PRI_STR_SIZE) {
+		opae_log_err("PCI address is too long\n");
+		return -2;
+	}
+
+	return 0;
+}
+
+static int get_rawdev_id(pcidev_id id, uint16_t *dev_id, int log)
+{
+	if (check_pcidev_id(id))
+		return -1;
+
+	if (rte_pmd_ifpga_get_dev_id(id->bdf, dev_id)) {
+		if (log)
+			opae_log_warn("%s is not probed by ifpga driver\n",
+				id->bdf);
+		return -1;
+	}
+
+	return 0;
+}
+
+static struct rte_pci_device *get_rte_pcidev(pcidev_id id, int log)
+{
+	const struct rte_pci_bus *pci_bus = NULL;
+	struct rte_pci_device *pci_dev = NULL;
+	struct rte_pci_addr addr;
+
+	if (check_pcidev_id(id))
+		return NULL;
+
+	if (rte_pci_addr_parse(id->bdf, &addr)) {
+		opae_log_err("PCI address %s is invalid\n", id->bdf);
+		return NULL;
+	}
+
+	pci_bus = rte_pmd_ifpga_get_pci_bus();
+	if (pci_bus) {
+		TAILQ_FOREACH(pci_dev, &pci_bus->device_list, next) {
+			if (!rte_pci_addr_cmp(&pci_dev->addr, &addr))
+				return pci_dev;
+		}
+	}
+
+	if (log)
+		opae_log_err("No rte_pci_device for %s\n", id->bdf);
+
+	return NULL;
+}
+
+int opae_load_rsu_status(pcidev_id id, uint32_t *status, uint32_t *progress)
+{
+	uint16_t dev_id = 0;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	return rte_pmd_ifpga_get_rsu_status(dev_id, status, progress);
+}
+
+int opae_store_rsu_status(pcidev_id id, uint32_t status, uint32_t progress)
+{
+	uint16_t dev_id = 0;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	return rte_pmd_ifpga_set_rsu_status(dev_id, status, progress);
+}
+
+static int get_pci_property(pcidev_id id, opae_pci_property *prop)
+{
+	char path[PATH_MAX] = {0};
+	int ret = 0;
+
+	if (!id || !prop)
+		return -EINVAL;
+
+	snprintf(path, PATH_MAX, "%s/%s", rte_pci_get_sysfs_path(), id->bdf);
+
+	ret = get_pci_id(path, &prop->id);
+	if (ret < 0)
+		return ret;
+
+	ret = get_pci_addr(id->bdf, &prop->addr);
+	if (ret < 0)
+		return ret;
+
+	snprintf(prop->pci_addr, OPAE_NAME_SIZE, "%s", id->bdf);
+	get_driver(id, prop->drv_name, sizeof(prop->drv_name));
+
+	return 0;
+}
+
+static int get_fme_property(rte_pmd_ifpga_common_prop *common,
+	opae_fme_property *prop)
+{
+	opae_bitstream_id bbs_id;
+
+	if (!common || !prop)
+		return -EINVAL;
+
+	prop->boot_page = common->boot_page;
+	prop->num_ports = common->num_ports;
+	prop->bitstream_id = common->bitstream_id;
+	prop->bitstream_metadata = common->bitstream_metadata;
+	memcpy(prop->pr_id.b, common->pr_id.b, sizeof(opae_uuid));
+
+	bbs_id.id = prop->bitstream_id;
+	if (bbs_id.major < sizeof(platform_name) / sizeof(char *)) {
+		snprintf(prop->platform_name,
+			sizeof(prop->platform_name), "%s",
+			platform_name[bbs_id.major]);
+	} else {
+		snprintf(prop->platform_name,
+			sizeof(prop->platform_name), "unknown");
+	}
+
+	snprintf(prop->dcp_version, sizeof(prop->dcp_version),
+		"DCP 1.%u", bbs_id.minor);
+
+	if (bbs_id.patch < sizeof(release_name)/sizeof(char *)) {
+		snprintf(prop->release_name, sizeof(prop->release_name),
+			"%s", release_name[bbs_id.patch]);
+	} else {
+		snprintf(prop->release_name, sizeof(prop->release_name),
+			"unknown");
+	}
+
+	if (bbs_id.major == 0) {  /* Vista Creek */
+		if (bbs_id.interface <
+			sizeof(interface_type) / sizeof(char *)) {
+			snprintf(prop->interface_type,
+				sizeof(prop->interface_type), "%s",
+				interface_type[bbs_id.interface]);
+		} else {
+			snprintf(prop->interface_type,
+				sizeof(prop->interface_type), "unknown");
+		}
+	} else {
+		snprintf(prop->interface_type,
+			sizeof(prop->interface_type), "unknown");
+	}
+
+	snprintf(prop->build_version, sizeof(prop->build_version),
+		"%u.%u.%u", bbs_id.build_major, bbs_id.build_minor,
+		bbs_id.build_patch);
+
+	return 0;
+}
+
+static int get_port_property(rte_pmd_ifpga_port_prop *port,
+	opae_port_property *prop)
+{
+	if (!port || !prop)
+		return -EINVAL;
+
+	memcpy(prop->afu_id.b, port->afu_id.b, sizeof(opae_uuid));
+	prop->type = port->type;
+
+	return 0;
+}
+
+static int get_bmc_property(rte_pmd_ifpga_common_prop *common,
+	opae_bmc_property *prop)
+{
+	opae_bmc_version ver;
+
+	if (!common || !prop)
+		return -EINVAL;
+
+	ver.version = common->bmc_version;
+	snprintf(prop->bmc_version, sizeof(prop->bmc_version), "%c.%u.%u.%u",
+		ver.board, ver.major, ver.minor, ver.micro);
+
+	ver.version = common->bmc_nios_version;
+	snprintf(prop->fw_version, sizeof(prop->fw_version), "%c.%u.%u.%u",
+		ver.board, ver.major, ver.minor, ver.micro);
+
+	return 0;
+}
+
+int opae_get_property(pcidev_id id, opae_fpga_property *prop, int type)
+{
+	uint16_t dev_id = 0;
+	uint32_t i = 0;
+	rte_pmd_ifpga_prop fpga_prop;
+	int ret = 0;
+
+	if (!prop) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	if (type == 0)
+		type = OPAE_PROP_ALL;
+
+	memset(prop, 0, sizeof(opae_fpga_property));
+
+	/* PCI properties */
+	if (type & OPAE_PROP_PCI) {
+		if (get_pci_property(id, &prop->pci) < 0) {
+			opae_log_err("Failed to get PCI property\n");
+			return -EAGAIN;
+		}
+	}
+
+	if (type == OPAE_PROP_PCI)
+		return 0;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	ret = rte_pmd_ifpga_get_property(dev_id, &fpga_prop);
+	if (ret) {
+		opae_log_err("Failed to get FPGA property [e:%d]\n", ret);
+		return -EAGAIN;
+	}
+
+	/* FME properties */
+	if (type & (OPAE_PROP_FME | OPAE_PROP_PORT))
+		get_fme_property(&fpga_prop.common, &prop->fme);
+
+	/* PORT properties */
+	if (type & OPAE_PROP_PORT) {
+		for (i = 0; i < prop->fme.num_ports; i++) {
+			prop->port[i].index = i;
+			get_port_property(&fpga_prop.port[i], &prop->port[i]);
+		}
+	}
+
+	/* BMC properties */
+	if (type & OPAE_PROP_BMC)
+		get_bmc_property(&fpga_prop.common, &prop->bmc);
+
+	return 0;
+}
+
+int opae_get_phy_info(pcidev_id id, opae_phy_info *info)
+{
+	uint16_t dev_id = 0;
+	rte_pmd_ifpga_phy_info phy_info;
+	int ret = 0;
+
+	if (!info)
+		return -EINVAL;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	ret = rte_pmd_ifpga_get_phy_info(dev_id, &phy_info);
+	if (ret) {
+		opae_log_err("Failed to get PHY information [e:%d]\n", ret);
+		return -EAGAIN;
+	}
+
+	info->num_retimers = phy_info.num_retimers;
+	info->link_speed = phy_info.link_speed;
+	info->link_status = phy_info.link_status;
+
+	return 0;
+}
+
+static int update_driver(pcidev_id id, char *drv_name)
+{
+	struct rte_pci_device *pci_dev = NULL;
+	char name[OPAE_NAME_SIZE] = {0};
+	int ret = 0;
+
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	if (drv_name) {
+		if (strlen(drv_name) >= OPAE_NAME_SIZE) {
+			opae_log_err("Driver name \'%s\' too long\n",
+				drv_name);
+			return -EINVAL;
+		}
+		strncpy(name, drv_name, sizeof(name));
+	} else {
+		ret = get_driver(id, name, sizeof(name));
+		if (ret < 0)
+			return ret;
+	}
+
+	pci_dev = get_rte_pcidev(id, 0);
+	if (pci_dev) {
+		if (strlen(name) == 0) {
+			pci_dev->kdrv = RTE_PCI_KDRV_NONE;
+		} else {
+			if (!strcmp(name, OPAE_KDRV_VFIO_PCI))
+				pci_dev->kdrv = RTE_PCI_KDRV_VFIO;
+			else if (!strcmp(name, OPAE_KDRV_IGB_UIO))
+				pci_dev->kdrv = RTE_PCI_KDRV_IGB_UIO;
+			else if (!strcmp(name, OPAE_KDRV_UIO_PCI))
+				pci_dev->kdrv = RTE_PCI_KDRV_UIO_GENERIC;
+			else
+				pci_dev->kdrv = RTE_PCI_KDRV_UNKNOWN;
+		}
+	}
+
+	return 0;
+}
+
+int opae_unbind_driver(pcidev_id id)
+{
+	uint16_t dev_id = 0;
+	char path[PATH_MAX] = {0};
+	char drv_name[OPAE_NAME_SIZE] = {0};
+	char null[] = {0};
+	int ret = 0;
+
+	if (!get_rawdev_id(id, &dev_id, 0)) {
+		opae_log_err("%s is probed, remove it first\n", id->bdf);
+		return -EBUSY;
+	}
+
+	ret = get_driver(id, drv_name, sizeof(drv_name));
+	if (ret < 0)
+		return ret;
+
+	if (strlen(drv_name) > 0) {
+		snprintf(path, PATH_MAX, "/sys/bus/pci/drivers/%s/unbind",
+			drv_name);
+		ret = write_file(path, id->bdf, strlen(id->bdf) + 1);
+		if (ret == 0)
+			ret = update_driver(id, null);
+	}
+
+	return ret;
+}
+
+static int check_driver(const char *drv_name)
+{
+	char path[PATH_MAX] = {0};
+	struct stat buf;
+
+	if (!drv_name)
+		return -EINVAL;
+
+	if (strlen(drv_name) > 0) {
+		snprintf(path, PATH_MAX, "/sys/bus/pci/drivers/%s", drv_name);
+		if ((stat(path, &buf) < 0) || ((buf.st_mode & S_IFDIR) == 0)) {
+			opae_log_warn("Driver %s is not installed\n",
+				drv_name);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+int opae_bind_driver(pcidev_id id, char *drv_name)
+{
+	char path[PATH_MAX] = {0};
+	char name[OPAE_NAME_SIZE] = {0};
+	char null[] = {0};
+	int ret = 0;
+
+	ret = check_driver(drv_name);
+	if (ret < 0)
+		return ret;
+
+	ret = get_driver(id, name, sizeof(name));
+	if (ret < 0)
+		return ret;
+
+	if (!strcmp(drv_name, name))   /* driver not change */
+		return 0;
+
+	ret = opae_unbind_driver(id);
+	if (ret < 0)
+		return ret;
+
+	if (strlen(drv_name) > 0) {
+		/* bind driver */
+		snprintf(path, PATH_MAX, "%s/%s/driver_override",
+			rte_pci_get_sysfs_path(), id->bdf);
+		ret = write_file(path, drv_name, strlen(drv_name) + 1);
+		if (ret < 0)
+			goto update_drv;
+
+		snprintf(path, PATH_MAX, "/sys/bus/pci/drivers/%s/bind",
+			drv_name);
+		ret = write_file(path, id->bdf, strlen(id->bdf) + 1);
+		if (ret < 0)
+			goto update_drv;
+
+		snprintf(path, PATH_MAX, "%s/%s/driver_override",
+			rte_pci_get_sysfs_path(), id->bdf);
+		ret = write_file(path, null, 1);
+		if (ret < 0)
+			goto update_drv;
+	}
+
+update_drv:
+	ret = update_driver(id, NULL);
+	if (ret < 0)
+		opae_log_err("Failed to update driver information of %s\n",
+			id->bdf);
+
+	return 0;
+}
+
+int opae_probe_device(pcidev_id id)
+{
+	const struct rte_pci_bus *pci_bus = NULL;
+	struct rte_pci_device *pci_dev = NULL;
+
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	/* make sure device is added in rte_pci_bus devices list */
+	pci_bus = rte_pmd_ifpga_get_pci_bus();
+	if (pci_bus && pci_bus->bus.scan)
+		pci_bus->bus.scan();
+
+	pci_dev = get_rte_pcidev(id, 1);
+	if (!pci_dev)
+		return -ENODEV;
+
+	if (pci_dev->kdrv != RTE_PCI_KDRV_VFIO) {
+		opae_log_err("vfio-pci driver is not bound to %s\n", id->bdf);
+		return -EINVAL;
+	}
+
+	if (!pci_bus || !pci_bus->bus.plug)
+		return -ENODEV;
+
+	return pci_bus->bus.plug(&pci_dev->device);
+}
+
+int opae_remove_device(pcidev_id id)
+{
+	struct rte_pci_device *pci_dev = NULL;
+	struct rte_pci_driver *pci_drv = NULL;
+	int ret = 0;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	pci_dev = get_rte_pcidev(id, 0);
+	if (pci_dev && pci_dev->driver) {
+		pci_drv = pci_dev->driver;
+		ret = pci_drv->remove(pci_dev);
+		if (ret < 0) {
+			opae_log_err("Failed to remove %s [e:%d]\n",
+				id->bdf, ret);
+			return ret;
+		}
+		pci_dev->driver = NULL;
+		pci_dev->device.driver = NULL;
+		if (pci_drv->drv_flags & RTE_PCI_DRV_NEED_MAPPING)
+			rte_pci_unmap_device(pci_dev);
+	}
+
+	return ret;
+}
+
+static int is_pac(pcidev_id id)
+{
+	char path[PATH_MAX] = {0};
+	opae_pci_id pci_id;
+
+	if (check_pcidev_id(id))
+		return 0;
+
+	snprintf(path, PATH_MAX, "%s/%s", rte_pci_get_sysfs_path(), id->bdf);
+	if (get_pci_id(path, &pci_id) < 0)
+		return 0;
+
+	if ((pci_id.vendor_id == 0x8086) && (pci_id.device_id == 0x0b30))
+		return 1;
+
+	return 0;
+}
+
+int opae_get_parent(pcidev_id id, pcidev_id parent)
+{
+	char path[PATH_MAX] = {0};
+	char link[PATH_MAX] = {0};
+	int ret = 0;
+
+	if (!id || !parent) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		ret = -EINVAL;
+		goto end;
+	}
+
+	snprintf(path, PATH_MAX, "%s/%s", rte_pci_get_sysfs_path(), id->bdf);
+	ret = readlink(path, link, PATH_MAX);
+	if (ret >= PATH_MAX) {
+		opae_log_err("Length of link path exceeds %u\n", PATH_MAX);
+		ret = -ENAMETOOLONG;
+		goto end;
+	}
+
+	if (ret > 0) {
+		ret = extract_path(link, 1, parent->bdf, sizeof(parent->bdf));
+		if (!strncmp(parent->bdf, "pci", 3)) {
+			parent->bdf[0] = 0;
+			ret = -ENODEV;
+		}
+	} else {
+		parent->bdf[0] = 0;
+		if (ret == 0)
+			opae_log_err("Length of link path is 0\n");
+		else
+			opae_log_err("No link path for \'%s\'\n", path);
+	}
+end:
+	if (ret <= 0)
+		opae_log_err("%s has no parent\n", id->bdf);
+
+	return ret;
+}
+
+int opae_get_child(pcidev_id id, pcidev_id child, int size)
+{
+	glob_t pglob = {.gl_pathc = 0, .gl_pathv = NULL};
+	char path[PATH_MAX] = {0};
+	int i, count = 0;
+	int len = 0;
+	int ret = 0;
+
+	if (!id || (size < 0)) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	snprintf(path, PATH_MAX, "%s/%s/*:*:*.?", rte_pci_get_sysfs_path(),
+		id->bdf);
+	ret = glob(path, 0, NULL, &pglob);
+	if (ret == 0) {
+		if (child && (size > 0)) {
+			for (i = 0; i < (int)pglob.gl_pathc; i++) {
+				len = extract_path(pglob.gl_pathv[i], 0,
+					child->bdf, sizeof(child->bdf));
+				if (len <= 0) {
+					child->bdf[0] = 0;
+					continue;
+				}
+				if (++count >= size)
+					break;
+				child++;
+			}
+		} else {
+			count = (int)pglob.gl_pathc;
+		}
+		globfree(&pglob);
+	} else {
+		if (pglob.gl_pathv)
+			globfree(&pglob);
+	}
+
+	return count;
+}
+
+int opae_get_pf1(pcidev_id id, pcidev_id peer, int size)
+{
+	opae_pci_device parent;
+	opae_pci_device child[4];
+	int n = 0;
+	int ret = 0;
+
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	if (!is_pac(id)) {
+		opae_log_info("%s has no peer function\n", id->bdf);
+		return -EINVAL;
+	}
+
+	ret = opae_get_parent(id, &parent);
+	if (ret < 0)
+		return -ENODEV;
+	ret = opae_get_parent(&parent, &parent);
+	if (ret < 0)
+		return -ENODEV;
+
+	n = opae_get_child(&parent, child,
+		sizeof(child) / sizeof(opae_pci_device));
+	/* there should have four downstream ports of PCI switch on board */
+	if (n == 4) {
+		n = opae_get_child(&child[3], peer, size);
+	} else {
+		peer->bdf[0] = 0;
+		opae_log_dbg("%s has %d child(s)\n", parent.bdf, n);
+		n = 0;
+	}
+
+	return n;
+}
+
+int opae_update_flash(pcidev_id id, const char *image, uint64_t *status)
+{
+	opae_img_info info;
+	uint16_t dev_id = 0;
+	int ret = 0;
+
+	ret = opae_get_image_info(image, &info);
+	if (ret < 0) {
+		opae_log_err("Failed to get image information [e:%d]\n", ret);
+		return -EINVAL;
+	}
+
+	if ((info.type != OPAE_IMG_TYPE_BBS) &&
+		(info.type != OPAE_IMG_TYPE_BMC)) {
+		opae_log_err("Image is not supported [t:%u]\n", info.type);
+		return -EOPNOTSUPP;
+	}
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	return rte_pmd_ifpga_update_flash(dev_id, image, status);
+}
+
+int opae_cancel_flash_update(pcidev_id id, int force)
+{
+	uint16_t dev_id = 0;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	return rte_pmd_ifpga_stop_update(dev_id, force);
+}
+
+static int find_pci_ecap(int fd, unsigned int cap)
+{
+	uint32_t header = 0;
+	int ttl = (RTE_PCI_CFG_SPACE_EXP_SIZE - RTE_PCI_CFG_SPACE_SIZE) / 8;
+	int pos = RTE_PCI_CFG_SPACE_SIZE;
+	int ret = 0;
+
+	ret = pread(fd, &header, sizeof(header), pos);
+	if (ret < 0) {
+		opae_log_err("Failed to read from PCI configuration space [e:%s]\n",
+			strerror(errno));
+		return ret;
+	}
+	opae_log_dbg("Read 0x%08x from PCI configuration space 0x%x\n",
+		header, pos);
+
+	if (header == 0) {
+		opae_log_err("Capability is empty\n");
+		return 0;
+	}
+
+	while (ttl-- > 0) {
+		if ((RTE_PCI_EXT_CAP_ID(header) == cap) && (pos != 0))
+			return pos;
+
+		pos = RTE_PCI_EXT_CAP_NEXT(header);
+		if (pos < RTE_PCI_CFG_SPACE_SIZE) {
+			opae_log_err("Position of capability is invalid"
+						 "[e:%d]\n", pos);
+			break;
+		}
+		ret = pread(fd, &header, sizeof(header), pos);
+		if (ret < 0) {
+			opae_log_err("Failed to read from PCI config space [e:%s]\n",
+				strerror(errno));
+			return ret;
+		}
+		opae_log_dbg("Read 0x%08x from PCI configuration space 0x%x\n",
+			header, pos);
+	}
+
+	return 0;
+}
+
+static int set_aer(pcidev_id id, uint32_t v1, uint32_t v2, int record)
+{
+	char path[PATH_MAX] = {0};
+	uint32_t val = 0;
+	int fd = -1;
+	int pos = 0;
+	int ret = 0;
+
+	if (!id)
+		return -EINVAL;
+
+	snprintf(path, PATH_MAX, "%s/%s/config",
+		rte_pci_get_sysfs_path(), id->bdf);
+	fd = open(path, O_RDWR);
+	if (fd < 0) {
+		opae_log_err("Failed to open \'%s\' for RDWR [e:%s]\n",
+			path, strerror(errno));
+		return -EIO;
+	}
+
+	pos = find_pci_ecap(fd, RTE_PCI_EXT_CAP_ID_ERR);
+	if (pos <= 0) {
+		opae_log_warn("AER capability is not present\n");
+		ret = -ENXIO;
+		goto close_fd;
+	}
+
+	if (record) {
+		ret = pread(fd, &val, sizeof(val), pos + 0x08);
+		if (ret < 0) {
+			opae_log_err("Failed to read from PCI config space [e:%s]\n",
+				strerror(errno));
+			goto close_fd;
+		}
+		opae_log_dbg("Read 0x%08x from PCI configuration space 0x%x\n",
+			val, pos + 0x08);
+		dev_aer[0] = val;
+
+		ret = pread(fd, &val, sizeof(val), pos + 0x14);
+		if (ret < 0) {
+			opae_log_err("Failed to read from PCI config space [e:%s]\n",
+				strerror(errno));
+			goto close_fd;
+		}
+		opae_log_dbg("Read 0x%08x from PCI configuration space 0x%x\n",
+			val, pos + 0x14);
+		dev_aer[1] = val;
+	}
+
+	opae_log_dbg("Write 0x%08x to PCI configuration space 0x%x\n",
+		v1, pos + 0x08);
+	ret = pwrite(fd, &v1, sizeof(v1), pos + 0x08);
+	if (ret < 0) {
+		opae_log_err("Failed to write to PCI config space 0x%x [e:%s]\n",
+			pos + 0x08, strerror(errno));
+		goto close_fd;
+	}
+
+	opae_log_dbg("Write 0x%08x to PCI configuration space 0x%x\n",
+		v2, pos + 0x14);
+	ret = pwrite(fd, &v2, sizeof(v2), pos + 0x14);
+	if (ret < 0) {
+		opae_log_err("Failed to write to PCI config space 0x%x [e:%s]\n",
+			pos + 0x14, strerror(errno));
+	}
+
+close_fd:
+	close(fd);
+	return ret < 0 ? ret : 0;
+}
+
+static int enable_aer(pcidev_id id)
+{
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	opae_log_info("Enable AER of %s\n", id->bdf);
+
+	return set_aer(id, dev_aer[0], dev_aer[1], 0);
+}
+
+static int disable_aer(pcidev_id id)
+{
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	opae_log_info("Disable AER of %s\n", id->bdf);
+
+	return set_aer(id, 0xffffffff, 0xffffffff, 1);
+}
+
+static int remove_tree(pcidev_id id)
+{
+	int i, n = 0;
+	pcidev_id child;
+	int ret = 0;
+
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	n = opae_get_child(id, NULL, 0);
+	if (n > 0) {
+		child = (pcidev_id)rte_zmalloc(NULL,
+			sizeof(opae_pci_device) * n, 0);
+		if (!child) {
+			opae_log_err("Failed to malloc for children of %s\n",
+				id->bdf);
+			ret = -ENOMEM;
+			goto end;
+		}
+
+		opae_get_child(id, child, n);
+		for (i = 0; i < n; i++)
+			remove_tree(&child[i]);
+		rte_free(child);
+	}
+
+end:
+	opae_remove_device(id);
+	return ret;
+}
+
+static int remove_device(pcidev_id id)
+{
+	char path[PATH_MAX] = {0};
+	char one[] = {'1', 0};
+	int ret = 0;
+
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	opae_log_info("Remove %s from system\n", id->bdf);
+
+	snprintf(path, PATH_MAX, "%s/%s/remove",
+		rte_pci_get_sysfs_path(), id->bdf);
+	ret = write_file(path, one, strlen(one));
+	if (ret < 0) {
+		opae_log_err("Failed to remove %s from system\n", id->bdf);
+		return ret;
+	}
+
+	remove_tree(id);
+
+	return 0;
+}
+
+static int scan_device(pcidev_id parent, pcidev_id id)
+{
+	char path[PATH_MAX] = {0};
+	char bus[8] = {0};
+	char one[] = {'1', 0};
+	char pwr[16] = {0};
+	char pwr_on[] = {'o', 'n', 0};
+	int pwr_on_failed = 0;
+	int ret = 0;
+
+	if (!parent) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+	opae_log_info("Rescan devices under %s\n", parent->bdf);
+
+	if (id) {   /* scan specified bus under parent device */
+		snprintf(path, PATH_MAX, "%s/%s/power/control",
+			rte_pci_get_sysfs_path(), parent->bdf);
+		ret = read_file(path, pwr, sizeof(pwr));
+		if (ret < 0)
+			return ret;
+
+		if (strcmp(pwr, "on")) {
+			ret = write_file(path, pwr_on, strlen(pwr_on));
+			if (ret < 0)
+				pwr_on_failed = 1;
+			else
+				sleep(1);
+		}
+
+		snprintf(bus, sizeof(bus), "%s", id->bdf);
+		snprintf(path, PATH_MAX, "%s/%s/pci_bus/%s/rescan",
+			rte_pci_get_sysfs_path(), parent->bdf, bus);
+		ret = write_file(path, one, strlen(one));
+		if (ret < 0)
+			return ret;
+
+		if (pwr_on_failed) {   /* workaround for power on failed */
+			ret = write_file(path, one, strlen(one));
+			if (ret < 0)
+				return ret;
+		}
+
+		if (strcmp(pwr, "on")) {
+			snprintf(path, PATH_MAX, "%s/%s/power/control",
+				rte_pci_get_sysfs_path(), parent->bdf);
+			ret = write_file(path, pwr, strlen(pwr));
+		}
+	} else {   /* scan all buses under parent device */
+		snprintf(path, PATH_MAX, "%s/%s/rescan",
+			rte_pci_get_sysfs_path(), parent->bdf);
+		ret = write_file(path, one, strlen(one));
+	}
+
+	return ret;
+}
+
+int opae_reboot_device(pcidev_id id, int type, int page)
+{
+	uint16_t dev_id = 0;
+	opae_pci_device fpga;    /* FPGA after reboot */
+	opae_pci_device parent;
+	opae_pci_device peer[2];   /* physical function 1 of FPGA */
+	opae_pci_device peer_parent;
+	opae_pci_device ups;   /* upstream port device */
+	opae_pci_device root;  /* port connected to PAC */
+	pcidev_id peer_primary = NULL;
+	char drv_name[OPAE_NAME_SIZE] = {0};
+	int n = 0;
+	int i = 0;
+	int ret = 0;
+
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	if (!is_pac(id)) {
+		opae_log_err("%s can not be rebooted\n", id->bdf);
+		return -EINVAL;
+	}
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	ret = opae_get_parent(id, &parent);
+	if (ret < 0)
+		return -ENODEV;
+	ret = opae_get_parent(&parent, &ups);
+	if (ret < 0)
+		return -ENODEV;
+	ret = opae_get_parent(&ups, &root);
+	if (ret < 0)
+		return -ENODEV;
+
+	n = opae_get_pf1(id, peer, sizeof(peer) / sizeof(opae_pci_device));
+	if (n <= 0) {
+		opae_log_err("PF1 of %s is not found\n", id->bdf);
+	} else {
+		peer_primary = &peer[0];
+		ret = opae_get_parent(peer_primary, &peer_parent);
+		if (ret < 0)
+			return -ENODEV;
+	}
+
+	get_driver(id, drv_name, sizeof(drv_name));  /* save original driver */
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (rte_pmd_ifpga_reboot_try(dev_id)) {
+		opae_log_warn("Update or reboot is in progress\n");
+		return -EAGAIN;
+	}
+
+	if (type == 0) {
+		/* disable AER */
+		ret = disable_aer(&parent);
+		if (ret < 0) {
+			opae_log_err("Failed to disable AER of %s\n",
+				parent.bdf);
+			goto reboot_end;
+		}
+		ret = disable_aer(&peer_parent);
+		if (ret < 0) {
+			opae_log_err("Failed to disable AER of %s\n",
+				peer_parent.bdf);
+			goto reboot_end;
+		}
+
+		/* trigger reconfiguration */
+		ret = rte_pmd_ifpga_reload(dev_id, type, page);
+		if (ret == 0) {
+			ret = remove_device(id);
+			for (i = 0; i < n; i++)
+				ret += remove_device(&peer[i]);
+			if (ret == 0) {
+				opae_log_info("Wait 10 seconds for FPGA reloading\n");
+				sleep(10);
+				ret = scan_device(&parent, id);
+				if (ret < 0)
+					opae_log_err("Failed to rescan %s\n",
+						id->bdf);
+				if (peer_primary) {
+					ret = scan_device(&peer_parent,
+						peer_primary);
+					if (ret < 0) {
+						opae_log_err("Failed to rescan %s\n",
+							peer_primary->bdf);
+					}
+				}
+			}
+		}
+
+		/* restore AER */
+		if (enable_aer(&parent) < 0) {
+			opae_log_err("Failed to enable AER of %s\n",
+				parent.bdf);
+		}
+		if (enable_aer(&peer_parent) < 0) {
+			opae_log_err("Failed to enable AER of %s\n",
+				peer_parent.bdf);
+		}
+	} else if (type == 1) {
+		/* disable AER */
+		ret = disable_aer(&root);
+		if (ret < 0) {
+			opae_log_err("Failed to disable AER of %s\n", root.bdf);
+			goto reboot_end;
+		}
+
+		/* trigger reconfiguration */
+		ret = rte_pmd_ifpga_reload(dev_id, type, page);
+		if (ret == 0) {
+			ret += remove_device(&ups);
+			if (ret == 0) {
+				opae_log_info("Wait 10 seconds for BMC reloading\n");
+				sleep(10);
+				ret = scan_device(&root, &ups);
+				if (ret < 0)
+					opae_log_err("Failed to rescan %s\n",
+						ups.bdf);
+			}
+		}
+
+		/* restore AER */
+		if (enable_aer(&root) < 0)
+			opae_log_err("Failed to enable AER of %s\n", root.bdf);
+	} else {
+		opae_log_err("Type of reboot is not supported [t:%d]\n", type);
+		ret = -EINVAL;
+		goto reboot_end;
+	}
+
+	/* update id if bdf changed after reboot */
+	if (opae_get_child(&parent, &fpga, 1) == 1) {
+		if (strcmp(id->bdf, fpga.bdf))
+			id = &fpga;
+	}
+
+	ret = opae_bind_driver(id, drv_name);
+	if (ret < 0)
+		opae_log_err("Failed to bind original driver of %s\n", id->bdf);
+
+	ret = opae_probe_device(id);
+	if (ret < 0)
+		opae_log_err("Failed to probe %s [e:%d]\n", id->bdf, ret);
+
+reboot_end:
+	rte_pmd_ifpga_set_rsu_status(dev_id, 0, 0);
+	return ret;
+}
+
+int opae_partial_reconfigure(pcidev_id id, int port, const char *gbs)
+{
+	uint16_t dev_id = 0;
+
+	if (!id || !gbs) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	return rte_pmd_ifpga_partial_reconfigure(dev_id, port, gbs);
+}
+
+int opae_read_pci_cfg(pcidev_id id, uint32_t address, uint32_t *value)
+{
+	char path[PATH_MAX] = {0};
+	int fd = -1;
+	int ret = 0;
+
+	if (!id || !value) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	snprintf(path, PATH_MAX, "%s/%s/config", rte_pci_get_sysfs_path(),
+		id->bdf);
+	fd = open(path, O_RDONLY);
+	if (fd < 0) {
+		opae_log_dbg("Failed to open \'%s\' for RDONLY [e:%s]\n",
+			path, strerror(errno));
+		return -EIO;
+	}
+
+	ret = pread(fd, value, 4, address);
+	if (ret < 0) {
+		opae_log_err("Failed to read from PCI device %s [e:%s]\n",
+			id->bdf, strerror(errno));
+		close(fd);
+		return ret;
+	}
+
+	opae_log_dbg("CONFIG+0x%08x -> 0x%08x\n", address, *value);
+	close(fd);
+	return 0;
+}
+
+int opae_write_pci_cfg(pcidev_id id, uint32_t address, uint32_t value)
+{
+	char path[PATH_MAX] = {0};
+	int fd = -1;
+	int ret = 0;
+
+	if (!id) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	snprintf(path, PATH_MAX, "%s/%s/config", rte_pci_get_sysfs_path(),
+		id->bdf);
+	fd = open(path, O_WRONLY);
+	if (fd < 0) {
+		opae_log_dbg("Failed to open \'%s\' for WRONLY [e:%s]\n",
+			path, strerror(errno));
+		return -EIO;
+	}
+
+	ret = pwrite(fd, &value, 4, address);
+	if (ret < 0) {
+		opae_log_err("Failed to write to PCI device %s [e:%s]\n",
+			id->bdf, strerror(errno));
+		close(fd);
+		return ret;
+	}
+
+	opae_log_dbg("CONFIG+0x%08x <- 0x%08x\n", address, value);
+	close(fd);
+	return 0;
+}
diff --git a/examples/ifpga/opae_api.h b/examples/ifpga/opae_api.h
new file mode 100644
index 0000000000..3b6bc01fbc
--- /dev/null
+++ b/examples/ifpga/opae_api.h
@@ -0,0 +1,244 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _OPAE_API_H
+#define _OPAE_API_H
+
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+extern int opae_log_level;
+extern FILE *opae_log_file;
+
+#define OPAE_LOG_API      0  /**< Critical conditions.              */
+#define OPAE_LOG_ERR      1  /**< Error conditions.                 */
+#define OPAE_LOG_WARN     2  /**< Warning conditions.               */
+#define OPAE_LOG_INFO     3  /**< Informational.                    */
+#define OPAE_LOG_DEBUG    4  /**< Debug-level messages.             */
+
+#define opae_log(type, fmt, args...)             \
+do {                                             \
+	if (opae_log_level >= OPAE_LOG_##type) {     \
+		printf(fmt, ##args);                     \
+		if (opae_log_file) {                     \
+			fprintf(opae_log_file, fmt, ##args); \
+			fflush(opae_log_file);               \
+		}                                        \
+	}                                            \
+} while (0)
+
+#define opae_log_api(fmt, args...)  opae_log(API, "OPAE-API: "fmt, ##args)
+#define opae_log_err(fmt, args...)  opae_log(ERR, "OPAE-ERR: "fmt, ##args)
+#define opae_log_dbg(fmt, args...)  opae_log(DEBUG, "OPAE-DBG: "fmt, ##args)
+#define opae_log_warn(fmt, args...) opae_log(WARN, "OPAE-WARN: "fmt, ##args)
+#define opae_log_info(fmt, args...) opae_log(INFO, "OPAE-INFO: "fmt, ##args)
+
+#define EAL_INIT_FUNCTION    "init"
+#define EAL_DEFAULT_OPTIONS  "--proc-type auto"
+
+#define OPAE_KDRV_UNKNOWN           "unknown"
+#define OPAE_KDRV_VFIO_PCI          "vfio-pci"
+#define OPAE_KDRV_IGB_UIO           "igb_uio"
+#define OPAE_KDRV_UIO_PCI           "uio_pci_generic"
+#define OPAE_KDRV_INTEL_FPGA_PCI    "intel-fpga-pci"
+
+typedef struct {
+	uint32_t major;
+	uint32_t minor;
+	uint32_t micro;
+} opae_api_version;
+
+#define OPAE_NAME_SIZE  32
+
+typedef struct {
+	char bdf[OPAE_NAME_SIZE];   /* segment:bus:device.function */
+} opae_pci_device;
+
+typedef opae_pci_device *pcidev_id;
+
+typedef struct {
+	uint32_t class_id;            /**< Class ID or RTE_CLASS_ANY_ID. */
+	uint16_t vendor_id;           /**< Vendor ID or PCI_ANY_ID. */
+	uint16_t device_id;           /**< Device ID or PCI_ANY_ID. */
+	uint16_t subsystem_vendor_id; /**< Subsystem vendor ID or PCI_ANY_ID. */
+	uint16_t subsystem_device_id; /**< Subsystem device ID or PCI_ANY_ID. */
+} opae_pci_id;
+
+typedef struct {
+	uint32_t domain;              /**< Device domain */
+	uint8_t bus;                  /**< Device bus */
+	uint8_t devid;                /**< Device ID */
+	uint8_t function;             /**< Device function. */
+} opae_pci_addr;
+
+typedef struct {
+	char pci_addr[OPAE_NAME_SIZE];  /* segment:bus:device.function */
+	char drv_name[OPAE_NAME_SIZE];  /* vfio-pci, intel-fpga-pci, etc. */
+	opae_pci_id id;
+	opae_pci_addr addr;
+} opae_pci_property;
+
+#define BIT_SET_8   0xFF
+#define BIT_SET_16  0xFFFF
+#define BIT_SET_32  0xFFFFFFFF
+
+typedef struct {
+	uint8_t b[16];
+} opae_uuid;
+
+typedef struct {
+	uint32_t boot_page;
+	uint32_t num_ports;
+	uint64_t bitstream_id;
+	uint64_t bitstream_metadata;
+	opae_uuid pr_id;
+	char platform_name[OPAE_NAME_SIZE];
+	char dcp_version[OPAE_NAME_SIZE];
+	char release_name[OPAE_NAME_SIZE];
+	char interface_type[OPAE_NAME_SIZE];
+	char build_version[OPAE_NAME_SIZE];
+} opae_fme_property;
+
+typedef struct {
+	opae_uuid afu_id;
+	uint32_t type;   /* AFU memory access control type */
+	uint32_t index;  /* PORT index */
+} opae_port_property;
+
+typedef struct {
+	char bmc_version[OPAE_NAME_SIZE];
+	char fw_version[OPAE_NAME_SIZE];
+} opae_bmc_property;
+
+typedef struct {
+	uint32_t num_retimers;
+	uint32_t link_speed;
+	uint32_t link_status;  /* each bit corresponding to one link status */
+} opae_phy_info;
+
+typedef struct {
+	union {
+		uint64_t id;
+		struct {
+			uint8_t build_patch;
+			uint8_t build_minor;
+			uint8_t build_major;
+			uint8_t fvl_bypass:1;
+			uint8_t mac_lightweight:1;
+			uint8_t disagregate:1;
+			uint8_t lightweiht:1;
+			uint8_t seu:1;
+			uint8_t ptp:1;
+			uint8_t reserve:2;
+			uint16_t interface:4;
+			uint16_t afu_revision:12;
+			uint16_t patch:4;
+			uint16_t minor:4;
+			uint16_t major:4;
+			uint16_t reserved:4;
+		};
+	};
+} opae_bitstream_id;
+
+typedef struct {
+	union {
+		uint32_t version;
+		struct {
+			uint8_t micro;
+			uint8_t minor;
+			uint8_t major;
+			uint8_t board;
+		};
+	};
+} opae_bmc_version;
+
+#define OPAE_MAX_PORT_NUM   4
+
+#define OPAE_PROP_PCI   0x01
+#define OPAE_PROP_FME   0x02
+#define OPAE_PROP_PORT  0x04
+#define OPAE_PROP_BMC   0x08
+#define OPAE_PROP_ALL   \
+	(OPAE_PROP_PCI | OPAE_PROP_FME | OPAE_PROP_PORT | OPAE_PROP_BMC)
+
+typedef struct {
+	opae_pci_property pci;
+	opae_fme_property fme;
+	opae_port_property port[OPAE_MAX_PORT_NUM];
+	opae_bmc_property bmc;
+} opae_fpga_property;
+
+typedef struct {
+	uint64_t guid_h;
+	uint64_t guid_l;
+	uint32_t metadata_len;
+} gbs_header;
+
+#define OPAE_IMG_TYPE_BBS       0
+#define OPAE_IMG_TYPE_BMC       1
+#define OPAE_IMG_TYPE_GBS       2
+#define OPAE_IMG_TYPE(t)        ((t) & 0xff)
+
+#define OPAE_IMG_SUBTYPE_UPDATE             0
+#define OPAE_IMG_SUBTYPE_CANCELLATION       1
+#define OPAE_IMG_SUBTYPE_ROOT_KEY_HASH_256  2
+#define OPAE_IMG_SUBTYPE_ROOT_KEY_HASH_384  3
+#define OPAE_IMG_SUBTYPE(t)     (((t) >> 8) & 0xff)
+
+#define OPAE_IMG_BLK0_SIZE      128
+#define OPAE_IMG_BLK0_MAGIC     0xb6eafd19
+#define OPAE_IMG_BLK1_SIZE      896
+#define OPAE_IMG_HDR_SIZE   (OPAE_IMG_BLK0_SIZE + OPAE_IMG_BLK1_SIZE)
+#define OPAE_IMG_PL_MIN_SIZE    128
+
+typedef struct {
+	uint32_t magic;
+	uint32_t payload_len;
+	uint32_t payload_type;
+} opae_img_hdr;
+
+typedef struct {
+	int type;
+	int subtype;
+	uint32_t total_len;
+	uint32_t payload_offset;
+	uint32_t payload_len;
+} opae_img_info;
+
+void opae_get_api_version(opae_api_version *version);
+int opae_set_log_level(int level);
+int opae_set_log_file(char *path, int clean);
+int opae_get_proc_type(void);
+int opae_get_parent(pcidev_id id, pcidev_id parent);
+int opae_get_child(pcidev_id id, pcidev_id child, int size);
+int opae_get_pf1(pcidev_id id, pcidev_id peer, int size);
+int opae_init_eal(int argc, char **argv);
+int opae_cleanup_eal(void);
+int opae_enumerate(opae_pci_id *filter, pcidev_id list, int size);
+int opae_probe_device(pcidev_id id);
+int opae_remove_device(pcidev_id id);
+int opae_unbind_driver(pcidev_id id);
+int opae_bind_driver(pcidev_id id, char *drv_name);
+int opae_get_property(pcidev_id id, opae_fpga_property *prop, int type);
+int opae_get_phy_info(pcidev_id id, opae_phy_info *info);
+int opae_partial_reconfigure(pcidev_id id, int port, const char *gbs);
+int opae_get_image_info(const char *image, opae_img_info *info);
+int opae_cancel_flash_update(pcidev_id id, int force);
+int opae_update_flash(pcidev_id id, const char *image, uint64_t *status);
+int opae_reboot_device(pcidev_id id, int type, int page);
+int opae_store_rsu_status(pcidev_id id, uint32_t status, uint32_t progress);
+int opae_load_rsu_status(pcidev_id id, uint32_t *status, uint32_t *progress);
+int opae_read_pci_cfg(pcidev_id id, uint32_t address, uint32_t *value);
+int opae_write_pci_cfg(pcidev_id id, uint32_t address, uint32_t value);
+
+#ifdef __cplusplus
+}
+#endif
+
+
+#endif  /* _OPAE_API_H */
diff --git a/examples/meson.build b/examples/meson.build
index b9ab24223f..b029715277 100644
--- a/examples/meson.build
+++ b/examples/meson.build
@@ -16,7 +16,7 @@ all_examples = [
 	'eventdev_pipeline',
 	'fips_validation', 'flow_classify',
 	'flow_filtering', 'helloworld',
-	'ioat',
+	'ioat', 'ifpga',
 	'ip_fragmentation', 'ip_pipeline',
 	'ip_reassembly', 'ipsec-secgw',
 	'ipv4_multicast', 'kni',
-- 
2.29.2

[dpdk-dev] [PATCH v15 0/4] raw/ifpga: add extra OPAE APIs

[Wei Huang]

Cyborg is part of OpenStack, it needs some OPAE APIs to manage
devices with Intel FPGA. The first three patches implement extra
APIs to meet Cyborg requirement. The last patch add an example
to show how to use these APIs.

Main changes from v14:
- fix compilation issue of format error

Wei Huang (4):
  raw/ifpga: add fpga rsu APIs
  raw/ifpga: add APIs to get fpga information
  raw/ifpga: add miscellaneous APIs
  examples/ifpga: add example for ifpga APIs

 MAINTAINERS                                |    3 +
 doc/api/doxy-api-index.md                  |    3 +-
 doc/api/doxy-api.conf.in                   |    1 +
 doc/guides/sample_app_ug/ifpga.rst         |  387 +++++
 doc/guides/sample_app_ug/index.rst         |    1 +
 drivers/raw/ifpga/base/ifpga_api.c         |   34 +
 drivers/raw/ifpga/base/ifpga_defines.h     |    1 +
 drivers/raw/ifpga/base/ifpga_feature_dev.c |   21 +
 drivers/raw/ifpga/base/ifpga_feature_dev.h |    1 +
 drivers/raw/ifpga/base/ifpga_fme.c         |   36 +-
 drivers/raw/ifpga/base/ifpga_fme_rsu.c     |  428 +++++
 drivers/raw/ifpga/base/ifpga_hw.h          |    1 +
 drivers/raw/ifpga/base/ifpga_sec_mgr.c     |  639 ++++++++
 drivers/raw/ifpga/base/ifpga_sec_mgr.h     |   93 ++
 drivers/raw/ifpga/base/meson.build         |    2 +
 drivers/raw/ifpga/base/opae_hw_api.c       |   77 +
 drivers/raw/ifpga/base/opae_hw_api.h       |   12 +
 drivers/raw/ifpga/base/opae_ifpga_hw_api.h |    1 +
 drivers/raw/ifpga/base/opae_intel_max10.c  |   48 +
 drivers/raw/ifpga/base/opae_intel_max10.h  |   44 +
 drivers/raw/ifpga/ifpga_rawdev.c           |   30 +
 drivers/raw/ifpga/ifpga_rawdev.h           |    7 +-
 drivers/raw/ifpga/meson.build              |    4 +-
 drivers/raw/ifpga/rte_pmd_ifpga.c          |  429 ++++++
 drivers/raw/ifpga/rte_pmd_ifpga.h          |  308 ++++
 drivers/raw/ifpga/version.map              |   18 +
 examples/ifpga/Makefile                    |   43 +
 examples/ifpga/commands.c                  | 1294 ++++++++++++++++
 examples/ifpga/commands.h                  |   16 +
 examples/ifpga/main.c                      |   38 +
 examples/ifpga/meson.build                 |   20 +
 examples/ifpga/opae_api.c                  | 1630 ++++++++++++++++++++
 examples/ifpga/opae_api.h                  |  244 +++
 examples/meson.build                       |    2 +-
 34 files changed, 5910 insertions(+), 6 deletions(-)
 create mode 100644 doc/guides/sample_app_ug/ifpga.rst
 create mode 100644 drivers/raw/ifpga/base/ifpga_fme_rsu.c
 create mode 100644 drivers/raw/ifpga/base/ifpga_sec_mgr.c
 create mode 100644 drivers/raw/ifpga/base/ifpga_sec_mgr.h
 create mode 100644 drivers/raw/ifpga/rte_pmd_ifpga.c
 create mode 100644 drivers/raw/ifpga/rte_pmd_ifpga.h
 create mode 100644 examples/ifpga/Makefile
 create mode 100644 examples/ifpga/commands.c
 create mode 100644 examples/ifpga/commands.h
 create mode 100644 examples/ifpga/main.c
 create mode 100644 examples/ifpga/meson.build
 create mode 100644 examples/ifpga/opae_api.c
 create mode 100644 examples/ifpga/opae_api.h

-- 
2.29.2

[dpdk-dev] [PATCH v15 1/4] raw/ifpga: add fpga rsu APIs

[Wei Huang]

RSU (Remote System Update) depends on secure manager which may be
different on various implementations, so a new secure manager device
is implemented for adapting such difference.
There are five APIs added:
1. rte_pmd_ifpga_get_dev_id() get raw device ID of ifpga device from
PCI address like 'Domain:Bus:Dev.Func'.
2. rte_pmd_ifpga_update_flash() update flash with specific image file.
3. rte_pmd_ifpga_stop_update() abort flash update process.
4. rte_pmd_ifpga_reboot_try() check current ifpga status and change it
to reboot status if it is idle.
5. rte_pmd_ifpga_reload() trigger full reconfiguration of ifpga device.

Signed-off-by: Wei Huang <wei.huang@intel.com>
Acked-by: Tianfei Zhang <tianfei.zhang@intel.com>
Acked-by: Rosen Xu <rosen.xu@intel.com>
---
v2: fix coding style issue in ifpga_fme_rsu.c and ifpga_sec_mgr.c
---
v3: fix compilation issues in ifpga_fme_rsu.c
---
v4: fix compilation issues in opae_intel_max10.c
---
v5: implement APIs in rte_pmd_ifpga.c
---
 doc/api/doxy-api-index.md                 |   3 +-
 drivers/raw/ifpga/base/ifpga_api.c        |  26 +
 drivers/raw/ifpga/base/ifpga_fme.c        |   8 +
 drivers/raw/ifpga/base/ifpga_fme_rsu.c    | 428 +++++++++++++++
 drivers/raw/ifpga/base/ifpga_hw.h         |   1 +
 drivers/raw/ifpga/base/ifpga_sec_mgr.c    | 639 ++++++++++++++++++++++
 drivers/raw/ifpga/base/ifpga_sec_mgr.h    |  93 ++++
 drivers/raw/ifpga/base/meson.build        |   2 +
 drivers/raw/ifpga/base/opae_hw_api.c      |  59 ++
 drivers/raw/ifpga/base/opae_hw_api.h      |  10 +
 drivers/raw/ifpga/base/opae_intel_max10.c |  48 ++
 drivers/raw/ifpga/base/opae_intel_max10.h |  44 ++
 drivers/raw/ifpga/meson.build             |   4 +-
 drivers/raw/ifpga/rte_pmd_ifpga.c         | 163 ++++++
 drivers/raw/ifpga/rte_pmd_ifpga.h         | 132 +++++
 drivers/raw/ifpga/version.map             |  11 +
 16 files changed, 1669 insertions(+), 2 deletions(-)
 create mode 100644 drivers/raw/ifpga/base/ifpga_fme_rsu.c
 create mode 100644 drivers/raw/ifpga/base/ifpga_sec_mgr.c
 create mode 100644 drivers/raw/ifpga/base/ifpga_sec_mgr.h
 create mode 100644 drivers/raw/ifpga/rte_pmd_ifpga.c
 create mode 100644 drivers/raw/ifpga/rte_pmd_ifpga.h

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 748514e243..8a48af1042 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -56,7 +56,8 @@ The public API headers are grouped by topics:
   [dpaa2_qdma]         (@ref rte_pmd_dpaa2_qdma.h),
   [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h),
   [dlb]                (@ref rte_pmd_dlb.h),
-  [dlb2]               (@ref rte_pmd_dlb2.h)
+  [dlb2]               (@ref rte_pmd_dlb2.h),
+  [ifpga]              (@ref rte_pmd_ifpga.h)
 
 - **memory**:
   [memseg]             (@ref rte_memory.h),
diff --git a/drivers/raw/ifpga/base/ifpga_api.c b/drivers/raw/ifpga/base/ifpga_api.c
index 1ff57fa188..1aedf150bc 100644
--- a/drivers/raw/ifpga/base/ifpga_api.c
+++ b/drivers/raw/ifpga/base/ifpga_api.c
@@ -5,6 +5,7 @@
 #include "ifpga_api.h"
 #include "ifpga_enumerate.h"
 #include "ifpga_feature_dev.h"
+#include "ifpga_sec_mgr.h"
 
 #include "opae_hw_api.h"
 
@@ -228,11 +229,36 @@ static int ifpga_mgr_get_board_info(struct opae_manager *mgr,
 	return 0;
 }
 
+static int ifpga_mgr_update_flash(struct opae_manager *mgr, const char *image,
+	u64 *status)
+{
+	struct ifpga_fme_hw *fme = mgr->data;
+
+	return fpga_update_flash(fme, image, status);
+}
+
+static int ifpga_mgr_stop_flash_update(struct opae_manager *mgr, int force)
+{
+	struct ifpga_fme_hw *fme = mgr->data;
+
+	return fpga_stop_flash_update(fme, force);
+}
+
+static int ifpga_mgr_reload(struct opae_manager *mgr, int type, int page)
+{
+	struct ifpga_fme_hw *fme = mgr->data;
+
+	return fpga_reload(fme, type, page);
+}
+
 struct opae_manager_ops ifpga_mgr_ops = {
 	.flash = ifpga_mgr_flash,
 	.get_eth_group_region_info = ifpga_mgr_get_eth_group_region_info,
 	.get_sensor_value = ifpga_mgr_get_sensor_value,
 	.get_board_info = ifpga_mgr_get_board_info,
+	.update_flash = ifpga_mgr_update_flash,
+	.stop_flash_update = ifpga_mgr_stop_flash_update,
+	.reload = ifpga_mgr_reload,
 };
 
 static int ifpga_mgr_read_mac_rom(struct opae_manager *mgr, int offset,
diff --git a/drivers/raw/ifpga/base/ifpga_fme.c b/drivers/raw/ifpga/base/ifpga_fme.c
index f29ff3159b..34fd9a818e 100644
--- a/drivers/raw/ifpga/base/ifpga_fme.c
+++ b/drivers/raw/ifpga/base/ifpga_fme.c
@@ -7,6 +7,7 @@
 #include "opae_intel_max10.h"
 #include "opae_i2c.h"
 #include "opae_at24_eeprom.h"
+#include "ifpga_sec_mgr.h"
 
 #define PWR_THRESHOLD_MAX       0x7F
 
@@ -1152,6 +1153,12 @@ static int fme_nios_spi_init(struct ifpga_feature *feature)
 	if (spi_self_checking(max10))
 		goto spi_fail;
 
+	ret = init_sec_mgr(fme);
+	if (ret) {
+		dev_err(fme, "security manager init fail\n");
+		goto spi_fail;
+	}
+
 	return ret;
 
 spi_fail:
@@ -1165,6 +1172,7 @@ static void fme_nios_spi_uinit(struct ifpga_feature *feature)
 {
 	struct ifpga_fme_hw *fme = (struct ifpga_fme_hw *)feature->parent;
 
+	release_sec_mgr(fme);
 	if (fme->max10_dev)
 		intel_max10_device_remove(fme->max10_dev);
 }
diff --git a/drivers/raw/ifpga/base/ifpga_fme_rsu.c b/drivers/raw/ifpga/base/ifpga_fme_rsu.c
new file mode 100644
index 0000000000..28198abd78
--- /dev/null
+++ b/drivers/raw/ifpga/base/ifpga_fme_rsu.c
@@ -0,0 +1,428 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <fcntl.h>
+#include <signal.h>
+#include <unistd.h>
+#include "ifpga_sec_mgr.h"
+
+static struct ifpga_sec_mgr *sec_mgr;
+
+static void set_rsu_control(struct ifpga_sec_mgr *smgr, uint32_t ctrl)
+{
+	if (smgr && smgr->rsu_control)
+		*smgr->rsu_control = ctrl;
+}
+
+static uint32_t get_rsu_control(struct ifpga_sec_mgr *smgr)
+{
+	if (smgr && smgr->rsu_control)
+		return *smgr->rsu_control;
+	return 0;
+}
+
+static void set_rsu_status(struct ifpga_sec_mgr *smgr, uint32_t status,
+	uint32_t progress)
+{
+	if (smgr && smgr->rsu_status)
+		*smgr->rsu_status = IFPGA_RSU_STATUS(status, progress);
+}
+
+static void get_rsu_status(struct ifpga_sec_mgr *smgr, uint32_t *status,
+	uint32_t *progress)
+{
+	if (smgr && smgr->rsu_status) {
+		if (status)
+			*status = IFPGA_RSU_GET_STAT(*smgr->rsu_status);
+		if (progress)
+			*progress = IFPGA_RSU_GET_PROG(*smgr->rsu_status);
+	}
+}
+
+static void sig_handler(int sig, siginfo_t *info, void *data)
+{
+	(void)(info);
+	(void)(data);
+
+	switch (sig) {
+	case SIGINT:
+		if (sec_mgr) {
+			dev_info(sec_mgr, "Interrupt secure flash update"
+				" by keyboard\n");
+			set_rsu_control(sec_mgr, IFPGA_RSU_ABORT);
+		}
+		break;
+	default:
+		break;
+	}
+}
+
+static void log_time(time_t t, const char *msg)
+{
+	uint32_t h = 0;
+	uint32_t m = 0;
+	uint32_t s = 0;
+
+	if (t < 60) {
+		s = (uint32_t)t;
+	} else if (t < 3600) {
+		s = (uint32_t)(t % 60);
+		m = (uint32_t)(t / 60);
+	} else {
+		s = (uint32_t)(t % 60);
+		m = (uint32_t)((t % 3600) / 60);
+		h = (uint32_t)(t / 3600);
+	}
+	printf("%s - %02u:%02u:%02u\n", msg, h, m, s);
+}
+
+static int start_flash_update(struct ifpga_sec_mgr *smgr)
+{
+	if (!smgr)
+		return -ENODEV;
+
+	if (!smgr->ops || !smgr->ops->prepare)
+		return -EINVAL;
+
+	return smgr->ops->prepare(smgr);
+}
+
+static int write_flash_image(struct ifpga_sec_mgr *smgr, const char *image,
+	uint32_t offset)
+{
+	void *buf = NULL;
+	int retry = 0;
+	uint32_t length = 0;
+	uint32_t to_transfer = 0;
+	uint32_t one_percent = 0;
+	uint32_t prog = 0;
+	uint32_t old_prog = -1;
+	ssize_t read_size = 0;
+	int fd = -1;
+	int ret = 0;
+
+	if (!smgr)
+		return -ENODEV;
+
+	if (!smgr->ops || !smgr->ops->write_blk)
+		return -EINVAL;
+
+	fd = open(image, O_RDONLY);
+	if (fd < 0) {
+		dev_err(smgr,
+			"Failed to open \'%s\' for RD [e:%s]\n",
+			image, strerror(errno));
+		return -EIO;
+	}
+
+	buf = malloc(IFPGA_RSU_DATA_BLK_SIZE);
+	if (!buf) {
+		dev_err(smgr, "Failed to allocate memory for flash update\n");
+		close(fd);
+		return -ENOMEM;
+	}
+
+	length = smgr->rsu_length;
+	one_percent = length / 100;
+	do {
+		to_transfer = (length > IFPGA_RSU_DATA_BLK_SIZE) ?
+			IFPGA_RSU_DATA_BLK_SIZE : length;
+		lseek(fd, offset, SEEK_SET);
+		read_size = read(fd, buf, to_transfer);
+		if (read_size < 0) {
+			dev_err(smgr, "Failed to read from \'%s\' [e:%s]\n",
+				image, strerror(errno));
+			ret = -EIO;
+			goto end;
+		}
+		if ((uint32_t)read_size != to_transfer) {
+			dev_err(smgr,
+				"Read length %zd is not expected [e:%u]\n",
+				read_size, to_transfer);
+			ret = -EIO;
+			goto end;
+		}
+
+		retry = 0;
+		do {
+			if (get_rsu_control(smgr) == IFPGA_RSU_ABORT) {
+				ret = -EAGAIN;
+				goto end;
+			}
+			ret = smgr->ops->write_blk(smgr, buf, offset,
+				to_transfer);
+			if (ret == 0)
+				break;
+			sleep(1);
+		} while (++retry <= IFPGA_RSU_WRITE_RETRY);
+		if (retry > IFPGA_RSU_WRITE_RETRY) {
+			dev_err(smgr, "Failed to write to staging area 0x%x\n",
+				offset);
+			ret = -EAGAIN;
+			goto end;
+		}
+
+		length -= to_transfer;
+		offset += to_transfer;
+		prog = offset / one_percent;
+		if (prog != old_prog) {
+			printf("\r%d%%", prog);
+			fflush(stdout);
+			set_rsu_status(smgr, IFPGA_RSU_READY, prog);
+			old_prog = prog;
+		}
+	} while (length > 0);
+	set_rsu_status(smgr, IFPGA_RSU_READY, 100);
+	printf("\n");
+
+end:
+	free(buf);
+	close(fd);
+	return ret;
+}
+
+static int apply_flash_update(struct ifpga_sec_mgr *smgr)
+{
+	uint32_t one_percent = 0;
+	uint32_t one_percent_time = 0;
+	uint32_t prog = 0;
+	uint32_t old_prog = -1;
+	uint32_t copy_time = 0;
+	int ret = 0;
+
+	if (!smgr)
+		return -ENODEV;
+
+	if (!smgr->ops || !smgr->ops->write_done || !smgr->ops->check_complete)
+		return -EINVAL;
+
+	if (smgr->ops->write_done(smgr) < 0) {
+		dev_err(smgr, "Failed to apply flash update\n");
+		return -EAGAIN;
+	}
+
+	one_percent = (smgr->rsu_length + 99) / 100;
+	if (smgr->copy_speed == 0)   /* avoid zero divide fault */
+		smgr->copy_speed = 1;
+	one_percent_time = (one_percent + smgr->copy_speed - 1) /
+		smgr->copy_speed;
+	if (one_percent_time == 0)   /* avoid zero divide fault */
+		one_percent_time = 1;
+
+	do {
+		ret = smgr->ops->check_complete(smgr);
+		if (ret != -EAGAIN)
+			break;
+		sleep(1);
+		copy_time += 1;
+		prog = copy_time / one_percent_time;
+		if (prog >= 100)
+			prog = 99;
+		if (prog != old_prog) {
+			printf("\r%d%%", prog);
+			fflush(stdout);
+			set_rsu_status(smgr, IFPGA_RSU_COPYING, prog);
+			old_prog = prog;
+		}
+	} while (true);
+
+	if (ret < 0) {
+		printf("\n");
+		dev_err(smgr, "Failed to complete secure flash update\n");
+	} else {
+		printf("\r100%%\n");
+		set_rsu_status(smgr, IFPGA_RSU_COPYING, 100);
+	}
+
+	return ret;
+}
+
+static int secure_update_cancel(struct ifpga_sec_mgr *smgr)
+{
+	if (!smgr)
+		return -ENODEV;
+
+	if (!smgr->ops || !smgr->ops->cancel)
+		return -EINVAL;
+
+	return smgr->ops->cancel(smgr);
+}
+
+static int secure_update_status(struct ifpga_sec_mgr *smgr, uint64_t *status)
+{
+	if (!smgr)
+		return -ENODEV;
+
+	if (!smgr->ops || !smgr->ops->get_hw_errinfo)
+		return -EINVAL;
+
+	if (status)
+		*status = smgr->ops->get_hw_errinfo(smgr);
+
+	return 0;
+}
+
+int fpga_update_flash(struct ifpga_fme_hw *fme, const char *image,
+	uint64_t *status)
+{
+	struct ifpga_hw *hw = NULL;
+	struct ifpga_sec_mgr *smgr = NULL;
+	uint32_t rsu_stat = 0;
+	int fd = -1;
+	struct sigaction old_sigint_action;
+	struct sigaction sa;
+	time_t start;
+	int ret = 0;
+
+	if (!fme || !image || !status) {
+		dev_err(fme, "Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	hw = (struct ifpga_hw *)fme->parent;
+	if (!hw) {
+		dev_err(fme, "Parent of FME not found\n");
+		return -ENODEV;
+	}
+
+	smgr = (struct ifpga_sec_mgr *)fme->sec_mgr;
+	if (!smgr || !smgr->max10_dev) {
+		dev_err(smgr, "Security manager not initialized\n");
+		return -ENODEV;
+	}
+
+	opae_adapter_lock(hw->adapter, -1);
+	get_rsu_status(smgr, &rsu_stat, NULL);
+	if (rsu_stat != IFPGA_RSU_IDLE) {
+		opae_adapter_unlock(hw->adapter);
+		if (rsu_stat == IFPGA_RSU_REBOOT)
+			dev_info(smgr, "Reboot is in progress\n");
+		else
+			dev_info(smgr, "Update is in progress\n");
+		return -EAGAIN;
+	}
+	set_rsu_control(smgr, 0);
+	set_rsu_status(smgr, IFPGA_RSU_PREPARE, 0);
+	opae_adapter_unlock(hw->adapter);
+
+	fd = open(image, O_RDONLY);
+	if (fd < 0) {
+		dev_err(smgr,
+			"Failed to open \'%s\' for RD [e:%s]\n",
+			image, strerror(errno));
+		return -EIO;
+	}
+	smgr->rsu_length = lseek(fd, 0, SEEK_END);
+	close(fd);
+
+	if (smgr->max10_dev->staging_area_size < smgr->rsu_length) {
+		dev_err(dev, "Size of staging area is small than image length "
+			"[%u<%u]\n", smgr->max10_dev->staging_area_size,
+			smgr->rsu_length);
+		return -EINVAL;
+	}
+
+	printf("Updating from file \'%s\' with size %u\n",
+		image, smgr->rsu_length);
+
+	sec_mgr = smgr;
+	memset(&sa, 0, sizeof(struct sigaction));
+	sa.sa_flags = SA_SIGINFO | SA_RESETHAND;
+	sa.sa_sigaction = sig_handler;
+	ret = sigaction(SIGINT, &sa, &old_sigint_action);
+	if (ret < 0) {
+		dev_warn(dev, "Failed to register signal handler"
+			" [e:%d]\n", ret);
+		sec_mgr = NULL;
+	}
+
+	start = time(NULL);
+	log_time(time(NULL) - start, "Starting secure flash update");
+	ret = start_flash_update(smgr);
+	if (ret < 0)
+		goto end;
+
+	set_rsu_status(smgr, IFPGA_RSU_READY, 0);
+	log_time(time(NULL) - start, "Writing to staging area");
+	ret = write_flash_image(smgr, image, 0);
+	if (ret < 0)
+		goto end;
+
+	set_rsu_status(smgr, IFPGA_RSU_COPYING, 0);
+	log_time(time(NULL) - start, "Applying secure flash update");
+	ret = apply_flash_update(smgr);
+
+end:
+	if (sec_mgr) {
+		sec_mgr = NULL;
+		if (sigaction(SIGINT, &old_sigint_action, NULL) < 0)
+			dev_err(smgr, "Failed to unregister signal handler\n");
+	}
+
+	secure_update_status(smgr, status);
+	if (ret < 0) {
+		log_time(time(NULL) - start, "Secure flash update ERROR");
+		if (ret == -EAGAIN)
+			secure_update_cancel(smgr);
+	} else {
+		log_time(time(NULL) - start, "Secure flash update OK");
+	}
+	set_rsu_status(smgr, IFPGA_RSU_IDLE, 0);
+
+	return ret;
+}
+
+int fpga_stop_flash_update(struct ifpga_fme_hw *fme, int force)
+{
+	struct ifpga_sec_mgr *smgr = NULL;
+	uint32_t status = 0;
+	int retry = IFPGA_RSU_CANCEL_RETRY;
+	int ret = 0;
+
+	if (!fme) {
+		dev_err(fme, "Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+	smgr = (struct ifpga_sec_mgr *)fme->sec_mgr;
+
+	get_rsu_status(smgr, &status, NULL);
+	if (status != IFPGA_RSU_IDLE) {
+		dev_info(smgr, "Cancel secure flash update\n");
+		set_rsu_control(smgr, IFPGA_RSU_ABORT);
+	}
+
+	if (force) {
+		sleep(2);
+		do {
+			get_rsu_status(smgr, &status, NULL);
+			if (status == IFPGA_RSU_IDLE)
+				break;
+			if (secure_update_cancel(smgr) == 0)
+				set_rsu_status(smgr, IFPGA_RSU_IDLE, 0);
+			sleep(1);
+		} while (--retry > 0);
+		if (retry <= 0) {
+			dev_err(smgr, "Failed to stop flash update\n");
+			ret = -EAGAIN;
+		}
+	}
+
+	return ret;
+}
+
+int fpga_reload(struct ifpga_fme_hw *fme, int type, int page)
+{
+	struct ifpga_sec_mgr *smgr = NULL;
+
+	if (!fme) {
+		dev_err(fme, "Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+	smgr = (struct ifpga_sec_mgr *)fme->sec_mgr;
+
+	if (!smgr || !smgr->ops || !smgr->ops->reload)
+		return -EINVAL;
+
+	return smgr->ops->reload(smgr, type, page);
+}
diff --git a/drivers/raw/ifpga/base/ifpga_hw.h b/drivers/raw/ifpga/base/ifpga_hw.h
index 7c3307fe77..ed5edc6016 100644
--- a/drivers/raw/ifpga/base/ifpga_hw.h
+++ b/drivers/raw/ifpga/base/ifpga_hw.h
@@ -91,6 +91,7 @@ struct ifpga_fme_hw {
 	struct opae_board_info board_info;
 	int nums_eth_dev;
 	unsigned int nums_acc_region;
+	void *sec_mgr;
 };
 
 enum ifpga_port_state {
diff --git a/drivers/raw/ifpga/base/ifpga_sec_mgr.c b/drivers/raw/ifpga/base/ifpga_sec_mgr.c
new file mode 100644
index 0000000000..4cf1db3049
--- /dev/null
+++ b/drivers/raw/ifpga/base/ifpga_sec_mgr.c
@@ -0,0 +1,639 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <fcntl.h>
+#include <signal.h>
+#include <unistd.h>
+#include "ifpga_sec_mgr.h"
+
+
+static const char * const rsu_prog[] = {"IDLE", "PREPARING", "SLEEPING",
+	"READY", "AUTHENTICATING", "COPYING", "CANCELLATION", "PROGRAMMING_KEY",
+	"DONE", "PKVL_DONE"};
+static const char * const rsu_statl[] = {"NORMAL", "TIMEOUT", "AUTH_FAIL",
+	"COPY_FAIL", "FATAL", "PKVL_REJECT", "NON_INCR", "ERASE_FAIL",
+	"WEAROUT"};
+static const char * const rsu_stath[] = {"NIOS_OK", "USER_OK", "FACTORY_OK",
+	"USER_FAIL", "FACTORY_FAIL", "NIOS_FLASH_ERR", "FPGA_FLASH_ERR"};
+
+static const char *rsu_progress_name(uint32_t prog)
+{
+	if (prog > SEC_PROGRESS_PKVL_PROM_DONE)
+		return "UNKNOWN";
+	else
+		return rsu_prog[prog];
+}
+
+static const char *rsu_status_name(uint32_t stat)
+{
+	if (stat >= SEC_STATUS_NIOS_OK) {
+		if (stat > SEC_STATUS_FPGA_FLASH_ERR)
+			return "UNKNOWN";
+		else
+			return rsu_stath[stat-SEC_STATUS_NIOS_OK];
+	} else {
+		if (stat > SEC_STATUS_WEAROUT)
+			return "UNKNOWN";
+		else
+			return rsu_statl[stat];
+	}
+}
+
+static bool secure_start_done(uint32_t doorbell)
+{
+	return (SEC_STATUS_G(doorbell) == SEC_STATUS_ERASE_FAIL ||
+		SEC_STATUS_G(doorbell) == SEC_STATUS_WEAROUT ||
+		(SEC_PROGRESS_G(doorbell) != SEC_PROGRESS_IDLE &&
+		SEC_PROGRESS_G(doorbell) != SEC_PROGRESS_RSU_DONE));
+}
+
+static bool secure_prog_ready(uint32_t doorbell)
+{
+	return (SEC_PROGRESS_G(doorbell) != SEC_PROGRESS_READY);
+}
+
+static int poll_timeout(struct intel_max10_device *dev, uint32_t offset,
+	bool (*cond)(uint32_t), uint32_t interval_ms, uint32_t timeout_ms)
+{
+	uint32_t val = 0;
+	int ret = 0;
+
+	for (;;) {
+		ret = max10_sys_read(dev, offset, &val);
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to read max10 register 0x%x [e:%d]\n",
+				offset, ret);
+			break;
+		}
+
+		if (cond(val)) {
+			dev_debug(dev,
+				"Read 0x%08x from max10 register 0x%x "
+				"[poll success]\n", val, offset);
+			ret = 0;
+			break;
+		}
+		if (timeout_ms > interval_ms)
+			timeout_ms -= interval_ms;
+		else
+			timeout_ms = 0;
+		if (timeout_ms == 0) {
+			dev_debug(dev,
+				"Read 0x%08x from max10 register 0x%x "
+				"[poll timeout]\n", val, offset);
+			ret = -ETIMEDOUT;
+			break;
+		}
+		msleep(interval_ms);
+	}
+
+	return ret;
+}
+
+static int n3000_secure_update_start(struct intel_max10_device *dev)
+{
+	uint32_t doorbell = 0;
+	uint32_t prog = 0;
+	uint32_t status = 0;
+	int ret = 0;
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	prog = SEC_PROGRESS_G(doorbell);
+	if ((prog != SEC_PROGRESS_IDLE) && (prog != SEC_PROGRESS_RSU_DONE)) {
+		dev_debug(dev, "Current RSU progress is %s\n",
+			rsu_progress_name(prog));
+		return -EBUSY;
+	}
+
+	ret = max10_sys_update_bits(dev, MAX10_DOORBELL,
+		RSU_REQUEST | HOST_STATUS, RSU_REQUEST);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to updt max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	ret = poll_timeout(dev, MAX10_DOORBELL, secure_start_done,
+		IFPGA_SEC_START_INTERVAL_MS, IFPGA_SEC_START_TIMEOUT_MS);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to poll max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	status = SEC_STATUS_G(doorbell);
+	if (status == SEC_STATUS_WEAROUT)
+		return -EAGAIN;
+
+	if (status == SEC_STATUS_ERASE_FAIL)
+		return -EIO;
+
+	return 0;
+}
+
+static int n3000_cancel(struct ifpga_sec_mgr *smgr)
+{
+	struct intel_max10_device *dev = NULL;
+	uint32_t doorbell = 0;
+	uint32_t prog = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+	dev = (struct intel_max10_device *)smgr->max10_dev;
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	prog = SEC_PROGRESS_G(doorbell);
+	if (prog == SEC_PROGRESS_IDLE)
+		return 0;
+	if (prog != SEC_PROGRESS_READY)
+		return -EBUSY;
+
+	return max10_sys_update_bits(dev, MAX10_DOORBELL, HOST_STATUS,
+		HOST_STATUS_S(HOST_STATUS_ABORT_RSU));
+}
+
+static int n3000_prepare(struct ifpga_sec_mgr *smgr)
+{
+	struct intel_max10_device *dev = NULL;
+	int retry = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+	dev = (struct intel_max10_device *)smgr->max10_dev;
+
+	ret = n3000_secure_update_start(dev);
+	if (ret == -EBUSY)
+		n3000_cancel(smgr);
+
+	while (ret) {
+		if (++retry > IFPGA_RSU_START_RETRY)
+			break;
+		msleep(1000);
+		ret = n3000_secure_update_start(dev);
+	}
+	if (retry > IFPGA_RSU_START_RETRY) {
+		dev_err(dev, "Failed to start secure flash update\n");
+		ret = -EAGAIN;
+	}
+
+	return ret;
+}
+
+static int n3000_bulk_write(struct intel_max10_device *dev, uint32_t addr,
+	char *buf, uint32_t len)
+{
+	uint32_t i = 0;
+	uint32_t n = 0;
+	uint32_t v = 0;
+	uint32_t p = 0;
+	int ret = 0;
+
+	if (len & 0x3) {
+		dev_err(dev,
+			"Length of data block is not 4 bytes aligned [e:%u]\n",
+			len);
+		return -EINVAL;
+	}
+
+	n = len >> 2;
+	for (i = 0; i < n; i++) {
+		p = i << 2;
+		v = *(uint32_t *)(buf + p);
+		ret = max10_reg_write(dev, addr + p, v);
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to write to staging area 0x%08x [e:%d]\n",
+				addr + p, ret);
+			return ret;
+		}
+		usleep(1);
+	}
+
+	return 0;
+}
+
+static int n3000_write_blk(struct ifpga_sec_mgr *smgr, char *buf,
+	uint32_t offset, uint32_t len)
+{
+	struct intel_max10_device *dev = NULL;
+	uint32_t doorbell = 0;
+	uint32_t prog = 0;
+	uint32_t m = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+	dev = (struct intel_max10_device *)smgr->max10_dev;
+
+	if (offset + len > dev->staging_area_size) {
+		dev_err(dev,
+			"Write position would be out of staging area [e:%u]\n",
+			dev->staging_area_size);
+		return -ENOMEM;
+	}
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	prog = SEC_PROGRESS_G(doorbell);
+	if (prog == SEC_PROGRESS_PREPARE)
+		return -EAGAIN;
+	else if (prog != SEC_PROGRESS_READY)
+		return -EBUSY;
+
+	m = len & 0x3;
+	if (m != 0)
+		len += 4 - m;   /* make length to 4 bytes align */
+
+	return n3000_bulk_write(dev, dev->staging_area_base + offset, buf, len);
+}
+
+static int n3000_write_done(struct ifpga_sec_mgr *smgr)
+{
+	struct intel_max10_device *dev = NULL;
+	uint32_t doorbell = 0;
+	uint32_t prog = 0;
+	uint32_t status = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+	dev = (struct intel_max10_device *)smgr->max10_dev;
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	prog = SEC_PROGRESS_G(doorbell);
+	if (prog != SEC_PROGRESS_READY)
+		return -EBUSY;
+
+	ret = max10_sys_update_bits(dev, MAX10_DOORBELL, HOST_STATUS,
+		HOST_STATUS_S(HOST_STATUS_WRITE_DONE));
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to update max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	ret = poll_timeout(dev, MAX10_DOORBELL, secure_prog_ready,
+		IFPGA_NIOS_HANDSHAKE_INTERVAL_MS,
+		IFPGA_NIOS_HANDSHAKE_TIMEOUT_MS);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to poll max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	status = SEC_STATUS_G(doorbell);
+	switch (status) {
+	case SEC_STATUS_NORMAL:
+	case SEC_STATUS_NIOS_OK:
+	case SEC_STATUS_USER_OK:
+	case SEC_STATUS_FACTORY_OK:
+		ret = 0;
+		break;
+	default:
+		ret = -EIO;
+		break;
+	}
+
+	return ret;
+}
+
+static int n3000_check_complete(struct ifpga_sec_mgr *smgr)
+{
+	struct intel_max10_device *dev = NULL;
+	uint32_t doorbell = 0;
+	uint32_t status = 0;
+	uint32_t prog = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+	dev = (struct intel_max10_device *)smgr->max10_dev;
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return ret;
+	}
+
+	status = SEC_STATUS_G(doorbell);
+	switch (status) {
+	case SEC_STATUS_NORMAL:
+	case SEC_STATUS_NIOS_OK:
+	case SEC_STATUS_USER_OK:
+	case SEC_STATUS_FACTORY_OK:
+	case SEC_STATUS_WEAROUT:
+		break;
+	default:
+		return -EIO;
+	}
+
+	prog = SEC_PROGRESS_G(doorbell);
+	switch (prog) {
+	case SEC_PROGRESS_IDLE:
+	case SEC_PROGRESS_RSU_DONE:
+		return 0;
+	case SEC_PROGRESS_AUTHENTICATING:
+	case SEC_PROGRESS_COPYING:
+	case SEC_PROGRESS_UPDATE_CANCEL:
+	case SEC_PROGRESS_PROGRAM_KEY_HASH:
+		return -EAGAIN;
+	case SEC_PROGRESS_PREPARE:
+	case SEC_PROGRESS_READY:
+		return -EBUSY;
+	default:
+		return -EIO;
+	}
+
+	return 0;
+}
+
+static int n3000_reload_fpga(struct intel_max10_device *dev, int page)
+{
+	int ret = 0;
+
+	dev_info(dev, "Reload FPGA\n");
+
+	if (!dev || ((page != 0) && (page != 1))) {
+		dev_err(dev, "Input parameter of %s is invalid\n", __func__);
+		ret = -EINVAL;
+		goto end;
+	}
+
+	if (dev->flags & MAX10_FLAGS_SECURE) {
+		ret = max10_sys_update_bits(dev, FPGA_RECONF_REG,
+			SFPGA_RP_LOAD, 0);
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to update max10 reconfig register [e:%d]\n",
+				ret);
+			goto end;
+		}
+		ret = max10_sys_update_bits(dev, FPGA_RECONF_REG,
+			SFPGA_RP_LOAD | SFPGA_RECONF_PAGE,
+			SFPGA_RP_LOAD | SFPGA_PAGE(page));
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to update max10 reconfig register [e:%d]\n",
+				ret);
+			goto end;
+		}
+	} else {
+		ret = max10_sys_update_bits(dev, RSU_REG, FPGA_RP_LOAD, 0);
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to update max10 rsu register [e:%d]\n",
+				ret);
+			goto end;
+		}
+		ret = max10_sys_update_bits(dev, RSU_REG,
+			FPGA_RP_LOAD | FPGA_RECONF_PAGE,
+			FPGA_RP_LOAD | FPGA_PAGE(page));
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to update max10 rsu register [e:%d]\n",
+				ret);
+			goto end;
+		}
+	}
+
+	ret = max10_sys_update_bits(dev, FPGA_RECONF_REG, COUNTDOWN_START, 0);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to update max10 reconfig register [e:%d]\n",
+			ret);
+		goto end;
+	}
+
+	ret = max10_sys_update_bits(dev, FPGA_RECONF_REG, COUNTDOWN_START,
+		COUNTDOWN_START);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to update max10 reconfig register [e:%d]\n",
+			ret);
+	}
+end:
+	if (ret < 0)
+		dev_err(dev, "Failed to reload FPGA\n");
+
+	return ret;
+}
+
+static int n3000_reload_bmc(struct intel_max10_device *dev, int page)
+{
+	uint32_t val = 0;
+	int ret = 0;
+
+	dev_info(dev, "Reload BMC\n");
+
+	if (!dev || ((page != 0) && (page != 1))) {
+		dev_err(dev, "Input parameter of %s is invalid\n", __func__);
+		ret = -EINVAL;
+		goto end;
+	}
+
+	if (dev->flags & MAX10_FLAGS_SECURE) {
+		ret = max10_sys_update_bits(dev, MAX10_DOORBELL,
+			CONFIG_SEL | REBOOT_REQ,
+			CONFIG_SEL_S(page) | REBOOT_REQ);
+	} else {
+		val = (page == 0) ? 0x1 : 0x3;
+		ret = max10_reg_write(dev, IFPGA_DUAL_CFG_CTRL1, val);
+		if (ret < 0) {
+			dev_err(dev,
+				"Failed to write to dual config1 register [e:%d]\n",
+				ret);
+			goto end;
+		}
+
+		ret = max10_reg_write(dev, IFPGA_DUAL_CFG_CTRL0, 0x1);
+		if (ret < 0) {
+			if (ret == -EIO) {
+				ret = 0;
+				goto end;
+			}
+			dev_err(dev,
+				"Failed to write to dual config0 register [e:%d]\n",
+				ret);
+		}
+	}
+
+end:
+	if (ret < 0)
+		dev_err(dev, "Failed to reload BMC\n");
+
+	return ret;
+}
+
+static int n3000_reload(struct ifpga_sec_mgr *smgr, int type, int page)
+{
+	int psel = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+
+	if (type == IFPGA_BOOT_TYPE_FPGA) {
+		psel = (page == IFPGA_BOOT_PAGE_FACTORY ? 0 : 1);
+		ret = n3000_reload_fpga(smgr->max10_dev, psel);
+	} else if (type == IFPGA_BOOT_TYPE_BMC) {
+		psel = (page == IFPGA_BOOT_PAGE_FACTORY ? 1 : 0);
+		ret = n3000_reload_bmc(smgr->max10_dev, psel);
+	} else {
+		ret = -EINVAL;
+	}
+
+	return ret;
+}
+
+static uint64_t n3000_get_hw_errinfo(struct ifpga_sec_mgr *smgr)
+{
+	struct intel_max10_device *dev = NULL;
+	uint32_t doorbell = 0;
+	uint32_t stat = 0;
+	uint32_t prog = 0;
+	uint32_t auth_result = 0;
+	int ret = 0;
+
+	if (!smgr || !smgr->max10_dev)
+		return -ENODEV;
+	dev = (struct intel_max10_device *)smgr->max10_dev;
+
+	ret = max10_sys_read(dev, MAX10_DOORBELL, &doorbell);
+	if (ret < 0) {
+		dev_err(dev, "Failed to read max10 doorbell register [e:%d]\n",
+			ret);
+		return -1;
+	}
+	stat = SEC_STATUS_G(doorbell);
+	prog = SEC_PROGRESS_G(doorbell);
+	dev_debug(dev, "Current RSU status is %s, progress is %s\n",
+		rsu_status_name(stat), rsu_progress_name(prog));
+
+	ret = max10_sys_read(dev, MAX10_AUTH_RESULT, &auth_result);
+	if (ret < 0) {
+		dev_err(dev,
+			"Failed to read authenticate result register [e:%d]\n",
+			ret);
+		return -1;
+	}
+
+	return (uint64_t)doorbell << 32 | (uint64_t)auth_result;
+}
+
+static const struct ifpga_sec_ops n3000_sec_ops = {
+	.prepare = n3000_prepare,
+	.write_blk = n3000_write_blk,
+	.write_done = n3000_write_done,
+	.check_complete = n3000_check_complete,
+	.reload = n3000_reload,
+	.cancel = n3000_cancel,
+	.cleanup = NULL,
+	.get_hw_errinfo = n3000_get_hw_errinfo,
+};
+
+int init_sec_mgr(struct ifpga_fme_hw *fme)
+{
+	struct ifpga_hw *hw = NULL;
+	opae_share_data *sd = NULL;
+	struct ifpga_sec_mgr *smgr = NULL;
+
+	if (!fme || !fme->max10_dev)
+		return -ENODEV;
+
+	smgr = (struct ifpga_sec_mgr *)malloc(sizeof(*smgr));
+	if (!smgr) {
+		dev_err(NULL, "Failed to allocate memory for security manager\n");
+		return -ENOMEM;
+	}
+	fme->sec_mgr = smgr;
+
+	hw = (struct ifpga_hw *)fme->parent;
+	if (hw && hw->adapter && hw->adapter->shm.ptr) {
+		sd = (opae_share_data *)hw->adapter->shm.ptr;
+		smgr->rsu_control = &sd->rsu_ctrl;
+		smgr->rsu_status = &sd->rsu_stat;
+	} else {
+		smgr->rsu_control = NULL;
+		smgr->rsu_status = NULL;
+	}
+
+	if ((hw->pci_data->device_id == IFPGA_N3000_DID) &&
+		(hw->pci_data->vendor_id == IFPGA_N3000_VID)) {
+		smgr->ops = &n3000_sec_ops;
+		smgr->copy_speed = IFPGA_N3000_COPY_SPEED;
+	} else {
+		dev_err(NULL, "No operation for security manager\n");
+		smgr->ops = NULL;
+	}
+
+	smgr->fme = fme;
+	smgr->max10_dev = fme->max10_dev;
+
+	return 0;
+}
+
+void release_sec_mgr(struct ifpga_fme_hw *fme)
+{
+	struct ifpga_sec_mgr *smgr = NULL;
+
+	if (fme) {
+		smgr = (struct ifpga_sec_mgr *)fme->sec_mgr;
+		if (smgr) {
+			fme->sec_mgr = NULL;
+			free(smgr);
+		}
+	}
+}
diff --git a/drivers/raw/ifpga/base/ifpga_sec_mgr.h b/drivers/raw/ifpga/base/ifpga_sec_mgr.h
new file mode 100644
index 0000000000..fbeba561f4
--- /dev/null
+++ b/drivers/raw/ifpga/base/ifpga_sec_mgr.h
@@ -0,0 +1,93 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _IFPGA_FME_RSU_H_
+#define _IFPGA_FME_RSU_H_
+
+
+#include "ifpga_hw.h"
+
+#define IFPGA_N3000_VID     0x8086
+#define IFPGA_N3000_DID     0x0b30
+
+#define IFPGA_BOOT_TYPE_FPGA     0
+#define IFPGA_BOOT_TYPE_BMC      1
+
+#define IFPGA_BOOT_PAGE_FACTORY  0
+#define IFPGA_BOOT_PAGE_USER     1
+
+#define IFPGA_RSU_DATA_BLK_SIZE  32768
+#define IFPGA_RSU_START_RETRY    120
+#define IFPGA_RSU_WRITE_RETRY    10
+#define IFPGA_RSU_CANCEL_RETRY   30
+
+#define IFPGA_N3000_COPY_SPEED   42700
+
+/* status */
+#define IFPGA_RSU_IDLE       0
+#define IFPGA_RSU_PREPARE    1
+#define IFPGA_RSU_READY      2
+#define IFPGA_RSU_COPYING    3
+#define IFPGA_RSU_REBOOT     4
+
+#define IFPGA_RSU_GET_STAT(v)  (((v) >> 16) & 0xffff)
+#define IFPGA_RSU_GET_PROG(v)  ((v) & 0xffff)
+#define IFPGA_RSU_STATUS(s, p) ((((s) << 16) & 0xffff0000) | ((p) & 0xffff))
+
+/* control */
+#define IFPGA_RSU_ABORT      1
+
+#define IFPGA_DUAL_CFG_CTRL0     0x200020
+#define IFPGA_DUAL_CFG_CTRL1     0x200024
+
+#define IFPGA_SEC_START_INTERVAL_MS       100
+#define IFPGA_SEC_START_TIMEOUT_MS        20000
+#define IFPGA_NIOS_HANDSHAKE_INTERVAL_MS  100
+#define IFPGA_NIOS_HANDSHAKE_TIMEOUT_MS   5000
+
+#define IFPGA_RSU_ERR_HW_ERROR		-1
+#define IFPGA_RSU_ERR_TIMEOUT		-2
+#define IFPGA_RSU_ERR_CANCELED		-3
+#define IFPGA_RSU_ERR_BUSY			-4
+#define IFPGA_RSU_ERR_INVALID_SIZE	-5
+#define IFPGA_RSU_ERR_RW_ERROR		-6
+#define IFPGA_RSU_ERR_WEAROUT		-7
+#define IFPGA_RSU_ERR_FILE_READ		-8
+
+struct ifpga_sec_mgr;
+
+struct ifpga_sec_ops {
+	int (*prepare)(struct ifpga_sec_mgr *smgr);
+	int (*write_blk)(struct ifpga_sec_mgr *smgr, char *buf, uint32_t offset,
+		uint32_t size);
+	int (*write_done)(struct ifpga_sec_mgr *smgr);
+	int (*check_complete)(struct ifpga_sec_mgr *smgr);
+	int (*reload)(struct ifpga_sec_mgr *smgr, int type, int page);
+	int (*cancel)(struct ifpga_sec_mgr *smgr);
+	void (*cleanup)(struct ifpga_sec_mgr *smgr);
+	u64 (*get_hw_errinfo)(struct ifpga_sec_mgr *smgr);
+};
+
+struct ifpga_sec_mgr {
+	struct ifpga_fme_hw *fme;
+	struct intel_max10_device *max10_dev;
+	unsigned int rsu_length;
+	/* number of bytes that copied from staging area to working area
+	 * in one second, which is calculated by experiment
+	 */
+	unsigned int copy_speed;
+	unsigned int *rsu_control;
+	unsigned int *rsu_status;
+	const struct ifpga_sec_ops *ops;
+};
+
+int init_sec_mgr(struct ifpga_fme_hw *fme);
+void release_sec_mgr(struct ifpga_fme_hw *fme);
+int fpga_update_flash(struct ifpga_fme_hw *fme, const char *image,
+	uint64_t *status);
+int fpga_stop_flash_update(struct ifpga_fme_hw *fme, int force);
+int fpga_reload(struct ifpga_fme_hw *fme, int type, int page);
+
+
+#endif /* _IFPGA_FME_RSU_H_ */
diff --git a/drivers/raw/ifpga/base/meson.build b/drivers/raw/ifpga/base/meson.build
index da2d6e33ca..3549afafa1 100644
--- a/drivers/raw/ifpga/base/meson.build
+++ b/drivers/raw/ifpga/base/meson.build
@@ -12,6 +12,8 @@ sources = [
 	'ifpga_port.c',
 	'ifpga_port_error.c',
 	'ifpga_fme_pr.c',
+	'ifpga_fme_rsu.c',
+	'ifpga_sec_mgr.c',
 	'opae_hw_api.c',
 	'opae_ifpga_hw_api.c',
 	'opae_debug.c',
diff --git a/drivers/raw/ifpga/base/opae_hw_api.c b/drivers/raw/ifpga/base/opae_hw_api.c
index d5cd5fe608..86ad88f720 100644
--- a/drivers/raw/ifpga/base/opae_hw_api.c
+++ b/drivers/raw/ifpga/base/opae_hw_api.c
@@ -470,6 +470,8 @@ static void opae_adapter_shm_init(struct opae_adapter *adapter)
 	opae_mutex_init(&sd->i2c_mutex);
 	sd->ref_cnt = 0;
 	sd->dtb_size = SHM_BLK_SIZE;
+	sd->rsu_ctrl = 0;
+	sd->rsu_stat = 0;
 }
 
 static void *opae_adapter_shm_alloc(struct opae_adapter *adapter)
@@ -964,3 +966,60 @@ opae_mgr_get_board_info(struct opae_manager *mgr,
 
 	return -ENOENT;
 }
+
+/**
+ * opae_mgr_update_flash -  update image in flash.
+ * @mgr: targeted manager
+ * @image: name of image file
+ * @status: status of update
+ *
+ * Return: 0 on success, otherwise error code.
+ */
+int opae_mgr_update_flash(struct opae_manager *mgr, const char *image,
+	uint64_t *status)
+{
+	if (!mgr)
+		return -EINVAL;
+
+	if (mgr->ops && mgr->ops->update_flash)
+		return mgr->ops->update_flash(mgr, image, status);
+
+	return -ENOENT;
+}
+
+/**
+ * opae_stop_flash_update -  stop flash update.
+ * @mgr: targeted manager
+ * @force: make sure the update process is stopped
+ *
+ * Return: 0 on success, otherwise error code.
+ */
+int opae_mgr_stop_flash_update(struct opae_manager *mgr, int force)
+{
+	if (!mgr)
+		return -EINVAL;
+
+	if (mgr->ops && mgr->ops->stop_flash_update)
+		return mgr->ops->stop_flash_update(mgr, force);
+
+	return -ENOENT;
+}
+
+/**
+ * opae_mgr_reload -  reload FPGA.
+ * @mgr: targeted manager
+ * @type: FPGA type
+ * @page: reload from which page
+ *
+ * Return: 0 on success, otherwise error code.
+ */
+int opae_mgr_reload(struct opae_manager *mgr, int type, int page)
+{
+	if (!mgr)
+		return -EINVAL;
+
+	if (mgr->ops && mgr->ops->reload)
+		return mgr->ops->reload(mgr, type, page);
+
+	return -ENOENT;
+}
diff --git a/drivers/raw/ifpga/base/opae_hw_api.h b/drivers/raw/ifpga/base/opae_hw_api.h
index e99ee4564c..91d26d9b5b 100644
--- a/drivers/raw/ifpga/base/opae_hw_api.h
+++ b/drivers/raw/ifpga/base/opae_hw_api.h
@@ -55,6 +55,10 @@ struct opae_manager_ops {
 			unsigned int *value);
 	int (*get_board_info)(struct opae_manager *mgr,
 			struct opae_board_info **info);
+	int (*update_flash)(struct opae_manager *mgr, const char *image,
+			u64 *status);
+	int (*stop_flash_update)(struct opae_manager *mgr, int force);
+	int (*reload)(struct opae_manager *mgr, int type, int page);
 };
 
 /* networking management ops in FME */
@@ -276,6 +280,8 @@ typedef struct {
 			pthread_mutex_t i2c_mutex;
 			u32 ref_cnt;    /* reference count of shared memory */
 			u32 dtb_size;   /* actual length of DTB data in byte */
+			u32 rsu_ctrl;   /* used to control rsu */
+			u32 rsu_stat;   /* used to report status for rsu */
 		};
 	};
 	u8 dtb[SHM_BLK_SIZE];   /* DTB data */
@@ -354,4 +360,8 @@ int opae_manager_eth_group_read_reg(struct opae_manager *mgr, u8 group_id,
 		u8 type, u8 index, u16 addr, u32 *data);
 int opae_mgr_get_board_info(struct opae_manager *mgr,
 		struct opae_board_info **info);
+int opae_mgr_update_flash(struct opae_manager *mgr, const char *image,
+		uint64_t *status);
+int opae_mgr_stop_flash_update(struct opae_manager *mgr, int force);
+int opae_mgr_reload(struct opae_manager *mgr, int type, int page);
 #endif /* _OPAE_HW_API_H_*/
diff --git a/drivers/raw/ifpga/base/opae_intel_max10.c b/drivers/raw/ifpga/base/opae_intel_max10.c
index 1a526ea549..443e248fb3 100644
--- a/drivers/raw/ifpga/base/opae_intel_max10.c
+++ b/drivers/raw/ifpga/base/opae_intel_max10.c
@@ -51,6 +51,22 @@ int max10_sys_write(struct intel_max10_device *dev,
 	return max10_reg_write(dev, dev->base + offset, val);
 }
 
+int max10_sys_update_bits(struct intel_max10_device *dev, unsigned int offset,
+					unsigned int msk, unsigned int val)
+{
+	int ret = 0;
+	unsigned int temp = 0;
+
+	ret = max10_sys_read(dev, offset, &temp);
+	if (ret < 0)
+		return ret;
+
+	temp &= ~msk;
+	temp |= val & msk;
+
+	return max10_sys_write(dev, offset, temp);
+}
+
 static struct max10_compatible_id max10_id_table[] = {
 	{.compatible = MAX10_PAC,},
 	{.compatible = MAX10_PAC_N3000,},
@@ -557,6 +573,36 @@ static int check_max10_version(struct intel_max10_device *dev)
 	return -ENODEV;
 }
 
+static int max10_staging_area_init(struct intel_max10_device *dev)
+{
+	char *fdt_root = dev->fdt_root;
+	int ret, offset = 0;
+	u64 start, size;
+
+	if (!fdt_root) {
+		dev_debug(dev,
+			"skip staging area init as not find Device Tree\n");
+		return -ENODEV;
+	}
+
+	dev->staging_area_size = 0;
+
+	fdt_for_each_subnode(offset, fdt_root, 0) {
+		if (fdt_node_check_compatible(fdt_root, offset,
+					      "ifpga-sec-mgr,staging-area"))
+			continue;
+
+		ret = fdt_get_reg(fdt_root, offset, 0, &start, &size);
+		if (!ret) {
+			dev->staging_area_base = start;
+			dev->staging_area_size = size;
+		}
+		return ret;
+	}
+
+	return -ENODEV;
+}
+
 static int
 max10_secure_hw_init(struct intel_max10_device *dev)
 {
@@ -581,6 +627,8 @@ max10_secure_hw_init(struct intel_max10_device *dev)
 
 	max10_sensor_init(dev, sysmgr_offset);
 
+	max10_staging_area_init(dev);
+
 	return 0;
 }
 
diff --git a/drivers/raw/ifpga/base/opae_intel_max10.h b/drivers/raw/ifpga/base/opae_intel_max10.h
index 123cdc48b9..670683f017 100644
--- a/drivers/raw/ifpga/base/opae_intel_max10.h
+++ b/drivers/raw/ifpga/base/opae_intel_max10.h
@@ -38,6 +38,8 @@ struct intel_max10_device {
 	unsigned int base; /* max10 base address */
 	u16 bus;
 	struct opae_sensor_list opae_sensor_list;
+	u32 staging_area_base;
+	u32 staging_area_size;
 };
 
 /* retimer speed */
@@ -98,6 +100,7 @@ struct opae_retimer_status {
 #define   MAX10_MAC_COUNT	GENMASK(23, 16)
 #define RSU_REG			0x2c
 #define   FPGA_RECONF_PAGE	GENMASK(2, 0)
+#define   FPGA_PAGE(p)		((p) & 0x1)
 #define   FPGA_RP_LOAD		BIT(3)
 #define   NIOS2_PRERESET	BIT(4)
 #define   NIOS2_HANG		BIT(5)
@@ -106,6 +109,9 @@ struct opae_retimer_status {
 #define   NIOS2_I2C2_POLL_STOP	BIT(13)
 #define   PKVL_EEPROM_LOAD	BIT(31)
 #define FPGA_RECONF_REG		0x30
+#define   SFPGA_RECONF_PAGE	GENMASK(22, 20)
+#define   SFPGA_PAGE(p)		(((p) & 0x1) << 20)
+#define   SFPGA_RP_LOAD		BIT(23)
 #define MAX10_TEST_REG		0x3c
 #define   COUNTDOWN_START	BIT(18)
 #define MAX10_BUILD_VER		0x68
@@ -118,8 +124,44 @@ struct opae_retimer_status {
 #define MAX10_DOORBELL		0x400
 #define   RSU_REQUEST		BIT(0)
 #define   SEC_PROGRESS		GENMASK(7, 4)
+#define   SEC_PROGRESS_G(v)	(((v) >> 4) & 0xf)
+#define   SEC_PROGRESS_IDLE				0x0
+#define   SEC_PROGRESS_PREPARE			0x1
+#define   SEC_PROGRESS_SLEEP			0x2
+#define   SEC_PROGRESS_READY			0x3
+#define   SEC_PROGRESS_AUTHENTICATING	0x4
+#define   SEC_PROGRESS_COPYING			0x5
+#define   SEC_PROGRESS_UPDATE_CANCEL	0x6
+#define   SEC_PROGRESS_PROGRAM_KEY_HASH	0x7
+#define   SEC_PROGRESS_RSU_DONE			0x8
+#define   SEC_PROGRESS_PKVL_PROM_DONE	0x9
 #define   HOST_STATUS		GENMASK(11, 8)
+#define   HOST_STATUS_S(v)	(((v) << 8) & 0xf00)
+#define   HOST_STATUS_IDLE			0x0
+#define   HOST_STATUS_WRITE_DONE	0x1
+#define   HOST_STATUS_ABORT_RSU		0x2
 #define   SEC_STATUS		GENMASK(23, 16)
+#define   SEC_STATUS_G(v)	(((v) >> 16) & 0xff)
+#define   SEC_STATUS_NORMAL			0x0
+#define   SEC_STATUS_TIMEOUT		0x1
+#define   SEC_STATUS_AUTH_FAIL		0x2
+#define   SEC_STATUS_COPY_FAIL		0x3
+#define   SEC_STATUS_FATAL			0x4
+#define   SEC_STATUS_PKVL_REJECT	0x5
+#define   SEC_STATUS_NON_INC		0x6
+#define   SEC_STATUS_ERASE_FAIL		0x7
+#define   SEC_STATUS_WEAROUT		0x8
+#define   SEC_STATUS_NIOS_OK		0x80
+#define   SEC_STATUS_USER_OK		0x81
+#define   SEC_STATUS_FACTORY_OK		0x82
+#define   SEC_STATUS_USER_FAIL		0x83
+#define   SEC_STATUS_FACTORY_FAIL	0x84
+#define   SEC_STATUS_NIOS_FLASH_ERR	0x85
+#define   SEC_STATUS_FPGA_FLASH_ERR	0x86
+#define   CONFIG_SEL		BIT(28)
+#define   CONFIG_SEL_S(v)	(((v) & 0x1) << 28)
+#define   REBOOT_REQ		BIT(29)
+#define MAX10_AUTH_RESULT	0x404
 
 /* PKVL related registers, in system register region */
 #define PKVL_POLLING_CTRL		0x80
@@ -149,6 +191,8 @@ int max10_sys_read(struct intel_max10_device *dev,
 	unsigned int offset, unsigned int *val);
 int max10_sys_write(struct intel_max10_device *dev,
 	unsigned int offset, unsigned int val);
+int max10_sys_update_bits(struct intel_max10_device *dev,
+	unsigned int offset, unsigned int msk, unsigned int val);
 struct intel_max10_device *
 intel_max10_device_probe(struct altera_spi_device *spi,
 		int chipselect);
diff --git a/drivers/raw/ifpga/meson.build b/drivers/raw/ifpga/meson.build
index 027ff80562..60ea59ae28 100644
--- a/drivers/raw/ifpga/meson.build
+++ b/drivers/raw/ifpga/meson.build
@@ -13,8 +13,10 @@ objs = [base_objs]
 deps += ['ethdev', 'rawdev', 'pci', 'bus_pci', 'kvargs',
 	'bus_vdev', 'bus_ifpga', 'net', 'net_i40e', 'net_ipn3ke']
 
-sources = files('ifpga_rawdev.c')
+sources = files('ifpga_rawdev.c', 'rte_pmd_ifpga.c')
 
 includes += include_directories('base')
 includes += include_directories('../../net/ipn3ke')
 includes += include_directories('../../net/i40e')
+
+headers = files('rte_pmd_ifpga.h')
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.c b/drivers/raw/ifpga/rte_pmd_ifpga.c
new file mode 100644
index 0000000000..af6f175e98
--- /dev/null
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.c
@@ -0,0 +1,163 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 Intel Corporation
+ */
+
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+#include <rte_rawdev.h>
+#include <rte_rawdev_pmd.h>
+#include "rte_pmd_ifpga.h"
+#include "ifpga_rawdev.h"
+#include "base/ifpga_sec_mgr.h"
+
+
+int
+rte_pmd_ifpga_get_dev_id(const char *pci_addr, uint16_t *dev_id)
+{
+	struct rte_pci_addr addr;
+	struct rte_rawdev *rdev = NULL;
+	char rdev_name[RTE_RAWDEV_NAME_MAX_LEN] = {0};
+
+	if (!pci_addr || !dev_id) {
+		IFPGA_RAWDEV_PMD_ERR("Input parameter is invalid.");
+		return -EINVAL;
+	}
+
+	if (strnlen(pci_addr, PCI_PRI_STR_SIZE) == PCI_PRI_STR_SIZE) {
+		IFPGA_RAWDEV_PMD_ERR("PCI address is too long.");
+		return -EINVAL;
+	}
+
+	if (rte_pci_addr_parse(pci_addr, &addr)) {
+		IFPGA_RAWDEV_PMD_ERR("PCI address %s is invalid.", pci_addr);
+		return -EINVAL;
+	}
+
+	snprintf(rdev_name, RTE_RAWDEV_NAME_MAX_LEN, "IFPGA:%02x:%02x.%x",
+		addr.bus, addr.devid, addr.function);
+	rdev = rte_rawdev_pmd_get_named_dev(rdev_name);
+	if (!rdev) {
+		IFPGA_RAWDEV_PMD_DEBUG("%s is not probed by ifpga driver.",
+			pci_addr);
+		return -ENODEV;
+	}
+	*dev_id = rdev->dev_id;
+
+	return 0;
+}
+
+static struct rte_rawdev *
+get_rte_rawdev(uint16_t dev_id)
+{
+	struct rte_rawdev *dev = NULL;
+
+	if (dev_id >= RTE_RAWDEV_MAX_DEVS)
+		return NULL;
+
+	dev = &rte_rawdevs[dev_id];
+	if (dev->attached == RTE_RAWDEV_ATTACHED)
+		return dev;
+
+	return NULL;
+}
+
+static struct opae_adapter *
+get_opae_adapter(uint16_t dev_id)
+{
+	struct rte_rawdev *dev = NULL;
+	struct opae_adapter *adapter = NULL;
+
+	dev = get_rte_rawdev(dev_id);
+	if (!dev) {
+		IFPGA_RAWDEV_PMD_ERR("Device ID %u is invalid.", dev_id);
+		return NULL;
+	}
+
+	adapter = ifpga_rawdev_get_priv(dev);
+	if (!adapter) {
+		IFPGA_RAWDEV_PMD_ERR("Adapter is not registered.");
+		return NULL;
+	}
+
+	return adapter;
+}
+
+static opae_share_data *
+get_share_data(struct opae_adapter *adapter)
+{
+	opae_share_data *sd = NULL;
+
+	if (!adapter)
+		return NULL;
+
+	sd = (opae_share_data *)adapter->shm.ptr;
+	if (!sd) {
+		IFPGA_RAWDEV_PMD_ERR("Share data is not initialized.");
+		return NULL;
+	}
+
+	return sd;
+}
+
+int
+rte_pmd_ifpga_update_flash(uint16_t dev_id, const char *image,
+	uint64_t *status)
+{
+	struct opae_adapter *adapter = NULL;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	return opae_mgr_update_flash(adapter->mgr, image, status);
+}
+
+int
+rte_pmd_ifpga_stop_update(uint16_t dev_id, int force)
+{
+	struct opae_adapter *adapter = NULL;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	return opae_mgr_stop_flash_update(adapter->mgr, force);
+}
+
+int
+rte_pmd_ifpga_reboot_try(uint16_t dev_id)
+{
+	struct opae_adapter *adapter = NULL;
+	opae_share_data *sd = NULL;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	sd = get_share_data(adapter);
+	if (!sd)
+		return -ENOMEM;
+
+	opae_adapter_lock(adapter, -1);
+	if (IFPGA_RSU_GET_STAT(sd->rsu_stat) != IFPGA_RSU_IDLE) {
+		opae_adapter_unlock(adapter);
+		IFPGA_RAWDEV_PMD_WARN("Update or reboot is in progress.");
+		return -EBUSY;
+	}
+	sd->rsu_stat = IFPGA_RSU_STATUS(IFPGA_RSU_REBOOT, 0);
+	opae_adapter_unlock(adapter);
+
+	return 0;
+}
+
+int
+rte_pmd_ifpga_reload(uint16_t dev_id, int type, int page)
+{
+	struct opae_adapter *adapter = NULL;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	return opae_mgr_reload(adapter->mgr, type, page);
+}
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.h b/drivers/raw/ifpga/rte_pmd_ifpga.h
new file mode 100644
index 0000000000..023a011116
--- /dev/null
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.h
@@ -0,0 +1,132 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 Intel Corporation
+ */
+
+#ifndef _RTE_PMD_IFPGA_H_
+#define _RTE_PMD_IFPGA_H_
+
+/**
+ * @file rte_pmd_ifpga.h
+ *
+ * ifpga PMD specific functions.
+ *
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Get raw device ID from PCI address string like 'Domain:Bus:Dev.Func'
+ *
+ * @param pci_addr
+ *    The PCI address of specified Intel FPGA device.
+ * @param dev_id
+ *    The buffer to output device ID.
+ * @return
+ *   - (0) if successful.
+ *   - (-EINVAL) if bad parameter.
+ *   - (-ENODEV) if FPGA is not probed by ifpga driver.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_get_dev_id(const char *pci_addr, uint16_t *dev_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Update image flash of specified Intel FPGA device
+ *
+ * @param dev_id
+ *   The raw device ID of specified Intel FPGA device.
+ * @param image
+ *   The image file name string.
+ * @param status
+ *   The detailed update status for debug.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-EINVAL) if bad parameter or staging area is not initialized.
+ *   - (-EBUSY) if FPGA is updating or rebooting.
+ *   - (-EIO) if failed to open image file.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_update_flash(uint16_t dev_id, const char *image,
+	uint64_t *status);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Stop flash update of specified Intel FPGA device
+ *
+ * @param dev_id
+ *   The raw device ID of specified Intel FPGA device.
+ * @param force
+ *   Abort the update process by writing register if set non-zero.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-EINVAL) if bad parameter.
+ *   - (-EAGAIN) if failed with force.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_stop_update(uint16_t dev_id, int force);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Check current Intel FPGA status and change it to reboot status if it is idle
+ *
+ * @param dev_id
+ *    The raw device ID of specified Intel FPGA device.
+ * @return
+ *   - (0) if FPGA is ready to reboot.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-ENOMEM) if share data is not initialized.
+ *   - (-EBUSY) if FPGA is updating or rebooting.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_reboot_try(uint16_t dev_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Trigger full reconfiguration of specified Intel FPGA device
+ *
+ * @param dev_id
+ *    The raw device ID of specified Intel FPGA device.
+ * @param type
+ *    Select reconfiguration type.
+ *    0 - reconfigure FPGA only.
+ *    1 - reboot the whole card including FPGA.
+ * @param page
+ *    Select image from which flash partition.
+ *    0 - factory partition.
+ *    1 - user partition.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-EINVAL) if bad parameter.
+ *   - (-EBUSY) if failed to access BMC register.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_reload(uint16_t dev_id, int type, int page);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMD_IFPGA_H_ */
diff --git a/drivers/raw/ifpga/version.map b/drivers/raw/ifpga/version.map
index 4a76d1d52d..16584f7fe4 100644
--- a/drivers/raw/ifpga/version.map
+++ b/drivers/raw/ifpga/version.map
@@ -1,3 +1,14 @@
 DPDK_21 {
 	local: *;
 };
+
+EXPERIMENTAL {
+	global:
+
+	# added in 21.05
+	rte_pmd_ifpga_get_dev_id;
+	rte_pmd_ifpga_update_flash;
+	rte_pmd_ifpga_stop_update;
+	rte_pmd_ifpga_reboot_try;
+	rte_pmd_ifpga_reload;
+};
-- 
2.29.2

[dpdk-dev] [PATCH v15 2/4] raw/ifpga: add APIs to get fpga information

[Wei Huang]

There are some information data can be got from FPGA, they are
implemented in below APIs:
1. rte_pmd_ifpga_get_property() get properties of FPGA (include BMC).
2. rte_pmd_ifpga_get_phy_info() get information of PHY connect to FPGA.
3. rte_pmd_ifpga_get_rsu_status() get status of rsu process.

Signed-off-by: Wei Huang <wei.huang@intel.com>
Acked-by: Tianfei Zhang <tianfei.zhang@intel.com>
Acked-by: Rosen Xu <rosen.xu@intel.com>
---
v2: implement APIs in rte_pmd_ifpga.c
---
 drivers/raw/ifpga/base/ifpga_api.c         |   8 +
 drivers/raw/ifpga/base/ifpga_defines.h     |   1 +
 drivers/raw/ifpga/base/ifpga_feature_dev.c |  21 ++
 drivers/raw/ifpga/base/ifpga_feature_dev.h |   1 +
 drivers/raw/ifpga/base/ifpga_fme.c         |  28 ++-
 drivers/raw/ifpga/base/opae_hw_api.c       |  18 ++
 drivers/raw/ifpga/base/opae_hw_api.h       |   2 +
 drivers/raw/ifpga/base/opae_ifpga_hw_api.h |   1 +
 drivers/raw/ifpga/rte_pmd_ifpga.c          | 221 +++++++++++++++++++++
 drivers/raw/ifpga/rte_pmd_ifpga.h          | 110 ++++++++++
 drivers/raw/ifpga/version.map              |   3 +
 11 files changed, 412 insertions(+), 2 deletions(-)

diff --git a/drivers/raw/ifpga/base/ifpga_api.c b/drivers/raw/ifpga/base/ifpga_api.c
index 1aedf150bc..4610ef101e 100644
--- a/drivers/raw/ifpga/base/ifpga_api.c
+++ b/drivers/raw/ifpga/base/ifpga_api.c
@@ -229,6 +229,13 @@ static int ifpga_mgr_get_board_info(struct opae_manager *mgr,
 	return 0;
 }
 
+static int ifpga_mgr_get_uuid(struct opae_manager *mgr, struct uuid *uuid)
+{
+	struct ifpga_fme_hw *fme = mgr->data;
+
+	return fpga_get_pr_uuid(fme, uuid);
+}
+
 static int ifpga_mgr_update_flash(struct opae_manager *mgr, const char *image,
 	u64 *status)
 {
@@ -256,6 +263,7 @@ struct opae_manager_ops ifpga_mgr_ops = {
 	.get_eth_group_region_info = ifpga_mgr_get_eth_group_region_info,
 	.get_sensor_value = ifpga_mgr_get_sensor_value,
 	.get_board_info = ifpga_mgr_get_board_info,
+	.get_uuid = ifpga_mgr_get_uuid,
 	.update_flash = ifpga_mgr_update_flash,
 	.stop_flash_update = ifpga_mgr_stop_flash_update,
 	.reload = ifpga_mgr_reload,
diff --git a/drivers/raw/ifpga/base/ifpga_defines.h b/drivers/raw/ifpga/base/ifpga_defines.h
index 9f0147d1ed..dca1518a83 100644
--- a/drivers/raw/ifpga/base/ifpga_defines.h
+++ b/drivers/raw/ifpga/base/ifpga_defines.h
@@ -1727,6 +1727,7 @@ struct opae_board_info {
 	u8 seu;
 	u8 ptp;
 
+	u32 boot_page;
 	u32 max10_version;
 	u32 nios_fw_version;
 	u32 nums_of_retimer;
diff --git a/drivers/raw/ifpga/base/ifpga_feature_dev.c b/drivers/raw/ifpga/base/ifpga_feature_dev.c
index 0f852a75ad..08135137ad 100644
--- a/drivers/raw/ifpga/base/ifpga_feature_dev.c
+++ b/drivers/raw/ifpga/base/ifpga_feature_dev.c
@@ -87,6 +87,27 @@ int fpga_get_afu_uuid(struct ifpga_port_hw *port, struct uuid *uuid)
 	return 0;
 }
 
+int fpga_get_pr_uuid(struct ifpga_fme_hw *fme, struct uuid *uuid)
+{
+	struct feature_fme_pr *fme_pr;
+	u64 guidl, guidh;
+
+	if (!fme || !uuid)
+		return -EINVAL;
+
+	fme_pr = get_fme_feature_ioaddr_by_index(fme, FME_FEATURE_ID_PR_MGMT);
+
+	spinlock_lock(&fme->lock);
+	guidl = readq(&fme_pr->fme_pr_intfc_id_l);
+	guidh = readq(&fme_pr->fme_pr_intfc_id_h);
+	spinlock_unlock(&fme->lock);
+
+	opae_memcpy(uuid->b, &guidl, sizeof(u64));
+	opae_memcpy(uuid->b + 8, &guidh, sizeof(u64));
+
+	return 0;
+}
+
 /* Mask / Unmask Port Errors by the Error Mask register. */
 void port_err_mask(struct ifpga_port_hw *port, bool mask)
 {
diff --git a/drivers/raw/ifpga/base/ifpga_feature_dev.h b/drivers/raw/ifpga/base/ifpga_feature_dev.h
index 2b1309b44a..b355d22b0e 100644
--- a/drivers/raw/ifpga/base/ifpga_feature_dev.h
+++ b/drivers/raw/ifpga/base/ifpga_feature_dev.h
@@ -103,6 +103,7 @@ is_port_feature_present(struct ifpga_port_hw *port, int index)
 }
 
 int fpga_get_afu_uuid(struct ifpga_port_hw *port, struct uuid *uuid);
+int fpga_get_pr_uuid(struct ifpga_fme_hw *fme, struct uuid *uuid);
 
 int __fpga_port_disable(struct ifpga_port_hw *port);
 void __fpga_port_enable(struct ifpga_port_hw *port);
diff --git a/drivers/raw/ifpga/base/ifpga_fme.c b/drivers/raw/ifpga/base/ifpga_fme.c
index 34fd9a818e..43c7b9c3dc 100644
--- a/drivers/raw/ifpga/base/ifpga_fme.c
+++ b/drivers/raw/ifpga/base/ifpga_fme.c
@@ -101,6 +101,24 @@ static int fme_hdr_get_ports_num(struct ifpga_fme_hw *fme, u64 *ports_num)
 	return 0;
 }
 
+static int fme_hdr_get_port_type(struct ifpga_fme_hw *fme, u64 *port_type)
+{
+	struct feature_fme_header *fme_hdr
+		= get_fme_feature_ioaddr_by_index(fme, FME_FEATURE_ID_HEADER);
+	struct feature_fme_port pt;
+	u32 port = (u32)((*port_type >> 32) & 0xffffffff);
+
+	pt.csr = readq(&fme_hdr->port[port]);
+	if (!pt.port_implemented)
+		return -ENODEV;
+	if (pt.afu_access_control)
+		*port_type |= 0x1;
+	else
+		*port_type &= ~0x1;
+
+	return 0;
+}
+
 static int fme_hdr_get_cache_size(struct ifpga_fme_hw *fme, u64 *cache_size)
 {
 	struct feature_fme_header *fme_hdr
@@ -179,6 +197,8 @@ fme_hdr_get_prop(struct ifpga_feature *feature, struct feature_prop *prop)
 		return fme_hdr_get_bitstream_id(fme, &prop->data);
 	case FME_HDR_PROP_BITSTREAM_METADATA:
 		return fme_hdr_get_bitstream_metadata(fme, &prop->data);
+	case FME_HDR_PROP_PORT_TYPE:
+		return fme_hdr_get_port_type(fme, &prop->data);
 	}
 
 	return -ENOENT;
@@ -891,13 +911,17 @@ static int fme_get_board_interface(struct ifpga_fme_hw *fme)
 			fme->board_info.nums_of_fvl,
 			fme->board_info.ports_per_fvl);
 
+	if (max10_sys_read(fme->max10_dev, FPGA_PAGE_INFO, &val))
+		return -EINVAL;
+	fme->board_info.boot_page = val & 0x7;
+
 	if (max10_sys_read(fme->max10_dev, MAX10_BUILD_VER, &val))
 		return -EINVAL;
-	fme->board_info.max10_version = val & 0xffffff;
+	fme->board_info.max10_version = val;
 
 	if (max10_sys_read(fme->max10_dev, NIOS2_FW_VERSION, &val))
 		return -EINVAL;
-	fme->board_info.nios_fw_version = val & 0xffffff;
+	fme->board_info.nios_fw_version = val;
 
 	dev_info(fme, "max10 version 0x%x, nios fw version 0x%x\n",
 		fme->board_info.max10_version,
diff --git a/drivers/raw/ifpga/base/opae_hw_api.c b/drivers/raw/ifpga/base/opae_hw_api.c
index 86ad88f720..11c9887c7f 100644
--- a/drivers/raw/ifpga/base/opae_hw_api.c
+++ b/drivers/raw/ifpga/base/opae_hw_api.c
@@ -967,6 +967,24 @@ opae_mgr_get_board_info(struct opae_manager *mgr,
 	return -ENOENT;
 }
 
+/**
+ * opae_mgr_get_uuid -  get manager's UUID.
+ * @mgr: targeted manager
+ * @uuid: a pointer to UUID
+ *
+ * Return: 0 on success, otherwise error code.
+ */
+int opae_mgr_get_uuid(struct opae_manager *mgr, struct uuid *uuid)
+{
+	if (!mgr || !uuid)
+		return -EINVAL;
+
+	if (mgr->ops && mgr->ops->get_uuid)
+		return mgr->ops->get_uuid(mgr, uuid);
+
+	return -ENOENT;
+}
+
 /**
  * opae_mgr_update_flash -  update image in flash.
  * @mgr: targeted manager
diff --git a/drivers/raw/ifpga/base/opae_hw_api.h b/drivers/raw/ifpga/base/opae_hw_api.h
index 91d26d9b5b..7e04b56471 100644
--- a/drivers/raw/ifpga/base/opae_hw_api.h
+++ b/drivers/raw/ifpga/base/opae_hw_api.h
@@ -55,6 +55,7 @@ struct opae_manager_ops {
 			unsigned int *value);
 	int (*get_board_info)(struct opae_manager *mgr,
 			struct opae_board_info **info);
+	int (*get_uuid)(struct opae_manager *mgr, struct uuid *uuid);
 	int (*update_flash)(struct opae_manager *mgr, const char *image,
 			u64 *status);
 	int (*stop_flash_update)(struct opae_manager *mgr, int force);
@@ -360,6 +361,7 @@ int opae_manager_eth_group_read_reg(struct opae_manager *mgr, u8 group_id,
 		u8 type, u8 index, u16 addr, u32 *data);
 int opae_mgr_get_board_info(struct opae_manager *mgr,
 		struct opae_board_info **info);
+int opae_mgr_get_uuid(struct opae_manager *mgr, struct uuid *uuid);
 int opae_mgr_update_flash(struct opae_manager *mgr, const char *image,
 		uint64_t *status);
 int opae_mgr_stop_flash_update(struct opae_manager *mgr, int force);
diff --git a/drivers/raw/ifpga/base/opae_ifpga_hw_api.h b/drivers/raw/ifpga/base/opae_ifpga_hw_api.h
index bab33862ee..ffdbebf704 100644
--- a/drivers/raw/ifpga/base/opae_ifpga_hw_api.h
+++ b/drivers/raw/ifpga/base/opae_ifpga_hw_api.h
@@ -61,6 +61,7 @@ struct feature_prop {
 #define FME_HDR_PROP_SOCKET_ID		0x5	/* RDONLY */
 #define FME_HDR_PROP_BITSTREAM_ID		0x6	/* RDONLY */
 #define FME_HDR_PROP_BITSTREAM_METADATA	0x7	/* RDONLY */
+#define FME_HDR_PROP_PORT_TYPE		0x8	/* RDWR */
 
 /* FME error reporting feature's properties */
 /* FME error reporting properties format */
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.c b/drivers/raw/ifpga/rte_pmd_ifpga.c
index af6f175e98..8e04e22d5f 100644
--- a/drivers/raw/ifpga/rte_pmd_ifpga.c
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.c
@@ -8,6 +8,7 @@
 #include <rte_rawdev_pmd.h>
 #include "rte_pmd_ifpga.h"
 #include "ifpga_rawdev.h"
+#include "base/ifpga_api.h"
 #include "base/ifpga_sec_mgr.h"
 
 
@@ -99,6 +100,226 @@ get_share_data(struct opae_adapter *adapter)
 	return sd;
 }
 
+int
+rte_pmd_ifpga_get_rsu_status(uint16_t dev_id, uint32_t *stat, uint32_t *prog)
+{
+	struct opae_adapter *adapter = NULL;
+	opae_share_data *sd = NULL;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	sd = get_share_data(adapter);
+	if (!sd)
+		return -ENOMEM;
+
+	if (stat)
+		*stat = IFPGA_RSU_GET_STAT(sd->rsu_stat);
+	if (prog)
+		*prog = IFPGA_RSU_GET_PROG(sd->rsu_stat);
+
+	return 0;
+}
+
+static int
+ifpga_is_rebooting(struct opae_adapter *adapter)
+{
+	opae_share_data *sd = NULL;
+
+	sd = get_share_data(adapter);
+	if (!sd)
+		return 1;
+
+	if (IFPGA_RSU_GET_STAT(sd->rsu_stat) == IFPGA_RSU_REBOOT) {
+		IFPGA_RAWDEV_PMD_WARN("Reboot is in progress.");
+		return 1;
+	}
+
+	return 0;
+}
+
+static int
+get_common_property(struct opae_adapter *adapter,
+	rte_pmd_ifpga_common_prop *prop)
+{
+	struct ifpga_fme_hw *fme = NULL;
+	struct opae_board_info *info = NULL;
+	struct feature_prop fp;
+	struct uuid pr_id;
+	int ret = 0;
+
+	if (!adapter || !prop)
+		return -EINVAL;
+
+	if (!adapter->mgr || !adapter->mgr->data) {
+		IFPGA_RAWDEV_PMD_ERR("Manager is not registered.");
+		return -ENODEV;
+	}
+
+	fme = adapter->mgr->data;
+	fp.feature_id = FME_FEATURE_ID_HEADER;
+	fp.prop_id = FME_HDR_PROP_PORTS_NUM;
+	ret = ifpga_get_prop(fme->parent, FEATURE_FIU_ID_FME, 0, &fp);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get port number.");
+		return ret;
+	}
+	prop->num_ports = fp.data;
+
+	fp.prop_id = FME_HDR_PROP_BITSTREAM_ID;
+	ret = ifpga_get_prop(fme->parent, FEATURE_FIU_ID_FME, 0, &fp);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get bitstream ID.");
+		return ret;
+	}
+	prop->bitstream_id = fp.data;
+
+	fp.prop_id = FME_HDR_PROP_BITSTREAM_METADATA;
+	ret = ifpga_get_prop(fme->parent, FEATURE_FIU_ID_FME, 0, &fp);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get bitstream metadata.");
+		return ret;
+	}
+	prop->bitstream_metadata = fp.data;
+
+	ret = opae_mgr_get_uuid(adapter->mgr, &pr_id);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get PR ID.");
+		return ret;
+	}
+	memcpy(prop->pr_id.b, pr_id.b, sizeof(rte_pmd_ifpga_uuid));
+
+	ret = opae_mgr_get_board_info(adapter->mgr, &info);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get board info.");
+		return ret;
+	}
+	prop->boot_page = info->boot_page;
+	prop->bmc_version = info->max10_version;
+	prop->bmc_nios_version = info->nios_fw_version;
+
+	return 0;
+}
+
+static int
+get_port_property(struct opae_adapter *adapter, uint16_t port,
+	rte_pmd_ifpga_port_prop *prop)
+{
+	struct ifpga_fme_hw *fme = NULL;
+	struct feature_prop fp;
+	struct opae_accelerator *acc = NULL;
+	struct uuid afu_id;
+	int ret = 0;
+
+	if (!adapter || !prop)
+		return -EINVAL;
+
+	if (!adapter->mgr || !adapter->mgr->data) {
+		IFPGA_RAWDEV_PMD_ERR("Manager is not registered.");
+		return -ENODEV;
+	}
+
+	fme = adapter->mgr->data;
+	fp.feature_id = FME_FEATURE_ID_HEADER;
+	fp.prop_id = FME_HDR_PROP_PORT_TYPE;
+	fp.data = port;
+	fp.data <<= 32;
+	ret = ifpga_get_prop(fme->parent, FEATURE_FIU_ID_FME, 0, &fp);
+	if (ret)
+		return ret;
+	prop->type = fp.data & 0xffffffff;
+
+	if (prop->type == 0) {
+		acc = opae_adapter_get_acc(adapter, port);
+		ret = opae_acc_get_uuid(acc, &afu_id);
+		if (ret) {
+			IFPGA_RAWDEV_PMD_ERR("Failed to get port%u AFU ID.",
+				port);
+			return ret;
+		}
+		memcpy(prop->afu_id.b, afu_id.b, sizeof(rte_pmd_ifpga_uuid));
+	}
+
+	return 0;
+}
+
+int
+rte_pmd_ifpga_get_property(uint16_t dev_id, rte_pmd_ifpga_prop *prop)
+{
+	struct opae_adapter *adapter = NULL;
+	uint32_t i = 0;
+	int ret = 0;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	opae_adapter_lock(adapter, -1);
+	if (ifpga_is_rebooting(adapter)) {
+		ret = -EBUSY;
+		goto unlock_dev;
+	}
+
+	ret = get_common_property(adapter, &prop->common);
+	if (ret) {
+		ret = -EIO;
+		goto unlock_dev;
+	}
+
+	for (i = 0; i < prop->common.num_ports; i++) {
+		ret = get_port_property(adapter, i, &prop->port[i]);
+		if (ret) {
+			ret = -EIO;
+			break;
+		}
+	}
+
+unlock_dev:
+	opae_adapter_unlock(adapter);
+	return ret;
+}
+
+int
+rte_pmd_ifpga_get_phy_info(uint16_t dev_id, rte_pmd_ifpga_phy_info *info)
+{
+	struct opae_adapter *adapter = NULL;
+	struct opae_retimer_info rtm_info;
+	struct opae_retimer_status rtm_status;
+	int ret = 0;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	opae_adapter_lock(adapter, -1);
+	if (ifpga_is_rebooting(adapter)) {
+		ret = -EBUSY;
+		goto unlock_dev;
+	}
+
+	ret = opae_manager_get_retimer_info(adapter->mgr, &rtm_info);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get retimer info.");
+		ret = -EIO;
+		goto unlock_dev;
+	}
+	info->num_retimers = rtm_info.nums_retimer;
+
+	ret = opae_manager_get_retimer_status(adapter->mgr, &rtm_status);
+	if (ret) {
+		IFPGA_RAWDEV_PMD_ERR("Failed to get retimer status.");
+		ret = -EIO;
+		goto unlock_dev;
+	}
+	info->link_speed = rtm_status.speed;
+	info->link_status = rtm_status.line_link_bitmap;
+
+unlock_dev:
+	opae_adapter_unlock(adapter);
+	return ret;
+}
+
 int
 rte_pmd_ifpga_update_flash(uint16_t dev_id, const char *image,
 	uint64_t *status)
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.h b/drivers/raw/ifpga/rte_pmd_ifpga.h
index 023a011116..633f6e9d1b 100644
--- a/drivers/raw/ifpga/rte_pmd_ifpga.h
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.h
@@ -18,6 +18,55 @@
 extern "C" {
 #endif
 
+#include <stdint.h>
+
+#define IFPGA_MAX_PORT_NUM   4
+
+/**
+ * UUID data structure.
+ */
+typedef struct {
+	uint8_t b[16];
+} rte_pmd_ifpga_uuid;
+
+/**
+ * FME property data structure.
+ */
+typedef struct {
+	uint32_t num_ports;
+	uint32_t boot_page;
+	uint64_t bitstream_id;
+	uint64_t bitstream_metadata;
+	rte_pmd_ifpga_uuid pr_id;
+	uint32_t bmc_version;
+	uint32_t bmc_nios_version;
+} rte_pmd_ifpga_common_prop;
+
+/**
+ * port property data structure.
+ */
+typedef struct {
+	rte_pmd_ifpga_uuid afu_id;
+	uint32_t type;   /* AFU memory access control type */
+} rte_pmd_ifpga_port_prop;
+
+/**
+ * FPGA property data structure.
+ */
+typedef struct {
+	rte_pmd_ifpga_common_prop  common;
+	rte_pmd_ifpga_port_prop    port[IFPGA_MAX_PORT_NUM];
+} rte_pmd_ifpga_prop;
+
+/**
+ * PHY information data structure.
+ */
+typedef struct {
+	uint32_t num_retimers;
+	uint32_t link_speed;
+	uint32_t link_status;
+} rte_pmd_ifpga_phy_info;
+
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
@@ -37,6 +86,67 @@ __rte_experimental
 int
 rte_pmd_ifpga_get_dev_id(const char *pci_addr, uint16_t *dev_id);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Get current RSU status of the specified Intel FPGA device
+ *
+ * @param dev_id
+ *    The raw device ID of specified Intel FPGA device.
+ * @param stat
+ *    The buffer to output RSU status.
+ * @param prog
+ *    The buffer to output RSU progress.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-ENOMEM) if share data is not initialized.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_get_rsu_status(uint16_t dev_id, uint32_t *stat, uint32_t *prog);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Get FPGA property of specified Intel FPGA device
+ *
+ * @param dev_id
+ *    The raw device ID of specified Intel FPGA device.
+ * @param prop
+ *    The data pointer of FPGA property buffer.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-EBUSY) if FPGA is rebooting.
+ *   - (-EIO) if failed to access hardware.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_get_property(uint16_t dev_id, rte_pmd_ifpga_prop *prop);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Get PHY information of specified Intel FPGA device
+ *
+ * @param dev_id
+ *    The raw device ID of specified Intel FPGA device.
+ * @param info
+ *    The data pointer of PHY information buffer.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-EBUSY) if FPGA is rebooting.
+ *   - (-EIO) if failed to access hardware.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_get_phy_info(uint16_t dev_id, rte_pmd_ifpga_phy_info *info);
+
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
diff --git a/drivers/raw/ifpga/version.map b/drivers/raw/ifpga/version.map
index 16584f7fe4..ca6f7f5810 100644
--- a/drivers/raw/ifpga/version.map
+++ b/drivers/raw/ifpga/version.map
@@ -7,6 +7,9 @@ EXPERIMENTAL {
 
 	# added in 21.05
 	rte_pmd_ifpga_get_dev_id;
+	rte_pmd_ifpga_get_rsu_status;
+	rte_pmd_ifpga_get_property;
+	rte_pmd_ifpga_get_phy_info;
 	rte_pmd_ifpga_update_flash;
 	rte_pmd_ifpga_stop_update;
 	rte_pmd_ifpga_reboot_try;
-- 
2.29.2

[dpdk-dev] [PATCH v15 3/4] raw/ifpga: add miscellaneous APIs

[Wei Huang]

Below miscellaneous APIs are used to implement OPAE application.
1. rte_pmd_ifpga_get_pci_bus() get PCI bus ifpga driver registered.
2. rte_pmd_ifpga_partial_reconfigure() do partial reconfiguration.
3. rte_pmd_ifpga_cleanup() free software resources allocated by driver.
4. rte_pmd_ifpga_set_rsu_status() set status of rsu process.

Signed-off-by: Wei Huang <wei.huang@intel.com>
Acked-by: Tianfei Zhang <tianfei.zhang@intel.com>
Acked-by: Rosen Xu <rosen.xu@intel.com>
---
v2: implement APIs in rte_pmd_ifpga.c
---
 drivers/raw/ifpga/ifpga_rawdev.c  | 30 ++++++++++++++
 drivers/raw/ifpga/ifpga_rawdev.h  |  7 +++-
 drivers/raw/ifpga/rte_pmd_ifpga.c | 45 +++++++++++++++++++++
 drivers/raw/ifpga/rte_pmd_ifpga.h | 66 +++++++++++++++++++++++++++++++
 drivers/raw/ifpga/version.map     |  4 ++
 5 files changed, 151 insertions(+), 1 deletion(-)

diff --git a/drivers/raw/ifpga/ifpga_rawdev.c b/drivers/raw/ifpga/ifpga_rawdev.c
index 27129b133e..05d79bfcc2 100644
--- a/drivers/raw/ifpga/ifpga_rawdev.c
+++ b/drivers/raw/ifpga/ifpga_rawdev.c
@@ -1737,3 +1737,33 @@ RTE_PMD_REGISTER_PARAM_STRING(ifpga_rawdev_cfg,
 	"ifpga=<string> "
 	"port=<int> "
 	"afu_bts=<path>");
+
+struct rte_pci_bus *ifpga_get_pci_bus(void)
+{
+	return rte_ifpga_rawdev_pmd.bus;
+}
+
+int ifpga_rawdev_partial_reconfigure(struct rte_rawdev *dev, int port,
+	const char *file)
+{
+	if (!dev) {
+		IFPGA_RAWDEV_PMD_ERR("Input parameter is invalid");
+		return -EINVAL;
+	}
+
+	return rte_fpga_do_pr(dev, port, file);
+}
+
+void ifpga_rawdev_cleanup(void)
+{
+	struct ifpga_rawdev *dev;
+	unsigned int i;
+
+	for (i = 0; i < IFPGA_RAWDEV_NUM; i++) {
+		dev = &ifpga_rawdevices[i];
+		if (dev->rawdev) {
+			rte_rawdev_pmd_release(dev->rawdev);
+			dev->rawdev = NULL;
+		}
+	}
+}
diff --git a/drivers/raw/ifpga/ifpga_rawdev.h b/drivers/raw/ifpga/ifpga_rawdev.h
index 7754beb02b..9bbe9a4278 100644
--- a/drivers/raw/ifpga/ifpga_rawdev.h
+++ b/drivers/raw/ifpga/ifpga_rawdev.h
@@ -43,7 +43,7 @@ enum ifpga_rawdev_device_state {
 static inline struct opae_adapter *
 ifpga_rawdev_get_priv(const struct rte_rawdev *rawdev)
 {
-	return rawdev->dev_private;
+	return (struct opae_adapter *)rawdev->dev_private;
 }
 
 #define IFPGA_RAWDEV_MSIX_IRQ_NUM 7
@@ -76,4 +76,9 @@ int
 ifpga_unregister_msix_irq(enum ifpga_irq_type type,
 		int vec_start, rte_intr_callback_fn handler, void *arg);
 
+struct rte_pci_bus *ifpga_get_pci_bus(void);
+int ifpga_rawdev_partial_reconfigure(struct rte_rawdev *dev, int port,
+	const char *file);
+void ifpga_rawdev_cleanup(void);
+
 #endif /* _IFPGA_RAWDEV_H_ */
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.c b/drivers/raw/ifpga/rte_pmd_ifpga.c
index 8e04e22d5f..6e23a2581a 100644
--- a/drivers/raw/ifpga/rte_pmd_ifpga.c
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.c
@@ -122,6 +122,25 @@ rte_pmd_ifpga_get_rsu_status(uint16_t dev_id, uint32_t *stat, uint32_t *prog)
 	return 0;
 }
 
+int
+rte_pmd_ifpga_set_rsu_status(uint16_t dev_id, uint32_t stat, uint32_t prog)
+{
+	struct opae_adapter *adapter = NULL;
+	opae_share_data *sd = NULL;
+
+	adapter = get_opae_adapter(dev_id);
+	if (!adapter)
+		return -ENODEV;
+
+	sd = get_share_data(adapter);
+	if (!sd)
+		return -ENOMEM;
+
+	sd->rsu_stat = IFPGA_RSU_STATUS(stat, prog);
+
+	return 0;
+}
+
 static int
 ifpga_is_rebooting(struct opae_adapter *adapter)
 {
@@ -382,3 +401,29 @@ rte_pmd_ifpga_reload(uint16_t dev_id, int type, int page)
 
 	return opae_mgr_reload(adapter->mgr, type, page);
 }
+
+const struct rte_pci_bus *
+rte_pmd_ifpga_get_pci_bus(void)
+{
+	return ifpga_get_pci_bus();
+}
+
+int
+rte_pmd_ifpga_partial_reconfigure(uint16_t dev_id, int port, const char *file)
+{
+	struct rte_rawdev *dev = NULL;
+
+	dev = get_rte_rawdev(dev_id);
+	if (!dev) {
+		IFPGA_RAWDEV_PMD_ERR("Device ID %u is invalid.", dev_id);
+		return -EINVAL;
+	}
+
+	return ifpga_rawdev_partial_reconfigure(dev, port, file);
+}
+
+void
+rte_pmd_ifpga_cleanup(void)
+{
+	ifpga_rawdev_cleanup();
+}
diff --git a/drivers/raw/ifpga/rte_pmd_ifpga.h b/drivers/raw/ifpga/rte_pmd_ifpga.h
index 633f6e9d1b..47d66ba655 100644
--- a/drivers/raw/ifpga/rte_pmd_ifpga.h
+++ b/drivers/raw/ifpga/rte_pmd_ifpga.h
@@ -107,6 +107,27 @@ __rte_experimental
 int
 rte_pmd_ifpga_get_rsu_status(uint16_t dev_id, uint32_t *stat, uint32_t *prog);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Set current RSU status of the specified Intel FPGA device
+ *
+ * @param dev_id
+ *    The raw device ID of specified Intel FPGA device.
+ * @param stat
+ *    The RSU status value to set.
+ * @param prog
+ *    The RSU progress value to set.
+ * @return
+ *   - (0) if successful.
+ *   - (-ENODEV) if dev_id is invalid.
+ *   - (-ENOMEM) if share data is not initialized.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_set_rsu_status(uint16_t dev_id, uint32_t stat, uint32_t prog);
+
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
@@ -235,6 +256,51 @@ __rte_experimental
 int
 rte_pmd_ifpga_reload(uint16_t dev_id, int type, int page);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Get PCI bus the Intel FPGA driver register to
+ *
+ * @return
+ *   - (valid pointer) if successful.
+ *   - (NULL) if the Intel FPGA driver is not registered to any PCI bus.
+ */
+__rte_experimental
+const struct rte_pci_bus *
+rte_pmd_ifpga_get_pci_bus(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Perform PR (partial reconfiguration) on specified Intel FPGA device
+ *
+ * @param dev_id
+ *   The raw device ID of specified Intel FPGA device.
+ * @param port
+ *   The port index of the partial reconfiguration area.
+ * @param file
+ *   The GBS (Green BitStream) image file name string.
+ * @return
+ *   - (0) if successful.
+ *   - (-EINVAL) if bad parameter or operation failed.
+ *   - (-ENOMEM) if failed to allocate memory.
+ */
+__rte_experimental
+int
+rte_pmd_ifpga_partial_reconfigure(uint16_t dev_id, int port, const char *file);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Free software resources allocated by Intel FPGA PMD
+ */
+__rte_experimental
+void
+rte_pmd_ifpga_cleanup(void);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/drivers/raw/ifpga/version.map b/drivers/raw/ifpga/version.map
index ca6f7f5810..995c419a9b 100644
--- a/drivers/raw/ifpga/version.map
+++ b/drivers/raw/ifpga/version.map
@@ -8,10 +8,14 @@ EXPERIMENTAL {
 	# added in 21.05
 	rte_pmd_ifpga_get_dev_id;
 	rte_pmd_ifpga_get_rsu_status;
+	rte_pmd_ifpga_set_rsu_status;
 	rte_pmd_ifpga_get_property;
 	rte_pmd_ifpga_get_phy_info;
 	rte_pmd_ifpga_update_flash;
 	rte_pmd_ifpga_stop_update;
 	rte_pmd_ifpga_reboot_try;
 	rte_pmd_ifpga_reload;
+	rte_pmd_ifpga_get_pci_bus;
+	rte_pmd_ifpga_partial_reconfigure;
+	rte_pmd_ifpga_cleanup;
 };
-- 
2.29.2

[dpdk-dev] [PATCH v15 4/4] examples/ifpga: add example for ifpga APIs

[Wei Huang]

This example application shows how to use ifpga APIs.
A set of OPAE APIs are implemented based on these ifpga APIs.
You can test each OPAE API by running corresponding command.
A guide is also added to show how to run the example.

Signed-off-by: Wei Huang <wei.huang@intel.com>
Acked-by: Tianfei Zhang <tianfei.zhang@intel.com>
Acked-by: Rosen Xu <rosen.xu@intel.com>
---
v2: fix coding style issue in commands.c
---
v3: add guide for running example
---
v4: fix compilation issue of ifpga.rst
---
v5: add ifpga.rst into sample_app_ug/index.rst
---
v6: implement OPAE APIs in example instead of in driver
---
v7: fix compilation issue of unused const variable 'kdrv'
---
v8: fix compilation issue of format error
---
 MAINTAINERS                        |    3 +
 doc/api/doxy-api.conf.in           |    1 +
 doc/guides/sample_app_ug/ifpga.rst |  387 +++++++
 doc/guides/sample_app_ug/index.rst |    1 +
 examples/ifpga/Makefile            |   43 +
 examples/ifpga/commands.c          | 1294 ++++++++++++++++++++++
 examples/ifpga/commands.h          |   16 +
 examples/ifpga/main.c              |   38 +
 examples/ifpga/meson.build         |   20 +
 examples/ifpga/opae_api.c          | 1630 ++++++++++++++++++++++++++++
 examples/ifpga/opae_api.h          |  244 +++++
 examples/meson.build               |    2 +-
 12 files changed, 3678 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/sample_app_ug/ifpga.rst
 create mode 100644 examples/ifpga/Makefile
 create mode 100644 examples/ifpga/commands.c
 create mode 100644 examples/ifpga/commands.h
 create mode 100644 examples/ifpga/main.c
 create mode 100644 examples/ifpga/meson.build
 create mode 100644 examples/ifpga/opae_api.c
 create mode 100644 examples/ifpga/opae_api.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 1a12916f56..48c2ea44d4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1236,9 +1236,12 @@ Rawdev Drivers
 Intel FPGA
 M: Rosen Xu <rosen.xu@intel.com>
 M: Tianfei zhang <tianfei.zhang@intel.com>
+M: Wei Huang <wei.huang@intel.com>
 T: git://dpdk.org/next/dpdk-next-net-intel
 F: drivers/raw/ifpga/
 F: doc/guides/rawdevs/ifpga.rst
+F: examples/ifpga/
+F: doc/guides/sample_app_ug/ifpga.rst
 
 IOAT Rawdev
 M: Bruce Richardson <bruce.richardson@intel.com>
diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index 5c883b613b..5eb31508fd 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -23,6 +23,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/drivers/net/softnic \
                           @TOPDIR@/drivers/raw/dpaa2_cmdif \
                           @TOPDIR@/drivers/raw/dpaa2_qdma \
+                          @TOPDIR@/drivers/raw/ifpga \
                           @TOPDIR@/drivers/raw/ioat \
                           @TOPDIR@/lib/librte_eal/include \
                           @TOPDIR@/lib/librte_eal/include/generic \
diff --git a/doc/guides/sample_app_ug/ifpga.rst b/doc/guides/sample_app_ug/ifpga.rst
new file mode 100644
index 0000000000..bc05153418
--- /dev/null
+++ b/doc/guides/sample_app_ug/ifpga.rst
@@ -0,0 +1,387 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020-2021 Intel Corporation.
+
+Intel FPGA Sample Application
+=============================
+
+The Intel FPGA sample application is an example of how to use rte_pmd_ifpga API
+to manage Intel FPGA.
+
+Overview
+--------
+
+The Intel FPGA sample application is a simple application that demonstrates
+the use of the APIs provided by ifpga driver in the DPDK.
+A set of OPAE APIs are implemented in this example with 'opae' prefix. These
+APIs use PCI address ('Domain:Bus:Dev.Func') as an unique ID of FPGA,
+application based on OPAE can manage specified FPGA easily with them.
+OPAE (Open Programmable Acceleration Engine) is a software framework for
+managing and accessing programmable accelerators (FPGAs). For more information
+about OPAE, please refer to https://opae.github.io/2.0.0/index.html .
+This application is a readline-like interface that can be used to manage
+Intel FPGA, in a Linux* application environment.
+
+Compiling the Application
+-------------------------
+
+To compile the sample application see :doc:`compiling`
+
+The application is located in the ``ifpga`` sub-directory.
+
+Running the Application
+-----------------------
+
+To run the application in linux environment, issue the following command:
+
+.. code-block:: console
+
+    $ ./<build_dir>/examples/dpdk-ifpga --proc-type=auto
+
+Refer to the *DPDK Getting Started Guide* for general information on running
+applications and the Environment Abstraction Layer (EAL) options.
+
+Commands Description
+--------------------
+
+The following sections provide some explanation of the commands.
+
+help command
+~~~~~~~~~~~~
+
+The application has on-line help for the commands that are available at runtime.
+
+.. code-block:: console
+
+   opae> help
+    get_api_version                        get OPAE API version
+    get_proc_type                          get DPDK process type
+    get_image_info <FILE>                  get information of image file
+    get_status <BDF>                       get current status & progress of FPGA
+    get_property <BDF> <0|1|2|4|8>         get property of FPGA
+    get_phy_info <BDF>                     get information of PHY
+    get_parent <BDF>                       get parent PCI device of FPGA
+    get_child <BDF>                        get child PCI device of FPGA
+    get_pf1 <BDF>                          get physical function 1 device of FPGA
+    set_log_level <0-4>                    set logging level
+    set_log_file <FILE>                    set logging file
+    set_status <BDF> <0-4> <0-100>         set current status & progress of FPGA
+    enumerate <VID> <DID>                  enumerate specified FPGA
+    bind <BDF> <DRIVER>                    bind FPGA with kernel driver
+    unbind <BDF>                           unbind FPGA from kernel driver
+    probe <BDF>                            probe FPGA with IFPGA driver
+    remove <BDF>                           remove FPGA from IFPGA driver
+    flash <BDF> <FILE>                     update flash of FPGA
+    pr <BDF> <PORT> <FILE>                 partial reconfigure FPGA
+    reboot <BDF> <fpga|bmc> <0-1>          reboot FPGA or MAX10
+    cancel <BDF>                           cancel flash update
+    pci_read <BDF> <0-1024>                read PCI configuration space
+    pci_write <BDF> <0-1024> <NUM>         write PCI configuration space
+    quit                                   exit DPDK application
+    help                                   show commands list
+
+get_api_version command
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Show OPAE API version which is same to the version of DPDK.
+
+.. code-block:: console
+
+   opae> get_api_version
+   21.2.0
+
+set_log_level command
+~~~~~~~~~~~~~~~~~~~~~
+
+Set logging level of OPAE API. Logging level is defined as below.
+0 - CRITICAL
+1 - ERROR
+2 - WARNING
+3 - INFORMATION
+4 - DEBUG
+
+.. code-block:: console
+
+   opae> set_log_level 4
+   OPAE-API: Current log level is DEBUG
+   Successful
+   opae> set_log_level 6
+   OPAE-API: Current log level is DEBUG
+   Failed
+
+set_log_file command
+~~~~~~~~~~~~~~~~~~~~
+
+Set logging file of OPAE API.
+
+.. code-block:: console
+
+   opae> set_log_file /tmp/ifpga.log
+   Successful
+
+get_proc_type command
+~~~~~~~~~~~~~~~~~~~~~
+
+Show the process type of DPDK. If you start multiple instances of the
+application, the process type of the first one is 'Primary', the others
+are 'Secondary'.
+
+.. code-block:: console
+
+   opae> get_proc_type
+   Primary
+
+get_image_info command
+~~~~~~~~~~~~~~~~~~~~~~
+
+Display information of FPGA image file.
+
+.. code-block:: console
+
+   opae> get_image_info /home/wei/a10.bin
+   Type:           FPGA_BBS
+   Action:         UPDATE
+   Total length:   58720256
+   Payload offset: 1024
+   Payload length: 58719232
+   opae> get_image_info /home/wei/data.bin
+   OPAE-ERR: Image '/home/wei/data.bin' can not be recognized
+   Invalid image file
+
+enumerate command
+~~~~~~~~~~~~~~~~~
+
+Display PCI address of FPGA with specified vendor ID and device ID. ID value can
+be set to 0xffff for arbitrary ID.
+
+.. code-block:: console
+
+   opae> enumerate 0x8086 0x0b30
+   0000:24:00.0
+
+get_property command
+~~~~~~~~~~~~~~~~~~~~
+
+Display property information of specified FPGA. Property type is defined as below.
+0 - All properties
+1 - PCI property
+2 - FME property
+4 - port property
+8 - BMC property
+PCI property is always available, other properties can only be displayed after
+ifpga driver is probed to the FPGA.
+
+.. code-block:: console
+
+   opae> get_property 24:00.0 0
+   PCI:
+    PCIe s:b:d.f     : 0000:24:00.0
+    kernel driver    : vfio-pci
+   FME:
+    platform         : Vista Creek
+    DCP version      : DCP 1.2
+    phase            : Beta
+    interface        : 2x2x25G
+    build version    : 0.0.2
+    ports num        : 1
+    boot page        : user
+    pr interface id  : a5d72a3c-c8b0-4939-912c-f715e5dc10ca
+   PORT0:
+    access type      : PF
+    accelerator id   : 8892c23e-2eed-4b44-8bb6-5c88606e07df
+   BMC:
+    MAX10 version    : D.2.0.5
+    NIOS FW version  : D.2.0.12
+
+get_phy_info command
+~~~~~~~~~~~~~~~~~~~~
+
+Display information and status of PHY connects to the specified FPGA.
+
+.. code-block:: console
+
+   opae> get_phy_info 24:00.0
+    retimers num     : 2
+    link speed       : 25G
+    link status      : 00
+
+get_parent command
+~~~~~~~~~~~~~~~~~~
+
+Display PCI address of upstream device connects to the specified FPGA.
+
+.. code-block:: console
+
+   opae> get_parent 24:00.0
+   0000:22:09.0
+
+get_child command
+~~~~~~~~~~~~~~~~~
+
+Display PCI address of downstream device connects to the specified FPGA.
+
+.. code-block:: console
+
+   opae> get_child 24:00.0
+   No child
+   opae> get_child 22:09.0
+   0000:24:00.0
+
+get_pf1 command
+~~~~~~~~~~~~~~~
+
+Display PCI address of PF1 (physical function 1) of specified FPGA.
+
+.. code-block:: console
+
+   opae> get_pf1 24:00.0
+   0000:26:00.0
+   0000:26:00.1
+
+get_status command
+~~~~~~~~~~~~~~~~~~
+
+Display current RSU status of specified FPGA.
+
+.. code-block:: console
+
+   opae> get_status 24:00.0
+   Status:   IDLE
+   Progress: 0%
+
+set_status command
+~~~~~~~~~~~~~~~~~~
+
+Set current RSU status of specified FPGA. This command is mainly used for debug
+purpose. Status value is defined as below.
+0 - IDLE
+1 - PREPARE
+2 - PROGRAM
+3 - COPY
+4 - REBOOT
+
+.. code-block:: console
+
+   opae> set_status 24:00.0 2 35
+   Successful
+   opae> get_status 24:00.0
+   Status:   PROGRAM
+   Progress: 35%
+
+unbind command
+~~~~~~~~~~~~~~
+
+Unbind kernel driver from specified FPGA.
+
+.. code-block:: console
+
+   opae> unbind 24:00.0
+   OPAE-ERR: 0000:24:00.0 is probed, remove it first
+   Failed
+   opae> remove 24:00.0
+   Successful
+   opae> unbind 24:00.0
+   Successful
+
+bind command
+~~~~~~~~~~~~
+
+Bind specified kernel driver to specified FPGA.
+
+.. code-block:: console
+
+   opae> bind 24:00.0 vfio-pci
+   Successful
+
+probe command
+~~~~~~~~~~~~~
+
+Probe specified FPGA with DPDK PMD driver.
+
+.. code-block:: console
+
+   opae> probe 24:00.0
+   Successful
+
+remove command
+~~~~~~~~~~~~~~
+
+Remove specified FPGA from DPDK PMD driver. It's a reverse operation to probe
+command.
+
+.. code-block:: console
+
+   opae> remove 24:00.0
+   Successful
+
+flash command
+~~~~~~~~~~~~~
+
+Update image in flash of specified FPGA.
+
+.. code-block:: console
+
+   opae> flash 24:00.0 /home/wei/a10.bin
+   Successful
+
+pr command
+~~~~~~~~~~
+
+Do partial reconfiguration of specified FPGA.
+
+.. code-block:: console
+
+   opae> pr 24:00.0 0 /home/wei/nlb0.gbs
+   Successful
+
+reboot command
+~~~~~~~~~~~~~~
+
+Reboot specified FPGA. Reboot type and page is defined as below.
+fpga - reboot FPGA only
+bmc - reboot whole card with FPGA
+0 - factory page
+1 - user page
+
+.. code-block:: console
+
+   opae> reboot 24:00.0 fpga 1
+   Successful
+
+cancel command
+~~~~~~~~~~~~~~
+
+Cancel flash programming of specified FPGA.
+
+.. code-block:: console
+
+   opae> cancel 24:00.0
+   Successful
+
+pci_read command
+~~~~~~~~~~~~~~~~
+
+Read PCI configuration space of specified FPGA.
+
+.. code-block:: console
+
+   opae> pci_read 24:00.0 0
+   0x0b308086
+
+pci_write command
+~~~~~~~~~~~~~~~~~
+
+Write PCI configuration space of specified FPGA.
+
+.. code-block:: console
+
+   opae> pci_write 24:00.0 4 0x100406
+   Successful
+
+quit command
+~~~~~~~~~~~~
+
+Exit this sample application.
+
+.. code-block:: console
+
+   opae> quit
diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
index e8db83d3a7..beb94ff3e7 100644
--- a/doc/guides/sample_app_ug/index.rst
+++ b/doc/guides/sample_app_ug/index.rst
@@ -23,6 +23,7 @@ Sample Applications User Guides
     kernel_nic_interface
     keep_alive
     ioat
+    ifpga
     l2_forward_crypto
     l2_forward_job_stats
     l2_forward_real_virtual
diff --git a/examples/ifpga/Makefile b/examples/ifpga/Makefile
new file mode 100644
index 0000000000..7d76479a36
--- /dev/null
+++ b/examples/ifpga/Makefile
@@ -0,0 +1,43 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020-2021 Intel Corporation
+
+# binary name
+APP = ifpga
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c opae_api.c
+
+# Build using pkg-config variables if possible
+ifneq ($(shell pkg-config --exists libdpdk && echo 0),0)
+$(error "no installation of DPDK found")
+endif
+
+all: shared
+.PHONY: shared static
+shared: build/$(APP)-shared
+	ln -sf $(APP)-shared build/$(APP)
+static: build/$(APP)-static
+	ln -sf $(APP)-static build/$(APP)
+
+PKGCONF ?= pkg-config
+
+PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk) -lrte_bus_pci -lrte_raw_ifpga
+LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
+
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
+
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
+
+build:
+	@mkdir -p $@
+
+.PHONY: clean
+clean:
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	test -d build && rmdir -p build || true
diff --git a/examples/ifpga/commands.c b/examples/ifpga/commands.c
new file mode 100644
index 0000000000..4209e23de6
--- /dev/null
+++ b/examples/ifpga/commands.c
@@ -0,0 +1,1294 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020-2021 Intel Corporation.
+ * All rights reserved.
+ */
+
+#include <stdio.h>
+#include <stdint.h>
+#include <string.h>
+#include <stdlib.h>
+#include <stdarg.h>
+#include <errno.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+
+#include "commands.h"
+
+static int parse_pciaddr(const char *bdf, opae_pci_device *id)
+{
+	size_t len = 0;
+	unsigned int domain = 0;
+	unsigned int bus = 0;
+	unsigned int devid = 0;
+	unsigned int function = 0;
+
+	if (!bdf || !id)
+		return -EINVAL;
+
+	len = strlen(bdf);
+	if ((len < 5) || (len > 12))
+		return -EINVAL;
+
+	len = sscanf(bdf, "%x:%x:%x.%d", &domain, &bus, &devid, &function);
+	if (len == 4) {
+		snprintf(id->bdf, sizeof(id->bdf), "%04x:%02x:%02x.%d",
+			domain, bus, devid, function);
+	} else {
+		len = sscanf(bdf, "%x:%x.%d", &bus, &devid, &function);
+		if (len == 3) {
+			snprintf(id->bdf, sizeof(id->bdf), "%04x:%02x:%02x.%d",
+				0, bus, devid, function);
+		} else {
+			return -EINVAL;
+		}
+	}
+	return 0;
+}
+
+static void uuid_to_str(opae_uuid *id, uuid_str *str)
+{
+	uint8_t *b = NULL;
+	char *p = NULL;
+	int i, j;
+
+	if (!id || !str)
+		return;
+
+	b = &id->b[15];
+	p = str->s;
+	for (i = 0; i < 4; i++, b--, p += 2)
+		sprintf(p, "%02x", *b);
+	sprintf(p++, "-");
+	for (i = 0; i < 3; i++) {
+		for (j = 0; j < 2; j++, b--, p += 2)
+			sprintf(p, "%02x", *b);
+		sprintf(p++, "-");
+	}
+	for (i = 0; i < 6; i++, b--, p += 2)
+		sprintf(p, "%02x", *b);
+}
+
+/* *** GET API VERSION *** */
+struct cmd_version_result {
+	cmdline_fixed_string_t cmd;
+};
+
+static void cmd_version_parsed(__rte_unused void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	opae_api_version ver;
+	opae_get_api_version(&ver);
+	cmdline_printf(cl, "%d.%d.%d\n", ver.major, ver.minor, ver.micro);
+}
+
+cmdline_parse_token_string_t cmd_version_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_version_result, cmd, "get_api_version");
+
+cmdline_parse_inst_t cmd_get_api_version = {
+	.f = cmd_version_parsed,
+	.data = NULL,
+	.help_str = "get OPAE API version",
+	.tokens = {
+		(void *)&cmd_version_cmd,
+		NULL,
+	},
+};
+
+/* *** GET PROC TYPE *** */
+struct cmd_proc_type_result {
+	cmdline_fixed_string_t cmd;
+};
+
+static void cmd_proc_type_parsed(__rte_unused void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	int type = opae_get_proc_type();
+
+	if (type == 0)
+		cmdline_printf(cl, "Primary\n");
+	else if (type == 1)
+		cmdline_printf(cl, "Secondary\n");
+	else
+		cmdline_printf(cl, "Unknown\n");
+}
+
+cmdline_parse_token_string_t cmd_proc_type_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_proc_type_result, cmd, "get_proc_type");
+
+cmdline_parse_inst_t cmd_get_proc_type = {
+	.f = cmd_proc_type_parsed,
+	.data = NULL,
+	.help_str = "get DPDK process type",
+	.tokens = {
+		(void *)&cmd_proc_type_cmd,
+		NULL,
+	},
+};
+
+/* *** GET IMAGE INFO *** */
+struct cmd_image_info_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t path;
+};
+
+static void cmd_image_info_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_image_info_result *res = parsed_result;
+	opae_img_info info;
+
+	if (opae_get_image_info(res->path, &info) == 0) {
+		cmdline_printf(cl, "%-16s", "Type:");
+		if (info.type == OPAE_IMG_TYPE_BBS)
+			cmdline_printf(cl, "FPGA_BBS\n");
+		else if (info.type == OPAE_IMG_TYPE_BMC)
+			cmdline_printf(cl, "BMC\n");
+		else if (info.type == OPAE_IMG_TYPE_GBS)
+			cmdline_printf(cl, "FGPA_GBS\n");
+		else
+			cmdline_printf(cl, "Unknown\n");
+		cmdline_printf(cl, "%-16s", "Action:");
+		if (info.subtype == OPAE_IMG_SUBTYPE_UPDATE)
+			cmdline_printf(cl, "UPDATE\n");
+		else if (info.subtype == OPAE_IMG_SUBTYPE_CANCELLATION)
+			cmdline_printf(cl, "CANCELLATION\n");
+		else if (info.subtype == OPAE_IMG_SUBTYPE_ROOT_KEY_HASH_256)
+			cmdline_printf(cl, "ROOT_HASH_256\n");
+		else if (info.subtype == OPAE_IMG_SUBTYPE_ROOT_KEY_HASH_384)
+			cmdline_printf(cl, "ROOT_HASH_384\n");
+		else
+			cmdline_printf(cl, "Unknown\n");
+		cmdline_printf(cl, "%-16s%u\n", "Total length:",
+			info.total_len);
+		cmdline_printf(cl, "%-16s%u\n", "Payload offset:",
+			info.payload_offset);
+		cmdline_printf(cl, "%-16s%u\n", "Payload length:",
+			info.payload_len);
+	} else {
+		cmdline_printf(cl, "Invalid image file\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_image_info_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_image_info_result, cmd,
+		"get_image_info");
+cmdline_parse_token_string_t cmd_image_info_path =
+	TOKEN_STRING_INITIALIZER(struct cmd_image_info_result, path, NULL);
+
+cmdline_parse_inst_t cmd_get_image_info = {
+	.f = cmd_image_info_parsed,
+	.data = NULL,
+	.help_str = "get information of image file",
+	.tokens = {
+		(void *)&cmd_image_info_cmd,
+		(void *)&cmd_image_info_path,
+		NULL,
+	},
+};
+
+/* *** GET STATUS *** */
+struct cmd_get_status_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_get_status_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_get_status_result *res = parsed_result;
+	opae_pci_device id;
+	uint32_t stat, prog;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_load_rsu_status(&id, &stat, &prog) == 0) {
+		cmdline_printf(cl, "%-10s", "Status:");
+		if (stat == 0)
+			cmdline_printf(cl, "IDLE\n");
+		else if (stat == 1)
+			cmdline_printf(cl, "PREPARE\n");
+		else if (stat == 2)
+			cmdline_printf(cl, "PROGRAM\n");
+		else if (stat == 3)
+			cmdline_printf(cl, "COPY\n");
+		else if (stat == 4)
+			cmdline_printf(cl, "REBOOT\n");
+		else
+			cmdline_printf(cl, "unknown\n");
+		cmdline_printf(cl, "%-10s%u%%\n", "Progress:", prog);
+	} else {
+		cmdline_printf(cl, "Failed\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_get_status_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_get_status_result, cmd, "get_status");
+cmdline_parse_token_string_t cmd_get_status_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_get_status_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_get_status = {
+	.f = cmd_get_status_parsed,
+	.data = NULL,
+	.help_str = "get current status & progress of FPGA",
+	.tokens = {
+		(void *)&cmd_get_status_cmd,
+		(void *)&cmd_get_status_bdf,
+		NULL,
+	},
+};
+
+/* *** GET PROPERTY *** */
+struct cmd_property_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	int32_t type;
+};
+
+static void cmd_property_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_property_result *res = parsed_result;
+	opae_pci_device id;
+	opae_fpga_property prop;
+	uuid_str str;
+	uint32_t port = 0;
+
+	switch (res->type) {
+	case 0:
+	case 1:
+	case 2:
+	case 4:
+	case 8:
+		break;
+	default:
+		cmdline_printf(cl, "%d is invalid type of property\n",
+			res->type);
+		return;
+	}
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_get_property(&id, &prop, res->type) == 0) {
+		if ((res->type == 0) || (res->type == 1)) {
+			cmdline_printf(cl, "%s:\n", "PCI");
+			cmdline_printf(cl, " %-16s : %s\n",
+				"PCIe s:b:d.f", prop.pci.pci_addr);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"kernel driver", prop.pci.drv_name);
+		}
+		if ((res->type == 0) || (res->type == 2)) {
+			cmdline_printf(cl, "%s:\n", "FME");
+			cmdline_printf(cl, " %-16s : %s\n",
+				"platform", prop.fme.platform_name);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"DCP version", prop.fme.dcp_version);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"phase", prop.fme.release_name);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"interface", prop.fme.interface_type);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"build version", prop.fme.build_version);
+			cmdline_printf(cl, " %-16s : %u\n",
+				"ports num", prop.fme.num_ports);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"boot page", prop.fme.boot_page ? "user" : "factory");
+			uuid_to_str(&prop.fme.pr_id, &str);
+			cmdline_printf(cl, " %-16s : %s\n", "pr interface id",
+				str.s);
+		}
+		if ((res->type == 0) || (res->type == 4)) {
+			for (port = 0; port < prop.fme.num_ports; port++) {
+				cmdline_printf(cl, "%s%d:\n", "PORT", port);
+				cmdline_printf(cl, " %-16s : %s\n",
+					"access type",
+					prop.port[port].type ? "VF" : "PF");
+				uuid_to_str(&prop.port[port].afu_id, &str);
+				cmdline_printf(cl, " %-16s : %s\n",
+					"accelerator id", str.s);
+			}
+		}
+		if ((res->type == 0) || (res->type == 8)) {
+			cmdline_printf(cl, "%s:\n", "BMC");
+			cmdline_printf(cl, " %-16s : %s\n",
+				"MAX10 version", prop.bmc.bmc_version);
+			cmdline_printf(cl, " %-16s : %s\n",
+				"NIOS FW version", prop.bmc.fw_version);
+		}
+	} else {
+		cmdline_printf(cl, "Failed\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_property_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_property_result, cmd, "get_property");
+cmdline_parse_token_string_t cmd_property_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_property_result, bdf, NULL);
+cmdline_parse_token_num_t cmd_property_type =
+	TOKEN_NUM_INITIALIZER(struct cmd_property_result, type, RTE_INT32);
+
+cmdline_parse_inst_t cmd_get_property = {
+	.f = cmd_property_parsed,
+	.data = NULL,
+	.help_str = "get property of FPGA",
+	.tokens = {
+		(void *)&cmd_property_cmd,
+		(void *)&cmd_property_bdf,
+		(void *)&cmd_property_type,
+		NULL,
+	},
+};
+
+/* *** GET PHY INFO *** */
+struct cmd_phy_info_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_phy_info_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_phy_info_result *res = parsed_result;
+	opae_pci_device id;
+	opae_phy_info info;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_get_phy_info(&id, &info) == 0) {
+		cmdline_printf(cl, " %-16s : %u\n",
+			"retimers num", info.num_retimers);
+		cmdline_printf(cl, " %-16s : %uG\n",
+			"link speed", info.link_speed);
+		cmdline_printf(cl, " %-16s : %02xh\n",
+			"link status", info.link_status);
+	} else {
+		cmdline_printf(cl, "Failed\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_phy_info_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_phy_info_result, cmd, "get_phy_info");
+cmdline_parse_token_string_t cmd_phy_info_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_phy_info_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_phy_info = {
+	.f = cmd_phy_info_parsed,
+	.data = NULL,
+	.help_str = "get information of PHY",
+	.tokens = {
+		(void *)&cmd_phy_info_cmd,
+		(void *)&cmd_phy_info_bdf,
+		NULL,
+	},
+};
+
+/* *** GET PARENT *** */
+struct cmd_parent_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_parent_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_parent_result *res = parsed_result;
+	opae_pci_device id;
+	opae_pci_device parent;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_get_parent(&id, &parent) > 0)
+		cmdline_printf(cl, "%s\n", parent.bdf);
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_parent_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_parent_result, cmd, "get_parent");
+cmdline_parse_token_string_t cmd_parent_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_parent_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_get_parent = {
+	.f = cmd_parent_parsed,
+	.data = NULL,
+	.help_str = "get parent PCI device of FPGA",
+	.tokens = {
+		(void *)&cmd_parent_cmd,
+		(void *)&cmd_parent_bdf,
+		NULL,
+	},
+};
+
+/* *** GET CHILD *** */
+struct cmd_child_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_child_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_child_result *res = parsed_result;
+	opae_pci_device id;
+	pcidev_id child;
+	int i, count = 0;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	count = opae_get_child(&id, NULL, 0);
+	if (count > 0) {
+		child = (pcidev_id)malloc(sizeof(opae_pci_device) * count);
+		if (child) {
+			opae_get_child(&id, child, count);
+			for (i = 0; i < count; i++)
+				cmdline_printf(cl, "%s\n", child[i].bdf);
+			free(child);
+		} else {
+			cmdline_printf(cl, "No memory\n");
+		}
+	} else if (count == 0) {
+		cmdline_printf(cl, "No child\n");
+	} else {
+		cmdline_printf(cl, "Failed\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_child_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_child_result, cmd, "get_child");
+cmdline_parse_token_string_t cmd_child_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_child_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_get_child = {
+	.f = cmd_child_parsed,
+	.data = NULL,
+	.help_str = "get child PCI device of FPGA",
+	.tokens = {
+		(void *)&cmd_child_cmd,
+		(void *)&cmd_child_bdf,
+		NULL,
+	},
+};
+
+/* *** GET PF1 *** */
+struct cmd_pf1_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_pf1_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_pf1_result *res = parsed_result;
+	opae_pci_device id;
+	pcidev_id peer;
+	int i, count = 0;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	count = opae_get_pf1(&id, NULL, 0);
+	if (count > 0) {
+		peer = (pcidev_id)malloc(sizeof(opae_pci_device) * count);
+		if (peer) {
+			opae_get_pf1(&id, peer, count);
+			for (i = 0; i < count; i++)
+				cmdline_printf(cl, "%s\n", peer[i].bdf);
+			free(peer);
+		} else {
+			cmdline_printf(cl, "No memory\n");
+		}
+	} else if (count == 0) {
+		cmdline_printf(cl, "No PF1\n");
+	} else {
+		cmdline_printf(cl, "Failed\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_pf1_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_pf1_result, cmd, "get_pf1");
+cmdline_parse_token_string_t cmd_pf1_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_pf1_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_get_pf1 = {
+	.f = cmd_pf1_parsed,
+	.data = NULL,
+	.help_str = "get physical function 1 device of FPGA",
+	.tokens = {
+		(void *)&cmd_pf1_cmd,
+		(void *)&cmd_pf1_bdf,
+		NULL,
+	},
+};
+
+/* *** SET LOG LEVEL *** */
+struct cmd_log_level_result {
+	cmdline_fixed_string_t cmd;
+	int32_t level;
+};
+
+static void cmd_log_level_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_log_level_result *res = parsed_result;
+	if (opae_set_log_level(res->level) == res->level)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_log_level_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_log_level_result, cmd, "set_log_level");
+cmdline_parse_token_num_t cmd_log_level_level =
+	TOKEN_NUM_INITIALIZER(struct cmd_log_level_result, level, RTE_INT32);
+
+cmdline_parse_inst_t cmd_set_log_level = {
+	.f = cmd_log_level_parsed,
+	.data = NULL,
+	.help_str = "set logging level",
+	.tokens = {
+		(void *)&cmd_log_level_cmd,
+		(void *)&cmd_log_level_level,
+		NULL,
+	},
+};
+
+/* *** SET LOG FILE *** */
+struct cmd_log_file_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t path;
+};
+
+static void cmd_log_file_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_log_file_result *res = parsed_result;
+	if (opae_set_log_file(res->path, 1) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_log_file_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_log_file_result, cmd, "set_log_file");
+cmdline_parse_token_string_t cmd_log_file_path =
+	TOKEN_STRING_INITIALIZER(struct cmd_log_file_result, path, NULL);
+
+cmdline_parse_inst_t cmd_set_log_file = {
+	.f = cmd_log_file_parsed,
+	.data = NULL,
+	.help_str = "set logging file",
+	.tokens = {
+		(void *)&cmd_log_file_cmd,
+		(void *)&cmd_log_file_path,
+		NULL,
+	},
+};
+
+/* *** SET STATUS *** */
+struct cmd_set_status_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	uint32_t stat;
+	uint32_t prog;
+};
+
+static void cmd_set_status_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_set_status_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+	if ((res->stat > 4) || (res->prog > 100)) {
+		cmdline_printf(cl, "%u,%u is invalid status\n", res->stat,
+			res->prog);
+		return;
+	}
+
+	if (opae_store_rsu_status(&id, res->stat, res->prog) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_set_status_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_set_status_result, cmd, "set_status");
+cmdline_parse_token_string_t cmd_set_status_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_set_status_result, bdf, NULL);
+cmdline_parse_token_num_t cmd_set_status_stat =
+	TOKEN_NUM_INITIALIZER(struct cmd_set_status_result, stat, RTE_UINT32);
+cmdline_parse_token_num_t cmd_set_status_prog =
+	TOKEN_NUM_INITIALIZER(struct cmd_set_status_result, prog, RTE_UINT32);
+
+cmdline_parse_inst_t cmd_set_status = {
+	.f = cmd_set_status_parsed,
+	.data = NULL,
+	.help_str = "set current status & progress of FPGA",
+	.tokens = {
+		(void *)&cmd_set_status_cmd,
+		(void *)&cmd_set_status_bdf,
+		(void *)&cmd_set_status_stat,
+		(void *)&cmd_set_status_prog,
+		NULL,
+	},
+};
+
+/* *** ENUMERATE *** */
+struct cmd_enumerate_result {
+	cmdline_fixed_string_t cmd;
+	uint32_t vid;
+	uint32_t did;
+};
+
+static void cmd_enumerate_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_enumerate_result *res = parsed_result;
+	opae_pci_id filter;
+	opae_pci_device *id;
+	int i, count = 0;
+
+	filter.vendor_id = res->vid;
+	filter.device_id = res->did;
+	filter.class_id = BIT_SET_32;
+	filter.subsystem_vendor_id = BIT_SET_16;
+	filter.subsystem_device_id = BIT_SET_16;
+
+	count = opae_enumerate(&filter, NULL, 0);
+	if (count > 0) {
+		id = (opae_pci_device *)malloc(sizeof(opae_pci_device) * count);
+		if (id) {
+			opae_enumerate(&filter, id, count);
+			for (i = 0; i < count; i++)
+				cmdline_printf(cl, "%s\n", id[i].bdf);
+			free(id);
+		} else {
+			cmdline_printf(cl, "No memory\n");
+		}
+	} else if (count == 0) {
+		cmdline_printf(cl, "Not found\n");
+	} else {
+		cmdline_printf(cl, "Failed\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_enumerate_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_enumerate_result, cmd, "enumerate");
+cmdline_parse_token_num_t cmd_enumerate_vid =
+	TOKEN_NUM_INITIALIZER(struct cmd_enumerate_result, vid, RTE_UINT32);
+cmdline_parse_token_num_t cmd_enumerate_did =
+	TOKEN_NUM_INITIALIZER(struct cmd_enumerate_result, did, RTE_UINT32);
+
+cmdline_parse_inst_t cmd_enumerate = {
+	.f = cmd_enumerate_parsed,
+	.data = NULL,
+	.help_str = "enumerate specified FPGA",
+	.tokens = {
+		(void *)&cmd_enumerate_cmd,
+		(void *)&cmd_enumerate_vid,
+		(void *)&cmd_enumerate_did,
+		NULL,
+	},
+};
+
+/* *** BIND *** */
+struct cmd_bind_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	cmdline_fixed_string_t drv;
+};
+
+static void cmd_bind_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_bind_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_bind_driver(&id, res->drv) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_bind_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_bind_result, cmd, "bind");
+cmdline_parse_token_string_t cmd_bind_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_bind_result, bdf, NULL);
+cmdline_parse_token_string_t cmd_bind_drv =
+	TOKEN_STRING_INITIALIZER(struct cmd_bind_result, drv, NULL);
+
+cmdline_parse_inst_t cmd_bind = {
+	.f = cmd_bind_parsed,
+	.data = NULL,
+	.help_str = "bind FPGA with kernel driver",
+	.tokens = {
+		(void *)&cmd_bind_cmd,
+		(void *)&cmd_bind_bdf,
+		(void *)&cmd_bind_drv,
+		NULL,
+	},
+};
+
+/* *** UNBIND *** */
+struct cmd_unbind_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_unbind_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_unbind_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_unbind_driver(&id) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_unbind_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_bind_result, cmd, "unbind");
+cmdline_parse_token_string_t cmd_unbind_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_bind_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_unbind = {
+	.f = cmd_unbind_parsed,
+	.data = NULL,
+	.help_str = "unbind FPGA from kernel driver",
+	.tokens = {
+		(void *)&cmd_unbind_cmd,
+		(void *)&cmd_unbind_bdf,
+		NULL,
+	},
+};
+
+/* *** PROBE *** */
+struct cmd_probe_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_probe_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_probe_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_probe_device(&id) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_probe_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_probe_result, cmd, "probe");
+cmdline_parse_token_string_t cmd_probe_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_probe_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_probe = {
+	.f = cmd_probe_parsed,
+	.data = NULL,
+	.help_str = "probe FPGA with IFPGA driver",
+	.tokens = {
+		(void *)&cmd_probe_cmd,
+		(void *)&cmd_probe_bdf,
+		NULL,
+	},
+};
+
+/* *** REMOVE *** */
+struct cmd_remove_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_remove_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_remove_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_remove_device(&id) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_remove_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_remove_result, cmd, "remove");
+cmdline_parse_token_string_t cmd_remove_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_remove_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_remove = {
+	.f = cmd_remove_parsed,
+	.data = NULL,
+	.help_str = "remove FPGA from IFPGA driver",
+	.tokens = {
+		(void *)&cmd_remove_cmd,
+		(void *)&cmd_remove_bdf,
+		NULL,
+	},
+};
+
+/* *** FLASH *** */
+struct cmd_flash_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	cmdline_fixed_string_t path;
+};
+
+static void cmd_flash_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_flash_result *res = parsed_result;
+	opae_pci_device id;
+	uint64_t stat = 0;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_update_flash(&id, res->path, &stat))
+		cmdline_printf(cl, "Error: 0x%lx\n", (unsigned long)stat);
+}
+
+cmdline_parse_token_string_t cmd_flash_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_flash_result, cmd, "flash");
+cmdline_parse_token_string_t cmd_flash_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_flash_result, bdf, NULL);
+cmdline_parse_token_string_t cmd_flash_path =
+	TOKEN_STRING_INITIALIZER(struct cmd_flash_result, path, NULL);
+
+cmdline_parse_inst_t cmd_flash = {
+	.f = cmd_flash_parsed,
+	.data = NULL,
+	.help_str = "update flash of FPGA",
+	.tokens = {
+		(void *)&cmd_flash_cmd,
+		(void *)&cmd_flash_bdf,
+		(void *)&cmd_flash_path,
+		NULL,
+	},
+};
+
+/* *** PR *** */
+struct cmd_pr_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	int32_t port;
+	cmdline_fixed_string_t path;
+};
+
+static void cmd_pr_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_pr_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_partial_reconfigure(&id, res->port, res->path) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_pr_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_pr_result, cmd, "pr");
+cmdline_parse_token_string_t cmd_pr_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_pr_result, bdf, NULL);
+cmdline_parse_token_num_t cmd_pr_port =
+	TOKEN_NUM_INITIALIZER(struct cmd_pr_result, port, RTE_INT32);
+cmdline_parse_token_string_t cmd_pr_path =
+	TOKEN_STRING_INITIALIZER(struct cmd_pr_result, path, NULL);
+
+cmdline_parse_inst_t cmd_pr = {
+	.f = cmd_pr_parsed,
+	.data = NULL,
+	.help_str = "partial reconfigure FPGA",
+	.tokens = {
+		(void *)&cmd_pr_cmd,
+		(void *)&cmd_pr_bdf,
+		(void *)&cmd_pr_port,
+		(void *)&cmd_pr_path,
+		NULL,
+	},
+};
+
+/* *** REBOOT *** */
+struct cmd_reboot_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	cmdline_fixed_string_t type;
+	int32_t page;
+};
+
+static void cmd_reboot_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_reboot_result *res = parsed_result;
+	opae_pci_device id;
+	int type = 0;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (!strcmp(res->type, "fpga")) {
+		type = 0;
+	} else if (!strcmp(res->type, "bmc")) {
+		type = 1;
+	} else {
+		cmdline_printf(cl, "%s is invalid reboot type\n", res->type);
+		return;
+	}
+
+	if (opae_reboot_device(&id, type, res->page) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_reboot_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_reboot_result, cmd, "reboot");
+cmdline_parse_token_string_t cmd_reboot_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_reboot_result, bdf, NULL);
+cmdline_parse_token_string_t cmd_reboot_type =
+	TOKEN_STRING_INITIALIZER(struct cmd_reboot_result, type, NULL);
+cmdline_parse_token_num_t cmd_reboot_page =
+	TOKEN_NUM_INITIALIZER(struct cmd_reboot_result, page, RTE_INT32);
+
+cmdline_parse_inst_t cmd_reboot = {
+	.f = cmd_reboot_parsed,
+	.data = NULL,
+	.help_str = "reboot FPGA or MAX10",
+	.tokens = {
+		(void *)&cmd_reboot_cmd,
+		(void *)&cmd_reboot_bdf,
+		(void *)&cmd_reboot_type,
+		(void *)&cmd_reboot_page,
+		NULL,
+	},
+};
+
+/* *** CANCEL *** */
+struct cmd_cancel_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_cancel_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_cancel_result *res = parsed_result;
+	opae_pci_device id;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (opae_cancel_flash_update(&id, 0) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_cancel_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_cancel_result, cmd, "cancel");
+cmdline_parse_token_string_t cmd_cancel_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_cancel_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_cancel = {
+	.f = cmd_cancel_parsed,
+	.data = NULL,
+	.help_str = "cancel flash update",
+	.tokens = {
+		(void *)&cmd_cancel_cmd,
+		(void *)&cmd_cancel_bdf,
+		NULL,
+	},
+};
+
+/* *** PCI READ *** */
+struct cmd_pci_read_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	uint32_t offset;
+};
+
+static void cmd_pci_read_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_pci_read_result *res = parsed_result;
+	opae_pci_device id;
+	uint32_t offset = 0;
+	uint32_t value = 0;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (res->offset & 0x3) {
+		offset = res->offset & ~3;
+		cmdline_printf(cl, "align offset to 0x%x\n", offset);
+	} else {
+		offset = res->offset;
+	}
+
+	if (opae_read_pci_cfg(&id, offset, &value) == 0)
+		cmdline_printf(cl, "0x%08x\n", value);
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_pci_read_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_pci_read_result, cmd, "pci_read");
+cmdline_parse_token_string_t cmd_pci_read_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_pci_read_result, bdf, NULL);
+cmdline_parse_token_num_t cmd_pci_read_offset =
+	TOKEN_NUM_INITIALIZER(struct cmd_pci_read_result, offset, RTE_UINT32);
+
+cmdline_parse_inst_t cmd_pci_read = {
+	.f = cmd_pci_read_parsed,
+	.data = NULL,
+	.help_str = "read PCI configuration space",
+	.tokens = {
+		(void *)&cmd_pci_read_cmd,
+		(void *)&cmd_pci_read_bdf,
+		(void *)&cmd_pci_read_offset,
+		NULL,
+	},
+};
+
+/* *** PCI WRITE *** */
+struct cmd_pci_write_result {
+	cmdline_fixed_string_t cmd;
+	cmdline_fixed_string_t bdf;
+	uint32_t offset;
+	uint32_t value;
+};
+
+static void cmd_pci_write_parsed(void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	struct cmd_pci_write_result *res = parsed_result;
+	opae_pci_device id;
+	uint32_t offset = 0;
+
+	if (parse_pciaddr(res->bdf, &id) < 0) {
+		cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf);
+		return;
+	}
+
+	if (res->offset & 0x3) {
+		offset = res->offset & ~3;
+		cmdline_printf(cl, "align offset to 0x%x\n", offset);
+	} else {
+		offset = res->offset;
+	}
+
+	if (opae_write_pci_cfg(&id, offset, res->value) == 0)
+		cmdline_printf(cl, "Successful\n");
+	else
+		cmdline_printf(cl, "Failed\n");
+}
+
+cmdline_parse_token_string_t cmd_pci_write_cmd =
+	TOKEN_STRING_INITIALIZER(struct cmd_pci_write_result, cmd, "pci_write");
+cmdline_parse_token_string_t cmd_pci_write_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_pci_write_result, bdf, NULL);
+cmdline_parse_token_num_t cmd_pci_write_offset =
+	TOKEN_NUM_INITIALIZER(struct cmd_pci_write_result, offset, RTE_UINT32);
+cmdline_parse_token_num_t cmd_pci_write_value =
+	TOKEN_NUM_INITIALIZER(struct cmd_pci_write_result, value, RTE_UINT32);
+
+cmdline_parse_inst_t cmd_pci_write = {
+	.f = cmd_pci_write_parsed,
+	.data = NULL,
+	.help_str = "write PCI configuration space",
+	.tokens = {
+		(void *)&cmd_pci_write_cmd,
+		(void *)&cmd_pci_write_bdf,
+		(void *)&cmd_pci_write_offset,
+		(void *)&cmd_pci_write_value,
+		NULL,
+	},
+};
+
+/* *** QUIT *** */
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__rte_unused void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,
+	.data = NULL,
+	.help_str = "exit DPDK application",
+	.tokens = {
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/* *** HELP *** */
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__rte_unused void *parsed_result,
+	struct cmdline *cl, __rte_unused void *data)
+{
+	cmdline_printf(cl,
+		" get_api_version               \t\t"
+			"get OPAE API version\n"
+		" get_proc_type                 \t\t"
+			"get DPDK process type\n"
+		" get_image_info <FILE>         \t\t"
+			"get information of image file\n"
+		" get_status <BDF>              \t\t"
+			"get current status & progress of FPGA\n"
+		" get_property <BDF> <0|1|2|4|8>\t\t"
+			"get property of FPGA\n"
+		" get_phy_info <BDF>            \t\t"
+			"get information of PHY\n"
+		" get_parent <BDF>              \t\t"
+			"get parent PCI device of FPGA\n"
+		" get_child <BDF>               \t\t"
+			"get child PCI device of FPGA\n"
+		" get_pf1 <BDF>                 \t\t"
+			"get physical function 1 device of FPGA\n"
+		" set_log_level <0-4>           \t\t"
+			"set logging level\n"
+		" set_log_file <FILE>           \t\t"
+			"set logging file\n"
+		" set_status <BDF> <0-4> <0-100>\t\t"
+			"set current status & progress of FPGA\n"
+		" enumerate <VID> <DID>         \t\t"
+			"enumerate specified FPGA\n"
+		" bind <BDF> <DRIVER>           \t\t"
+			"bind FPGA with kernel driver\n"
+		" unbind <BDF>                  \t\t"
+			"unbind FPGA from kernel driver\n"
+		" probe <BDF>                   \t\t"
+			"probe FPGA with IFPGA driver\n"
+		" remove <BDF>                  \t\t"
+			"remove FPGA from IFPGA driver\n"
+		" flash <BDF> <FILE>            \t\t"
+			"update flash of FPGA\n"
+		" pr <BDF> <PORT> <FILE>        \t\t"
+			"partial reconfigure FPGA\n"
+		" reboot <BDF> <fpga|bmc> <0-1> \t\t"
+			"reboot FPGA or MAX10\n"
+		" cancel <BDF>                  \t\t"
+			"cancel flash update\n"
+		" pci_read <BDF> <0-1024>       \t\t"
+			"read PCI configuration space\n"
+		" pci_write <BDF> <0-1024> <NUM>\t\t"
+			"write PCI configuration space\n"
+		" quit                          \t\t"
+			"exit DPDK application\n"
+		" help                          \t\t"
+			"show commands list\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,
+	.data = NULL,
+	.help_str = "show commands list",
+	.tokens = {
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/****** CONTEXT (list of commands) */
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_get_image_info,
+	(cmdline_parse_inst_t *)&cmd_get_api_version,
+	(cmdline_parse_inst_t *)&cmd_get_proc_type,
+	(cmdline_parse_inst_t *)&cmd_get_status,
+	(cmdline_parse_inst_t *)&cmd_get_property,
+	(cmdline_parse_inst_t *)&cmd_phy_info,
+	(cmdline_parse_inst_t *)&cmd_get_parent,
+	(cmdline_parse_inst_t *)&cmd_get_child,
+	(cmdline_parse_inst_t *)&cmd_get_pf1,
+	(cmdline_parse_inst_t *)&cmd_set_log_level,
+	(cmdline_parse_inst_t *)&cmd_set_log_file,
+	(cmdline_parse_inst_t *)&cmd_set_status,
+	(cmdline_parse_inst_t *)&cmd_enumerate,
+	(cmdline_parse_inst_t *)&cmd_bind,
+	(cmdline_parse_inst_t *)&cmd_unbind,
+	(cmdline_parse_inst_t *)&cmd_probe,
+	(cmdline_parse_inst_t *)&cmd_remove,
+	(cmdline_parse_inst_t *)&cmd_flash,
+	(cmdline_parse_inst_t *)&cmd_pr,
+	(cmdline_parse_inst_t *)&cmd_reboot,
+	(cmdline_parse_inst_t *)&cmd_cancel,
+	(cmdline_parse_inst_t *)&cmd_pci_read,
+	(cmdline_parse_inst_t *)&cmd_pci_write,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_help,
+	NULL,
+};
diff --git a/examples/ifpga/commands.h b/examples/ifpga/commands.h
new file mode 100644
index 0000000000..06fe9a68b2
--- /dev/null
+++ b/examples/ifpga/commands.h
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020-2021 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+#include "opae_api.h"
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+typedef struct {
+	char s[38];
+} uuid_str;
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/ifpga/main.c b/examples/ifpga/main.c
new file mode 100644
index 0000000000..e9380d581a
--- /dev/null
+++ b/examples/ifpga/main.c
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020-2021 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+
+int main(int argc, char **argv)
+{
+	struct cmdline *cl;
+	int ret;
+
+	ret = opae_init_eal(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+	cl = cmdline_stdin_new(main_ctx, "opae> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	opae_cleanup_eal();
+	cmdline_stdin_exit(cl);
+	return 0;
+}
diff --git a/examples/ifpga/meson.build b/examples/ifpga/meson.build
new file mode 100644
index 0000000000..b88a37a7a5
--- /dev/null
+++ b/examples/ifpga/meson.build
@@ -0,0 +1,20 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020-2021 Intel Corporation
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+# require the raw_ifpga library
+build = dpdk_conf.has('RTE_RAW_IFPGA')
+if not build
+	subdir_done()
+endif
+
+deps += 'raw_ifpga'
+allow_experimental_apis = true
+sources = files(
+	'main.c', 'commands.c', 'opae_api.c'
+)
+cflags += '-fPIC'
diff --git a/examples/ifpga/opae_api.c b/examples/ifpga/opae_api.c
new file mode 100644
index 0000000000..7c27a7a278
--- /dev/null
+++ b/examples/ifpga/opae_api.c
@@ -0,0 +1,1630 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <stdlib.h>
+#include <string.h>
+#include <dirent.h>
+#include <fcntl.h>
+#include <glob.h>
+#include <unistd.h>
+#include <sys/stat.h>
+#include <rte_eal.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+#include <rte_rawdev_pmd.h>
+#include "rte_pmd_ifpga.h"
+#include "opae_api.h"
+
+
+int opae_log_level;
+FILE *opae_log_file;
+
+static opae_api_version api_ver = {21, 5, 0};
+static int eal_inited;
+static uint32_t dev_aer[2] = {0};
+
+static const char * const log_level_name[] = {"CRITICAL", "ERROR",
+	"WARNING", "INFORMATION", "DEBUG"};
+static const char * const proc_type_name[] = {"NON-DPDK", "PRIMARY",
+	"SECONDARY"};
+static const char * const platform_name[] = {"Vista Creek", "Rush Creek",
+	"Darby Creek", "Lightning Creek"};
+static const char * const release_name[] = {"Pre-Alpha", "Alpha", "Beta", "PV"};
+static const char * const interface_type[] = {"8x10G", "4x25G", "2x1x25G",
+	"4x25G+2x25G", "2x2x25G", "2x1x25Gx2FVL", "1x2x25G"};
+
+void opae_get_api_version(opae_api_version *version)
+{
+	if (version)
+		memcpy(version, &api_ver, sizeof(opae_api_version));
+	opae_log_info("API version is %u.%u.%u\n",
+		api_ver.major, api_ver.minor, api_ver.micro);
+}
+
+int opae_set_log_level(int level)
+{
+	if ((level >= OPAE_LOG_API) && (level <= OPAE_LOG_DEBUG))
+		opae_log_level = level;
+	opae_log_api("Current log level is %s\n",
+		log_level_name[opae_log_level]);
+	return opae_log_level;
+}
+
+int opae_set_log_file(char *path, int clean)
+{
+	FILE *f = NULL;
+	time_t start;
+	struct tm *lt = NULL;
+
+	if (path) {
+		if (clean)
+			f = fopen(path, "w+");
+		else
+			f = fopen(path, "a+");
+
+		if (f) {
+			if (opae_log_file) {
+				fclose(opae_log_file);
+				opae_log_file = NULL;
+			}
+			time(&start);
+			lt = localtime(&start);
+			if (lt)
+				fprintf(f, "================%d-%02d-%02d "
+					"%02d:%02d:%02d================\n",
+					1900 + lt->tm_year, 1 + lt->tm_mon,
+					lt->tm_mday,
+					lt->tm_hour, lt->tm_min, lt->tm_sec);
+			fflush(f);
+			opae_log_file = f;
+		} else {
+			opae_log_err("failed to open log file \'%s\'\n", path);
+			return -1;
+		}
+	} else {
+		if (opae_log_file) {
+			fclose(opae_log_file);
+			opae_log_file = NULL;
+		}
+	}
+
+	return 0;
+}
+
+int opae_get_image_info(const char *image, opae_img_info *info)
+{
+	int fd = -1;
+	off_t file_size = 0;
+	opae_img_hdr hdr;
+	ssize_t read_size = 0;
+	int ret = 0;
+
+	if (!image || !info) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	fd = open(image, O_RDONLY);
+	if (fd < 0) {
+		opae_log_err("Failed to open \'%s\' for RD [e:%s]\n",
+			image, strerror(errno));
+		return -EIO;
+	}
+
+	file_size = lseek(fd, 0, SEEK_END);
+	if (file_size < (OPAE_IMG_HDR_SIZE + OPAE_IMG_PL_MIN_SIZE)) {
+		opae_log_err("Size of \'%s\' is less than expected [e:%u]\n",
+			image, OPAE_IMG_HDR_SIZE + OPAE_IMG_PL_MIN_SIZE);
+		ret = -EINVAL;
+		goto close_fd;
+	}
+
+	/* read image header */
+	lseek(fd, 0, SEEK_SET);
+	read_size = read(fd, (void *)&hdr, sizeof(opae_img_hdr));
+	if (read_size < 0) {
+		opae_log_err("Failed to read from \'%s\' [e:%s]\n",
+			image, strerror(errno));
+		ret = -EIO;
+		goto close_fd;
+	}
+	if ((size_t)read_size != sizeof(opae_img_hdr)) {
+		opae_log_err("Read length %zd is not expected [e:%zu]\n",
+			read_size, sizeof(opae_img_hdr));
+		ret = -EIO;
+		goto close_fd;
+	}
+
+	info->total_len = file_size;
+	/* check signed image header */
+	if (hdr.magic == OPAE_IMG_BLK0_MAGIC) {
+		info->type = OPAE_IMG_TYPE(hdr.payload_type);
+		info->subtype = OPAE_IMG_SUBTYPE(hdr.payload_type);
+		info->payload_offset = OPAE_IMG_HDR_SIZE;
+		info->payload_len = hdr.payload_len;
+	} else {
+		opae_log_err("Image \'%s\' can not be recognized\n", image);
+		ret = -EINVAL;
+	}
+close_fd:
+	close(fd);
+	return ret;
+}
+
+static int write_file(char *path, char *buf, int size)
+{
+	int fd = -1;
+	ssize_t n = 0;
+
+	if (!path || !buf || (size <= 0))
+		return -EINVAL;
+
+	fd = open(path, O_WRONLY);
+	if (fd < 0) {
+		opae_log_err("Failed to open \'%s\' for WR [e:%s]\n",
+			path, strerror(errno));
+		return -EIO;
+	}
+	opae_log_dbg("Write \"%s\" to \'%s\'\n", buf, path);
+	n = write(fd, buf, size);
+	if (n < size)  {
+		opae_log_err("Failed to write to \'%s\' [e:%s]\n",
+			path, strerror(errno));
+		close(fd);
+		return -EIO;
+	}
+	close(fd);
+
+	return 0;
+}
+
+static int read_file(char *path, char *buf, int size)
+{
+	int fd = -1;
+	ssize_t n = 0;
+
+	if (!path || !buf || (size <= 0))
+		return -EINVAL;
+
+	fd = open(path, O_RDONLY);
+	if (fd < 0) {
+		opae_log_err("Failed to open \'%s\' for RD [e:%s]\n",
+			path, strerror(errno));
+		return -EIO;
+	}
+	n = read(fd, buf, size);
+	if (n < 0)  {
+		opae_log_err("Failed to read from \'%s\' [e:%s]\n",
+			path, strerror(errno));
+		close(fd);
+		return -EIO;
+	}
+	close(fd);
+
+	if (n > 0)
+		buf[n-1] = 0;
+
+	opae_log_dbg("Read \"%s\" from \'%s\'\n", buf, path);
+	return 0;
+}
+
+int opae_get_proc_type(void)
+{
+	int type = -1;
+
+	if (eal_inited) {
+		if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+			type = 0;
+		else
+			type = 1;
+	}
+	opae_log_info("Current process type is %s\n", proc_type_name[type+1]);
+
+	return type;
+}
+
+static bool check_eal(int inited)
+{
+	if (!eal_inited) {
+		if (inited) {
+			opae_log_warn("EAL is not initialized\n");
+			return 0;
+		}
+	} else {
+		if (!inited) {
+			opae_log_warn("EAL is already initialized\n");
+			return 0;
+		}
+	}
+
+	return 1;
+}
+
+int opae_init_eal(int argc, char **argv)
+{
+	int ret = 0;
+
+	if (!check_eal(0))
+		return ret;
+
+	opae_log_level = OPAE_LOG_ERR;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0) {
+		if (rte_errno == EALREADY) {
+			eal_inited = 1;
+			return 0;
+		}
+		opae_log_err("Cannot initialize EAL [e:%d]\n", ret);
+		if (rte_eal_cleanup())
+			opae_log_warn("EAL could not release all resources\n");
+	} else {
+		eal_inited = 1;
+		opae_log_info("Initialize EAL done\n");
+	}
+
+	return ret;
+}
+
+int opae_cleanup_eal(void)
+{
+	int ret = 0;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	rte_pmd_ifpga_cleanup();
+
+	ret = rte_eal_cleanup();
+	if (ret)
+		opae_log_err("Failed to cleanup EAL [e:%d]\n", ret);
+
+	if (opae_log_file) {
+		fclose(opae_log_file);
+		opae_log_file = NULL;
+	}
+
+	return ret;
+}
+
+static int compare_pci_id(opae_pci_id *id, opae_pci_id *expected_id)
+{
+	if ((expected_id->class_id != BIT_SET_32) &&
+		(expected_id->class_id != id->class_id))
+		return -1;
+	if ((expected_id->vendor_id != BIT_SET_16) &&
+		(expected_id->vendor_id != id->vendor_id))
+		return -1;
+	if ((expected_id->device_id != BIT_SET_16) &&
+		(expected_id->device_id != id->device_id))
+		return -1;
+	if ((expected_id->subsystem_vendor_id != BIT_SET_16) &&
+		(expected_id->subsystem_vendor_id != id->subsystem_vendor_id))
+		return -1;
+	if ((expected_id->subsystem_device_id != BIT_SET_16) &&
+		(expected_id->subsystem_device_id != id->subsystem_device_id))
+		return -1;
+
+	return 0;
+}
+
+static int parse_sysfs_value(char *node, uint32_t *val)
+{
+	char buf[16];
+	char *end = NULL;
+	int ret = 0;
+
+	ret = read_file(node, buf, sizeof(buf));
+	if (ret < 0)
+		return ret;
+
+	*val = (uint32_t)strtoul(buf, &end, 0);
+	return 0;
+}
+
+static int get_pci_id(const char *dev_path, opae_pci_id *id)
+{
+	char path[PATH_MAX] = {0};
+	uint32_t tmp;
+
+	if (!dev_path || !id)
+		return -EINVAL;
+
+	snprintf(path, sizeof(path), "%s/vendor", dev_path);
+	if (parse_sysfs_value(path, &tmp) < 0)
+		return -ENODEV;
+	id->vendor_id = (uint16_t)tmp;
+
+	snprintf(path, sizeof(path), "%s/device", dev_path);
+	if (parse_sysfs_value(path, &tmp) < 0)
+		return -ENODEV;
+	id->device_id = (uint16_t)tmp;
+
+	snprintf(path, sizeof(path), "%s/subsystem_vendor", dev_path);
+	if (parse_sysfs_value(path, &tmp) < 0)
+		return -ENODEV;
+	id->subsystem_vendor_id = (uint16_t)tmp;
+
+	snprintf(path, sizeof(path), "%s/subsystem_device", dev_path);
+	if (parse_sysfs_value(path, &tmp) < 0)
+		return -ENODEV;
+	id->subsystem_device_id = (uint16_t)tmp;
+
+	snprintf(path, sizeof(path), "%s/class", dev_path);
+	if (parse_sysfs_value(path, &tmp) < 0)
+		return -ENODEV;
+	id->class_id = (uint32_t)tmp & RTE_CLASS_ANY_ID;
+
+	return 0;
+}
+
+static int extract_path(char *in, int ridx, char *out, uint32_t size)
+{
+	char src[PATH_MAX] = {0};
+	char *p = NULL;
+	int ret = 0;
+
+	if (!in || (strlen(in) > PATH_MAX) || (ridx < 0) || !out)
+		return -EINVAL;
+
+	strncpy(src, in, sizeof(src));
+	*out = 0;
+
+	while (1) {
+		p = strrchr(src, '/');
+		if (p) {
+			*p++ = 0;
+			if (*p) {
+				if (ridx-- <= 0) {
+					if (size > strlen(p)) {
+						strncpy(out, p, size);
+						ret = strlen(p);
+					}
+					break;
+				}
+			}
+		} else {
+			break;
+		}
+	}
+
+	return ret;
+}
+
+int opae_enumerate(opae_pci_id *filter, pcidev_id list, int size)
+{
+	DIR *dir = NULL;
+	struct dirent *dirent = NULL;
+	char path[PATH_MAX] = {0};
+	opae_pci_id id;
+	int n = 0;
+
+	if (!filter || (size < 0) || (!list && (size > 0))) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	dir = opendir(rte_pci_get_sysfs_path());
+	if (!dir) {
+		opae_log_err("Failed to open \'%s\'\n",
+			rte_pci_get_sysfs_path());
+		return -EINVAL;
+	}
+	while ((dirent = readdir(dir))) {
+		if (!strcmp(dirent->d_name, "."))
+			continue;
+		if (!strcmp(dirent->d_name, ".."))
+			continue;
+
+		snprintf(path, PATH_MAX, "%s/%s", rte_pci_get_sysfs_path(),
+			dirent->d_name);
+		if (get_pci_id(path, &id) < 0)
+			continue;
+		if (compare_pci_id(&id, filter) < 0)
+			continue;
+
+		if (n++ < size) {
+			snprintf(list->bdf, sizeof(list->bdf), "%s",
+				dirent->d_name);
+			list++;
+		}
+	}
+	closedir(dir);
+
+	return n;
+}
+
+static int get_driver(pcidev_id id, char *drv_name, uint32_t size)
+{
+	char path[PATH_MAX] = {0};
+	char link[PATH_MAX] = {0};
+	int ret = 0;
+
+	if (!id || !drv_name) {
+		ret = -EINVAL;
+		goto end;
+	}
+	size--;   /* reserve one byte for the end of string */
+
+	snprintf(path, PATH_MAX, "%s/%s/driver",
+		rte_pci_get_sysfs_path(), id->bdf);
+	ret = readlink(path, link, PATH_MAX);
+	if (ret >= PATH_MAX) {
+		opae_log_err("Link path too long [%d]\n", ret);
+		ret = -ENAMETOOLONG;
+		goto end;
+	}
+	if (ret > 0) {
+		ret = extract_path(link, 0, drv_name, size);
+	} else {
+		*drv_name = 0;
+		opae_log_info("No link path for \'%s\'\n", path);
+		ret = 0;
+	}
+
+end:
+	if (ret < 0)
+		opae_log_err("Failed to get driver of %s\n", id->bdf);
+
+	return ret;
+}
+
+static int get_pci_addr(const char *bdf, opae_pci_addr *addr)
+{
+	struct rte_pci_addr pci_addr;
+	int ret = 0;
+
+	if (!bdf || !addr)
+		return -EINVAL;
+
+	ret = rte_pci_addr_parse(bdf, &pci_addr);
+	if (ret == 0) {
+		addr->domain = pci_addr.domain;
+		addr->bus = pci_addr.bus;
+		addr->devid = pci_addr.devid;
+		addr->function = pci_addr.function;
+	}
+
+	return ret;
+}
+
+static int check_pcidev_id(pcidev_id id)
+{
+	if (!id) {
+		opae_log_err("ID is NULL\n");
+		return -1;
+	}
+
+	if (strnlen(id->bdf, PCI_PRI_STR_SIZE) == PCI_PRI_STR_SIZE) {
+		opae_log_err("PCI address is too long\n");
+		return -2;
+	}
+
+	return 0;
+}
+
+static int get_rawdev_id(pcidev_id id, uint16_t *dev_id, int log)
+{
+	if (check_pcidev_id(id))
+		return -1;
+
+	if (rte_pmd_ifpga_get_dev_id(id->bdf, dev_id)) {
+		if (log)
+			opae_log_warn("%s is not probed by ifpga driver\n",
+				id->bdf);
+		return -1;
+	}
+
+	return 0;
+}
+
+static struct rte_pci_device *get_rte_pcidev(pcidev_id id, int log)
+{
+	const struct rte_pci_bus *pci_bus = NULL;
+	struct rte_pci_device *pci_dev = NULL;
+	struct rte_pci_addr addr;
+
+	if (check_pcidev_id(id))
+		return NULL;
+
+	if (rte_pci_addr_parse(id->bdf, &addr)) {
+		opae_log_err("PCI address %s is invalid\n", id->bdf);
+		return NULL;
+	}
+
+	pci_bus = rte_pmd_ifpga_get_pci_bus();
+	if (pci_bus) {
+		TAILQ_FOREACH(pci_dev, &pci_bus->device_list, next) {
+			if (!rte_pci_addr_cmp(&pci_dev->addr, &addr))
+				return pci_dev;
+		}
+	}
+
+	if (log)
+		opae_log_err("No rte_pci_device for %s\n", id->bdf);
+
+	return NULL;
+}
+
+int opae_load_rsu_status(pcidev_id id, uint32_t *status, uint32_t *progress)
+{
+	uint16_t dev_id = 0;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	return rte_pmd_ifpga_get_rsu_status(dev_id, status, progress);
+}
+
+int opae_store_rsu_status(pcidev_id id, uint32_t status, uint32_t progress)
+{
+	uint16_t dev_id = 0;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	return rte_pmd_ifpga_set_rsu_status(dev_id, status, progress);
+}
+
+static int get_pci_property(pcidev_id id, opae_pci_property *prop)
+{
+	char path[PATH_MAX] = {0};
+	int ret = 0;
+
+	if (!id || !prop)
+		return -EINVAL;
+
+	snprintf(path, PATH_MAX, "%s/%s", rte_pci_get_sysfs_path(), id->bdf);
+
+	ret = get_pci_id(path, &prop->id);
+	if (ret < 0)
+		return ret;
+
+	ret = get_pci_addr(id->bdf, &prop->addr);
+	if (ret < 0)
+		return ret;
+
+	snprintf(prop->pci_addr, OPAE_NAME_SIZE, "%s", id->bdf);
+	get_driver(id, prop->drv_name, sizeof(prop->drv_name));
+
+	return 0;
+}
+
+static int get_fme_property(rte_pmd_ifpga_common_prop *common,
+	opae_fme_property *prop)
+{
+	opae_bitstream_id bbs_id;
+
+	if (!common || !prop)
+		return -EINVAL;
+
+	prop->boot_page = common->boot_page;
+	prop->num_ports = common->num_ports;
+	prop->bitstream_id = common->bitstream_id;
+	prop->bitstream_metadata = common->bitstream_metadata;
+	memcpy(prop->pr_id.b, common->pr_id.b, sizeof(opae_uuid));
+
+	bbs_id.id = prop->bitstream_id;
+	if (bbs_id.major < sizeof(platform_name) / sizeof(char *)) {
+		snprintf(prop->platform_name,
+			sizeof(prop->platform_name), "%s",
+			platform_name[bbs_id.major]);
+	} else {
+		snprintf(prop->platform_name,
+			sizeof(prop->platform_name), "unknown");
+	}
+
+	snprintf(prop->dcp_version, sizeof(prop->dcp_version),
+		"DCP 1.%u", bbs_id.minor);
+
+	if (bbs_id.patch < sizeof(release_name)/sizeof(char *)) {
+		snprintf(prop->release_name, sizeof(prop->release_name),
+			"%s", release_name[bbs_id.patch]);
+	} else {
+		snprintf(prop->release_name, sizeof(prop->release_name),
+			"unknown");
+	}
+
+	if (bbs_id.major == 0) {  /* Vista Creek */
+		if (bbs_id.interface <
+			sizeof(interface_type) / sizeof(char *)) {
+			snprintf(prop->interface_type,
+				sizeof(prop->interface_type), "%s",
+				interface_type[bbs_id.interface]);
+		} else {
+			snprintf(prop->interface_type,
+				sizeof(prop->interface_type), "unknown");
+		}
+	} else {
+		snprintf(prop->interface_type,
+			sizeof(prop->interface_type), "unknown");
+	}
+
+	snprintf(prop->build_version, sizeof(prop->build_version),
+		"%u.%u.%u", bbs_id.build_major, bbs_id.build_minor,
+		bbs_id.build_patch);
+
+	return 0;
+}
+
+static int get_port_property(rte_pmd_ifpga_port_prop *port,
+	opae_port_property *prop)
+{
+	if (!port || !prop)
+		return -EINVAL;
+
+	memcpy(prop->afu_id.b, port->afu_id.b, sizeof(opae_uuid));
+	prop->type = port->type;
+
+	return 0;
+}
+
+static int get_bmc_property(rte_pmd_ifpga_common_prop *common,
+	opae_bmc_property *prop)
+{
+	opae_bmc_version ver;
+
+	if (!common || !prop)
+		return -EINVAL;
+
+	ver.version = common->bmc_version;
+	snprintf(prop->bmc_version, sizeof(prop->bmc_version), "%c.%u.%u.%u",
+		ver.board, ver.major, ver.minor, ver.micro);
+
+	ver.version = common->bmc_nios_version;
+	snprintf(prop->fw_version, sizeof(prop->fw_version), "%c.%u.%u.%u",
+		ver.board, ver.major, ver.minor, ver.micro);
+
+	return 0;
+}
+
+int opae_get_property(pcidev_id id, opae_fpga_property *prop, int type)
+{
+	uint16_t dev_id = 0;
+	uint32_t i = 0;
+	rte_pmd_ifpga_prop fpga_prop;
+	int ret = 0;
+
+	if (!prop) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	if (type == 0)
+		type = OPAE_PROP_ALL;
+
+	memset(prop, 0, sizeof(opae_fpga_property));
+
+	/* PCI properties */
+	if (type & OPAE_PROP_PCI) {
+		if (get_pci_property(id, &prop->pci) < 0) {
+			opae_log_err("Failed to get PCI property\n");
+			return -EAGAIN;
+		}
+	}
+
+	if (type == OPAE_PROP_PCI)
+		return 0;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	ret = rte_pmd_ifpga_get_property(dev_id, &fpga_prop);
+	if (ret) {
+		opae_log_err("Failed to get FPGA property [e:%d]\n", ret);
+		return -EAGAIN;
+	}
+
+	/* FME properties */
+	if (type & (OPAE_PROP_FME | OPAE_PROP_PORT))
+		get_fme_property(&fpga_prop.common, &prop->fme);
+
+	/* PORT properties */
+	if (type & OPAE_PROP_PORT) {
+		for (i = 0; i < prop->fme.num_ports; i++) {
+			prop->port[i].index = i;
+			get_port_property(&fpga_prop.port[i], &prop->port[i]);
+		}
+	}
+
+	/* BMC properties */
+	if (type & OPAE_PROP_BMC)
+		get_bmc_property(&fpga_prop.common, &prop->bmc);
+
+	return 0;
+}
+
+int opae_get_phy_info(pcidev_id id, opae_phy_info *info)
+{
+	uint16_t dev_id = 0;
+	rte_pmd_ifpga_phy_info phy_info;
+	int ret = 0;
+
+	if (!info)
+		return -EINVAL;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	ret = rte_pmd_ifpga_get_phy_info(dev_id, &phy_info);
+	if (ret) {
+		opae_log_err("Failed to get PHY information [e:%d]\n", ret);
+		return -EAGAIN;
+	}
+
+	info->num_retimers = phy_info.num_retimers;
+	info->link_speed = phy_info.link_speed;
+	info->link_status = phy_info.link_status;
+
+	return 0;
+}
+
+static int update_driver(pcidev_id id, char *drv_name)
+{
+	struct rte_pci_device *pci_dev = NULL;
+	char name[OPAE_NAME_SIZE] = {0};
+	int ret = 0;
+
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	if (drv_name) {
+		if (strlen(drv_name) >= OPAE_NAME_SIZE) {
+			opae_log_err("Driver name \'%s\' too long\n",
+				drv_name);
+			return -EINVAL;
+		}
+		strncpy(name, drv_name, sizeof(name));
+	} else {
+		ret = get_driver(id, name, sizeof(name));
+		if (ret < 0)
+			return ret;
+	}
+
+	pci_dev = get_rte_pcidev(id, 0);
+	if (pci_dev) {
+		if (strlen(name) == 0) {
+			pci_dev->kdrv = RTE_PCI_KDRV_NONE;
+		} else {
+			if (!strcmp(name, OPAE_KDRV_VFIO_PCI))
+				pci_dev->kdrv = RTE_PCI_KDRV_VFIO;
+			else if (!strcmp(name, OPAE_KDRV_IGB_UIO))
+				pci_dev->kdrv = RTE_PCI_KDRV_IGB_UIO;
+			else if (!strcmp(name, OPAE_KDRV_UIO_PCI))
+				pci_dev->kdrv = RTE_PCI_KDRV_UIO_GENERIC;
+			else
+				pci_dev->kdrv = RTE_PCI_KDRV_UNKNOWN;
+		}
+	}
+
+	return 0;
+}
+
+int opae_unbind_driver(pcidev_id id)
+{
+	uint16_t dev_id = 0;
+	char path[PATH_MAX] = {0};
+	char drv_name[OPAE_NAME_SIZE] = {0};
+	char null[] = {0};
+	int ret = 0;
+
+	if (!get_rawdev_id(id, &dev_id, 0)) {
+		opae_log_err("%s is probed, remove it first\n", id->bdf);
+		return -EBUSY;
+	}
+
+	ret = get_driver(id, drv_name, sizeof(drv_name));
+	if (ret < 0)
+		return ret;
+
+	if (strlen(drv_name) > 0) {
+		snprintf(path, PATH_MAX, "/sys/bus/pci/drivers/%s/unbind",
+			drv_name);
+		ret = write_file(path, id->bdf, strlen(id->bdf) + 1);
+		if (ret == 0)
+			ret = update_driver(id, null);
+	}
+
+	return ret;
+}
+
+static int check_driver(const char *drv_name)
+{
+	char path[PATH_MAX] = {0};
+	struct stat buf;
+
+	if (!drv_name)
+		return -EINVAL;
+
+	if (strlen(drv_name) > 0) {
+		snprintf(path, PATH_MAX, "/sys/bus/pci/drivers/%s", drv_name);
+		if ((stat(path, &buf) < 0) || ((buf.st_mode & S_IFDIR) == 0)) {
+			opae_log_warn("Driver %s is not installed\n",
+				drv_name);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+int opae_bind_driver(pcidev_id id, char *drv_name)
+{
+	char path[PATH_MAX] = {0};
+	char name[OPAE_NAME_SIZE] = {0};
+	char null[] = {0};
+	int ret = 0;
+
+	ret = check_driver(drv_name);
+	if (ret < 0)
+		return ret;
+
+	ret = get_driver(id, name, sizeof(name));
+	if (ret < 0)
+		return ret;
+
+	if (!strcmp(drv_name, name))   /* driver not change */
+		return 0;
+
+	ret = opae_unbind_driver(id);
+	if (ret < 0)
+		return ret;
+
+	if (strlen(drv_name) > 0) {
+		/* bind driver */
+		snprintf(path, PATH_MAX, "%s/%s/driver_override",
+			rte_pci_get_sysfs_path(), id->bdf);
+		ret = write_file(path, drv_name, strlen(drv_name) + 1);
+		if (ret < 0)
+			goto update_drv;
+
+		snprintf(path, PATH_MAX, "/sys/bus/pci/drivers/%s/bind",
+			drv_name);
+		ret = write_file(path, id->bdf, strlen(id->bdf) + 1);
+		if (ret < 0)
+			goto update_drv;
+
+		snprintf(path, PATH_MAX, "%s/%s/driver_override",
+			rte_pci_get_sysfs_path(), id->bdf);
+		ret = write_file(path, null, 1);
+		if (ret < 0)
+			goto update_drv;
+	}
+
+update_drv:
+	ret = update_driver(id, NULL);
+	if (ret < 0)
+		opae_log_err("Failed to update driver information of %s\n",
+			id->bdf);
+
+	return 0;
+}
+
+int opae_probe_device(pcidev_id id)
+{
+	const struct rte_pci_bus *pci_bus = NULL;
+	struct rte_pci_device *pci_dev = NULL;
+
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	/* make sure device is added in rte_pci_bus devices list */
+	pci_bus = rte_pmd_ifpga_get_pci_bus();
+	if (pci_bus && pci_bus->bus.scan)
+		pci_bus->bus.scan();
+
+	pci_dev = get_rte_pcidev(id, 1);
+	if (!pci_dev)
+		return -ENODEV;
+
+	if (pci_dev->kdrv != RTE_PCI_KDRV_VFIO) {
+		opae_log_err("vfio-pci driver is not bound to %s\n", id->bdf);
+		return -EINVAL;
+	}
+
+	if (!pci_bus || !pci_bus->bus.plug)
+		return -ENODEV;
+
+	return pci_bus->bus.plug(&pci_dev->device);
+}
+
+int opae_remove_device(pcidev_id id)
+{
+	struct rte_pci_device *pci_dev = NULL;
+	struct rte_pci_driver *pci_drv = NULL;
+	int ret = 0;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	pci_dev = get_rte_pcidev(id, 0);
+	if (pci_dev && pci_dev->driver) {
+		pci_drv = pci_dev->driver;
+		ret = pci_drv->remove(pci_dev);
+		if (ret < 0) {
+			opae_log_err("Failed to remove %s [e:%d]\n",
+				id->bdf, ret);
+			return ret;
+		}
+		pci_dev->driver = NULL;
+		pci_dev->device.driver = NULL;
+		if (pci_drv->drv_flags & RTE_PCI_DRV_NEED_MAPPING)
+			rte_pci_unmap_device(pci_dev);
+	}
+
+	return ret;
+}
+
+static int is_pac(pcidev_id id)
+{
+	char path[PATH_MAX] = {0};
+	opae_pci_id pci_id;
+
+	if (check_pcidev_id(id))
+		return 0;
+
+	snprintf(path, PATH_MAX, "%s/%s", rte_pci_get_sysfs_path(), id->bdf);
+	if (get_pci_id(path, &pci_id) < 0)
+		return 0;
+
+	if ((pci_id.vendor_id == 0x8086) && (pci_id.device_id == 0x0b30))
+		return 1;
+
+	return 0;
+}
+
+int opae_get_parent(pcidev_id id, pcidev_id parent)
+{
+	char path[PATH_MAX] = {0};
+	char link[PATH_MAX] = {0};
+	int ret = 0;
+
+	if (!id || !parent) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		ret = -EINVAL;
+		goto end;
+	}
+
+	snprintf(path, PATH_MAX, "%s/%s", rte_pci_get_sysfs_path(), id->bdf);
+	ret = readlink(path, link, PATH_MAX);
+	if (ret >= PATH_MAX) {
+		opae_log_err("Length of link path exceeds %u\n", PATH_MAX);
+		ret = -ENAMETOOLONG;
+		goto end;
+	}
+
+	if (ret > 0) {
+		ret = extract_path(link, 1, parent->bdf, sizeof(parent->bdf));
+		if (!strncmp(parent->bdf, "pci", 3)) {
+			parent->bdf[0] = 0;
+			ret = -ENODEV;
+		}
+	} else {
+		parent->bdf[0] = 0;
+		if (ret == 0)
+			opae_log_err("Length of link path is 0\n");
+		else
+			opae_log_err("No link path for \'%s\'\n", path);
+	}
+end:
+	if (ret <= 0)
+		opae_log_err("%s has no parent\n", id->bdf);
+
+	return ret;
+}
+
+int opae_get_child(pcidev_id id, pcidev_id child, int size)
+{
+	glob_t pglob = {.gl_pathc = 0, .gl_pathv = NULL};
+	char path[PATH_MAX] = {0};
+	int i, count = 0;
+	int len = 0;
+	int ret = 0;
+
+	if (!id || (size < 0)) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	snprintf(path, PATH_MAX, "%s/%s/*:*:*.?", rte_pci_get_sysfs_path(),
+		id->bdf);
+	ret = glob(path, 0, NULL, &pglob);
+	if (ret == 0) {
+		if (child && (size > 0)) {
+			for (i = 0; i < (int)pglob.gl_pathc; i++) {
+				len = extract_path(pglob.gl_pathv[i], 0,
+					child->bdf, sizeof(child->bdf));
+				if (len <= 0) {
+					child->bdf[0] = 0;
+					continue;
+				}
+				if (++count >= size)
+					break;
+				child++;
+			}
+		} else {
+			count = (int)pglob.gl_pathc;
+		}
+		globfree(&pglob);
+	} else {
+		if (pglob.gl_pathv)
+			globfree(&pglob);
+	}
+
+	return count;
+}
+
+int opae_get_pf1(pcidev_id id, pcidev_id peer, int size)
+{
+	opae_pci_device parent;
+	opae_pci_device child[4];
+	int n = 0;
+	int ret = 0;
+
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	if (!is_pac(id)) {
+		opae_log_info("%s has no peer function\n", id->bdf);
+		return -EINVAL;
+	}
+
+	ret = opae_get_parent(id, &parent);
+	if (ret < 0)
+		return -ENODEV;
+	ret = opae_get_parent(&parent, &parent);
+	if (ret < 0)
+		return -ENODEV;
+
+	n = opae_get_child(&parent, child,
+		sizeof(child) / sizeof(opae_pci_device));
+	/* there should have four downstream ports of PCI switch on board */
+	if (n == 4) {
+		n = opae_get_child(&child[3], peer, size);
+	} else {
+		peer->bdf[0] = 0;
+		opae_log_dbg("%s has %d child(s)\n", parent.bdf, n);
+		n = 0;
+	}
+
+	return n;
+}
+
+int opae_update_flash(pcidev_id id, const char *image, uint64_t *status)
+{
+	opae_img_info info;
+	uint16_t dev_id = 0;
+	int ret = 0;
+
+	ret = opae_get_image_info(image, &info);
+	if (ret < 0) {
+		opae_log_err("Failed to get image information [e:%d]\n", ret);
+		return -EINVAL;
+	}
+
+	if ((info.type != OPAE_IMG_TYPE_BBS) &&
+		(info.type != OPAE_IMG_TYPE_BMC)) {
+		opae_log_err("Image is not supported [t:%u]\n", info.type);
+		return -EOPNOTSUPP;
+	}
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	return rte_pmd_ifpga_update_flash(dev_id, image, status);
+}
+
+int opae_cancel_flash_update(pcidev_id id, int force)
+{
+	uint16_t dev_id = 0;
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	return rte_pmd_ifpga_stop_update(dev_id, force);
+}
+
+static int find_pci_ecap(int fd, unsigned int cap)
+{
+	uint32_t header = 0;
+	int ttl = (RTE_PCI_CFG_SPACE_EXP_SIZE - RTE_PCI_CFG_SPACE_SIZE) / 8;
+	int pos = RTE_PCI_CFG_SPACE_SIZE;
+	int ret = 0;
+
+	ret = pread(fd, &header, sizeof(header), pos);
+	if (ret < 0) {
+		opae_log_err("Failed to read from PCI configuration space [e:%s]\n",
+			strerror(errno));
+		return ret;
+	}
+	opae_log_dbg("Read 0x%08x from PCI configuration space 0x%x\n",
+		header, pos);
+
+	if (header == 0) {
+		opae_log_err("Capability is empty\n");
+		return 0;
+	}
+
+	while (ttl-- > 0) {
+		if ((RTE_PCI_EXT_CAP_ID(header) == cap) && (pos != 0))
+			return pos;
+
+		pos = RTE_PCI_EXT_CAP_NEXT(header);
+		if (pos < RTE_PCI_CFG_SPACE_SIZE) {
+			opae_log_err("Position of capability is invalid"
+						 "[e:%d]\n", pos);
+			break;
+		}
+		ret = pread(fd, &header, sizeof(header), pos);
+		if (ret < 0) {
+			opae_log_err("Failed to read from PCI config space [e:%s]\n",
+				strerror(errno));
+			return ret;
+		}
+		opae_log_dbg("Read 0x%08x from PCI configuration space 0x%x\n",
+			header, pos);
+	}
+
+	return 0;
+}
+
+static int set_aer(pcidev_id id, uint32_t v1, uint32_t v2, int record)
+{
+	char path[PATH_MAX] = {0};
+	uint32_t val = 0;
+	int fd = -1;
+	int pos = 0;
+	int ret = 0;
+
+	if (!id)
+		return -EINVAL;
+
+	snprintf(path, PATH_MAX, "%s/%s/config",
+		rte_pci_get_sysfs_path(), id->bdf);
+	fd = open(path, O_RDWR);
+	if (fd < 0) {
+		opae_log_err("Failed to open \'%s\' for RDWR [e:%s]\n",
+			path, strerror(errno));
+		return -EIO;
+	}
+
+	pos = find_pci_ecap(fd, RTE_PCI_EXT_CAP_ID_ERR);
+	if (pos <= 0) {
+		opae_log_warn("AER capability is not present\n");
+		ret = -ENXIO;
+		goto close_fd;
+	}
+
+	if (record) {
+		ret = pread(fd, &val, sizeof(val), pos + 0x08);
+		if (ret < 0) {
+			opae_log_err("Failed to read from PCI config space [e:%s]\n",
+				strerror(errno));
+			goto close_fd;
+		}
+		opae_log_dbg("Read 0x%08x from PCI configuration space 0x%x\n",
+			val, pos + 0x08);
+		dev_aer[0] = val;
+
+		ret = pread(fd, &val, sizeof(val), pos + 0x14);
+		if (ret < 0) {
+			opae_log_err("Failed to read from PCI config space [e:%s]\n",
+				strerror(errno));
+			goto close_fd;
+		}
+		opae_log_dbg("Read 0x%08x from PCI configuration space 0x%x\n",
+			val, pos + 0x14);
+		dev_aer[1] = val;
+	}
+
+	opae_log_dbg("Write 0x%08x to PCI configuration space 0x%x\n",
+		v1, pos + 0x08);
+	ret = pwrite(fd, &v1, sizeof(v1), pos + 0x08);
+	if (ret < 0) {
+		opae_log_err("Failed to write to PCI config space 0x%x [e:%s]\n",
+			pos + 0x08, strerror(errno));
+		goto close_fd;
+	}
+
+	opae_log_dbg("Write 0x%08x to PCI configuration space 0x%x\n",
+		v2, pos + 0x14);
+	ret = pwrite(fd, &v2, sizeof(v2), pos + 0x14);
+	if (ret < 0) {
+		opae_log_err("Failed to write to PCI config space 0x%x [e:%s]\n",
+			pos + 0x14, strerror(errno));
+	}
+
+close_fd:
+	close(fd);
+	return ret < 0 ? ret : 0;
+}
+
+static int enable_aer(pcidev_id id)
+{
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	opae_log_info("Enable AER of %s\n", id->bdf);
+
+	return set_aer(id, dev_aer[0], dev_aer[1], 0);
+}
+
+static int disable_aer(pcidev_id id)
+{
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	opae_log_info("Disable AER of %s\n", id->bdf);
+
+	return set_aer(id, 0xffffffff, 0xffffffff, 1);
+}
+
+static int remove_tree(pcidev_id id)
+{
+	int i, n = 0;
+	pcidev_id child;
+	int ret = 0;
+
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	n = opae_get_child(id, NULL, 0);
+	if (n > 0) {
+		child = (pcidev_id)rte_zmalloc(NULL,
+			sizeof(opae_pci_device) * n, 0);
+		if (!child) {
+			opae_log_err("Failed to malloc for children of %s\n",
+				id->bdf);
+			ret = -ENOMEM;
+			goto end;
+		}
+
+		opae_get_child(id, child, n);
+		for (i = 0; i < n; i++)
+			remove_tree(&child[i]);
+		rte_free(child);
+	}
+
+end:
+	opae_remove_device(id);
+	return ret;
+}
+
+static int remove_device(pcidev_id id)
+{
+	char path[PATH_MAX] = {0};
+	char one[] = {'1', 0};
+	int ret = 0;
+
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	opae_log_info("Remove %s from system\n", id->bdf);
+
+	snprintf(path, PATH_MAX, "%s/%s/remove",
+		rte_pci_get_sysfs_path(), id->bdf);
+	ret = write_file(path, one, strlen(one));
+	if (ret < 0) {
+		opae_log_err("Failed to remove %s from system\n", id->bdf);
+		return ret;
+	}
+
+	remove_tree(id);
+
+	return 0;
+}
+
+static int scan_device(pcidev_id parent, pcidev_id id)
+{
+	char path[PATH_MAX] = {0};
+	char bus[8] = {0};
+	char one[] = {'1', 0};
+	char pwr[16] = {0};
+	char pwr_on[] = {'o', 'n', 0};
+	int pwr_on_failed = 0;
+	int ret = 0;
+
+	if (!parent) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+	opae_log_info("Rescan devices under %s\n", parent->bdf);
+
+	if (id) {   /* scan specified bus under parent device */
+		snprintf(path, PATH_MAX, "%s/%s/power/control",
+			rte_pci_get_sysfs_path(), parent->bdf);
+		ret = read_file(path, pwr, sizeof(pwr));
+		if (ret < 0)
+			return ret;
+
+		if (strcmp(pwr, "on")) {
+			ret = write_file(path, pwr_on, strlen(pwr_on));
+			if (ret < 0)
+				pwr_on_failed = 1;
+			else
+				sleep(1);
+		}
+
+		snprintf(bus, sizeof(bus), "%s", id->bdf);
+		snprintf(path, PATH_MAX, "%s/%s/pci_bus/%s/rescan",
+			rte_pci_get_sysfs_path(), parent->bdf, bus);
+		ret = write_file(path, one, strlen(one));
+		if (ret < 0)
+			return ret;
+
+		if (pwr_on_failed) {   /* workaround for power on failed */
+			ret = write_file(path, one, strlen(one));
+			if (ret < 0)
+				return ret;
+		}
+
+		if (strcmp(pwr, "on")) {
+			snprintf(path, PATH_MAX, "%s/%s/power/control",
+				rte_pci_get_sysfs_path(), parent->bdf);
+			ret = write_file(path, pwr, strlen(pwr));
+		}
+	} else {   /* scan all buses under parent device */
+		snprintf(path, PATH_MAX, "%s/%s/rescan",
+			rte_pci_get_sysfs_path(), parent->bdf);
+		ret = write_file(path, one, strlen(one));
+	}
+
+	return ret;
+}
+
+int opae_reboot_device(pcidev_id id, int type, int page)
+{
+	uint16_t dev_id = 0;
+	opae_pci_device fpga;    /* FPGA after reboot */
+	opae_pci_device parent;
+	opae_pci_device peer[2];   /* physical function 1 of FPGA */
+	opae_pci_device peer_parent;
+	opae_pci_device ups;   /* upstream port device */
+	opae_pci_device root;  /* port connected to PAC */
+	pcidev_id peer_primary = NULL;
+	char drv_name[OPAE_NAME_SIZE] = {0};
+	int n = 0;
+	int i = 0;
+	int ret = 0;
+
+	if (check_pcidev_id(id))
+		return -EINVAL;
+
+	if (!is_pac(id)) {
+		opae_log_err("%s can not be rebooted\n", id->bdf);
+		return -EINVAL;
+	}
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	ret = opae_get_parent(id, &parent);
+	if (ret < 0)
+		return -ENODEV;
+	ret = opae_get_parent(&parent, &ups);
+	if (ret < 0)
+		return -ENODEV;
+	ret = opae_get_parent(&ups, &root);
+	if (ret < 0)
+		return -ENODEV;
+
+	n = opae_get_pf1(id, peer, sizeof(peer) / sizeof(opae_pci_device));
+	if (n <= 0) {
+		opae_log_err("PF1 of %s is not found\n", id->bdf);
+	} else {
+		peer_primary = &peer[0];
+		ret = opae_get_parent(peer_primary, &peer_parent);
+		if (ret < 0)
+			return -ENODEV;
+	}
+
+	get_driver(id, drv_name, sizeof(drv_name));  /* save original driver */
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (rte_pmd_ifpga_reboot_try(dev_id)) {
+		opae_log_warn("Update or reboot is in progress\n");
+		return -EAGAIN;
+	}
+
+	if (type == 0) {
+		/* disable AER */
+		ret = disable_aer(&parent);
+		if (ret < 0) {
+			opae_log_err("Failed to disable AER of %s\n",
+				parent.bdf);
+			goto reboot_end;
+		}
+		ret = disable_aer(&peer_parent);
+		if (ret < 0) {
+			opae_log_err("Failed to disable AER of %s\n",
+				peer_parent.bdf);
+			goto reboot_end;
+		}
+
+		/* trigger reconfiguration */
+		ret = rte_pmd_ifpga_reload(dev_id, type, page);
+		if (ret == 0) {
+			ret = remove_device(id);
+			for (i = 0; i < n; i++)
+				ret += remove_device(&peer[i]);
+			if (ret == 0) {
+				opae_log_info("Wait 10 seconds for FPGA reloading\n");
+				sleep(10);
+				ret = scan_device(&parent, id);
+				if (ret < 0)
+					opae_log_err("Failed to rescan %s\n",
+						id->bdf);
+				if (peer_primary) {
+					ret = scan_device(&peer_parent,
+						peer_primary);
+					if (ret < 0) {
+						opae_log_err("Failed to rescan %s\n",
+							peer_primary->bdf);
+					}
+				}
+			}
+		}
+
+		/* restore AER */
+		if (enable_aer(&parent) < 0) {
+			opae_log_err("Failed to enable AER of %s\n",
+				parent.bdf);
+		}
+		if (enable_aer(&peer_parent) < 0) {
+			opae_log_err("Failed to enable AER of %s\n",
+				peer_parent.bdf);
+		}
+	} else if (type == 1) {
+		/* disable AER */
+		ret = disable_aer(&root);
+		if (ret < 0) {
+			opae_log_err("Failed to disable AER of %s\n", root.bdf);
+			goto reboot_end;
+		}
+
+		/* trigger reconfiguration */
+		ret = rte_pmd_ifpga_reload(dev_id, type, page);
+		if (ret == 0) {
+			ret += remove_device(&ups);
+			if (ret == 0) {
+				opae_log_info("Wait 10 seconds for BMC reloading\n");
+				sleep(10);
+				ret = scan_device(&root, &ups);
+				if (ret < 0)
+					opae_log_err("Failed to rescan %s\n",
+						ups.bdf);
+			}
+		}
+
+		/* restore AER */
+		if (enable_aer(&root) < 0)
+			opae_log_err("Failed to enable AER of %s\n", root.bdf);
+	} else {
+		opae_log_err("Type of reboot is not supported [t:%d]\n", type);
+		ret = -EINVAL;
+		goto reboot_end;
+	}
+
+	/* update id if bdf changed after reboot */
+	if (opae_get_child(&parent, &fpga, 1) == 1) {
+		if (strcmp(id->bdf, fpga.bdf))
+			id = &fpga;
+	}
+
+	ret = opae_bind_driver(id, drv_name);
+	if (ret < 0)
+		opae_log_err("Failed to bind original driver of %s\n", id->bdf);
+
+	ret = opae_probe_device(id);
+	if (ret < 0)
+		opae_log_err("Failed to probe %s [e:%d]\n", id->bdf, ret);
+
+reboot_end:
+	rte_pmd_ifpga_set_rsu_status(dev_id, 0, 0);
+	return ret;
+}
+
+int opae_partial_reconfigure(pcidev_id id, int port, const char *gbs)
+{
+	uint16_t dev_id = 0;
+
+	if (!id || !gbs) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	if (!check_eal(1))
+		return -EPERM;
+
+	if (get_rawdev_id(id, &dev_id, 1))
+		return -ENODEV;
+
+	return rte_pmd_ifpga_partial_reconfigure(dev_id, port, gbs);
+}
+
+int opae_read_pci_cfg(pcidev_id id, uint32_t address, uint32_t *value)
+{
+	char path[PATH_MAX] = {0};
+	int fd = -1;
+	int ret = 0;
+
+	if (!id || !value) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	snprintf(path, PATH_MAX, "%s/%s/config", rte_pci_get_sysfs_path(),
+		id->bdf);
+	fd = open(path, O_RDONLY);
+	if (fd < 0) {
+		opae_log_dbg("Failed to open \'%s\' for RDONLY [e:%s]\n",
+			path, strerror(errno));
+		return -EIO;
+	}
+
+	ret = pread(fd, value, 4, address);
+	if (ret < 0) {
+		opae_log_err("Failed to read from PCI device %s [e:%s]\n",
+			id->bdf, strerror(errno));
+		close(fd);
+		return ret;
+	}
+
+	opae_log_dbg("CONFIG+0x%08x -> 0x%08x\n", address, *value);
+	close(fd);
+	return 0;
+}
+
+int opae_write_pci_cfg(pcidev_id id, uint32_t address, uint32_t value)
+{
+	char path[PATH_MAX] = {0};
+	int fd = -1;
+	int ret = 0;
+
+	if (!id) {
+		opae_log_err("Input parameter of %s is invalid\n", __func__);
+		return -EINVAL;
+	}
+
+	snprintf(path, PATH_MAX, "%s/%s/config", rte_pci_get_sysfs_path(),
+		id->bdf);
+	fd = open(path, O_WRONLY);
+	if (fd < 0) {
+		opae_log_dbg("Failed to open \'%s\' for WRONLY [e:%s]\n",
+			path, strerror(errno));
+		return -EIO;
+	}
+
+	ret = pwrite(fd, &value, 4, address);
+	if (ret < 0) {
+		opae_log_err("Failed to write to PCI device %s [e:%s]\n",
+			id->bdf, strerror(errno));
+		close(fd);
+		return ret;
+	}
+
+	opae_log_dbg("CONFIG+0x%08x <- 0x%08x\n", address, value);
+	close(fd);
+	return 0;
+}
diff --git a/examples/ifpga/opae_api.h b/examples/ifpga/opae_api.h
new file mode 100644
index 0000000000..3b6bc01fbc
--- /dev/null
+++ b/examples/ifpga/opae_api.h
@@ -0,0 +1,244 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _OPAE_API_H
+#define _OPAE_API_H
+
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+extern int opae_log_level;
+extern FILE *opae_log_file;
+
+#define OPAE_LOG_API      0  /**< Critical conditions.              */
+#define OPAE_LOG_ERR      1  /**< Error conditions.                 */
+#define OPAE_LOG_WARN     2  /**< Warning conditions.               */
+#define OPAE_LOG_INFO     3  /**< Informational.                    */
+#define OPAE_LOG_DEBUG    4  /**< Debug-level messages.             */
+
+#define opae_log(type, fmt, args...)             \
+do {                                             \
+	if (opae_log_level >= OPAE_LOG_##type) {     \
+		printf(fmt, ##args);                     \
+		if (opae_log_file) {                     \
+			fprintf(opae_log_file, fmt, ##args); \
+			fflush(opae_log_file);               \
+		}                                        \
+	}                                            \
+} while (0)
+
+#define opae_log_api(fmt, args...)  opae_log(API, "OPAE-API: "fmt, ##args)
+#define opae_log_err(fmt, args...)  opae_log(ERR, "OPAE-ERR: "fmt, ##args)
+#define opae_log_dbg(fmt, args...)  opae_log(DEBUG, "OPAE-DBG: "fmt, ##args)
+#define opae_log_warn(fmt, args...) opae_log(WARN, "OPAE-WARN: "fmt, ##args)
+#define opae_log_info(fmt, args...) opae_log(INFO, "OPAE-INFO: "fmt, ##args)
+
+#define EAL_INIT_FUNCTION    "init"
+#define EAL_DEFAULT_OPTIONS  "--proc-type auto"
+
+#define OPAE_KDRV_UNKNOWN           "unknown"
+#define OPAE_KDRV_VFIO_PCI          "vfio-pci"
+#define OPAE_KDRV_IGB_UIO           "igb_uio"
+#define OPAE_KDRV_UIO_PCI           "uio_pci_generic"
+#define OPAE_KDRV_INTEL_FPGA_PCI    "intel-fpga-pci"
+
+typedef struct {
+	uint32_t major;
+	uint32_t minor;
+	uint32_t micro;
+} opae_api_version;
+
+#define OPAE_NAME_SIZE  32
+
+typedef struct {
+	char bdf[OPAE_NAME_SIZE];   /* segment:bus:device.function */
+} opae_pci_device;
+
+typedef opae_pci_device *pcidev_id;
+
+typedef struct {
+	uint32_t class_id;            /**< Class ID or RTE_CLASS_ANY_ID. */
+	uint16_t vendor_id;           /**< Vendor ID or PCI_ANY_ID. */
+	uint16_t device_id;           /**< Device ID or PCI_ANY_ID. */
+	uint16_t subsystem_vendor_id; /**< Subsystem vendor ID or PCI_ANY_ID. */
+	uint16_t subsystem_device_id; /**< Subsystem device ID or PCI_ANY_ID. */
+} opae_pci_id;
+
+typedef struct {
+	uint32_t domain;              /**< Device domain */
+	uint8_t bus;                  /**< Device bus */
+	uint8_t devid;                /**< Device ID */
+	uint8_t function;             /**< Device function. */
+} opae_pci_addr;
+
+typedef struct {
+	char pci_addr[OPAE_NAME_SIZE];  /* segment:bus:device.function */
+	char drv_name[OPAE_NAME_SIZE];  /* vfio-pci, intel-fpga-pci, etc. */
+	opae_pci_id id;
+	opae_pci_addr addr;
+} opae_pci_property;
+
+#define BIT_SET_8   0xFF
+#define BIT_SET_16  0xFFFF
+#define BIT_SET_32  0xFFFFFFFF
+
+typedef struct {
+	uint8_t b[16];
+} opae_uuid;
+
+typedef struct {
+	uint32_t boot_page;
+	uint32_t num_ports;
+	uint64_t bitstream_id;
+	uint64_t bitstream_metadata;
+	opae_uuid pr_id;
+	char platform_name[OPAE_NAME_SIZE];
+	char dcp_version[OPAE_NAME_SIZE];
+	char release_name[OPAE_NAME_SIZE];
+	char interface_type[OPAE_NAME_SIZE];
+	char build_version[OPAE_NAME_SIZE];
+} opae_fme_property;
+
+typedef struct {
+	opae_uuid afu_id;
+	uint32_t type;   /* AFU memory access control type */
+	uint32_t index;  /* PORT index */
+} opae_port_property;
+
+typedef struct {
+	char bmc_version[OPAE_NAME_SIZE];
+	char fw_version[OPAE_NAME_SIZE];
+} opae_bmc_property;
+
+typedef struct {
+	uint32_t num_retimers;
+	uint32_t link_speed;
+	uint32_t link_status;  /* each bit corresponding to one link status */
+} opae_phy_info;
+
+typedef struct {
+	union {
+		uint64_t id;
+		struct {
+			uint8_t build_patch;
+			uint8_t build_minor;
+			uint8_t build_major;
+			uint8_t fvl_bypass:1;
+			uint8_t mac_lightweight:1;
+			uint8_t disagregate:1;
+			uint8_t lightweiht:1;
+			uint8_t seu:1;
+			uint8_t ptp:1;
+			uint8_t reserve:2;
+			uint16_t interface:4;
+			uint16_t afu_revision:12;
+			uint16_t patch:4;
+			uint16_t minor:4;
+			uint16_t major:4;
+			uint16_t reserved:4;
+		};
+	};
+} opae_bitstream_id;
+
+typedef struct {
+	union {
+		uint32_t version;
+		struct {
+			uint8_t micro;
+			uint8_t minor;
+			uint8_t major;
+			uint8_t board;
+		};
+	};
+} opae_bmc_version;
+
+#define OPAE_MAX_PORT_NUM   4
+
+#define OPAE_PROP_PCI   0x01
+#define OPAE_PROP_FME   0x02
+#define OPAE_PROP_PORT  0x04
+#define OPAE_PROP_BMC   0x08
+#define OPAE_PROP_ALL   \
+	(OPAE_PROP_PCI | OPAE_PROP_FME | OPAE_PROP_PORT | OPAE_PROP_BMC)
+
+typedef struct {
+	opae_pci_property pci;
+	opae_fme_property fme;
+	opae_port_property port[OPAE_MAX_PORT_NUM];
+	opae_bmc_property bmc;
+} opae_fpga_property;
+
+typedef struct {
+	uint64_t guid_h;
+	uint64_t guid_l;
+	uint32_t metadata_len;
+} gbs_header;
+
+#define OPAE_IMG_TYPE_BBS       0
+#define OPAE_IMG_TYPE_BMC       1
+#define OPAE_IMG_TYPE_GBS       2
+#define OPAE_IMG_TYPE(t)        ((t) & 0xff)
+
+#define OPAE_IMG_SUBTYPE_UPDATE             0
+#define OPAE_IMG_SUBTYPE_CANCELLATION       1
+#define OPAE_IMG_SUBTYPE_ROOT_KEY_HASH_256  2
+#define OPAE_IMG_SUBTYPE_ROOT_KEY_HASH_384  3
+#define OPAE_IMG_SUBTYPE(t)     (((t) >> 8) & 0xff)
+
+#define OPAE_IMG_BLK0_SIZE      128
+#define OPAE_IMG_BLK0_MAGIC     0xb6eafd19
+#define OPAE_IMG_BLK1_SIZE      896
+#define OPAE_IMG_HDR_SIZE   (OPAE_IMG_BLK0_SIZE + OPAE_IMG_BLK1_SIZE)
+#define OPAE_IMG_PL_MIN_SIZE    128
+
+typedef struct {
+	uint32_t magic;
+	uint32_t payload_len;
+	uint32_t payload_type;
+} opae_img_hdr;
+
+typedef struct {
+	int type;
+	int subtype;
+	uint32_t total_len;
+	uint32_t payload_offset;
+	uint32_t payload_len;
+} opae_img_info;
+
+void opae_get_api_version(opae_api_version *version);
+int opae_set_log_level(int level);
+int opae_set_log_file(char *path, int clean);
+int opae_get_proc_type(void);
+int opae_get_parent(pcidev_id id, pcidev_id parent);
+int opae_get_child(pcidev_id id, pcidev_id child, int size);
+int opae_get_pf1(pcidev_id id, pcidev_id peer, int size);
+int opae_init_eal(int argc, char **argv);
+int opae_cleanup_eal(void);
+int opae_enumerate(opae_pci_id *filter, pcidev_id list, int size);
+int opae_probe_device(pcidev_id id);
+int opae_remove_device(pcidev_id id);
+int opae_unbind_driver(pcidev_id id);
+int opae_bind_driver(pcidev_id id, char *drv_name);
+int opae_get_property(pcidev_id id, opae_fpga_property *prop, int type);
+int opae_get_phy_info(pcidev_id id, opae_phy_info *info);
+int opae_partial_reconfigure(pcidev_id id, int port, const char *gbs);
+int opae_get_image_info(const char *image, opae_img_info *info);
+int opae_cancel_flash_update(pcidev_id id, int force);
+int opae_update_flash(pcidev_id id, const char *image, uint64_t *status);
+int opae_reboot_device(pcidev_id id, int type, int page);
+int opae_store_rsu_status(pcidev_id id, uint32_t status, uint32_t progress);
+int opae_load_rsu_status(pcidev_id id, uint32_t *status, uint32_t *progress);
+int opae_read_pci_cfg(pcidev_id id, uint32_t address, uint32_t *value);
+int opae_write_pci_cfg(pcidev_id id, uint32_t address, uint32_t value);
+
+#ifdef __cplusplus
+}
+#endif
+
+
+#endif  /* _OPAE_API_H */
diff --git a/examples/meson.build b/examples/meson.build
index b9ab24223f..b029715277 100644
--- a/examples/meson.build
+++ b/examples/meson.build
@@ -16,7 +16,7 @@ all_examples = [
 	'eventdev_pipeline',
 	'fips_validation', 'flow_classify',
 	'flow_filtering', 'helloworld',
-	'ioat',
+	'ioat', 'ifpga',
 	'ip_fragmentation', 'ip_pipeline',
 	'ip_reassembly', 'ipsec-secgw',
 	'ipv4_multicast', 'kni',
-- 
2.29.2