From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A2639A034F; Mon, 22 Feb 2021 08:36:26 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B299B22A2A4; Mon, 22 Feb 2021 08:35:59 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 6C7D122A28A; Mon, 22 Feb 2021 08:35:56 +0100 (CET) IronPort-SDR: Ga54g2SmJwdaWRzg4ScNcu6jvbel2H6Fvc9zCahSwrtDSU6+gpCbrZl9t5VGDUqet8BJp0NwBT B1ppzIffGqDA== X-IronPort-AV: E=McAfee;i="6000,8403,9902"; a="183640866" X-IronPort-AV: E=Sophos;i="5.81,196,1610438400"; d="scan'208";a="183640866" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Feb 2021 23:35:56 -0800 IronPort-SDR: CkkTiSpOg+CoushUNOhRlWTMa7+MWtC6ttnZgI1VYCwPQLsR/NgD9q8zHE0mMKAjlhRfD0K7TR QNAxnY8atRcA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.81,196,1610438400"; d="scan'208";a="498391236" Received: from unknown (HELO sh_lab5_1.sh.intel.com) ([10.238.175.190]) by fmsmga001.fm.intel.com with ESMTP; 21 Feb 2021 23:35:53 -0800 From: Wei Huang To: dev@dpdk.org, rosen.xu@intel.com, qi.z.zhang@intel.com Cc: stable@dpdk.org, tianfei.zhang@intel.com, ferruh.yigit@intel.com, Wei Huang Date: Mon, 22 Feb 2021 02:35:52 -0500 Message-Id: <1613979352-4864-5-git-send-email-wei.huang@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1613979352-4864-1-git-send-email-wei.huang@intel.com> References: <1613963747-30047-1-git-send-email-wei.huang@intel.com> <1613979352-4864-1-git-send-email-wei.huang@intel.com> Subject: [dpdk-dev] [PATCH v15 4/4] examples/ifpga: add example for ifpga APIs X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This example application shows how to use ifpga APIs. A set of OPAE APIs are implemented based on these ifpga APIs. You can test each OPAE API by running corresponding command. A guide is also added to show how to run the example. Signed-off-by: Wei Huang Acked-by: Tianfei Zhang Acked-by: Rosen Xu --- v2: fix coding style issue in commands.c --- v3: add guide for running example --- v4: fix compilation issue of ifpga.rst --- v5: add ifpga.rst into sample_app_ug/index.rst --- v6: implement OPAE APIs in example instead of in driver --- v7: fix compilation issue of unused const variable 'kdrv' --- v8: fix compilation issue of format error --- MAINTAINERS | 3 + doc/api/doxy-api.conf.in | 1 + doc/guides/sample_app_ug/ifpga.rst | 387 +++++++ doc/guides/sample_app_ug/index.rst | 1 + examples/ifpga/Makefile | 43 + examples/ifpga/commands.c | 1294 ++++++++++++++++++++++ examples/ifpga/commands.h | 16 + examples/ifpga/main.c | 38 + examples/ifpga/meson.build | 20 + examples/ifpga/opae_api.c | 1630 ++++++++++++++++++++++++++++ examples/ifpga/opae_api.h | 244 +++++ examples/meson.build | 2 +- 12 files changed, 3678 insertions(+), 1 deletion(-) create mode 100644 doc/guides/sample_app_ug/ifpga.rst create mode 100644 examples/ifpga/Makefile create mode 100644 examples/ifpga/commands.c create mode 100644 examples/ifpga/commands.h create mode 100644 examples/ifpga/main.c create mode 100644 examples/ifpga/meson.build create mode 100644 examples/ifpga/opae_api.c create mode 100644 examples/ifpga/opae_api.h diff --git a/MAINTAINERS b/MAINTAINERS index 1a12916f56..48c2ea44d4 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1236,9 +1236,12 @@ Rawdev Drivers Intel FPGA M: Rosen Xu M: Tianfei zhang +M: Wei Huang T: git://dpdk.org/next/dpdk-next-net-intel F: drivers/raw/ifpga/ F: doc/guides/rawdevs/ifpga.rst +F: examples/ifpga/ +F: doc/guides/sample_app_ug/ifpga.rst IOAT Rawdev M: Bruce Richardson diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in index 5c883b613b..5eb31508fd 100644 --- a/doc/api/doxy-api.conf.in +++ b/doc/api/doxy-api.conf.in @@ -23,6 +23,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/drivers/net/softnic \ @TOPDIR@/drivers/raw/dpaa2_cmdif \ @TOPDIR@/drivers/raw/dpaa2_qdma \ + @TOPDIR@/drivers/raw/ifpga \ @TOPDIR@/drivers/raw/ioat \ @TOPDIR@/lib/librte_eal/include \ @TOPDIR@/lib/librte_eal/include/generic \ diff --git a/doc/guides/sample_app_ug/ifpga.rst b/doc/guides/sample_app_ug/ifpga.rst new file mode 100644 index 0000000000..bc05153418 --- /dev/null +++ b/doc/guides/sample_app_ug/ifpga.rst @@ -0,0 +1,387 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2020-2021 Intel Corporation. + +Intel FPGA Sample Application +============================= + +The Intel FPGA sample application is an example of how to use rte_pmd_ifpga API +to manage Intel FPGA. + +Overview +-------- + +The Intel FPGA sample application is a simple application that demonstrates +the use of the APIs provided by ifpga driver in the DPDK. +A set of OPAE APIs are implemented in this example with 'opae' prefix. These +APIs use PCI address ('Domain:Bus:Dev.Func') as an unique ID of FPGA, +application based on OPAE can manage specified FPGA easily with them. +OPAE (Open Programmable Acceleration Engine) is a software framework for +managing and accessing programmable accelerators (FPGAs). For more information +about OPAE, please refer to https://opae.github.io/2.0.0/index.html . +This application is a readline-like interface that can be used to manage +Intel FPGA, in a Linux* application environment. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling` + +The application is located in the ``ifpga`` sub-directory. + +Running the Application +----------------------- + +To run the application in linux environment, issue the following command: + +.. code-block:: console + + $ .//examples/dpdk-ifpga --proc-type=auto + +Refer to the *DPDK Getting Started Guide* for general information on running +applications and the Environment Abstraction Layer (EAL) options. + +Commands Description +-------------------- + +The following sections provide some explanation of the commands. + +help command +~~~~~~~~~~~~ + +The application has on-line help for the commands that are available at runtime. + +.. code-block:: console + + opae> help + get_api_version get OPAE API version + get_proc_type get DPDK process type + get_image_info get information of image file + get_status get current status & progress of FPGA + get_property <0|1|2|4|8> get property of FPGA + get_phy_info get information of PHY + get_parent get parent PCI device of FPGA + get_child get child PCI device of FPGA + get_pf1 get physical function 1 device of FPGA + set_log_level <0-4> set logging level + set_log_file set logging file + set_status <0-4> <0-100> set current status & progress of FPGA + enumerate enumerate specified FPGA + bind bind FPGA with kernel driver + unbind unbind FPGA from kernel driver + probe probe FPGA with IFPGA driver + remove remove FPGA from IFPGA driver + flash update flash of FPGA + pr partial reconfigure FPGA + reboot <0-1> reboot FPGA or MAX10 + cancel cancel flash update + pci_read <0-1024> read PCI configuration space + pci_write <0-1024> write PCI configuration space + quit exit DPDK application + help show commands list + +get_api_version command +~~~~~~~~~~~~~~~~~~~~~~~ + +Show OPAE API version which is same to the version of DPDK. + +.. code-block:: console + + opae> get_api_version + 21.2.0 + +set_log_level command +~~~~~~~~~~~~~~~~~~~~~ + +Set logging level of OPAE API. Logging level is defined as below. +0 - CRITICAL +1 - ERROR +2 - WARNING +3 - INFORMATION +4 - DEBUG + +.. code-block:: console + + opae> set_log_level 4 + OPAE-API: Current log level is DEBUG + Successful + opae> set_log_level 6 + OPAE-API: Current log level is DEBUG + Failed + +set_log_file command +~~~~~~~~~~~~~~~~~~~~ + +Set logging file of OPAE API. + +.. code-block:: console + + opae> set_log_file /tmp/ifpga.log + Successful + +get_proc_type command +~~~~~~~~~~~~~~~~~~~~~ + +Show the process type of DPDK. If you start multiple instances of the +application, the process type of the first one is 'Primary', the others +are 'Secondary'. + +.. code-block:: console + + opae> get_proc_type + Primary + +get_image_info command +~~~~~~~~~~~~~~~~~~~~~~ + +Display information of FPGA image file. + +.. code-block:: console + + opae> get_image_info /home/wei/a10.bin + Type: FPGA_BBS + Action: UPDATE + Total length: 58720256 + Payload offset: 1024 + Payload length: 58719232 + opae> get_image_info /home/wei/data.bin + OPAE-ERR: Image '/home/wei/data.bin' can not be recognized + Invalid image file + +enumerate command +~~~~~~~~~~~~~~~~~ + +Display PCI address of FPGA with specified vendor ID and device ID. ID value can +be set to 0xffff for arbitrary ID. + +.. code-block:: console + + opae> enumerate 0x8086 0x0b30 + 0000:24:00.0 + +get_property command +~~~~~~~~~~~~~~~~~~~~ + +Display property information of specified FPGA. Property type is defined as below. +0 - All properties +1 - PCI property +2 - FME property +4 - port property +8 - BMC property +PCI property is always available, other properties can only be displayed after +ifpga driver is probed to the FPGA. + +.. code-block:: console + + opae> get_property 24:00.0 0 + PCI: + PCIe s:b:d.f : 0000:24:00.0 + kernel driver : vfio-pci + FME: + platform : Vista Creek + DCP version : DCP 1.2 + phase : Beta + interface : 2x2x25G + build version : 0.0.2 + ports num : 1 + boot page : user + pr interface id : a5d72a3c-c8b0-4939-912c-f715e5dc10ca + PORT0: + access type : PF + accelerator id : 8892c23e-2eed-4b44-8bb6-5c88606e07df + BMC: + MAX10 version : D.2.0.5 + NIOS FW version : D.2.0.12 + +get_phy_info command +~~~~~~~~~~~~~~~~~~~~ + +Display information and status of PHY connects to the specified FPGA. + +.. code-block:: console + + opae> get_phy_info 24:00.0 + retimers num : 2 + link speed : 25G + link status : 00 + +get_parent command +~~~~~~~~~~~~~~~~~~ + +Display PCI address of upstream device connects to the specified FPGA. + +.. code-block:: console + + opae> get_parent 24:00.0 + 0000:22:09.0 + +get_child command +~~~~~~~~~~~~~~~~~ + +Display PCI address of downstream device connects to the specified FPGA. + +.. code-block:: console + + opae> get_child 24:00.0 + No child + opae> get_child 22:09.0 + 0000:24:00.0 + +get_pf1 command +~~~~~~~~~~~~~~~ + +Display PCI address of PF1 (physical function 1) of specified FPGA. + +.. code-block:: console + + opae> get_pf1 24:00.0 + 0000:26:00.0 + 0000:26:00.1 + +get_status command +~~~~~~~~~~~~~~~~~~ + +Display current RSU status of specified FPGA. + +.. code-block:: console + + opae> get_status 24:00.0 + Status: IDLE + Progress: 0% + +set_status command +~~~~~~~~~~~~~~~~~~ + +Set current RSU status of specified FPGA. This command is mainly used for debug +purpose. Status value is defined as below. +0 - IDLE +1 - PREPARE +2 - PROGRAM +3 - COPY +4 - REBOOT + +.. code-block:: console + + opae> set_status 24:00.0 2 35 + Successful + opae> get_status 24:00.0 + Status: PROGRAM + Progress: 35% + +unbind command +~~~~~~~~~~~~~~ + +Unbind kernel driver from specified FPGA. + +.. code-block:: console + + opae> unbind 24:00.0 + OPAE-ERR: 0000:24:00.0 is probed, remove it first + Failed + opae> remove 24:00.0 + Successful + opae> unbind 24:00.0 + Successful + +bind command +~~~~~~~~~~~~ + +Bind specified kernel driver to specified FPGA. + +.. code-block:: console + + opae> bind 24:00.0 vfio-pci + Successful + +probe command +~~~~~~~~~~~~~ + +Probe specified FPGA with DPDK PMD driver. + +.. code-block:: console + + opae> probe 24:00.0 + Successful + +remove command +~~~~~~~~~~~~~~ + +Remove specified FPGA from DPDK PMD driver. It's a reverse operation to probe +command. + +.. code-block:: console + + opae> remove 24:00.0 + Successful + +flash command +~~~~~~~~~~~~~ + +Update image in flash of specified FPGA. + +.. code-block:: console + + opae> flash 24:00.0 /home/wei/a10.bin + Successful + +pr command +~~~~~~~~~~ + +Do partial reconfiguration of specified FPGA. + +.. code-block:: console + + opae> pr 24:00.0 0 /home/wei/nlb0.gbs + Successful + +reboot command +~~~~~~~~~~~~~~ + +Reboot specified FPGA. Reboot type and page is defined as below. +fpga - reboot FPGA only +bmc - reboot whole card with FPGA +0 - factory page +1 - user page + +.. code-block:: console + + opae> reboot 24:00.0 fpga 1 + Successful + +cancel command +~~~~~~~~~~~~~~ + +Cancel flash programming of specified FPGA. + +.. code-block:: console + + opae> cancel 24:00.0 + Successful + +pci_read command +~~~~~~~~~~~~~~~~ + +Read PCI configuration space of specified FPGA. + +.. code-block:: console + + opae> pci_read 24:00.0 0 + 0x0b308086 + +pci_write command +~~~~~~~~~~~~~~~~~ + +Write PCI configuration space of specified FPGA. + +.. code-block:: console + + opae> pci_write 24:00.0 4 0x100406 + Successful + +quit command +~~~~~~~~~~~~ + +Exit this sample application. + +.. code-block:: console + + opae> quit diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst index e8db83d3a7..beb94ff3e7 100644 --- a/doc/guides/sample_app_ug/index.rst +++ b/doc/guides/sample_app_ug/index.rst @@ -23,6 +23,7 @@ Sample Applications User Guides kernel_nic_interface keep_alive ioat + ifpga l2_forward_crypto l2_forward_job_stats l2_forward_real_virtual diff --git a/examples/ifpga/Makefile b/examples/ifpga/Makefile new file mode 100644 index 0000000000..7d76479a36 --- /dev/null +++ b/examples/ifpga/Makefile @@ -0,0 +1,43 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2020-2021 Intel Corporation + +# binary name +APP = ifpga + +# all source are stored in SRCS-y +SRCS-y := main.c commands.c opae_api.c + +# Build using pkg-config variables if possible +ifneq ($(shell pkg-config --exists libdpdk && echo 0),0) +$(error "no installation of DPDK found") +endif + +all: shared +.PHONY: shared static +shared: build/$(APP)-shared + ln -sf $(APP)-shared build/$(APP) +static: build/$(APP)-static + ln -sf $(APP)-static build/$(APP) + +PKGCONF ?= pkg-config + +PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null) +CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) +LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk) -lrte_bus_pci -lrte_raw_ifpga +LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk) + +CFLAGS += -DALLOW_EXPERIMENTAL_API + +build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build + $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED) + +build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build + $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC) + +build: + @mkdir -p $@ + +.PHONY: clean +clean: + rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared + test -d build && rmdir -p build || true diff --git a/examples/ifpga/commands.c b/examples/ifpga/commands.c new file mode 100644 index 0000000000..4209e23de6 --- /dev/null +++ b/examples/ifpga/commands.c @@ -0,0 +1,1294 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2020-2021 Intel Corporation. + * All rights reserved. + */ + +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +#include "commands.h" + +static int parse_pciaddr(const char *bdf, opae_pci_device *id) +{ + size_t len = 0; + unsigned int domain = 0; + unsigned int bus = 0; + unsigned int devid = 0; + unsigned int function = 0; + + if (!bdf || !id) + return -EINVAL; + + len = strlen(bdf); + if ((len < 5) || (len > 12)) + return -EINVAL; + + len = sscanf(bdf, "%x:%x:%x.%d", &domain, &bus, &devid, &function); + if (len == 4) { + snprintf(id->bdf, sizeof(id->bdf), "%04x:%02x:%02x.%d", + domain, bus, devid, function); + } else { + len = sscanf(bdf, "%x:%x.%d", &bus, &devid, &function); + if (len == 3) { + snprintf(id->bdf, sizeof(id->bdf), "%04x:%02x:%02x.%d", + 0, bus, devid, function); + } else { + return -EINVAL; + } + } + return 0; +} + +static void uuid_to_str(opae_uuid *id, uuid_str *str) +{ + uint8_t *b = NULL; + char *p = NULL; + int i, j; + + if (!id || !str) + return; + + b = &id->b[15]; + p = str->s; + for (i = 0; i < 4; i++, b--, p += 2) + sprintf(p, "%02x", *b); + sprintf(p++, "-"); + for (i = 0; i < 3; i++) { + for (j = 0; j < 2; j++, b--, p += 2) + sprintf(p, "%02x", *b); + sprintf(p++, "-"); + } + for (i = 0; i < 6; i++, b--, p += 2) + sprintf(p, "%02x", *b); +} + +/* *** GET API VERSION *** */ +struct cmd_version_result { + cmdline_fixed_string_t cmd; +}; + +static void cmd_version_parsed(__rte_unused void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + opae_api_version ver; + opae_get_api_version(&ver); + cmdline_printf(cl, "%d.%d.%d\n", ver.major, ver.minor, ver.micro); +} + +cmdline_parse_token_string_t cmd_version_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_version_result, cmd, "get_api_version"); + +cmdline_parse_inst_t cmd_get_api_version = { + .f = cmd_version_parsed, + .data = NULL, + .help_str = "get OPAE API version", + .tokens = { + (void *)&cmd_version_cmd, + NULL, + }, +}; + +/* *** GET PROC TYPE *** */ +struct cmd_proc_type_result { + cmdline_fixed_string_t cmd; +}; + +static void cmd_proc_type_parsed(__rte_unused void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + int type = opae_get_proc_type(); + + if (type == 0) + cmdline_printf(cl, "Primary\n"); + else if (type == 1) + cmdline_printf(cl, "Secondary\n"); + else + cmdline_printf(cl, "Unknown\n"); +} + +cmdline_parse_token_string_t cmd_proc_type_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_proc_type_result, cmd, "get_proc_type"); + +cmdline_parse_inst_t cmd_get_proc_type = { + .f = cmd_proc_type_parsed, + .data = NULL, + .help_str = "get DPDK process type", + .tokens = { + (void *)&cmd_proc_type_cmd, + NULL, + }, +}; + +/* *** GET IMAGE INFO *** */ +struct cmd_image_info_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t path; +}; + +static void cmd_image_info_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_image_info_result *res = parsed_result; + opae_img_info info; + + if (opae_get_image_info(res->path, &info) == 0) { + cmdline_printf(cl, "%-16s", "Type:"); + if (info.type == OPAE_IMG_TYPE_BBS) + cmdline_printf(cl, "FPGA_BBS\n"); + else if (info.type == OPAE_IMG_TYPE_BMC) + cmdline_printf(cl, "BMC\n"); + else if (info.type == OPAE_IMG_TYPE_GBS) + cmdline_printf(cl, "FGPA_GBS\n"); + else + cmdline_printf(cl, "Unknown\n"); + cmdline_printf(cl, "%-16s", "Action:"); + if (info.subtype == OPAE_IMG_SUBTYPE_UPDATE) + cmdline_printf(cl, "UPDATE\n"); + else if (info.subtype == OPAE_IMG_SUBTYPE_CANCELLATION) + cmdline_printf(cl, "CANCELLATION\n"); + else if (info.subtype == OPAE_IMG_SUBTYPE_ROOT_KEY_HASH_256) + cmdline_printf(cl, "ROOT_HASH_256\n"); + else if (info.subtype == OPAE_IMG_SUBTYPE_ROOT_KEY_HASH_384) + cmdline_printf(cl, "ROOT_HASH_384\n"); + else + cmdline_printf(cl, "Unknown\n"); + cmdline_printf(cl, "%-16s%u\n", "Total length:", + info.total_len); + cmdline_printf(cl, "%-16s%u\n", "Payload offset:", + info.payload_offset); + cmdline_printf(cl, "%-16s%u\n", "Payload length:", + info.payload_len); + } else { + cmdline_printf(cl, "Invalid image file\n"); + } +} + +cmdline_parse_token_string_t cmd_image_info_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_image_info_result, cmd, + "get_image_info"); +cmdline_parse_token_string_t cmd_image_info_path = + TOKEN_STRING_INITIALIZER(struct cmd_image_info_result, path, NULL); + +cmdline_parse_inst_t cmd_get_image_info = { + .f = cmd_image_info_parsed, + .data = NULL, + .help_str = "get information of image file", + .tokens = { + (void *)&cmd_image_info_cmd, + (void *)&cmd_image_info_path, + NULL, + }, +}; + +/* *** GET STATUS *** */ +struct cmd_get_status_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t bdf; +}; + +static void cmd_get_status_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_get_status_result *res = parsed_result; + opae_pci_device id; + uint32_t stat, prog; + + if (parse_pciaddr(res->bdf, &id) < 0) { + cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf); + return; + } + + if (opae_load_rsu_status(&id, &stat, &prog) == 0) { + cmdline_printf(cl, "%-10s", "Status:"); + if (stat == 0) + cmdline_printf(cl, "IDLE\n"); + else if (stat == 1) + cmdline_printf(cl, "PREPARE\n"); + else if (stat == 2) + cmdline_printf(cl, "PROGRAM\n"); + else if (stat == 3) + cmdline_printf(cl, "COPY\n"); + else if (stat == 4) + cmdline_printf(cl, "REBOOT\n"); + else + cmdline_printf(cl, "unknown\n"); + cmdline_printf(cl, "%-10s%u%%\n", "Progress:", prog); + } else { + cmdline_printf(cl, "Failed\n"); + } +} + +cmdline_parse_token_string_t cmd_get_status_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_get_status_result, cmd, "get_status"); +cmdline_parse_token_string_t cmd_get_status_bdf = + TOKEN_STRING_INITIALIZER(struct cmd_get_status_result, bdf, NULL); + +cmdline_parse_inst_t cmd_get_status = { + .f = cmd_get_status_parsed, + .data = NULL, + .help_str = "get current status & progress of FPGA", + .tokens = { + (void *)&cmd_get_status_cmd, + (void *)&cmd_get_status_bdf, + NULL, + }, +}; + +/* *** GET PROPERTY *** */ +struct cmd_property_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t bdf; + int32_t type; +}; + +static void cmd_property_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_property_result *res = parsed_result; + opae_pci_device id; + opae_fpga_property prop; + uuid_str str; + uint32_t port = 0; + + switch (res->type) { + case 0: + case 1: + case 2: + case 4: + case 8: + break; + default: + cmdline_printf(cl, "%d is invalid type of property\n", + res->type); + return; + } + if (parse_pciaddr(res->bdf, &id) < 0) { + cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf); + return; + } + + if (opae_get_property(&id, &prop, res->type) == 0) { + if ((res->type == 0) || (res->type == 1)) { + cmdline_printf(cl, "%s:\n", "PCI"); + cmdline_printf(cl, " %-16s : %s\n", + "PCIe s:b:d.f", prop.pci.pci_addr); + cmdline_printf(cl, " %-16s : %s\n", + "kernel driver", prop.pci.drv_name); + } + if ((res->type == 0) || (res->type == 2)) { + cmdline_printf(cl, "%s:\n", "FME"); + cmdline_printf(cl, " %-16s : %s\n", + "platform", prop.fme.platform_name); + cmdline_printf(cl, " %-16s : %s\n", + "DCP version", prop.fme.dcp_version); + cmdline_printf(cl, " %-16s : %s\n", + "phase", prop.fme.release_name); + cmdline_printf(cl, " %-16s : %s\n", + "interface", prop.fme.interface_type); + cmdline_printf(cl, " %-16s : %s\n", + "build version", prop.fme.build_version); + cmdline_printf(cl, " %-16s : %u\n", + "ports num", prop.fme.num_ports); + cmdline_printf(cl, " %-16s : %s\n", + "boot page", prop.fme.boot_page ? "user" : "factory"); + uuid_to_str(&prop.fme.pr_id, &str); + cmdline_printf(cl, " %-16s : %s\n", "pr interface id", + str.s); + } + if ((res->type == 0) || (res->type == 4)) { + for (port = 0; port < prop.fme.num_ports; port++) { + cmdline_printf(cl, "%s%d:\n", "PORT", port); + cmdline_printf(cl, " %-16s : %s\n", + "access type", + prop.port[port].type ? "VF" : "PF"); + uuid_to_str(&prop.port[port].afu_id, &str); + cmdline_printf(cl, " %-16s : %s\n", + "accelerator id", str.s); + } + } + if ((res->type == 0) || (res->type == 8)) { + cmdline_printf(cl, "%s:\n", "BMC"); + cmdline_printf(cl, " %-16s : %s\n", + "MAX10 version", prop.bmc.bmc_version); + cmdline_printf(cl, " %-16s : %s\n", + "NIOS FW version", prop.bmc.fw_version); + } + } else { + cmdline_printf(cl, "Failed\n"); + } +} + +cmdline_parse_token_string_t cmd_property_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_property_result, cmd, "get_property"); +cmdline_parse_token_string_t cmd_property_bdf = + TOKEN_STRING_INITIALIZER(struct cmd_property_result, bdf, NULL); +cmdline_parse_token_num_t cmd_property_type = + TOKEN_NUM_INITIALIZER(struct cmd_property_result, type, RTE_INT32); + +cmdline_parse_inst_t cmd_get_property = { + .f = cmd_property_parsed, + .data = NULL, + .help_str = "get property of FPGA", + .tokens = { + (void *)&cmd_property_cmd, + (void *)&cmd_property_bdf, + (void *)&cmd_property_type, + NULL, + }, +}; + +/* *** GET PHY INFO *** */ +struct cmd_phy_info_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t bdf; +}; + +static void cmd_phy_info_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_phy_info_result *res = parsed_result; + opae_pci_device id; + opae_phy_info info; + + if (parse_pciaddr(res->bdf, &id) < 0) { + cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf); + return; + } + + if (opae_get_phy_info(&id, &info) == 0) { + cmdline_printf(cl, " %-16s : %u\n", + "retimers num", info.num_retimers); + cmdline_printf(cl, " %-16s : %uG\n", + "link speed", info.link_speed); + cmdline_printf(cl, " %-16s : %02xh\n", + "link status", info.link_status); + } else { + cmdline_printf(cl, "Failed\n"); + } +} + +cmdline_parse_token_string_t cmd_phy_info_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_phy_info_result, cmd, "get_phy_info"); +cmdline_parse_token_string_t cmd_phy_info_bdf = + TOKEN_STRING_INITIALIZER(struct cmd_phy_info_result, bdf, NULL); + +cmdline_parse_inst_t cmd_phy_info = { + .f = cmd_phy_info_parsed, + .data = NULL, + .help_str = "get information of PHY", + .tokens = { + (void *)&cmd_phy_info_cmd, + (void *)&cmd_phy_info_bdf, + NULL, + }, +}; + +/* *** GET PARENT *** */ +struct cmd_parent_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t bdf; +}; + +static void cmd_parent_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_parent_result *res = parsed_result; + opae_pci_device id; + opae_pci_device parent; + + if (parse_pciaddr(res->bdf, &id) < 0) { + cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf); + return; + } + + if (opae_get_parent(&id, &parent) > 0) + cmdline_printf(cl, "%s\n", parent.bdf); + else + cmdline_printf(cl, "Failed\n"); +} + +cmdline_parse_token_string_t cmd_parent_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_parent_result, cmd, "get_parent"); +cmdline_parse_token_string_t cmd_parent_bdf = + TOKEN_STRING_INITIALIZER(struct cmd_parent_result, bdf, NULL); + +cmdline_parse_inst_t cmd_get_parent = { + .f = cmd_parent_parsed, + .data = NULL, + .help_str = "get parent PCI device of FPGA", + .tokens = { + (void *)&cmd_parent_cmd, + (void *)&cmd_parent_bdf, + NULL, + }, +}; + +/* *** GET CHILD *** */ +struct cmd_child_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t bdf; +}; + +static void cmd_child_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_child_result *res = parsed_result; + opae_pci_device id; + pcidev_id child; + int i, count = 0; + + if (parse_pciaddr(res->bdf, &id) < 0) { + cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf); + return; + } + + count = opae_get_child(&id, NULL, 0); + if (count > 0) { + child = (pcidev_id)malloc(sizeof(opae_pci_device) * count); + if (child) { + opae_get_child(&id, child, count); + for (i = 0; i < count; i++) + cmdline_printf(cl, "%s\n", child[i].bdf); + free(child); + } else { + cmdline_printf(cl, "No memory\n"); + } + } else if (count == 0) { + cmdline_printf(cl, "No child\n"); + } else { + cmdline_printf(cl, "Failed\n"); + } +} + +cmdline_parse_token_string_t cmd_child_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_child_result, cmd, "get_child"); +cmdline_parse_token_string_t cmd_child_bdf = + TOKEN_STRING_INITIALIZER(struct cmd_child_result, bdf, NULL); + +cmdline_parse_inst_t cmd_get_child = { + .f = cmd_child_parsed, + .data = NULL, + .help_str = "get child PCI device of FPGA", + .tokens = { + (void *)&cmd_child_cmd, + (void *)&cmd_child_bdf, + NULL, + }, +}; + +/* *** GET PF1 *** */ +struct cmd_pf1_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t bdf; +}; + +static void cmd_pf1_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_pf1_result *res = parsed_result; + opae_pci_device id; + pcidev_id peer; + int i, count = 0; + + if (parse_pciaddr(res->bdf, &id) < 0) { + cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf); + return; + } + + count = opae_get_pf1(&id, NULL, 0); + if (count > 0) { + peer = (pcidev_id)malloc(sizeof(opae_pci_device) * count); + if (peer) { + opae_get_pf1(&id, peer, count); + for (i = 0; i < count; i++) + cmdline_printf(cl, "%s\n", peer[i].bdf); + free(peer); + } else { + cmdline_printf(cl, "No memory\n"); + } + } else if (count == 0) { + cmdline_printf(cl, "No PF1\n"); + } else { + cmdline_printf(cl, "Failed\n"); + } +} + +cmdline_parse_token_string_t cmd_pf1_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_pf1_result, cmd, "get_pf1"); +cmdline_parse_token_string_t cmd_pf1_bdf = + TOKEN_STRING_INITIALIZER(struct cmd_pf1_result, bdf, NULL); + +cmdline_parse_inst_t cmd_get_pf1 = { + .f = cmd_pf1_parsed, + .data = NULL, + .help_str = "get physical function 1 device of FPGA", + .tokens = { + (void *)&cmd_pf1_cmd, + (void *)&cmd_pf1_bdf, + NULL, + }, +}; + +/* *** SET LOG LEVEL *** */ +struct cmd_log_level_result { + cmdline_fixed_string_t cmd; + int32_t level; +}; + +static void cmd_log_level_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_log_level_result *res = parsed_result; + if (opae_set_log_level(res->level) == res->level) + cmdline_printf(cl, "Successful\n"); + else + cmdline_printf(cl, "Failed\n"); +} + +cmdline_parse_token_string_t cmd_log_level_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_log_level_result, cmd, "set_log_level"); +cmdline_parse_token_num_t cmd_log_level_level = + TOKEN_NUM_INITIALIZER(struct cmd_log_level_result, level, RTE_INT32); + +cmdline_parse_inst_t cmd_set_log_level = { + .f = cmd_log_level_parsed, + .data = NULL, + .help_str = "set logging level", + .tokens = { + (void *)&cmd_log_level_cmd, + (void *)&cmd_log_level_level, + NULL, + }, +}; + +/* *** SET LOG FILE *** */ +struct cmd_log_file_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t path; +}; + +static void cmd_log_file_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_log_file_result *res = parsed_result; + if (opae_set_log_file(res->path, 1) == 0) + cmdline_printf(cl, "Successful\n"); + else + cmdline_printf(cl, "Failed\n"); +} + +cmdline_parse_token_string_t cmd_log_file_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_log_file_result, cmd, "set_log_file"); +cmdline_parse_token_string_t cmd_log_file_path = + TOKEN_STRING_INITIALIZER(struct cmd_log_file_result, path, NULL); + +cmdline_parse_inst_t cmd_set_log_file = { + .f = cmd_log_file_parsed, + .data = NULL, + .help_str = "set logging file", + .tokens = { + (void *)&cmd_log_file_cmd, + (void *)&cmd_log_file_path, + NULL, + }, +}; + +/* *** SET STATUS *** */ +struct cmd_set_status_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t bdf; + uint32_t stat; + uint32_t prog; +}; + +static void cmd_set_status_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_set_status_result *res = parsed_result; + opae_pci_device id; + + if (parse_pciaddr(res->bdf, &id) < 0) { + cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf); + return; + } + if ((res->stat > 4) || (res->prog > 100)) { + cmdline_printf(cl, "%u,%u is invalid status\n", res->stat, + res->prog); + return; + } + + if (opae_store_rsu_status(&id, res->stat, res->prog) == 0) + cmdline_printf(cl, "Successful\n"); + else + cmdline_printf(cl, "Failed\n"); +} + +cmdline_parse_token_string_t cmd_set_status_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_set_status_result, cmd, "set_status"); +cmdline_parse_token_string_t cmd_set_status_bdf = + TOKEN_STRING_INITIALIZER(struct cmd_set_status_result, bdf, NULL); +cmdline_parse_token_num_t cmd_set_status_stat = + TOKEN_NUM_INITIALIZER(struct cmd_set_status_result, stat, RTE_UINT32); +cmdline_parse_token_num_t cmd_set_status_prog = + TOKEN_NUM_INITIALIZER(struct cmd_set_status_result, prog, RTE_UINT32); + +cmdline_parse_inst_t cmd_set_status = { + .f = cmd_set_status_parsed, + .data = NULL, + .help_str = "set current status & progress of FPGA", + .tokens = { + (void *)&cmd_set_status_cmd, + (void *)&cmd_set_status_bdf, + (void *)&cmd_set_status_stat, + (void *)&cmd_set_status_prog, + NULL, + }, +}; + +/* *** ENUMERATE *** */ +struct cmd_enumerate_result { + cmdline_fixed_string_t cmd; + uint32_t vid; + uint32_t did; +}; + +static void cmd_enumerate_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_enumerate_result *res = parsed_result; + opae_pci_id filter; + opae_pci_device *id; + int i, count = 0; + + filter.vendor_id = res->vid; + filter.device_id = res->did; + filter.class_id = BIT_SET_32; + filter.subsystem_vendor_id = BIT_SET_16; + filter.subsystem_device_id = BIT_SET_16; + + count = opae_enumerate(&filter, NULL, 0); + if (count > 0) { + id = (opae_pci_device *)malloc(sizeof(opae_pci_device) * count); + if (id) { + opae_enumerate(&filter, id, count); + for (i = 0; i < count; i++) + cmdline_printf(cl, "%s\n", id[i].bdf); + free(id); + } else { + cmdline_printf(cl, "No memory\n"); + } + } else if (count == 0) { + cmdline_printf(cl, "Not found\n"); + } else { + cmdline_printf(cl, "Failed\n"); + } +} + +cmdline_parse_token_string_t cmd_enumerate_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_enumerate_result, cmd, "enumerate"); +cmdline_parse_token_num_t cmd_enumerate_vid = + TOKEN_NUM_INITIALIZER(struct cmd_enumerate_result, vid, RTE_UINT32); +cmdline_parse_token_num_t cmd_enumerate_did = + TOKEN_NUM_INITIALIZER(struct cmd_enumerate_result, did, RTE_UINT32); + +cmdline_parse_inst_t cmd_enumerate = { + .f = cmd_enumerate_parsed, + .data = NULL, + .help_str = "enumerate specified FPGA", + .tokens = { + (void *)&cmd_enumerate_cmd, + (void *)&cmd_enumerate_vid, + (void *)&cmd_enumerate_did, + NULL, + }, +}; + +/* *** BIND *** */ +struct cmd_bind_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t bdf; + cmdline_fixed_string_t drv; +}; + +static void cmd_bind_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_bind_result *res = parsed_result; + opae_pci_device id; + + if (parse_pciaddr(res->bdf, &id) < 0) { + cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf); + return; + } + + if (opae_bind_driver(&id, res->drv) == 0) + cmdline_printf(cl, "Successful\n"); + else + cmdline_printf(cl, "Failed\n"); +} + +cmdline_parse_token_string_t cmd_bind_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_bind_result, cmd, "bind"); +cmdline_parse_token_string_t cmd_bind_bdf = + TOKEN_STRING_INITIALIZER(struct cmd_bind_result, bdf, NULL); +cmdline_parse_token_string_t cmd_bind_drv = + TOKEN_STRING_INITIALIZER(struct cmd_bind_result, drv, NULL); + +cmdline_parse_inst_t cmd_bind = { + .f = cmd_bind_parsed, + .data = NULL, + .help_str = "bind FPGA with kernel driver", + .tokens = { + (void *)&cmd_bind_cmd, + (void *)&cmd_bind_bdf, + (void *)&cmd_bind_drv, + NULL, + }, +}; + +/* *** UNBIND *** */ +struct cmd_unbind_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t bdf; +}; + +static void cmd_unbind_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_unbind_result *res = parsed_result; + opae_pci_device id; + + if (parse_pciaddr(res->bdf, &id) < 0) { + cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf); + return; + } + + if (opae_unbind_driver(&id) == 0) + cmdline_printf(cl, "Successful\n"); + else + cmdline_printf(cl, "Failed\n"); +} + +cmdline_parse_token_string_t cmd_unbind_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_bind_result, cmd, "unbind"); +cmdline_parse_token_string_t cmd_unbind_bdf = + TOKEN_STRING_INITIALIZER(struct cmd_bind_result, bdf, NULL); + +cmdline_parse_inst_t cmd_unbind = { + .f = cmd_unbind_parsed, + .data = NULL, + .help_str = "unbind FPGA from kernel driver", + .tokens = { + (void *)&cmd_unbind_cmd, + (void *)&cmd_unbind_bdf, + NULL, + }, +}; + +/* *** PROBE *** */ +struct cmd_probe_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t bdf; +}; + +static void cmd_probe_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_probe_result *res = parsed_result; + opae_pci_device id; + + if (parse_pciaddr(res->bdf, &id) < 0) { + cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf); + return; + } + + if (opae_probe_device(&id) == 0) + cmdline_printf(cl, "Successful\n"); + else + cmdline_printf(cl, "Failed\n"); +} + +cmdline_parse_token_string_t cmd_probe_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_probe_result, cmd, "probe"); +cmdline_parse_token_string_t cmd_probe_bdf = + TOKEN_STRING_INITIALIZER(struct cmd_probe_result, bdf, NULL); + +cmdline_parse_inst_t cmd_probe = { + .f = cmd_probe_parsed, + .data = NULL, + .help_str = "probe FPGA with IFPGA driver", + .tokens = { + (void *)&cmd_probe_cmd, + (void *)&cmd_probe_bdf, + NULL, + }, +}; + +/* *** REMOVE *** */ +struct cmd_remove_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t bdf; +}; + +static void cmd_remove_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_remove_result *res = parsed_result; + opae_pci_device id; + + if (parse_pciaddr(res->bdf, &id) < 0) { + cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf); + return; + } + + if (opae_remove_device(&id) == 0) + cmdline_printf(cl, "Successful\n"); + else + cmdline_printf(cl, "Failed\n"); +} + +cmdline_parse_token_string_t cmd_remove_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_remove_result, cmd, "remove"); +cmdline_parse_token_string_t cmd_remove_bdf = + TOKEN_STRING_INITIALIZER(struct cmd_remove_result, bdf, NULL); + +cmdline_parse_inst_t cmd_remove = { + .f = cmd_remove_parsed, + .data = NULL, + .help_str = "remove FPGA from IFPGA driver", + .tokens = { + (void *)&cmd_remove_cmd, + (void *)&cmd_remove_bdf, + NULL, + }, +}; + +/* *** FLASH *** */ +struct cmd_flash_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t bdf; + cmdline_fixed_string_t path; +}; + +static void cmd_flash_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_flash_result *res = parsed_result; + opae_pci_device id; + uint64_t stat = 0; + + if (parse_pciaddr(res->bdf, &id) < 0) { + cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf); + return; + } + + if (opae_update_flash(&id, res->path, &stat)) + cmdline_printf(cl, "Error: 0x%lx\n", (unsigned long)stat); +} + +cmdline_parse_token_string_t cmd_flash_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_flash_result, cmd, "flash"); +cmdline_parse_token_string_t cmd_flash_bdf = + TOKEN_STRING_INITIALIZER(struct cmd_flash_result, bdf, NULL); +cmdline_parse_token_string_t cmd_flash_path = + TOKEN_STRING_INITIALIZER(struct cmd_flash_result, path, NULL); + +cmdline_parse_inst_t cmd_flash = { + .f = cmd_flash_parsed, + .data = NULL, + .help_str = "update flash of FPGA", + .tokens = { + (void *)&cmd_flash_cmd, + (void *)&cmd_flash_bdf, + (void *)&cmd_flash_path, + NULL, + }, +}; + +/* *** PR *** */ +struct cmd_pr_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t bdf; + int32_t port; + cmdline_fixed_string_t path; +}; + +static void cmd_pr_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_pr_result *res = parsed_result; + opae_pci_device id; + + if (parse_pciaddr(res->bdf, &id) < 0) { + cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf); + return; + } + + if (opae_partial_reconfigure(&id, res->port, res->path) == 0) + cmdline_printf(cl, "Successful\n"); + else + cmdline_printf(cl, "Failed\n"); +} + +cmdline_parse_token_string_t cmd_pr_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_pr_result, cmd, "pr"); +cmdline_parse_token_string_t cmd_pr_bdf = + TOKEN_STRING_INITIALIZER(struct cmd_pr_result, bdf, NULL); +cmdline_parse_token_num_t cmd_pr_port = + TOKEN_NUM_INITIALIZER(struct cmd_pr_result, port, RTE_INT32); +cmdline_parse_token_string_t cmd_pr_path = + TOKEN_STRING_INITIALIZER(struct cmd_pr_result, path, NULL); + +cmdline_parse_inst_t cmd_pr = { + .f = cmd_pr_parsed, + .data = NULL, + .help_str = "partial reconfigure FPGA", + .tokens = { + (void *)&cmd_pr_cmd, + (void *)&cmd_pr_bdf, + (void *)&cmd_pr_port, + (void *)&cmd_pr_path, + NULL, + }, +}; + +/* *** REBOOT *** */ +struct cmd_reboot_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t bdf; + cmdline_fixed_string_t type; + int32_t page; +}; + +static void cmd_reboot_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_reboot_result *res = parsed_result; + opae_pci_device id; + int type = 0; + + if (parse_pciaddr(res->bdf, &id) < 0) { + cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf); + return; + } + + if (!strcmp(res->type, "fpga")) { + type = 0; + } else if (!strcmp(res->type, "bmc")) { + type = 1; + } else { + cmdline_printf(cl, "%s is invalid reboot type\n", res->type); + return; + } + + if (opae_reboot_device(&id, type, res->page) == 0) + cmdline_printf(cl, "Successful\n"); + else + cmdline_printf(cl, "Failed\n"); +} + +cmdline_parse_token_string_t cmd_reboot_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_reboot_result, cmd, "reboot"); +cmdline_parse_token_string_t cmd_reboot_bdf = + TOKEN_STRING_INITIALIZER(struct cmd_reboot_result, bdf, NULL); +cmdline_parse_token_string_t cmd_reboot_type = + TOKEN_STRING_INITIALIZER(struct cmd_reboot_result, type, NULL); +cmdline_parse_token_num_t cmd_reboot_page = + TOKEN_NUM_INITIALIZER(struct cmd_reboot_result, page, RTE_INT32); + +cmdline_parse_inst_t cmd_reboot = { + .f = cmd_reboot_parsed, + .data = NULL, + .help_str = "reboot FPGA or MAX10", + .tokens = { + (void *)&cmd_reboot_cmd, + (void *)&cmd_reboot_bdf, + (void *)&cmd_reboot_type, + (void *)&cmd_reboot_page, + NULL, + }, +}; + +/* *** CANCEL *** */ +struct cmd_cancel_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t bdf; +}; + +static void cmd_cancel_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_cancel_result *res = parsed_result; + opae_pci_device id; + + if (parse_pciaddr(res->bdf, &id) < 0) { + cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf); + return; + } + + if (opae_cancel_flash_update(&id, 0) == 0) + cmdline_printf(cl, "Successful\n"); + else + cmdline_printf(cl, "Failed\n"); +} + +cmdline_parse_token_string_t cmd_cancel_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_cancel_result, cmd, "cancel"); +cmdline_parse_token_string_t cmd_cancel_bdf = + TOKEN_STRING_INITIALIZER(struct cmd_cancel_result, bdf, NULL); + +cmdline_parse_inst_t cmd_cancel = { + .f = cmd_cancel_parsed, + .data = NULL, + .help_str = "cancel flash update", + .tokens = { + (void *)&cmd_cancel_cmd, + (void *)&cmd_cancel_bdf, + NULL, + }, +}; + +/* *** PCI READ *** */ +struct cmd_pci_read_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t bdf; + uint32_t offset; +}; + +static void cmd_pci_read_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_pci_read_result *res = parsed_result; + opae_pci_device id; + uint32_t offset = 0; + uint32_t value = 0; + + if (parse_pciaddr(res->bdf, &id) < 0) { + cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf); + return; + } + + if (res->offset & 0x3) { + offset = res->offset & ~3; + cmdline_printf(cl, "align offset to 0x%x\n", offset); + } else { + offset = res->offset; + } + + if (opae_read_pci_cfg(&id, offset, &value) == 0) + cmdline_printf(cl, "0x%08x\n", value); + else + cmdline_printf(cl, "Failed\n"); +} + +cmdline_parse_token_string_t cmd_pci_read_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_pci_read_result, cmd, "pci_read"); +cmdline_parse_token_string_t cmd_pci_read_bdf = + TOKEN_STRING_INITIALIZER(struct cmd_pci_read_result, bdf, NULL); +cmdline_parse_token_num_t cmd_pci_read_offset = + TOKEN_NUM_INITIALIZER(struct cmd_pci_read_result, offset, RTE_UINT32); + +cmdline_parse_inst_t cmd_pci_read = { + .f = cmd_pci_read_parsed, + .data = NULL, + .help_str = "read PCI configuration space", + .tokens = { + (void *)&cmd_pci_read_cmd, + (void *)&cmd_pci_read_bdf, + (void *)&cmd_pci_read_offset, + NULL, + }, +}; + +/* *** PCI WRITE *** */ +struct cmd_pci_write_result { + cmdline_fixed_string_t cmd; + cmdline_fixed_string_t bdf; + uint32_t offset; + uint32_t value; +}; + +static void cmd_pci_write_parsed(void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + struct cmd_pci_write_result *res = parsed_result; + opae_pci_device id; + uint32_t offset = 0; + + if (parse_pciaddr(res->bdf, &id) < 0) { + cmdline_printf(cl, "%s is invalid PCI address\n", res->bdf); + return; + } + + if (res->offset & 0x3) { + offset = res->offset & ~3; + cmdline_printf(cl, "align offset to 0x%x\n", offset); + } else { + offset = res->offset; + } + + if (opae_write_pci_cfg(&id, offset, res->value) == 0) + cmdline_printf(cl, "Successful\n"); + else + cmdline_printf(cl, "Failed\n"); +} + +cmdline_parse_token_string_t cmd_pci_write_cmd = + TOKEN_STRING_INITIALIZER(struct cmd_pci_write_result, cmd, "pci_write"); +cmdline_parse_token_string_t cmd_pci_write_bdf = + TOKEN_STRING_INITIALIZER(struct cmd_pci_write_result, bdf, NULL); +cmdline_parse_token_num_t cmd_pci_write_offset = + TOKEN_NUM_INITIALIZER(struct cmd_pci_write_result, offset, RTE_UINT32); +cmdline_parse_token_num_t cmd_pci_write_value = + TOKEN_NUM_INITIALIZER(struct cmd_pci_write_result, value, RTE_UINT32); + +cmdline_parse_inst_t cmd_pci_write = { + .f = cmd_pci_write_parsed, + .data = NULL, + .help_str = "write PCI configuration space", + .tokens = { + (void *)&cmd_pci_write_cmd, + (void *)&cmd_pci_write_bdf, + (void *)&cmd_pci_write_offset, + (void *)&cmd_pci_write_value, + NULL, + }, +}; + +/* *** QUIT *** */ +struct cmd_quit_result { + cmdline_fixed_string_t quit; +}; + +static void cmd_quit_parsed(__rte_unused void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + cmdline_quit(cl); +} + +cmdline_parse_token_string_t cmd_quit_quit = + TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit"); + +cmdline_parse_inst_t cmd_quit = { + .f = cmd_quit_parsed, + .data = NULL, + .help_str = "exit DPDK application", + .tokens = { + (void *)&cmd_quit_quit, + NULL, + }, +}; + +/* *** HELP *** */ +struct cmd_help_result { + cmdline_fixed_string_t help; +}; + +static void cmd_help_parsed(__rte_unused void *parsed_result, + struct cmdline *cl, __rte_unused void *data) +{ + cmdline_printf(cl, + " get_api_version \t\t" + "get OPAE API version\n" + " get_proc_type \t\t" + "get DPDK process type\n" + " get_image_info \t\t" + "get information of image file\n" + " get_status \t\t" + "get current status & progress of FPGA\n" + " get_property <0|1|2|4|8>\t\t" + "get property of FPGA\n" + " get_phy_info \t\t" + "get information of PHY\n" + " get_parent \t\t" + "get parent PCI device of FPGA\n" + " get_child \t\t" + "get child PCI device of FPGA\n" + " get_pf1 \t\t" + "get physical function 1 device of FPGA\n" + " set_log_level <0-4> \t\t" + "set logging level\n" + " set_log_file \t\t" + "set logging file\n" + " set_status <0-4> <0-100>\t\t" + "set current status & progress of FPGA\n" + " enumerate \t\t" + "enumerate specified FPGA\n" + " bind \t\t" + "bind FPGA with kernel driver\n" + " unbind \t\t" + "unbind FPGA from kernel driver\n" + " probe \t\t" + "probe FPGA with IFPGA driver\n" + " remove \t\t" + "remove FPGA from IFPGA driver\n" + " flash \t\t" + "update flash of FPGA\n" + " pr \t\t" + "partial reconfigure FPGA\n" + " reboot <0-1> \t\t" + "reboot FPGA or MAX10\n" + " cancel \t\t" + "cancel flash update\n" + " pci_read <0-1024> \t\t" + "read PCI configuration space\n" + " pci_write <0-1024> \t\t" + "write PCI configuration space\n" + " quit \t\t" + "exit DPDK application\n" + " help \t\t" + "show commands list\n"); +} + +cmdline_parse_token_string_t cmd_help_help = + TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help"); + +cmdline_parse_inst_t cmd_help = { + .f = cmd_help_parsed, + .data = NULL, + .help_str = "show commands list", + .tokens = { + (void *)&cmd_help_help, + NULL, + }, +}; + +/****** CONTEXT (list of commands) */ +cmdline_parse_ctx_t main_ctx[] = { + (cmdline_parse_inst_t *)&cmd_get_image_info, + (cmdline_parse_inst_t *)&cmd_get_api_version, + (cmdline_parse_inst_t *)&cmd_get_proc_type, + (cmdline_parse_inst_t *)&cmd_get_status, + (cmdline_parse_inst_t *)&cmd_get_property, + (cmdline_parse_inst_t *)&cmd_phy_info, + (cmdline_parse_inst_t *)&cmd_get_parent, + (cmdline_parse_inst_t *)&cmd_get_child, + (cmdline_parse_inst_t *)&cmd_get_pf1, + (cmdline_parse_inst_t *)&cmd_set_log_level, + (cmdline_parse_inst_t *)&cmd_set_log_file, + (cmdline_parse_inst_t *)&cmd_set_status, + (cmdline_parse_inst_t *)&cmd_enumerate, + (cmdline_parse_inst_t *)&cmd_bind, + (cmdline_parse_inst_t *)&cmd_unbind, + (cmdline_parse_inst_t *)&cmd_probe, + (cmdline_parse_inst_t *)&cmd_remove, + (cmdline_parse_inst_t *)&cmd_flash, + (cmdline_parse_inst_t *)&cmd_pr, + (cmdline_parse_inst_t *)&cmd_reboot, + (cmdline_parse_inst_t *)&cmd_cancel, + (cmdline_parse_inst_t *)&cmd_pci_read, + (cmdline_parse_inst_t *)&cmd_pci_write, + (cmdline_parse_inst_t *)&cmd_quit, + (cmdline_parse_inst_t *)&cmd_help, + NULL, +}; diff --git a/examples/ifpga/commands.h b/examples/ifpga/commands.h new file mode 100644 index 0000000000..06fe9a68b2 --- /dev/null +++ b/examples/ifpga/commands.h @@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2020-2021 Intel Corporation + */ + +#ifndef _COMMANDS_H_ +#define _COMMANDS_H_ + +#include "opae_api.h" + +extern cmdline_parse_ctx_t main_ctx[]; + +typedef struct { + char s[38]; +} uuid_str; + +#endif /* _COMMANDS_H_ */ diff --git a/examples/ifpga/main.c b/examples/ifpga/main.c new file mode 100644 index 0000000000..e9380d581a --- /dev/null +++ b/examples/ifpga/main.c @@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2020-2021 Intel Corporation + */ + +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include +#include +#include + +#include "commands.h" + + +int main(int argc, char **argv) +{ + struct cmdline *cl; + int ret; + + ret = opae_init_eal(argc, argv); + if (ret < 0) + rte_panic("Cannot init EAL\n"); + cl = cmdline_stdin_new(main_ctx, "opae> "); + if (cl == NULL) + rte_panic("Cannot create cmdline instance\n"); + cmdline_interact(cl); + opae_cleanup_eal(); + cmdline_stdin_exit(cl); + return 0; +} diff --git a/examples/ifpga/meson.build b/examples/ifpga/meson.build new file mode 100644 index 0000000000..b88a37a7a5 --- /dev/null +++ b/examples/ifpga/meson.build @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2020-2021 Intel Corporation + +# meson file, for building this example as part of a main DPDK build. +# +# To build this example as a standalone application with an already-installed +# DPDK instance, use 'make' + +# require the raw_ifpga library +build = dpdk_conf.has('RTE_RAW_IFPGA') +if not build + subdir_done() +endif + +deps += 'raw_ifpga' +allow_experimental_apis = true +sources = files( + 'main.c', 'commands.c', 'opae_api.c' +) +cflags += '-fPIC' diff --git a/examples/ifpga/opae_api.c b/examples/ifpga/opae_api.c new file mode 100644 index 0000000000..7c27a7a278 --- /dev/null +++ b/examples/ifpga/opae_api.c @@ -0,0 +1,1630 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2020 Intel Corporation + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "rte_pmd_ifpga.h" +#include "opae_api.h" + + +int opae_log_level; +FILE *opae_log_file; + +static opae_api_version api_ver = {21, 5, 0}; +static int eal_inited; +static uint32_t dev_aer[2] = {0}; + +static const char * const log_level_name[] = {"CRITICAL", "ERROR", + "WARNING", "INFORMATION", "DEBUG"}; +static const char * const proc_type_name[] = {"NON-DPDK", "PRIMARY", + "SECONDARY"}; +static const char * const platform_name[] = {"Vista Creek", "Rush Creek", + "Darby Creek", "Lightning Creek"}; +static const char * const release_name[] = {"Pre-Alpha", "Alpha", "Beta", "PV"}; +static const char * const interface_type[] = {"8x10G", "4x25G", "2x1x25G", + "4x25G+2x25G", "2x2x25G", "2x1x25Gx2FVL", "1x2x25G"}; + +void opae_get_api_version(opae_api_version *version) +{ + if (version) + memcpy(version, &api_ver, sizeof(opae_api_version)); + opae_log_info("API version is %u.%u.%u\n", + api_ver.major, api_ver.minor, api_ver.micro); +} + +int opae_set_log_level(int level) +{ + if ((level >= OPAE_LOG_API) && (level <= OPAE_LOG_DEBUG)) + opae_log_level = level; + opae_log_api("Current log level is %s\n", + log_level_name[opae_log_level]); + return opae_log_level; +} + +int opae_set_log_file(char *path, int clean) +{ + FILE *f = NULL; + time_t start; + struct tm *lt = NULL; + + if (path) { + if (clean) + f = fopen(path, "w+"); + else + f = fopen(path, "a+"); + + if (f) { + if (opae_log_file) { + fclose(opae_log_file); + opae_log_file = NULL; + } + time(&start); + lt = localtime(&start); + if (lt) + fprintf(f, "================%d-%02d-%02d " + "%02d:%02d:%02d================\n", + 1900 + lt->tm_year, 1 + lt->tm_mon, + lt->tm_mday, + lt->tm_hour, lt->tm_min, lt->tm_sec); + fflush(f); + opae_log_file = f; + } else { + opae_log_err("failed to open log file \'%s\'\n", path); + return -1; + } + } else { + if (opae_log_file) { + fclose(opae_log_file); + opae_log_file = NULL; + } + } + + return 0; +} + +int opae_get_image_info(const char *image, opae_img_info *info) +{ + int fd = -1; + off_t file_size = 0; + opae_img_hdr hdr; + ssize_t read_size = 0; + int ret = 0; + + if (!image || !info) { + opae_log_err("Input parameter of %s is invalid\n", __func__); + return -EINVAL; + } + + fd = open(image, O_RDONLY); + if (fd < 0) { + opae_log_err("Failed to open \'%s\' for RD [e:%s]\n", + image, strerror(errno)); + return -EIO; + } + + file_size = lseek(fd, 0, SEEK_END); + if (file_size < (OPAE_IMG_HDR_SIZE + OPAE_IMG_PL_MIN_SIZE)) { + opae_log_err("Size of \'%s\' is less than expected [e:%u]\n", + image, OPAE_IMG_HDR_SIZE + OPAE_IMG_PL_MIN_SIZE); + ret = -EINVAL; + goto close_fd; + } + + /* read image header */ + lseek(fd, 0, SEEK_SET); + read_size = read(fd, (void *)&hdr, sizeof(opae_img_hdr)); + if (read_size < 0) { + opae_log_err("Failed to read from \'%s\' [e:%s]\n", + image, strerror(errno)); + ret = -EIO; + goto close_fd; + } + if ((size_t)read_size != sizeof(opae_img_hdr)) { + opae_log_err("Read length %zd is not expected [e:%zu]\n", + read_size, sizeof(opae_img_hdr)); + ret = -EIO; + goto close_fd; + } + + info->total_len = file_size; + /* check signed image header */ + if (hdr.magic == OPAE_IMG_BLK0_MAGIC) { + info->type = OPAE_IMG_TYPE(hdr.payload_type); + info->subtype = OPAE_IMG_SUBTYPE(hdr.payload_type); + info->payload_offset = OPAE_IMG_HDR_SIZE; + info->payload_len = hdr.payload_len; + } else { + opae_log_err("Image \'%s\' can not be recognized\n", image); + ret = -EINVAL; + } +close_fd: + close(fd); + return ret; +} + +static int write_file(char *path, char *buf, int size) +{ + int fd = -1; + ssize_t n = 0; + + if (!path || !buf || (size <= 0)) + return -EINVAL; + + fd = open(path, O_WRONLY); + if (fd < 0) { + opae_log_err("Failed to open \'%s\' for WR [e:%s]\n", + path, strerror(errno)); + return -EIO; + } + opae_log_dbg("Write \"%s\" to \'%s\'\n", buf, path); + n = write(fd, buf, size); + if (n < size) { + opae_log_err("Failed to write to \'%s\' [e:%s]\n", + path, strerror(errno)); + close(fd); + return -EIO; + } + close(fd); + + return 0; +} + +static int read_file(char *path, char *buf, int size) +{ + int fd = -1; + ssize_t n = 0; + + if (!path || !buf || (size <= 0)) + return -EINVAL; + + fd = open(path, O_RDONLY); + if (fd < 0) { + opae_log_err("Failed to open \'%s\' for RD [e:%s]\n", + path, strerror(errno)); + return -EIO; + } + n = read(fd, buf, size); + if (n < 0) { + opae_log_err("Failed to read from \'%s\' [e:%s]\n", + path, strerror(errno)); + close(fd); + return -EIO; + } + close(fd); + + if (n > 0) + buf[n-1] = 0; + + opae_log_dbg("Read \"%s\" from \'%s\'\n", buf, path); + return 0; +} + +int opae_get_proc_type(void) +{ + int type = -1; + + if (eal_inited) { + if (rte_eal_process_type() == RTE_PROC_PRIMARY) + type = 0; + else + type = 1; + } + opae_log_info("Current process type is %s\n", proc_type_name[type+1]); + + return type; +} + +static bool check_eal(int inited) +{ + if (!eal_inited) { + if (inited) { + opae_log_warn("EAL is not initialized\n"); + return 0; + } + } else { + if (!inited) { + opae_log_warn("EAL is already initialized\n"); + return 0; + } + } + + return 1; +} + +int opae_init_eal(int argc, char **argv) +{ + int ret = 0; + + if (!check_eal(0)) + return ret; + + opae_log_level = OPAE_LOG_ERR; + + ret = rte_eal_init(argc, argv); + if (ret < 0) { + if (rte_errno == EALREADY) { + eal_inited = 1; + return 0; + } + opae_log_err("Cannot initialize EAL [e:%d]\n", ret); + if (rte_eal_cleanup()) + opae_log_warn("EAL could not release all resources\n"); + } else { + eal_inited = 1; + opae_log_info("Initialize EAL done\n"); + } + + return ret; +} + +int opae_cleanup_eal(void) +{ + int ret = 0; + + if (!check_eal(1)) + return -EPERM; + + rte_pmd_ifpga_cleanup(); + + ret = rte_eal_cleanup(); + if (ret) + opae_log_err("Failed to cleanup EAL [e:%d]\n", ret); + + if (opae_log_file) { + fclose(opae_log_file); + opae_log_file = NULL; + } + + return ret; +} + +static int compare_pci_id(opae_pci_id *id, opae_pci_id *expected_id) +{ + if ((expected_id->class_id != BIT_SET_32) && + (expected_id->class_id != id->class_id)) + return -1; + if ((expected_id->vendor_id != BIT_SET_16) && + (expected_id->vendor_id != id->vendor_id)) + return -1; + if ((expected_id->device_id != BIT_SET_16) && + (expected_id->device_id != id->device_id)) + return -1; + if ((expected_id->subsystem_vendor_id != BIT_SET_16) && + (expected_id->subsystem_vendor_id != id->subsystem_vendor_id)) + return -1; + if ((expected_id->subsystem_device_id != BIT_SET_16) && + (expected_id->subsystem_device_id != id->subsystem_device_id)) + return -1; + + return 0; +} + +static int parse_sysfs_value(char *node, uint32_t *val) +{ + char buf[16]; + char *end = NULL; + int ret = 0; + + ret = read_file(node, buf, sizeof(buf)); + if (ret < 0) + return ret; + + *val = (uint32_t)strtoul(buf, &end, 0); + return 0; +} + +static int get_pci_id(const char *dev_path, opae_pci_id *id) +{ + char path[PATH_MAX] = {0}; + uint32_t tmp; + + if (!dev_path || !id) + return -EINVAL; + + snprintf(path, sizeof(path), "%s/vendor", dev_path); + if (parse_sysfs_value(path, &tmp) < 0) + return -ENODEV; + id->vendor_id = (uint16_t)tmp; + + snprintf(path, sizeof(path), "%s/device", dev_path); + if (parse_sysfs_value(path, &tmp) < 0) + return -ENODEV; + id->device_id = (uint16_t)tmp; + + snprintf(path, sizeof(path), "%s/subsystem_vendor", dev_path); + if (parse_sysfs_value(path, &tmp) < 0) + return -ENODEV; + id->subsystem_vendor_id = (uint16_t)tmp; + + snprintf(path, sizeof(path), "%s/subsystem_device", dev_path); + if (parse_sysfs_value(path, &tmp) < 0) + return -ENODEV; + id->subsystem_device_id = (uint16_t)tmp; + + snprintf(path, sizeof(path), "%s/class", dev_path); + if (parse_sysfs_value(path, &tmp) < 0) + return -ENODEV; + id->class_id = (uint32_t)tmp & RTE_CLASS_ANY_ID; + + return 0; +} + +static int extract_path(char *in, int ridx, char *out, uint32_t size) +{ + char src[PATH_MAX] = {0}; + char *p = NULL; + int ret = 0; + + if (!in || (strlen(in) > PATH_MAX) || (ridx < 0) || !out) + return -EINVAL; + + strncpy(src, in, sizeof(src)); + *out = 0; + + while (1) { + p = strrchr(src, '/'); + if (p) { + *p++ = 0; + if (*p) { + if (ridx-- <= 0) { + if (size > strlen(p)) { + strncpy(out, p, size); + ret = strlen(p); + } + break; + } + } + } else { + break; + } + } + + return ret; +} + +int opae_enumerate(opae_pci_id *filter, pcidev_id list, int size) +{ + DIR *dir = NULL; + struct dirent *dirent = NULL; + char path[PATH_MAX] = {0}; + opae_pci_id id; + int n = 0; + + if (!filter || (size < 0) || (!list && (size > 0))) { + opae_log_err("Input parameter of %s is invalid\n", __func__); + return -EINVAL; + } + + dir = opendir(rte_pci_get_sysfs_path()); + if (!dir) { + opae_log_err("Failed to open \'%s\'\n", + rte_pci_get_sysfs_path()); + return -EINVAL; + } + while ((dirent = readdir(dir))) { + if (!strcmp(dirent->d_name, ".")) + continue; + if (!strcmp(dirent->d_name, "..")) + continue; + + snprintf(path, PATH_MAX, "%s/%s", rte_pci_get_sysfs_path(), + dirent->d_name); + if (get_pci_id(path, &id) < 0) + continue; + if (compare_pci_id(&id, filter) < 0) + continue; + + if (n++ < size) { + snprintf(list->bdf, sizeof(list->bdf), "%s", + dirent->d_name); + list++; + } + } + closedir(dir); + + return n; +} + +static int get_driver(pcidev_id id, char *drv_name, uint32_t size) +{ + char path[PATH_MAX] = {0}; + char link[PATH_MAX] = {0}; + int ret = 0; + + if (!id || !drv_name) { + ret = -EINVAL; + goto end; + } + size--; /* reserve one byte for the end of string */ + + snprintf(path, PATH_MAX, "%s/%s/driver", + rte_pci_get_sysfs_path(), id->bdf); + ret = readlink(path, link, PATH_MAX); + if (ret >= PATH_MAX) { + opae_log_err("Link path too long [%d]\n", ret); + ret = -ENAMETOOLONG; + goto end; + } + if (ret > 0) { + ret = extract_path(link, 0, drv_name, size); + } else { + *drv_name = 0; + opae_log_info("No link path for \'%s\'\n", path); + ret = 0; + } + +end: + if (ret < 0) + opae_log_err("Failed to get driver of %s\n", id->bdf); + + return ret; +} + +static int get_pci_addr(const char *bdf, opae_pci_addr *addr) +{ + struct rte_pci_addr pci_addr; + int ret = 0; + + if (!bdf || !addr) + return -EINVAL; + + ret = rte_pci_addr_parse(bdf, &pci_addr); + if (ret == 0) { + addr->domain = pci_addr.domain; + addr->bus = pci_addr.bus; + addr->devid = pci_addr.devid; + addr->function = pci_addr.function; + } + + return ret; +} + +static int check_pcidev_id(pcidev_id id) +{ + if (!id) { + opae_log_err("ID is NULL\n"); + return -1; + } + + if (strnlen(id->bdf, PCI_PRI_STR_SIZE) == PCI_PRI_STR_SIZE) { + opae_log_err("PCI address is too long\n"); + return -2; + } + + return 0; +} + +static int get_rawdev_id(pcidev_id id, uint16_t *dev_id, int log) +{ + if (check_pcidev_id(id)) + return -1; + + if (rte_pmd_ifpga_get_dev_id(id->bdf, dev_id)) { + if (log) + opae_log_warn("%s is not probed by ifpga driver\n", + id->bdf); + return -1; + } + + return 0; +} + +static struct rte_pci_device *get_rte_pcidev(pcidev_id id, int log) +{ + const struct rte_pci_bus *pci_bus = NULL; + struct rte_pci_device *pci_dev = NULL; + struct rte_pci_addr addr; + + if (check_pcidev_id(id)) + return NULL; + + if (rte_pci_addr_parse(id->bdf, &addr)) { + opae_log_err("PCI address %s is invalid\n", id->bdf); + return NULL; + } + + pci_bus = rte_pmd_ifpga_get_pci_bus(); + if (pci_bus) { + TAILQ_FOREACH(pci_dev, &pci_bus->device_list, next) { + if (!rte_pci_addr_cmp(&pci_dev->addr, &addr)) + return pci_dev; + } + } + + if (log) + opae_log_err("No rte_pci_device for %s\n", id->bdf); + + return NULL; +} + +int opae_load_rsu_status(pcidev_id id, uint32_t *status, uint32_t *progress) +{ + uint16_t dev_id = 0; + + if (!check_eal(1)) + return -EPERM; + + if (get_rawdev_id(id, &dev_id, 1)) + return -ENODEV; + + return rte_pmd_ifpga_get_rsu_status(dev_id, status, progress); +} + +int opae_store_rsu_status(pcidev_id id, uint32_t status, uint32_t progress) +{ + uint16_t dev_id = 0; + + if (!check_eal(1)) + return -EPERM; + + if (get_rawdev_id(id, &dev_id, 1)) + return -ENODEV; + + return rte_pmd_ifpga_set_rsu_status(dev_id, status, progress); +} + +static int get_pci_property(pcidev_id id, opae_pci_property *prop) +{ + char path[PATH_MAX] = {0}; + int ret = 0; + + if (!id || !prop) + return -EINVAL; + + snprintf(path, PATH_MAX, "%s/%s", rte_pci_get_sysfs_path(), id->bdf); + + ret = get_pci_id(path, &prop->id); + if (ret < 0) + return ret; + + ret = get_pci_addr(id->bdf, &prop->addr); + if (ret < 0) + return ret; + + snprintf(prop->pci_addr, OPAE_NAME_SIZE, "%s", id->bdf); + get_driver(id, prop->drv_name, sizeof(prop->drv_name)); + + return 0; +} + +static int get_fme_property(rte_pmd_ifpga_common_prop *common, + opae_fme_property *prop) +{ + opae_bitstream_id bbs_id; + + if (!common || !prop) + return -EINVAL; + + prop->boot_page = common->boot_page; + prop->num_ports = common->num_ports; + prop->bitstream_id = common->bitstream_id; + prop->bitstream_metadata = common->bitstream_metadata; + memcpy(prop->pr_id.b, common->pr_id.b, sizeof(opae_uuid)); + + bbs_id.id = prop->bitstream_id; + if (bbs_id.major < sizeof(platform_name) / sizeof(char *)) { + snprintf(prop->platform_name, + sizeof(prop->platform_name), "%s", + platform_name[bbs_id.major]); + } else { + snprintf(prop->platform_name, + sizeof(prop->platform_name), "unknown"); + } + + snprintf(prop->dcp_version, sizeof(prop->dcp_version), + "DCP 1.%u", bbs_id.minor); + + if (bbs_id.patch < sizeof(release_name)/sizeof(char *)) { + snprintf(prop->release_name, sizeof(prop->release_name), + "%s", release_name[bbs_id.patch]); + } else { + snprintf(prop->release_name, sizeof(prop->release_name), + "unknown"); + } + + if (bbs_id.major == 0) { /* Vista Creek */ + if (bbs_id.interface < + sizeof(interface_type) / sizeof(char *)) { + snprintf(prop->interface_type, + sizeof(prop->interface_type), "%s", + interface_type[bbs_id.interface]); + } else { + snprintf(prop->interface_type, + sizeof(prop->interface_type), "unknown"); + } + } else { + snprintf(prop->interface_type, + sizeof(prop->interface_type), "unknown"); + } + + snprintf(prop->build_version, sizeof(prop->build_version), + "%u.%u.%u", bbs_id.build_major, bbs_id.build_minor, + bbs_id.build_patch); + + return 0; +} + +static int get_port_property(rte_pmd_ifpga_port_prop *port, + opae_port_property *prop) +{ + if (!port || !prop) + return -EINVAL; + + memcpy(prop->afu_id.b, port->afu_id.b, sizeof(opae_uuid)); + prop->type = port->type; + + return 0; +} + +static int get_bmc_property(rte_pmd_ifpga_common_prop *common, + opae_bmc_property *prop) +{ + opae_bmc_version ver; + + if (!common || !prop) + return -EINVAL; + + ver.version = common->bmc_version; + snprintf(prop->bmc_version, sizeof(prop->bmc_version), "%c.%u.%u.%u", + ver.board, ver.major, ver.minor, ver.micro); + + ver.version = common->bmc_nios_version; + snprintf(prop->fw_version, sizeof(prop->fw_version), "%c.%u.%u.%u", + ver.board, ver.major, ver.minor, ver.micro); + + return 0; +} + +int opae_get_property(pcidev_id id, opae_fpga_property *prop, int type) +{ + uint16_t dev_id = 0; + uint32_t i = 0; + rte_pmd_ifpga_prop fpga_prop; + int ret = 0; + + if (!prop) { + opae_log_err("Input parameter of %s is invalid\n", __func__); + return -EINVAL; + } + + if (type == 0) + type = OPAE_PROP_ALL; + + memset(prop, 0, sizeof(opae_fpga_property)); + + /* PCI properties */ + if (type & OPAE_PROP_PCI) { + if (get_pci_property(id, &prop->pci) < 0) { + opae_log_err("Failed to get PCI property\n"); + return -EAGAIN; + } + } + + if (type == OPAE_PROP_PCI) + return 0; + + if (!check_eal(1)) + return -EPERM; + + if (get_rawdev_id(id, &dev_id, 1)) + return -ENODEV; + + ret = rte_pmd_ifpga_get_property(dev_id, &fpga_prop); + if (ret) { + opae_log_err("Failed to get FPGA property [e:%d]\n", ret); + return -EAGAIN; + } + + /* FME properties */ + if (type & (OPAE_PROP_FME | OPAE_PROP_PORT)) + get_fme_property(&fpga_prop.common, &prop->fme); + + /* PORT properties */ + if (type & OPAE_PROP_PORT) { + for (i = 0; i < prop->fme.num_ports; i++) { + prop->port[i].index = i; + get_port_property(&fpga_prop.port[i], &prop->port[i]); + } + } + + /* BMC properties */ + if (type & OPAE_PROP_BMC) + get_bmc_property(&fpga_prop.common, &prop->bmc); + + return 0; +} + +int opae_get_phy_info(pcidev_id id, opae_phy_info *info) +{ + uint16_t dev_id = 0; + rte_pmd_ifpga_phy_info phy_info; + int ret = 0; + + if (!info) + return -EINVAL; + + if (get_rawdev_id(id, &dev_id, 1)) + return -ENODEV; + + ret = rte_pmd_ifpga_get_phy_info(dev_id, &phy_info); + if (ret) { + opae_log_err("Failed to get PHY information [e:%d]\n", ret); + return -EAGAIN; + } + + info->num_retimers = phy_info.num_retimers; + info->link_speed = phy_info.link_speed; + info->link_status = phy_info.link_status; + + return 0; +} + +static int update_driver(pcidev_id id, char *drv_name) +{ + struct rte_pci_device *pci_dev = NULL; + char name[OPAE_NAME_SIZE] = {0}; + int ret = 0; + + if (check_pcidev_id(id)) + return -EINVAL; + + if (drv_name) { + if (strlen(drv_name) >= OPAE_NAME_SIZE) { + opae_log_err("Driver name \'%s\' too long\n", + drv_name); + return -EINVAL; + } + strncpy(name, drv_name, sizeof(name)); + } else { + ret = get_driver(id, name, sizeof(name)); + if (ret < 0) + return ret; + } + + pci_dev = get_rte_pcidev(id, 0); + if (pci_dev) { + if (strlen(name) == 0) { + pci_dev->kdrv = RTE_PCI_KDRV_NONE; + } else { + if (!strcmp(name, OPAE_KDRV_VFIO_PCI)) + pci_dev->kdrv = RTE_PCI_KDRV_VFIO; + else if (!strcmp(name, OPAE_KDRV_IGB_UIO)) + pci_dev->kdrv = RTE_PCI_KDRV_IGB_UIO; + else if (!strcmp(name, OPAE_KDRV_UIO_PCI)) + pci_dev->kdrv = RTE_PCI_KDRV_UIO_GENERIC; + else + pci_dev->kdrv = RTE_PCI_KDRV_UNKNOWN; + } + } + + return 0; +} + +int opae_unbind_driver(pcidev_id id) +{ + uint16_t dev_id = 0; + char path[PATH_MAX] = {0}; + char drv_name[OPAE_NAME_SIZE] = {0}; + char null[] = {0}; + int ret = 0; + + if (!get_rawdev_id(id, &dev_id, 0)) { + opae_log_err("%s is probed, remove it first\n", id->bdf); + return -EBUSY; + } + + ret = get_driver(id, drv_name, sizeof(drv_name)); + if (ret < 0) + return ret; + + if (strlen(drv_name) > 0) { + snprintf(path, PATH_MAX, "/sys/bus/pci/drivers/%s/unbind", + drv_name); + ret = write_file(path, id->bdf, strlen(id->bdf) + 1); + if (ret == 0) + ret = update_driver(id, null); + } + + return ret; +} + +static int check_driver(const char *drv_name) +{ + char path[PATH_MAX] = {0}; + struct stat buf; + + if (!drv_name) + return -EINVAL; + + if (strlen(drv_name) > 0) { + snprintf(path, PATH_MAX, "/sys/bus/pci/drivers/%s", drv_name); + if ((stat(path, &buf) < 0) || ((buf.st_mode & S_IFDIR) == 0)) { + opae_log_warn("Driver %s is not installed\n", + drv_name); + return -EINVAL; + } + } + + return 0; +} + +int opae_bind_driver(pcidev_id id, char *drv_name) +{ + char path[PATH_MAX] = {0}; + char name[OPAE_NAME_SIZE] = {0}; + char null[] = {0}; + int ret = 0; + + ret = check_driver(drv_name); + if (ret < 0) + return ret; + + ret = get_driver(id, name, sizeof(name)); + if (ret < 0) + return ret; + + if (!strcmp(drv_name, name)) /* driver not change */ + return 0; + + ret = opae_unbind_driver(id); + if (ret < 0) + return ret; + + if (strlen(drv_name) > 0) { + /* bind driver */ + snprintf(path, PATH_MAX, "%s/%s/driver_override", + rte_pci_get_sysfs_path(), id->bdf); + ret = write_file(path, drv_name, strlen(drv_name) + 1); + if (ret < 0) + goto update_drv; + + snprintf(path, PATH_MAX, "/sys/bus/pci/drivers/%s/bind", + drv_name); + ret = write_file(path, id->bdf, strlen(id->bdf) + 1); + if (ret < 0) + goto update_drv; + + snprintf(path, PATH_MAX, "%s/%s/driver_override", + rte_pci_get_sysfs_path(), id->bdf); + ret = write_file(path, null, 1); + if (ret < 0) + goto update_drv; + } + +update_drv: + ret = update_driver(id, NULL); + if (ret < 0) + opae_log_err("Failed to update driver information of %s\n", + id->bdf); + + return 0; +} + +int opae_probe_device(pcidev_id id) +{ + const struct rte_pci_bus *pci_bus = NULL; + struct rte_pci_device *pci_dev = NULL; + + if (check_pcidev_id(id)) + return -EINVAL; + + if (!check_eal(1)) + return -EPERM; + + /* make sure device is added in rte_pci_bus devices list */ + pci_bus = rte_pmd_ifpga_get_pci_bus(); + if (pci_bus && pci_bus->bus.scan) + pci_bus->bus.scan(); + + pci_dev = get_rte_pcidev(id, 1); + if (!pci_dev) + return -ENODEV; + + if (pci_dev->kdrv != RTE_PCI_KDRV_VFIO) { + opae_log_err("vfio-pci driver is not bound to %s\n", id->bdf); + return -EINVAL; + } + + if (!pci_bus || !pci_bus->bus.plug) + return -ENODEV; + + return pci_bus->bus.plug(&pci_dev->device); +} + +int opae_remove_device(pcidev_id id) +{ + struct rte_pci_device *pci_dev = NULL; + struct rte_pci_driver *pci_drv = NULL; + int ret = 0; + + if (!check_eal(1)) + return -EPERM; + + pci_dev = get_rte_pcidev(id, 0); + if (pci_dev && pci_dev->driver) { + pci_drv = pci_dev->driver; + ret = pci_drv->remove(pci_dev); + if (ret < 0) { + opae_log_err("Failed to remove %s [e:%d]\n", + id->bdf, ret); + return ret; + } + pci_dev->driver = NULL; + pci_dev->device.driver = NULL; + if (pci_drv->drv_flags & RTE_PCI_DRV_NEED_MAPPING) + rte_pci_unmap_device(pci_dev); + } + + return ret; +} + +static int is_pac(pcidev_id id) +{ + char path[PATH_MAX] = {0}; + opae_pci_id pci_id; + + if (check_pcidev_id(id)) + return 0; + + snprintf(path, PATH_MAX, "%s/%s", rte_pci_get_sysfs_path(), id->bdf); + if (get_pci_id(path, &pci_id) < 0) + return 0; + + if ((pci_id.vendor_id == 0x8086) && (pci_id.device_id == 0x0b30)) + return 1; + + return 0; +} + +int opae_get_parent(pcidev_id id, pcidev_id parent) +{ + char path[PATH_MAX] = {0}; + char link[PATH_MAX] = {0}; + int ret = 0; + + if (!id || !parent) { + opae_log_err("Input parameter of %s is invalid\n", __func__); + ret = -EINVAL; + goto end; + } + + snprintf(path, PATH_MAX, "%s/%s", rte_pci_get_sysfs_path(), id->bdf); + ret = readlink(path, link, PATH_MAX); + if (ret >= PATH_MAX) { + opae_log_err("Length of link path exceeds %u\n", PATH_MAX); + ret = -ENAMETOOLONG; + goto end; + } + + if (ret > 0) { + ret = extract_path(link, 1, parent->bdf, sizeof(parent->bdf)); + if (!strncmp(parent->bdf, "pci", 3)) { + parent->bdf[0] = 0; + ret = -ENODEV; + } + } else { + parent->bdf[0] = 0; + if (ret == 0) + opae_log_err("Length of link path is 0\n"); + else + opae_log_err("No link path for \'%s\'\n", path); + } +end: + if (ret <= 0) + opae_log_err("%s has no parent\n", id->bdf); + + return ret; +} + +int opae_get_child(pcidev_id id, pcidev_id child, int size) +{ + glob_t pglob = {.gl_pathc = 0, .gl_pathv = NULL}; + char path[PATH_MAX] = {0}; + int i, count = 0; + int len = 0; + int ret = 0; + + if (!id || (size < 0)) { + opae_log_err("Input parameter of %s is invalid\n", __func__); + return -EINVAL; + } + + snprintf(path, PATH_MAX, "%s/%s/*:*:*.?", rte_pci_get_sysfs_path(), + id->bdf); + ret = glob(path, 0, NULL, &pglob); + if (ret == 0) { + if (child && (size > 0)) { + for (i = 0; i < (int)pglob.gl_pathc; i++) { + len = extract_path(pglob.gl_pathv[i], 0, + child->bdf, sizeof(child->bdf)); + if (len <= 0) { + child->bdf[0] = 0; + continue; + } + if (++count >= size) + break; + child++; + } + } else { + count = (int)pglob.gl_pathc; + } + globfree(&pglob); + } else { + if (pglob.gl_pathv) + globfree(&pglob); + } + + return count; +} + +int opae_get_pf1(pcidev_id id, pcidev_id peer, int size) +{ + opae_pci_device parent; + opae_pci_device child[4]; + int n = 0; + int ret = 0; + + if (check_pcidev_id(id)) + return -EINVAL; + + if (!is_pac(id)) { + opae_log_info("%s has no peer function\n", id->bdf); + return -EINVAL; + } + + ret = opae_get_parent(id, &parent); + if (ret < 0) + return -ENODEV; + ret = opae_get_parent(&parent, &parent); + if (ret < 0) + return -ENODEV; + + n = opae_get_child(&parent, child, + sizeof(child) / sizeof(opae_pci_device)); + /* there should have four downstream ports of PCI switch on board */ + if (n == 4) { + n = opae_get_child(&child[3], peer, size); + } else { + peer->bdf[0] = 0; + opae_log_dbg("%s has %d child(s)\n", parent.bdf, n); + n = 0; + } + + return n; +} + +int opae_update_flash(pcidev_id id, const char *image, uint64_t *status) +{ + opae_img_info info; + uint16_t dev_id = 0; + int ret = 0; + + ret = opae_get_image_info(image, &info); + if (ret < 0) { + opae_log_err("Failed to get image information [e:%d]\n", ret); + return -EINVAL; + } + + if ((info.type != OPAE_IMG_TYPE_BBS) && + (info.type != OPAE_IMG_TYPE_BMC)) { + opae_log_err("Image is not supported [t:%u]\n", info.type); + return -EOPNOTSUPP; + } + + if (!check_eal(1)) + return -EPERM; + + if (get_rawdev_id(id, &dev_id, 1)) + return -ENODEV; + + return rte_pmd_ifpga_update_flash(dev_id, image, status); +} + +int opae_cancel_flash_update(pcidev_id id, int force) +{ + uint16_t dev_id = 0; + + if (!check_eal(1)) + return -EPERM; + + if (get_rawdev_id(id, &dev_id, 1)) + return -ENODEV; + + return rte_pmd_ifpga_stop_update(dev_id, force); +} + +static int find_pci_ecap(int fd, unsigned int cap) +{ + uint32_t header = 0; + int ttl = (RTE_PCI_CFG_SPACE_EXP_SIZE - RTE_PCI_CFG_SPACE_SIZE) / 8; + int pos = RTE_PCI_CFG_SPACE_SIZE; + int ret = 0; + + ret = pread(fd, &header, sizeof(header), pos); + if (ret < 0) { + opae_log_err("Failed to read from PCI configuration space [e:%s]\n", + strerror(errno)); + return ret; + } + opae_log_dbg("Read 0x%08x from PCI configuration space 0x%x\n", + header, pos); + + if (header == 0) { + opae_log_err("Capability is empty\n"); + return 0; + } + + while (ttl-- > 0) { + if ((RTE_PCI_EXT_CAP_ID(header) == cap) && (pos != 0)) + return pos; + + pos = RTE_PCI_EXT_CAP_NEXT(header); + if (pos < RTE_PCI_CFG_SPACE_SIZE) { + opae_log_err("Position of capability is invalid" + "[e:%d]\n", pos); + break; + } + ret = pread(fd, &header, sizeof(header), pos); + if (ret < 0) { + opae_log_err("Failed to read from PCI config space [e:%s]\n", + strerror(errno)); + return ret; + } + opae_log_dbg("Read 0x%08x from PCI configuration space 0x%x\n", + header, pos); + } + + return 0; +} + +static int set_aer(pcidev_id id, uint32_t v1, uint32_t v2, int record) +{ + char path[PATH_MAX] = {0}; + uint32_t val = 0; + int fd = -1; + int pos = 0; + int ret = 0; + + if (!id) + return -EINVAL; + + snprintf(path, PATH_MAX, "%s/%s/config", + rte_pci_get_sysfs_path(), id->bdf); + fd = open(path, O_RDWR); + if (fd < 0) { + opae_log_err("Failed to open \'%s\' for RDWR [e:%s]\n", + path, strerror(errno)); + return -EIO; + } + + pos = find_pci_ecap(fd, RTE_PCI_EXT_CAP_ID_ERR); + if (pos <= 0) { + opae_log_warn("AER capability is not present\n"); + ret = -ENXIO; + goto close_fd; + } + + if (record) { + ret = pread(fd, &val, sizeof(val), pos + 0x08); + if (ret < 0) { + opae_log_err("Failed to read from PCI config space [e:%s]\n", + strerror(errno)); + goto close_fd; + } + opae_log_dbg("Read 0x%08x from PCI configuration space 0x%x\n", + val, pos + 0x08); + dev_aer[0] = val; + + ret = pread(fd, &val, sizeof(val), pos + 0x14); + if (ret < 0) { + opae_log_err("Failed to read from PCI config space [e:%s]\n", + strerror(errno)); + goto close_fd; + } + opae_log_dbg("Read 0x%08x from PCI configuration space 0x%x\n", + val, pos + 0x14); + dev_aer[1] = val; + } + + opae_log_dbg("Write 0x%08x to PCI configuration space 0x%x\n", + v1, pos + 0x08); + ret = pwrite(fd, &v1, sizeof(v1), pos + 0x08); + if (ret < 0) { + opae_log_err("Failed to write to PCI config space 0x%x [e:%s]\n", + pos + 0x08, strerror(errno)); + goto close_fd; + } + + opae_log_dbg("Write 0x%08x to PCI configuration space 0x%x\n", + v2, pos + 0x14); + ret = pwrite(fd, &v2, sizeof(v2), pos + 0x14); + if (ret < 0) { + opae_log_err("Failed to write to PCI config space 0x%x [e:%s]\n", + pos + 0x14, strerror(errno)); + } + +close_fd: + close(fd); + return ret < 0 ? ret : 0; +} + +static int enable_aer(pcidev_id id) +{ + if (check_pcidev_id(id)) + return -EINVAL; + + opae_log_info("Enable AER of %s\n", id->bdf); + + return set_aer(id, dev_aer[0], dev_aer[1], 0); +} + +static int disable_aer(pcidev_id id) +{ + if (check_pcidev_id(id)) + return -EINVAL; + + opae_log_info("Disable AER of %s\n", id->bdf); + + return set_aer(id, 0xffffffff, 0xffffffff, 1); +} + +static int remove_tree(pcidev_id id) +{ + int i, n = 0; + pcidev_id child; + int ret = 0; + + if (check_pcidev_id(id)) + return -EINVAL; + + n = opae_get_child(id, NULL, 0); + if (n > 0) { + child = (pcidev_id)rte_zmalloc(NULL, + sizeof(opae_pci_device) * n, 0); + if (!child) { + opae_log_err("Failed to malloc for children of %s\n", + id->bdf); + ret = -ENOMEM; + goto end; + } + + opae_get_child(id, child, n); + for (i = 0; i < n; i++) + remove_tree(&child[i]); + rte_free(child); + } + +end: + opae_remove_device(id); + return ret; +} + +static int remove_device(pcidev_id id) +{ + char path[PATH_MAX] = {0}; + char one[] = {'1', 0}; + int ret = 0; + + if (check_pcidev_id(id)) + return -EINVAL; + + opae_log_info("Remove %s from system\n", id->bdf); + + snprintf(path, PATH_MAX, "%s/%s/remove", + rte_pci_get_sysfs_path(), id->bdf); + ret = write_file(path, one, strlen(one)); + if (ret < 0) { + opae_log_err("Failed to remove %s from system\n", id->bdf); + return ret; + } + + remove_tree(id); + + return 0; +} + +static int scan_device(pcidev_id parent, pcidev_id id) +{ + char path[PATH_MAX] = {0}; + char bus[8] = {0}; + char one[] = {'1', 0}; + char pwr[16] = {0}; + char pwr_on[] = {'o', 'n', 0}; + int pwr_on_failed = 0; + int ret = 0; + + if (!parent) { + opae_log_err("Input parameter of %s is invalid\n", __func__); + return -EINVAL; + } + opae_log_info("Rescan devices under %s\n", parent->bdf); + + if (id) { /* scan specified bus under parent device */ + snprintf(path, PATH_MAX, "%s/%s/power/control", + rte_pci_get_sysfs_path(), parent->bdf); + ret = read_file(path, pwr, sizeof(pwr)); + if (ret < 0) + return ret; + + if (strcmp(pwr, "on")) { + ret = write_file(path, pwr_on, strlen(pwr_on)); + if (ret < 0) + pwr_on_failed = 1; + else + sleep(1); + } + + snprintf(bus, sizeof(bus), "%s", id->bdf); + snprintf(path, PATH_MAX, "%s/%s/pci_bus/%s/rescan", + rte_pci_get_sysfs_path(), parent->bdf, bus); + ret = write_file(path, one, strlen(one)); + if (ret < 0) + return ret; + + if (pwr_on_failed) { /* workaround for power on failed */ + ret = write_file(path, one, strlen(one)); + if (ret < 0) + return ret; + } + + if (strcmp(pwr, "on")) { + snprintf(path, PATH_MAX, "%s/%s/power/control", + rte_pci_get_sysfs_path(), parent->bdf); + ret = write_file(path, pwr, strlen(pwr)); + } + } else { /* scan all buses under parent device */ + snprintf(path, PATH_MAX, "%s/%s/rescan", + rte_pci_get_sysfs_path(), parent->bdf); + ret = write_file(path, one, strlen(one)); + } + + return ret; +} + +int opae_reboot_device(pcidev_id id, int type, int page) +{ + uint16_t dev_id = 0; + opae_pci_device fpga; /* FPGA after reboot */ + opae_pci_device parent; + opae_pci_device peer[2]; /* physical function 1 of FPGA */ + opae_pci_device peer_parent; + opae_pci_device ups; /* upstream port device */ + opae_pci_device root; /* port connected to PAC */ + pcidev_id peer_primary = NULL; + char drv_name[OPAE_NAME_SIZE] = {0}; + int n = 0; + int i = 0; + int ret = 0; + + if (check_pcidev_id(id)) + return -EINVAL; + + if (!is_pac(id)) { + opae_log_err("%s can not be rebooted\n", id->bdf); + return -EINVAL; + } + + if (get_rawdev_id(id, &dev_id, 1)) + return -ENODEV; + + ret = opae_get_parent(id, &parent); + if (ret < 0) + return -ENODEV; + ret = opae_get_parent(&parent, &ups); + if (ret < 0) + return -ENODEV; + ret = opae_get_parent(&ups, &root); + if (ret < 0) + return -ENODEV; + + n = opae_get_pf1(id, peer, sizeof(peer) / sizeof(opae_pci_device)); + if (n <= 0) { + opae_log_err("PF1 of %s is not found\n", id->bdf); + } else { + peer_primary = &peer[0]; + ret = opae_get_parent(peer_primary, &peer_parent); + if (ret < 0) + return -ENODEV; + } + + get_driver(id, drv_name, sizeof(drv_name)); /* save original driver */ + + if (!check_eal(1)) + return -EPERM; + + if (rte_pmd_ifpga_reboot_try(dev_id)) { + opae_log_warn("Update or reboot is in progress\n"); + return -EAGAIN; + } + + if (type == 0) { + /* disable AER */ + ret = disable_aer(&parent); + if (ret < 0) { + opae_log_err("Failed to disable AER of %s\n", + parent.bdf); + goto reboot_end; + } + ret = disable_aer(&peer_parent); + if (ret < 0) { + opae_log_err("Failed to disable AER of %s\n", + peer_parent.bdf); + goto reboot_end; + } + + /* trigger reconfiguration */ + ret = rte_pmd_ifpga_reload(dev_id, type, page); + if (ret == 0) { + ret = remove_device(id); + for (i = 0; i < n; i++) + ret += remove_device(&peer[i]); + if (ret == 0) { + opae_log_info("Wait 10 seconds for FPGA reloading\n"); + sleep(10); + ret = scan_device(&parent, id); + if (ret < 0) + opae_log_err("Failed to rescan %s\n", + id->bdf); + if (peer_primary) { + ret = scan_device(&peer_parent, + peer_primary); + if (ret < 0) { + opae_log_err("Failed to rescan %s\n", + peer_primary->bdf); + } + } + } + } + + /* restore AER */ + if (enable_aer(&parent) < 0) { + opae_log_err("Failed to enable AER of %s\n", + parent.bdf); + } + if (enable_aer(&peer_parent) < 0) { + opae_log_err("Failed to enable AER of %s\n", + peer_parent.bdf); + } + } else if (type == 1) { + /* disable AER */ + ret = disable_aer(&root); + if (ret < 0) { + opae_log_err("Failed to disable AER of %s\n", root.bdf); + goto reboot_end; + } + + /* trigger reconfiguration */ + ret = rte_pmd_ifpga_reload(dev_id, type, page); + if (ret == 0) { + ret += remove_device(&ups); + if (ret == 0) { + opae_log_info("Wait 10 seconds for BMC reloading\n"); + sleep(10); + ret = scan_device(&root, &ups); + if (ret < 0) + opae_log_err("Failed to rescan %s\n", + ups.bdf); + } + } + + /* restore AER */ + if (enable_aer(&root) < 0) + opae_log_err("Failed to enable AER of %s\n", root.bdf); + } else { + opae_log_err("Type of reboot is not supported [t:%d]\n", type); + ret = -EINVAL; + goto reboot_end; + } + + /* update id if bdf changed after reboot */ + if (opae_get_child(&parent, &fpga, 1) == 1) { + if (strcmp(id->bdf, fpga.bdf)) + id = &fpga; + } + + ret = opae_bind_driver(id, drv_name); + if (ret < 0) + opae_log_err("Failed to bind original driver of %s\n", id->bdf); + + ret = opae_probe_device(id); + if (ret < 0) + opae_log_err("Failed to probe %s [e:%d]\n", id->bdf, ret); + +reboot_end: + rte_pmd_ifpga_set_rsu_status(dev_id, 0, 0); + return ret; +} + +int opae_partial_reconfigure(pcidev_id id, int port, const char *gbs) +{ + uint16_t dev_id = 0; + + if (!id || !gbs) { + opae_log_err("Input parameter of %s is invalid\n", __func__); + return -EINVAL; + } + + if (!check_eal(1)) + return -EPERM; + + if (get_rawdev_id(id, &dev_id, 1)) + return -ENODEV; + + return rte_pmd_ifpga_partial_reconfigure(dev_id, port, gbs); +} + +int opae_read_pci_cfg(pcidev_id id, uint32_t address, uint32_t *value) +{ + char path[PATH_MAX] = {0}; + int fd = -1; + int ret = 0; + + if (!id || !value) { + opae_log_err("Input parameter of %s is invalid\n", __func__); + return -EINVAL; + } + + snprintf(path, PATH_MAX, "%s/%s/config", rte_pci_get_sysfs_path(), + id->bdf); + fd = open(path, O_RDONLY); + if (fd < 0) { + opae_log_dbg("Failed to open \'%s\' for RDONLY [e:%s]\n", + path, strerror(errno)); + return -EIO; + } + + ret = pread(fd, value, 4, address); + if (ret < 0) { + opae_log_err("Failed to read from PCI device %s [e:%s]\n", + id->bdf, strerror(errno)); + close(fd); + return ret; + } + + opae_log_dbg("CONFIG+0x%08x -> 0x%08x\n", address, *value); + close(fd); + return 0; +} + +int opae_write_pci_cfg(pcidev_id id, uint32_t address, uint32_t value) +{ + char path[PATH_MAX] = {0}; + int fd = -1; + int ret = 0; + + if (!id) { + opae_log_err("Input parameter of %s is invalid\n", __func__); + return -EINVAL; + } + + snprintf(path, PATH_MAX, "%s/%s/config", rte_pci_get_sysfs_path(), + id->bdf); + fd = open(path, O_WRONLY); + if (fd < 0) { + opae_log_dbg("Failed to open \'%s\' for WRONLY [e:%s]\n", + path, strerror(errno)); + return -EIO; + } + + ret = pwrite(fd, &value, 4, address); + if (ret < 0) { + opae_log_err("Failed to write to PCI device %s [e:%s]\n", + id->bdf, strerror(errno)); + close(fd); + return ret; + } + + opae_log_dbg("CONFIG+0x%08x <- 0x%08x\n", address, value); + close(fd); + return 0; +} diff --git a/examples/ifpga/opae_api.h b/examples/ifpga/opae_api.h new file mode 100644 index 0000000000..3b6bc01fbc --- /dev/null +++ b/examples/ifpga/opae_api.h @@ -0,0 +1,244 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2020 Intel Corporation + */ + +#ifndef _OPAE_API_H +#define _OPAE_API_H + + +#ifdef __cplusplus +extern "C" { +#endif + +#include + +extern int opae_log_level; +extern FILE *opae_log_file; + +#define OPAE_LOG_API 0 /**< Critical conditions. */ +#define OPAE_LOG_ERR 1 /**< Error conditions. */ +#define OPAE_LOG_WARN 2 /**< Warning conditions. */ +#define OPAE_LOG_INFO 3 /**< Informational. */ +#define OPAE_LOG_DEBUG 4 /**< Debug-level messages. */ + +#define opae_log(type, fmt, args...) \ +do { \ + if (opae_log_level >= OPAE_LOG_##type) { \ + printf(fmt, ##args); \ + if (opae_log_file) { \ + fprintf(opae_log_file, fmt, ##args); \ + fflush(opae_log_file); \ + } \ + } \ +} while (0) + +#define opae_log_api(fmt, args...) opae_log(API, "OPAE-API: "fmt, ##args) +#define opae_log_err(fmt, args...) opae_log(ERR, "OPAE-ERR: "fmt, ##args) +#define opae_log_dbg(fmt, args...) opae_log(DEBUG, "OPAE-DBG: "fmt, ##args) +#define opae_log_warn(fmt, args...) opae_log(WARN, "OPAE-WARN: "fmt, ##args) +#define opae_log_info(fmt, args...) opae_log(INFO, "OPAE-INFO: "fmt, ##args) + +#define EAL_INIT_FUNCTION "init" +#define EAL_DEFAULT_OPTIONS "--proc-type auto" + +#define OPAE_KDRV_UNKNOWN "unknown" +#define OPAE_KDRV_VFIO_PCI "vfio-pci" +#define OPAE_KDRV_IGB_UIO "igb_uio" +#define OPAE_KDRV_UIO_PCI "uio_pci_generic" +#define OPAE_KDRV_INTEL_FPGA_PCI "intel-fpga-pci" + +typedef struct { + uint32_t major; + uint32_t minor; + uint32_t micro; +} opae_api_version; + +#define OPAE_NAME_SIZE 32 + +typedef struct { + char bdf[OPAE_NAME_SIZE]; /* segment:bus:device.function */ +} opae_pci_device; + +typedef opae_pci_device *pcidev_id; + +typedef struct { + uint32_t class_id; /**< Class ID or RTE_CLASS_ANY_ID. */ + uint16_t vendor_id; /**< Vendor ID or PCI_ANY_ID. */ + uint16_t device_id; /**< Device ID or PCI_ANY_ID. */ + uint16_t subsystem_vendor_id; /**< Subsystem vendor ID or PCI_ANY_ID. */ + uint16_t subsystem_device_id; /**< Subsystem device ID or PCI_ANY_ID. */ +} opae_pci_id; + +typedef struct { + uint32_t domain; /**< Device domain */ + uint8_t bus; /**< Device bus */ + uint8_t devid; /**< Device ID */ + uint8_t function; /**< Device function. */ +} opae_pci_addr; + +typedef struct { + char pci_addr[OPAE_NAME_SIZE]; /* segment:bus:device.function */ + char drv_name[OPAE_NAME_SIZE]; /* vfio-pci, intel-fpga-pci, etc. */ + opae_pci_id id; + opae_pci_addr addr; +} opae_pci_property; + +#define BIT_SET_8 0xFF +#define BIT_SET_16 0xFFFF +#define BIT_SET_32 0xFFFFFFFF + +typedef struct { + uint8_t b[16]; +} opae_uuid; + +typedef struct { + uint32_t boot_page; + uint32_t num_ports; + uint64_t bitstream_id; + uint64_t bitstream_metadata; + opae_uuid pr_id; + char platform_name[OPAE_NAME_SIZE]; + char dcp_version[OPAE_NAME_SIZE]; + char release_name[OPAE_NAME_SIZE]; + char interface_type[OPAE_NAME_SIZE]; + char build_version[OPAE_NAME_SIZE]; +} opae_fme_property; + +typedef struct { + opae_uuid afu_id; + uint32_t type; /* AFU memory access control type */ + uint32_t index; /* PORT index */ +} opae_port_property; + +typedef struct { + char bmc_version[OPAE_NAME_SIZE]; + char fw_version[OPAE_NAME_SIZE]; +} opae_bmc_property; + +typedef struct { + uint32_t num_retimers; + uint32_t link_speed; + uint32_t link_status; /* each bit corresponding to one link status */ +} opae_phy_info; + +typedef struct { + union { + uint64_t id; + struct { + uint8_t build_patch; + uint8_t build_minor; + uint8_t build_major; + uint8_t fvl_bypass:1; + uint8_t mac_lightweight:1; + uint8_t disagregate:1; + uint8_t lightweiht:1; + uint8_t seu:1; + uint8_t ptp:1; + uint8_t reserve:2; + uint16_t interface:4; + uint16_t afu_revision:12; + uint16_t patch:4; + uint16_t minor:4; + uint16_t major:4; + uint16_t reserved:4; + }; + }; +} opae_bitstream_id; + +typedef struct { + union { + uint32_t version; + struct { + uint8_t micro; + uint8_t minor; + uint8_t major; + uint8_t board; + }; + }; +} opae_bmc_version; + +#define OPAE_MAX_PORT_NUM 4 + +#define OPAE_PROP_PCI 0x01 +#define OPAE_PROP_FME 0x02 +#define OPAE_PROP_PORT 0x04 +#define OPAE_PROP_BMC 0x08 +#define OPAE_PROP_ALL \ + (OPAE_PROP_PCI | OPAE_PROP_FME | OPAE_PROP_PORT | OPAE_PROP_BMC) + +typedef struct { + opae_pci_property pci; + opae_fme_property fme; + opae_port_property port[OPAE_MAX_PORT_NUM]; + opae_bmc_property bmc; +} opae_fpga_property; + +typedef struct { + uint64_t guid_h; + uint64_t guid_l; + uint32_t metadata_len; +} gbs_header; + +#define OPAE_IMG_TYPE_BBS 0 +#define OPAE_IMG_TYPE_BMC 1 +#define OPAE_IMG_TYPE_GBS 2 +#define OPAE_IMG_TYPE(t) ((t) & 0xff) + +#define OPAE_IMG_SUBTYPE_UPDATE 0 +#define OPAE_IMG_SUBTYPE_CANCELLATION 1 +#define OPAE_IMG_SUBTYPE_ROOT_KEY_HASH_256 2 +#define OPAE_IMG_SUBTYPE_ROOT_KEY_HASH_384 3 +#define OPAE_IMG_SUBTYPE(t) (((t) >> 8) & 0xff) + +#define OPAE_IMG_BLK0_SIZE 128 +#define OPAE_IMG_BLK0_MAGIC 0xb6eafd19 +#define OPAE_IMG_BLK1_SIZE 896 +#define OPAE_IMG_HDR_SIZE (OPAE_IMG_BLK0_SIZE + OPAE_IMG_BLK1_SIZE) +#define OPAE_IMG_PL_MIN_SIZE 128 + +typedef struct { + uint32_t magic; + uint32_t payload_len; + uint32_t payload_type; +} opae_img_hdr; + +typedef struct { + int type; + int subtype; + uint32_t total_len; + uint32_t payload_offset; + uint32_t payload_len; +} opae_img_info; + +void opae_get_api_version(opae_api_version *version); +int opae_set_log_level(int level); +int opae_set_log_file(char *path, int clean); +int opae_get_proc_type(void); +int opae_get_parent(pcidev_id id, pcidev_id parent); +int opae_get_child(pcidev_id id, pcidev_id child, int size); +int opae_get_pf1(pcidev_id id, pcidev_id peer, int size); +int opae_init_eal(int argc, char **argv); +int opae_cleanup_eal(void); +int opae_enumerate(opae_pci_id *filter, pcidev_id list, int size); +int opae_probe_device(pcidev_id id); +int opae_remove_device(pcidev_id id); +int opae_unbind_driver(pcidev_id id); +int opae_bind_driver(pcidev_id id, char *drv_name); +int opae_get_property(pcidev_id id, opae_fpga_property *prop, int type); +int opae_get_phy_info(pcidev_id id, opae_phy_info *info); +int opae_partial_reconfigure(pcidev_id id, int port, const char *gbs); +int opae_get_image_info(const char *image, opae_img_info *info); +int opae_cancel_flash_update(pcidev_id id, int force); +int opae_update_flash(pcidev_id id, const char *image, uint64_t *status); +int opae_reboot_device(pcidev_id id, int type, int page); +int opae_store_rsu_status(pcidev_id id, uint32_t status, uint32_t progress); +int opae_load_rsu_status(pcidev_id id, uint32_t *status, uint32_t *progress); +int opae_read_pci_cfg(pcidev_id id, uint32_t address, uint32_t *value); +int opae_write_pci_cfg(pcidev_id id, uint32_t address, uint32_t value); + +#ifdef __cplusplus +} +#endif + + +#endif /* _OPAE_API_H */ diff --git a/examples/meson.build b/examples/meson.build index b9ab24223f..b029715277 100644 --- a/examples/meson.build +++ b/examples/meson.build @@ -16,7 +16,7 @@ all_examples = [ 'eventdev_pipeline', 'fips_validation', 'flow_classify', 'flow_filtering', 'helloworld', - 'ioat', + 'ioat', 'ifpga', 'ip_fragmentation', 'ip_pipeline', 'ip_reassembly', 'ipsec-secgw', 'ipv4_multicast', 'kni', -- 2.29.2