From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by dpdk.org (Postfix) with ESMTP id 38F4A2952 for ; Thu, 2 Mar 2017 12:07:30 +0100 (CET) Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga105.jf.intel.com with ESMTP; 02 Mar 2017 03:07:29 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.35,230,1484035200"; d="scan'208";a="231421866" Received: from unknown (HELO localhost.localdomain.sh.intel.com) ([10.239.128.150]) by fmsmga004.fm.intel.com with ESMTP; 02 Mar 2017 03:07:27 -0800 From: "Chen Jing D(Mark)" To: dev@dpdk.org Cc: cunming.liang@intel.com, gerald.rogers@intel.com, keith.wiles@intel.com, bruce.richardson@intel.com, "Chen, Jing D" , John Mcnamara Date: Thu, 2 Mar 2017 12:03:58 +0800 Message-Id: <1488427438-13646-7-git-send-email-jing.d.chen@intel.com> X-Mailer: git-send-email 1.9.3 In-Reply-To: <1488427438-13646-1-git-send-email-jing.d.chen@intel.com> References: <1488427438-13646-1-git-send-email-jing.d.chen@intel.com> Subject: [dpdk-dev] [PATCH 6/6] doc: introduction to prgdev X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Mar 2017 11:07:31 -0000 From: "Chen, Jing D" This is the documentation to describe what prgdev is, how to use prgdev API and accomplish an image download. Signed-off-by: Chen Jing D(Mark) Signed-off-by: Gerald Rogers Signed-off-by: John Mcnamara --- doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/prgdev_lib.rst | 465 ++++++++++++++++++++++++++++++++++ 2 files changed, 466 insertions(+), 0 deletions(-) create mode 100644 doc/guides/prog_guide/prgdev_lib.rst diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst index 77f427e..4a24264 100644 --- a/doc/guides/prog_guide/index.rst +++ b/doc/guides/prog_guide/index.rst @@ -60,6 +60,7 @@ Programmer's Guide qos_framework power_man packet_classif_access_ctrl + prgdev_lib packet_framework vhost_lib port_hotplug_framework diff --git a/doc/guides/prog_guide/prgdev_lib.rst b/doc/guides/prog_guide/prgdev_lib.rst new file mode 100644 index 0000000..e2d1e2e --- /dev/null +++ b/doc/guides/prog_guide/prgdev_lib.rst @@ -0,0 +1,465 @@ +Overview +======== + +More and more devices start to support on-die programming, which include +NIC, GPU, FPGA, etc. This capability has a requirement to drivers that +introduce a API for application to download/upload image to/from the HW. +So, it's necessary to provide a generic API to perform image download, upload, +validation, etc. + +This RFC patch intends to introduce a DPDK generic programming device layer, +called prgdev below, to provide an abstract, generic APIs for applications to +program hardware without knowing the details of programmable devices. From +driver's perspective, they'll try to adapt their functions to the abstract +APIs defined in prgdev. + +The major purpose of prgdev is to help DPDK users to load/upgrade RTL images +for FPGA devices, or upgrade firmware for programmble NICs. + +But those devices that have the capability to do on-die programming may +expose versatile talents to apps. Add/delete/update entries in flow table, +update/overwrite the firmware/microcode, add a new algorithm, update profiles, +etc, which is hard to be abstracted as generic behaviors. So, prgdev won't +include the APIs to perform device-unique actions in current stage until it +became popular. On the contratry, it only provides the simple capability to +download a segment of buffer(image) from host to on-die device, or vice versa. +The image could be an algorithm, a flow table in the on-die SRAM, a firmware +image running in the device, a function within a pipeline, etc. prgdev won't +try to inteprete the content and it's transparent to prgdev. Apps and devices +can communicate with a language they can understand each other, which is not +provided by prgdev to decide what needs to be programmed. + +When the set of APIs is introduced, a general question is why we need it in +DPDK community? Why we can't use offline tool to perform same thing? The answer +is the prgdev provide a generic, online API to applications, in the meanwile, +offers a capability to switch driver dynamically when downloaded image changed +personality and a new driver is required. Comparing offline tool, it have online +programmability (see below examples); Comparing online tool, it provides a +generic API for many HWs; Comparing generic online tool/API for many products, +it provides a capability to switch driver dynamically. + +There are various means to reach same goal, we'll try to find the best/better +way to approach. All above advantages will help prgdev to be a 'better choice'. + + +Scope +----- + +The target devices include but not limited to programmable NIC, FPGA, GPU. +Any devices, having the capability to store/load a piece of info to/from +the deivce then changed hardware behavior, and applicable to prgdev programming +model, could be registered as a prgdev device. + +The device can be programmed (dynamic) within DPDK or have been prior programmed +(static) prior to a DPDK application being launched. + +Static example: In a static example a program, bitstream or other firmware +would have been loaded prior to the start of a DPDK application. The prgdev +would recognize the firmware loaded, and instantiate higher level interfaces +(example : PMD, L3 forwarding, flow, etc.). + +Dynamic example: In a dynamic example, the programmability of the program, +bitstream or other firmware would be loaded by the DPDK application upon start +, or during execution. The prgdev would offer the interface to allow the DPDK +application to program the device. + + +Dyanamic bind/unbind +-------------------- + +Besides the simple API to upload/download image, the prgdev also introduces a +mechanism internally to switch drivers and register/unregister device on the +fly. With this mechanism, apps can use the programmable device, unbind other +personalities on demand, then program it and bind it back with new personalities. +Apps can follow below examples to simply complete the whole process. + +Note that bind/unbind actions are different concept from that a whole device +attach/detach. For example, ``rte_eal_dev_detach()``, which will try to detach the +drivers with device and remove the whole device from specific class of devices +(ethdev, for example). Then, the whole device is invisible until a new 'probe' +is activated. + +During the whole procedure of image upload/download, prgdev handler is always +valid and apps can use it operate on programmable device. The reason why unbind +is necessary is it may break the hardware when programming is in progress while +other functions are active. Using programmble NIC as an example, it's a little +impossible to receive/forward packets for hardware while updating firmware. In +this case, apps need to detach ethdev function before firmware upgrading. Again, +prgdev handler is still valid, it's function-level detach, different from +device-level detach. After finishing image download, original function needs to +attach back, either use same or different drivers, depends on personalities +changed or not. For a programmble NIC, the driver may be the same. For FPGA, it +may not, see below examples to get more details. + +Another reason to provide bind/unbind action is programmble devices, like FPGA, +are not identified driver by 'vendor ID' and 'device ID', they might not be +changed in all the ways, even FPGA is fully programmed. So, it depends on +internal mechanism of FPGA, not 'vendor ID' and 'device ID' to identify proper +drivers, either a register value or signature, depending on FPGA hardware +design. In this case, EAL or other bus driver doesn't have the capability to +identify proper driver for FPGA device. With prgdev introduced, while FPGA is +always a prgdev, FPGA can use prgdev as primary driver, to find proper function +driver. + +The bind/unbind actions are not trying to break the general initialization +procedure in DPDK. prgdev still follow current EAL initializatio model to probe +drivers for devices. See below examples for details. + +It's not true that bind/unbind is always necessary for programming. If device +has a capability to perform specific functions and program in parallel, then +apps needn't do bind/unbind actions. See 'dynamic program' section below. + +Authentication +-------------- + +Authentication and permissions for programming the device are outside the scope +of this API. It is dependent upon a different mechanism not to be supported by +DPDK. Application can utilize a different mechanism to authenticate user and +or 'program' as validated for loading. + +Misc +---- + +PCI-E Vendor & Device ID are fixed in some devices such as the FPGA and GPU. The +device can be loaded with a personality, and it would be up to this programmable +device to load the appropriate "personality driver" based upon an identification +mechanism to work with the loaded personality. + + +prgdev API +========== + +Below is the major APIs exposed to apps. + +- ``rte_prgdev_get_info``: + Acquire general info of the prgdev. +- ``rte_prgdev_open``: + Open a prgdev for later programming. If failed, needs to call below 'unbind' + function to detach other functions except prgdev first. +- ``rte_prgdev_img_download``: + Download a piece of buffer from host to the device after 'open' succeded. +- ``rte_prgdev_img_upload``: + Upload a piece of information from the device to host. +- ``rte_prgdev_check_status``: + Even download process is successful, it's necessary to check the status of the + downloaded image if worked in expected way by this function. Harware or driver + may not have the capability to check that status, so it's optional to driver. +- ``rte_prgdev_close``: + After upload/download actions, close the prgdev. +- ``rte_prgdev_bind``: + After 'close' action, app can call it to attach the 'new' personality with + proper drivers, then expose to app with new functions. prgdev driver will be + responsible for finding proper drivers to drive the functions. Desired driver + will be responsible for exposing the functions to apps. +- ``rte_prgdev_unbind``: + When app are using the device, they need to call this function to unbind the + misc functions except prgdev with drivers prior to call 'open' function. + + +Initialization procedure +======================== + +In this section, there 2 examples, FPGA and programmable NIC, used as examples +to describe the initialization procedure on how programmable devices are probed +, initialized and registered as different devices. In which, FPGA has 2 +different possible cases, blank and prior programmed FPGAs. Note that prgdev +depends and follow on EAL initialization process to probe drivers for prgdev. + +Blank FPGA +---------- + +Below is the diagram on how Blank FPGA are probed and registered as a prgdev +device that expose to apps. + +When PCI probe seeks driver for the FPGA device, FPGA drive will compare vendor +ID and device ID, then perform HW initialization if both ID match. After then, +it registered in prgdev abstract layer as a new device with a set of helper +function ops. + +After that, apps will observe single device: prgdev device. Until now, the FPGA +device is a simply device without any functionalities from application's +perspective. + +In the following action, app will call ``prgdev bind`` function with the device +ID of the FPGA device to bind the function within the FPGA to specific driver. +FPGA driver will read AFU (Accelerated Function Unit) or signature from hardware +and find potential function. Since it's a blank without any prior programming, a +simply SUCCESS return:: + + Abstraction FPGA driver Hardware + layer + + + + + | | | + +-----eal pci probe-----> | + | | | +------------+ + | | | | FPGA dev | + | +--HW initialization----> | | + +------------+ | | | |+----------+| + | prgdev dev | <--prgdev_ops_register--+ | || AFU id || + +------------+ | | | |+----------+| + | | | +------------+ + +-----prgdev bind-------> | + | | | + | +--Read and parse AFU---> + | | | + | <--No personality found-+ + | | | + <-------SUCCESS---------+ | + | | | + + + + + +prior programmed FPGA +--------------------- + +For a prior programmed FPGA, all execution flow is similar to blank FPGA case, +except the action of reading signature, as below flow table indicates. It will +read valid functions, crypto and ethdev, as an example, then find proper driver +to initialize the functions, then register as different ports. + +After initialization, app will observe 3 kinds of devices, a prgdev device, a +ethdev device and a cryptodev device, which all comes from a single PCIE device:: + + + Abstraction FPGA driver Hardware + layer + + + + + | | | + +----eal pci probe------> | + | | | +-------------+ + | | | | FPGA dev | + | +--HW initialization----> | | + +------------+ | | | +-------------+ + | prgdev dev | <--prgdev_ops_register--+ | || AFU id || + +------------+ | | | |-------------| + | | | +-------------+ + +-----prgdev bind-------> | + | | | + | +--Read and parse AFU---> + | | | + | <--eth and crypto found-+ + +------------+ | | | + | ethdev port| <---ethdev attach-------+ | + +------------+ | | | + | | | + +------------+ | | | + | crypto port| <---crypto attach-------+ | + +------------+ | | | + | | | + + + + + + +Programmable NIC +---------------- + +Comparing FPGA, programmable NIC has a nature of being a ethdev device. The +initialization procedure is much more simpler than FPGA. Below is a flow table +to describe this. + +The 'eal pci probe' procedure is similar, it registered itself as a prgdev after +checking that the NIC is programmable. What is different from FPGA is it +registered itself as a ethdev device in the following action due to the NIC +nature. + +Then, app still follow the common process to call ``prgdev bind``, while the NIC +driver does nothing and just return a SUCCESS:: + + Abstraction NIC driver Hardware + layer + + + + + | | | + +----eal pci probe------> | + | | | +-------------+ + | +--HW initialization----> |Programmable | + | | | | NIC | + +------------+ | +---Capability check----> +-------------+ + | prgdev dev | <--prgdev_ops_register--+ | + +------------+ | | | + | | | + +------------+ | | | + | ethdev port| <---ethdev attach-------+ | + +------------+ | | | + | | | + +-----prgdev bind-------> | + | | | + | | | + <--------SUCCESS--------+ | + | | | + + + + + + +Program procedure +================= + +This section will describe how apps finish an image downloading procedure with +the prgdev API. Similar to previous section, it will use several FPGA and NIC +examples to show the common and different steps. + +Needs to point out, ``rte_prgdev_img_download`` function didn't carry the info +of the image, or where downloading the image to. The interpretion info of the +image can be hidden into the buffer itself. + +Common programming steps +------------------------ + +Before reaching details, it needs to introduce the common steps for apps to +follow in order to download the image to the hardware. + +Before programming, apps need to call function ``rte_prgdev_open`` to open the +programming interface. If failed, either functions residing in PCI-E device is +in use or the PCI-E device is not ready, the programming steps are different. + +Below is the steps that 'open' succeeds: + +#. rte_prgdev_open --> SUCCESS +#. rte_prgdev_img_download +#. rte_prgdev_check_status +#. rte_prgdev_close +#. rte_prgdev_bind + +If failed to open the programming interface, apps need to follow below steps: + +#. rte_prgdev_open --> FAILURE +#. Close all related functions +#. ret_prgdev_unbind +#. rte_prgdev_open --> SUCCESS +#. rte_prgdev_img_download +#. rte_prgdev_check_status +#. rte_prgdev_close +#. rte_prgdev_bind + + +Detach functions +---------------- + +As discussed before, apps need to follow steps to detach all the other functions +except prgdev when meeting error for 'open' operation. Below is a FPGA example +with 'ethdev' and 'cryptodev' function attached. + +After meeting error, app needs to close ethdev port and cryptodev port first. +Then, use 'prgdev unbind' to detach all the functions that registered previously +. FPGA driver will call each 'detach' function for each registered device class +respectively. After then, the 'ethdev' and 'cryptodev' ports will not be +available:: + + + Abstraction FPGA driver Hardware + layer + + + + + +------------+ | | | + | prgdev dev | +-------prgdev open-----> | + | | <-------ERR_IN_USE------+ | +-------------+ + +------------+ | | | | FPGA dev | + | | | | +---------+ | + +------------+ | | | | | AFU id | | + | ethdev port| +---rte_ethdev_close----> | | +---------+ | + | | +------ethdev close-----> +-------------+ + +------------+ <---------SUCCESS-------+ | + | | | + +------------+ | | | + | crypto port| +---crypto port close---> | + | | | +---crypto func close---> + +------------+ <---------SUCCESS-------+ | + | | | + +-----prgdev unbind-----> | + | | | + + - - - - - -+ | | | + ethdev port <-----ethdev detach-----+ | + +- - - - - - + | | | + | | | + + - - - - - -+ | | | + crypto port <----cryptodev detach---+ | + +- - - - - - + | | | + + + + + +Device programming +------------------ + +After 'open' prgdev sucessfully, apps follow 'image download', 'check status', +'close' sequence respectively. After then, they need to 'bind' new functions +back with 'prgdev bind' call. + +Still re-use previous FPGA example, below is the execution sequence. After image +downloading and 'bind' action, 2 new 'virtio' and 'flow' functions are +registered and 2 new ports are presented to apps:: + + + Abstraction FPGA driver Hardware + layer + + + + +-------------+ + +------------+ | | | | FPGA dev | + | | +-----prgdev open-------> | | +---------+ | + | | <-------SUCCESS---------+ | | | ethdev | | + | | | | | | +---------+ | + | | +--prgdev_img_download--> | | +---------+ | + | | | +----Download image-----> | | crypto | | + | | | <--------SUCCESS--------+ | +---------+ | + | | <---------SUCCESS-------+ | +-------------+ + | prgdev dev | | | | | + | | +---prgdev_chk_status---> | | + | | | +--Verify image status--> | + | | | <--------SUCCESS--------+ | + | | <--------SUCCESS--------+ | | + | | +-------prgdev close----> | v + | | | | | +-------------+ + | | +-------prgdev bind-----> | | FPGA dev | + +------------+ | | | | +---------+ | + | <--Virtio and flow dev--+ | | virtio | | + +------------+ | | | | +---------+ | + | ethdev port| <-----ethdev attach-----+ | | +---------+ | + +------------+ | | | | |Flow dev | | + | | | | +---------+ | + +------------+ | | | +-------------+ + | Flow port | <-----Flow dev attach---+ | + +------------+ | | | + + + + + + +Dyanamic program +---------------- + +If prgdev device has a capability to be programmed while functions are in use, +then we say it supports dynamic program. It's possible to update a profile while +device is used as a ethdev device. + +The device can claim this capability by set a DYNAMIC_PROGRAM flag in function +``rte_prgdev_get_info``. + +The dynamic programming procedure is the same as previous section shows, while +ignore the steps to 'close' and 'unbind' actions when device is in use. + +Note that prgdev driver needs to consider if it can support dynamic program to +avoid any hardware or software error. + + +Source code layout +================== + +Below is the suggested source code layout. + +- 'prg NIC' under 'net' is the PMD NIC directory that support prgdev. This + doesn't change programmable NIC's directory and remains the same. The PMD + driver should include 2 parts, one part is the code to support rte_ethdev API, + another part is the code to support rte_prgdev API. +- 'librte_prgdev' under 'lib' is the new directory store definition and + implementation of the generic prgdev API. +- 'prg' under 'drivers' is the directory to store specfic implementation + driver to support rte_prgdev. Each sub-directory could be a prgdev driver to + support specific hardware product. + +:: + + +---------+ +---------+ + | drivers | | lib | + +---------+ +----+----+ + | | + +----------+-----------+ | + | | | + +---v---+ +---v----+ +-------v-------+ + | net | | prg | | librte_prgdev | + +---+---+ +---+----+ +---------------+ + | | + +-----+----+ +-----+----+ + | | | | + +---v----+ +---v----+ +---v----+ +---v----+ + | i40e | | prg NIC| |FPGA ABC| |GPU XYZ | + +--------+ +--------+ +--------+ +--------+ -- 1.7.7.6