From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id BDE93A0613 for ; Fri, 27 Sep 2019 18:55:04 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B4D461BF50; Fri, 27 Sep 2019 18:55:01 +0200 (CEST) Received: from relay0213.mxlogin.com (relay0213.mxlogin.com [199.181.239.213]) by dpdk.org (Postfix) with ESMTP id E7ED21BF4D for ; Fri, 27 Sep 2019 18:54:59 +0200 (CEST) Received: from filter002.mxroute.com (unknown [94.130.183.33]) by relay0213.mxlogin.com (Postfix) with ESMTP id C35B8CD1030F; Fri, 27 Sep 2019 11:54:58 -0500 (CDT) Received: from galaxy.mxroute.com (unknown [23.92.70.113]) by filter002.mxroute.com (Postfix) with ESMTPS id A30F43F0B8; Fri, 27 Sep 2019 16:54:51 +0000 (UTC) Received: from irdmzpr02-ext.ir.intel.com ([192.198.151.37] helo=localhost) by galaxy.mxroute.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.91) (envelope-from ) id 1iDtN8-0008JY-1h; Fri, 27 Sep 2019 12:45:34 -0400 From: Ray Kinsella To: dev@dpdk.org Cc: mdr@ashroe.eu, thomas@monjalon.net, stephen@networkplumber.org, bruce.richardson@intel.com, ferruh.yigit@intel.com, konstantin.ananyev@intel.com, jerinj@marvell.com, olivier.matz@6wind.com, nhorman@tuxdriver.com, maxime.coquelin@redhat.com, john.mcnamara@intel.com, marko.kovacevic@intel.com, hemant.agrawal@nxp.com, ktraynor@redhat.com, aconole@redhat.com Date: Fri, 27 Sep 2019 17:54:40 +0100 Message-Id: <1569603283-1857-2-git-send-email-mdr@ashroe.eu> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1569603283-1857-1-git-send-email-mdr@ashroe.eu> References: <1569603283-1857-1-git-send-email-mdr@ashroe.eu> X-AuthUser: mdr@ashroe.eu Subject: [dpdk-dev] [PATCH v6 1/4] doc: separate versioning.rst into version and policy X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Separate versioning.rst into abi versioning and abi policy guidance, in preparation for adding more detail to the abi policy. Signed-off-by: Ray Kinsella --- doc/guides/contributing/abi_policy.rst | 169 +++++++++ doc/guides/contributing/abi_versioning.rst | 427 +++++++++++++++++++++ doc/guides/contributing/index.rst | 3 +- doc/guides/contributing/versioning.rst | 591 ----------------------------- 4 files changed, 598 insertions(+), 592 deletions(-) create mode 100644 doc/guides/contributing/abi_policy.rst create mode 100644 doc/guides/contributing/abi_versioning.rst delete mode 100644 doc/guides/contributing/versioning.rst diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst new file mode 100644 index 0000000..55bacb4 --- /dev/null +++ b/doc/guides/contributing/abi_policy.rst @@ -0,0 +1,169 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2018 The DPDK contributors + +.. abi_api_policy: + +DPDK ABI/API policy +=================== + +Description +----------- + +This document details some methods for handling ABI management in the DPDK. + +General Guidelines +------------------ + +#. Whenever possible, ABI should be preserved +#. ABI/API may be changed with a deprecation process +#. The modification of symbols can generally be managed with versioning +#. Libraries or APIs marked in ``experimental`` state may change without constraint +#. New APIs will be marked as ``experimental`` for at least one release to allow + any issues found by users of the new API to be fixed quickly +#. The addition of symbols is generally not problematic +#. The removal of symbols generally is an ABI break and requires bumping of the + LIBABIVER macro +#. Updates to the minimum hardware requirements, which drop support for hardware which + was previously supported, should be treated as an ABI change. + +What is an ABI +~~~~~~~~~~~~~~ + +An ABI (Application Binary Interface) is the set of runtime interfaces exposed +by a library. It is similar to an API (Application Programming Interface) but +is the result of compilation. It is also effectively cloned when applications +link to dynamic libraries. That is to say when an application is compiled to +link against dynamic libraries, it is assumed that the ABI remains constant +between the time the application is compiled/linked, and the time that it runs. +Therefore, in the case of dynamic linking, it is critical that an ABI is +preserved, or (when modified), done in such a way that the application is unable +to behave improperly or in an unexpected fashion. + + +ABI/API Deprecation +------------------- + +The DPDK ABI policy +~~~~~~~~~~~~~~~~~~~ + +ABI versions are set at the time of major release labeling, and the ABI may +change multiple times, without warning, between the last release label and the +HEAD label of the git tree. + +ABI versions, once released, are available until such time as their +deprecation has been noted in the Release Notes for at least one major release +cycle. For example consider the case where the ABI for DPDK 2.0 has been +shipped and then a decision is made to modify it during the development of +DPDK 2.1. The decision will be recorded in the Release Notes for the DPDK 2.1 +release and the modification will be made available in the DPDK 2.2 release. + +ABI versions may be deprecated in whole or in part as needed by a given +update. + +Some ABI changes may be too significant to reasonably maintain multiple +versions. In those cases ABI's may be updated without backward compatibility +being provided. The requirements for doing so are: + +#. At least 3 acknowledgments of the need to do so must be made on the + dpdk.org mailing list. + + - The acknowledgment of the maintainer of the component is mandatory, or if + no maintainer is available for the component, the tree/sub-tree maintainer + for that component must acknowledge the ABI change instead. + + - It is also recommended that acknowledgments from different "areas of + interest" be sought for each deprecation, for example: from NIC vendors, + CPU vendors, end-users, etc. + +#. The changes (including an alternative map file) can be included with + deprecation notice, in wrapped way by the ``RTE_NEXT_ABI`` option, + to provide more details about oncoming changes. + ``RTE_NEXT_ABI`` wrapper will be removed when it become the default ABI. + More preferred way to provide this information is sending the feature + as a separate patch and reference it in deprecation notice. + +#. A full deprecation cycle, as explained above, must be made to offer + downstream consumers sufficient warning of the change. + +Note that the above process for ABI deprecation should not be undertaken +lightly. ABI stability is extremely important for downstream consumers of the +DPDK, especially when distributed in shared object form. Every effort should +be made to preserve the ABI whenever possible. The ABI should only be changed +for significant reasons, such as performance enhancements. ABI breakage due to +changes such as reorganizing public structure fields for aesthetic or +readability purposes should be avoided. + +.. note:: + + Updates to the minimum hardware requirements, which drop support for hardware + which was previously supported, should be treated as an ABI change, and + follow the relevant deprecation policy procedures as above: 3 acks and + announcement at least one release in advance. + +Examples of Deprecation Notices +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The following are some examples of ABI deprecation notices which would be +added to the Release Notes: + +* The Macro ``#RTE_FOO`` is deprecated and will be removed with version 2.0, + to be replaced with the inline function ``rte_foo()``. + +* The function ``rte_mbuf_grok()`` has been updated to include a new parameter + in version 2.0. Backwards compatibility will be maintained for this function + until the release of version 2.1 + +* The members of ``struct rte_foo`` have been reorganized in release 2.0 for + performance reasons. Existing binary applications will have backwards + compatibility in release 2.0, while newly built binaries will need to + reference the new structure variant ``struct rte_foo2``. Compatibility will + be removed in release 2.2, and all applications will require updating and + rebuilding to the new structure at that time, which will be renamed to the + original ``struct rte_foo``. + +* Significant ABI changes are planned for the ``librte_dostuff`` library. The + upcoming release 2.0 will not contain these changes, but release 2.1 will, + and no backwards compatibility is planned due to the extensive nature of + these changes. Binaries using this library built prior to version 2.1 will + require updating and recompilation. + +New API replacing previous one +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If a new API proposed functionally replaces an existing one, when the new API +becomes non-experimental then the old one is marked with ``__rte_deprecated``. +Deprecated APIs are removed completely just after the next LTS. + +Reminder that old API should follow deprecation process to be removed. + + +Experimental APIs +----------------- + +APIs marked as ``experimental`` are not considered part of the ABI and may +change without warning at any time. Since changes to APIs are most likely +immediately after their introduction, as users begin to take advantage of +those new APIs and start finding issues with them, new DPDK APIs will be +automatically marked as ``experimental`` to allow for a period of stabilization +before they become part of a tracked ABI. + +Note that marking an API as experimental is a multi step process. +To mark an API as experimental, the symbols which are desired to be exported +must be placed in an EXPERIMENTAL version block in the corresponding libraries' +version map script. +Secondly, the corresponding prototypes of those exported functions (in the +development header files), must be marked with the ``__rte_experimental`` tag +(see ``rte_compat.h``). +The DPDK build makefiles perform a check to ensure that the map file and the +C code reflect the same list of symbols. +This check can be circumvented by defining ``ALLOW_EXPERIMENTAL_API`` +during compilation in the corresponding library Makefile. + +In addition to tagging the code with ``__rte_experimental``, +the doxygen markup must also contain the EXPERIMENTAL string, +and the MAINTAINERS file should note the EXPERIMENTAL libraries. + +For removing the experimental tag associated with an API, deprecation notice +is not required. Though, an API should remain in experimental state for at least +one release. Thereafter, normal process of posting patch for review to mailing +list can be followed. diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst new file mode 100644 index 0000000..53e6ac0 --- /dev/null +++ b/doc/guides/contributing/abi_versioning.rst @@ -0,0 +1,427 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2018 The DPDK contributors + +.. library_versioning: + +Library versioning +------------------ + +Downstreams might want to provide different DPDK releases at the same time to +support multiple consumers of DPDK linked against older and newer sonames. + +Also due to the interdependencies that DPDK libraries can have applications +might end up with an executable space in which multiple versions of a library +are mapped by ld.so. + +Think of LibA that got an ABI bump and LibB that did not get an ABI bump but is +depending on LibA. + +.. note:: + + Application + \-> LibA.old + \-> LibB.new -> LibA.new + +That is a conflict which can be avoided by setting ``CONFIG_RTE_MAJOR_ABI``. +If set, the value of ``CONFIG_RTE_MAJOR_ABI`` overwrites all - otherwise per +library - versions defined in the libraries ``LIBABIVER``. +An example might be ``CONFIG_RTE_MAJOR_ABI=16.11`` which will make all libraries +``librte.so.16.11`` instead of ``librte.so.``. + + +ABI versioning +-------------- + +Versioning Macros +~~~~~~~~~~~~~~~~~ + +When a symbol is exported from a library to provide an API, it also provides a +calling convention (ABI) that is embodied in its name, return type and +arguments. Occasionally that function may need to change to accommodate new +functionality or behavior. When that occurs, it is desirable to allow for +backward compatibility for a time with older binaries that are dynamically +linked to the DPDK. + +To support backward compatibility the ``rte_compat.h`` +header file provides macros to use when updating exported functions. These +macros are used in conjunction with the ``rte__version.map`` file for +a given library to allow multiple versions of a symbol to exist in a shared +library so that older binaries need not be immediately recompiled. + +The macros exported are: + +* ``VERSION_SYMBOL(b, e, n)``: Creates a symbol version table entry binding + versioned symbol ``b@DPDK_n`` to the internal function ``b_e``. + +* ``BIND_DEFAULT_SYMBOL(b, e, n)``: Creates a symbol version entry instructing + the linker to bind references to symbol ``b`` to the internal symbol + ``b_e``. + +* ``MAP_STATIC_SYMBOL(f, p)``: Declare the prototype ``f``, and map it to the + fully qualified function ``p``, so that if a symbol becomes versioned, it + can still be mapped back to the public symbol name. + +Examples of ABI Macro use +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Updating a public API +_____________________ + +Assume we have a function as follows + +.. code-block:: c + + /* + * Create an acl context object for apps to + * manipulate + */ + struct rte_acl_ctx * + rte_acl_create(const struct rte_acl_param *param) + { + ... + } + + +Assume that struct rte_acl_ctx is a private structure, and that a developer +wishes to enhance the acl api so that a debugging flag can be enabled on a +per-context basis. This requires an addition to the structure (which, being +private, is safe), but it also requires modifying the code as follows + +.. code-block:: c + + /* + * Create an acl context object for apps to + * manipulate + */ + struct rte_acl_ctx * + rte_acl_create(const struct rte_acl_param *param, int debug) + { + ... + } + + +Note also that, being a public function, the header file prototype must also be +changed, as must all the call sites, to reflect the new ABI footprint. We will +maintain previous ABI versions that are accessible only to previously compiled +binaries + +The addition of a parameter to the function is ABI breaking as the function is +public, and existing application may use it in its current form. However, the +compatibility macros in DPDK allow a developer to use symbol versioning so that +multiple functions can be mapped to the same public symbol based on when an +application was linked to it. To see how this is done, we start with the +requisite libraries version map file. Initially the version map file for the +acl library looks like this + +.. code-block:: none + + DPDK_2.0 { + global: + + rte_acl_add_rules; + rte_acl_build; + rte_acl_classify; + rte_acl_classify_alg; + rte_acl_classify_scalar; + rte_acl_create; + rte_acl_dump; + rte_acl_find_existing; + rte_acl_free; + rte_acl_ipv4vlan_add_rules; + rte_acl_ipv4vlan_build; + rte_acl_list_dump; + rte_acl_reset; + rte_acl_reset_rules; + rte_acl_set_ctx_classify; + + local: *; + }; + +This file needs to be modified as follows + +.. code-block:: none + + DPDK_2.0 { + global: + + rte_acl_add_rules; + rte_acl_build; + rte_acl_classify; + rte_acl_classify_alg; + rte_acl_classify_scalar; + rte_acl_create; + rte_acl_dump; + rte_acl_find_existing; + rte_acl_free; + rte_acl_ipv4vlan_add_rules; + rte_acl_ipv4vlan_build; + rte_acl_list_dump; + rte_acl_reset; + rte_acl_reset_rules; + rte_acl_set_ctx_classify; + + local: *; + }; + + DPDK_2.1 { + global: + rte_acl_create; + + } DPDK_2.0; + +The addition of the new block tells the linker that a new version node is +available (DPDK_2.1), which contains the symbol rte_acl_create, and inherits the +symbols from the DPDK_2.0 node. This list is directly translated into a list of +exported symbols when DPDK is compiled as a shared library + +Next, we need to specify in the code which function map to the rte_acl_create +symbol at which versions. First, at the site of the initial symbol definition, +we need to update the function so that it is uniquely named, and not in conflict +with the public symbol name + +.. code-block:: c + + struct rte_acl_ctx * + -rte_acl_create(const struct rte_acl_param *param) + +rte_acl_create_v20(const struct rte_acl_param *param) + { + size_t sz; + struct rte_acl_ctx *ctx; + ... + +Note that the base name of the symbol was kept intact, as this is conducive to +the macros used for versioning symbols. That is our next step, mapping this new +symbol name to the initial symbol name at version node 2.0. Immediately after +the function, we add this line of code + +.. code-block:: c + + VERSION_SYMBOL(rte_acl_create, _v20, 2.0); + +Remembering to also add the rte_compat.h header to the requisite c file where +these changes are being made. The above macro instructs the linker to create a +new symbol ``rte_acl_create@DPDK_2.0``, which matches the symbol created in older +builds, but now points to the above newly named function. We have now mapped +the original rte_acl_create symbol to the original function (but with a new +name) + +Next, we need to create the 2.1 version of the symbol. We create a new function +name, with a different suffix, and implement it appropriately + +.. code-block:: c + + struct rte_acl_ctx * + rte_acl_create_v21(const struct rte_acl_param *param, int debug); + { + struct rte_acl_ctx *ctx = rte_acl_create_v20(param); + + ctx->debug = debug; + + return ctx; + } + +This code serves as our new API call. Its the same as our old call, but adds +the new parameter in place. Next we need to map this function to the symbol +``rte_acl_create@DPDK_2.1``. To do this, we modify the public prototype of the call +in the header file, adding the macro there to inform all including applications, +that on re-link, the default rte_acl_create symbol should point to this +function. Note that we could do this by simply naming the function above +rte_acl_create, and the linker would chose the most recent version tag to apply +in the version script, but we can also do this in the header file + +.. code-block:: c + + struct rte_acl_ctx * + -rte_acl_create(const struct rte_acl_param *param); + +rte_acl_create(const struct rte_acl_param *param, int debug); + +BIND_DEFAULT_SYMBOL(rte_acl_create, _v21, 2.1); + +The BIND_DEFAULT_SYMBOL macro explicitly tells applications that include this +header, to link to the rte_acl_create_v21 function and apply the DPDK_2.1 +version node to it. This method is more explicit and flexible than just +re-implementing the exact symbol name, and allows for other features (such as +linking to the old symbol version by default, when the new ABI is to be opt-in +for a period. + +One last thing we need to do. Note that we've taken what was a public symbol, +and duplicated it into two uniquely and differently named symbols. We've then +mapped each of those back to the public symbol ``rte_acl_create`` with different +version tags. This only applies to dynamic linking, as static linking has no +notion of versioning. That leaves this code in a position of no longer having a +symbol simply named ``rte_acl_create`` and a static build will fail on that +missing symbol. + +To correct this, we can simply map a function of our choosing back to the public +symbol in the static build with the ``MAP_STATIC_SYMBOL`` macro. Generally the +assumption is that the most recent version of the symbol is the one you want to +map. So, back in the C file where, immediately after ``rte_acl_create_v21`` is +defined, we add this + +.. code-block:: c + + struct rte_acl_ctx * + rte_acl_create_v21(const struct rte_acl_param *param, int debug) + { + ... + } + MAP_STATIC_SYMBOL(struct rte_acl_ctx *rte_acl_create(const struct rte_acl_param *param, int debug), rte_acl_create_v21); + +That tells the compiler that, when building a static library, any calls to the +symbol ``rte_acl_create`` should be linked to ``rte_acl_create_v21`` + +That's it, on the next shared library rebuild, there will be two versions of +rte_acl_create, an old DPDK_2.0 version, used by previously built applications, +and a new DPDK_2.1 version, used by future built applications. + + +Deprecating part of a public API +________________________________ + +Lets assume that you've done the above update, and after a few releases have +passed you decide you would like to retire the old version of the function. +After having gone through the ABI deprecation announcement process, removal is +easy. Start by removing the symbol from the requisite version map file: + +.. code-block:: none + + DPDK_2.0 { + global: + + rte_acl_add_rules; + rte_acl_build; + rte_acl_classify; + rte_acl_classify_alg; + rte_acl_classify_scalar; + rte_acl_dump; + - rte_acl_create + rte_acl_find_existing; + rte_acl_free; + rte_acl_ipv4vlan_add_rules; + rte_acl_ipv4vlan_build; + rte_acl_list_dump; + rte_acl_reset; + rte_acl_reset_rules; + rte_acl_set_ctx_classify; + + local: *; + }; + + DPDK_2.1 { + global: + rte_acl_create; + } DPDK_2.0; + + +Next remove the corresponding versioned export. + +.. code-block:: c + + -VERSION_SYMBOL(rte_acl_create, _v20, 2.0); + + +Note that the internal function definition could also be removed, but its used +in our example by the newer version _v21, so we leave it in place. This is a +coding style choice. + +Lastly, we need to bump the LIBABIVER number for this library in the Makefile to +indicate to applications doing dynamic linking that this is a later, and +possibly incompatible library version: + +.. code-block:: c + + -LIBABIVER := 1 + +LIBABIVER := 2 + +Deprecating an entire ABI version +_________________________________ + +While removing a symbol from and ABI may be useful, it is often more practical +to remove an entire version node at once. If a version node completely +specifies an API, then removing part of it, typically makes it incomplete. In +those cases it is better to remove the entire node + +To do this, start by modifying the version map file, such that all symbols from +the node to be removed are merged into the next node in the map + +In the case of our map above, it would transform to look as follows + +.. code-block:: none + + DPDK_2.1 { + global: + + rte_acl_add_rules; + rte_acl_build; + rte_acl_classify; + rte_acl_classify_alg; + rte_acl_classify_scalar; + rte_acl_dump; + rte_acl_create + rte_acl_find_existing; + rte_acl_free; + rte_acl_ipv4vlan_add_rules; + rte_acl_ipv4vlan_build; + rte_acl_list_dump; + rte_acl_reset; + rte_acl_reset_rules; + rte_acl_set_ctx_classify; + + local: *; + }; + +Then any uses of BIND_DEFAULT_SYMBOL that pointed to the old node should be +updated to point to the new version node in any header files for all affected +symbols. + +.. code-block:: c + + -BIND_DEFAULT_SYMBOL(rte_acl_create, _v20, 2.0); + +BIND_DEFAULT_SYMBOL(rte_acl_create, _v21, 2.1); + +Lastly, any VERSION_SYMBOL macros that point to the old version node should be +removed, taking care to keep, where need old code in place to support newer +versions of the symbol. + + +Running the ABI Validator +------------------------- + +The ``devtools`` directory in the DPDK source tree contains a utility program, +``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI +Compliance Checker +`_. + +This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper`` +utilities which can be installed via a package manager. For example:: + + sudo yum install abi-compliance-checker + sudo yum install abi-dumper + +The syntax of the ``validate-abi.sh`` utility is:: + + ./devtools/validate-abi.sh + +Where ``REV1`` and ``REV2`` are valid gitrevisions(7) +https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html +on the local repo. + +For example:: + + # Check between the previous and latest commit: + ./devtools/validate-abi.sh HEAD~1 HEAD + + # Check on a specific compilation target: + ./devtools/validate-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD + + # Check between two tags: + ./devtools/validate-abi.sh v2.0.0 v2.1.0 + + # Check between git master and local topic-branch "vhost-hacking": + ./devtools/validate-abi.sh master vhost-hacking + +After the validation script completes (it can take a while since it need to +compile both tags) it will create compatibility reports in the +``./abi-check/compat_report`` directory. Listed incompatibilities can be found +as follows:: + + grep -lr Incompatible abi-check/compat_reports/ diff --git a/doc/guides/contributing/index.rst b/doc/guides/contributing/index.rst index e2608d3..2fefd91 100644 --- a/doc/guides/contributing/index.rst +++ b/doc/guides/contributing/index.rst @@ -10,7 +10,8 @@ Contributor's Guidelines coding_style design - versioning + abi_policy + abi_versioning documentation patches vulnerability diff --git a/doc/guides/contributing/versioning.rst b/doc/guides/contributing/versioning.rst deleted file mode 100644 index 3ab2c43..0000000 --- a/doc/guides/contributing/versioning.rst +++ /dev/null @@ -1,591 +0,0 @@ -.. SPDX-License-Identifier: BSD-3-Clause - Copyright 2018 The DPDK contributors - -DPDK ABI/API policy -=================== - -Description ------------ - -This document details some methods for handling ABI management in the DPDK. - -General Guidelines ------------------- - -#. Whenever possible, ABI should be preserved -#. ABI/API may be changed with a deprecation process -#. The modification of symbols can generally be managed with versioning -#. Libraries or APIs marked in ``experimental`` state may change without constraint -#. New APIs will be marked as ``experimental`` for at least one release to allow - any issues found by users of the new API to be fixed quickly -#. The addition of symbols is generally not problematic -#. The removal of symbols generally is an ABI break and requires bumping of the - LIBABIVER macro -#. Updates to the minimum hardware requirements, which drop support for hardware which - was previously supported, should be treated as an ABI change. - -What is an ABI -~~~~~~~~~~~~~~ - -An ABI (Application Binary Interface) is the set of runtime interfaces exposed -by a library. It is similar to an API (Application Programming Interface) but -is the result of compilation. It is also effectively cloned when applications -link to dynamic libraries. That is to say when an application is compiled to -link against dynamic libraries, it is assumed that the ABI remains constant -between the time the application is compiled/linked, and the time that it runs. -Therefore, in the case of dynamic linking, it is critical that an ABI is -preserved, or (when modified), done in such a way that the application is unable -to behave improperly or in an unexpected fashion. - - -ABI/API Deprecation -------------------- - -The DPDK ABI policy -~~~~~~~~~~~~~~~~~~~ - -ABI versions are set at the time of major release labeling, and the ABI may -change multiple times, without warning, between the last release label and the -HEAD label of the git tree. - -ABI versions, once released, are available until such time as their -deprecation has been noted in the Release Notes for at least one major release -cycle. For example consider the case where the ABI for DPDK 2.0 has been -shipped and then a decision is made to modify it during the development of -DPDK 2.1. The decision will be recorded in the Release Notes for the DPDK 2.1 -release and the modification will be made available in the DPDK 2.2 release. - -ABI versions may be deprecated in whole or in part as needed by a given -update. - -Some ABI changes may be too significant to reasonably maintain multiple -versions. In those cases ABI's may be updated without backward compatibility -being provided. The requirements for doing so are: - -#. At least 3 acknowledgments of the need to do so must be made on the - dpdk.org mailing list. - - - The acknowledgment of the maintainer of the component is mandatory, or if - no maintainer is available for the component, the tree/sub-tree maintainer - for that component must acknowledge the ABI change instead. - - - It is also recommended that acknowledgments from different "areas of - interest" be sought for each deprecation, for example: from NIC vendors, - CPU vendors, end-users, etc. - -#. The changes (including an alternative map file) can be included with - deprecation notice, in wrapped way by the ``RTE_NEXT_ABI`` option, - to provide more details about oncoming changes. - ``RTE_NEXT_ABI`` wrapper will be removed when it become the default ABI. - More preferred way to provide this information is sending the feature - as a separate patch and reference it in deprecation notice. - -#. A full deprecation cycle, as explained above, must be made to offer - downstream consumers sufficient warning of the change. - -Note that the above process for ABI deprecation should not be undertaken -lightly. ABI stability is extremely important for downstream consumers of the -DPDK, especially when distributed in shared object form. Every effort should -be made to preserve the ABI whenever possible. The ABI should only be changed -for significant reasons, such as performance enhancements. ABI breakage due to -changes such as reorganizing public structure fields for aesthetic or -readability purposes should be avoided. - -.. note:: - - Updates to the minimum hardware requirements, which drop support for hardware - which was previously supported, should be treated as an ABI change, and - follow the relevant deprecation policy procedures as above: 3 acks and - announcement at least one release in advance. - -Examples of Deprecation Notices -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The following are some examples of ABI deprecation notices which would be -added to the Release Notes: - -* The Macro ``#RTE_FOO`` is deprecated and will be removed with version 2.0, - to be replaced with the inline function ``rte_foo()``. - -* The function ``rte_mbuf_grok()`` has been updated to include a new parameter - in version 2.0. Backwards compatibility will be maintained for this function - until the release of version 2.1 - -* The members of ``struct rte_foo`` have been reorganized in release 2.0 for - performance reasons. Existing binary applications will have backwards - compatibility in release 2.0, while newly built binaries will need to - reference the new structure variant ``struct rte_foo2``. Compatibility will - be removed in release 2.2, and all applications will require updating and - rebuilding to the new structure at that time, which will be renamed to the - original ``struct rte_foo``. - -* Significant ABI changes are planned for the ``librte_dostuff`` library. The - upcoming release 2.0 will not contain these changes, but release 2.1 will, - and no backwards compatibility is planned due to the extensive nature of - these changes. Binaries using this library built prior to version 2.1 will - require updating and recompilation. - -New API replacing previous one -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If a new API proposed functionally replaces an existing one, when the new API -becomes non-experimental then the old one is marked with ``__rte_deprecated``. -Deprecated APIs are removed completely just after the next LTS. - -Reminder that old API should follow deprecation process to be removed. - - -Experimental APIs ------------------ - -APIs marked as ``experimental`` are not considered part of the ABI and may -change without warning at any time. Since changes to APIs are most likely -immediately after their introduction, as users begin to take advantage of -those new APIs and start finding issues with them, new DPDK APIs will be -automatically marked as ``experimental`` to allow for a period of stabilization -before they become part of a tracked ABI. - -Note that marking an API as experimental is a multi step process. -To mark an API as experimental, the symbols which are desired to be exported -must be placed in an EXPERIMENTAL version block in the corresponding libraries' -version map script. -Secondly, the corresponding prototypes of those exported functions (in the -development header files), must be marked with the ``__rte_experimental`` tag -(see ``rte_compat.h``). -The DPDK build makefiles perform a check to ensure that the map file and the -C code reflect the same list of symbols. -This check can be circumvented by defining ``ALLOW_EXPERIMENTAL_API`` -during compilation in the corresponding library Makefile. - -In addition to tagging the code with ``__rte_experimental``, -the doxygen markup must also contain the EXPERIMENTAL string, -and the MAINTAINERS file should note the EXPERIMENTAL libraries. - -For removing the experimental tag associated with an API, deprecation notice -is not required. Though, an API should remain in experimental state for at least -one release. Thereafter, normal process of posting patch for review to mailing -list can be followed. - - -Library versioning ------------------- - -Downstreams might want to provide different DPDK releases at the same time to -support multiple consumers of DPDK linked against older and newer sonames. - -Also due to the interdependencies that DPDK libraries can have applications -might end up with an executable space in which multiple versions of a library -are mapped by ld.so. - -Think of LibA that got an ABI bump and LibB that did not get an ABI bump but is -depending on LibA. - -.. note:: - - Application - \-> LibA.old - \-> LibB.new -> LibA.new - -That is a conflict which can be avoided by setting ``CONFIG_RTE_MAJOR_ABI``. -If set, the value of ``CONFIG_RTE_MAJOR_ABI`` overwrites all - otherwise per -library - versions defined in the libraries ``LIBABIVER``. -An example might be ``CONFIG_RTE_MAJOR_ABI=16.11`` which will make all libraries -``librte.so.16.11`` instead of ``librte.so.``. - - -ABI versioning --------------- - -Versioning Macros -~~~~~~~~~~~~~~~~~ - -When a symbol is exported from a library to provide an API, it also provides a -calling convention (ABI) that is embodied in its name, return type and -arguments. Occasionally that function may need to change to accommodate new -functionality or behavior. When that occurs, it is desirable to allow for -backward compatibility for a time with older binaries that are dynamically -linked to the DPDK. - -To support backward compatibility the ``rte_compat.h`` -header file provides macros to use when updating exported functions. These -macros are used in conjunction with the ``rte__version.map`` file for -a given library to allow multiple versions of a symbol to exist in a shared -library so that older binaries need not be immediately recompiled. - -The macros exported are: - -* ``VERSION_SYMBOL(b, e, n)``: Creates a symbol version table entry binding - versioned symbol ``b@DPDK_n`` to the internal function ``b_e``. - -* ``BIND_DEFAULT_SYMBOL(b, e, n)``: Creates a symbol version entry instructing - the linker to bind references to symbol ``b`` to the internal symbol - ``b_e``. - -* ``MAP_STATIC_SYMBOL(f, p)``: Declare the prototype ``f``, and map it to the - fully qualified function ``p``, so that if a symbol becomes versioned, it - can still be mapped back to the public symbol name. - -Examples of ABI Macro use -^^^^^^^^^^^^^^^^^^^^^^^^^ - -Updating a public API -_____________________ - -Assume we have a function as follows - -.. code-block:: c - - /* - * Create an acl context object for apps to - * manipulate - */ - struct rte_acl_ctx * - rte_acl_create(const struct rte_acl_param *param) - { - ... - } - - -Assume that struct rte_acl_ctx is a private structure, and that a developer -wishes to enhance the acl api so that a debugging flag can be enabled on a -per-context basis. This requires an addition to the structure (which, being -private, is safe), but it also requires modifying the code as follows - -.. code-block:: c - - /* - * Create an acl context object for apps to - * manipulate - */ - struct rte_acl_ctx * - rte_acl_create(const struct rte_acl_param *param, int debug) - { - ... - } - - -Note also that, being a public function, the header file prototype must also be -changed, as must all the call sites, to reflect the new ABI footprint. We will -maintain previous ABI versions that are accessible only to previously compiled -binaries - -The addition of a parameter to the function is ABI breaking as the function is -public, and existing application may use it in its current form. However, the -compatibility macros in DPDK allow a developer to use symbol versioning so that -multiple functions can be mapped to the same public symbol based on when an -application was linked to it. To see how this is done, we start with the -requisite libraries version map file. Initially the version map file for the -acl library looks like this - -.. code-block:: none - - DPDK_2.0 { - global: - - rte_acl_add_rules; - rte_acl_build; - rte_acl_classify; - rte_acl_classify_alg; - rte_acl_classify_scalar; - rte_acl_create; - rte_acl_dump; - rte_acl_find_existing; - rte_acl_free; - rte_acl_ipv4vlan_add_rules; - rte_acl_ipv4vlan_build; - rte_acl_list_dump; - rte_acl_reset; - rte_acl_reset_rules; - rte_acl_set_ctx_classify; - - local: *; - }; - -This file needs to be modified as follows - -.. code-block:: none - - DPDK_2.0 { - global: - - rte_acl_add_rules; - rte_acl_build; - rte_acl_classify; - rte_acl_classify_alg; - rte_acl_classify_scalar; - rte_acl_create; - rte_acl_dump; - rte_acl_find_existing; - rte_acl_free; - rte_acl_ipv4vlan_add_rules; - rte_acl_ipv4vlan_build; - rte_acl_list_dump; - rte_acl_reset; - rte_acl_reset_rules; - rte_acl_set_ctx_classify; - - local: *; - }; - - DPDK_2.1 { - global: - rte_acl_create; - - } DPDK_2.0; - -The addition of the new block tells the linker that a new version node is -available (DPDK_2.1), which contains the symbol rte_acl_create, and inherits the -symbols from the DPDK_2.0 node. This list is directly translated into a list of -exported symbols when DPDK is compiled as a shared library - -Next, we need to specify in the code which function map to the rte_acl_create -symbol at which versions. First, at the site of the initial symbol definition, -we need to update the function so that it is uniquely named, and not in conflict -with the public symbol name - -.. code-block:: c - - struct rte_acl_ctx * - -rte_acl_create(const struct rte_acl_param *param) - +rte_acl_create_v20(const struct rte_acl_param *param) - { - size_t sz; - struct rte_acl_ctx *ctx; - ... - -Note that the base name of the symbol was kept intact, as this is conducive to -the macros used for versioning symbols. That is our next step, mapping this new -symbol name to the initial symbol name at version node 2.0. Immediately after -the function, we add this line of code - -.. code-block:: c - - VERSION_SYMBOL(rte_acl_create, _v20, 2.0); - -Remembering to also add the rte_compat.h header to the requisite c file where -these changes are being made. The above macro instructs the linker to create a -new symbol ``rte_acl_create@DPDK_2.0``, which matches the symbol created in older -builds, but now points to the above newly named function. We have now mapped -the original rte_acl_create symbol to the original function (but with a new -name) - -Next, we need to create the 2.1 version of the symbol. We create a new function -name, with a different suffix, and implement it appropriately - -.. code-block:: c - - struct rte_acl_ctx * - rte_acl_create_v21(const struct rte_acl_param *param, int debug); - { - struct rte_acl_ctx *ctx = rte_acl_create_v20(param); - - ctx->debug = debug; - - return ctx; - } - -This code serves as our new API call. Its the same as our old call, but adds -the new parameter in place. Next we need to map this function to the symbol -``rte_acl_create@DPDK_2.1``. To do this, we modify the public prototype of the call -in the header file, adding the macro there to inform all including applications, -that on re-link, the default rte_acl_create symbol should point to this -function. Note that we could do this by simply naming the function above -rte_acl_create, and the linker would chose the most recent version tag to apply -in the version script, but we can also do this in the header file - -.. code-block:: c - - struct rte_acl_ctx * - -rte_acl_create(const struct rte_acl_param *param); - +rte_acl_create(const struct rte_acl_param *param, int debug); - +BIND_DEFAULT_SYMBOL(rte_acl_create, _v21, 2.1); - -The BIND_DEFAULT_SYMBOL macro explicitly tells applications that include this -header, to link to the rte_acl_create_v21 function and apply the DPDK_2.1 -version node to it. This method is more explicit and flexible than just -re-implementing the exact symbol name, and allows for other features (such as -linking to the old symbol version by default, when the new ABI is to be opt-in -for a period. - -One last thing we need to do. Note that we've taken what was a public symbol, -and duplicated it into two uniquely and differently named symbols. We've then -mapped each of those back to the public symbol ``rte_acl_create`` with different -version tags. This only applies to dynamic linking, as static linking has no -notion of versioning. That leaves this code in a position of no longer having a -symbol simply named ``rte_acl_create`` and a static build will fail on that -missing symbol. - -To correct this, we can simply map a function of our choosing back to the public -symbol in the static build with the ``MAP_STATIC_SYMBOL`` macro. Generally the -assumption is that the most recent version of the symbol is the one you want to -map. So, back in the C file where, immediately after ``rte_acl_create_v21`` is -defined, we add this - -.. code-block:: c - - struct rte_acl_ctx * - rte_acl_create_v21(const struct rte_acl_param *param, int debug) - { - ... - } - MAP_STATIC_SYMBOL(struct rte_acl_ctx *rte_acl_create(const struct rte_acl_param *param, int debug), rte_acl_create_v21); - -That tells the compiler that, when building a static library, any calls to the -symbol ``rte_acl_create`` should be linked to ``rte_acl_create_v21`` - -That's it, on the next shared library rebuild, there will be two versions of -rte_acl_create, an old DPDK_2.0 version, used by previously built applications, -and a new DPDK_2.1 version, used by future built applications. - - -Deprecating part of a public API -________________________________ - -Lets assume that you've done the above update, and after a few releases have -passed you decide you would like to retire the old version of the function. -After having gone through the ABI deprecation announcement process, removal is -easy. Start by removing the symbol from the requisite version map file: - -.. code-block:: none - - DPDK_2.0 { - global: - - rte_acl_add_rules; - rte_acl_build; - rte_acl_classify; - rte_acl_classify_alg; - rte_acl_classify_scalar; - rte_acl_dump; - - rte_acl_create - rte_acl_find_existing; - rte_acl_free; - rte_acl_ipv4vlan_add_rules; - rte_acl_ipv4vlan_build; - rte_acl_list_dump; - rte_acl_reset; - rte_acl_reset_rules; - rte_acl_set_ctx_classify; - - local: *; - }; - - DPDK_2.1 { - global: - rte_acl_create; - } DPDK_2.0; - - -Next remove the corresponding versioned export. - -.. code-block:: c - - -VERSION_SYMBOL(rte_acl_create, _v20, 2.0); - - -Note that the internal function definition could also be removed, but its used -in our example by the newer version _v21, so we leave it in place. This is a -coding style choice. - -Lastly, we need to bump the LIBABIVER number for this library in the Makefile to -indicate to applications doing dynamic linking that this is a later, and -possibly incompatible library version: - -.. code-block:: c - - -LIBABIVER := 1 - +LIBABIVER := 2 - -Deprecating an entire ABI version -_________________________________ - -While removing a symbol from and ABI may be useful, it is often more practical -to remove an entire version node at once. If a version node completely -specifies an API, then removing part of it, typically makes it incomplete. In -those cases it is better to remove the entire node - -To do this, start by modifying the version map file, such that all symbols from -the node to be removed are merged into the next node in the map - -In the case of our map above, it would transform to look as follows - -.. code-block:: none - - DPDK_2.1 { - global: - - rte_acl_add_rules; - rte_acl_build; - rte_acl_classify; - rte_acl_classify_alg; - rte_acl_classify_scalar; - rte_acl_dump; - rte_acl_create - rte_acl_find_existing; - rte_acl_free; - rte_acl_ipv4vlan_add_rules; - rte_acl_ipv4vlan_build; - rte_acl_list_dump; - rte_acl_reset; - rte_acl_reset_rules; - rte_acl_set_ctx_classify; - - local: *; - }; - -Then any uses of BIND_DEFAULT_SYMBOL that pointed to the old node should be -updated to point to the new version node in any header files for all affected -symbols. - -.. code-block:: c - - -BIND_DEFAULT_SYMBOL(rte_acl_create, _v20, 2.0); - +BIND_DEFAULT_SYMBOL(rte_acl_create, _v21, 2.1); - -Lastly, any VERSION_SYMBOL macros that point to the old version node should be -removed, taking care to keep, where need old code in place to support newer -versions of the symbol. - - -Running the ABI Validator -------------------------- - -The ``devtools`` directory in the DPDK source tree contains a utility program, -``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI -Compliance Checker -`_. - -This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper`` -utilities which can be installed via a package manager. For example:: - - sudo yum install abi-compliance-checker - sudo yum install abi-dumper - -The syntax of the ``validate-abi.sh`` utility is:: - - ./devtools/validate-abi.sh - -Where ``REV1`` and ``REV2`` are valid gitrevisions(7) -https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html -on the local repo. - -For example:: - - # Check between the previous and latest commit: - ./devtools/validate-abi.sh HEAD~1 HEAD - - # Check on a specific compilation target: - ./devtools/validate-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD - - # Check between two tags: - ./devtools/validate-abi.sh v2.0.0 v2.1.0 - - # Check between git master and local topic-branch "vhost-hacking": - ./devtools/validate-abi.sh master vhost-hacking - -After the validation script completes (it can take a while since it need to -compile both tags) it will create compatibility reports in the -``./abi-check/compat_report`` directory. Listed incompatibilities can be found -as follows:: - - grep -lr Incompatible abi-check/compat_reports/ -- 2.7.4