From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2215E4574A; Tue, 6 Aug 2024 13:18:44 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0DD8842E5D; Tue, 6 Aug 2024 13:18:10 +0200 (CEST) Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) by mails.dpdk.org (Postfix) with ESMTP id A3F6A42E55 for ; Tue, 6 Aug 2024 13:18:06 +0200 (CEST) Received: by mail-lf1-f51.google.com with SMTP id 2adb3069b0e04-52f04150796so955279e87.3 for ; Tue, 06 Aug 2024 04:18:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1722943086; x=1723547886; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IWsoya+Fs/NLJxnz4B5CgC1XF3IqTYG/fwnHFmHs7Ps=; b=afzM1m061rEvw0NBgO8vF87f4yZ2GtW1W65vM4lVvp7zGDxEzfmfWhhXA4plJwoQjq l45WqtnQf4OxKDM1xTHtUoarFc1aZMSdwZ5JSr9yq51NH7KYb2XS+RLpaAM9uJ68QWv8 pWvdSpAH2uJ1Ger1UkbKQ4SOAjWtBop++t9SE+HVJek1pLg5Ka6Lfa/pbwgkoJddqTgr 010cOPY5SZb2wnREr/q5V8e2SyDkMbSOd8Po/0OQZExCyo5BUOE3MHWQkMirtBjkICZW 7JtW6iICu7MP0ADCvKhFZAdwEBx8beUc9cbXr2npPp9JlQzqHg+tAnPwzQx4xcUMbJZl pguA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722943086; x=1723547886; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IWsoya+Fs/NLJxnz4B5CgC1XF3IqTYG/fwnHFmHs7Ps=; b=ATWCTpYWDe8Zd2Gzn7I40RQV54+gcq1AjZz/AA3PysW+XkYMLM7Jq4NVcwJSn0YKfv yhIH7oFIjFkCqvp/S/3u+rGVT+qq3hBGr7bS/J8veNoazsNarQE5TzAe6L1rEBx/1svA ndMTvNZuYt0xRXkXn1wCTrbofooqUqyH66PiWLl3hNDRNa34UGNzTdhFkZJ/K5gnww/C FtggA3t/P5j1An8bCNbwxewm9pv03xHDr6xE44BgNLXafRN8dMChaMcJtTmeDrh9YzUM NS0YCELGtfdZqw8LO2zAzAAH554YUs3cWJ9YjfjImkmfv70jYGHu16CqaKFJ609K3ZNM 0QFw== X-Gm-Message-State: AOJu0Yx+LeDypP/4uzDR1e6lqR6DYOUTnK1Sy8+VG0CK6Vfg0ZCUQiPH aUwipbO8lo93ghr1DuzSbydY0fMZAdvR3jWISbAu/ePpWaI81vt4yEfcy53Nr7o= X-Google-Smtp-Source: AGHT+IGJxLN1FWweuhrmD8U8x8sAiIkjYIm3rdWU+QHnXgEHlqKPHYx18yeqKyxE+DeWkUXjlp7KyA== X-Received: by 2002:ac2:4e0b:0:b0:52e:faf0:40c with SMTP id 2adb3069b0e04-530bb367688mr10728038e87.3.1722943085708; Tue, 06 Aug 2024 04:18:05 -0700 (PDT) Received: from jlinkes-PT-Latitude-5530.pantheon.local (81.89.53.154.host.vnet.sk. [81.89.53.154]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5bb885c8b57sm2624520a12.96.2024.08.06.04.18.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 04:18:04 -0700 (PDT) From: =?UTF-8?q?Juraj=20Linke=C5=A1?= To: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, bruce.richardson@intel.com, jspewock@iol.unh.edu, probb@iol.unh.edu, paul.szczepanek@arm.com, Luca.Vizzarro@arm.com, npratte@iol.unh.edu Cc: dev@dpdk.org, =?UTF-8?q?Juraj=20Linke=C5=A1?= Subject: [PATCH v14 6/6] dts: add API doc generation Date: Tue, 6 Aug 2024 13:17:56 +0200 Message-Id: <20240806111756.314055-7-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240806111756.314055-1-juraj.linkes@pantheon.tech> References: <20231115133606.42081-1-juraj.linkes@pantheon.tech> <20240806111756.314055-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The tool used to generate DTS API docs is Sphinx, which is already in use in DPDK. The same configuration is used to preserve style with one DTS-specific configuration (so that the DPDK docs are unchanged) that modifies how the sidebar displays the content. Sphinx generates the documentation from Python docstrings. The docstring format is the Google format [0] which requires the sphinx.ext.napoleon extension. The other extension, sphinx.ext.intersphinx, enables linking to objects in external documentations, such as the Python documentation. There is one requirement for building DTS docs - the same Python version as DTS or higher, because Sphinx's autodoc extension imports the code. The dependencies needed to import the code don't have to be satisfied, as the autodoc extension allows us to mock the imports. The missing packages are taken from the DTS pyproject.toml file. The generated DTS API docs are linked with the DPDK API docs according to their placement after installing them with 'meson install'. However, the build path differs from the install path, requiring a symlink from DPDK API doc build path to DTS API build path to produce the proper link in the build directory. [0] https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings Signed-off-by: Juraj Linkeš --- buildtools/call-sphinx-build.py | 10 ++- buildtools/get-dts-deps.py | 78 +++++++++++++++++++++++ buildtools/meson.build | 1 + doc/api/doxy-api-index.md | 3 + doc/api/doxy-api.conf.in | 2 + doc/api/meson.build | 1 + doc/guides/conf.py | 41 +++++++++++- doc/guides/contributing/documentation.rst | 2 + doc/guides/contributing/patches.rst | 4 ++ doc/guides/meson.build | 1 + doc/guides/tools/dts.rst | 39 +++++++++++- dts/doc/meson.build | 43 +++++++++++++ dts/meson.build | 15 +++++ meson.build | 1 + 14 files changed, 238 insertions(+), 3 deletions(-) create mode 100755 buildtools/get-dts-deps.py create mode 100644 dts/doc/meson.build create mode 100644 dts/meson.build diff --git a/buildtools/call-sphinx-build.py b/buildtools/call-sphinx-build.py index 623e7363ee..5dd59907cd 100755 --- a/buildtools/call-sphinx-build.py +++ b/buildtools/call-sphinx-build.py @@ -15,6 +15,11 @@ # set the version in environment for sphinx to pick up os.environ['DPDK_VERSION'] = version +conf_src = src +if src.find('dts') != -1: + if '-c' in extra_args: + conf_src = extra_args[extra_args.index('-c') + 1] + os.environ['DTS_BUILD'] = "y" sphinx_cmd = [sphinx] + extra_args @@ -23,6 +28,9 @@ for root, dirs, files in os.walk(src): srcfiles.extend([join(root, f) for f in files]) +if not os.path.exists(dst): + os.makedirs(dst) + # run sphinx, putting the html output in a "html" directory with open(join(dst, 'sphinx_html.out'), 'w') as out: process = run(sphinx_cmd + ['-b', 'html', src, join(dst, 'html')], @@ -34,7 +42,7 @@ # copy custom CSS file css = 'custom.css' -src_css = join(src, css) +src_css = join(conf_src, css) dst_css = join(dst, 'html', '_static', 'css', css) if not os.path.exists(dst_css) or not filecmp.cmp(src_css, dst_css): os.makedirs(os.path.dirname(dst_css), exist_ok=True) diff --git a/buildtools/get-dts-deps.py b/buildtools/get-dts-deps.py new file mode 100755 index 0000000000..309b83cb5c --- /dev/null +++ b/buildtools/get-dts-deps.py @@ -0,0 +1,78 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2024 PANTHEON.tech s.r.o. +# + +"""Utilities for DTS dependencies. + +The module can be used as an executable script, +which verifies that the running Python version meets the version requirement of DTS. +The script returns the standard exit codes in this mode (0 is success, 1 is failure). + +The module also contains a function, get_missing_imports, +which looks for runtime and doc generation dependencies in the DTS pyproject.toml file +a returns a list of module names used in an import statement that are missing. +""" + +import configparser +import importlib.metadata +import importlib.util +import os.path +import platform + +_VERSION_COMPARISON_CHARS = '^<>=' +_EXTRA_DEPS = {'invoke': '>=1.3', 'paramiko': '>=2.4'} +_DPDK_ROOT = os.path.dirname(os.path.dirname(__file__)) +_DTS_DEP_FILE_PATH = os.path.join(_DPDK_ROOT, 'dts', 'pyproject.toml') + + +def _get_version_tuple(version_str): + return tuple(map(int, version_str.split("."))) + + +def _get_dependencies(cfg_file_path): + cfg = configparser.ConfigParser() + with open(cfg_file_path) as f: + dts_deps_file_str = f.read() + dts_deps_file_str = dts_deps_file_str.replace("\n]", "]") + cfg.read_string(dts_deps_file_str) + + deps_section = cfg['tool.poetry.dependencies'] + deps = {dep: deps_section[dep].strip('"\'') for dep in deps_section} + doc_deps_section = cfg['tool.poetry.group.docs.dependencies'] + doc_deps = {dep: doc_deps_section[dep].strip("\"'") for dep in doc_deps_section} + + return deps | doc_deps + + +def get_missing_imports(): + missing_imports = [] + req_deps = _get_dependencies(_DTS_DEP_FILE_PATH) + req_deps.pop('python') + + for req_dep, req_ver in (req_deps | _EXTRA_DEPS).items(): + try: + req_ver = _get_version_tuple(req_ver.strip(_VERSION_COMPARISON_CHARS)) + found_dep_ver = _get_version_tuple(importlib.metadata.version(req_dep)) + if found_dep_ver < req_ver: + print( + f'The version "{found_dep_ver}" of package "{req_dep}" ' + f'is lower than required "{req_ver}".' + ) + except importlib.metadata.PackageNotFoundError: + print(f'Package "{req_dep}" not found.') + missing_imports.append(req_dep.lower().replace('-', '_')) + + return missing_imports + + +if __name__ == '__main__': + python_version = _get_dependencies(_DTS_DEP_FILE_PATH).pop('python') + if python_version: + sys_ver = _get_version_tuple(platform.python_version()) + req_ver = _get_version_tuple(python_version.strip(_VERSION_COMPARISON_CHARS)) + if sys_ver < req_ver: + print( + f'The available Python version "{sys_ver}" is lower than required "{req_ver}".' + ) + exit(1) diff --git a/buildtools/meson.build b/buildtools/meson.build index 3adf34e1a8..599653bea4 100644 --- a/buildtools/meson.build +++ b/buildtools/meson.build @@ -24,6 +24,7 @@ get_numa_count_cmd = py3 + files('get-numa-count.py') get_test_suites_cmd = py3 + files('get-test-suites.py') has_hugepages_cmd = py3 + files('has-hugepages.py') cmdline_gen_cmd = py3 + files('dpdk-cmdline-gen.py') +get_dts_deps = py3 + files('get-dts-deps.py') # install any build tools that end-users might want also install_data([ diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md index f9f0300126..ab223bcdf7 100644 --- a/doc/api/doxy-api-index.md +++ b/doc/api/doxy-api-index.md @@ -245,3 +245,6 @@ The public API headers are grouped by topics: [experimental APIs](@ref rte_compat.h), [ABI versioning](@ref rte_function_versioning.h), [version](@ref rte_version.h) + +- **tests**: + [**DTS**](@dts_api_main_page) diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in index a8823c046f..c94f02d411 100644 --- a/doc/api/doxy-api.conf.in +++ b/doc/api/doxy-api.conf.in @@ -124,6 +124,8 @@ SEARCHENGINE = YES SORT_MEMBER_DOCS = NO SOURCE_BROWSER = YES +ALIASES = "dts_api_main_page=@DTS_API_MAIN_PAGE@" + EXAMPLE_PATH = @TOPDIR@/examples EXAMPLE_PATTERNS = *.c EXAMPLE_RECURSIVE = YES diff --git a/doc/api/meson.build b/doc/api/meson.build index b828b1ed66..b893931b92 100644 --- a/doc/api/meson.build +++ b/doc/api/meson.build @@ -41,6 +41,7 @@ cdata.set('WARN_AS_ERROR', 'NO') if get_option('werror') cdata.set('WARN_AS_ERROR', 'YES') endif +cdata.set('DTS_API_MAIN_PAGE', join_paths('..', 'dts', 'html', 'index.html')) # configure HTML Doxygen run html_cdata = configuration_data() diff --git a/doc/guides/conf.py b/doc/guides/conf.py index 0f7ff5282d..eab3387874 100644 --- a/doc/guides/conf.py +++ b/doc/guides/conf.py @@ -10,7 +10,7 @@ from os.path import basename from os.path import dirname from os.path import join as path_join -from sys import argv, stderr +from sys import argv, stderr, path import configparser @@ -24,6 +24,45 @@ file=stderr) pass +# Napoleon enables the Google format of Python doscstrings, used in DTS. +# Intersphinx allows linking to external projects, such as Python docs, also used in DTS. +extensions = ['sphinx.ext.napoleon', 'sphinx.ext.intersphinx'] + +# DTS Python docstring options. +autodoc_default_options = { + 'members': True, + 'member-order': 'bysource', + 'show-inheritance': True, +} +autodoc_class_signature = 'separated' +autodoc_typehints = 'both' +autodoc_typehints_format = 'short' +autodoc_typehints_description_target = 'documented' +napoleon_numpy_docstring = False +napoleon_attr_annotations = True +napoleon_preprocess_types = True +add_module_names = False +toc_object_entries = True +toc_object_entries_show_parents = 'hide' +intersphinx_mapping = {'python': ('https://docs.python.org/3', None)} + +if environ.get('DTS_BUILD'): + # Add path to DTS sources so that Sphinx can find them. + dpdk_root = dirname(dirname(dirname(__file__))) + path.append(path_join(dpdk_root, 'dts')) + + # Get missing DTS dependencies. Add path to buildtools to find the get_missing_imports function. + path.append(path_join(dpdk_root, 'buildtools')) + import importlib + # Ignore missing imports from DTS dependencies. + autodoc_mock_imports = importlib.import_module('get-dts-deps').get_missing_imports() + + # DTS Sidebar config. + html_theme_options = { + 'collapse_navigation': False, + 'navigation_depth': -1, # unlimited depth + } + stop_on_error = ('-W' in argv) project = 'Data Plane Development Kit' diff --git a/doc/guides/contributing/documentation.rst b/doc/guides/contributing/documentation.rst index 68454ae0d5..7b287ce631 100644 --- a/doc/guides/contributing/documentation.rst +++ b/doc/guides/contributing/documentation.rst @@ -133,6 +133,8 @@ added to by the developer. Building the Documentation -------------------------- +.. _doc_dependencies: + Dependencies ~~~~~~~~~~~~ diff --git a/doc/guides/contributing/patches.rst b/doc/guides/contributing/patches.rst index 04c66bebc4..6629928bee 100644 --- a/doc/guides/contributing/patches.rst +++ b/doc/guides/contributing/patches.rst @@ -499,6 +499,10 @@ The script usage is:: For both of the above scripts, the -n option is used to specify a number of commits from HEAD, and the -r option allows the user specify a ``git log`` range. +Additionally, when contributing to the DTS tool, patches should also be checked using +the ``dts-check-format.sh`` script in the ``devtools`` directory of the DPDK repo. +To run the script, extra :ref:`Python dependencies ` are needed. + .. _contrib_check_compilation: Checking Compilation diff --git a/doc/guides/meson.build b/doc/guides/meson.build index f8bbfba9f5..b34b7b8eb0 100644 --- a/doc/guides/meson.build +++ b/doc/guides/meson.build @@ -1,6 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Intel Corporation +doc_guides_source_dir = meson.current_source_dir() sphinx = find_program('sphinx-build', required: get_option('enable_docs')) if not sphinx.found() diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst index 515b15e4d8..bd715f8072 100644 --- a/doc/guides/tools/dts.rst +++ b/doc/guides/tools/dts.rst @@ -54,6 +54,7 @@ DTS uses Poetry as its Python dependency management. Python build/development and runtime environments are the same and DTS development environment, DTS runtime environment or just plain DTS environment are used interchangeably. +.. _dts_deps: Setting up DTS environment ~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -291,8 +292,15 @@ When adding code to the DTS framework, pay attention to the rest of the code and try not to divert much from it. The :ref:`DTS developer tools ` will issue warnings when some of the basics are not met. +You should also build the :ref:`API documentation ` +to address any issues found during the build. -The code must be properly documented with docstrings. +The API documentation, which is a helpful reference when developing, may be accessed +in the code directly or generated with the :ref:`API docs build steps `. +When adding new files or modifying the directory structure, +the corresponding changes must be made to DTS api doc sources in ``dts/doc``. + +Speaking of which, the code must be properly documented with docstrings. The style must conform to the `Google style `_. See an example of the style `here @@ -427,6 +435,35 @@ the DTS code check and format script. Refer to the script for usage: ``devtools/dts-check-format.sh -h``. +.. _building_api_docs: + +Building DTS API docs +--------------------- + +The documentation is built using the standard DPDK build system. +See :doc:`../linux_gsg/build_dpdk` for more details on compiling DPDK with meson. + +The :ref:`doc build dependencies ` may be installed with Poetry: + +.. code-block:: console + + poetry install --no-root --only docs + poetry install --no-root --with docs # an alternative that will also install DTS dependencies + poetry shell + +After executing the meson command, build the documentation with: + +.. code-block:: console + + ninja -C build dts-doc + +The output is generated in ``build/doc/api/dts/html``. + +.. note:: + + Make sure to fix any Sphinx warnings when adding or updating docstrings. + + Configuration Schema -------------------- diff --git a/dts/doc/meson.build b/dts/doc/meson.build new file mode 100644 index 0000000000..d48a7f2003 --- /dev/null +++ b/dts/doc/meson.build @@ -0,0 +1,43 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2023 PANTHEON.tech s.r.o. + +sphinx = find_program('sphinx-build', required: get_option('enable_docs')) +if not sphinx.found() + subdir_done() +endif + +python_ver_satisfied = run_command(get_dts_deps).returncode() +if python_ver_satisfied != 0 + subdir_done() +endif + +dts_doc_api_build_dir = join_paths(doc_api_build_dir, 'dts') + +extra_sphinx_args = ['-E', '-c', doc_guides_source_dir] +if get_option('werror') + extra_sphinx_args += '-W' +endif + +htmldir = join_paths(get_option('datadir'), 'doc', 'dpdk', 'dts') +dts_api_html = custom_target('dts_api_html', + output: 'html', + command: [sphinx_wrapper, sphinx, meson.project_version(), + meson.current_source_dir(), meson.current_build_dir(), extra_sphinx_args], + build_by_default: get_option('enable_docs'), + install: get_option('enable_docs'), + install_dir: htmldir) + +doc_targets += dts_api_html +doc_target_names += 'DTS_API_HTML' + +build_html_dir = join_paths(meson.current_build_dir(), 'html') +dts_api_symlink = custom_target('dts_api_symlink', + output: 'symlink', + depends: dts_api_html, + command: ['mkdir', '-p', dts_doc_api_build_dir, '&&', + 'ln', '-sf', build_html_dir, dts_doc_api_build_dir], + build_by_default: get_option('enable_docs'), + install: false) + +doc_targets += dts_api_symlink +doc_target_names += 'DTS_API_SYMLINK' diff --git a/dts/meson.build b/dts/meson.build new file mode 100644 index 0000000000..6ed3c93fe1 --- /dev/null +++ b/dts/meson.build @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2023 PANTHEON.tech s.r.o. + +doc_targets = [] +doc_target_names = [] + +subdir('doc') + +if doc_targets.length() == 0 + message = 'No docs targets found' +else + message = 'Built docs:' +endif +run_target('dts-doc', command: [echo, message, doc_target_names], + depends: doc_targets) diff --git a/meson.build b/meson.build index 8b248d4505..835973a0ce 100644 --- a/meson.build +++ b/meson.build @@ -87,6 +87,7 @@ subdir('app') # build docs subdir('doc') +subdir('dts') # build any examples explicitly requested - useful for developers - and # install any example code into the appropriate install path -- 2.34.1