Re: [PATCH v19 0/5] DTS API docs generation

["Juraj Linkeš" <juraj.linkes@pantheon.tech>]

On 21. 8. 2024 17:02, Juraj Linkeš wrote:
> The generation is done with Sphinx, which DPDK already uses, with
> slightly modified configuration of the sidebar present in an if block.
> 
> DTS dependencies do not need to be installed, but there is the option to
> install doc build dependencies with Poetry:
> poetry install --with docs
> 
> The build itself may be run with:
> meson setup <meson_build_dir> -Denable_docs=true
> ninja -C <meson_build_dir>
> 
> The above will do a full DPDK build with docs. To build just docs:
> meson setup <meson_build_dir>
> ninja -C <meson_build_dir> doc
> 
> Python3.10 is required to build the DTS API docs.
> 
> The patchset contains the .rst sources which Sphinx uses to generate the
> html pages. These were first generated with the sphinx-apidoc utility
> and modified to provide a better look. The documentation just doesn't
> look that good without the modifications and there isn't enough
> configuration options to achieve that without manual changes to the .rst
> files. This introduces extra maintenance which involves adding new .rst
> files when a new Python module is added or changing the .rst structure
> if the Python directory/file structure is changed (moved, renamed
> files). This small maintenance burden is outweighed by the flexibility
> afforded by the ability to make manual changes to the .rst files.
> 
> v10:
> Fix dts doc generation issue: Only copy the custom rss file if it exists.
> 
> v11:
> Added the config option autodoc_mock_imports, which eliminates the need
> for DTS dependencies. Added a script that find out which imports need to
> be added to autodoc_mock_imports. The script also check the required
> Python version for building DTS docs.
> Removed tags from the two affected patches which will need to be
> reviewed again.
> 
> v12:
> Added paramiko to the required dependencies of get-dts-deps.py.
> 
> v13:
> Fixed build error:
> TypeError: unsupported operand type(s) for |: 'NoneType' and 'Transport'
> 
> v14:
> Fixed install error:
> ERROR: File 'dts/doc/html' could not be found
> This required me to put the built docs into dts/doc which is outside the
> DPDK API doc dir, resulting in linking between DPDK and DTS api docs not
> working properly. I addressed this by adding a symlink to the build dir.
> This way the link works after installing the docs and the symlink is
> just one extra file in the build dir.
> 
> v15:
> Moved DTS API sources to doc/api/dts. This simplifies a lot of things in
> the build, but mainly makes a lot of sense. Now the source, build and
> install paths are the same so there isn't any need for any symlinks or
> other workarounds.
> Also added a symlink to the custom.css file so that it works with
> call-sphinx-build.py without any modifications.
> 
> v16:
> Renamed the dependency python file to get-dts-runtime-deps.py a modified
> it to only get runtime dependencies. We don't need to check docs
> dependencies (Sphinx) as we don't need to mock those.
> Also moved all new Sphinx configuration into the DTS if branch to make
> sure it won't ever affect the DPDK doc build.
> 
> v17:
> Removed the dts-doc build target to mirror the functionality of using
> -Denable_docs=true.
> Moved DTS-specific meson build code to doc/api/dts/meson.build.
> Added comments to get_missing_imports() and the top level docstring of
> get-dts-runtime-deps.py to explain the function is there to be imported.
> 
> v18:
> Added PyYAML to get-dts-runtime-deps.py.
> 
> v19:
> Fixed a regression in get-dts-runtime-deps.py introduced in v18:
> AttributeError: 'dict' object has no attribute 'strip'
> 
> Juraj Linkeš (5):
>    dts: update params and parser docstrings
>    dts: replace the or operator in third party types
>    dts: add doc generation dependencies
>    dts: add API doc sources
>    dts: add API doc generation
> 
>   buildtools/call-sphinx-build.py               |   2 +
>   buildtools/get-dts-runtime-deps.py            |  95 ++++
>   buildtools/meson.build                        |   1 +
>   doc/api/doxy-api-index.md                     |   3 +
>   doc/api/doxy-api.conf.in                      |   2 +
>   doc/api/dts/conf_yaml_schema.json             |   1 +
>   doc/api/dts/custom.css                        |   1 +
>   doc/api/dts/framework.config.rst              |  12 +
>   doc/api/dts/framework.config.types.rst        |   6 +
>   doc/api/dts/framework.exception.rst           |   6 +
>   doc/api/dts/framework.logger.rst              |   6 +
>   doc/api/dts/framework.params.eal.rst          |   6 +
>   doc/api/dts/framework.params.rst              |  14 +
>   doc/api/dts/framework.params.testpmd.rst      |   6 +
>   doc/api/dts/framework.params.types.rst        |   6 +
>   doc/api/dts/framework.parser.rst              |   6 +
>   .../framework.remote_session.dpdk_shell.rst   |   6 +
>   ...ote_session.interactive_remote_session.rst |   6 +
>   ...ework.remote_session.interactive_shell.rst |   6 +
>   .../framework.remote_session.python_shell.rst |   6 +
>   ...ramework.remote_session.remote_session.rst |   6 +
>   doc/api/dts/framework.remote_session.rst      |  18 +
>   .../framework.remote_session.ssh_session.rst  |   6 +
>   ...framework.remote_session.testpmd_shell.rst |   6 +
>   doc/api/dts/framework.runner.rst              |   6 +
>   doc/api/dts/framework.settings.rst            |   6 +
>   doc/api/dts/framework.test_result.rst         |   6 +
>   doc/api/dts/framework.test_suite.rst          |   6 +
>   doc/api/dts/framework.testbed_model.cpu.rst   |   6 +
>   .../framework.testbed_model.linux_session.rst |   6 +
>   doc/api/dts/framework.testbed_model.node.rst  |   6 +
>   .../framework.testbed_model.os_session.rst    |   6 +
>   doc/api/dts/framework.testbed_model.port.rst  |   6 +
>   .../framework.testbed_model.posix_session.rst |   6 +
>   doc/api/dts/framework.testbed_model.rst       |  26 +
>   .../dts/framework.testbed_model.sut_node.rst  |   6 +
>   .../dts/framework.testbed_model.tg_node.rst   |   6 +
>   ..._generator.capturing_traffic_generator.rst |   6 +
>   ...mework.testbed_model.traffic_generator.rst |  14 +
>   ....testbed_model.traffic_generator.scapy.rst |   6 +
>   ...el.traffic_generator.traffic_generator.rst |   6 +
>   ...framework.testbed_model.virtual_device.rst |   6 +
>   doc/api/dts/framework.utils.rst               |   6 +
>   doc/api/dts/index.rst                         |  43 ++
>   doc/api/dts/meson.build                       |  31 ++
>   doc/api/meson.build                           |   6 +-
>   doc/guides/conf.py                            |  44 +-
>   doc/guides/contributing/documentation.rst     |   2 +
>   doc/guides/contributing/patches.rst           |   4 +
>   doc/guides/tools/dts.rst                      |  39 +-
>   doc/meson.build                               |   1 +
>   dts/framework/params/__init__.py              |   4 +-
>   dts/framework/params/eal.py                   |   7 +-
>   dts/framework/params/types.py                 |   3 +-
>   dts/framework/parser.py                       |   4 +-
>   .../interactive_remote_session.py             |   3 +-
>   dts/poetry.lock                               | 521 +++++++++++++++++-
>   dts/pyproject.toml                            |   8 +
>   58 files changed, 1072 insertions(+), 23 deletions(-)
>   create mode 100755 buildtools/get-dts-runtime-deps.py
>   create mode 120000 doc/api/dts/conf_yaml_schema.json
>   create mode 120000 doc/api/dts/custom.css
>   create mode 100644 doc/api/dts/framework.config.rst
>   create mode 100644 doc/api/dts/framework.config.types.rst
>   create mode 100644 doc/api/dts/framework.exception.rst
>   create mode 100644 doc/api/dts/framework.logger.rst
>   create mode 100644 doc/api/dts/framework.params.eal.rst
>   create mode 100644 doc/api/dts/framework.params.rst
>   create mode 100644 doc/api/dts/framework.params.testpmd.rst
>   create mode 100644 doc/api/dts/framework.params.types.rst
>   create mode 100644 doc/api/dts/framework.parser.rst
>   create mode 100644 doc/api/dts/framework.remote_session.dpdk_shell.rst
>   create mode 100644 doc/api/dts/framework.remote_session.interactive_remote_session.rst
>   create mode 100644 doc/api/dts/framework.remote_session.interactive_shell.rst
>   create mode 100644 doc/api/dts/framework.remote_session.python_shell.rst
>   create mode 100644 doc/api/dts/framework.remote_session.remote_session.rst
>   create mode 100644 doc/api/dts/framework.remote_session.rst
>   create mode 100644 doc/api/dts/framework.remote_session.ssh_session.rst
>   create mode 100644 doc/api/dts/framework.remote_session.testpmd_shell.rst
>   create mode 100644 doc/api/dts/framework.runner.rst
>   create mode 100644 doc/api/dts/framework.settings.rst
>   create mode 100644 doc/api/dts/framework.test_result.rst
>   create mode 100644 doc/api/dts/framework.test_suite.rst
>   create mode 100644 doc/api/dts/framework.testbed_model.cpu.rst
>   create mode 100644 doc/api/dts/framework.testbed_model.linux_session.rst
>   create mode 100644 doc/api/dts/framework.testbed_model.node.rst
>   create mode 100644 doc/api/dts/framework.testbed_model.os_session.rst
>   create mode 100644 doc/api/dts/framework.testbed_model.port.rst
>   create mode 100644 doc/api/dts/framework.testbed_model.posix_session.rst
>   create mode 100644 doc/api/dts/framework.testbed_model.rst
>   create mode 100644 doc/api/dts/framework.testbed_model.sut_node.rst
>   create mode 100644 doc/api/dts/framework.testbed_model.tg_node.rst
>   create mode 100644 doc/api/dts/framework.testbed_model.traffic_generator.capturing_traffic_generator.rst
>   create mode 100644 doc/api/dts/framework.testbed_model.traffic_generator.rst
>   create mode 100644 doc/api/dts/framework.testbed_model.traffic_generator.scapy.rst
>   create mode 100644 doc/api/dts/framework.testbed_model.traffic_generator.traffic_generator.rst
>   create mode 100644 doc/api/dts/framework.testbed_model.virtual_device.rst
>   create mode 100644 doc/api/dts/framework.utils.rst
>   create mode 100644 doc/api/dts/index.rst
>   create mode 100644 doc/api/dts/meson.build
> 

Applied to dpdk-next-dts with slight modifications suggested by Thomas 
in v19. Thanks everyone for reviews.