[dts] [RFC] Reorganization of DTS

[dts] [RFC] Reorganization of DTS

[Owen Hilyard]

[-- Attachment #1: Type: text/plain, Size: 3040 bytes --]

Hello everyone,

The DTS working group has decided that, conditional upon no issues being
discovered, DTS should be re-organized into a more standard python
project format. We have a few reasons for this.

How DTS currently is structured is only possible due to modifications of
the python interpreter search path. The specific modifications are adding
all of the directories with python code in them, such as framework, nic,
and tests, to the search path. This enables an import statement like
"import logging" to work in a test suite, as opposed to the "import
framework.logging as logging" that would otherwise be needed. However, this
change is not detected by most python tooling. As a result, features like
autocompletion and jump-to-definition are broken in many IDEs, and static
analysis tools can only do surface-level inspections, which also produce a
lot of false positives due to undefined variables. There are two core
reasons the DTS working group has decided to move in this direction.

1. Developer Convenience

There has been a long-running discussion around making DTS submissions for
DPDK patches that introduce new features mandatory. We feel that making DTS
look more like a standard python project would help ease the transition
period. In addition, as mentioned before, IDE features like autocomplete
and jump-to-definition would be functional, and we feel that those are very
useful for learning a codebase. This will also enable any future changes to
be made with the aid of automated refactoring tools, reducing the risk of
breaking changes.

2. Automated Tooling

Most automated tooling also cannot process the additions to the python
path. Some tools, like formatters, are file-oriented and work perfectly
fine right now, but others, such as flake8, do not. Due to python's nature
as a duck typed language, not knowing the base type of something (because
it can't be imported inside of the tool), means that everything it
interacts with also has an unknown type. This results in some tooling
choosing to treat it as always the correct type, and some tooling to treat
it as though it could be any type in the project. Both of these approaches
reduce the accuracy of the tooling.


What will change:

* The main file, currently located at dts/framework/main.py, will be moved
to dts/main.py.
* __init__.py files will be added to all modules to allow for exporting the
types the modules needs to expose.
* Most import paths will be updated, it is expected that most imports will
change from "import logging" to "import framework.logging as logging", to
avoid needing to change any source code during the reorganization.

What we need from the community:

What the DTS working group needs from the community is to let us know if
there are other changes that should be made, or if this change would cause
issues for you. We anticipate that there will not be many issues with this
reorganization unless there is external tooling attached to DTS that is
incompatible with a standard python module format.

Owen Hilyard

[-- Attachment #2: Type: text/html, Size: 3404 bytes --]

Re: [dts] [RFC] Reorganization of DTS

[Owen Hilyard]

[-- Attachment #1: Type: text/plain, Size: 3378 bytes --]

Hello everyone,

Since there was no feedback for a week, the DTS working group has decided
to move ahead with this proposal as written.

Owen Hilyard

On Thu, Aug 5, 2021 at 2:49 PM Owen Hilyard <ohilyard@iol.unh.edu> wrote:

> Hello everyone,
>
> The DTS working group has decided that, conditional upon no issues being
> discovered, DTS should be re-organized into a more standard python
> project format. We have a few reasons for this.
>
> How DTS currently is structured is only possible due to modifications of
> the python interpreter search path. The specific modifications are adding
> all of the directories with python code in them, such as framework, nic,
> and tests, to the search path. This enables an import statement like
> "import logging" to work in a test suite, as opposed to the "import
> framework.logging as logging" that would otherwise be needed. However, this
> change is not detected by most python tooling. As a result, features like
> autocompletion and jump-to-definition are broken in many IDEs, and static
> analysis tools can only do surface-level inspections, which also produce a
> lot of false positives due to undefined variables. There are two core
> reasons the DTS working group has decided to move in this direction.
>
> 1. Developer Convenience
>
> There has been a long-running discussion around making DTS submissions for
> DPDK patches that introduce new features mandatory. We feel that making DTS
> look more like a standard python project would help ease the transition
> period. In addition, as mentioned before, IDE features like autocomplete
> and jump-to-definition would be functional, and we feel that those are very
> useful for learning a codebase. This will also enable any future changes to
> be made with the aid of automated refactoring tools, reducing the risk of
> breaking changes.
>
> 2. Automated Tooling
>
> Most automated tooling also cannot process the additions to the python
> path. Some tools, like formatters, are file-oriented and work perfectly
> fine right now, but others, such as flake8, do not. Due to python's nature
> as a duck typed language, not knowing the base type of something (because
> it can't be imported inside of the tool), means that everything it
> interacts with also has an unknown type. This results in some tooling
> choosing to treat it as always the correct type, and some tooling to treat
> it as though it could be any type in the project. Both of these approaches
> reduce the accuracy of the tooling.
>
>
> What will change:
>
> * The main file, currently located at dts/framework/main.py, will be moved
> to dts/main.py.
> * __init__.py files will be added to all modules to allow for exporting
> the types the modules needs to expose.
> * Most import paths will be updated, it is expected that most imports will
> change from "import logging" to "import framework.logging as logging", to
> avoid needing to change any source code during the reorganization.
>
> What we need from the community:
>
> What the DTS working group needs from the community is to let us know if
> there are other changes that should be made, or if this change would cause
> issues for you. We anticipate that there will not be many issues with this
> reorganization unless there is external tooling attached to DTS that is
> incompatible with a standard python module format.
>
> Owen Hilyard
>

[-- Attachment #2: Type: text/html, Size: 3962 bytes --]