Re: [dpdk-dev] [RFC PATCH 0/2] pmdinfogen: rewrite in Python

[Neil Horman <nhorman@tuxdriver.com>]

On Mon, Jun 22, 2020 at 03:45:01AM +0300, Dmitry Kozlyuk wrote:
> This is a PoC rewrite of pmdinfogen in Python with missing bits
> described below and in commits. Community input is desired.
> 
> Pros:
> 
> 1. Simpler build process without host apps.
> 2. Less build requirements (host libelf).
> 3. Easier debugging and maintenance with a high-level language.
> 4. Easier porting on Windows (only add new object format).
> 
Front matter: It seems reasonable to me to implement this in python, just for
ease of maintenence.

> Cons:
> 
> 1. No standard ELF or COFF module for Python
>     (amount of Python code without libelf on par with C code using it).
Thats not really true, pyelftools is pretty widely used, and packaged for
most(all) major distributions.  Requiring this kicks the can for (2) above down
the road a bit, but I would prefer to see that used rather than a custom
implementation, just to reduce the maint. burden

> 2. struct rte_pci_id must be synchronized with header file
>     (it's a few lines that never change).
> 
I think you can just use ctypes and the native python C interface to just import
the rte_pci_id structure from the native dpdk header to avoid this, no?

> There are no built-in or widely used Python libraries for ELF or COFF.
> Some ELF-parsing libraries exist on PyPI, but they're not very handy for
> the task and their installation would complicate build requirements.
> Thus, elf.py implements its own parsing.
> 
> COFF support is underway, it's just not included in this RFC.
> Amount of code is similar to elf.py.
> 
> Build is only tested on Linux x64_64.
> 
> If the community deems this RFC worth finishing, there are a few opens:
> 
> 1. Support for >65K section headers seems present in current pmdinfogen.
>     However, the data it reads is not used after. Is it really needed?
> 
Its used.
The support you're referring to is primarily the ability to extract the true
number of section headers from the ELF file (in the event that its more than
64k).  Without that, on very large binaries, you may not be able to find the
symbol table in the ELF file.

> 2. How much error-handling is required? This is a build-time tool,
>     and Python gives nice stacktraces. However, segfaults are possible
>     in Python version due to pointer handling. IMO, error checking
>     must be just sufficient to prevent silent segfaults.
> 
I'm not really sure what the question is here.  You need to handle errors in the
parsing process, we can't just have the tool crashing during the build.  How
thats handled is an implementation detail I would think.

> 3. On Unix, pmdinfogen is called for each object file extracted with ar
>     from an .a library by a shell script. On Windows, other tools
>     have to be used, shell script will not work. On the other hand, COFF
>     library format is quite simple. Would it be appropriate for
>     pmdinfogen to handle it to avoid intermediate script?
> 
I suppose you could, but extracting objects from archives seems outside the
scope of what pmdinfogen normally does.  What is the problem with using a shell
script on windows?  I thought WSL had support for executing bash syntax shell
scripts there?  Why not key off an environment variable to make the relevant
scripts do the right thing on your environment?

> ---
> Dmitry Kozlyuk (2):
>   pmdinfogen: prototype in Python
>   build: use Python pmdinfogen
> 
>  buildtools/elf.py        | 194 +++++++++++++++++++++++++++++++++++++++
>  buildtools/meson.build   |   3 +-
>  buildtools/pmdinfogen.py | 144 +++++++++++++++++++++++++++++
>  drivers/meson.build      |   2 +-
>  4 files changed, 340 insertions(+), 3 deletions(-)
>  create mode 100644 buildtools/elf.py
>  create mode 100755 buildtools/pmdinfogen.py
> 
> -- 
> 2.25.4
> 
>