[dpdk-dev] [RFC PATCH 0/2] pmdinfogen: rewrite in Python

[dpdk-dev] [RFC PATCH 0/2] pmdinfogen: rewrite in Python

[Dmitry Kozlyuk]

This is a PoC rewrite of pmdinfogen in Python with missing bits
described below and in commits. Community input is desired.

Pros:

1. Simpler build process without host apps.
2. Less build requirements (host libelf).
3. Easier debugging and maintenance with a high-level language.
4. Easier porting on Windows (only add new object format).

Cons:

1. No standard ELF or COFF module for Python
    (amount of Python code without libelf on par with C code using it).
2. struct rte_pci_id must be synchronized with header file
    (it's a few lines that never change).

There are no built-in or widely used Python libraries for ELF or COFF.
Some ELF-parsing libraries exist on PyPI, but they're not very handy for
the task and their installation would complicate build requirements.
Thus, elf.py implements its own parsing.

COFF support is underway, it's just not included in this RFC.
Amount of code is similar to elf.py.

Build is only tested on Linux x64_64.

If the community deems this RFC worth finishing, there are a few opens:

1. Support for >65K section headers seems present in current pmdinfogen.
    However, the data it reads is not used after. Is it really needed?

2. How much error-handling is required? This is a build-time tool,
    and Python gives nice stacktraces. However, segfaults are possible
    in Python version due to pointer handling. IMO, error checking
    must be just sufficient to prevent silent segfaults.

3. On Unix, pmdinfogen is called for each object file extracted with ar
    from an .a library by a shell script. On Windows, other tools
    have to be used, shell script will not work. On the other hand, COFF
    library format is quite simple. Would it be appropriate for
    pmdinfogen to handle it to avoid intermediate script?

---
Dmitry Kozlyuk (2):
  pmdinfogen: prototype in Python
  build: use Python pmdinfogen

 buildtools/elf.py        | 194 +++++++++++++++++++++++++++++++++++++++
 buildtools/meson.build   |   3 +-
 buildtools/pmdinfogen.py | 144 +++++++++++++++++++++++++++++
 drivers/meson.build      |   2 +-
 4 files changed, 340 insertions(+), 3 deletions(-)
 create mode 100644 buildtools/elf.py
 create mode 100755 buildtools/pmdinfogen.py

-- 
2.25.4

[dpdk-dev] [RFC PATCH 1/2] pmdinfogen: prototype in Python

[Dmitry Kozlyuk]

* No user-friendly error handling and no bounds checking yet.
* No support for >65K sections case (is it needed?).
* The order of definitions is reversed.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 buildtools/elf.py        | 194 +++++++++++++++++++++++++++++++++++++++
 buildtools/pmdinfogen.py | 144 +++++++++++++++++++++++++++++
 2 files changed, 338 insertions(+)
 create mode 100644 buildtools/elf.py
 create mode 100755 buildtools/pmdinfogen.py

diff --git a/buildtools/elf.py b/buildtools/elf.py
new file mode 100644
index 000000000..4085d547b
--- /dev/null
+++ b/buildtools/elf.py
@@ -0,0 +1,194 @@
+import ctypes
+
+
+class ElfIdent(ctypes.Structure):
+    _pack_ = True
+    _fields_ = [
+        ("magic", ctypes.c_char * 4),
+        ("class_", ctypes.c_uint8),
+        ("data", ctypes.c_uint8),
+        ("version", ctypes.c_uint8),
+        ("os_abi", ctypes.c_uint8),
+        ("abi_version", ctypes.c_uint8),
+        ("pad", ctypes.c_uint8 * 7),
+    ]
+
+    @property
+    def is_magic_ok(self):
+        return self.magic.value == b"\x7fELF"
+
+    @property
+    def is_32bit(self):
+        return self.class_ == 1
+
+    @property
+    def is_big_endian(self):
+        return self.data == 2
+
+    def define_structures(self):
+        base_type = ctypes.LittleEndianStructure
+        if self.is_big_endian:
+            base = ctypes.BigEndianStructure
+
+        size_type = ctypes.c_uint64
+        if self.is_32bit:
+            size_type = ctypes.c_uint32
+
+        class FileHeader(base_type):
+            _pack_ = True
+            _fields_ = [
+                ("e_ident", ElfIdent),
+                ("e_type", ctypes.c_uint16),
+                ("e_machine", ctypes.c_uint16),
+                ("e_version", ctypes.c_uint32),
+                ("e_entry", size_type),
+                ("e_phoff", size_type),
+                ("e_shoff", size_type),
+                ("e_flags", ctypes.c_uint32),
+                ("e_ehsize", ctypes.c_uint16),
+                ("e_phentsize", ctypes.c_uint16),
+                ("e_phnum", ctypes.c_uint16),
+                ("e_shentsize", ctypes.c_uint16),
+                ("e_shnum", ctypes.c_uint16),
+                ("e_shstrndx", ctypes.c_uint16),
+            ]
+
+        class SectionHeader(base_type):
+            _pack_ = True
+            _fields_ = [
+                ("sh_name", ctypes.c_uint32),
+                ("sh_type", ctypes.c_uint32),
+                ("sh_flags", size_type),
+                ("sh_addr", size_type),
+                ("sh_offset", size_type),
+                ("sh_size", size_type),
+                ("sh_link", ctypes.c_uint32),
+                ("sh_info", ctypes.c_uint32),
+                ("sh_addralign", size_type),
+                ("sh_entsize", size_type),
+            ]
+
+        class Symbol32(base_type):
+            _pack_ = True
+            _fields_ = [
+                ("st_name", ctypes.c_uint32),
+                ("st_value", ctypes.c_uint32),
+                ("st_size", ctypes.c_uint32),
+                ("st_info", ctypes.c_uint8),
+                ("st_other", ctypes.c_uint8),
+                ("st_shndx", ctypes.c_uint16),
+            ]
+
+        class Symbol64(base_type):
+            _pack_ = True
+            _fields_ = [
+                ("st_name", ctypes.c_uint32),
+                ("st_info", ctypes.c_uint8),
+                ("st_other", ctypes.c_uint8),
+                ("st_shndx", ctypes.c_uint16),
+                ("st_value", ctypes.c_uint64),
+                ("st_size", ctypes.c_uint64),
+            ]
+
+        Symbol = Symbol32 if self.is_32bit else Symbol64
+
+        return FileHeader, SectionHeader, Symbol
+
+
+class Symbol:
+    def __init__(self, image, elf):
+        self._image = image
+        self._elf = elf
+
+
+    @property
+    def address(self):
+        base = self._image._sections[self._elf.st_shndx].sh_offset
+        offset = base + self._elf.st_value
+        memory = ctypes.c_char.from_buffer(self._image._data, offset)
+        return ctypes.addressof(memory)
+
+
+class Image:
+    def __init__(self, data):
+        SHN_UNDEF = 0x0000
+        SHN_XINDEX = 0xFFFF
+
+        ident = ElfIdent.from_buffer(data)
+        ElfFileHeader, ElfSectionHeader, ElfSymbol = ident.define_structures()
+
+        header = ElfFileHeader.from_buffer(data)
+
+        if header.e_shnum == SHN_UNDEF:
+            section = ElfSectionHeader.from_buffer(data, header.e_shoff)
+            sections_num = section.sh_size
+        else:
+            sections_num = header.e_shnum
+        sections_desc = ElfSectionHeader * sections_num
+        sections = sections_desc.from_buffer(data, header.e_shoff)
+
+        if header.e_shstrndx == SHN_XINDEX:
+            strings_index = sections[0].sh_link
+        else:
+            strings_index = header.e_shstrndx
+
+        symtab, strtab = Image._find_symbol_table(data, sections, ElfSymbol)
+
+        self._data = data
+        self._header = header
+        self._sections = sections
+        self._strings = sections[strings_index]
+        self._symtab = symtab
+        self._strtab = strtab
+
+
+    @staticmethod
+    def _find_symbol_table(data, sections, symbol_type):
+        SHT_SYMTAB = 2
+        SHT_SYMTAB_SHNDX = 18
+
+        for section in sections:
+            if section.sh_type == SHT_SYMTAB:
+                symbol_count = section.sh_size // ctypes.sizeof(symbol_type)
+                symtab_desc = symbol_type * symbol_count
+                symtab_offset = section.sh_offset
+                symtab = symtab_desc.from_buffer(data, symtab_offset)
+
+                strtab = sections[section.sh_link].sh_offset
+
+                return symtab, strtab
+
+            # TODO: SHT_SYMTAB_SHNDX?
+
+        raise Exception('no symbol table')
+
+
+    @property
+    def is_big_endian(self):
+        return self._header.e_ident.is_big_endian
+
+
+    def find_symbol(self, name: str, start: Symbol = None):
+        name_size = len(name)
+        name_desc = ctypes.c_char * name_size
+
+        i = 0
+        if start:
+            table_address = ctypes.addressof(self._symtab)
+            start_address = ctypes.addressof(start._elf)
+            i = (start_address - table_address) // ctypes.sizeof(start._elf) + 1
+
+        while i < len(self._symtab):
+            symbol = self._symtab[i]
+
+            name_offset = self._strtab + symbol.st_name
+            if name_offset + name_size > len(self._data):
+                break
+
+            symbol_name = name_desc.from_buffer(self._data, name_offset).value
+            if symbol_name == name.encode():
+                return Symbol(self, symbol)
+
+            i += 1
+
+        return None
diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
new file mode 100755
index 000000000..74d2e5285
--- /dev/null
+++ b/buildtools/pmdinfogen.py
@@ -0,0 +1,144 @@
+#!/usr/bin/env python3
+
+import argparse
+import ctypes
+import elf
+import json
+import mmap
+import os
+import sys
+import tempfile
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = ctypes.string_at(symbol.address).decode()
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_symbol(f"__{name}{suffix}")
+            if option_symbol:
+                value = ctypes.string_at(option_symbol.address).decode()
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_symbol(f"__{name}_pci_tbl_export")
+        if not pci_table_name_symbol:
+            return driver
+
+        pci_table_name = ctypes.string_at(pci_table_name_symbol.address).decode()
+
+        pci_table_symbol = image.find_symbol(pci_table_name)
+        if not pci_table_symbol:
+            raise Exception('PCI table declared but not defined')
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id = rte_pci_id.from_address(pci_table_symbol.address)
+        while pci_id.device_id:
+            driver.pci_ids.append(
+                [
+                    pci_id.vendor_id,
+                    pci_id.device_id,
+                    pci_id.subsystem_vendor_id,
+                    pci_id.subsystem_device_id,
+                ]
+            )
+            pci_id = rte_pci_id.from_address(
+                ctypes.addressof(pci_id) + ctypes.sizeof(pci_id)
+            )
+
+        return driver
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            f"const char {self.name}_pmd_info[] __attribute__((used)) = "
+            f'"PMD_INFO_STRING= {escaped}";',
+            file=file,
+        )
+
+
+def get_symbols_by_prefix(image, name):
+    symbol = image.find_symbol(name)
+    while symbol:
+        yield symbol
+        symbol = image.find_symbol(name, symbol)
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in get_symbols_by_prefix(image, "this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    for driver in drivers:
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path")
+    parser.add_argument("output", help="output C file path")
+    return parser.parse_args()
+
+
+def map_input(path):
+    if path == '-':
+        fd, name = tempfile.mkstemp()
+    else:
+        fd = os.open(path, os.O_RDONLY)
+    return mmap.mmap(fd, 0, access=mmap.ACCESS_COPY)
+
+
+def open_output(path):
+    if path == '-':
+        return sys.stdout
+    return open(path, 'w')
+
+
+def main():
+    args = parse_args()
+    memory = map_input(args.input)
+    image = elf.Image(memory)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
-- 
2.25.4

[dpdk-dev] [RFC PATCH 2/2] build: use Python pmdinfogen

[Dmitry Kozlyuk]

* C pmdinfogen not removed.
* No Makefile support yet.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 buildtools/meson.build | 3 +--
 drivers/meson.build    | 2 +-
 2 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/buildtools/meson.build b/buildtools/meson.build
index d5f8291be..1af835a9c 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -1,9 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017-2019 Intel Corporation
 
-subdir('pmdinfogen')
-
 pkgconf = find_program('pkg-config', 'pkgconf', required: false)
+pmdinfogen = find_program('pmdinfogen.py')
 pmdinfo = find_program('gen-pmdinfo-cfile.sh')
 list_dir_globs = find_program('list-dir-globs.py')
 check_symbols = find_program('check-symbols.sh')
diff --git a/drivers/meson.build b/drivers/meson.build
index cfb6a833c..be521398e 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -122,7 +122,7 @@ foreach class:dpdk_driver_classes
 					command: [pmdinfo, tmp_lib.full_path(),
 						'@OUTPUT@', pmdinfogen],
 					output: out_filename,
-					depends: [pmdinfogen, tmp_lib])
+					depends: [tmp_lib])
 
 			version_map = '@0@/@1@/@2@_version.map'.format(
 					meson.current_source_dir(),
-- 
2.25.4

Re: [dpdk-dev] [RFC PATCH 0/2] pmdinfogen: rewrite in Python

[Neil Horman]

On Mon, Jun 22, 2020 at 03:45:01AM +0300, Dmitry Kozlyuk wrote:
> This is a PoC rewrite of pmdinfogen in Python with missing bits
> described below and in commits. Community input is desired.
> 
> Pros:
> 
> 1. Simpler build process without host apps.
> 2. Less build requirements (host libelf).
> 3. Easier debugging and maintenance with a high-level language.
> 4. Easier porting on Windows (only add new object format).
> 
Front matter: It seems reasonable to me to implement this in python, just for
ease of maintenence.

> Cons:
> 
> 1. No standard ELF or COFF module for Python
>     (amount of Python code without libelf on par with C code using it).
Thats not really true, pyelftools is pretty widely used, and packaged for
most(all) major distributions.  Requiring this kicks the can for (2) above down
the road a bit, but I would prefer to see that used rather than a custom
implementation, just to reduce the maint. burden

> 2. struct rte_pci_id must be synchronized with header file
>     (it's a few lines that never change).
> 
I think you can just use ctypes and the native python C interface to just import
the rte_pci_id structure from the native dpdk header to avoid this, no?

> There are no built-in or widely used Python libraries for ELF or COFF.
> Some ELF-parsing libraries exist on PyPI, but they're not very handy for
> the task and their installation would complicate build requirements.
> Thus, elf.py implements its own parsing.
> 
> COFF support is underway, it's just not included in this RFC.
> Amount of code is similar to elf.py.
> 
> Build is only tested on Linux x64_64.
> 
> If the community deems this RFC worth finishing, there are a few opens:
> 
> 1. Support for >65K section headers seems present in current pmdinfogen.
>     However, the data it reads is not used after. Is it really needed?
> 
Its used.
The support you're referring to is primarily the ability to extract the true
number of section headers from the ELF file (in the event that its more than
64k).  Without that, on very large binaries, you may not be able to find the
symbol table in the ELF file.

> 2. How much error-handling is required? This is a build-time tool,
>     and Python gives nice stacktraces. However, segfaults are possible
>     in Python version due to pointer handling. IMO, error checking
>     must be just sufficient to prevent silent segfaults.
> 
I'm not really sure what the question is here.  You need to handle errors in the
parsing process, we can't just have the tool crashing during the build.  How
thats handled is an implementation detail I would think.

> 3. On Unix, pmdinfogen is called for each object file extracted with ar
>     from an .a library by a shell script. On Windows, other tools
>     have to be used, shell script will not work. On the other hand, COFF
>     library format is quite simple. Would it be appropriate for
>     pmdinfogen to handle it to avoid intermediate script?
> 
I suppose you could, but extracting objects from archives seems outside the
scope of what pmdinfogen normally does.  What is the problem with using a shell
script on windows?  I thought WSL had support for executing bash syntax shell
scripts there?  Why not key off an environment variable to make the relevant
scripts do the right thing on your environment?

> ---
> Dmitry Kozlyuk (2):
>   pmdinfogen: prototype in Python
>   build: use Python pmdinfogen
> 
>  buildtools/elf.py        | 194 +++++++++++++++++++++++++++++++++++++++
>  buildtools/meson.build   |   3 +-
>  buildtools/pmdinfogen.py | 144 +++++++++++++++++++++++++++++
>  drivers/meson.build      |   2 +-
>  4 files changed, 340 insertions(+), 3 deletions(-)
>  create mode 100644 buildtools/elf.py
>  create mode 100755 buildtools/pmdinfogen.py
> 
> -- 
> 2.25.4
> 
> 

Re: [dpdk-dev] [RFC PATCH 0/2] pmdinfogen: rewrite in Python

[Dmitry Kozlyuk]

On Mon, 22 Jun 2020 08:41:17 -0400, Neil Horman wrote:
> On Mon, Jun 22, 2020 at 03:45:01AM +0300, Dmitry Kozlyuk wrote:
[snip]
> > 1. No standard ELF or COFF module for Python
> >     (amount of Python code without libelf on par with C code using it).  
> Thats not really true, pyelftools is pretty widely used, and packaged for
> most(all) major distributions.  Requiring this kicks the can for (2) above down
> the road a bit, but I would prefer to see that used rather than a custom
> implementation, just to reduce the maint. burden

I must have underestimated pyelftools spread. I'll look into using it. The
less new code the better, I agree here.  Windows users can get it via PyPI.


> > 2. struct rte_pci_id must be synchronized with header file
> >     (it's a few lines that never change).
> >   
> I think you can just use ctypes and the native python C interface to just import
> the rte_pci_id structure from the native dpdk header to avoid this, no?

Not sure I follow. RFC script uses ctypes to describe struct rte_pci_id
in Python. If you're suggesting to create a Python module in C just to include
a header with a single small structure, I'd say it's not worth the trouble.


> > 1. Support for >65K section headers seems present in current pmdinfogen.
> >     However, the data it reads is not used after. Is it really needed?
> >   
> Its used.
> The support you're referring to is primarily the ability to extract the true
> number of section headers from the ELF file (in the event that its more than
> 64k).  Without that, on very large binaries, you may not be able to find the
> symbol table in the ELF file.

I was talking about these fields of struct elf_info:

	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
	 * take shndx from symtab_shndx_start[N] instead
	 */
	Elf32_Word   *symtab_shndx_start;
	Elf32_Word   *symtab_shndx_stop;

They are not used after being filled in parse_elf() and their endianness
fixed in the end despite the comment.


> > 2. How much error-handling is required? This is a build-time tool,
> >     and Python gives nice stacktraces. However, segfaults are possible
> >     in Python version due to pointer handling. IMO, error checking
> >     must be just sufficient to prevent silent segfaults.
> >   
> I'm not really sure what the question is here.  You need to handle errors in the
> parsing process, we can't just have the tool crashing during the build.  How
> thats handled is an implementation detail I would think.

Consider a snippet from pmdinfogen.c:

	/* Check if file offset is correct */
	if (hdr->e_shoff > info->size) {
		fprintf(stderr, "section header offset=%lu in file '%s' "
		      "is bigger than filesize=%lu\n",
		      (unsigned long)hdr->e_shoff,
		      filename, info->size);
		return 0;
	}

It is required in C, because otherwise pmdinfogen would crash without
diagnostic. With Python, most errors like this result in a crash with a trace
to the error line and a message from ctypes (or ELF parsing library). I'm
arguing for leaving only the checks that prevent *silent* crashes (this can
happen with ctypes) which are hard to diagnose. Motivation is to keep the
code simple.


> > 3. On Unix, pmdinfogen is called for each object file extracted with ar
> >     from an .a library by a shell script. On Windows, other tools
> >     have to be used, shell script will not work. On the other hand, COFF
> >     library format is quite simple. Would it be appropriate for
> >     pmdinfogen to handle it to avoid intermediate script?
> >   
> I suppose you could, but extracting objects from archives seems outside the
> scope of what pmdinfogen normally does.  What is the problem with using a shell
> script on windows?  I thought WSL had support for executing bash syntax shell
> scripts there?  Why not key off an environment variable to make the relevant
> scripts do the right thing on your environment?

WSL2 is a Linux VM integrated in Windows, you can only cross-compile from
there. Native builds can use Python or PowerShell, which is as capable as Unix
shells, but incompatible with them. To call lib.exe (MSVC analog of ar) is
probably simpler then parsing COFF, yes. So meson could select one of the
following for a Windows target (objects are COFF):

Host     Toolchain  Archive  Script      Extractor
-------  ---------  -------  ---------   ---------
Linux    MinGW      .a       sh          ar
Windows  MinGW      .a       PowerShell  ar
Windows  Clang      .lib     PowerShell  lib

-- 
Dmitry Kozlyuk

Re: [dpdk-dev] [RFC PATCH 0/2] pmdinfogen: rewrite in Python

[Neil Horman]

On Mon, Jun 22, 2020 at 10:39:46PM +0300, Dmitry Kozlyuk wrote:
> On Mon, 22 Jun 2020 08:41:17 -0400, Neil Horman wrote:
> > On Mon, Jun 22, 2020 at 03:45:01AM +0300, Dmitry Kozlyuk wrote:
> [snip]
> > > 1. No standard ELF or COFF module for Python
> > >     (amount of Python code without libelf on par with C code using it).  
> > Thats not really true, pyelftools is pretty widely used, and packaged for
> > most(all) major distributions.  Requiring this kicks the can for (2) above down
> > the road a bit, but I would prefer to see that used rather than a custom
> > implementation, just to reduce the maint. burden
> 
> I must have underestimated pyelftools spread. I'll look into using it. The
> less new code the better, I agree here.  Windows users can get it via PyPI.
> 
Ack.

> 
> > > 2. struct rte_pci_id must be synchronized with header file
> > >     (it's a few lines that never change).
> > >   
> > I think you can just use ctypes and the native python C interface to just import
> > the rte_pci_id structure from the native dpdk header to avoid this, no?
> 
> Not sure I follow. RFC script uses ctypes to describe struct rte_pci_id
> in Python. If you're suggesting to create a Python module in C just to include
> a header with a single small structure, I'd say it's not worth the trouble.
> 
Yeah, apologies, i misread what you were saying here, and didn't bother to check
the code.  what you're doing makes sense.

> 
> > > 1. Support for >65K section headers seems present in current pmdinfogen.
> > >     However, the data it reads is not used after. Is it really needed?
> > >   
> > Its used.
> > The support you're referring to is primarily the ability to extract the true
> > number of section headers from the ELF file (in the event that its more than
> > 64k).  Without that, on very large binaries, you may not be able to find the
> > symbol table in the ELF file.
> 
> I was talking about these fields of struct elf_info:
> 
> 	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
> 	 * take shndx from symtab_shndx_start[N] instead
> 	 */
> 	Elf32_Word   *symtab_shndx_start;
> 	Elf32_Word   *symtab_shndx_stop;
> 
> They are not used after being filled in parse_elf() and their endianness
> fixed in the end despite the comment.
> 
Its been a while since I wrote this, so I need to go back and look closely, but
as you say, these values aren't used after parse_elf completes, but they are(I
think) used to correct the endianess of the section header index table, so that
get_sym_value works properly.  You would (again I think), only note a problem if
you were parsing an ELF file for an architecture with an endianess opposite that
of what you are building on (i.e. an x86 system cross compiling for a powerpc
target).

> 
> > > 2. How much error-handling is required? This is a build-time tool,
> > >     and Python gives nice stacktraces. However, segfaults are possible
> > >     in Python version due to pointer handling. IMO, error checking
> > >     must be just sufficient to prevent silent segfaults.
> > >   
> > I'm not really sure what the question is here.  You need to handle errors in the
> > parsing process, we can't just have the tool crashing during the build.  How
> > thats handled is an implementation detail I would think.
> 
> Consider a snippet from pmdinfogen.c:
> 
> 	/* Check if file offset is correct */
> 	if (hdr->e_shoff > info->size) {
> 		fprintf(stderr, "section header offset=%lu in file '%s' "
> 		      "is bigger than filesize=%lu\n",
> 		      (unsigned long)hdr->e_shoff,
> 		      filename, info->size);
> 		return 0;
> 	}
> 
> It is required in C, because otherwise pmdinfogen would crash without
> diagnostic. With Python, most errors like this result in a crash with a trace
> to the error line and a message from ctypes (or ELF parsing library). I'm
> arguing for leaving only the checks that prevent *silent* crashes (this can
> happen with ctypes) which are hard to diagnose. Motivation is to keep the
> code simple.
> 
Hmm, I'd defer to others opinions on that.  As a build tool it may be ok to just
crash with a backtrace, but I'm personally not a fan of that.  Id rather see a
diagnostic error and have the tool exits with an appropriate exit code, but lets
see what others say.

> 
> > > 3. On Unix, pmdinfogen is called for each object file extracted with ar
> > >     from an .a library by a shell script. On Windows, other tools
> > >     have to be used, shell script will not work. On the other hand, COFF
> > >     library format is quite simple. Would it be appropriate for
> > >     pmdinfogen to handle it to avoid intermediate script?
> > >   
> > I suppose you could, but extracting objects from archives seems outside the
> > scope of what pmdinfogen normally does.  What is the problem with using a shell
> > script on windows?  I thought WSL had support for executing bash syntax shell
> > scripts there?  Why not key off an environment variable to make the relevant
> > scripts do the right thing on your environment?
> 
> WSL2 is a Linux VM integrated in Windows, you can only cross-compile from
> there. Native builds can use Python or PowerShell, which is as capable as Unix
> shells, but incompatible with them. To call lib.exe (MSVC analog of ar) is
> probably simpler then parsing COFF, yes. So meson could select one of the
> following for a Windows target (objects are COFF):
> 
> Host     Toolchain  Archive  Script      Extractor
> -------  ---------  -------  ---------   ---------
> Linux    MinGW      .a       sh          ar
> Windows  MinGW      .a       PowerShell  ar
> Windows  Clang      .lib     PowerShell  lib
I think if I read you right, what you're suggesting here is that meson setup a
build time marco $ARCHIVETOOL, and set it to ar or lib.exe depending on the
build environment, and just have the script in question use $ARCHIVER?  If so,
I'd be good with that.  Do we have to worry about the use of divergent command
line options for each tool as well?

Neil

> 
> -- 
> Dmitry Kozlyuk
> 

Re: [dpdk-dev] [RFC PATCH 0/2] pmdinfogen: rewrite in Python

[Bruce Richardson]

On Tue, Jun 23, 2020 at 07:28:06AM -0400, Neil Horman wrote:
> On Mon, Jun 22, 2020 at 10:39:46PM +0300, Dmitry Kozlyuk wrote:
> > On Mon, 22 Jun 2020 08:41:17 -0400, Neil Horman wrote:
> > > On Mon, Jun 22, 2020 at 03:45:01AM +0300, Dmitry Kozlyuk wrote:
> > [snip]

> > 
> > > > 2. How much error-handling is required? This is a build-time tool,
> > > >     and Python gives nice stacktraces. However, segfaults are possible
> > > >     in Python version due to pointer handling. IMO, error checking
> > > >     must be just sufficient to prevent silent segfaults.
> > > >   
> > > I'm not really sure what the question is here.  You need to handle errors in the
> > > parsing process, we can't just have the tool crashing during the build.  How
> > > thats handled is an implementation detail I would think.
> > 
> > Consider a snippet from pmdinfogen.c:
> > 
> > 	/* Check if file offset is correct */
> > 	if (hdr->e_shoff > info->size) {
> > 		fprintf(stderr, "section header offset=%lu in file '%s' "
> > 		      "is bigger than filesize=%lu\n",
> > 		      (unsigned long)hdr->e_shoff,
> > 		      filename, info->size);
> > 		return 0;
> > 	}
> > 
> > It is required in C, because otherwise pmdinfogen would crash without
> > diagnostic. With Python, most errors like this result in a crash with a trace
> > to the error line and a message from ctypes (or ELF parsing library). I'm
> > arguing for leaving only the checks that prevent *silent* crashes (this can
> > happen with ctypes) which are hard to diagnose. Motivation is to keep the
> > code simple.
> > 
> Hmm, I'd defer to others opinions on that.  As a build tool it may be ok to just
> crash with a backtrace, but I'm personally not a fan of that.  Id rather see a
> diagnostic error and have the tool exits with an appropriate exit code, but lets
> see what others say.
>
I'd take a wait-and-see approach here. Obviously individualized errors
messages are better, but for a case like the above where we have a
malformed elf header, I would think that this is such a rare event that
just letting python exception handling manage it should be ok.
 
> > 
> > > > 3. On Unix, pmdinfogen is called for each object file extracted with ar
> > > >     from an .a library by a shell script. On Windows, other tools
> > > >     have to be used, shell script will not work. On the other hand, COFF
> > > >     library format is quite simple. Would it be appropriate for
> > > >     pmdinfogen to handle it to avoid intermediate script?
> > > >   
> > > I suppose you could, but extracting objects from archives seems outside the
> > > scope of what pmdinfogen normally does.  What is the problem with using a shell
> > > script on windows?  I thought WSL had support for executing bash syntax shell
> > > scripts there?  Why not key off an environment variable to make the relevant
> > > scripts do the right thing on your environment?
> > 
> > WSL2 is a Linux VM integrated in Windows, you can only cross-compile from
> > there. Native builds can use Python or PowerShell, which is as capable as Unix
> > shells, but incompatible with them. To call lib.exe (MSVC analog of ar) is
> > probably simpler then parsing COFF, yes. So meson could select one of the
> > following for a Windows target (objects are COFF):
> > 
> > Host     Toolchain  Archive  Script      Extractor
> > -------  ---------  -------  ---------   ---------
> > Linux    MinGW      .a       sh          ar
> > Windows  MinGW      .a       PowerShell  ar
> > Windows  Clang      .lib     PowerShell  lib
> I think if I read you right, what you're suggesting here is that meson setup a
> build time marco $ARCHIVETOOL, and set it to ar or lib.exe depending on the
> build environment, and just have the script in question use $ARCHIVER?  If so,
> I'd be good with that.  Do we have to worry about the use of divergent command
> line options for each tool as well?
> 
It might be better just to do that detection in the python script itself,
rather than in meson, since the script may have to do other behavioural
changes based on the platform.

Regards,
/Bruce

[dpdk-dev] [RFC PATCH v2 0/2] pmdinfogen: rewrite in Python

[Dmitry Kozlyuk]

The goals are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf.

Changes in v2:

1. pyelftools is used instead of hand-written ELF parser.
2. Makefiles are adjusted.
3. f-strings replaced with % formatting to support Python 3.5.
4. Wrapper script does not hide stderr from ar and pmdinfogen.

Only logic errors, like missing symbols or invalid content, are checked.
This seems enough for debugging and, to my understanding, there are no
strong objections.  All errors raise exceptions printed to stderr.

While discussion of future wrapper logic and Windows support continues,
this patchset is ready for testing and review.

Dmitry Kozlyuk (3):
  pmdinfogen: prototype in Python
  build: use Python pmdinfogen
  doc/linux_gsg: require pyelftools for pmdinfogen

 GNUmakefile                       |   2 +-
 buildtools/Makefile               |   9 --
 buildtools/gen-pmdinfo-cfile.sh   |   3 +-
 buildtools/meson.build            |   3 +-
 buildtools/pmdinfogen.py          | 191 ++++++++++++++++++++++++++++++
 doc/guides/linux_gsg/sys_reqs.rst |   6 +
 drivers/meson.build               |   2 +-
 mk/internal/rte.compile-pre.mk    |   2 +-
 mk/rte.sdkinstall.mk              |   2 -
 9 files changed, 202 insertions(+), 18 deletions(-)
 delete mode 100644 buildtools/Makefile
 create mode 100755 buildtools/pmdinfogen.py

-- 
2.25.4

[dpdk-dev] [RFC PATCH v2 1/3] pmdinfogen: prototype in Python

[Dmitry Kozlyuk]

Requires pyelfutils, but no docs yet.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 buildtools/pmdinfogen.py | 191 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 191 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py

diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
new file mode 100755
index 000000000..05252e7ce
--- /dev/null
+++ b/buildtools/pmdinfogen.py
@@ -0,0 +1,191 @@
+#!/usr/bin/env python3
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image._image.get_section(self._symbol["st_shndx"]).data()
+        base = self._symbol["st_value"]
+        return data[base : base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        if len(value):
+            return value[:-1].decode()
+        return ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        if symbol:
+            return ELFSymbol(self, symbol[0])
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    def _load_pci_ids(self, image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        if table_symbol.size % pci_id_size != 0:
+            raise Exception(
+                "PCI table size (%d) is not a multiple of rte_pci_id size (%d)"
+                % (table_symbol.size, pci_id_size)
+            )
+
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            self.pci_ids.append(
+                [
+                    pci_id.vendor_id,
+                    pci_id.device_id,
+                    pci_id.subsystem_vendor_id,
+                    pci_id.subsystem_device_id,
+                ]
+            )
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
-- 
2.25.4

[dpdk-dev] [RFC PATCH v2 2/3] build: use Python pmdinfogen

[Dmitry Kozlyuk]

Also don't suppress stderr from ar and pmdinfogen.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 GNUmakefile                     | 2 +-
 buildtools/Makefile             | 9 ---------
 buildtools/gen-pmdinfo-cfile.sh | 3 +--
 buildtools/meson.build          | 3 +--
 drivers/meson.build             | 2 +-
 mk/internal/rte.compile-pre.mk  | 2 +-
 mk/rte.sdkinstall.mk            | 2 --
 7 files changed, 5 insertions(+), 18 deletions(-)
 delete mode 100644 buildtools/Makefile

diff --git a/GNUmakefile b/GNUmakefile
index e8de422df..242d30d2e 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -12,6 +12,6 @@ export RTE_SDK
 # directory list
 #
 
-ROOTDIRS-y := buildtools lib kernel drivers app
+ROOTDIRS-y := lib kernel drivers app
 
 include $(RTE_SDK)/mk/rte.sdkroot.mk
diff --git a/buildtools/Makefile b/buildtools/Makefile
deleted file mode 100644
index 7f76fd7d6..000000000
--- a/buildtools/Makefile
+++ /dev/null
@@ -1,9 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2016 Neil Horman <nhorman@tuxdriver.com>
-# All rights reserved.
-
-include $(RTE_SDK)/mk/rte.vars.mk
-
-DIRS-y += pmdinfogen
-
-include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/buildtools/gen-pmdinfo-cfile.sh b/buildtools/gen-pmdinfo-cfile.sh
index 43059cf36..eeeab678e 100755
--- a/buildtools/gen-pmdinfo-cfile.sh
+++ b/buildtools/gen-pmdinfo-cfile.sh
@@ -9,6 +9,5 @@ pmdinfogen=$3
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
diff --git a/buildtools/meson.build b/buildtools/meson.build
index f9d2fdf74..88c238ca4 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -1,9 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017-2019 Intel Corporation
 
-subdir('pmdinfogen')
-
 pkgconf = find_program('pkg-config', 'pkgconf', required: false)
+pmdinfogen = find_program('pmdinfogen.py')
 pmdinfo = find_program('gen-pmdinfo-cfile.sh')
 list_dir_globs = find_program('list-dir-globs.py')
 check_symbols = find_program('check-symbols.sh')
diff --git a/drivers/meson.build b/drivers/meson.build
index e78c76c55..37822537a 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -119,7 +119,7 @@ foreach class:dpdk_driver_classes
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 			version_map = '@0@/@1@/@2@_version.map'.format(
 					meson.current_source_dir(),
diff --git a/mk/internal/rte.compile-pre.mk b/mk/internal/rte.compile-pre.mk
index df05b5576..bb2ab0725 100644
--- a/mk/internal/rte.compile-pre.mk
+++ b/mk/internal/rte.compile-pre.mk
@@ -59,7 +59,7 @@ endif
 CHECK_SYMBOLS_SCRIPT = $(RTE_SDK)/buildtools/check-symbols.sh
 CHECK_SYMBOLS = $(CHECK_SYMBOLS_SCRIPT) $(SRCDIR)/$(EXPORT_MAP) $@
 
-PMDINFO_GEN = $(RTE_SDK_BIN)/app/dpdk-pmdinfogen $@ $@.pmd.c
+PMDINFO_GEN = $(RTE_SDK)/buildtools/pmdinfogen.py $@ $@.pmd.c
 PMDINFO_CC = $(CC) $(CPPFLAGS) $(CFLAGS) $(EXTRA_CFLAGS) -c -o $@.pmd.o $@.pmd.c
 PMDINFO_LD = $(CROSS)ld -r $(filter-out -export-dynamic,$(LDFLAGS)) -o $@.o $@.pmd.o $@
 PMDINFO_TO_O = if grep -q 'RTE_PMD_REGISTER_.*(.*)' $<; then \
diff --git a/mk/rte.sdkinstall.mk b/mk/rte.sdkinstall.mk
index 32bed5d95..875a64f04 100644
--- a/mk/rte.sdkinstall.mk
+++ b/mk/rte.sdkinstall.mk
@@ -98,7 +98,6 @@ install-runtime:
 	$(Q)cp $(CP_FLAGS)    $O/lib/* $(DESTDIR)$(libdir)
 	$(Q)$(call rte_mkdir, $(DESTDIR)$(bindir))
 	$(Q)tar -cf -      -C $O --exclude 'app/*.map' \
-		--exclude app/dpdk-pmdinfogen \
 		--exclude 'app/cmdline*' --exclude app/test \
 		--exclude app/testacl --exclude app/testpipeline app | \
 	    tar -xf -      -C $(DESTDIR)$(bindir) $(TAR_X_FLAGS)
@@ -134,7 +133,6 @@ install-sdk:
 	$(Q)cp $(CP_FLAGS)      $(RTE_SDK)/buildtools    $(DESTDIR)$(sdkdir)
 	$(Q)$(call rte_mkdir,                            $(DESTDIR)$(targetdir)/app)
 	$(Q)cp $(CP_FLAGS)      $O/.config               $(DESTDIR)$(targetdir)
-	$(Q)cp $(CP_FLAGS)      $O/app/dpdk-pmdinfogen   $(DESTDIR)$(targetdir)/app
 	$(Q)$(call rte_symlink, $(DESTDIR)$(includedir), $(DESTDIR)$(targetdir)/include)
 	$(Q)$(call rte_symlink, $(DESTDIR)$(libdir),     $(DESTDIR)$(targetdir)/lib)
 
-- 
2.25.4

[dpdk-dev] [RFC PATCH v2 3/3] doc/linux_gsg: require pyelftools for pmdinfogen

[Dmitry Kozlyuk]

The package is available on all major distributions.
FreeBSD has no system requirements section in its GSG,
Windows currently neither uses pmdinfogen not is supported by it.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
index a124656bc..ef2037ca7 100644
--- a/doc/guides/linux_gsg/sys_reqs.rst
+++ b/doc/guides/linux_gsg/sys_reqs.rst
@@ -56,6 +56,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
-- 
2.25.4

Re: [dpdk-dev] [RFC PATCH 0/2] pmdinfogen: rewrite in Python

[Dmitry Kozlyuk]

On Tue, 23 Jun 2020 07:28:06 -0400 Neil Horman wrote:
> On Mon, Jun 22, 2020 at 10:39:46PM +0300, Dmitry Kozlyuk wrote:
> > On Mon, 22 Jun 2020 08:41:17 -0400, Neil Horman wrote:  
> > > On Mon, Jun 22, 2020 at 03:45:01AM +0300, Dmitry Kozlyuk wrote:  
[snip]  
> > I was talking about these fields of struct elf_info:
> > 
> > 	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
> > 	 * take shndx from symtab_shndx_start[N] instead
> > 	 */
> > 	Elf32_Word   *symtab_shndx_start;
> > 	Elf32_Word   *symtab_shndx_stop;
> > 
> > They are not used after being filled in parse_elf() and their endianness
> > fixed in the end despite the comment.
> >   
> Its been a while since I wrote this, so I need to go back and look closely, but
> as you say, these values aren't used after parse_elf completes, but they are(I
> think) used to correct the endianess of the section header index table, so that
> get_sym_value works properly.  You would (again I think), only note a problem if
> you were parsing an ELF file for an architecture with an endianess opposite that
> of what you are building on (i.e. an x86 system cross compiling for a powerpc
> target).

Thanks for the explanation. Anyway, it's yet another point to prefer
pyelftools to parsing ELF manually.


[snip]
> > Native builds can use Python or PowerShell, which is as capable as Unix
> > shells, but incompatible with them. To call lib.exe (MSVC analog of ar) is
> > probably simpler then parsing COFF, yes. So meson could select one of the
> > following for a Windows target (objects are COFF):
> > 
> > Host     Toolchain  Archive  Script      Extractor
> > -------  ---------  -------  ---------   ---------
> > Linux    MinGW      .a       sh          ar
> > Windows  MinGW      .a       PowerShell  ar
> > Windows  Clang      .lib     PowerShell  lib  
> I think if I read you right, what you're suggesting here is that meson setup a
> build time marco $ARCHIVETOOL, and set it to ar or lib.exe depending on the
> build environment, and just have the script in question use $ARCHIVER?  If so,
> I'd be good with that.  Do we have to worry about the use of divergent command
> line options for each tool as well?

As I said, PowerShell cannot consume the same shell script. My suggestion was
to have two scripts, one existing for shell and ar (used by Linux host), and
one new PowerShell script for both lib.exe and ar (used by Windows host).
Meson then would select which script to call. Another option is to have one
wrapper Python script for all hosts: a few extra lines to launch archiver,
but less files and no need to know shell/PowerShell when changing this part.
(I believe Bruce means the same.)  But it will still be a separate script.

-- 
Dmitry Kozlyuk

Re: [dpdk-dev] [RFC PATCH v2 3/3] doc/linux_gsg: require pyelftools for pmdinfogen

[Neil Horman]

On Thu, Jul 02, 2020 at 03:02:32AM +0300, Dmitry Kozlyuk wrote:
> The package is available on all major distributions.
> FreeBSD has no system requirements section in its GSG,
> Windows currently neither uses pmdinfogen not is supported by it.
> 
Why doesn't windows use pmdinfogen?  That tool doesn't relate to DSO function
versioning, it just generates additional symbols for binaries so that we can see
what pmd support is in a given ELF file

Neil

> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> ---
>  doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
> index a124656bc..ef2037ca7 100644
> --- a/doc/guides/linux_gsg/sys_reqs.rst
> +++ b/doc/guides/linux_gsg/sys_reqs.rst
> @@ -56,6 +56,12 @@ Compilation of the DPDK
>      * If the packaged version is below the minimum version, the latest versions
>        can be installed from Python's "pip" repository: ``pip3 install meson ninja``
>  
> +*   ``pyelftools`` (version 0.22+)
> +
> +    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
> +
> +    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
> +
>  *   Library for handling NUMA (Non Uniform Memory Access).
>  
>      * ``numactl-devel`` in RHEL/Fedora;
> -- 
> 2.25.4
> 
> 

Re: [dpdk-dev] [RFC PATCH v2 3/3] doc/linux_gsg: require pyelftools for pmdinfogen

[Dmitry Kozlyuk]

On Mon, 6 Jul 2020 08:52:57 -0400, Neil Horman wrote:
> On Thu, Jul 02, 2020 at 03:02:32AM +0300, Dmitry Kozlyuk wrote:
> > The package is available on all major distributions.
> > FreeBSD has no system requirements section in its GSG,
> > Windows currently neither uses pmdinfogen not is supported by it.
> >   
> Why doesn't windows use pmdinfogen?  That tool doesn't relate to DSO function
> versioning, it just generates additional symbols for binaries so that we can see
> what pmd support is in a given ELF file

Windows uses COFF instead of ELF. I'm planning to add COFF support to
pmdinfogen in a separate patch and also put wrapper script there. In this
series I'd like to focus on just rewriting existing functionality.

-- 
Dmitry Kozlyuk

Re: [dpdk-dev] [RFC PATCH v2 3/3] doc/linux_gsg: require pyelftools for pmdinfogen

[Neil Horman]

On Mon, Jul 06, 2020 at 04:24:29PM +0300, Dmitry Kozlyuk wrote:
> On Mon, 6 Jul 2020 08:52:57 -0400, Neil Horman wrote:
> > On Thu, Jul 02, 2020 at 03:02:32AM +0300, Dmitry Kozlyuk wrote:
> > > The package is available on all major distributions.
> > > FreeBSD has no system requirements section in its GSG,
> > > Windows currently neither uses pmdinfogen not is supported by it.
> > >   
> > Why doesn't windows use pmdinfogen?  That tool doesn't relate to DSO function
> > versioning, it just generates additional symbols for binaries so that we can see
> > what pmd support is in a given ELF file
> 
> Windows uses COFF instead of ELF. I'm planning to add COFF support to
> pmdinfogen in a separate patch and also put wrapper script there. In this
> series I'd like to focus on just rewriting existing functionality.
> 
Ah, understood, thanks for the clarification

Neil

> -- 
> Dmitry Kozlyuk
> 

[dpdk-dev] [PATCH v3 0/4] pmdinfogen: rewrite in Python

[Dmitry Kozlyuk]

This patchset implements existing pmdinfogen logic in Python, replaces
and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf.

Travis CI script is adjusted to install python3-pyelftools, but other CI
systems may need similar tweaking. Particularly, testing on FreeBSD and
big-endian targets is desired.

---
Changes in v3:

1. Rebase on ToT, add package to CI.
2. Add copyrights, fixup maintainers file.
3. Remove C implementation.

Changes in v2:

1. pyelftools is used instead of hand-written ELF parser.
2. Makefiles are adjusted.
3. f-strings replaced with % formatting to support Python 3.5.
4. Wrapper script does not hide stderr from ar and pmdinfogen.

---
Dmitry Kozlyuk (4):
  pmdinfogen: add Python implementation
  build: use Python pmdinfogen
  doc/linux_gsg: require pyelftools for pmdinfogen
  pmdinfogen: remove C implementation

 .travis.yml                        |   2 +-
 GNUmakefile                        |   2 +-
 MAINTAINERS                        |   2 +-
 buildtools/Makefile                |   9 -
 buildtools/gen-pmdinfo-cfile.sh    |   6 +-
 buildtools/meson.build             |   1 +
 buildtools/pmdinfogen.py           | 194 ++++++++++++
 buildtools/pmdinfogen/Makefile     |  20 --
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 doc/guides/linux_gsg/sys_reqs.rst  |   6 +
 drivers/meson.build                |   2 +-
 meson.build                        |   1 -
 mk/internal/rte.compile-pre.mk     |   2 +-
 mk/rte.sdkinstall.mk               |   2 -
 16 files changed, 209 insertions(+), 629 deletions(-)
 delete mode 100644 buildtools/Makefile
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/Makefile
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

-- 
2.25.4

[dpdk-dev] [PATCH v3 1/4] pmdinfogen: add Python implementation

[Dmitry Kozlyuk]

Using a high-level, interpreted language simplifies maintenance and
build process. Furthermore, ELF handling is delegated to pyelftools
package. Original logic is kept, the copyright recognizes that.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 buildtools/pmdinfogen.py | 194 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 194 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py

diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
new file mode 100755
index 000000000..c32654a4a
--- /dev/null
+++ b/buildtools/pmdinfogen.py
@@ -0,0 +1,194 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <nhorman@tuxdriver.com>
+# Copyright (c) 2020 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image._image.get_section(self._symbol["st_shndx"]).data()
+        base = self._symbol["st_value"]
+        return data[base : base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        if len(value):
+            return value[:-1].decode()
+        return ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        if symbol:
+            return ELFSymbol(self, symbol[0])
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    def _load_pci_ids(self, image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        if table_symbol.size % pci_id_size != 0:
+            raise Exception(
+                "PCI table size (%d) is not a multiple of rte_pci_id size (%d)"
+                % (table_symbol.size, pci_id_size)
+            )
+
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            self.pci_ids.append(
+                [
+                    pci_id.vendor_id,
+                    pci_id.device_id,
+                    pci_id.subsystem_vendor_id,
+                    pci_id.subsystem_device_id,
+                ]
+            )
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
-- 
2.25.4

[dpdk-dev] [PATCH v3 2/4] build: use Python pmdinfogen

[Dmitry Kozlyuk]

Like for other build scripts, use Python interpreter to run pmdinfogen.
Adjust wrapper script accordingly and also don't suppress stderr from ar
and pmdinfogen.

Add python3-pyelftools to CI configuration.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 .travis.yml                     | 2 +-
 GNUmakefile                     | 2 +-
 buildtools/Makefile             | 9 ---------
 buildtools/gen-pmdinfo-cfile.sh | 6 +++---
 buildtools/meson.build          | 1 +
 drivers/meson.build             | 2 +-
 meson.build                     | 1 -
 mk/internal/rte.compile-pre.mk  | 2 +-
 mk/rte.sdkinstall.mk            | 2 --
 9 files changed, 8 insertions(+), 19 deletions(-)
 delete mode 100644 buildtools/Makefile

diff --git a/.travis.yml b/.travis.yml
index 14f812423..28b559a25 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -14,7 +14,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
diff --git a/GNUmakefile b/GNUmakefile
index e8de422df..242d30d2e 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -12,6 +12,6 @@ export RTE_SDK
 # directory list
 #
 
-ROOTDIRS-y := buildtools lib kernel drivers app
+ROOTDIRS-y := lib kernel drivers app
 
 include $(RTE_SDK)/mk/rte.sdkroot.mk
diff --git a/buildtools/Makefile b/buildtools/Makefile
deleted file mode 100644
index 7f76fd7d6..000000000
--- a/buildtools/Makefile
+++ /dev/null
@@ -1,9 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2016 Neil Horman <nhorman@tuxdriver.com>
-# All rights reserved.
-
-include $(RTE_SDK)/mk/rte.vars.mk
-
-DIRS-y += pmdinfogen
-
-include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/buildtools/gen-pmdinfo-cfile.sh b/buildtools/gen-pmdinfo-cfile.sh
index 43059cf36..109ee461e 100755
--- a/buildtools/gen-pmdinfo-cfile.sh
+++ b/buildtools/gen-pmdinfo-cfile.sh
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 04808dabc..3a64b28b7 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -17,3 +17,4 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
diff --git a/drivers/meson.build b/drivers/meson.build
index 161cfda04..3c4d4b700 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -119,7 +119,7 @@ foreach class:dpdk_driver_classes
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
diff --git a/meson.build b/meson.build
index 61d9a4f5f..a6e8c09c2 100644
--- a/meson.build
+++ b/meson.build
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
diff --git a/mk/internal/rte.compile-pre.mk b/mk/internal/rte.compile-pre.mk
index df05b5576..bb2ab0725 100644
--- a/mk/internal/rte.compile-pre.mk
+++ b/mk/internal/rte.compile-pre.mk
@@ -59,7 +59,7 @@ endif
 CHECK_SYMBOLS_SCRIPT = $(RTE_SDK)/buildtools/check-symbols.sh
 CHECK_SYMBOLS = $(CHECK_SYMBOLS_SCRIPT) $(SRCDIR)/$(EXPORT_MAP) $@
 
-PMDINFO_GEN = $(RTE_SDK_BIN)/app/dpdk-pmdinfogen $@ $@.pmd.c
+PMDINFO_GEN = $(RTE_SDK)/buildtools/pmdinfogen.py $@ $@.pmd.c
 PMDINFO_CC = $(CC) $(CPPFLAGS) $(CFLAGS) $(EXTRA_CFLAGS) -c -o $@.pmd.o $@.pmd.c
 PMDINFO_LD = $(CROSS)ld -r $(filter-out -export-dynamic,$(LDFLAGS)) -o $@.o $@.pmd.o $@
 PMDINFO_TO_O = if grep -q 'RTE_PMD_REGISTER_.*(.*)' $<; then \
diff --git a/mk/rte.sdkinstall.mk b/mk/rte.sdkinstall.mk
index 32bed5d95..875a64f04 100644
--- a/mk/rte.sdkinstall.mk
+++ b/mk/rte.sdkinstall.mk
@@ -98,7 +98,6 @@ install-runtime:
 	$(Q)cp $(CP_FLAGS)    $O/lib/* $(DESTDIR)$(libdir)
 	$(Q)$(call rte_mkdir, $(DESTDIR)$(bindir))
 	$(Q)tar -cf -      -C $O --exclude 'app/*.map' \
-		--exclude app/dpdk-pmdinfogen \
 		--exclude 'app/cmdline*' --exclude app/test \
 		--exclude app/testacl --exclude app/testpipeline app | \
 	    tar -xf -      -C $(DESTDIR)$(bindir) $(TAR_X_FLAGS)
@@ -134,7 +133,6 @@ install-sdk:
 	$(Q)cp $(CP_FLAGS)      $(RTE_SDK)/buildtools    $(DESTDIR)$(sdkdir)
 	$(Q)$(call rte_mkdir,                            $(DESTDIR)$(targetdir)/app)
 	$(Q)cp $(CP_FLAGS)      $O/.config               $(DESTDIR)$(targetdir)
-	$(Q)cp $(CP_FLAGS)      $O/app/dpdk-pmdinfogen   $(DESTDIR)$(targetdir)/app
 	$(Q)$(call rte_symlink, $(DESTDIR)$(includedir), $(DESTDIR)$(targetdir)/include)
 	$(Q)$(call rte_symlink, $(DESTDIR)$(libdir),     $(DESTDIR)$(targetdir)/lib)
 
-- 
2.25.4

[dpdk-dev] [PATCH v3 3/4] doc/linux_gsg: require pyelftools for pmdinfogen

[Dmitry Kozlyuk]

The package is available on all major distributions.
FreeBSD has no system requirements section in its GSG,
Windows currently neither uses pmdinfogen not is supported by it.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
index a124656bc..ef2037ca7 100644
--- a/doc/guides/linux_gsg/sys_reqs.rst
+++ b/doc/guides/linux_gsg/sys_reqs.rst
@@ -56,6 +56,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
-- 
2.25.4

[dpdk-dev] [PATCH v3 4/4] pmdinfogen: remove C implementation

[Dmitry Kozlyuk]

Delete the files no longer used in build process.  Keep Neil Horman
maintainer of the entire "Driver information" section, only adjust
paths.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 MAINTAINERS                        |   2 +-
 buildtools/pmdinfogen/Makefile     |  20 --
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 5 files changed, 1 insertion(+), 610 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/Makefile
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 6ba02d73a..93d209101 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -157,7 +157,7 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <nhorman@tuxdriver.com>
-F: buildtools/pmdinfogen/
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
diff --git a/buildtools/pmdinfogen/Makefile b/buildtools/pmdinfogen/Makefile
deleted file mode 100644
index a97a7648f..000000000
--- a/buildtools/pmdinfogen/Makefile
+++ /dev/null
@@ -1,20 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2016 Neil Horman <nhorman@tuxdriver.com>
-# All rights reserved.
-
-include $(RTE_SDK)/mk/rte.vars.mk
-
-#
-# library name
-#
-HOSTAPP = dpdk-pmdinfogen
-
-#
-# all sources are stored in SRCS-y
-#
-SRCS-y += pmdinfogen.c
-
-HOST_CFLAGS += $(HOST_WERROR_FLAGS) -g
-HOST_CFLAGS += -I$(RTE_OUTPUT)/include
-
-include $(RTE_SDK)/mk/rte.hostapp.mk
diff --git a/buildtools/pmdinfogen/meson.build b/buildtools/pmdinfogen/meson.build
deleted file mode 100644
index 670528fac..000000000
--- a/buildtools/pmdinfogen/meson.build
+++ /dev/null
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
diff --git a/buildtools/pmdinfogen/pmdinfogen.c b/buildtools/pmdinfogen/pmdinfogen.c
deleted file mode 100644
index a68d1ea99..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.c
+++ /dev/null
@@ -1,456 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <ctype.h>
-#include <string.h>
-#include <limits.h>
-#include <stdbool.h>
-#include <errno.h>
-#include <libgen.h>
-
-#include <rte_common.h>
-#include "pmdinfogen.h"
-
-#ifdef RTE_ARCH_64
-#define ADDR_SIZE 64
-#else
-#define ADDR_SIZE 32
-#endif
-
-static int use_stdin, use_stdout;
-
-static const char *sym_name(struct elf_info *elf, Elf_Sym *sym)
-{
-	if (sym)
-		return elf->strtab + sym->st_name;
-	else
-		return "(unknown)";
-}
-
-static void *grab_file(const char *filename, unsigned long *size)
-{
-	struct stat st;
-	void *map = MAP_FAILED;
-	int fd = -1;
-
-	if (!use_stdin) {
-		fd = open(filename, O_RDONLY);
-		if (fd < 0)
-			return NULL;
-	} else {
-		/* from stdin, use a temporary file to mmap */
-		FILE *infile;
-		char buffer[1024];
-		int n;
-
-		infile = tmpfile();
-		if (infile == NULL) {
-			perror("tmpfile");
-			return NULL;
-		}
-		fd = dup(fileno(infile));
-		fclose(infile);
-		if (fd < 0)
-			return NULL;
-
-		n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		while (n > 0) {
-			if (write(fd, buffer, n) != n)
-				goto failed;
-			n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		}
-	}
-
-	if (fstat(fd, &st))
-		goto failed;
-
-	*size = st.st_size;
-	map = mmap(NULL, *size, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);
-
-failed:
-	close(fd);
-	if (map == MAP_FAILED)
-		return NULL;
-	return map;
-}
-
-/**
-  * Return a copy of the next line in a mmap'ed file.
-  * spaces in the beginning of the line is trimmed away.
-  * Return a pointer to a static buffer.
-  **/
-static void release_file(void *file, unsigned long size)
-{
-	munmap(file, size);
-}
-
-
-static void *get_sym_value(struct elf_info *info, const Elf_Sym *sym)
-{
-	return RTE_PTR_ADD(info->hdr,
-		info->sechdrs[sym->st_shndx].sh_offset + sym->st_value);
-}
-
-static Elf_Sym *find_sym_in_symtab(struct elf_info *info,
-				   const char *name, Elf_Sym *last)
-{
-	Elf_Sym *idx;
-	if (last)
-		idx = last+1;
-	else
-		idx = info->symtab_start;
-
-	for (; idx < info->symtab_stop; idx++) {
-		const char *n = sym_name(info, idx);
-		if (!strncmp(n, name, strlen(name)))
-			return idx;
-	}
-	return NULL;
-}
-
-static int parse_elf(struct elf_info *info, const char *filename)
-{
-	unsigned int i;
-	Elf_Ehdr *hdr;
-	Elf_Shdr *sechdrs;
-	Elf_Sym  *sym;
-	int endian;
-	unsigned int symtab_idx = ~0U, symtab_shndx_idx = ~0U;
-
-	hdr = grab_file(filename, &info->size);
-	if (!hdr) {
-		perror(filename);
-		exit(1);
-	}
-	info->hdr = hdr;
-	if (info->size < sizeof(*hdr)) {
-		/* file too small, assume this is an empty .o file */
-		return 0;
-	}
-	/* Is this a valid ELF file? */
-	if ((hdr->e_ident[EI_MAG0] != ELFMAG0) ||
-	    (hdr->e_ident[EI_MAG1] != ELFMAG1) ||
-	    (hdr->e_ident[EI_MAG2] != ELFMAG2) ||
-	    (hdr->e_ident[EI_MAG3] != ELFMAG3)) {
-		/* Not an ELF file - silently ignore it */
-		return 0;
-	}
-
-	if (!hdr->e_ident[EI_DATA]) {
-		/* Unknown endian */
-		return 0;
-	}
-
-	endian = hdr->e_ident[EI_DATA];
-
-	/* Fix endianness in ELF header */
-	hdr->e_type      = TO_NATIVE(endian, 16, hdr->e_type);
-	hdr->e_machine   = TO_NATIVE(endian, 16, hdr->e_machine);
-	hdr->e_version   = TO_NATIVE(endian, 32, hdr->e_version);
-	hdr->e_entry     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_entry);
-	hdr->e_phoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_phoff);
-	hdr->e_shoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_shoff);
-	hdr->e_flags     = TO_NATIVE(endian, 32, hdr->e_flags);
-	hdr->e_ehsize    = TO_NATIVE(endian, 16, hdr->e_ehsize);
-	hdr->e_phentsize = TO_NATIVE(endian, 16, hdr->e_phentsize);
-	hdr->e_phnum     = TO_NATIVE(endian, 16, hdr->e_phnum);
-	hdr->e_shentsize = TO_NATIVE(endian, 16, hdr->e_shentsize);
-	hdr->e_shnum     = TO_NATIVE(endian, 16, hdr->e_shnum);
-	hdr->e_shstrndx  = TO_NATIVE(endian, 16, hdr->e_shstrndx);
-
-	sechdrs = RTE_PTR_ADD(hdr, hdr->e_shoff);
-	info->sechdrs = sechdrs;
-
-	/* Check if file offset is correct */
-	if (hdr->e_shoff > info->size) {
-		fprintf(stderr, "section header offset=%lu in file '%s' "
-		      "is bigger than filesize=%lu\n",
-		      (unsigned long)hdr->e_shoff,
-		      filename, info->size);
-		return 0;
-	}
-
-	if (hdr->e_shnum == SHN_UNDEF) {
-		/*
-		 * There are more than 64k sections,
-		 * read count from .sh_size.
-		 */
-		info->num_sections =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[0].sh_size);
-	} else {
-		info->num_sections = hdr->e_shnum;
-	}
-	if (hdr->e_shstrndx == SHN_XINDEX)
-		info->secindex_strings =
-			TO_NATIVE(endian, 32, sechdrs[0].sh_link);
-	else
-		info->secindex_strings = hdr->e_shstrndx;
-
-	/* Fix endianness in section headers */
-	for (i = 0; i < info->num_sections; i++) {
-		sechdrs[i].sh_name      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_name);
-		sechdrs[i].sh_type      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_type);
-		sechdrs[i].sh_flags     =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_flags);
-		sechdrs[i].sh_addr      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addr);
-		sechdrs[i].sh_offset    =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_offset);
-		sechdrs[i].sh_size      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_size);
-		sechdrs[i].sh_link      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_link);
-		sechdrs[i].sh_info      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_info);
-		sechdrs[i].sh_addralign =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addralign);
-		sechdrs[i].sh_entsize   =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_entsize);
-	}
-	/* Find symbol table. */
-	for (i = 1; i < info->num_sections; i++) {
-		int nobits = sechdrs[i].sh_type == SHT_NOBITS;
-
-		if (!nobits && sechdrs[i].sh_offset > info->size) {
-			fprintf(stderr, "%s is truncated. "
-			      "sechdrs[i].sh_offset=%lu > sizeof(*hrd)=%zu\n",
-			      filename, (unsigned long)sechdrs[i].sh_offset,
-			      sizeof(*hdr));
-			return 0;
-		}
-
-		if (sechdrs[i].sh_type == SHT_SYMTAB) {
-			unsigned int sh_link_idx;
-			symtab_idx = i;
-			info->symtab_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-			sh_link_idx = sechdrs[i].sh_link;
-			info->strtab       = RTE_PTR_ADD(hdr,
-				sechdrs[sh_link_idx].sh_offset);
-		}
-
-		/* 32bit section no. table? ("more than 64k sections") */
-		if (sechdrs[i].sh_type == SHT_SYMTAB_SHNDX) {
-			symtab_shndx_idx = i;
-			info->symtab_shndx_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_shndx_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-		}
-	}
-	if (!info->symtab_start)
-		fprintf(stderr, "%s has no symtab?\n", filename);
-	else {
-		/* Fix endianness in symbols */
-		for (sym = info->symtab_start; sym < info->symtab_stop; sym++) {
-			sym->st_shndx = TO_NATIVE(endian, 16, sym->st_shndx);
-			sym->st_name  = TO_NATIVE(endian, 32, sym->st_name);
-			sym->st_value = TO_NATIVE(endian, ADDR_SIZE, sym->st_value);
-			sym->st_size  = TO_NATIVE(endian, ADDR_SIZE, sym->st_size);
-		}
-	}
-
-	if (symtab_shndx_idx != ~0U) {
-		Elf32_Word *p;
-		if (symtab_idx != sechdrs[symtab_shndx_idx].sh_link)
-			fprintf(stderr,
-			      "%s: SYMTAB_SHNDX has bad sh_link: %u!=%u\n",
-			      filename, sechdrs[symtab_shndx_idx].sh_link,
-			      symtab_idx);
-		/* Fix endianness */
-		for (p = info->symtab_shndx_start; p < info->symtab_shndx_stop;
-		     p++)
-			*p = TO_NATIVE(endian, 32, *p);
-	}
-
-	return 1;
-}
-
-static void parse_elf_finish(struct elf_info *info)
-{
-	struct pmd_driver *tmp, *idx = info->drivers;
-	release_file(info->hdr, info->size);
-	while (idx) {
-		tmp = idx->next;
-		free(idx);
-		idx = tmp;
-	}
-}
-
-struct opt_tag {
-	const char *suffix;
-	const char *json_id;
-};
-
-static const struct opt_tag opt_tags[] = {
-	{"_param_string_export", "params"},
-	{"_kmod_dep_export", "kmod"},
-};
-
-static int complete_pmd_entry(struct elf_info *info, struct pmd_driver *drv)
-{
-	const char *tname;
-	int i;
-	char tmpsymname[128];
-	Elf_Sym *tmpsym;
-
-	drv->name = get_sym_value(info, drv->name_sym);
-
-	for (i = 0; i < PMD_OPT_MAX; i++) {
-		memset(tmpsymname, 0, 128);
-		sprintf(tmpsymname, "__%s%s", drv->name, opt_tags[i].suffix);
-		tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-		if (!tmpsym)
-			continue;
-		drv->opt_vals[i] = get_sym_value(info, tmpsym);
-	}
-
-	memset(tmpsymname, 0, 128);
-	sprintf(tmpsymname, "__%s_pci_tbl_export", drv->name);
-
-	tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-
-
-	/*
-	 * If this returns NULL, then this is a PMD_VDEV, because
-	 * it has no pci table reference
-	 */
-	if (!tmpsym) {
-		drv->pci_tbl = NULL;
-		return 0;
-	}
-
-	tname = get_sym_value(info, tmpsym);
-	tmpsym = find_sym_in_symtab(info, tname, NULL);
-	if (!tmpsym)
-		return -ENOENT;
-
-	drv->pci_tbl = (struct rte_pci_id *)get_sym_value(info, tmpsym);
-	if (!drv->pci_tbl)
-		return -ENOENT;
-
-	return 0;
-}
-
-static int locate_pmd_entries(struct elf_info *info)
-{
-	Elf_Sym *last = NULL;
-	struct pmd_driver *new;
-
-	info->drivers = NULL;
-
-	do {
-		new = calloc(sizeof(struct pmd_driver), 1);
-		if (new == NULL) {
-			fprintf(stderr, "Failed to calloc memory\n");
-			return -1;
-		}
-		new->name_sym = find_sym_in_symtab(info, "this_pmd_name", last);
-		last = new->name_sym;
-		if (!new->name_sym)
-			free(new);
-		else {
-			if (complete_pmd_entry(info, new)) {
-				fprintf(stderr,
-					"Failed to complete pmd entry\n");
-				free(new);
-			} else {
-				new->next = info->drivers;
-				info->drivers = new;
-			}
-		}
-	} while (last);
-
-	return 0;
-}
-
-static void output_pmd_info_string(struct elf_info *info, char *outfile)
-{
-	FILE *ofd;
-	struct pmd_driver *drv;
-	struct rte_pci_id *pci_ids;
-	int idx = 0;
-
-	if (use_stdout)
-		ofd = stdout;
-	else {
-		ofd = fopen(outfile, "w+");
-		if (!ofd) {
-			fprintf(stderr, "Unable to open output file\n");
-			return;
-		}
-	}
-
-	drv = info->drivers;
-
-	while (drv) {
-		fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
-			"\"PMD_INFO_STRING= {",
-			drv->name);
-		fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);
-
-		for (idx = 0; idx < PMD_OPT_MAX; idx++) {
-			if (drv->opt_vals[idx])
-				fprintf(ofd, "\\\"%s\\\" : \\\"%s\\\", ",
-					opt_tags[idx].json_id,
-					drv->opt_vals[idx]);
-		}
-
-		pci_ids = drv->pci_tbl;
-		fprintf(ofd, "\\\"pci_ids\\\" : [");
-
-		while (pci_ids && pci_ids->device_id) {
-			fprintf(ofd, "[%d, %d, %d, %d]",
-				pci_ids->vendor_id, pci_ids->device_id,
-				pci_ids->subsystem_vendor_id,
-				pci_ids->subsystem_device_id);
-			pci_ids++;
-			if (pci_ids->device_id)
-				fprintf(ofd, ",");
-			else
-				fprintf(ofd, " ");
-		}
-		fprintf(ofd, "]}\";\n");
-		drv = drv->next;
-	}
-
-	fclose(ofd);
-}
-
-int main(int argc, char **argv)
-{
-	struct elf_info info = {0};
-	int rc = 1;
-
-	if (argc < 3) {
-		fprintf(stderr,
-			"usage: %s <object file> <c output file>\n",
-			basename(argv[0]));
-		exit(127);
-	}
-	use_stdin = !strcmp(argv[1], "-");
-	use_stdout = !strcmp(argv[2], "-");
-	parse_elf(&info, argv[1]);
-
-	if (locate_pmd_entries(&info) < 0)
-		exit(1);
-
-	if (info.drivers) {
-		output_pmd_info_string(&info, argv[2]);
-		rc = 0;
-	} else {
-		fprintf(stderr, "No drivers registered\n");
-	}
-
-	parse_elf_finish(&info);
-	exit(rc);
-}
diff --git a/buildtools/pmdinfogen/pmdinfogen.h b/buildtools/pmdinfogen/pmdinfogen.h
deleted file mode 100644
index 93930e454..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.h
+++ /dev/null
@@ -1,119 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <stdarg.h>
-#include <string.h>
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#ifdef __linux__
-#include <endian.h>
-#else
-#include <sys/endian.h>
-#endif
-#include <fcntl.h>
-#include <unistd.h>
-#include <elf.h>
-#include <rte_pci.h>
-
-/* On BSD-alike OSes elf.h defines these according to host's word size */
-#undef ELF_ST_BIND
-#undef ELF_ST_TYPE
-#undef ELF_R_SYM
-#undef ELF_R_TYPE
-
-/*
- * Define ELF64_* to ELF_*, the latter being defined in both 32 and 64 bit
- * flavors in elf.h.  This makes our code a bit more generic between arches
- * and allows us to support 32 bit code in the future should we ever want to
- */
-#ifdef RTE_ARCH_64
-#define Elf_Ehdr    Elf64_Ehdr
-#define Elf_Shdr    Elf64_Shdr
-#define Elf_Sym     Elf64_Sym
-#define Elf_Addr    Elf64_Addr
-#define Elf_Sword   Elf64_Sxword
-#define Elf_Section Elf64_Half
-#define ELF_ST_BIND ELF64_ST_BIND
-#define ELF_ST_TYPE ELF64_ST_TYPE
-
-#define Elf_Rel     Elf64_Rel
-#define Elf_Rela    Elf64_Rela
-#define ELF_R_SYM   ELF64_R_SYM
-#define ELF_R_TYPE  ELF64_R_TYPE
-#else
-#define Elf_Ehdr    Elf32_Ehdr
-#define Elf_Shdr    Elf32_Shdr
-#define Elf_Sym     Elf32_Sym
-#define Elf_Addr    Elf32_Addr
-#define Elf_Sword   Elf32_Sxword
-#define Elf_Section Elf32_Half
-#define ELF_ST_BIND ELF32_ST_BIND
-#define ELF_ST_TYPE ELF32_ST_TYPE
-
-#define Elf_Rel     Elf32_Rel
-#define Elf_Rela    Elf32_Rela
-#define ELF_R_SYM   ELF32_R_SYM
-#define ELF_R_TYPE  ELF32_R_TYPE
-#endif
-
-
-/*
- * Note, it seems odd that we have both a CONVERT_NATIVE and a TO_NATIVE macro
- * below.  We do this because the values passed to TO_NATIVE may themselves be
- * macros and need both macros here to get expanded.  Specifically its the width
- * variable we are concerned with, because it needs to get expanded prior to
- * string concatenation
- */
-#define CONVERT_NATIVE(fend, width, x) ({ \
-typeof(x) ___x; \
-if ((fend) == ELFDATA2LSB) \
-	___x = le##width##toh(x); \
-else \
-	___x = be##width##toh(x); \
-	___x; \
-})
-
-#define TO_NATIVE(fend, width, x) CONVERT_NATIVE(fend, width, x)
-
-enum opt_params {
-	PMD_PARAM_STRING = 0,
-	PMD_KMOD_DEP,
-	PMD_OPT_MAX
-};
-
-struct pmd_driver {
-	Elf_Sym *name_sym;
-	const char *name;
-	struct rte_pci_id *pci_tbl;
-	struct pmd_driver *next;
-
-	const char *opt_vals[PMD_OPT_MAX];
-};
-
-struct elf_info {
-	unsigned long size;
-	Elf_Ehdr     *hdr;
-	Elf_Shdr     *sechdrs;
-	Elf_Sym      *symtab_start;
-	Elf_Sym      *symtab_stop;
-	char         *strtab;
-
-	/* support for 32bit section numbers */
-
-	unsigned int num_sections; /* max_secindex + 1 */
-	unsigned int secindex_strings;
-	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
-	 * take shndx from symtab_shndx_start[N] instead
-	 */
-	Elf32_Word   *symtab_shndx_start;
-	Elf32_Word   *symtab_shndx_stop;
-
-	struct pmd_driver *drivers;
-};
-- 
2.25.4

[dpdk-dev] [PATCH v4 0/4] pmdinfogen: rewrite in Python

[Dmitry Kozlyuk]

This patchset implements existing pmdinfogen logic in Python, replaces
and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf.

Travis CI script is adjusted to install python3-pyelftools, but other CI
systems may need similar tweaking. Particularly, testing on FreeBSD and
big-endian targets is desired.

---
Changes in v4:

Fix build by removing redundant check.

Changes in v3:

1. Rebase on ToT, add package to CI.
2. Add copyrights, fixup maintainers file.
3. Remove C implementation.

Changes in v2:

1. pyelftools is used instead of hand-written ELF parser.
2. Makefiles are adjusted.
3. f-strings replaced with % formatting to support Python 3.5.
4. Wrapper script does not hide stderr from ar and pmdinfogen.

---
Dmitry Kozlyuk (4):
  pmdinfogen: add Python implementation
  build: use Python pmdinfogen
  doc/linux_gsg: require pyelftools for pmdinfogen
  pmdinfogen: remove C implementation

 .travis.yml                        |   2 +-
 GNUmakefile                        |   2 +-
 MAINTAINERS                        |   2 +-
 buildtools/Makefile                |   9 -
 buildtools/gen-pmdinfo-cfile.sh    |   6 +-
 buildtools/meson.build             |   1 +
 buildtools/pmdinfogen.py           | 188 ++++++++++++
 buildtools/pmdinfogen/Makefile     |  20 --
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 doc/guides/linux_gsg/sys_reqs.rst  |   6 +
 drivers/meson.build                |   2 +-
 meson.build                        |   1 -
 mk/internal/rte.compile-pre.mk     |   2 +-
 mk/rte.sdkinstall.mk               |   2 -
 16 files changed, 203 insertions(+), 629 deletions(-)
 delete mode 100644 buildtools/Makefile
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/Makefile
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

-- 
2.25.4

[dpdk-dev] [PATCH v4 1/4] pmdinfogen: add Python implementation

[Dmitry Kozlyuk]

Using a high-level, interpreted language simplifies maintenance and
build process. Furthermore, ELF handling is delegated to pyelftools
package. Original logic is kept, the copyright recognizes that.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 buildtools/pmdinfogen.py | 188 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 188 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py

diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
new file mode 100755
index 000000000..a9c2643a4
--- /dev/null
+++ b/buildtools/pmdinfogen.py
@@ -0,0 +1,188 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <nhorman@tuxdriver.com>
+# Copyright (c) 2020 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image._image.get_section(self._symbol["st_shndx"]).data()
+        base = self._symbol["st_value"]
+        return data[base : base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        if len(value):
+            return value[:-1].decode()
+        return ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        if symbol:
+            return ELFSymbol(self, symbol[0])
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    def _load_pci_ids(self, image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            self.pci_ids.append(
+                [
+                    pci_id.vendor_id,
+                    pci_id.device_id,
+                    pci_id.subsystem_vendor_id,
+                    pci_id.subsystem_device_id,
+                ]
+            )
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
-- 
2.25.4

[dpdk-dev] [PATCH v4 2/4] build: use Python pmdinfogen

[Dmitry Kozlyuk]

Like for other build scripts, use Python interpreter to run pmdinfogen.
Adjust wrapper script accordingly and also don't suppress stderr from ar
and pmdinfogen.

Add python3-pyelftools to CI configuration.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 .travis.yml                     | 2 +-
 GNUmakefile                     | 2 +-
 buildtools/Makefile             | 9 ---------
 buildtools/gen-pmdinfo-cfile.sh | 6 +++---
 buildtools/meson.build          | 1 +
 drivers/meson.build             | 2 +-
 meson.build                     | 1 -
 mk/internal/rte.compile-pre.mk  | 2 +-
 mk/rte.sdkinstall.mk            | 2 --
 9 files changed, 8 insertions(+), 19 deletions(-)
 delete mode 100644 buildtools/Makefile

diff --git a/.travis.yml b/.travis.yml
index 14f812423..28b559a25 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -14,7 +14,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
diff --git a/GNUmakefile b/GNUmakefile
index e8de422df..242d30d2e 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -12,6 +12,6 @@ export RTE_SDK
 # directory list
 #
 
-ROOTDIRS-y := buildtools lib kernel drivers app
+ROOTDIRS-y := lib kernel drivers app
 
 include $(RTE_SDK)/mk/rte.sdkroot.mk
diff --git a/buildtools/Makefile b/buildtools/Makefile
deleted file mode 100644
index 7f76fd7d6..000000000
--- a/buildtools/Makefile
+++ /dev/null
@@ -1,9 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2016 Neil Horman <nhorman@tuxdriver.com>
-# All rights reserved.
-
-include $(RTE_SDK)/mk/rte.vars.mk
-
-DIRS-y += pmdinfogen
-
-include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/buildtools/gen-pmdinfo-cfile.sh b/buildtools/gen-pmdinfo-cfile.sh
index 43059cf36..109ee461e 100755
--- a/buildtools/gen-pmdinfo-cfile.sh
+++ b/buildtools/gen-pmdinfo-cfile.sh
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 04808dabc..3a64b28b7 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -17,3 +17,4 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
diff --git a/drivers/meson.build b/drivers/meson.build
index 161cfda04..3c4d4b700 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -119,7 +119,7 @@ foreach class:dpdk_driver_classes
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
diff --git a/meson.build b/meson.build
index 61d9a4f5f..a6e8c09c2 100644
--- a/meson.build
+++ b/meson.build
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
diff --git a/mk/internal/rte.compile-pre.mk b/mk/internal/rte.compile-pre.mk
index df05b5576..bb2ab0725 100644
--- a/mk/internal/rte.compile-pre.mk
+++ b/mk/internal/rte.compile-pre.mk
@@ -59,7 +59,7 @@ endif
 CHECK_SYMBOLS_SCRIPT = $(RTE_SDK)/buildtools/check-symbols.sh
 CHECK_SYMBOLS = $(CHECK_SYMBOLS_SCRIPT) $(SRCDIR)/$(EXPORT_MAP) $@
 
-PMDINFO_GEN = $(RTE_SDK_BIN)/app/dpdk-pmdinfogen $@ $@.pmd.c
+PMDINFO_GEN = $(RTE_SDK)/buildtools/pmdinfogen.py $@ $@.pmd.c
 PMDINFO_CC = $(CC) $(CPPFLAGS) $(CFLAGS) $(EXTRA_CFLAGS) -c -o $@.pmd.o $@.pmd.c
 PMDINFO_LD = $(CROSS)ld -r $(filter-out -export-dynamic,$(LDFLAGS)) -o $@.o $@.pmd.o $@
 PMDINFO_TO_O = if grep -q 'RTE_PMD_REGISTER_.*(.*)' $<; then \
diff --git a/mk/rte.sdkinstall.mk b/mk/rte.sdkinstall.mk
index 32bed5d95..875a64f04 100644
--- a/mk/rte.sdkinstall.mk
+++ b/mk/rte.sdkinstall.mk
@@ -98,7 +98,6 @@ install-runtime:
 	$(Q)cp $(CP_FLAGS)    $O/lib/* $(DESTDIR)$(libdir)
 	$(Q)$(call rte_mkdir, $(DESTDIR)$(bindir))
 	$(Q)tar -cf -      -C $O --exclude 'app/*.map' \
-		--exclude app/dpdk-pmdinfogen \
 		--exclude 'app/cmdline*' --exclude app/test \
 		--exclude app/testacl --exclude app/testpipeline app | \
 	    tar -xf -      -C $(DESTDIR)$(bindir) $(TAR_X_FLAGS)
@@ -134,7 +133,6 @@ install-sdk:
 	$(Q)cp $(CP_FLAGS)      $(RTE_SDK)/buildtools    $(DESTDIR)$(sdkdir)
 	$(Q)$(call rte_mkdir,                            $(DESTDIR)$(targetdir)/app)
 	$(Q)cp $(CP_FLAGS)      $O/.config               $(DESTDIR)$(targetdir)
-	$(Q)cp $(CP_FLAGS)      $O/app/dpdk-pmdinfogen   $(DESTDIR)$(targetdir)/app
 	$(Q)$(call rte_symlink, $(DESTDIR)$(includedir), $(DESTDIR)$(targetdir)/include)
 	$(Q)$(call rte_symlink, $(DESTDIR)$(libdir),     $(DESTDIR)$(targetdir)/lib)
 
-- 
2.25.4

[dpdk-dev] [PATCH v4 3/4] doc/linux_gsg: require pyelftools for pmdinfogen

[Dmitry Kozlyuk]

The package is available on all major distributions.
FreeBSD has no system requirements section in its GSG,
Windows currently neither uses pmdinfogen not is supported by it.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
index a124656bc..ef2037ca7 100644
--- a/doc/guides/linux_gsg/sys_reqs.rst
+++ b/doc/guides/linux_gsg/sys_reqs.rst
@@ -56,6 +56,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
-- 
2.25.4

[dpdk-dev] [PATCH v4 4/4] pmdinfogen: remove C implementation

[Dmitry Kozlyuk]

Delete the files no longer used in build process.  Keep Neil Horman
maintainer of the entire "Driver information" section, only adjust
paths.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 MAINTAINERS                        |   2 +-
 buildtools/pmdinfogen/Makefile     |  20 --
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 5 files changed, 1 insertion(+), 610 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/Makefile
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 2bb8583f3..68046aaad 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -157,7 +157,7 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <nhorman@tuxdriver.com>
-F: buildtools/pmdinfogen/
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
diff --git a/buildtools/pmdinfogen/Makefile b/buildtools/pmdinfogen/Makefile
deleted file mode 100644
index a97a7648f..000000000
--- a/buildtools/pmdinfogen/Makefile
+++ /dev/null
@@ -1,20 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2016 Neil Horman <nhorman@tuxdriver.com>
-# All rights reserved.
-
-include $(RTE_SDK)/mk/rte.vars.mk
-
-#
-# library name
-#
-HOSTAPP = dpdk-pmdinfogen
-
-#
-# all sources are stored in SRCS-y
-#
-SRCS-y += pmdinfogen.c
-
-HOST_CFLAGS += $(HOST_WERROR_FLAGS) -g
-HOST_CFLAGS += -I$(RTE_OUTPUT)/include
-
-include $(RTE_SDK)/mk/rte.hostapp.mk
diff --git a/buildtools/pmdinfogen/meson.build b/buildtools/pmdinfogen/meson.build
deleted file mode 100644
index 670528fac..000000000
--- a/buildtools/pmdinfogen/meson.build
+++ /dev/null
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
diff --git a/buildtools/pmdinfogen/pmdinfogen.c b/buildtools/pmdinfogen/pmdinfogen.c
deleted file mode 100644
index a68d1ea99..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.c
+++ /dev/null
@@ -1,456 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <ctype.h>
-#include <string.h>
-#include <limits.h>
-#include <stdbool.h>
-#include <errno.h>
-#include <libgen.h>
-
-#include <rte_common.h>
-#include "pmdinfogen.h"
-
-#ifdef RTE_ARCH_64
-#define ADDR_SIZE 64
-#else
-#define ADDR_SIZE 32
-#endif
-
-static int use_stdin, use_stdout;
-
-static const char *sym_name(struct elf_info *elf, Elf_Sym *sym)
-{
-	if (sym)
-		return elf->strtab + sym->st_name;
-	else
-		return "(unknown)";
-}
-
-static void *grab_file(const char *filename, unsigned long *size)
-{
-	struct stat st;
-	void *map = MAP_FAILED;
-	int fd = -1;
-
-	if (!use_stdin) {
-		fd = open(filename, O_RDONLY);
-		if (fd < 0)
-			return NULL;
-	} else {
-		/* from stdin, use a temporary file to mmap */
-		FILE *infile;
-		char buffer[1024];
-		int n;
-
-		infile = tmpfile();
-		if (infile == NULL) {
-			perror("tmpfile");
-			return NULL;
-		}
-		fd = dup(fileno(infile));
-		fclose(infile);
-		if (fd < 0)
-			return NULL;
-
-		n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		while (n > 0) {
-			if (write(fd, buffer, n) != n)
-				goto failed;
-			n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		}
-	}
-
-	if (fstat(fd, &st))
-		goto failed;
-
-	*size = st.st_size;
-	map = mmap(NULL, *size, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);
-
-failed:
-	close(fd);
-	if (map == MAP_FAILED)
-		return NULL;
-	return map;
-}
-
-/**
-  * Return a copy of the next line in a mmap'ed file.
-  * spaces in the beginning of the line is trimmed away.
-  * Return a pointer to a static buffer.
-  **/
-static void release_file(void *file, unsigned long size)
-{
-	munmap(file, size);
-}
-
-
-static void *get_sym_value(struct elf_info *info, const Elf_Sym *sym)
-{
-	return RTE_PTR_ADD(info->hdr,
-		info->sechdrs[sym->st_shndx].sh_offset + sym->st_value);
-}
-
-static Elf_Sym *find_sym_in_symtab(struct elf_info *info,
-				   const char *name, Elf_Sym *last)
-{
-	Elf_Sym *idx;
-	if (last)
-		idx = last+1;
-	else
-		idx = info->symtab_start;
-
-	for (; idx < info->symtab_stop; idx++) {
-		const char *n = sym_name(info, idx);
-		if (!strncmp(n, name, strlen(name)))
-			return idx;
-	}
-	return NULL;
-}
-
-static int parse_elf(struct elf_info *info, const char *filename)
-{
-	unsigned int i;
-	Elf_Ehdr *hdr;
-	Elf_Shdr *sechdrs;
-	Elf_Sym  *sym;
-	int endian;
-	unsigned int symtab_idx = ~0U, symtab_shndx_idx = ~0U;
-
-	hdr = grab_file(filename, &info->size);
-	if (!hdr) {
-		perror(filename);
-		exit(1);
-	}
-	info->hdr = hdr;
-	if (info->size < sizeof(*hdr)) {
-		/* file too small, assume this is an empty .o file */
-		return 0;
-	}
-	/* Is this a valid ELF file? */
-	if ((hdr->e_ident[EI_MAG0] != ELFMAG0) ||
-	    (hdr->e_ident[EI_MAG1] != ELFMAG1) ||
-	    (hdr->e_ident[EI_MAG2] != ELFMAG2) ||
-	    (hdr->e_ident[EI_MAG3] != ELFMAG3)) {
-		/* Not an ELF file - silently ignore it */
-		return 0;
-	}
-
-	if (!hdr->e_ident[EI_DATA]) {
-		/* Unknown endian */
-		return 0;
-	}
-
-	endian = hdr->e_ident[EI_DATA];
-
-	/* Fix endianness in ELF header */
-	hdr->e_type      = TO_NATIVE(endian, 16, hdr->e_type);
-	hdr->e_machine   = TO_NATIVE(endian, 16, hdr->e_machine);
-	hdr->e_version   = TO_NATIVE(endian, 32, hdr->e_version);
-	hdr->e_entry     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_entry);
-	hdr->e_phoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_phoff);
-	hdr->e_shoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_shoff);
-	hdr->e_flags     = TO_NATIVE(endian, 32, hdr->e_flags);
-	hdr->e_ehsize    = TO_NATIVE(endian, 16, hdr->e_ehsize);
-	hdr->e_phentsize = TO_NATIVE(endian, 16, hdr->e_phentsize);
-	hdr->e_phnum     = TO_NATIVE(endian, 16, hdr->e_phnum);
-	hdr->e_shentsize = TO_NATIVE(endian, 16, hdr->e_shentsize);
-	hdr->e_shnum     = TO_NATIVE(endian, 16, hdr->e_shnum);
-	hdr->e_shstrndx  = TO_NATIVE(endian, 16, hdr->e_shstrndx);
-
-	sechdrs = RTE_PTR_ADD(hdr, hdr->e_shoff);
-	info->sechdrs = sechdrs;
-
-	/* Check if file offset is correct */
-	if (hdr->e_shoff > info->size) {
-		fprintf(stderr, "section header offset=%lu in file '%s' "
-		      "is bigger than filesize=%lu\n",
-		      (unsigned long)hdr->e_shoff,
-		      filename, info->size);
-		return 0;
-	}
-
-	if (hdr->e_shnum == SHN_UNDEF) {
-		/*
-		 * There are more than 64k sections,
-		 * read count from .sh_size.
-		 */
-		info->num_sections =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[0].sh_size);
-	} else {
-		info->num_sections = hdr->e_shnum;
-	}
-	if (hdr->e_shstrndx == SHN_XINDEX)
-		info->secindex_strings =
-			TO_NATIVE(endian, 32, sechdrs[0].sh_link);
-	else
-		info->secindex_strings = hdr->e_shstrndx;
-
-	/* Fix endianness in section headers */
-	for (i = 0; i < info->num_sections; i++) {
-		sechdrs[i].sh_name      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_name);
-		sechdrs[i].sh_type      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_type);
-		sechdrs[i].sh_flags     =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_flags);
-		sechdrs[i].sh_addr      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addr);
-		sechdrs[i].sh_offset    =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_offset);
-		sechdrs[i].sh_size      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_size);
-		sechdrs[i].sh_link      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_link);
-		sechdrs[i].sh_info      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_info);
-		sechdrs[i].sh_addralign =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addralign);
-		sechdrs[i].sh_entsize   =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_entsize);
-	}
-	/* Find symbol table. */
-	for (i = 1; i < info->num_sections; i++) {
-		int nobits = sechdrs[i].sh_type == SHT_NOBITS;
-
-		if (!nobits && sechdrs[i].sh_offset > info->size) {
-			fprintf(stderr, "%s is truncated. "
-			      "sechdrs[i].sh_offset=%lu > sizeof(*hrd)=%zu\n",
-			      filename, (unsigned long)sechdrs[i].sh_offset,
-			      sizeof(*hdr));
-			return 0;
-		}
-
-		if (sechdrs[i].sh_type == SHT_SYMTAB) {
-			unsigned int sh_link_idx;
-			symtab_idx = i;
-			info->symtab_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-			sh_link_idx = sechdrs[i].sh_link;
-			info->strtab       = RTE_PTR_ADD(hdr,
-				sechdrs[sh_link_idx].sh_offset);
-		}
-
-		/* 32bit section no. table? ("more than 64k sections") */
-		if (sechdrs[i].sh_type == SHT_SYMTAB_SHNDX) {
-			symtab_shndx_idx = i;
-			info->symtab_shndx_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_shndx_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-		}
-	}
-	if (!info->symtab_start)
-		fprintf(stderr, "%s has no symtab?\n", filename);
-	else {
-		/* Fix endianness in symbols */
-		for (sym = info->symtab_start; sym < info->symtab_stop; sym++) {
-			sym->st_shndx = TO_NATIVE(endian, 16, sym->st_shndx);
-			sym->st_name  = TO_NATIVE(endian, 32, sym->st_name);
-			sym->st_value = TO_NATIVE(endian, ADDR_SIZE, sym->st_value);
-			sym->st_size  = TO_NATIVE(endian, ADDR_SIZE, sym->st_size);
-		}
-	}
-
-	if (symtab_shndx_idx != ~0U) {
-		Elf32_Word *p;
-		if (symtab_idx != sechdrs[symtab_shndx_idx].sh_link)
-			fprintf(stderr,
-			      "%s: SYMTAB_SHNDX has bad sh_link: %u!=%u\n",
-			      filename, sechdrs[symtab_shndx_idx].sh_link,
-			      symtab_idx);
-		/* Fix endianness */
-		for (p = info->symtab_shndx_start; p < info->symtab_shndx_stop;
-		     p++)
-			*p = TO_NATIVE(endian, 32, *p);
-	}
-
-	return 1;
-}
-
-static void parse_elf_finish(struct elf_info *info)
-{
-	struct pmd_driver *tmp, *idx = info->drivers;
-	release_file(info->hdr, info->size);
-	while (idx) {
-		tmp = idx->next;
-		free(idx);
-		idx = tmp;
-	}
-}
-
-struct opt_tag {
-	const char *suffix;
-	const char *json_id;
-};
-
-static const struct opt_tag opt_tags[] = {
-	{"_param_string_export", "params"},
-	{"_kmod_dep_export", "kmod"},
-};
-
-static int complete_pmd_entry(struct elf_info *info, struct pmd_driver *drv)
-{
-	const char *tname;
-	int i;
-	char tmpsymname[128];
-	Elf_Sym *tmpsym;
-
-	drv->name = get_sym_value(info, drv->name_sym);
-
-	for (i = 0; i < PMD_OPT_MAX; i++) {
-		memset(tmpsymname, 0, 128);
-		sprintf(tmpsymname, "__%s%s", drv->name, opt_tags[i].suffix);
-		tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-		if (!tmpsym)
-			continue;
-		drv->opt_vals[i] = get_sym_value(info, tmpsym);
-	}
-
-	memset(tmpsymname, 0, 128);
-	sprintf(tmpsymname, "__%s_pci_tbl_export", drv->name);
-
-	tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-
-
-	/*
-	 * If this returns NULL, then this is a PMD_VDEV, because
-	 * it has no pci table reference
-	 */
-	if (!tmpsym) {
-		drv->pci_tbl = NULL;
-		return 0;
-	}
-
-	tname = get_sym_value(info, tmpsym);
-	tmpsym = find_sym_in_symtab(info, tname, NULL);
-	if (!tmpsym)
-		return -ENOENT;
-
-	drv->pci_tbl = (struct rte_pci_id *)get_sym_value(info, tmpsym);
-	if (!drv->pci_tbl)
-		return -ENOENT;
-
-	return 0;
-}
-
-static int locate_pmd_entries(struct elf_info *info)
-{
-	Elf_Sym *last = NULL;
-	struct pmd_driver *new;
-
-	info->drivers = NULL;
-
-	do {
-		new = calloc(sizeof(struct pmd_driver), 1);
-		if (new == NULL) {
-			fprintf(stderr, "Failed to calloc memory\n");
-			return -1;
-		}
-		new->name_sym = find_sym_in_symtab(info, "this_pmd_name", last);
-		last = new->name_sym;
-		if (!new->name_sym)
-			free(new);
-		else {
-			if (complete_pmd_entry(info, new)) {
-				fprintf(stderr,
-					"Failed to complete pmd entry\n");
-				free(new);
-			} else {
-				new->next = info->drivers;
-				info->drivers = new;
-			}
-		}
-	} while (last);
-
-	return 0;
-}
-
-static void output_pmd_info_string(struct elf_info *info, char *outfile)
-{
-	FILE *ofd;
-	struct pmd_driver *drv;
-	struct rte_pci_id *pci_ids;
-	int idx = 0;
-
-	if (use_stdout)
-		ofd = stdout;
-	else {
-		ofd = fopen(outfile, "w+");
-		if (!ofd) {
-			fprintf(stderr, "Unable to open output file\n");
-			return;
-		}
-	}
-
-	drv = info->drivers;
-
-	while (drv) {
-		fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
-			"\"PMD_INFO_STRING= {",
-			drv->name);
-		fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);
-
-		for (idx = 0; idx < PMD_OPT_MAX; idx++) {
-			if (drv->opt_vals[idx])
-				fprintf(ofd, "\\\"%s\\\" : \\\"%s\\\", ",
-					opt_tags[idx].json_id,
-					drv->opt_vals[idx]);
-		}
-
-		pci_ids = drv->pci_tbl;
-		fprintf(ofd, "\\\"pci_ids\\\" : [");
-
-		while (pci_ids && pci_ids->device_id) {
-			fprintf(ofd, "[%d, %d, %d, %d]",
-				pci_ids->vendor_id, pci_ids->device_id,
-				pci_ids->subsystem_vendor_id,
-				pci_ids->subsystem_device_id);
-			pci_ids++;
-			if (pci_ids->device_id)
-				fprintf(ofd, ",");
-			else
-				fprintf(ofd, " ");
-		}
-		fprintf(ofd, "]}\";\n");
-		drv = drv->next;
-	}
-
-	fclose(ofd);
-}
-
-int main(int argc, char **argv)
-{
-	struct elf_info info = {0};
-	int rc = 1;
-
-	if (argc < 3) {
-		fprintf(stderr,
-			"usage: %s <object file> <c output file>\n",
-			basename(argv[0]));
-		exit(127);
-	}
-	use_stdin = !strcmp(argv[1], "-");
-	use_stdout = !strcmp(argv[2], "-");
-	parse_elf(&info, argv[1]);
-
-	if (locate_pmd_entries(&info) < 0)
-		exit(1);
-
-	if (info.drivers) {
-		output_pmd_info_string(&info, argv[2]);
-		rc = 0;
-	} else {
-		fprintf(stderr, "No drivers registered\n");
-	}
-
-	parse_elf_finish(&info);
-	exit(rc);
-}
diff --git a/buildtools/pmdinfogen/pmdinfogen.h b/buildtools/pmdinfogen/pmdinfogen.h
deleted file mode 100644
index 93930e454..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.h
+++ /dev/null
@@ -1,119 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <stdarg.h>
-#include <string.h>
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#ifdef __linux__
-#include <endian.h>
-#else
-#include <sys/endian.h>
-#endif
-#include <fcntl.h>
-#include <unistd.h>
-#include <elf.h>
-#include <rte_pci.h>
-
-/* On BSD-alike OSes elf.h defines these according to host's word size */
-#undef ELF_ST_BIND
-#undef ELF_ST_TYPE
-#undef ELF_R_SYM
-#undef ELF_R_TYPE
-
-/*
- * Define ELF64_* to ELF_*, the latter being defined in both 32 and 64 bit
- * flavors in elf.h.  This makes our code a bit more generic between arches
- * and allows us to support 32 bit code in the future should we ever want to
- */
-#ifdef RTE_ARCH_64
-#define Elf_Ehdr    Elf64_Ehdr
-#define Elf_Shdr    Elf64_Shdr
-#define Elf_Sym     Elf64_Sym
-#define Elf_Addr    Elf64_Addr
-#define Elf_Sword   Elf64_Sxword
-#define Elf_Section Elf64_Half
-#define ELF_ST_BIND ELF64_ST_BIND
-#define ELF_ST_TYPE ELF64_ST_TYPE
-
-#define Elf_Rel     Elf64_Rel
-#define Elf_Rela    Elf64_Rela
-#define ELF_R_SYM   ELF64_R_SYM
-#define ELF_R_TYPE  ELF64_R_TYPE
-#else
-#define Elf_Ehdr    Elf32_Ehdr
-#define Elf_Shdr    Elf32_Shdr
-#define Elf_Sym     Elf32_Sym
-#define Elf_Addr    Elf32_Addr
-#define Elf_Sword   Elf32_Sxword
-#define Elf_Section Elf32_Half
-#define ELF_ST_BIND ELF32_ST_BIND
-#define ELF_ST_TYPE ELF32_ST_TYPE
-
-#define Elf_Rel     Elf32_Rel
-#define Elf_Rela    Elf32_Rela
-#define ELF_R_SYM   ELF32_R_SYM
-#define ELF_R_TYPE  ELF32_R_TYPE
-#endif
-
-
-/*
- * Note, it seems odd that we have both a CONVERT_NATIVE and a TO_NATIVE macro
- * below.  We do this because the values passed to TO_NATIVE may themselves be
- * macros and need both macros here to get expanded.  Specifically its the width
- * variable we are concerned with, because it needs to get expanded prior to
- * string concatenation
- */
-#define CONVERT_NATIVE(fend, width, x) ({ \
-typeof(x) ___x; \
-if ((fend) == ELFDATA2LSB) \
-	___x = le##width##toh(x); \
-else \
-	___x = be##width##toh(x); \
-	___x; \
-})
-
-#define TO_NATIVE(fend, width, x) CONVERT_NATIVE(fend, width, x)
-
-enum opt_params {
-	PMD_PARAM_STRING = 0,
-	PMD_KMOD_DEP,
-	PMD_OPT_MAX
-};
-
-struct pmd_driver {
-	Elf_Sym *name_sym;
-	const char *name;
-	struct rte_pci_id *pci_tbl;
-	struct pmd_driver *next;
-
-	const char *opt_vals[PMD_OPT_MAX];
-};
-
-struct elf_info {
-	unsigned long size;
-	Elf_Ehdr     *hdr;
-	Elf_Shdr     *sechdrs;
-	Elf_Sym      *symtab_start;
-	Elf_Sym      *symtab_stop;
-	char         *strtab;
-
-	/* support for 32bit section numbers */
-
-	unsigned int num_sections; /* max_secindex + 1 */
-	unsigned int secindex_strings;
-	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
-	 * take shndx from symtab_shndx_start[N] instead
-	 */
-	Elf32_Word   *symtab_shndx_start;
-	Elf32_Word   *symtab_shndx_stop;
-
-	struct pmd_driver *drivers;
-};
-- 
2.25.4

Re: [dpdk-dev] [PATCH v4 0/4] pmdinfogen: rewrite in Python

[Neil Horman]

On Thu, Jul 09, 2020 at 12:23:31AM +0300, Dmitry Kozlyuk wrote:
> This patchset implements existing pmdinfogen logic in Python, replaces
> and removes the old code. The goals of rewriting are:
> 
> * easier maintenance by using a more high-level language,
> * simpler build process without host application and libelf.
> 
> Travis CI script is adjusted to install python3-pyelftools, but other CI
> systems may need similar tweaking. Particularly, testing on FreeBSD and
> big-endian targets is desired.
> 
> ---
> Changes in v4:
> 
> Fix build by removing redundant check.
> 
> Changes in v3:
> 
> 1. Rebase on ToT, add package to CI.
> 2. Add copyrights, fixup maintainers file.
> 3. Remove C implementation.
> 
> Changes in v2:
> 
> 1. pyelftools is used instead of hand-written ELF parser.
> 2. Makefiles are adjusted.
> 3. f-strings replaced with % formatting to support Python 3.5.
> 4. Wrapper script does not hide stderr from ar and pmdinfogen.
> 
> ---
> Dmitry Kozlyuk (4):
>   pmdinfogen: add Python implementation
>   build: use Python pmdinfogen
>   doc/linux_gsg: require pyelftools for pmdinfogen
>   pmdinfogen: remove C implementation
> 

I'm not currently setup to test with cross compiling, but for direct targets
(tested on power and x86), it seems to work well

Acked-by: Neil Horman <nhorman@tuxdriver.com>

Re: [dpdk-dev] [PATCH v4 3/4] doc/linux_gsg: require pyelftools for pmdinfogen

[Bruce Richardson]

On Thu, Jul 09, 2020 at 12:23:34AM +0300, Dmitry Kozlyuk wrote:
> The package is available on all major distributions.
> FreeBSD has no system requirements section in its GSG,
> Windows currently neither uses pmdinfogen not is supported by it.
> 
> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> ---
>  doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
> index a124656bc..ef2037ca7 100644
> --- a/doc/guides/linux_gsg/sys_reqs.rst
> +++ b/doc/guides/linux_gsg/sys_reqs.rst
> @@ -56,6 +56,12 @@ Compilation of the DPDK
>      * If the packaged version is below the minimum version, the latest versions
>        can be installed from Python's "pip" repository: ``pip3 install meson ninja``
>  
> +*   ``pyelftools`` (version 0.22+)
> +
> +    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
> +
> +    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
> +

If you want for completeness you can add:
 * For FreeBSD it can be installed using ``pkg install py37-pyelftools``, 
   where py37 refers to the python version in use.

>  *   Library for handling NUMA (Non Uniform Memory Access).
>  
>      * ``numactl-devel`` in RHEL/Fedora;
> -- 
> 2.25.4
> 

Re: [dpdk-dev] [PATCH v4 0/4] pmdinfogen: rewrite in Python

[Bruce Richardson]

On Thu, Jul 09, 2020 at 12:23:31AM +0300, Dmitry Kozlyuk wrote:
> This patchset implements existing pmdinfogen logic in Python, replaces
> and removes the old code. The goals of rewriting are:
> 
> * easier maintenance by using a more high-level language,
> * simpler build process without host application and libelf.
> 
> Travis CI script is adjusted to install python3-pyelftools, but other CI
> systems may need similar tweaking. Particularly, testing on FreeBSD and
> big-endian targets is desired.
> 
> ---
> Changes in v4:
> 
> Fix build by removing redundant check.
> 
> Changes in v3:
> 
> 1. Rebase on ToT, add package to CI.
> 2. Add copyrights, fixup maintainers file.
> 3. Remove C implementation.
> 
> Changes in v2:
> 
> 1. pyelftools is used instead of hand-written ELF parser.
> 2. Makefiles are adjusted.
> 3. f-strings replaced with % formatting to support Python 3.5.
> 4. Wrapper script does not hide stderr from ar and pmdinfogen.
> 
> ---
> Dmitry Kozlyuk (4):
>   pmdinfogen: add Python implementation
>   build: use Python pmdinfogen
>   doc/linux_gsg: require pyelftools for pmdinfogen
>   pmdinfogen: remove C implementation
> 
I've done builds with this patchset on linux and FreeBSD and checked the
results with dpdk-pmdinfo script. All looks ok to me.

Series-tested-by: Bruce Richardson <bruce.richardson@intel.com>

Re: [dpdk-dev] [PATCH v4 2/4] build: use Python pmdinfogen

[Bruce Richardson]

On Thu, Jul 09, 2020 at 12:23:33AM +0300, Dmitry Kozlyuk wrote:
> Like for other build scripts, use Python interpreter to run pmdinfogen.
> Adjust wrapper script accordingly and also don't suppress stderr from ar
> and pmdinfogen.
> 
> Add python3-pyelftools to CI configuration.
> 
> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> ---
>  .travis.yml                     | 2 +-
>  GNUmakefile                     | 2 +-
>  buildtools/Makefile             | 9 ---------
>  buildtools/gen-pmdinfo-cfile.sh | 6 +++---
>  buildtools/meson.build          | 1 +
>  drivers/meson.build             | 2 +-
>  meson.build                     | 1 -
>  mk/internal/rte.compile-pre.mk  | 2 +-
>  mk/rte.sdkinstall.mk            | 2 --
>  9 files changed, 8 insertions(+), 19 deletions(-)
>  delete mode 100644 buildtools/Makefile
> 
> diff --git a/.travis.yml b/.travis.yml
> index 14f812423..28b559a25 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -14,7 +14,7 @@ addons:
>    apt:
>      update: true
>      packages: &required_packages
> -      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
> +      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
>        - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
>  
>  _aarch64_packages: &aarch64_packages
> diff --git a/GNUmakefile b/GNUmakefile
> index e8de422df..242d30d2e 100644
> --- a/GNUmakefile
> +++ b/GNUmakefile
> @@ -12,6 +12,6 @@ export RTE_SDK
>  # directory list
>  #
>  
> -ROOTDIRS-y := buildtools lib kernel drivers app
> +ROOTDIRS-y := lib kernel drivers app
>  
>  include $(RTE_SDK)/mk/rte.sdkroot.mk
> diff --git a/buildtools/Makefile b/buildtools/Makefile
> deleted file mode 100644
> index 7f76fd7d6..000000000
> --- a/buildtools/Makefile
> +++ /dev/null
> @@ -1,9 +0,0 @@
> -# SPDX-License-Identifier: BSD-3-Clause
> -# Copyright(c) 2016 Neil Horman <nhorman@tuxdriver.com>
> -# All rights reserved.
> -
> -include $(RTE_SDK)/mk/rte.vars.mk
> -
> -DIRS-y += pmdinfogen
> -
> -include $(RTE_SDK)/mk/rte.subdir.mk
> diff --git a/buildtools/gen-pmdinfo-cfile.sh b/buildtools/gen-pmdinfo-cfile.sh
> index 43059cf36..109ee461e 100755
> --- a/buildtools/gen-pmdinfo-cfile.sh
> +++ b/buildtools/gen-pmdinfo-cfile.sh
> @@ -4,11 +4,11 @@
>  
>  arfile=$1
>  output=$2
> -pmdinfogen=$3
> +shift 2
> +pmdinfogen=$*
>  
>  # The generated file must not be empty if compiled in pedantic mode
>  echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
>  for ofile in `ar t $arfile` ; do
> -	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
> +	ar p $arfile $ofile | $pmdinfogen - - >> $output
>  done
> -exit 0

For 20.11, since make build system is being removed, I think we should look
to merge this into the python script rather than having an extra layer of
wrapper and having meson pass the command from one script to another.
However, since out-of-the-box python doesn't seem to have support for
reading .a files, this is fine for now.


> diff --git a/buildtools/meson.build b/buildtools/meson.build
> index 04808dabc..3a64b28b7 100644
> --- a/buildtools/meson.build
> +++ b/buildtools/meson.build
> @@ -17,3 +17,4 @@ else
>  endif
>  map_to_win_cmd = py3 + files('map_to_win.py')
>  sphinx_wrapper = py3 + files('call-sphinx-build.py')
> +pmdinfogen = py3 + files('pmdinfogen.py')
> diff --git a/drivers/meson.build b/drivers/meson.build
> index 161cfda04..3c4d4b700 100644
> --- a/drivers/meson.build
> +++ b/drivers/meson.build
> @@ -119,7 +119,7 @@ foreach class:dpdk_driver_classes
>  						command: [pmdinfo, tmp_lib.full_path(),
>  							'@OUTPUT@', pmdinfogen],
>  						output: out_filename,
> -						depends: [pmdinfogen, tmp_lib])
> +						depends: [tmp_lib])
>  			endif
>  
>  			# now build the static driver
> diff --git a/meson.build b/meson.build
> index 61d9a4f5f..a6e8c09c2 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -45,7 +45,6 @@ subdir('buildtools')
>  subdir('config')
>  
>  # build libs and drivers
> -subdir('buildtools/pmdinfogen')
>  subdir('lib')
>  subdir('drivers')
>  
> diff --git a/mk/internal/rte.compile-pre.mk b/mk/internal/rte.compile-pre.mk
> index df05b5576..bb2ab0725 100644
> --- a/mk/internal/rte.compile-pre.mk
> +++ b/mk/internal/rte.compile-pre.mk
> @@ -59,7 +59,7 @@ endif
>  CHECK_SYMBOLS_SCRIPT = $(RTE_SDK)/buildtools/check-symbols.sh
>  CHECK_SYMBOLS = $(CHECK_SYMBOLS_SCRIPT) $(SRCDIR)/$(EXPORT_MAP) $@
>  
> -PMDINFO_GEN = $(RTE_SDK_BIN)/app/dpdk-pmdinfogen $@ $@.pmd.c
> +PMDINFO_GEN = $(RTE_SDK)/buildtools/pmdinfogen.py $@ $@.pmd.c
>  PMDINFO_CC = $(CC) $(CPPFLAGS) $(CFLAGS) $(EXTRA_CFLAGS) -c -o $@.pmd.o $@.pmd.c
>  PMDINFO_LD = $(CROSS)ld -r $(filter-out -export-dynamic,$(LDFLAGS)) -o $@.o $@.pmd.o $@
>  PMDINFO_TO_O = if grep -q 'RTE_PMD_REGISTER_.*(.*)' $<; then \
> diff --git a/mk/rte.sdkinstall.mk b/mk/rte.sdkinstall.mk
> index 32bed5d95..875a64f04 100644
> --- a/mk/rte.sdkinstall.mk
> +++ b/mk/rte.sdkinstall.mk
> @@ -98,7 +98,6 @@ install-runtime:
>  	$(Q)cp $(CP_FLAGS)    $O/lib/* $(DESTDIR)$(libdir)
>  	$(Q)$(call rte_mkdir, $(DESTDIR)$(bindir))
>  	$(Q)tar -cf -      -C $O --exclude 'app/*.map' \
> -		--exclude app/dpdk-pmdinfogen \
>  		--exclude 'app/cmdline*' --exclude app/test \
>  		--exclude app/testacl --exclude app/testpipeline app | \
>  	    tar -xf -      -C $(DESTDIR)$(bindir) $(TAR_X_FLAGS)
> @@ -134,7 +133,6 @@ install-sdk:
>  	$(Q)cp $(CP_FLAGS)      $(RTE_SDK)/buildtools    $(DESTDIR)$(sdkdir)
>  	$(Q)$(call rte_mkdir,                            $(DESTDIR)$(targetdir)/app)
>  	$(Q)cp $(CP_FLAGS)      $O/.config               $(DESTDIR)$(targetdir)
> -	$(Q)cp $(CP_FLAGS)      $O/app/dpdk-pmdinfogen   $(DESTDIR)$(targetdir)/app
>  	$(Q)$(call rte_symlink, $(DESTDIR)$(includedir), $(DESTDIR)$(targetdir)/include)
>  	$(Q)$(call rte_symlink, $(DESTDIR)$(libdir),     $(DESTDIR)$(targetdir)/lib)
>  
> -- 
> 2.25.4
> 

For the meson build changes here:

Reviewed-by: Bruce Richardson <bruce.richardson@intel.com>

Re: [dpdk-dev] [PATCH v4 3/4] doc/linux_gsg: require pyelftools for pmdinfogen

[Bruce Richardson]

On Thu, Jul 09, 2020 at 12:23:34AM +0300, Dmitry Kozlyuk wrote:
> The package is available on all major distributions.
> FreeBSD has no system requirements section in its GSG,
> Windows currently neither uses pmdinfogen not is supported by it.
> 
> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> ---
>  doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
> index a124656bc..ef2037ca7 100644
> --- a/doc/guides/linux_gsg/sys_reqs.rst
> +++ b/doc/guides/linux_gsg/sys_reqs.rst
> @@ -56,6 +56,12 @@ Compilation of the DPDK
>      * If the packaged version is below the minimum version, the latest versions
>        can be installed from Python's "pip" repository: ``pip3 install meson ninja``
>  
> +*   ``pyelftools`` (version 0.22+)
> +
> +    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
> +
> +    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
> +
>  *   Library for handling NUMA (Non Uniform Memory Access).
>  
>      * ``numactl-devel`` in RHEL/Fedora;
> -- 
This can be merged into patch 2, I think.

Re: [dpdk-dev] [PATCH v4 3/4] doc/linux_gsg: require pyelftools for pmdinfogen

[Bruce Richardson]

On Tue, Jul 21, 2020 at 02:39:35PM +0100, Bruce Richardson wrote:
> On Thu, Jul 09, 2020 at 12:23:34AM +0300, Dmitry Kozlyuk wrote:
> > The package is available on all major distributions.
> > FreeBSD has no system requirements section in its GSG,
> > Windows currently neither uses pmdinfogen not is supported by it.
> > 
> > Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> > ---
> >  doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
> >  1 file changed, 6 insertions(+)
> > 
> > diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
> > index a124656bc..ef2037ca7 100644
> > --- a/doc/guides/linux_gsg/sys_reqs.rst
> > +++ b/doc/guides/linux_gsg/sys_reqs.rst
> > @@ -56,6 +56,12 @@ Compilation of the DPDK
> >      * If the packaged version is below the minimum version, the latest versions
> >        can be installed from Python's "pip" repository: ``pip3 install meson ninja``
> >  
> > +*   ``pyelftools`` (version 0.22+)
> > +
> > +    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
> > +
> > +    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
> > +
> 
> If you want for completeness you can add:
>  * For FreeBSD it can be installed using ``pkg install py37-pyelftools``, 
>    where py37 refers to the python version in use.
> 

Sorry, missed the fact that this was the Linux GSG. Never mind...

Re: [dpdk-dev] [PATCH v4 2/4] build: use Python pmdinfogen

[Dmitry Kozlyuk]

On Tue, 21 Jul 2020 15:04:11 +0100, Bruce Richardson wrote:
> On Thu, Jul 09, 2020 at 12:23:33AM +0300, Dmitry Kozlyuk wrote:
[snip]
> > diff --git a/buildtools/gen-pmdinfo-cfile.sh b/buildtools/gen-pmdinfo-cfile.sh
> > index 43059cf36..109ee461e 100755
> > --- a/buildtools/gen-pmdinfo-cfile.sh
> > +++ b/buildtools/gen-pmdinfo-cfile.sh
> > @@ -4,11 +4,11 @@
> >  
> >  arfile=$1
> >  output=$2
> > -pmdinfogen=$3
> > +shift 2
> > +pmdinfogen=$*
> >  
> >  # The generated file must not be empty if compiled in pedantic mode
> >  echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
> >  for ofile in `ar t $arfile` ; do
> > -	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
> > +	ar p $arfile $ofile | $pmdinfogen - - >> $output
> >  done
> > -exit 0  
> 
> For 20.11, since make build system is being removed, I think we should look
> to merge this into the python script rather than having an extra layer of
> wrapper and having meson pass the command from one script to another.
> However, since out-of-the-box python doesn't seem to have support for
> reading .a files, this is fine for now.

Or Meson can only tell the Python script whether to process ELF of COFF and
the path to an archiver tool (ar or lib.exe). This way we'd get rid of shell
(which is non-portable to Windows) and need no libraries to read .a or .lib.

[dpdk-dev] [PATCH v5 0/3] pmdinfogen: rewrite in Python

[Dmitry Kozlyuk]

This patchset implements existing pmdinfogen logic in Python, replaces
and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf.

Travis CI script is adjusted to install python3-pyelftools, but other CI
systems may need similar tweaking. Particularly, testing on FreeBSD and
big-endian targets is desired.

From v4:

Acked-by: Neil Horman <nhorman@tuxdriver.com>

---
Changes in v5:

1. Rebase on ToT, remove Makefile changes.
2. Squash doc changes into patch 2/3.

Changes in v4:

Fix build by removing redundant check.

Changes in v3:

1. Rebase on ToT, add package to CI.
2. Add copyrights, fixup maintainers file.
3. Remove C implementation.

Changes in v2:

1. pyelftools is used instead of hand-written ELF parser.
2. Makefiles are adjusted.
3. f-strings replaced with % formatting to support Python 3.5.
4. Wrapper script does not hide stderr from ar and pmdinfogen.

Dmitry Kozlyuk (3):
  pmdinfogen: add Python implementation
  build: use Python pmdinfogen
  pmdinfogen: remove C implementation

 .travis.yml                        |   2 +-
 MAINTAINERS                        |   2 +-
 buildtools/gen-pmdinfo-cfile.sh    |   6 +-
 buildtools/meson.build             |   1 +
 buildtools/pmdinfogen.py           | 188 ++++++++++++
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 doc/guides/linux_gsg/sys_reqs.rst  |   6 +
 drivers/meson.build                |   2 +-
 meson.build                        |   1 -
 11 files changed, 201 insertions(+), 596 deletions(-)
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

-- 
2.28.0

[dpdk-dev] [PATCH v5 1/3] pmdinfogen: add Python implementation

[Dmitry Kozlyuk]

Using a high-level, interpreted language simplifies maintenance and
build process. Furthermore, ELF handling is delegated to pyelftools
package. Original logic is kept, the copyright recognizes that.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 buildtools/pmdinfogen.py | 188 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 188 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py

diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
new file mode 100755
index 000000000..a9c2643a4
--- /dev/null
+++ b/buildtools/pmdinfogen.py
@@ -0,0 +1,188 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <nhorman@tuxdriver.com>
+# Copyright (c) 2020 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image._image.get_section(self._symbol["st_shndx"]).data()
+        base = self._symbol["st_value"]
+        return data[base : base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        if len(value):
+            return value[:-1].decode()
+        return ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        if symbol:
+            return ELFSymbol(self, symbol[0])
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    def _load_pci_ids(self, image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            self.pci_ids.append(
+                [
+                    pci_id.vendor_id,
+                    pci_id.device_id,
+                    pci_id.subsystem_vendor_id,
+                    pci_id.subsystem_device_id,
+                ]
+            )
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
-- 
2.28.0

[dpdk-dev] [PATCH v5 2/3] build: use Python pmdinfogen

[Dmitry Kozlyuk]

Like for other build scripts, use Python interpreter to run pmdinfogen.
Adjust wrapper script accordingly and also don't suppress stderr from ar
and pmdinfogen.

Add python3-pyelftools to CI configuration. The package is available on
all major distributions. FreeBSD has no system requirements section in
its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE)
supported.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 .travis.yml                       | 2 +-
 buildtools/gen-pmdinfo-cfile.sh   | 6 +++---
 buildtools/meson.build            | 1 +
 doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
 drivers/meson.build               | 2 +-
 meson.build                       | 1 -
 6 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index d6eeab371..226db7590 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -11,7 +11,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
diff --git a/buildtools/gen-pmdinfo-cfile.sh b/buildtools/gen-pmdinfo-cfile.sh
index 43059cf36..109ee461e 100755
--- a/buildtools/gen-pmdinfo-cfile.sh
+++ b/buildtools/gen-pmdinfo-cfile.sh
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 04808dabc..3a64b28b7 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -17,3 +17,4 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
index 6b9f573fc..200212081 100644
--- a/doc/guides/linux_gsg/sys_reqs.rst
+++ b/doc/guides/linux_gsg/sys_reqs.rst
@@ -56,6 +56,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
diff --git a/drivers/meson.build b/drivers/meson.build
index 5f9526557..88364259d 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -137,7 +137,7 @@ foreach subpath:subdirs
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
diff --git a/meson.build b/meson.build
index 61d9a4f5f..a6e8c09c2 100644
--- a/meson.build
+++ b/meson.build
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
-- 
2.28.0

[dpdk-dev] [PATCH v5 3/3] pmdinfogen: remove C implementation

[Dmitry Kozlyuk]

Delete the files no longer used in build process.  Keep Neil Horman
maintainer of the entire "Driver information" section, only adjust
paths.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 MAINTAINERS                        |   2 +-
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 4 files changed, 1 insertion(+), 590 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 652712ccb..38e994388 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -135,7 +135,7 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <nhorman@tuxdriver.com>
-F: buildtools/pmdinfogen/
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
diff --git a/buildtools/pmdinfogen/meson.build b/buildtools/pmdinfogen/meson.build
deleted file mode 100644
index 670528fac..000000000
--- a/buildtools/pmdinfogen/meson.build
+++ /dev/null
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
diff --git a/buildtools/pmdinfogen/pmdinfogen.c b/buildtools/pmdinfogen/pmdinfogen.c
deleted file mode 100644
index a68d1ea99..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.c
+++ /dev/null
@@ -1,456 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <ctype.h>
-#include <string.h>
-#include <limits.h>
-#include <stdbool.h>
-#include <errno.h>
-#include <libgen.h>
-
-#include <rte_common.h>
-#include "pmdinfogen.h"
-
-#ifdef RTE_ARCH_64
-#define ADDR_SIZE 64
-#else
-#define ADDR_SIZE 32
-#endif
-
-static int use_stdin, use_stdout;
-
-static const char *sym_name(struct elf_info *elf, Elf_Sym *sym)
-{
-	if (sym)
-		return elf->strtab + sym->st_name;
-	else
-		return "(unknown)";
-}
-
-static void *grab_file(const char *filename, unsigned long *size)
-{
-	struct stat st;
-	void *map = MAP_FAILED;
-	int fd = -1;
-
-	if (!use_stdin) {
-		fd = open(filename, O_RDONLY);
-		if (fd < 0)
-			return NULL;
-	} else {
-		/* from stdin, use a temporary file to mmap */
-		FILE *infile;
-		char buffer[1024];
-		int n;
-
-		infile = tmpfile();
-		if (infile == NULL) {
-			perror("tmpfile");
-			return NULL;
-		}
-		fd = dup(fileno(infile));
-		fclose(infile);
-		if (fd < 0)
-			return NULL;
-
-		n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		while (n > 0) {
-			if (write(fd, buffer, n) != n)
-				goto failed;
-			n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		}
-	}
-
-	if (fstat(fd, &st))
-		goto failed;
-
-	*size = st.st_size;
-	map = mmap(NULL, *size, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);
-
-failed:
-	close(fd);
-	if (map == MAP_FAILED)
-		return NULL;
-	return map;
-}
-
-/**
-  * Return a copy of the next line in a mmap'ed file.
-  * spaces in the beginning of the line is trimmed away.
-  * Return a pointer to a static buffer.
-  **/
-static void release_file(void *file, unsigned long size)
-{
-	munmap(file, size);
-}
-
-
-static void *get_sym_value(struct elf_info *info, const Elf_Sym *sym)
-{
-	return RTE_PTR_ADD(info->hdr,
-		info->sechdrs[sym->st_shndx].sh_offset + sym->st_value);
-}
-
-static Elf_Sym *find_sym_in_symtab(struct elf_info *info,
-				   const char *name, Elf_Sym *last)
-{
-	Elf_Sym *idx;
-	if (last)
-		idx = last+1;
-	else
-		idx = info->symtab_start;
-
-	for (; idx < info->symtab_stop; idx++) {
-		const char *n = sym_name(info, idx);
-		if (!strncmp(n, name, strlen(name)))
-			return idx;
-	}
-	return NULL;
-}
-
-static int parse_elf(struct elf_info *info, const char *filename)
-{
-	unsigned int i;
-	Elf_Ehdr *hdr;
-	Elf_Shdr *sechdrs;
-	Elf_Sym  *sym;
-	int endian;
-	unsigned int symtab_idx = ~0U, symtab_shndx_idx = ~0U;
-
-	hdr = grab_file(filename, &info->size);
-	if (!hdr) {
-		perror(filename);
-		exit(1);
-	}
-	info->hdr = hdr;
-	if (info->size < sizeof(*hdr)) {
-		/* file too small, assume this is an empty .o file */
-		return 0;
-	}
-	/* Is this a valid ELF file? */
-	if ((hdr->e_ident[EI_MAG0] != ELFMAG0) ||
-	    (hdr->e_ident[EI_MAG1] != ELFMAG1) ||
-	    (hdr->e_ident[EI_MAG2] != ELFMAG2) ||
-	    (hdr->e_ident[EI_MAG3] != ELFMAG3)) {
-		/* Not an ELF file - silently ignore it */
-		return 0;
-	}
-
-	if (!hdr->e_ident[EI_DATA]) {
-		/* Unknown endian */
-		return 0;
-	}
-
-	endian = hdr->e_ident[EI_DATA];
-
-	/* Fix endianness in ELF header */
-	hdr->e_type      = TO_NATIVE(endian, 16, hdr->e_type);
-	hdr->e_machine   = TO_NATIVE(endian, 16, hdr->e_machine);
-	hdr->e_version   = TO_NATIVE(endian, 32, hdr->e_version);
-	hdr->e_entry     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_entry);
-	hdr->e_phoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_phoff);
-	hdr->e_shoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_shoff);
-	hdr->e_flags     = TO_NATIVE(endian, 32, hdr->e_flags);
-	hdr->e_ehsize    = TO_NATIVE(endian, 16, hdr->e_ehsize);
-	hdr->e_phentsize = TO_NATIVE(endian, 16, hdr->e_phentsize);
-	hdr->e_phnum     = TO_NATIVE(endian, 16, hdr->e_phnum);
-	hdr->e_shentsize = TO_NATIVE(endian, 16, hdr->e_shentsize);
-	hdr->e_shnum     = TO_NATIVE(endian, 16, hdr->e_shnum);
-	hdr->e_shstrndx  = TO_NATIVE(endian, 16, hdr->e_shstrndx);
-
-	sechdrs = RTE_PTR_ADD(hdr, hdr->e_shoff);
-	info->sechdrs = sechdrs;
-
-	/* Check if file offset is correct */
-	if (hdr->e_shoff > info->size) {
-		fprintf(stderr, "section header offset=%lu in file '%s' "
-		      "is bigger than filesize=%lu\n",
-		      (unsigned long)hdr->e_shoff,
-		      filename, info->size);
-		return 0;
-	}
-
-	if (hdr->e_shnum == SHN_UNDEF) {
-		/*
-		 * There are more than 64k sections,
-		 * read count from .sh_size.
-		 */
-		info->num_sections =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[0].sh_size);
-	} else {
-		info->num_sections = hdr->e_shnum;
-	}
-	if (hdr->e_shstrndx == SHN_XINDEX)
-		info->secindex_strings =
-			TO_NATIVE(endian, 32, sechdrs[0].sh_link);
-	else
-		info->secindex_strings = hdr->e_shstrndx;
-
-	/* Fix endianness in section headers */
-	for (i = 0; i < info->num_sections; i++) {
-		sechdrs[i].sh_name      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_name);
-		sechdrs[i].sh_type      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_type);
-		sechdrs[i].sh_flags     =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_flags);
-		sechdrs[i].sh_addr      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addr);
-		sechdrs[i].sh_offset    =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_offset);
-		sechdrs[i].sh_size      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_size);
-		sechdrs[i].sh_link      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_link);
-		sechdrs[i].sh_info      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_info);
-		sechdrs[i].sh_addralign =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addralign);
-		sechdrs[i].sh_entsize   =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_entsize);
-	}
-	/* Find symbol table. */
-	for (i = 1; i < info->num_sections; i++) {
-		int nobits = sechdrs[i].sh_type == SHT_NOBITS;
-
-		if (!nobits && sechdrs[i].sh_offset > info->size) {
-			fprintf(stderr, "%s is truncated. "
-			      "sechdrs[i].sh_offset=%lu > sizeof(*hrd)=%zu\n",
-			      filename, (unsigned long)sechdrs[i].sh_offset,
-			      sizeof(*hdr));
-			return 0;
-		}
-
-		if (sechdrs[i].sh_type == SHT_SYMTAB) {
-			unsigned int sh_link_idx;
-			symtab_idx = i;
-			info->symtab_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-			sh_link_idx = sechdrs[i].sh_link;
-			info->strtab       = RTE_PTR_ADD(hdr,
-				sechdrs[sh_link_idx].sh_offset);
-		}
-
-		/* 32bit section no. table? ("more than 64k sections") */
-		if (sechdrs[i].sh_type == SHT_SYMTAB_SHNDX) {
-			symtab_shndx_idx = i;
-			info->symtab_shndx_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_shndx_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-		}
-	}
-	if (!info->symtab_start)
-		fprintf(stderr, "%s has no symtab?\n", filename);
-	else {
-		/* Fix endianness in symbols */
-		for (sym = info->symtab_start; sym < info->symtab_stop; sym++) {
-			sym->st_shndx = TO_NATIVE(endian, 16, sym->st_shndx);
-			sym->st_name  = TO_NATIVE(endian, 32, sym->st_name);
-			sym->st_value = TO_NATIVE(endian, ADDR_SIZE, sym->st_value);
-			sym->st_size  = TO_NATIVE(endian, ADDR_SIZE, sym->st_size);
-		}
-	}
-
-	if (symtab_shndx_idx != ~0U) {
-		Elf32_Word *p;
-		if (symtab_idx != sechdrs[symtab_shndx_idx].sh_link)
-			fprintf(stderr,
-			      "%s: SYMTAB_SHNDX has bad sh_link: %u!=%u\n",
-			      filename, sechdrs[symtab_shndx_idx].sh_link,
-			      symtab_idx);
-		/* Fix endianness */
-		for (p = info->symtab_shndx_start; p < info->symtab_shndx_stop;
-		     p++)
-			*p = TO_NATIVE(endian, 32, *p);
-	}
-
-	return 1;
-}
-
-static void parse_elf_finish(struct elf_info *info)
-{
-	struct pmd_driver *tmp, *idx = info->drivers;
-	release_file(info->hdr, info->size);
-	while (idx) {
-		tmp = idx->next;
-		free(idx);
-		idx = tmp;
-	}
-}
-
-struct opt_tag {
-	const char *suffix;
-	const char *json_id;
-};
-
-static const struct opt_tag opt_tags[] = {
-	{"_param_string_export", "params"},
-	{"_kmod_dep_export", "kmod"},
-};
-
-static int complete_pmd_entry(struct elf_info *info, struct pmd_driver *drv)
-{
-	const char *tname;
-	int i;
-	char tmpsymname[128];
-	Elf_Sym *tmpsym;
-
-	drv->name = get_sym_value(info, drv->name_sym);
-
-	for (i = 0; i < PMD_OPT_MAX; i++) {
-		memset(tmpsymname, 0, 128);
-		sprintf(tmpsymname, "__%s%s", drv->name, opt_tags[i].suffix);
-		tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-		if (!tmpsym)
-			continue;
-		drv->opt_vals[i] = get_sym_value(info, tmpsym);
-	}
-
-	memset(tmpsymname, 0, 128);
-	sprintf(tmpsymname, "__%s_pci_tbl_export", drv->name);
-
-	tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-
-
-	/*
-	 * If this returns NULL, then this is a PMD_VDEV, because
-	 * it has no pci table reference
-	 */
-	if (!tmpsym) {
-		drv->pci_tbl = NULL;
-		return 0;
-	}
-
-	tname = get_sym_value(info, tmpsym);
-	tmpsym = find_sym_in_symtab(info, tname, NULL);
-	if (!tmpsym)
-		return -ENOENT;
-
-	drv->pci_tbl = (struct rte_pci_id *)get_sym_value(info, tmpsym);
-	if (!drv->pci_tbl)
-		return -ENOENT;
-
-	return 0;
-}
-
-static int locate_pmd_entries(struct elf_info *info)
-{
-	Elf_Sym *last = NULL;
-	struct pmd_driver *new;
-
-	info->drivers = NULL;
-
-	do {
-		new = calloc(sizeof(struct pmd_driver), 1);
-		if (new == NULL) {
-			fprintf(stderr, "Failed to calloc memory\n");
-			return -1;
-		}
-		new->name_sym = find_sym_in_symtab(info, "this_pmd_name", last);
-		last = new->name_sym;
-		if (!new->name_sym)
-			free(new);
-		else {
-			if (complete_pmd_entry(info, new)) {
-				fprintf(stderr,
-					"Failed to complete pmd entry\n");
-				free(new);
-			} else {
-				new->next = info->drivers;
-				info->drivers = new;
-			}
-		}
-	} while (last);
-
-	return 0;
-}
-
-static void output_pmd_info_string(struct elf_info *info, char *outfile)
-{
-	FILE *ofd;
-	struct pmd_driver *drv;
-	struct rte_pci_id *pci_ids;
-	int idx = 0;
-
-	if (use_stdout)
-		ofd = stdout;
-	else {
-		ofd = fopen(outfile, "w+");
-		if (!ofd) {
-			fprintf(stderr, "Unable to open output file\n");
-			return;
-		}
-	}
-
-	drv = info->drivers;
-
-	while (drv) {
-		fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
-			"\"PMD_INFO_STRING= {",
-			drv->name);
-		fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);
-
-		for (idx = 0; idx < PMD_OPT_MAX; idx++) {
-			if (drv->opt_vals[idx])
-				fprintf(ofd, "\\\"%s\\\" : \\\"%s\\\", ",
-					opt_tags[idx].json_id,
-					drv->opt_vals[idx]);
-		}
-
-		pci_ids = drv->pci_tbl;
-		fprintf(ofd, "\\\"pci_ids\\\" : [");
-
-		while (pci_ids && pci_ids->device_id) {
-			fprintf(ofd, "[%d, %d, %d, %d]",
-				pci_ids->vendor_id, pci_ids->device_id,
-				pci_ids->subsystem_vendor_id,
-				pci_ids->subsystem_device_id);
-			pci_ids++;
-			if (pci_ids->device_id)
-				fprintf(ofd, ",");
-			else
-				fprintf(ofd, " ");
-		}
-		fprintf(ofd, "]}\";\n");
-		drv = drv->next;
-	}
-
-	fclose(ofd);
-}
-
-int main(int argc, char **argv)
-{
-	struct elf_info info = {0};
-	int rc = 1;
-
-	if (argc < 3) {
-		fprintf(stderr,
-			"usage: %s <object file> <c output file>\n",
-			basename(argv[0]));
-		exit(127);
-	}
-	use_stdin = !strcmp(argv[1], "-");
-	use_stdout = !strcmp(argv[2], "-");
-	parse_elf(&info, argv[1]);
-
-	if (locate_pmd_entries(&info) < 0)
-		exit(1);
-
-	if (info.drivers) {
-		output_pmd_info_string(&info, argv[2]);
-		rc = 0;
-	} else {
-		fprintf(stderr, "No drivers registered\n");
-	}
-
-	parse_elf_finish(&info);
-	exit(rc);
-}
diff --git a/buildtools/pmdinfogen/pmdinfogen.h b/buildtools/pmdinfogen/pmdinfogen.h
deleted file mode 100644
index 93930e454..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.h
+++ /dev/null
@@ -1,119 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <stdarg.h>
-#include <string.h>
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#ifdef __linux__
-#include <endian.h>
-#else
-#include <sys/endian.h>
-#endif
-#include <fcntl.h>
-#include <unistd.h>
-#include <elf.h>
-#include <rte_pci.h>
-
-/* On BSD-alike OSes elf.h defines these according to host's word size */
-#undef ELF_ST_BIND
-#undef ELF_ST_TYPE
-#undef ELF_R_SYM
-#undef ELF_R_TYPE
-
-/*
- * Define ELF64_* to ELF_*, the latter being defined in both 32 and 64 bit
- * flavors in elf.h.  This makes our code a bit more generic between arches
- * and allows us to support 32 bit code in the future should we ever want to
- */
-#ifdef RTE_ARCH_64
-#define Elf_Ehdr    Elf64_Ehdr
-#define Elf_Shdr    Elf64_Shdr
-#define Elf_Sym     Elf64_Sym
-#define Elf_Addr    Elf64_Addr
-#define Elf_Sword   Elf64_Sxword
-#define Elf_Section Elf64_Half
-#define ELF_ST_BIND ELF64_ST_BIND
-#define ELF_ST_TYPE ELF64_ST_TYPE
-
-#define Elf_Rel     Elf64_Rel
-#define Elf_Rela    Elf64_Rela
-#define ELF_R_SYM   ELF64_R_SYM
-#define ELF_R_TYPE  ELF64_R_TYPE
-#else
-#define Elf_Ehdr    Elf32_Ehdr
-#define Elf_Shdr    Elf32_Shdr
-#define Elf_Sym     Elf32_Sym
-#define Elf_Addr    Elf32_Addr
-#define Elf_Sword   Elf32_Sxword
-#define Elf_Section Elf32_Half
-#define ELF_ST_BIND ELF32_ST_BIND
-#define ELF_ST_TYPE ELF32_ST_TYPE
-
-#define Elf_Rel     Elf32_Rel
-#define Elf_Rela    Elf32_Rela
-#define ELF_R_SYM   ELF32_R_SYM
-#define ELF_R_TYPE  ELF32_R_TYPE
-#endif
-
-
-/*
- * Note, it seems odd that we have both a CONVERT_NATIVE and a TO_NATIVE macro
- * below.  We do this because the values passed to TO_NATIVE may themselves be
- * macros and need both macros here to get expanded.  Specifically its the width
- * variable we are concerned with, because it needs to get expanded prior to
- * string concatenation
- */
-#define CONVERT_NATIVE(fend, width, x) ({ \
-typeof(x) ___x; \
-if ((fend) == ELFDATA2LSB) \
-	___x = le##width##toh(x); \
-else \
-	___x = be##width##toh(x); \
-	___x; \
-})
-
-#define TO_NATIVE(fend, width, x) CONVERT_NATIVE(fend, width, x)
-
-enum opt_params {
-	PMD_PARAM_STRING = 0,
-	PMD_KMOD_DEP,
-	PMD_OPT_MAX
-};
-
-struct pmd_driver {
-	Elf_Sym *name_sym;
-	const char *name;
-	struct rte_pci_id *pci_tbl;
-	struct pmd_driver *next;
-
-	const char *opt_vals[PMD_OPT_MAX];
-};
-
-struct elf_info {
-	unsigned long size;
-	Elf_Ehdr     *hdr;
-	Elf_Shdr     *sechdrs;
-	Elf_Sym      *symtab_start;
-	Elf_Sym      *symtab_stop;
-	char         *strtab;
-
-	/* support for 32bit section numbers */
-
-	unsigned int num_sections; /* max_secindex + 1 */
-	unsigned int secindex_strings;
-	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
-	 * take shndx from symtab_shndx_start[N] instead
-	 */
-	Elf32_Word   *symtab_shndx_start;
-	Elf32_Word   *symtab_shndx_stop;
-
-	struct pmd_driver *drivers;
-};
-- 
2.28.0

Re: [dpdk-dev] [PATCH v5 1/3] pmdinfogen: add Python implementation

[Stephen Hemminger]

On Mon, 28 Sep 2020 00:47:30 +0300
Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:

> Using a high-level, interpreted language simplifies maintenance and
> build process. Furthermore, ELF handling is delegated to pyelftools
> package. Original logic is kept, the copyright recognizes that.
> 
> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>

Looks good, a couple little nits.

Yapf flags this as unnecessary spacing:


+    @property
+    def value(self):
+        data = self._image._image.get_section(self._symbol["st_shndx"]).data()
+        base = self._symbol["st_value"]
+        return data[base : base + self.size]
                         ^^^

Python lint complains about missing docstrings, which doesn't matter but it also shows:

$ pylint3 buildtools/pmdinfogen.py  | grep -v docstring
************* Module pmdinfogen
buildtools/pmdinfogen.py:27:15: W0212: Access to a protected member _image of a client class (protected-access)
buildtools/pmdinfogen.py:34:11: C1801: Do not use `len(SEQUENCE)` to determine if a sequence is empty (len-as-condition)
buildtools/pmdinfogen.py:50:4: R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
buildtools/pmdinfogen.py:107:12: W0212: Access to a protected member _load_pci_ids of a client class (protected-access)

------------------------------------------------------------------
Your code has been rated at 7.93/10 (previous run: 7.93/10, +0.00)

Re: [dpdk-dev] [PATCH v5 3/3] pmdinfogen: remove C implementation

[Thomas Monjalon]

> Delete the files no longer used in build process.  Keep Neil Horman
> maintainer of the entire "Driver information" section, only adjust
> paths.

Dmitry, don't you think you should be co-maintainer?

Re: [dpdk-dev] [PATCH v5 0/3] pmdinfogen: rewrite in Python

[David Marchand]

Hello Dmitry,

On Sun, Sep 27, 2020 at 11:48 PM Dmitry Kozlyuk
<dmitry.kozliuk@gmail.com> wrote:
>
> This patchset implements existing pmdinfogen logic in Python, replaces
> and removes the old code. The goals of rewriting are:
>

* support pmdinfo on Windows ? :-)

> * easier maintenance by using a more high-level language,
> * simpler build process without host application and libelf.
>
> Travis CI script is adjusted to install python3-pyelftools, but other CI
> systems may need similar tweaking. Particularly, testing on FreeBSD and
> big-endian targets is desired.

This new requirement will impact mainly distribution packagers,
developers and the CI.
While I think this is fine for the first two categories
(python-elftools is now available on most distributions, and you can
also pip install it), the last category is a concern.

If we want to move forward with this patch, this must be addressed in
a timely manner as we don't want to break the CI.


-- 
David Marchand

[dpdk-dev] [PATCH v6 0/3] pmdinfogen: rewrite in Python

[Dmitry Kozlyuk]

This patchset implements existing pmdinfogen logic in Python, replaces
and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.

Travis CI script is adjusted to install python3-pyelftools, but other CI
systems may need similar tweaking. Particularly, testing on FreeBSD and
big-endian targets is desired.

Acked-by: Neil Horman <nhorman@tuxdriver.com>

---
Changes in v6:

1. Address yapf and pylint warnings (Stephen Hemminger).
2. Add myself as new implementation maintainer (Thomas Monjalon).

Dmitry Kozlyuk (3):
  pmdinfogen: add Python implementation
  build: use Python pmdinfogen
  pmdinfogen: remove C implementation

 .travis.yml                        |   2 +-
 MAINTAINERS                        |   3 +-
 buildtools/gen-pmdinfo-cfile.sh    |   6 +-
 buildtools/meson.build             |   1 +
 buildtools/pmdinfogen.py           | 189 ++++++++++++
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 doc/guides/linux_gsg/sys_reqs.rst  |   6 +
 drivers/meson.build                |   2 +-
 meson.build                        |   1 -
 11 files changed, 203 insertions(+), 596 deletions(-)
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

-- 
2.28.0

[dpdk-dev] [PATCH v6 1/3] pmdinfogen: add Python implementation

[Dmitry Kozlyuk]

Using a high-level, interpreted language simplifies maintenance and
build process. Furthermore, ELF handling is delegated to pyelftools
package. Original logic is kept, the copyright recognizes that.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 buildtools/pmdinfogen.py | 189 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 189 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py

diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
new file mode 100755
index 000000000..0cca47ff1
--- /dev/null
+++ b/buildtools/pmdinfogen.py
@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <nhorman@tuxdriver.com>
+# Copyright (c) 2020 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image.get_section_data(self._symbol["st_shndx"])
+        base = self._symbol["st_value"]
+        return data[base:base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        return value[:-1].decode() if value else ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def get_section_data(self, name):
+        return self._image.get_section(name).data()
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        return ELFSymbol(self, symbol[0]) if symbol else None
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver.pci_ids = cls._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    @staticmethod
+    def _load_pci_ids(image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        result = []
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            result.append([
+                pci_id.vendor_id,
+                pci_id.device_id,
+                pci_id.subsystem_vendor_id,
+                pci_id.subsystem_device_id,
+                ])
+        return result
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
-- 
2.28.0

[dpdk-dev] [PATCH v6 2/3] build: use Python pmdinfogen

[Dmitry Kozlyuk]

Like for other build scripts, use Python interpreter to run pmdinfogen.
Adjust wrapper script accordingly and also don't suppress stderr from ar
and pmdinfogen.

Add python3-pyelftools to CI configuration. The package is available on
all major distributions. FreeBSD has no system requirements section in
its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE)
supported.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 .travis.yml                       | 2 +-
 buildtools/gen-pmdinfo-cfile.sh   | 6 +++---
 buildtools/meson.build            | 1 +
 doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
 drivers/meson.build               | 2 +-
 meson.build                       | 1 -
 6 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 5e12db23b..6744f6fc1 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -11,7 +11,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
diff --git a/buildtools/gen-pmdinfo-cfile.sh b/buildtools/gen-pmdinfo-cfile.sh
index 43059cf36..109ee461e 100755
--- a/buildtools/gen-pmdinfo-cfile.sh
+++ b/buildtools/gen-pmdinfo-cfile.sh
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 04808dabc..3a64b28b7 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -17,3 +17,4 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
index 6ecdc04aa..89c9f2570 100644
--- a/doc/guides/linux_gsg/sys_reqs.rst
+++ b/doc/guides/linux_gsg/sys_reqs.rst
@@ -52,6 +52,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
diff --git a/drivers/meson.build b/drivers/meson.build
index 5f9526557..88364259d 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -137,7 +137,7 @@ foreach subpath:subdirs
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
diff --git a/meson.build b/meson.build
index 61d9a4f5f..a6e8c09c2 100644
--- a/meson.build
+++ b/meson.build
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
-- 
2.28.0

[dpdk-dev] [PATCH v6 3/3] pmdinfogen: remove C implementation

[Dmitry Kozlyuk]

Delete the files no longer used in build process.
Add myself as maintainer of new implementation.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 MAINTAINERS                        |   3 +-
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 4 files changed, 2 insertions(+), 590 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 681093d94..24a8a160c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -133,7 +133,8 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <nhorman@tuxdriver.com>
-F: buildtools/pmdinfogen/
+M: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
diff --git a/buildtools/pmdinfogen/meson.build b/buildtools/pmdinfogen/meson.build
deleted file mode 100644
index 670528fac..000000000
--- a/buildtools/pmdinfogen/meson.build
+++ /dev/null
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
diff --git a/buildtools/pmdinfogen/pmdinfogen.c b/buildtools/pmdinfogen/pmdinfogen.c
deleted file mode 100644
index a68d1ea99..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.c
+++ /dev/null
@@ -1,456 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <ctype.h>
-#include <string.h>
-#include <limits.h>
-#include <stdbool.h>
-#include <errno.h>
-#include <libgen.h>
-
-#include <rte_common.h>
-#include "pmdinfogen.h"
-
-#ifdef RTE_ARCH_64
-#define ADDR_SIZE 64
-#else
-#define ADDR_SIZE 32
-#endif
-
-static int use_stdin, use_stdout;
-
-static const char *sym_name(struct elf_info *elf, Elf_Sym *sym)
-{
-	if (sym)
-		return elf->strtab + sym->st_name;
-	else
-		return "(unknown)";
-}
-
-static void *grab_file(const char *filename, unsigned long *size)
-{
-	struct stat st;
-	void *map = MAP_FAILED;
-	int fd = -1;
-
-	if (!use_stdin) {
-		fd = open(filename, O_RDONLY);
-		if (fd < 0)
-			return NULL;
-	} else {
-		/* from stdin, use a temporary file to mmap */
-		FILE *infile;
-		char buffer[1024];
-		int n;
-
-		infile = tmpfile();
-		if (infile == NULL) {
-			perror("tmpfile");
-			return NULL;
-		}
-		fd = dup(fileno(infile));
-		fclose(infile);
-		if (fd < 0)
-			return NULL;
-
-		n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		while (n > 0) {
-			if (write(fd, buffer, n) != n)
-				goto failed;
-			n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		}
-	}
-
-	if (fstat(fd, &st))
-		goto failed;
-
-	*size = st.st_size;
-	map = mmap(NULL, *size, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);
-
-failed:
-	close(fd);
-	if (map == MAP_FAILED)
-		return NULL;
-	return map;
-}
-
-/**
-  * Return a copy of the next line in a mmap'ed file.
-  * spaces in the beginning of the line is trimmed away.
-  * Return a pointer to a static buffer.
-  **/
-static void release_file(void *file, unsigned long size)
-{
-	munmap(file, size);
-}
-
-
-static void *get_sym_value(struct elf_info *info, const Elf_Sym *sym)
-{
-	return RTE_PTR_ADD(info->hdr,
-		info->sechdrs[sym->st_shndx].sh_offset + sym->st_value);
-}
-
-static Elf_Sym *find_sym_in_symtab(struct elf_info *info,
-				   const char *name, Elf_Sym *last)
-{
-	Elf_Sym *idx;
-	if (last)
-		idx = last+1;
-	else
-		idx = info->symtab_start;
-
-	for (; idx < info->symtab_stop; idx++) {
-		const char *n = sym_name(info, idx);
-		if (!strncmp(n, name, strlen(name)))
-			return idx;
-	}
-	return NULL;
-}
-
-static int parse_elf(struct elf_info *info, const char *filename)
-{
-	unsigned int i;
-	Elf_Ehdr *hdr;
-	Elf_Shdr *sechdrs;
-	Elf_Sym  *sym;
-	int endian;
-	unsigned int symtab_idx = ~0U, symtab_shndx_idx = ~0U;
-
-	hdr = grab_file(filename, &info->size);
-	if (!hdr) {
-		perror(filename);
-		exit(1);
-	}
-	info->hdr = hdr;
-	if (info->size < sizeof(*hdr)) {
-		/* file too small, assume this is an empty .o file */
-		return 0;
-	}
-	/* Is this a valid ELF file? */
-	if ((hdr->e_ident[EI_MAG0] != ELFMAG0) ||
-	    (hdr->e_ident[EI_MAG1] != ELFMAG1) ||
-	    (hdr->e_ident[EI_MAG2] != ELFMAG2) ||
-	    (hdr->e_ident[EI_MAG3] != ELFMAG3)) {
-		/* Not an ELF file - silently ignore it */
-		return 0;
-	}
-
-	if (!hdr->e_ident[EI_DATA]) {
-		/* Unknown endian */
-		return 0;
-	}
-
-	endian = hdr->e_ident[EI_DATA];
-
-	/* Fix endianness in ELF header */
-	hdr->e_type      = TO_NATIVE(endian, 16, hdr->e_type);
-	hdr->e_machine   = TO_NATIVE(endian, 16, hdr->e_machine);
-	hdr->e_version   = TO_NATIVE(endian, 32, hdr->e_version);
-	hdr->e_entry     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_entry);
-	hdr->e_phoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_phoff);
-	hdr->e_shoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_shoff);
-	hdr->e_flags     = TO_NATIVE(endian, 32, hdr->e_flags);
-	hdr->e_ehsize    = TO_NATIVE(endian, 16, hdr->e_ehsize);
-	hdr->e_phentsize = TO_NATIVE(endian, 16, hdr->e_phentsize);
-	hdr->e_phnum     = TO_NATIVE(endian, 16, hdr->e_phnum);
-	hdr->e_shentsize = TO_NATIVE(endian, 16, hdr->e_shentsize);
-	hdr->e_shnum     = TO_NATIVE(endian, 16, hdr->e_shnum);
-	hdr->e_shstrndx  = TO_NATIVE(endian, 16, hdr->e_shstrndx);
-
-	sechdrs = RTE_PTR_ADD(hdr, hdr->e_shoff);
-	info->sechdrs = sechdrs;
-
-	/* Check if file offset is correct */
-	if (hdr->e_shoff > info->size) {
-		fprintf(stderr, "section header offset=%lu in file '%s' "
-		      "is bigger than filesize=%lu\n",
-		      (unsigned long)hdr->e_shoff,
-		      filename, info->size);
-		return 0;
-	}
-
-	if (hdr->e_shnum == SHN_UNDEF) {
-		/*
-		 * There are more than 64k sections,
-		 * read count from .sh_size.
-		 */
-		info->num_sections =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[0].sh_size);
-	} else {
-		info->num_sections = hdr->e_shnum;
-	}
-	if (hdr->e_shstrndx == SHN_XINDEX)
-		info->secindex_strings =
-			TO_NATIVE(endian, 32, sechdrs[0].sh_link);
-	else
-		info->secindex_strings = hdr->e_shstrndx;
-
-	/* Fix endianness in section headers */
-	for (i = 0; i < info->num_sections; i++) {
-		sechdrs[i].sh_name      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_name);
-		sechdrs[i].sh_type      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_type);
-		sechdrs[i].sh_flags     =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_flags);
-		sechdrs[i].sh_addr      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addr);
-		sechdrs[i].sh_offset    =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_offset);
-		sechdrs[i].sh_size      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_size);
-		sechdrs[i].sh_link      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_link);
-		sechdrs[i].sh_info      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_info);
-		sechdrs[i].sh_addralign =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addralign);
-		sechdrs[i].sh_entsize   =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_entsize);
-	}
-	/* Find symbol table. */
-	for (i = 1; i < info->num_sections; i++) {
-		int nobits = sechdrs[i].sh_type == SHT_NOBITS;
-
-		if (!nobits && sechdrs[i].sh_offset > info->size) {
-			fprintf(stderr, "%s is truncated. "
-			      "sechdrs[i].sh_offset=%lu > sizeof(*hrd)=%zu\n",
-			      filename, (unsigned long)sechdrs[i].sh_offset,
-			      sizeof(*hdr));
-			return 0;
-		}
-
-		if (sechdrs[i].sh_type == SHT_SYMTAB) {
-			unsigned int sh_link_idx;
-			symtab_idx = i;
-			info->symtab_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-			sh_link_idx = sechdrs[i].sh_link;
-			info->strtab       = RTE_PTR_ADD(hdr,
-				sechdrs[sh_link_idx].sh_offset);
-		}
-
-		/* 32bit section no. table? ("more than 64k sections") */
-		if (sechdrs[i].sh_type == SHT_SYMTAB_SHNDX) {
-			symtab_shndx_idx = i;
-			info->symtab_shndx_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_shndx_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-		}
-	}
-	if (!info->symtab_start)
-		fprintf(stderr, "%s has no symtab?\n", filename);
-	else {
-		/* Fix endianness in symbols */
-		for (sym = info->symtab_start; sym < info->symtab_stop; sym++) {
-			sym->st_shndx = TO_NATIVE(endian, 16, sym->st_shndx);
-			sym->st_name  = TO_NATIVE(endian, 32, sym->st_name);
-			sym->st_value = TO_NATIVE(endian, ADDR_SIZE, sym->st_value);
-			sym->st_size  = TO_NATIVE(endian, ADDR_SIZE, sym->st_size);
-		}
-	}
-
-	if (symtab_shndx_idx != ~0U) {
-		Elf32_Word *p;
-		if (symtab_idx != sechdrs[symtab_shndx_idx].sh_link)
-			fprintf(stderr,
-			      "%s: SYMTAB_SHNDX has bad sh_link: %u!=%u\n",
-			      filename, sechdrs[symtab_shndx_idx].sh_link,
-			      symtab_idx);
-		/* Fix endianness */
-		for (p = info->symtab_shndx_start; p < info->symtab_shndx_stop;
-		     p++)
-			*p = TO_NATIVE(endian, 32, *p);
-	}
-
-	return 1;
-}
-
-static void parse_elf_finish(struct elf_info *info)
-{
-	struct pmd_driver *tmp, *idx = info->drivers;
-	release_file(info->hdr, info->size);
-	while (idx) {
-		tmp = idx->next;
-		free(idx);
-		idx = tmp;
-	}
-}
-
-struct opt_tag {
-	const char *suffix;
-	const char *json_id;
-};
-
-static const struct opt_tag opt_tags[] = {
-	{"_param_string_export", "params"},
-	{"_kmod_dep_export", "kmod"},
-};
-
-static int complete_pmd_entry(struct elf_info *info, struct pmd_driver *drv)
-{
-	const char *tname;
-	int i;
-	char tmpsymname[128];
-	Elf_Sym *tmpsym;
-
-	drv->name = get_sym_value(info, drv->name_sym);
-
-	for (i = 0; i < PMD_OPT_MAX; i++) {
-		memset(tmpsymname, 0, 128);
-		sprintf(tmpsymname, "__%s%s", drv->name, opt_tags[i].suffix);
-		tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-		if (!tmpsym)
-			continue;
-		drv->opt_vals[i] = get_sym_value(info, tmpsym);
-	}
-
-	memset(tmpsymname, 0, 128);
-	sprintf(tmpsymname, "__%s_pci_tbl_export", drv->name);
-
-	tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-
-
-	/*
-	 * If this returns NULL, then this is a PMD_VDEV, because
-	 * it has no pci table reference
-	 */
-	if (!tmpsym) {
-		drv->pci_tbl = NULL;
-		return 0;
-	}
-
-	tname = get_sym_value(info, tmpsym);
-	tmpsym = find_sym_in_symtab(info, tname, NULL);
-	if (!tmpsym)
-		return -ENOENT;
-
-	drv->pci_tbl = (struct rte_pci_id *)get_sym_value(info, tmpsym);
-	if (!drv->pci_tbl)
-		return -ENOENT;
-
-	return 0;
-}
-
-static int locate_pmd_entries(struct elf_info *info)
-{
-	Elf_Sym *last = NULL;
-	struct pmd_driver *new;
-
-	info->drivers = NULL;
-
-	do {
-		new = calloc(sizeof(struct pmd_driver), 1);
-		if (new == NULL) {
-			fprintf(stderr, "Failed to calloc memory\n");
-			return -1;
-		}
-		new->name_sym = find_sym_in_symtab(info, "this_pmd_name", last);
-		last = new->name_sym;
-		if (!new->name_sym)
-			free(new);
-		else {
-			if (complete_pmd_entry(info, new)) {
-				fprintf(stderr,
-					"Failed to complete pmd entry\n");
-				free(new);
-			} else {
-				new->next = info->drivers;
-				info->drivers = new;
-			}
-		}
-	} while (last);
-
-	return 0;
-}
-
-static void output_pmd_info_string(struct elf_info *info, char *outfile)
-{
-	FILE *ofd;
-	struct pmd_driver *drv;
-	struct rte_pci_id *pci_ids;
-	int idx = 0;
-
-	if (use_stdout)
-		ofd = stdout;
-	else {
-		ofd = fopen(outfile, "w+");
-		if (!ofd) {
-			fprintf(stderr, "Unable to open output file\n");
-			return;
-		}
-	}
-
-	drv = info->drivers;
-
-	while (drv) {
-		fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
-			"\"PMD_INFO_STRING= {",
-			drv->name);
-		fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);
-
-		for (idx = 0; idx < PMD_OPT_MAX; idx++) {
-			if (drv->opt_vals[idx])
-				fprintf(ofd, "\\\"%s\\\" : \\\"%s\\\", ",
-					opt_tags[idx].json_id,
-					drv->opt_vals[idx]);
-		}
-
-		pci_ids = drv->pci_tbl;
-		fprintf(ofd, "\\\"pci_ids\\\" : [");
-
-		while (pci_ids && pci_ids->device_id) {
-			fprintf(ofd, "[%d, %d, %d, %d]",
-				pci_ids->vendor_id, pci_ids->device_id,
-				pci_ids->subsystem_vendor_id,
-				pci_ids->subsystem_device_id);
-			pci_ids++;
-			if (pci_ids->device_id)
-				fprintf(ofd, ",");
-			else
-				fprintf(ofd, " ");
-		}
-		fprintf(ofd, "]}\";\n");
-		drv = drv->next;
-	}
-
-	fclose(ofd);
-}
-
-int main(int argc, char **argv)
-{
-	struct elf_info info = {0};
-	int rc = 1;
-
-	if (argc < 3) {
-		fprintf(stderr,
-			"usage: %s <object file> <c output file>\n",
-			basename(argv[0]));
-		exit(127);
-	}
-	use_stdin = !strcmp(argv[1], "-");
-	use_stdout = !strcmp(argv[2], "-");
-	parse_elf(&info, argv[1]);
-
-	if (locate_pmd_entries(&info) < 0)
-		exit(1);
-
-	if (info.drivers) {
-		output_pmd_info_string(&info, argv[2]);
-		rc = 0;
-	} else {
-		fprintf(stderr, "No drivers registered\n");
-	}
-
-	parse_elf_finish(&info);
-	exit(rc);
-}
diff --git a/buildtools/pmdinfogen/pmdinfogen.h b/buildtools/pmdinfogen/pmdinfogen.h
deleted file mode 100644
index 93930e454..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.h
+++ /dev/null
@@ -1,119 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <stdarg.h>
-#include <string.h>
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#ifdef __linux__
-#include <endian.h>
-#else
-#include <sys/endian.h>
-#endif
-#include <fcntl.h>
-#include <unistd.h>
-#include <elf.h>
-#include <rte_pci.h>
-
-/* On BSD-alike OSes elf.h defines these according to host's word size */
-#undef ELF_ST_BIND
-#undef ELF_ST_TYPE
-#undef ELF_R_SYM
-#undef ELF_R_TYPE
-
-/*
- * Define ELF64_* to ELF_*, the latter being defined in both 32 and 64 bit
- * flavors in elf.h.  This makes our code a bit more generic between arches
- * and allows us to support 32 bit code in the future should we ever want to
- */
-#ifdef RTE_ARCH_64
-#define Elf_Ehdr    Elf64_Ehdr
-#define Elf_Shdr    Elf64_Shdr
-#define Elf_Sym     Elf64_Sym
-#define Elf_Addr    Elf64_Addr
-#define Elf_Sword   Elf64_Sxword
-#define Elf_Section Elf64_Half
-#define ELF_ST_BIND ELF64_ST_BIND
-#define ELF_ST_TYPE ELF64_ST_TYPE
-
-#define Elf_Rel     Elf64_Rel
-#define Elf_Rela    Elf64_Rela
-#define ELF_R_SYM   ELF64_R_SYM
-#define ELF_R_TYPE  ELF64_R_TYPE
-#else
-#define Elf_Ehdr    Elf32_Ehdr
-#define Elf_Shdr    Elf32_Shdr
-#define Elf_Sym     Elf32_Sym
-#define Elf_Addr    Elf32_Addr
-#define Elf_Sword   Elf32_Sxword
-#define Elf_Section Elf32_Half
-#define ELF_ST_BIND ELF32_ST_BIND
-#define ELF_ST_TYPE ELF32_ST_TYPE
-
-#define Elf_Rel     Elf32_Rel
-#define Elf_Rela    Elf32_Rela
-#define ELF_R_SYM   ELF32_R_SYM
-#define ELF_R_TYPE  ELF32_R_TYPE
-#endif
-
-
-/*
- * Note, it seems odd that we have both a CONVERT_NATIVE and a TO_NATIVE macro
- * below.  We do this because the values passed to TO_NATIVE may themselves be
- * macros and need both macros here to get expanded.  Specifically its the width
- * variable we are concerned with, because it needs to get expanded prior to
- * string concatenation
- */
-#define CONVERT_NATIVE(fend, width, x) ({ \
-typeof(x) ___x; \
-if ((fend) == ELFDATA2LSB) \
-	___x = le##width##toh(x); \
-else \
-	___x = be##width##toh(x); \
-	___x; \
-})
-
-#define TO_NATIVE(fend, width, x) CONVERT_NATIVE(fend, width, x)
-
-enum opt_params {
-	PMD_PARAM_STRING = 0,
-	PMD_KMOD_DEP,
-	PMD_OPT_MAX
-};
-
-struct pmd_driver {
-	Elf_Sym *name_sym;
-	const char *name;
-	struct rte_pci_id *pci_tbl;
-	struct pmd_driver *next;
-
-	const char *opt_vals[PMD_OPT_MAX];
-};
-
-struct elf_info {
-	unsigned long size;
-	Elf_Ehdr     *hdr;
-	Elf_Shdr     *sechdrs;
-	Elf_Sym      *symtab_start;
-	Elf_Sym      *symtab_stop;
-	char         *strtab;
-
-	/* support for 32bit section numbers */
-
-	unsigned int num_sections; /* max_secindex + 1 */
-	unsigned int secindex_strings;
-	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
-	 * take shndx from symtab_shndx_start[N] instead
-	 */
-	Elf32_Word   *symtab_shndx_start;
-	Elf32_Word   *symtab_shndx_stop;
-
-	struct pmd_driver *drivers;
-};
-- 
2.28.0

Re: [dpdk-dev] [PATCH v6 0/3] pmdinfogen: rewrite in Python

[Maxime Coquelin]

Hi Dmitry,

On 10/4/20 3:59 AM, Dmitry Kozlyuk wrote:
> This patchset implements existing pmdinfogen logic in Python, replaces
> and removes the old code. The goals of rewriting are:
> 
> * easier maintenance by using a more high-level language,
> * simpler build process without host application and libelf,
> * foundation for adding Windows support.
> 
> Travis CI script is adjusted to install python3-pyelftools, but other CI
> systems may need similar tweaking. Particularly, testing on FreeBSD and
> big-endian targets is desired.
> 
> Acked-by: Neil Horman <nhorman@tuxdriver.com>
> 
> ---
> Changes in v6:
> 
> 1. Address yapf and pylint warnings (Stephen Hemminger).
> 2. Add myself as new implementation maintainer (Thomas Monjalon).
> 
> Dmitry Kozlyuk (3):
>   pmdinfogen: add Python implementation
>   build: use Python pmdinfogen
>   pmdinfogen: remove C implementation
> 
>  .travis.yml                        |   2 +-
>  MAINTAINERS                        |   3 +-
>  buildtools/gen-pmdinfo-cfile.sh    |   6 +-
>  buildtools/meson.build             |   1 +
>  buildtools/pmdinfogen.py           | 189 ++++++++++++
>  buildtools/pmdinfogen/meson.build  |  14 -
>  buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
>  buildtools/pmdinfogen/pmdinfogen.h | 119 --------
>  doc/guides/linux_gsg/sys_reqs.rst  |   6 +
>  drivers/meson.build                |   2 +-
>  meson.build                        |   1 -
>  11 files changed, 203 insertions(+), 596 deletions(-)
>  create mode 100755 buildtools/pmdinfogen.py
>  delete mode 100644 buildtools/pmdinfogen/meson.build
>  delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
>  delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
> 

Ninja fails when pyelftools is not installed.
Shouldn't meson detect and report it before build time?

Thanks,
Maxime

Re: [dpdk-dev] [PATCH v6 0/3] pmdinfogen: rewrite in Python

[Dmitry Kozlyuk]

Hi Maxime,

> Ninja fails when pyelftools is not installed.
> Shouldn't meson detect and report it before build time? 

Will add in v7. Meson 0.51 introduced a terse way to check for Python
dependencies, but it's not hard even with Meson 0.47.1.

https://github.com/mesonbuild/meson/issues/3551#issuecomment-518320640

[dpdk-dev] [PATCH v7 0/3] pmdinfogen: rewrite in Python

[Dmitry Kozlyuk]

This patchset implements existing pmdinfogen logic in Python, replaces
and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.

Travis CI script is adjusted to install python3-pyelftools, but other CI
systems may need similar tweaking. Particularly, testing on FreeBSD and
big-endian targets is desired.

Acked-by: Neil Horman <nhorman@tuxdriver.com>

---
Changes in v7:

    Add configure-time check for elftools module (Maxime Coquelin).

Dmitry Kozlyuk (3):
  pmdinfogen: add Python implementation
  build: use Python pmdinfogen
  pmdinfogen: remove C implementation

 .travis.yml                        |   2 +-
 MAINTAINERS                        |   3 +-
 buildtools/gen-pmdinfo-cfile.sh    |   6 +-
 buildtools/meson.build             |  15 +
 buildtools/pmdinfogen.py           | 189 ++++++++++++
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 doc/guides/linux_gsg/sys_reqs.rst  |   6 +
 drivers/meson.build                |   2 +-
 meson.build                        |   1 -
 11 files changed, 217 insertions(+), 596 deletions(-)
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

-- 
2.28.0

[dpdk-dev] [PATCH v7 1/3] pmdinfogen: add Python implementation

[Dmitry Kozlyuk]

Using a high-level, interpreted language simplifies maintenance and
build process. Furthermore, ELF handling is delegated to pyelftools
package. Original logic is kept, the copyright recognizes that.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 buildtools/pmdinfogen.py | 189 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 189 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py

diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
new file mode 100755
index 000000000..0cca47ff1
--- /dev/null
+++ b/buildtools/pmdinfogen.py
@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <nhorman@tuxdriver.com>
+# Copyright (c) 2020 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image.get_section_data(self._symbol["st_shndx"])
+        base = self._symbol["st_value"]
+        return data[base:base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        return value[:-1].decode() if value else ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def get_section_data(self, name):
+        return self._image.get_section(name).data()
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        return ELFSymbol(self, symbol[0]) if symbol else None
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver.pci_ids = cls._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    @staticmethod
+    def _load_pci_ids(image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        result = []
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            result.append([
+                pci_id.vendor_id,
+                pci_id.device_id,
+                pci_id.subsystem_vendor_id,
+                pci_id.subsystem_device_id,
+                ])
+        return result
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
-- 
2.28.0

[dpdk-dev] [PATCH v7 2/3] build: use Python pmdinfogen

[Dmitry Kozlyuk]

Use the same interpreter to run pmdinfogen as for other build scripts.
Adjust wrapper script accordingly and also don't suppress stderr from ar
and pmdinfogen.

Add configure-time check for elftools Python module.

Add python3-pyelftools to CI configuration. The package is available on
all major distributions. FreeBSD has no system requirements section in
its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE)
supported.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 .travis.yml                       |  2 +-
 buildtools/gen-pmdinfo-cfile.sh   |  6 +++---
 buildtools/meson.build            | 15 +++++++++++++++
 doc/guides/linux_gsg/sys_reqs.rst |  6 ++++++
 drivers/meson.build               |  2 +-
 meson.build                       |  1 -
 6 files changed, 26 insertions(+), 6 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 5e12db23b..6744f6fc1 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -11,7 +11,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
diff --git a/buildtools/gen-pmdinfo-cfile.sh b/buildtools/gen-pmdinfo-cfile.sh
index 43059cf36..109ee461e 100755
--- a/buildtools/gen-pmdinfo-cfile.sh
+++ b/buildtools/gen-pmdinfo-cfile.sh
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 04808dabc..0c668eb46 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -17,3 +17,18 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
+
+# TODO: starting from Meson 0.51.0 use
+# 	python3 = import('python').find_installation('python',
+#		modules : python3_required_modules)
+python3_required_modules = ['elftools']
+foreach module : python3_required_modules
+	script = \
+		'import importlib.util;' + \
+		'import sys;' + \
+		'exit(importlib.util.find_spec("@0@") is None)'.format(module)
+	if run_command(py3, '-c', script).returncode() != 0
+		error('missing python module: @0@'.format(module))
+	endif
+endforeach
diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
index 6ecdc04aa..89c9f2570 100644
--- a/doc/guides/linux_gsg/sys_reqs.rst
+++ b/doc/guides/linux_gsg/sys_reqs.rst
@@ -52,6 +52,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
diff --git a/drivers/meson.build b/drivers/meson.build
index 5f9526557..88364259d 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -137,7 +137,7 @@ foreach subpath:subdirs
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
diff --git a/meson.build b/meson.build
index 61d9a4f5f..a6e8c09c2 100644
--- a/meson.build
+++ b/meson.build
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
-- 
2.28.0

[dpdk-dev] [PATCH v7 3/3] pmdinfogen: remove C implementation

[Dmitry Kozlyuk]

Delete the files no longer used in build process.
Add myself as maintainer of new implementation.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 MAINTAINERS                        |   3 +-
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 4 files changed, 2 insertions(+), 590 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 499c7d297..5f5c599d2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -133,7 +133,8 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <nhorman@tuxdriver.com>
-F: buildtools/pmdinfogen/
+M: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
diff --git a/buildtools/pmdinfogen/meson.build b/buildtools/pmdinfogen/meson.build
deleted file mode 100644
index 670528fac..000000000
--- a/buildtools/pmdinfogen/meson.build
+++ /dev/null
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
diff --git a/buildtools/pmdinfogen/pmdinfogen.c b/buildtools/pmdinfogen/pmdinfogen.c
deleted file mode 100644
index a68d1ea99..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.c
+++ /dev/null
@@ -1,456 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <ctype.h>
-#include <string.h>
-#include <limits.h>
-#include <stdbool.h>
-#include <errno.h>
-#include <libgen.h>
-
-#include <rte_common.h>
-#include "pmdinfogen.h"
-
-#ifdef RTE_ARCH_64
-#define ADDR_SIZE 64
-#else
-#define ADDR_SIZE 32
-#endif
-
-static int use_stdin, use_stdout;
-
-static const char *sym_name(struct elf_info *elf, Elf_Sym *sym)
-{
-	if (sym)
-		return elf->strtab + sym->st_name;
-	else
-		return "(unknown)";
-}
-
-static void *grab_file(const char *filename, unsigned long *size)
-{
-	struct stat st;
-	void *map = MAP_FAILED;
-	int fd = -1;
-
-	if (!use_stdin) {
-		fd = open(filename, O_RDONLY);
-		if (fd < 0)
-			return NULL;
-	} else {
-		/* from stdin, use a temporary file to mmap */
-		FILE *infile;
-		char buffer[1024];
-		int n;
-
-		infile = tmpfile();
-		if (infile == NULL) {
-			perror("tmpfile");
-			return NULL;
-		}
-		fd = dup(fileno(infile));
-		fclose(infile);
-		if (fd < 0)
-			return NULL;
-
-		n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		while (n > 0) {
-			if (write(fd, buffer, n) != n)
-				goto failed;
-			n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		}
-	}
-
-	if (fstat(fd, &st))
-		goto failed;
-
-	*size = st.st_size;
-	map = mmap(NULL, *size, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);
-
-failed:
-	close(fd);
-	if (map == MAP_FAILED)
-		return NULL;
-	return map;
-}
-
-/**
-  * Return a copy of the next line in a mmap'ed file.
-  * spaces in the beginning of the line is trimmed away.
-  * Return a pointer to a static buffer.
-  **/
-static void release_file(void *file, unsigned long size)
-{
-	munmap(file, size);
-}
-
-
-static void *get_sym_value(struct elf_info *info, const Elf_Sym *sym)
-{
-	return RTE_PTR_ADD(info->hdr,
-		info->sechdrs[sym->st_shndx].sh_offset + sym->st_value);
-}
-
-static Elf_Sym *find_sym_in_symtab(struct elf_info *info,
-				   const char *name, Elf_Sym *last)
-{
-	Elf_Sym *idx;
-	if (last)
-		idx = last+1;
-	else
-		idx = info->symtab_start;
-
-	for (; idx < info->symtab_stop; idx++) {
-		const char *n = sym_name(info, idx);
-		if (!strncmp(n, name, strlen(name)))
-			return idx;
-	}
-	return NULL;
-}
-
-static int parse_elf(struct elf_info *info, const char *filename)
-{
-	unsigned int i;
-	Elf_Ehdr *hdr;
-	Elf_Shdr *sechdrs;
-	Elf_Sym  *sym;
-	int endian;
-	unsigned int symtab_idx = ~0U, symtab_shndx_idx = ~0U;
-
-	hdr = grab_file(filename, &info->size);
-	if (!hdr) {
-		perror(filename);
-		exit(1);
-	}
-	info->hdr = hdr;
-	if (info->size < sizeof(*hdr)) {
-		/* file too small, assume this is an empty .o file */
-		return 0;
-	}
-	/* Is this a valid ELF file? */
-	if ((hdr->e_ident[EI_MAG0] != ELFMAG0) ||
-	    (hdr->e_ident[EI_MAG1] != ELFMAG1) ||
-	    (hdr->e_ident[EI_MAG2] != ELFMAG2) ||
-	    (hdr->e_ident[EI_MAG3] != ELFMAG3)) {
-		/* Not an ELF file - silently ignore it */
-		return 0;
-	}
-
-	if (!hdr->e_ident[EI_DATA]) {
-		/* Unknown endian */
-		return 0;
-	}
-
-	endian = hdr->e_ident[EI_DATA];
-
-	/* Fix endianness in ELF header */
-	hdr->e_type      = TO_NATIVE(endian, 16, hdr->e_type);
-	hdr->e_machine   = TO_NATIVE(endian, 16, hdr->e_machine);
-	hdr->e_version   = TO_NATIVE(endian, 32, hdr->e_version);
-	hdr->e_entry     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_entry);
-	hdr->e_phoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_phoff);
-	hdr->e_shoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_shoff);
-	hdr->e_flags     = TO_NATIVE(endian, 32, hdr->e_flags);
-	hdr->e_ehsize    = TO_NATIVE(endian, 16, hdr->e_ehsize);
-	hdr->e_phentsize = TO_NATIVE(endian, 16, hdr->e_phentsize);
-	hdr->e_phnum     = TO_NATIVE(endian, 16, hdr->e_phnum);
-	hdr->e_shentsize = TO_NATIVE(endian, 16, hdr->e_shentsize);
-	hdr->e_shnum     = TO_NATIVE(endian, 16, hdr->e_shnum);
-	hdr->e_shstrndx  = TO_NATIVE(endian, 16, hdr->e_shstrndx);
-
-	sechdrs = RTE_PTR_ADD(hdr, hdr->e_shoff);
-	info->sechdrs = sechdrs;
-
-	/* Check if file offset is correct */
-	if (hdr->e_shoff > info->size) {
-		fprintf(stderr, "section header offset=%lu in file '%s' "
-		      "is bigger than filesize=%lu\n",
-		      (unsigned long)hdr->e_shoff,
-		      filename, info->size);
-		return 0;
-	}
-
-	if (hdr->e_shnum == SHN_UNDEF) {
-		/*
-		 * There are more than 64k sections,
-		 * read count from .sh_size.
-		 */
-		info->num_sections =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[0].sh_size);
-	} else {
-		info->num_sections = hdr->e_shnum;
-	}
-	if (hdr->e_shstrndx == SHN_XINDEX)
-		info->secindex_strings =
-			TO_NATIVE(endian, 32, sechdrs[0].sh_link);
-	else
-		info->secindex_strings = hdr->e_shstrndx;
-
-	/* Fix endianness in section headers */
-	for (i = 0; i < info->num_sections; i++) {
-		sechdrs[i].sh_name      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_name);
-		sechdrs[i].sh_type      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_type);
-		sechdrs[i].sh_flags     =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_flags);
-		sechdrs[i].sh_addr      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addr);
-		sechdrs[i].sh_offset    =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_offset);
-		sechdrs[i].sh_size      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_size);
-		sechdrs[i].sh_link      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_link);
-		sechdrs[i].sh_info      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_info);
-		sechdrs[i].sh_addralign =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addralign);
-		sechdrs[i].sh_entsize   =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_entsize);
-	}
-	/* Find symbol table. */
-	for (i = 1; i < info->num_sections; i++) {
-		int nobits = sechdrs[i].sh_type == SHT_NOBITS;
-
-		if (!nobits && sechdrs[i].sh_offset > info->size) {
-			fprintf(stderr, "%s is truncated. "
-			      "sechdrs[i].sh_offset=%lu > sizeof(*hrd)=%zu\n",
-			      filename, (unsigned long)sechdrs[i].sh_offset,
-			      sizeof(*hdr));
-			return 0;
-		}
-
-		if (sechdrs[i].sh_type == SHT_SYMTAB) {
-			unsigned int sh_link_idx;
-			symtab_idx = i;
-			info->symtab_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-			sh_link_idx = sechdrs[i].sh_link;
-			info->strtab       = RTE_PTR_ADD(hdr,
-				sechdrs[sh_link_idx].sh_offset);
-		}
-
-		/* 32bit section no. table? ("more than 64k sections") */
-		if (sechdrs[i].sh_type == SHT_SYMTAB_SHNDX) {
-			symtab_shndx_idx = i;
-			info->symtab_shndx_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_shndx_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-		}
-	}
-	if (!info->symtab_start)
-		fprintf(stderr, "%s has no symtab?\n", filename);
-	else {
-		/* Fix endianness in symbols */
-		for (sym = info->symtab_start; sym < info->symtab_stop; sym++) {
-			sym->st_shndx = TO_NATIVE(endian, 16, sym->st_shndx);
-			sym->st_name  = TO_NATIVE(endian, 32, sym->st_name);
-			sym->st_value = TO_NATIVE(endian, ADDR_SIZE, sym->st_value);
-			sym->st_size  = TO_NATIVE(endian, ADDR_SIZE, sym->st_size);
-		}
-	}
-
-	if (symtab_shndx_idx != ~0U) {
-		Elf32_Word *p;
-		if (symtab_idx != sechdrs[symtab_shndx_idx].sh_link)
-			fprintf(stderr,
-			      "%s: SYMTAB_SHNDX has bad sh_link: %u!=%u\n",
-			      filename, sechdrs[symtab_shndx_idx].sh_link,
-			      symtab_idx);
-		/* Fix endianness */
-		for (p = info->symtab_shndx_start; p < info->symtab_shndx_stop;
-		     p++)
-			*p = TO_NATIVE(endian, 32, *p);
-	}
-
-	return 1;
-}
-
-static void parse_elf_finish(struct elf_info *info)
-{
-	struct pmd_driver *tmp, *idx = info->drivers;
-	release_file(info->hdr, info->size);
-	while (idx) {
-		tmp = idx->next;
-		free(idx);
-		idx = tmp;
-	}
-}
-
-struct opt_tag {
-	const char *suffix;
-	const char *json_id;
-};
-
-static const struct opt_tag opt_tags[] = {
-	{"_param_string_export", "params"},
-	{"_kmod_dep_export", "kmod"},
-};
-
-static int complete_pmd_entry(struct elf_info *info, struct pmd_driver *drv)
-{
-	const char *tname;
-	int i;
-	char tmpsymname[128];
-	Elf_Sym *tmpsym;
-
-	drv->name = get_sym_value(info, drv->name_sym);
-
-	for (i = 0; i < PMD_OPT_MAX; i++) {
-		memset(tmpsymname, 0, 128);
-		sprintf(tmpsymname, "__%s%s", drv->name, opt_tags[i].suffix);
-		tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-		if (!tmpsym)
-			continue;
-		drv->opt_vals[i] = get_sym_value(info, tmpsym);
-	}
-
-	memset(tmpsymname, 0, 128);
-	sprintf(tmpsymname, "__%s_pci_tbl_export", drv->name);
-
-	tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-
-
-	/*
-	 * If this returns NULL, then this is a PMD_VDEV, because
-	 * it has no pci table reference
-	 */
-	if (!tmpsym) {
-		drv->pci_tbl = NULL;
-		return 0;
-	}
-
-	tname = get_sym_value(info, tmpsym);
-	tmpsym = find_sym_in_symtab(info, tname, NULL);
-	if (!tmpsym)
-		return -ENOENT;
-
-	drv->pci_tbl = (struct rte_pci_id *)get_sym_value(info, tmpsym);
-	if (!drv->pci_tbl)
-		return -ENOENT;
-
-	return 0;
-}
-
-static int locate_pmd_entries(struct elf_info *info)
-{
-	Elf_Sym *last = NULL;
-	struct pmd_driver *new;
-
-	info->drivers = NULL;
-
-	do {
-		new = calloc(sizeof(struct pmd_driver), 1);
-		if (new == NULL) {
-			fprintf(stderr, "Failed to calloc memory\n");
-			return -1;
-		}
-		new->name_sym = find_sym_in_symtab(info, "this_pmd_name", last);
-		last = new->name_sym;
-		if (!new->name_sym)
-			free(new);
-		else {
-			if (complete_pmd_entry(info, new)) {
-				fprintf(stderr,
-					"Failed to complete pmd entry\n");
-				free(new);
-			} else {
-				new->next = info->drivers;
-				info->drivers = new;
-			}
-		}
-	} while (last);
-
-	return 0;
-}
-
-static void output_pmd_info_string(struct elf_info *info, char *outfile)
-{
-	FILE *ofd;
-	struct pmd_driver *drv;
-	struct rte_pci_id *pci_ids;
-	int idx = 0;
-
-	if (use_stdout)
-		ofd = stdout;
-	else {
-		ofd = fopen(outfile, "w+");
-		if (!ofd) {
-			fprintf(stderr, "Unable to open output file\n");
-			return;
-		}
-	}
-
-	drv = info->drivers;
-
-	while (drv) {
-		fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
-			"\"PMD_INFO_STRING= {",
-			drv->name);
-		fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);
-
-		for (idx = 0; idx < PMD_OPT_MAX; idx++) {
-			if (drv->opt_vals[idx])
-				fprintf(ofd, "\\\"%s\\\" : \\\"%s\\\", ",
-					opt_tags[idx].json_id,
-					drv->opt_vals[idx]);
-		}
-
-		pci_ids = drv->pci_tbl;
-		fprintf(ofd, "\\\"pci_ids\\\" : [");
-
-		while (pci_ids && pci_ids->device_id) {
-			fprintf(ofd, "[%d, %d, %d, %d]",
-				pci_ids->vendor_id, pci_ids->device_id,
-				pci_ids->subsystem_vendor_id,
-				pci_ids->subsystem_device_id);
-			pci_ids++;
-			if (pci_ids->device_id)
-				fprintf(ofd, ",");
-			else
-				fprintf(ofd, " ");
-		}
-		fprintf(ofd, "]}\";\n");
-		drv = drv->next;
-	}
-
-	fclose(ofd);
-}
-
-int main(int argc, char **argv)
-{
-	struct elf_info info = {0};
-	int rc = 1;
-
-	if (argc < 3) {
-		fprintf(stderr,
-			"usage: %s <object file> <c output file>\n",
-			basename(argv[0]));
-		exit(127);
-	}
-	use_stdin = !strcmp(argv[1], "-");
-	use_stdout = !strcmp(argv[2], "-");
-	parse_elf(&info, argv[1]);
-
-	if (locate_pmd_entries(&info) < 0)
-		exit(1);
-
-	if (info.drivers) {
-		output_pmd_info_string(&info, argv[2]);
-		rc = 0;
-	} else {
-		fprintf(stderr, "No drivers registered\n");
-	}
-
-	parse_elf_finish(&info);
-	exit(rc);
-}
diff --git a/buildtools/pmdinfogen/pmdinfogen.h b/buildtools/pmdinfogen/pmdinfogen.h
deleted file mode 100644
index 38fd3ddcc..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.h
+++ /dev/null
@@ -1,119 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <stdarg.h>
-#include <string.h>
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#ifdef __linux__
-#include <endian.h>
-#else
-#include <sys/endian.h>
-#endif
-#include <fcntl.h>
-#include <unistd.h>
-#include <elf.h>
-#include <rte_pci.h>
-
-/* On BSD-alike OSes elf.h defines these according to host's word size */
-#undef ELF_ST_BIND
-#undef ELF_ST_TYPE
-#undef ELF_R_SYM
-#undef ELF_R_TYPE
-
-/*
- * Define ELF64_* to ELF_*, the latter being defined in both 32 and 64 bit
- * flavors in elf.h.  This makes our code a bit more generic between arches
- * and allows us to support 32 bit code in the future should we ever want to
- */
-#ifdef RTE_ARCH_64
-#define Elf_Ehdr    Elf64_Ehdr
-#define Elf_Shdr    Elf64_Shdr
-#define Elf_Sym     Elf64_Sym
-#define Elf_Addr    Elf64_Addr
-#define Elf_Sword   Elf64_Sxword
-#define Elf_Section Elf64_Half
-#define ELF_ST_BIND ELF64_ST_BIND
-#define ELF_ST_TYPE ELF64_ST_TYPE
-
-#define Elf_Rel     Elf64_Rel
-#define Elf_Rela    Elf64_Rela
-#define ELF_R_SYM   ELF64_R_SYM
-#define ELF_R_TYPE  ELF64_R_TYPE
-#else
-#define Elf_Ehdr    Elf32_Ehdr
-#define Elf_Shdr    Elf32_Shdr
-#define Elf_Sym     Elf32_Sym
-#define Elf_Addr    Elf32_Addr
-#define Elf_Sword   Elf32_Sxword
-#define Elf_Section Elf32_Half
-#define ELF_ST_BIND ELF32_ST_BIND
-#define ELF_ST_TYPE ELF32_ST_TYPE
-
-#define Elf_Rel     Elf32_Rel
-#define Elf_Rela    Elf32_Rela
-#define ELF_R_SYM   ELF32_R_SYM
-#define ELF_R_TYPE  ELF32_R_TYPE
-#endif
-
-
-/*
- * Note, it seems odd that we have both a CONVERT_NATIVE and a TO_NATIVE macro
- * below.  We do this because the values passed to TO_NATIVE may themselves be
- * macros and need both macros here to get expanded.  Specifically its the width
- * variable we are concerned with, because it needs to get expanded prior to
- * string concatenation
- */
-#define CONVERT_NATIVE(fend, width, x) ({ \
-typeof(x) ___x; \
-if ((fend) == ELFDATA2LSB) \
-	___x = le##width##toh(x); \
-else \
-	___x = be##width##toh(x); \
-___x; \
-})
-
-#define TO_NATIVE(fend, width, x) CONVERT_NATIVE(fend, width, x)
-
-enum opt_params {
-	PMD_PARAM_STRING = 0,
-	PMD_KMOD_DEP,
-	PMD_OPT_MAX
-};
-
-struct pmd_driver {
-	Elf_Sym *name_sym;
-	const char *name;
-	struct rte_pci_id *pci_tbl;
-	struct pmd_driver *next;
-
-	const char *opt_vals[PMD_OPT_MAX];
-};
-
-struct elf_info {
-	unsigned long size;
-	Elf_Ehdr     *hdr;
-	Elf_Shdr     *sechdrs;
-	Elf_Sym      *symtab_start;
-	Elf_Sym      *symtab_stop;
-	char         *strtab;
-
-	/* support for 32bit section numbers */
-
-	unsigned int num_sections; /* max_secindex + 1 */
-	unsigned int secindex_strings;
-	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
-	 * take shndx from symtab_shndx_start[N] instead
-	 */
-	Elf32_Word   *symtab_shndx_start;
-	Elf32_Word   *symtab_shndx_stop;
-
-	struct pmd_driver *drivers;
-};
-- 
2.28.0

Re: [dpdk-dev] [PATCH v7 0/3] pmdinfogen: rewrite in Python

[David Marchand]

Hello Dmitry,

On Wed, Oct 14, 2020 at 8:31 PM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
>
> This patchset implements existing pmdinfogen logic in Python, replaces
> and removes the old code. The goals of rewriting are:
>
> * easier maintenance by using a more high-level language,
> * simpler build process without host application and libelf,
> * foundation for adding Windows support.
>
> Travis CI script is adjusted to install python3-pyelftools, but other CI
> systems may need similar tweaking. Particularly, testing on FreeBSD and
> big-endian targets is desired.
>

- Travis reported an error in meson:
https://travis-ci.com/github/ovsrobot/dpdk/jobs/399762460#L478

Build machine cpu: x86_64
buildtools/meson.build:27:10: ERROR:  Expecting endforeach got eol_cont.
script = \
          ^
For a block that started at 26,0
foreach module : python3_required_modules
^


- In a thread on ci@dpdk.org
http://inbox.dpdk.org/ci/CAOeXdvbosqRGouSX06_phVTCam57XUt0s=sf_kpe-VmiwFTgMA@mail.gmail.com/T/#m9bb193c74ca979c151f2dbfa35664f95093a9e71,
UNH people confirmed that python3-elftools should be available, but
the last test had an error too, which might be linked to incorrect
meson syntax:
http://mails.dpdk.org/archives/test-report/2020-October/159038.html

ninja: error: loading 'build.ninja': No such file or directory


- As far as Intel CI systems are concerned, there might be an issue
with the Windows system:
http://mails.dpdk.org/archives/test-report/2020-October/159123.html

OS: WIN10-64
Target: x86_64-windows-clang
Program binutils-avx512-check.sh found: YES
Program python3 found: YES (c:\program files\python37\python.exe)

buildtools\meson.build:32:2: ERROR: Problem encountered: missing
python module: elftools

A full log can be found at C:\windows-dpdk\dpdk\build\meson-logs\meson-log.txt
ninja: Entering directory `build'
DPDK STV team



-- 
David Marchand

[dpdk-dev] [PATCH v8 0/3] pmdinfogen: rewrite in Python

[Dmitry Kozlyuk]

This patchset implements existing pmdinfogen logic in Python, replaces
and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.

Travis CI script is adjusted to install python3-pyelftools, but other CI
systems may need similar tweaking. Particularly, testing on FreeBSD and
big-endian targets is desired.

Acked-by: Neil Horman <nhorman@tuxdriver.com>

---
Changes in v8:

    * Fix syntax for meson 0.47.1 (CI).
    * Disable configure-time check for pyelftools for Windows (CI).

Dmitry Kozlyuk (3):
  pmdinfogen: add Python implementation
  build: use Python pmdinfogen
  pmdinfogen: remove C implementation

 .travis.yml                        |   2 +-
 MAINTAINERS                        |   3 +-
 buildtools/gen-pmdinfo-cfile.sh    |   6 +-
 buildtools/meson.build             |  15 +
 buildtools/pmdinfogen.py           | 189 ++++++++++++
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 doc/guides/linux_gsg/sys_reqs.rst  |   6 +
 drivers/meson.build                |   2 +-
 meson.build                        |   1 -
 11 files changed, 217 insertions(+), 596 deletions(-)
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

-- 
2.28.0

[dpdk-dev] [PATCH v8 1/3] pmdinfogen: add Python implementation

[Dmitry Kozlyuk]

Using a high-level, interpreted language simplifies maintenance and
build process. Furthermore, ELF handling is delegated to pyelftools
package. Original logic is kept, the copyright recognizes that.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 buildtools/pmdinfogen.py | 189 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 189 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py

diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
new file mode 100755
index 000000000..0cca47ff1
--- /dev/null
+++ b/buildtools/pmdinfogen.py
@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <nhorman@tuxdriver.com>
+# Copyright (c) 2020 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image.get_section_data(self._symbol["st_shndx"])
+        base = self._symbol["st_value"]
+        return data[base:base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        return value[:-1].decode() if value else ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def get_section_data(self, name):
+        return self._image.get_section(name).data()
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        return ELFSymbol(self, symbol[0]) if symbol else None
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver.pci_ids = cls._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    @staticmethod
+    def _load_pci_ids(image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        result = []
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            result.append([
+                pci_id.vendor_id,
+                pci_id.device_id,
+                pci_id.subsystem_vendor_id,
+                pci_id.subsystem_device_id,
+                ])
+        return result
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
-- 
2.28.0

[dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

[Dmitry Kozlyuk]

Use the same interpreter to run pmdinfogen as for other build scripts.
Adjust wrapper script accordingly and also don't suppress stderr from ar
and pmdinfogen.

Add configure-time check for elftools Python module for Unix hosts.

Add python3-pyelftools to CI configuration. The package is available on
all major distributions. FreeBSD has no system requirements section in
its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE)
supported.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 .travis.yml                       |  2 +-
 buildtools/gen-pmdinfo-cfile.sh   |  6 +++---
 buildtools/meson.build            | 15 +++++++++++++++
 doc/guides/linux_gsg/sys_reqs.rst |  6 ++++++
 drivers/meson.build               |  2 +-
 meson.build                       |  1 -
 6 files changed, 26 insertions(+), 6 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 5e12db23b..6744f6fc1 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -11,7 +11,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
diff --git a/buildtools/gen-pmdinfo-cfile.sh b/buildtools/gen-pmdinfo-cfile.sh
index 43059cf36..109ee461e 100755
--- a/buildtools/gen-pmdinfo-cfile.sh
+++ b/buildtools/gen-pmdinfo-cfile.sh
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 04808dabc..dd4c0f640 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -17,3 +17,18 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
+
+# TODO: starting from Meson 0.51.0 use
+# 	python3 = import('python').find_installation('python',
+#		modules : python3_required_modules)
+python3_required_modules = []
+if host_machine.system() != 'windows'
+	python3_required_modules = ['elftools']
+endif
+foreach module : python3_required_modules
+	script = 'import importlib.util; import sys; exit(importlib.util.find_spec("@0@") is None)'
+	if run_command(py3, '-c', script.format(module)).returncode() != 0
+		error('missing python module: @0@'.format(module))
+	endif
+endforeach
diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
index 6ecdc04aa..89c9f2570 100644
--- a/doc/guides/linux_gsg/sys_reqs.rst
+++ b/doc/guides/linux_gsg/sys_reqs.rst
@@ -52,6 +52,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
diff --git a/drivers/meson.build b/drivers/meson.build
index a5a6fed06..94403f7ac 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -152,7 +152,7 @@ foreach subpath:subdirs
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
diff --git a/meson.build b/meson.build
index 61d9a4f5f..a6e8c09c2 100644
--- a/meson.build
+++ b/meson.build
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
-- 
2.28.0

[dpdk-dev] [PATCH v8 3/3] pmdinfogen: remove C implementation

[Dmitry Kozlyuk]

Delete the files no longer used in build process.
Add myself as maintainer of new implementation.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 MAINTAINERS                        |   3 +-
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 4 files changed, 2 insertions(+), 590 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

diff --git a/MAINTAINERS b/MAINTAINERS
index a3d192747..996f732fd 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -133,7 +133,8 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <nhorman@tuxdriver.com>
-F: buildtools/pmdinfogen/
+M: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
diff --git a/buildtools/pmdinfogen/meson.build b/buildtools/pmdinfogen/meson.build
deleted file mode 100644
index 670528fac..000000000
--- a/buildtools/pmdinfogen/meson.build
+++ /dev/null
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
diff --git a/buildtools/pmdinfogen/pmdinfogen.c b/buildtools/pmdinfogen/pmdinfogen.c
deleted file mode 100644
index a68d1ea99..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.c
+++ /dev/null
@@ -1,456 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <ctype.h>
-#include <string.h>
-#include <limits.h>
-#include <stdbool.h>
-#include <errno.h>
-#include <libgen.h>
-
-#include <rte_common.h>
-#include "pmdinfogen.h"
-
-#ifdef RTE_ARCH_64
-#define ADDR_SIZE 64
-#else
-#define ADDR_SIZE 32
-#endif
-
-static int use_stdin, use_stdout;
-
-static const char *sym_name(struct elf_info *elf, Elf_Sym *sym)
-{
-	if (sym)
-		return elf->strtab + sym->st_name;
-	else
-		return "(unknown)";
-}
-
-static void *grab_file(const char *filename, unsigned long *size)
-{
-	struct stat st;
-	void *map = MAP_FAILED;
-	int fd = -1;
-
-	if (!use_stdin) {
-		fd = open(filename, O_RDONLY);
-		if (fd < 0)
-			return NULL;
-	} else {
-		/* from stdin, use a temporary file to mmap */
-		FILE *infile;
-		char buffer[1024];
-		int n;
-
-		infile = tmpfile();
-		if (infile == NULL) {
-			perror("tmpfile");
-			return NULL;
-		}
-		fd = dup(fileno(infile));
-		fclose(infile);
-		if (fd < 0)
-			return NULL;
-
-		n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		while (n > 0) {
-			if (write(fd, buffer, n) != n)
-				goto failed;
-			n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		}
-	}
-
-	if (fstat(fd, &st))
-		goto failed;
-
-	*size = st.st_size;
-	map = mmap(NULL, *size, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);
-
-failed:
-	close(fd);
-	if (map == MAP_FAILED)
-		return NULL;
-	return map;
-}
-
-/**
-  * Return a copy of the next line in a mmap'ed file.
-  * spaces in the beginning of the line is trimmed away.
-  * Return a pointer to a static buffer.
-  **/
-static void release_file(void *file, unsigned long size)
-{
-	munmap(file, size);
-}
-
-
-static void *get_sym_value(struct elf_info *info, const Elf_Sym *sym)
-{
-	return RTE_PTR_ADD(info->hdr,
-		info->sechdrs[sym->st_shndx].sh_offset + sym->st_value);
-}
-
-static Elf_Sym *find_sym_in_symtab(struct elf_info *info,
-				   const char *name, Elf_Sym *last)
-{
-	Elf_Sym *idx;
-	if (last)
-		idx = last+1;
-	else
-		idx = info->symtab_start;
-
-	for (; idx < info->symtab_stop; idx++) {
-		const char *n = sym_name(info, idx);
-		if (!strncmp(n, name, strlen(name)))
-			return idx;
-	}
-	return NULL;
-}
-
-static int parse_elf(struct elf_info *info, const char *filename)
-{
-	unsigned int i;
-	Elf_Ehdr *hdr;
-	Elf_Shdr *sechdrs;
-	Elf_Sym  *sym;
-	int endian;
-	unsigned int symtab_idx = ~0U, symtab_shndx_idx = ~0U;
-
-	hdr = grab_file(filename, &info->size);
-	if (!hdr) {
-		perror(filename);
-		exit(1);
-	}
-	info->hdr = hdr;
-	if (info->size < sizeof(*hdr)) {
-		/* file too small, assume this is an empty .o file */
-		return 0;
-	}
-	/* Is this a valid ELF file? */
-	if ((hdr->e_ident[EI_MAG0] != ELFMAG0) ||
-	    (hdr->e_ident[EI_MAG1] != ELFMAG1) ||
-	    (hdr->e_ident[EI_MAG2] != ELFMAG2) ||
-	    (hdr->e_ident[EI_MAG3] != ELFMAG3)) {
-		/* Not an ELF file - silently ignore it */
-		return 0;
-	}
-
-	if (!hdr->e_ident[EI_DATA]) {
-		/* Unknown endian */
-		return 0;
-	}
-
-	endian = hdr->e_ident[EI_DATA];
-
-	/* Fix endianness in ELF header */
-	hdr->e_type      = TO_NATIVE(endian, 16, hdr->e_type);
-	hdr->e_machine   = TO_NATIVE(endian, 16, hdr->e_machine);
-	hdr->e_version   = TO_NATIVE(endian, 32, hdr->e_version);
-	hdr->e_entry     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_entry);
-	hdr->e_phoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_phoff);
-	hdr->e_shoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_shoff);
-	hdr->e_flags     = TO_NATIVE(endian, 32, hdr->e_flags);
-	hdr->e_ehsize    = TO_NATIVE(endian, 16, hdr->e_ehsize);
-	hdr->e_phentsize = TO_NATIVE(endian, 16, hdr->e_phentsize);
-	hdr->e_phnum     = TO_NATIVE(endian, 16, hdr->e_phnum);
-	hdr->e_shentsize = TO_NATIVE(endian, 16, hdr->e_shentsize);
-	hdr->e_shnum     = TO_NATIVE(endian, 16, hdr->e_shnum);
-	hdr->e_shstrndx  = TO_NATIVE(endian, 16, hdr->e_shstrndx);
-
-	sechdrs = RTE_PTR_ADD(hdr, hdr->e_shoff);
-	info->sechdrs = sechdrs;
-
-	/* Check if file offset is correct */
-	if (hdr->e_shoff > info->size) {
-		fprintf(stderr, "section header offset=%lu in file '%s' "
-		      "is bigger than filesize=%lu\n",
-		      (unsigned long)hdr->e_shoff,
-		      filename, info->size);
-		return 0;
-	}
-
-	if (hdr->e_shnum == SHN_UNDEF) {
-		/*
-		 * There are more than 64k sections,
-		 * read count from .sh_size.
-		 */
-		info->num_sections =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[0].sh_size);
-	} else {
-		info->num_sections = hdr->e_shnum;
-	}
-	if (hdr->e_shstrndx == SHN_XINDEX)
-		info->secindex_strings =
-			TO_NATIVE(endian, 32, sechdrs[0].sh_link);
-	else
-		info->secindex_strings = hdr->e_shstrndx;
-
-	/* Fix endianness in section headers */
-	for (i = 0; i < info->num_sections; i++) {
-		sechdrs[i].sh_name      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_name);
-		sechdrs[i].sh_type      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_type);
-		sechdrs[i].sh_flags     =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_flags);
-		sechdrs[i].sh_addr      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addr);
-		sechdrs[i].sh_offset    =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_offset);
-		sechdrs[i].sh_size      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_size);
-		sechdrs[i].sh_link      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_link);
-		sechdrs[i].sh_info      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_info);
-		sechdrs[i].sh_addralign =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addralign);
-		sechdrs[i].sh_entsize   =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_entsize);
-	}
-	/* Find symbol table. */
-	for (i = 1; i < info->num_sections; i++) {
-		int nobits = sechdrs[i].sh_type == SHT_NOBITS;
-
-		if (!nobits && sechdrs[i].sh_offset > info->size) {
-			fprintf(stderr, "%s is truncated. "
-			      "sechdrs[i].sh_offset=%lu > sizeof(*hrd)=%zu\n",
-			      filename, (unsigned long)sechdrs[i].sh_offset,
-			      sizeof(*hdr));
-			return 0;
-		}
-
-		if (sechdrs[i].sh_type == SHT_SYMTAB) {
-			unsigned int sh_link_idx;
-			symtab_idx = i;
-			info->symtab_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-			sh_link_idx = sechdrs[i].sh_link;
-			info->strtab       = RTE_PTR_ADD(hdr,
-				sechdrs[sh_link_idx].sh_offset);
-		}
-
-		/* 32bit section no. table? ("more than 64k sections") */
-		if (sechdrs[i].sh_type == SHT_SYMTAB_SHNDX) {
-			symtab_shndx_idx = i;
-			info->symtab_shndx_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_shndx_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-		}
-	}
-	if (!info->symtab_start)
-		fprintf(stderr, "%s has no symtab?\n", filename);
-	else {
-		/* Fix endianness in symbols */
-		for (sym = info->symtab_start; sym < info->symtab_stop; sym++) {
-			sym->st_shndx = TO_NATIVE(endian, 16, sym->st_shndx);
-			sym->st_name  = TO_NATIVE(endian, 32, sym->st_name);
-			sym->st_value = TO_NATIVE(endian, ADDR_SIZE, sym->st_value);
-			sym->st_size  = TO_NATIVE(endian, ADDR_SIZE, sym->st_size);
-		}
-	}
-
-	if (symtab_shndx_idx != ~0U) {
-		Elf32_Word *p;
-		if (symtab_idx != sechdrs[symtab_shndx_idx].sh_link)
-			fprintf(stderr,
-			      "%s: SYMTAB_SHNDX has bad sh_link: %u!=%u\n",
-			      filename, sechdrs[symtab_shndx_idx].sh_link,
-			      symtab_idx);
-		/* Fix endianness */
-		for (p = info->symtab_shndx_start; p < info->symtab_shndx_stop;
-		     p++)
-			*p = TO_NATIVE(endian, 32, *p);
-	}
-
-	return 1;
-}
-
-static void parse_elf_finish(struct elf_info *info)
-{
-	struct pmd_driver *tmp, *idx = info->drivers;
-	release_file(info->hdr, info->size);
-	while (idx) {
-		tmp = idx->next;
-		free(idx);
-		idx = tmp;
-	}
-}
-
-struct opt_tag {
-	const char *suffix;
-	const char *json_id;
-};
-
-static const struct opt_tag opt_tags[] = {
-	{"_param_string_export", "params"},
-	{"_kmod_dep_export", "kmod"},
-};
-
-static int complete_pmd_entry(struct elf_info *info, struct pmd_driver *drv)
-{
-	const char *tname;
-	int i;
-	char tmpsymname[128];
-	Elf_Sym *tmpsym;
-
-	drv->name = get_sym_value(info, drv->name_sym);
-
-	for (i = 0; i < PMD_OPT_MAX; i++) {
-		memset(tmpsymname, 0, 128);
-		sprintf(tmpsymname, "__%s%s", drv->name, opt_tags[i].suffix);
-		tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-		if (!tmpsym)
-			continue;
-		drv->opt_vals[i] = get_sym_value(info, tmpsym);
-	}
-
-	memset(tmpsymname, 0, 128);
-	sprintf(tmpsymname, "__%s_pci_tbl_export", drv->name);
-
-	tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-
-
-	/*
-	 * If this returns NULL, then this is a PMD_VDEV, because
-	 * it has no pci table reference
-	 */
-	if (!tmpsym) {
-		drv->pci_tbl = NULL;
-		return 0;
-	}
-
-	tname = get_sym_value(info, tmpsym);
-	tmpsym = find_sym_in_symtab(info, tname, NULL);
-	if (!tmpsym)
-		return -ENOENT;
-
-	drv->pci_tbl = (struct rte_pci_id *)get_sym_value(info, tmpsym);
-	if (!drv->pci_tbl)
-		return -ENOENT;
-
-	return 0;
-}
-
-static int locate_pmd_entries(struct elf_info *info)
-{
-	Elf_Sym *last = NULL;
-	struct pmd_driver *new;
-
-	info->drivers = NULL;
-
-	do {
-		new = calloc(sizeof(struct pmd_driver), 1);
-		if (new == NULL) {
-			fprintf(stderr, "Failed to calloc memory\n");
-			return -1;
-		}
-		new->name_sym = find_sym_in_symtab(info, "this_pmd_name", last);
-		last = new->name_sym;
-		if (!new->name_sym)
-			free(new);
-		else {
-			if (complete_pmd_entry(info, new)) {
-				fprintf(stderr,
-					"Failed to complete pmd entry\n");
-				free(new);
-			} else {
-				new->next = info->drivers;
-				info->drivers = new;
-			}
-		}
-	} while (last);
-
-	return 0;
-}
-
-static void output_pmd_info_string(struct elf_info *info, char *outfile)
-{
-	FILE *ofd;
-	struct pmd_driver *drv;
-	struct rte_pci_id *pci_ids;
-	int idx = 0;
-
-	if (use_stdout)
-		ofd = stdout;
-	else {
-		ofd = fopen(outfile, "w+");
-		if (!ofd) {
-			fprintf(stderr, "Unable to open output file\n");
-			return;
-		}
-	}
-
-	drv = info->drivers;
-
-	while (drv) {
-		fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
-			"\"PMD_INFO_STRING= {",
-			drv->name);
-		fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);
-
-		for (idx = 0; idx < PMD_OPT_MAX; idx++) {
-			if (drv->opt_vals[idx])
-				fprintf(ofd, "\\\"%s\\\" : \\\"%s\\\", ",
-					opt_tags[idx].json_id,
-					drv->opt_vals[idx]);
-		}
-
-		pci_ids = drv->pci_tbl;
-		fprintf(ofd, "\\\"pci_ids\\\" : [");
-
-		while (pci_ids && pci_ids->device_id) {
-			fprintf(ofd, "[%d, %d, %d, %d]",
-				pci_ids->vendor_id, pci_ids->device_id,
-				pci_ids->subsystem_vendor_id,
-				pci_ids->subsystem_device_id);
-			pci_ids++;
-			if (pci_ids->device_id)
-				fprintf(ofd, ",");
-			else
-				fprintf(ofd, " ");
-		}
-		fprintf(ofd, "]}\";\n");
-		drv = drv->next;
-	}
-
-	fclose(ofd);
-}
-
-int main(int argc, char **argv)
-{
-	struct elf_info info = {0};
-	int rc = 1;
-
-	if (argc < 3) {
-		fprintf(stderr,
-			"usage: %s <object file> <c output file>\n",
-			basename(argv[0]));
-		exit(127);
-	}
-	use_stdin = !strcmp(argv[1], "-");
-	use_stdout = !strcmp(argv[2], "-");
-	parse_elf(&info, argv[1]);
-
-	if (locate_pmd_entries(&info) < 0)
-		exit(1);
-
-	if (info.drivers) {
-		output_pmd_info_string(&info, argv[2]);
-		rc = 0;
-	} else {
-		fprintf(stderr, "No drivers registered\n");
-	}
-
-	parse_elf_finish(&info);
-	exit(rc);
-}
diff --git a/buildtools/pmdinfogen/pmdinfogen.h b/buildtools/pmdinfogen/pmdinfogen.h
deleted file mode 100644
index 38fd3ddcc..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.h
+++ /dev/null
@@ -1,119 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <stdarg.h>
-#include <string.h>
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#ifdef __linux__
-#include <endian.h>
-#else
-#include <sys/endian.h>
-#endif
-#include <fcntl.h>
-#include <unistd.h>
-#include <elf.h>
-#include <rte_pci.h>
-
-/* On BSD-alike OSes elf.h defines these according to host's word size */
-#undef ELF_ST_BIND
-#undef ELF_ST_TYPE
-#undef ELF_R_SYM
-#undef ELF_R_TYPE
-
-/*
- * Define ELF64_* to ELF_*, the latter being defined in both 32 and 64 bit
- * flavors in elf.h.  This makes our code a bit more generic between arches
- * and allows us to support 32 bit code in the future should we ever want to
- */
-#ifdef RTE_ARCH_64
-#define Elf_Ehdr    Elf64_Ehdr
-#define Elf_Shdr    Elf64_Shdr
-#define Elf_Sym     Elf64_Sym
-#define Elf_Addr    Elf64_Addr
-#define Elf_Sword   Elf64_Sxword
-#define Elf_Section Elf64_Half
-#define ELF_ST_BIND ELF64_ST_BIND
-#define ELF_ST_TYPE ELF64_ST_TYPE
-
-#define Elf_Rel     Elf64_Rel
-#define Elf_Rela    Elf64_Rela
-#define ELF_R_SYM   ELF64_R_SYM
-#define ELF_R_TYPE  ELF64_R_TYPE
-#else
-#define Elf_Ehdr    Elf32_Ehdr
-#define Elf_Shdr    Elf32_Shdr
-#define Elf_Sym     Elf32_Sym
-#define Elf_Addr    Elf32_Addr
-#define Elf_Sword   Elf32_Sxword
-#define Elf_Section Elf32_Half
-#define ELF_ST_BIND ELF32_ST_BIND
-#define ELF_ST_TYPE ELF32_ST_TYPE
-
-#define Elf_Rel     Elf32_Rel
-#define Elf_Rela    Elf32_Rela
-#define ELF_R_SYM   ELF32_R_SYM
-#define ELF_R_TYPE  ELF32_R_TYPE
-#endif
-
-
-/*
- * Note, it seems odd that we have both a CONVERT_NATIVE and a TO_NATIVE macro
- * below.  We do this because the values passed to TO_NATIVE may themselves be
- * macros and need both macros here to get expanded.  Specifically its the width
- * variable we are concerned with, because it needs to get expanded prior to
- * string concatenation
- */
-#define CONVERT_NATIVE(fend, width, x) ({ \
-typeof(x) ___x; \
-if ((fend) == ELFDATA2LSB) \
-	___x = le##width##toh(x); \
-else \
-	___x = be##width##toh(x); \
-___x; \
-})
-
-#define TO_NATIVE(fend, width, x) CONVERT_NATIVE(fend, width, x)
-
-enum opt_params {
-	PMD_PARAM_STRING = 0,
-	PMD_KMOD_DEP,
-	PMD_OPT_MAX
-};
-
-struct pmd_driver {
-	Elf_Sym *name_sym;
-	const char *name;
-	struct rte_pci_id *pci_tbl;
-	struct pmd_driver *next;
-
-	const char *opt_vals[PMD_OPT_MAX];
-};
-
-struct elf_info {
-	unsigned long size;
-	Elf_Ehdr     *hdr;
-	Elf_Shdr     *sechdrs;
-	Elf_Sym      *symtab_start;
-	Elf_Sym      *symtab_stop;
-	char         *strtab;
-
-	/* support for 32bit section numbers */
-
-	unsigned int num_sections; /* max_secindex + 1 */
-	unsigned int secindex_strings;
-	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
-	 * take shndx from symtab_shndx_start[N] instead
-	 */
-	Elf32_Word   *symtab_shndx_start;
-	Elf32_Word   *symtab_shndx_stop;
-
-	struct pmd_driver *drivers;
-};
-- 
2.28.0

Re: [dpdk-dev] [PATCH v7 0/3] pmdinfogen: rewrite in Python

[Dmitry Kozlyuk]

Hello David,

> - Travis reported an error in meson:
> https://travis-ci.com/github/ovsrobot/dpdk/jobs/399762460#L478
> 
> Build machine cpu: x86_64
> buildtools/meson.build:27:10: ERROR:  Expecting endforeach got eol_cont.
> script = \
>           ^
> For a block that started at 26,0
> foreach module : python3_required_modules
> ^

My bad, meson 0.47.1 doesn't understand this indeed, fixed in v8.

> - In a thread on ci@dpdk.org
> http://inbox.dpdk.org/ci/CAOeXdvbosqRGouSX06_phVTCam57XUt0s=sf_kpe-VmiwFTgMA@mail.gmail.com/T/#m9bb193c74ca979c151f2dbfa35664f95093a9e71,
> UNH people confirmed that python3-elftools should be available, but
> the last test had an error too, which might be linked to incorrect
> meson syntax:
> http://mails.dpdk.org/archives/test-report/2020-October/159038.html
> 
> ninja: error: loading 'build.ninja': No such file or directory

Makes sense.

> - As far as Intel CI systems are concerned, there might be an issue
> with the Windows system:
> http://mails.dpdk.org/archives/test-report/2020-October/159123.html
>
> OS: WIN10-64
> Target: x86_64-windows-clang
> Program binutils-avx512-check.sh found: YES
> Program python3 found: YES (c:\program files\python37\python.exe)
> 
> buildtools\meson.build:32:2: ERROR: Problem encountered: missing
> python module: elftools

This is the check added in the last revision per Maxime's suggestion.
Disabled the check on Windows in v8 until pmdinfogen supports COFF (and this
will probably introduce an alternative dependency).

I'll watch new CI results, haven't been doing so closely because failures were
expected for these patches. Tested v8 build on Linux->Linux, Linux->Windows,
Windows->Windows (GCC & clang, without pyelftools installed).

Re: [dpdk-dev] [PATCH v7 0/3] pmdinfogen: rewrite in Python

[Dmitry Kozlyuk]

Hi IOL team,

The following error persists in v8:

> - In a thread on ci@dpdk.org
> http://inbox.dpdk.org/ci/CAOeXdvbosqRGouSX06_phVTCam57XUt0s=sf_kpe-VmiwFTgMA@mail.gmail.com/T/#m9bb193c74ca979c151f2dbfa35664f95093a9e71,
> UNH people confirmed that python3-elftools should be available, but
> the last test had an error too, which might be linked to incorrect
> meson syntax:
> http://mails.dpdk.org/archives/test-report/2020-October/159038.html
> 
> ninja: error: loading 'build.ninja': No such file or directory

https://inbox.dpdk.org/test-report/20201020181426.1BC8788FC9@noxus.dpdklab.iol.unh.edu/T/#u

The message isn't very informative. I suspect meson failed to configure,
but there's no way to tell what went wrong. How can I get more information
about this failure or specification of your environment to reproduce it?

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

[Bruce Richardson]

On Tue, Oct 20, 2020 at 08:44:02PM +0300, Dmitry Kozlyuk wrote:
> Use the same interpreter to run pmdinfogen as for other build scripts.
> Adjust wrapper script accordingly and also don't suppress stderr from ar
> and pmdinfogen.
> 
> Add configure-time check for elftools Python module for Unix hosts.
> 
> Add python3-pyelftools to CI configuration. The package is available on
> all major distributions. FreeBSD has no system requirements section in
> its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE)
> supported.
> 
> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> ---

From a meson viewpoint, this all looks ok. I'd hope in future that we could
merge the shell script wrapper into the python script directly, rather than
having two levels of scripting, but it's not something with any urgency.

Acked-by: Bruce Richardson <bruce.richardson@intel.com>

>  .travis.yml                       |  2 +-
>  buildtools/gen-pmdinfo-cfile.sh   |  6 +++---
>  buildtools/meson.build            | 15 +++++++++++++++
>  doc/guides/linux_gsg/sys_reqs.rst |  6 ++++++
>  drivers/meson.build               |  2 +-
>  meson.build                       |  1 -
>  6 files changed, 26 insertions(+), 6 deletions(-)
> 

Re: [dpdk-dev] [PATCH v8 3/3] pmdinfogen: remove C implementation

[Jie Zhou]

Tested-by: Jie Zhou <jizh@linux.microsoft.com>

On Tue, Oct 20, 2020 at 08:44:03PM +0300, Dmitry Kozlyuk wrote:
> Delete the files no longer used in build process.
> Add myself as maintainer of new implementation.
> 
> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> ---
>  MAINTAINERS                        |   3 +-
>  buildtools/pmdinfogen/meson.build  |  14 -
>  buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
>  buildtools/pmdinfogen/pmdinfogen.h | 119 --------
>  4 files changed, 2 insertions(+), 590 deletions(-)
>  delete mode 100644 buildtools/pmdinfogen/meson.build
>  delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
>  delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index a3d192747..996f732fd 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -133,7 +133,8 @@ F: lib/*/*.map
>  
>  Driver information
>  M: Neil Horman <nhorman@tuxdriver.com>
> -F: buildtools/pmdinfogen/
> +M: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> +F: buildtools/pmdinfogen.py
>  F: usertools/dpdk-pmdinfo.py
>  F: doc/guides/tools/pmdinfo.rst
>  
> diff --git a/buildtools/pmdinfogen/meson.build b/buildtools/pmdinfogen/meson.build
> deleted file mode 100644
> index 670528fac..000000000
> --- a/buildtools/pmdinfogen/meson.build
> +++ /dev/null
> @@ -1,14 +0,0 @@
> -# SPDX-License-Identifier: BSD-3-Clause
> -# Copyright(c) 2017 Intel Corporation
> -
> -if is_windows
> -	subdir_done()
> -endif
> -
> -pmdinfogen_inc = [global_inc]
> -pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
> -pmdinfogen_inc += include_directories('../../lib/librte_pci')
> -pmdinfogen = executable('pmdinfogen',
> -	'pmdinfogen.c',
> -	include_directories: pmdinfogen_inc,
> -	native: true)
> diff --git a/buildtools/pmdinfogen/pmdinfogen.c b/buildtools/pmdinfogen/pmdinfogen.c
> deleted file mode 100644
> index a68d1ea99..000000000
> --- a/buildtools/pmdinfogen/pmdinfogen.c
> +++ /dev/null
> @@ -1,456 +0,0 @@
> -/* SPDX-License-Identifier: GPL-2.0
> - * Postprocess pmd object files to export hw support
> - *
> - * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
> - * Based in part on modpost.c from the linux kernel
> - */
> -
> -#include <stdio.h>
> -#include <ctype.h>
> -#include <string.h>
> -#include <limits.h>
> -#include <stdbool.h>
> -#include <errno.h>
> -#include <libgen.h>
> -
> -#include <rte_common.h>
> -#include "pmdinfogen.h"
> -
> -#ifdef RTE_ARCH_64
> -#define ADDR_SIZE 64
> -#else
> -#define ADDR_SIZE 32
> -#endif
> -
> -static int use_stdin, use_stdout;
> -
> -static const char *sym_name(struct elf_info *elf, Elf_Sym *sym)
> -{
> -	if (sym)
> -		return elf->strtab + sym->st_name;
> -	else
> -		return "(unknown)";
> -}
> -
> -static void *grab_file(const char *filename, unsigned long *size)
> -{
> -	struct stat st;
> -	void *map = MAP_FAILED;
> -	int fd = -1;
> -
> -	if (!use_stdin) {
> -		fd = open(filename, O_RDONLY);
> -		if (fd < 0)
> -			return NULL;
> -	} else {
> -		/* from stdin, use a temporary file to mmap */
> -		FILE *infile;
> -		char buffer[1024];
> -		int n;
> -
> -		infile = tmpfile();
> -		if (infile == NULL) {
> -			perror("tmpfile");
> -			return NULL;
> -		}
> -		fd = dup(fileno(infile));
> -		fclose(infile);
> -		if (fd < 0)
> -			return NULL;
> -
> -		n = read(STDIN_FILENO, buffer, sizeof(buffer));
> -		while (n > 0) {
> -			if (write(fd, buffer, n) != n)
> -				goto failed;
> -			n = read(STDIN_FILENO, buffer, sizeof(buffer));
> -		}
> -	}
> -
> -	if (fstat(fd, &st))
> -		goto failed;
> -
> -	*size = st.st_size;
> -	map = mmap(NULL, *size, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);
> -
> -failed:
> -	close(fd);
> -	if (map == MAP_FAILED)
> -		return NULL;
> -	return map;
> -}
> -
> -/**
> -  * Return a copy of the next line in a mmap'ed file.
> -  * spaces in the beginning of the line is trimmed away.
> -  * Return a pointer to a static buffer.
> -  **/
> -static void release_file(void *file, unsigned long size)
> -{
> -	munmap(file, size);
> -}
> -
> -
> -static void *get_sym_value(struct elf_info *info, const Elf_Sym *sym)
> -{
> -	return RTE_PTR_ADD(info->hdr,
> -		info->sechdrs[sym->st_shndx].sh_offset + sym->st_value);
> -}
> -
> -static Elf_Sym *find_sym_in_symtab(struct elf_info *info,
> -				   const char *name, Elf_Sym *last)
> -{
> -	Elf_Sym *idx;
> -	if (last)
> -		idx = last+1;
> -	else
> -		idx = info->symtab_start;
> -
> -	for (; idx < info->symtab_stop; idx++) {
> -		const char *n = sym_name(info, idx);
> -		if (!strncmp(n, name, strlen(name)))
> -			return idx;
> -	}
> -	return NULL;
> -}
> -
> -static int parse_elf(struct elf_info *info, const char *filename)
> -{
> -	unsigned int i;
> -	Elf_Ehdr *hdr;
> -	Elf_Shdr *sechdrs;
> -	Elf_Sym  *sym;
> -	int endian;
> -	unsigned int symtab_idx = ~0U, symtab_shndx_idx = ~0U;
> -
> -	hdr = grab_file(filename, &info->size);
> -	if (!hdr) {
> -		perror(filename);
> -		exit(1);
> -	}
> -	info->hdr = hdr;
> -	if (info->size < sizeof(*hdr)) {
> -		/* file too small, assume this is an empty .o file */
> -		return 0;
> -	}
> -	/* Is this a valid ELF file? */
> -	if ((hdr->e_ident[EI_MAG0] != ELFMAG0) ||
> -	    (hdr->e_ident[EI_MAG1] != ELFMAG1) ||
> -	    (hdr->e_ident[EI_MAG2] != ELFMAG2) ||
> -	    (hdr->e_ident[EI_MAG3] != ELFMAG3)) {
> -		/* Not an ELF file - silently ignore it */
> -		return 0;
> -	}
> -
> -	if (!hdr->e_ident[EI_DATA]) {
> -		/* Unknown endian */
> -		return 0;
> -	}
> -
> -	endian = hdr->e_ident[EI_DATA];
> -
> -	/* Fix endianness in ELF header */
> -	hdr->e_type      = TO_NATIVE(endian, 16, hdr->e_type);
> -	hdr->e_machine   = TO_NATIVE(endian, 16, hdr->e_machine);
> -	hdr->e_version   = TO_NATIVE(endian, 32, hdr->e_version);
> -	hdr->e_entry     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_entry);
> -	hdr->e_phoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_phoff);
> -	hdr->e_shoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_shoff);
> -	hdr->e_flags     = TO_NATIVE(endian, 32, hdr->e_flags);
> -	hdr->e_ehsize    = TO_NATIVE(endian, 16, hdr->e_ehsize);
> -	hdr->e_phentsize = TO_NATIVE(endian, 16, hdr->e_phentsize);
> -	hdr->e_phnum     = TO_NATIVE(endian, 16, hdr->e_phnum);
> -	hdr->e_shentsize = TO_NATIVE(endian, 16, hdr->e_shentsize);
> -	hdr->e_shnum     = TO_NATIVE(endian, 16, hdr->e_shnum);
> -	hdr->e_shstrndx  = TO_NATIVE(endian, 16, hdr->e_shstrndx);
> -
> -	sechdrs = RTE_PTR_ADD(hdr, hdr->e_shoff);
> -	info->sechdrs = sechdrs;
> -
> -	/* Check if file offset is correct */
> -	if (hdr->e_shoff > info->size) {
> -		fprintf(stderr, "section header offset=%lu in file '%s' "
> -		      "is bigger than filesize=%lu\n",
> -		      (unsigned long)hdr->e_shoff,
> -		      filename, info->size);
> -		return 0;
> -	}
> -
> -	if (hdr->e_shnum == SHN_UNDEF) {
> -		/*
> -		 * There are more than 64k sections,
> -		 * read count from .sh_size.
> -		 */
> -		info->num_sections =
> -			TO_NATIVE(endian, ADDR_SIZE, sechdrs[0].sh_size);
> -	} else {
> -		info->num_sections = hdr->e_shnum;
> -	}
> -	if (hdr->e_shstrndx == SHN_XINDEX)
> -		info->secindex_strings =
> -			TO_NATIVE(endian, 32, sechdrs[0].sh_link);
> -	else
> -		info->secindex_strings = hdr->e_shstrndx;
> -
> -	/* Fix endianness in section headers */
> -	for (i = 0; i < info->num_sections; i++) {
> -		sechdrs[i].sh_name      =
> -			TO_NATIVE(endian, 32, sechdrs[i].sh_name);
> -		sechdrs[i].sh_type      =
> -			TO_NATIVE(endian, 32, sechdrs[i].sh_type);
> -		sechdrs[i].sh_flags     =
> -			TO_NATIVE(endian, 32, sechdrs[i].sh_flags);
> -		sechdrs[i].sh_addr      =
> -			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addr);
> -		sechdrs[i].sh_offset    =
> -			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_offset);
> -		sechdrs[i].sh_size      =
> -			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_size);
> -		sechdrs[i].sh_link      =
> -			TO_NATIVE(endian, 32, sechdrs[i].sh_link);
> -		sechdrs[i].sh_info      =
> -			TO_NATIVE(endian, 32, sechdrs[i].sh_info);
> -		sechdrs[i].sh_addralign =
> -			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addralign);
> -		sechdrs[i].sh_entsize   =
> -			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_entsize);
> -	}
> -	/* Find symbol table. */
> -	for (i = 1; i < info->num_sections; i++) {
> -		int nobits = sechdrs[i].sh_type == SHT_NOBITS;
> -
> -		if (!nobits && sechdrs[i].sh_offset > info->size) {
> -			fprintf(stderr, "%s is truncated. "
> -			      "sechdrs[i].sh_offset=%lu > sizeof(*hrd)=%zu\n",
> -			      filename, (unsigned long)sechdrs[i].sh_offset,
> -			      sizeof(*hdr));
> -			return 0;
> -		}
> -
> -		if (sechdrs[i].sh_type == SHT_SYMTAB) {
> -			unsigned int sh_link_idx;
> -			symtab_idx = i;
> -			info->symtab_start = RTE_PTR_ADD(hdr,
> -				sechdrs[i].sh_offset);
> -			info->symtab_stop  = RTE_PTR_ADD(hdr,
> -				sechdrs[i].sh_offset + sechdrs[i].sh_size);
> -			sh_link_idx = sechdrs[i].sh_link;
> -			info->strtab       = RTE_PTR_ADD(hdr,
> -				sechdrs[sh_link_idx].sh_offset);
> -		}
> -
> -		/* 32bit section no. table? ("more than 64k sections") */
> -		if (sechdrs[i].sh_type == SHT_SYMTAB_SHNDX) {
> -			symtab_shndx_idx = i;
> -			info->symtab_shndx_start = RTE_PTR_ADD(hdr,
> -				sechdrs[i].sh_offset);
> -			info->symtab_shndx_stop  = RTE_PTR_ADD(hdr,
> -				sechdrs[i].sh_offset + sechdrs[i].sh_size);
> -		}
> -	}
> -	if (!info->symtab_start)
> -		fprintf(stderr, "%s has no symtab?\n", filename);
> -	else {
> -		/* Fix endianness in symbols */
> -		for (sym = info->symtab_start; sym < info->symtab_stop; sym++) {
> -			sym->st_shndx = TO_NATIVE(endian, 16, sym->st_shndx);
> -			sym->st_name  = TO_NATIVE(endian, 32, sym->st_name);
> -			sym->st_value = TO_NATIVE(endian, ADDR_SIZE, sym->st_value);
> -			sym->st_size  = TO_NATIVE(endian, ADDR_SIZE, sym->st_size);
> -		}
> -	}
> -
> -	if (symtab_shndx_idx != ~0U) {
> -		Elf32_Word *p;
> -		if (symtab_idx != sechdrs[symtab_shndx_idx].sh_link)
> -			fprintf(stderr,
> -			      "%s: SYMTAB_SHNDX has bad sh_link: %u!=%u\n",
> -			      filename, sechdrs[symtab_shndx_idx].sh_link,
> -			      symtab_idx);
> -		/* Fix endianness */
> -		for (p = info->symtab_shndx_start; p < info->symtab_shndx_stop;
> -		     p++)
> -			*p = TO_NATIVE(endian, 32, *p);
> -	}
> -
> -	return 1;
> -}
> -
> -static void parse_elf_finish(struct elf_info *info)
> -{
> -	struct pmd_driver *tmp, *idx = info->drivers;
> -	release_file(info->hdr, info->size);
> -	while (idx) {
> -		tmp = idx->next;
> -		free(idx);
> -		idx = tmp;
> -	}
> -}
> -
> -struct opt_tag {
> -	const char *suffix;
> -	const char *json_id;
> -};
> -
> -static const struct opt_tag opt_tags[] = {
> -	{"_param_string_export", "params"},
> -	{"_kmod_dep_export", "kmod"},
> -};
> -
> -static int complete_pmd_entry(struct elf_info *info, struct pmd_driver *drv)
> -{
> -	const char *tname;
> -	int i;
> -	char tmpsymname[128];
> -	Elf_Sym *tmpsym;
> -
> -	drv->name = get_sym_value(info, drv->name_sym);
> -
> -	for (i = 0; i < PMD_OPT_MAX; i++) {
> -		memset(tmpsymname, 0, 128);
> -		sprintf(tmpsymname, "__%s%s", drv->name, opt_tags[i].suffix);
> -		tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
> -		if (!tmpsym)
> -			continue;
> -		drv->opt_vals[i] = get_sym_value(info, tmpsym);
> -	}
> -
> -	memset(tmpsymname, 0, 128);
> -	sprintf(tmpsymname, "__%s_pci_tbl_export", drv->name);
> -
> -	tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
> -
> -
> -	/*
> -	 * If this returns NULL, then this is a PMD_VDEV, because
> -	 * it has no pci table reference
> -	 */
> -	if (!tmpsym) {
> -		drv->pci_tbl = NULL;
> -		return 0;
> -	}
> -
> -	tname = get_sym_value(info, tmpsym);
> -	tmpsym = find_sym_in_symtab(info, tname, NULL);
> -	if (!tmpsym)
> -		return -ENOENT;
> -
> -	drv->pci_tbl = (struct rte_pci_id *)get_sym_value(info, tmpsym);
> -	if (!drv->pci_tbl)
> -		return -ENOENT;
> -
> -	return 0;
> -}
> -
> -static int locate_pmd_entries(struct elf_info *info)
> -{
> -	Elf_Sym *last = NULL;
> -	struct pmd_driver *new;
> -
> -	info->drivers = NULL;
> -
> -	do {
> -		new = calloc(sizeof(struct pmd_driver), 1);
> -		if (new == NULL) {
> -			fprintf(stderr, "Failed to calloc memory\n");
> -			return -1;
> -		}
> -		new->name_sym = find_sym_in_symtab(info, "this_pmd_name", last);
> -		last = new->name_sym;
> -		if (!new->name_sym)
> -			free(new);
> -		else {
> -			if (complete_pmd_entry(info, new)) {
> -				fprintf(stderr,
> -					"Failed to complete pmd entry\n");
> -				free(new);
> -			} else {
> -				new->next = info->drivers;
> -				info->drivers = new;
> -			}
> -		}
> -	} while (last);
> -
> -	return 0;
> -}
> -
> -static void output_pmd_info_string(struct elf_info *info, char *outfile)
> -{
> -	FILE *ofd;
> -	struct pmd_driver *drv;
> -	struct rte_pci_id *pci_ids;
> -	int idx = 0;
> -
> -	if (use_stdout)
> -		ofd = stdout;
> -	else {
> -		ofd = fopen(outfile, "w+");
> -		if (!ofd) {
> -			fprintf(stderr, "Unable to open output file\n");
> -			return;
> -		}
> -	}
> -
> -	drv = info->drivers;
> -
> -	while (drv) {
> -		fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
> -			"\"PMD_INFO_STRING= {",
> -			drv->name);
> -		fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);
> -
> -		for (idx = 0; idx < PMD_OPT_MAX; idx++) {
> -			if (drv->opt_vals[idx])
> -				fprintf(ofd, "\\\"%s\\\" : \\\"%s\\\", ",
> -					opt_tags[idx].json_id,
> -					drv->opt_vals[idx]);
> -		}
> -
> -		pci_ids = drv->pci_tbl;
> -		fprintf(ofd, "\\\"pci_ids\\\" : [");
> -
> -		while (pci_ids && pci_ids->device_id) {
> -			fprintf(ofd, "[%d, %d, %d, %d]",
> -				pci_ids->vendor_id, pci_ids->device_id,
> -				pci_ids->subsystem_vendor_id,
> -				pci_ids->subsystem_device_id);
> -			pci_ids++;
> -			if (pci_ids->device_id)
> -				fprintf(ofd, ",");
> -			else
> -				fprintf(ofd, " ");
> -		}
> -		fprintf(ofd, "]}\";\n");
> -		drv = drv->next;
> -	}
> -
> -	fclose(ofd);
> -}
> -
> -int main(int argc, char **argv)
> -{
> -	struct elf_info info = {0};
> -	int rc = 1;
> -
> -	if (argc < 3) {
> -		fprintf(stderr,
> -			"usage: %s <object file> <c output file>\n",
> -			basename(argv[0]));
> -		exit(127);
> -	}
> -	use_stdin = !strcmp(argv[1], "-");
> -	use_stdout = !strcmp(argv[2], "-");
> -	parse_elf(&info, argv[1]);
> -
> -	if (locate_pmd_entries(&info) < 0)
> -		exit(1);
> -
> -	if (info.drivers) {
> -		output_pmd_info_string(&info, argv[2]);
> -		rc = 0;
> -	} else {
> -		fprintf(stderr, "No drivers registered\n");
> -	}
> -
> -	parse_elf_finish(&info);
> -	exit(rc);
> -}
> diff --git a/buildtools/pmdinfogen/pmdinfogen.h b/buildtools/pmdinfogen/pmdinfogen.h
> deleted file mode 100644
> index 38fd3ddcc..000000000
> --- a/buildtools/pmdinfogen/pmdinfogen.h
> +++ /dev/null
> @@ -1,119 +0,0 @@
> -/* SPDX-License-Identifier: GPL-2.0
> - * Postprocess pmd object files to export hw support
> - *
> - * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
> - * Based in part on modpost.c from the linux kernel
> - */
> -
> -#include <stdio.h>
> -#include <stdlib.h>
> -#include <stdarg.h>
> -#include <string.h>
> -#include <sys/types.h>
> -#include <sys/stat.h>
> -#include <sys/mman.h>
> -#ifdef __linux__
> -#include <endian.h>
> -#else
> -#include <sys/endian.h>
> -#endif
> -#include <fcntl.h>
> -#include <unistd.h>
> -#include <elf.h>
> -#include <rte_pci.h>
> -
> -/* On BSD-alike OSes elf.h defines these according to host's word size */
> -#undef ELF_ST_BIND
> -#undef ELF_ST_TYPE
> -#undef ELF_R_SYM
> -#undef ELF_R_TYPE
> -
> -/*
> - * Define ELF64_* to ELF_*, the latter being defined in both 32 and 64 bit
> - * flavors in elf.h.  This makes our code a bit more generic between arches
> - * and allows us to support 32 bit code in the future should we ever want to
> - */
> -#ifdef RTE_ARCH_64
> -#define Elf_Ehdr    Elf64_Ehdr
> -#define Elf_Shdr    Elf64_Shdr
> -#define Elf_Sym     Elf64_Sym
> -#define Elf_Addr    Elf64_Addr
> -#define Elf_Sword   Elf64_Sxword
> -#define Elf_Section Elf64_Half
> -#define ELF_ST_BIND ELF64_ST_BIND
> -#define ELF_ST_TYPE ELF64_ST_TYPE
> -
> -#define Elf_Rel     Elf64_Rel
> -#define Elf_Rela    Elf64_Rela
> -#define ELF_R_SYM   ELF64_R_SYM
> -#define ELF_R_TYPE  ELF64_R_TYPE
> -#else
> -#define Elf_Ehdr    Elf32_Ehdr
> -#define Elf_Shdr    Elf32_Shdr
> -#define Elf_Sym     Elf32_Sym
> -#define Elf_Addr    Elf32_Addr
> -#define Elf_Sword   Elf32_Sxword
> -#define Elf_Section Elf32_Half
> -#define ELF_ST_BIND ELF32_ST_BIND
> -#define ELF_ST_TYPE ELF32_ST_TYPE
> -
> -#define Elf_Rel     Elf32_Rel
> -#define Elf_Rela    Elf32_Rela
> -#define ELF_R_SYM   ELF32_R_SYM
> -#define ELF_R_TYPE  ELF32_R_TYPE
> -#endif
> -
> -
> -/*
> - * Note, it seems odd that we have both a CONVERT_NATIVE and a TO_NATIVE macro
> - * below.  We do this because the values passed to TO_NATIVE may themselves be
> - * macros and need both macros here to get expanded.  Specifically its the width
> - * variable we are concerned with, because it needs to get expanded prior to
> - * string concatenation
> - */
> -#define CONVERT_NATIVE(fend, width, x) ({ \
> -typeof(x) ___x; \
> -if ((fend) == ELFDATA2LSB) \
> -	___x = le##width##toh(x); \
> -else \
> -	___x = be##width##toh(x); \
> -___x; \
> -})
> -
> -#define TO_NATIVE(fend, width, x) CONVERT_NATIVE(fend, width, x)
> -
> -enum opt_params {
> -	PMD_PARAM_STRING = 0,
> -	PMD_KMOD_DEP,
> -	PMD_OPT_MAX
> -};
> -
> -struct pmd_driver {
> -	Elf_Sym *name_sym;
> -	const char *name;
> -	struct rte_pci_id *pci_tbl;
> -	struct pmd_driver *next;
> -
> -	const char *opt_vals[PMD_OPT_MAX];
> -};
> -
> -struct elf_info {
> -	unsigned long size;
> -	Elf_Ehdr     *hdr;
> -	Elf_Shdr     *sechdrs;
> -	Elf_Sym      *symtab_start;
> -	Elf_Sym      *symtab_stop;
> -	char         *strtab;
> -
> -	/* support for 32bit section numbers */
> -
> -	unsigned int num_sections; /* max_secindex + 1 */
> -	unsigned int secindex_strings;
> -	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
> -	 * take shndx from symtab_shndx_start[N] instead
> -	 */
> -	Elf32_Word   *symtab_shndx_start;
> -	Elf32_Word   *symtab_shndx_stop;
> -
> -	struct pmd_driver *drivers;
> -};
> -- 
> 2.28.0

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

[Thomas Monjalon]

This is now the right timeframe to introduce this change
with the new Python module dependency.
Unfortunately, the ABI check is returning an issue:

'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c


Few more comments below:

20/10/2020 19:44, Dmitry Kozlyuk:
> --- a/buildtools/meson.build
> +++ b/buildtools/meson.build
> +if host_machine.system() != 'windows'

You can use "is_windows".


> --- a/doc/guides/linux_gsg/sys_reqs.rst
> +++ b/doc/guides/linux_gsg/sys_reqs.rst
> +*   ``pyelftools`` (version 0.22+)

This requirement is missing in doc/guides/freebsd_gsg/build_dpdk.rst


> --- a/meson.build
> +++ b/meson.build
> -subdir('buildtools/pmdinfogen')

This could be in patch 3 (removing the code).

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

[Dmitry Kozlyuk]

On Wed, 20 Jan 2021 01:05:59 +0100, Thomas Monjalon wrote:
> This is now the right timeframe to introduce this change
> with the new Python module dependency.
> Unfortunately, the ABI check is returning an issue:
> 
> 'const char mlx5_common_pci_pmd_info[62]' was changed
> to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c

Will investigate and fix ASAP.
 
> Few more comments below:
> 
> 20/10/2020 19:44, Dmitry Kozlyuk:
> > --- a/buildtools/meson.build
> > +++ b/buildtools/meson.build
> > +if host_machine.system() != 'windows'  
> 
> You can use "is_windows".

It's defined by config/meson.build, which is processed after
buidtools/meson.build, because of the dependency, if swapped:

	config/x86/meson.build:6:1: ERROR: Unknown variable
	"binutils_avx512_check".

> > --- a/doc/guides/linux_gsg/sys_reqs.rst
> > +++ b/doc/guides/linux_gsg/sys_reqs.rst
> > +*   ``pyelftools`` (version 0.22+)  
> 
> This requirement is missing in doc/guides/freebsd_gsg/build_dpdk.rst

OK.

> > --- a/meson.build
> > +++ b/meson.build
> > -subdir('buildtools/pmdinfogen')  
> 
> This could be in patch 3 (removing the code).

It would redefine "pmdinfogen" variable to old pmdinfogen.
Besides, why build what's not used at this patch already?

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

[Thomas Monjalon]

20/01/2021 08:23, Dmitry Kozlyuk:
> On Wed, 20 Jan 2021 01:05:59 +0100, Thomas Monjalon wrote:
> > This is now the right timeframe to introduce this change
> > with the new Python module dependency.
> > Unfortunately, the ABI check is returning an issue:
> > 
> > 'const char mlx5_common_pci_pmd_info[62]' was changed
> > to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c
> 
> Will investigate and fix ASAP.
>  
> > Few more comments below:
> > 
> > 20/10/2020 19:44, Dmitry Kozlyuk:
> > > --- a/buildtools/meson.build
> > > +++ b/buildtools/meson.build
> > > +if host_machine.system() != 'windows'  
> > 
> > You can use "is_windows".
> 
> It's defined by config/meson.build, which is processed after
> buidtools/meson.build, because of the dependency, if swapped:
> 
> 	config/x86/meson.build:6:1: ERROR: Unknown variable
> 	"binutils_avx512_check".

OK

> > > --- a/doc/guides/linux_gsg/sys_reqs.rst
> > > +++ b/doc/guides/linux_gsg/sys_reqs.rst
> > > +*   ``pyelftools`` (version 0.22+)  
> > 
> > This requirement is missing in doc/guides/freebsd_gsg/build_dpdk.rst
> 
> OK.
> 
> > > --- a/meson.build
> > > +++ b/meson.build
> > > -subdir('buildtools/pmdinfogen')  
> > 
> > This could be in patch 3 (removing the code).
> 
> It would redefine "pmdinfogen" variable to old pmdinfogen.
> Besides, why build what's not used at this patch already?

Just trying to find the best patch split.
If needed, OK to keep as is.

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

[Dmitry Kozlyuk]

On Wed, 20 Jan 2021 11:24:21 +0100, Thomas Monjalon wrote:
> 20/01/2021 08:23, Dmitry Kozlyuk:
> > On Wed, 20 Jan 2021 01:05:59 +0100, Thomas Monjalon wrote:  
> > > This is now the right timeframe to introduce this change
> > > with the new Python module dependency.
> > > Unfortunately, the ABI check is returning an issue:
> > > 
> > > 'const char mlx5_common_pci_pmd_info[62]' was changed
> > > to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c  
> > 
> > Will investigate and fix ASAP.

Now that I think of it: strings like this change every time new PCI IDs are
added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage,
is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID")
added 2020-07-08, i.e. clearly outside of ABI change window.

"xxx_pmd_info" changes are due to JSON formatting (new is more canonical),
which can be worked around easily, if the above is wrong.

> > > > --- a/meson.build
> > > > +++ b/meson.build
> > > > -subdir('buildtools/pmdinfogen')    
> > > 
> > > This could be in patch 3 (removing the code).  
> > 
> > It would redefine "pmdinfogen" variable to old pmdinfogen.
> > Besides, why build what's not used at this patch already?  
> 
> Just trying to find the best patch split.
> If needed, OK to keep as is.

I even don't mind squashing all three commits into one.
The split is done to ease the review.

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

[Thomas Monjalon]

22/01/2021 21:31, Dmitry Kozlyuk:
> On Wed, 20 Jan 2021 11:24:21 +0100, Thomas Monjalon wrote:
> > 20/01/2021 08:23, Dmitry Kozlyuk:
> > > On Wed, 20 Jan 2021 01:05:59 +0100, Thomas Monjalon wrote:  
> > > > This is now the right timeframe to introduce this change
> > > > with the new Python module dependency.
> > > > Unfortunately, the ABI check is returning an issue:
> > > > 
> > > > 'const char mlx5_common_pci_pmd_info[62]' was changed
> > > > to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c  
> > > 
> > > Will investigate and fix ASAP.
> 
> Now that I think of it: strings like this change every time new PCI IDs are
> added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage,
> is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID")
> added 2020-07-08, i.e. clearly outside of ABI change window.

You're right.

> "xxx_pmd_info" changes are due to JSON formatting (new is more canonical),
> which can be worked around easily, if the above is wrong.

If the new format is better, please keep it.
What we need is an exception for the pmdinfo symbols
in the file devtools/libabigail.abignore.
You can probably use a regex for these symbols.


> > > > > --- a/meson.build
> > > > > +++ b/meson.build
> > > > > -subdir('buildtools/pmdinfogen')    
> > > > 
> > > > This could be in patch 3 (removing the code).  
> > > 
> > > It would redefine "pmdinfogen" variable to old pmdinfogen.
> > > Besides, why build what's not used at this patch already?  
> > 
> > Just trying to find the best patch split.
> > If needed, OK to keep as is.
> 
> I even don't mind squashing all three commits into one.
> The split is done to ease the review.

I think the split is good as is.

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

[Dmitry Kozlyuk]

On Fri, 22 Jan 2021 21:57:15 +0100, Thomas Monjalon wrote:
> 22/01/2021 21:31, Dmitry Kozlyuk:
> > On Wed, 20 Jan 2021 11:24:21 +0100, Thomas Monjalon wrote:  
> > > 20/01/2021 08:23, Dmitry Kozlyuk:  
> > > > On Wed, 20 Jan 2021 01:05:59 +0100, Thomas Monjalon wrote:    
> > > > > This is now the right timeframe to introduce this change
> > > > > with the new Python module dependency.
> > > > > Unfortunately, the ABI check is returning an issue:
> > > > > 
> > > > > 'const char mlx5_common_pci_pmd_info[62]' was changed
> > > > > to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c    
> > > > 
> > > > Will investigate and fix ASAP.  
> > 
> > Now that I think of it: strings like this change every time new PCI IDs are
> > added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage,
> > is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID")
> > added 2020-07-08, i.e. clearly outside of ABI change window.  
> 
> You're right.
> 
> > "xxx_pmd_info" changes are due to JSON formatting (new is more canonical),
> > which can be worked around easily, if the above is wrong.  
> 
> If the new format is better, please keep it.
> What we need is an exception for the pmdinfo symbols
> in the file devtools/libabigail.abignore.
> You can probably use a regex for these symbols.

This would allow real breakages to pass ABI check, abidiff doesn't analyze
variable content and it's not easy to compare. Maybe later a script can be
added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of
5494 relevant commits between 19.11 and 20.11, though.

To verify there are no meaningful changes I ensured empty diff between
results of the following command for "main" and the branch:

	find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py

[dpdk-dev] [PATCH v9 0/3] pmdinfogen: rewrite in Python

[Dmitry Kozlyuk]

This patchset implements existing pmdinfogen logic in Python, replaces
and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.

Canonical JSON formatting of generated strings raises ABI warnings.
There are no meaningful changes, which can be checked by comparing
output of pmdinfo before and after the patch:

    find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py

Acked-by: Neil Horman <nhorman@tuxdriver.com>
Tested-by: Jie Zhou <jizh@linux.microsoft.com>

---
Changes in v9:

    * Document pyelftools requirement for FreeBSD (Thomas).
    * Add pyelftools to GitHub workflow.

Dmitry Kozlyuk (3):
  pmdinfogen: add Python implementation
  build: use Python pmdinfogen
  pmdinfogen: remove C implementation

 .github/workflows/build.yml           |   4 +-
 .travis.yml                           |   2 +-
 MAINTAINERS                           |   3 +-
 buildtools/gen-pmdinfo-cfile.sh       |   6 +-
 buildtools/meson.build                |  15 +
 buildtools/pmdinfogen.py              | 189 +++++++++++
 buildtools/pmdinfogen/meson.build     |  14 -
 buildtools/pmdinfogen/pmdinfogen.c    | 456 --------------------------
 buildtools/pmdinfogen/pmdinfogen.h    | 119 -------
 doc/guides/freebsd_gsg/build_dpdk.rst |   3 +-
 doc/guides/linux_gsg/sys_reqs.rst     |   6 +
 drivers/meson.build                   |   2 +-
 meson.build                           |   1 -
 13 files changed, 221 insertions(+), 599 deletions(-)
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

-- 
2.29.2

[dpdk-dev] [PATCH v9 1/3] pmdinfogen: add Python implementation

[Dmitry Kozlyuk]

Using a high-level, interpreted language simplifies maintenance and
build process. Furthermore, ELF handling is delegated to pyelftools
package. Original logic is kept, the copyright recognizes that.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 buildtools/pmdinfogen.py | 189 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 189 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py

diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
new file mode 100755
index 000000000..0cca47ff1
--- /dev/null
+++ b/buildtools/pmdinfogen.py
@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <nhorman@tuxdriver.com>
+# Copyright (c) 2020 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image.get_section_data(self._symbol["st_shndx"])
+        base = self._symbol["st_value"]
+        return data[base:base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        return value[:-1].decode() if value else ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def get_section_data(self, name):
+        return self._image.get_section(name).data()
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        return ELFSymbol(self, symbol[0]) if symbol else None
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver.pci_ids = cls._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    @staticmethod
+    def _load_pci_ids(image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        result = []
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            result.append([
+                pci_id.vendor_id,
+                pci_id.device_id,
+                pci_id.subsystem_vendor_id,
+                pci_id.subsystem_device_id,
+                ])
+        return result
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
-- 
2.29.2

[dpdk-dev] [PATCH v9 2/3] build: use Python pmdinfogen

[Dmitry Kozlyuk]

Use the same interpreter to run pmdinfogen as for other build scripts.
Adjust wrapper script accordingly and also don't suppress stderr from ar
and pmdinfogen.

Add configure-time check for elftools Python module for Unix hosts.

Add pyelftools to CI configuration and build requirements for Linux and
FreeBSD. Currently neither Windows uses pmdinfogen, nor is COFF (PE)
supported.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 .github/workflows/build.yml           |  4 ++--
 .travis.yml                           |  2 +-
 buildtools/gen-pmdinfo-cfile.sh       |  6 +++---
 buildtools/meson.build                | 15 +++++++++++++++
 doc/guides/freebsd_gsg/build_dpdk.rst |  3 ++-
 doc/guides/linux_gsg/sys_reqs.rst     |  6 ++++++
 drivers/meson.build                   |  2 +-
 meson.build                           |  1 -
 8 files changed, 30 insertions(+), 9 deletions(-)

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 0b72df0eb..a5b579add 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -91,8 +91,8 @@ jobs:
       run: sudo apt update
     - name: Install packages
       run: sudo apt install -y ccache libnuma-dev python3-setuptools
-        python3-wheel python3-pip ninja-build libbsd-dev libpcap-dev
-        libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
+        python3-wheel python3-pip python3-pyelftools ninja-build libbsd-dev
+        libpcap-dev libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
     - name: Install libabigail build dependencies if no cache is available
       if: env.ABI_CHECKS == 'true' && steps.libabigail-cache.outputs.cache-hit != 'true'
       run: sudo apt install -y autoconf automake libtool pkg-config libxml2-dev
diff --git a/.travis.yml b/.travis.yml
index 5aa7ad49f..4391af1d5 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -14,7 +14,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
diff --git a/buildtools/gen-pmdinfo-cfile.sh b/buildtools/gen-pmdinfo-cfile.sh
index 43059cf36..109ee461e 100755
--- a/buildtools/gen-pmdinfo-cfile.sh
+++ b/buildtools/gen-pmdinfo-cfile.sh
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 04808dabc..dd4c0f640 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -17,3 +17,18 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
+
+# TODO: starting from Meson 0.51.0 use
+# 	python3 = import('python').find_installation('python',
+#		modules : python3_required_modules)
+python3_required_modules = []
+if host_machine.system() != 'windows'
+	python3_required_modules = ['elftools']
+endif
+foreach module : python3_required_modules
+	script = 'import importlib.util; import sys; exit(importlib.util.find_spec("@0@") is None)'
+	if run_command(py3, '-c', script.format(module)).returncode() != 0
+		error('missing python module: @0@'.format(module))
+	endif
+endforeach
diff --git a/doc/guides/freebsd_gsg/build_dpdk.rst b/doc/guides/freebsd_gsg/build_dpdk.rst
index e3005a7f3..bed353473 100644
--- a/doc/guides/freebsd_gsg/build_dpdk.rst
+++ b/doc/guides/freebsd_gsg/build_dpdk.rst
@@ -14,10 +14,11 @@ The following FreeBSD packages are required to build DPDK:
 * meson
 * ninja
 * pkgconf
+* py37-pyelftools
 
 These can be installed using (as root)::
 
-  pkg install meson pkgconf
+  pkg install meson pkgconf py37-pyelftools
 
 To compile the required kernel modules for memory management and working
 with physical NIC devices, the kernel sources for FreeBSD also
diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
index be714adf2..a05b5bd81 100644
--- a/doc/guides/linux_gsg/sys_reqs.rst
+++ b/doc/guides/linux_gsg/sys_reqs.rst
@@ -52,6 +52,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
diff --git a/drivers/meson.build b/drivers/meson.build
index 77f65fa90..ff5cdb952 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -132,7 +132,7 @@ foreach subpath:subdirs
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
diff --git a/meson.build b/meson.build
index 45d974cd2..2b9c37eb4 100644
--- a/meson.build
+++ b/meson.build
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
-- 
2.29.2

[dpdk-dev] [PATCH v9 3/3] pmdinfogen: remove C implementation

[Dmitry Kozlyuk]

Delete the files no longer used in build process.
Add myself as maintainer of new implementation.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 MAINTAINERS                        |   3 +-
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 4 files changed, 2 insertions(+), 590 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

diff --git a/MAINTAINERS b/MAINTAINERS
index aa973a396..65f6fffd1 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -133,7 +133,8 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <nhorman@tuxdriver.com>
-F: buildtools/pmdinfogen/
+M: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
diff --git a/buildtools/pmdinfogen/meson.build b/buildtools/pmdinfogen/meson.build
deleted file mode 100644
index 670528fac..000000000
--- a/buildtools/pmdinfogen/meson.build
+++ /dev/null
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
diff --git a/buildtools/pmdinfogen/pmdinfogen.c b/buildtools/pmdinfogen/pmdinfogen.c
deleted file mode 100644
index a68d1ea99..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.c
+++ /dev/null
@@ -1,456 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <ctype.h>
-#include <string.h>
-#include <limits.h>
-#include <stdbool.h>
-#include <errno.h>
-#include <libgen.h>
-
-#include <rte_common.h>
-#include "pmdinfogen.h"
-
-#ifdef RTE_ARCH_64
-#define ADDR_SIZE 64
-#else
-#define ADDR_SIZE 32
-#endif
-
-static int use_stdin, use_stdout;
-
-static const char *sym_name(struct elf_info *elf, Elf_Sym *sym)
-{
-	if (sym)
-		return elf->strtab + sym->st_name;
-	else
-		return "(unknown)";
-}
-
-static void *grab_file(const char *filename, unsigned long *size)
-{
-	struct stat st;
-	void *map = MAP_FAILED;
-	int fd = -1;
-
-	if (!use_stdin) {
-		fd = open(filename, O_RDONLY);
-		if (fd < 0)
-			return NULL;
-	} else {
-		/* from stdin, use a temporary file to mmap */
-		FILE *infile;
-		char buffer[1024];
-		int n;
-
-		infile = tmpfile();
-		if (infile == NULL) {
-			perror("tmpfile");
-			return NULL;
-		}
-		fd = dup(fileno(infile));
-		fclose(infile);
-		if (fd < 0)
-			return NULL;
-
-		n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		while (n > 0) {
-			if (write(fd, buffer, n) != n)
-				goto failed;
-			n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		}
-	}
-
-	if (fstat(fd, &st))
-		goto failed;
-
-	*size = st.st_size;
-	map = mmap(NULL, *size, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);
-
-failed:
-	close(fd);
-	if (map == MAP_FAILED)
-		return NULL;
-	return map;
-}
-
-/**
-  * Return a copy of the next line in a mmap'ed file.
-  * spaces in the beginning of the line is trimmed away.
-  * Return a pointer to a static buffer.
-  **/
-static void release_file(void *file, unsigned long size)
-{
-	munmap(file, size);
-}
-
-
-static void *get_sym_value(struct elf_info *info, const Elf_Sym *sym)
-{
-	return RTE_PTR_ADD(info->hdr,
-		info->sechdrs[sym->st_shndx].sh_offset + sym->st_value);
-}
-
-static Elf_Sym *find_sym_in_symtab(struct elf_info *info,
-				   const char *name, Elf_Sym *last)
-{
-	Elf_Sym *idx;
-	if (last)
-		idx = last+1;
-	else
-		idx = info->symtab_start;
-
-	for (; idx < info->symtab_stop; idx++) {
-		const char *n = sym_name(info, idx);
-		if (!strncmp(n, name, strlen(name)))
-			return idx;
-	}
-	return NULL;
-}
-
-static int parse_elf(struct elf_info *info, const char *filename)
-{
-	unsigned int i;
-	Elf_Ehdr *hdr;
-	Elf_Shdr *sechdrs;
-	Elf_Sym  *sym;
-	int endian;
-	unsigned int symtab_idx = ~0U, symtab_shndx_idx = ~0U;
-
-	hdr = grab_file(filename, &info->size);
-	if (!hdr) {
-		perror(filename);
-		exit(1);
-	}
-	info->hdr = hdr;
-	if (info->size < sizeof(*hdr)) {
-		/* file too small, assume this is an empty .o file */
-		return 0;
-	}
-	/* Is this a valid ELF file? */
-	if ((hdr->e_ident[EI_MAG0] != ELFMAG0) ||
-	    (hdr->e_ident[EI_MAG1] != ELFMAG1) ||
-	    (hdr->e_ident[EI_MAG2] != ELFMAG2) ||
-	    (hdr->e_ident[EI_MAG3] != ELFMAG3)) {
-		/* Not an ELF file - silently ignore it */
-		return 0;
-	}
-
-	if (!hdr->e_ident[EI_DATA]) {
-		/* Unknown endian */
-		return 0;
-	}
-
-	endian = hdr->e_ident[EI_DATA];
-
-	/* Fix endianness in ELF header */
-	hdr->e_type      = TO_NATIVE(endian, 16, hdr->e_type);
-	hdr->e_machine   = TO_NATIVE(endian, 16, hdr->e_machine);
-	hdr->e_version   = TO_NATIVE(endian, 32, hdr->e_version);
-	hdr->e_entry     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_entry);
-	hdr->e_phoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_phoff);
-	hdr->e_shoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_shoff);
-	hdr->e_flags     = TO_NATIVE(endian, 32, hdr->e_flags);
-	hdr->e_ehsize    = TO_NATIVE(endian, 16, hdr->e_ehsize);
-	hdr->e_phentsize = TO_NATIVE(endian, 16, hdr->e_phentsize);
-	hdr->e_phnum     = TO_NATIVE(endian, 16, hdr->e_phnum);
-	hdr->e_shentsize = TO_NATIVE(endian, 16, hdr->e_shentsize);
-	hdr->e_shnum     = TO_NATIVE(endian, 16, hdr->e_shnum);
-	hdr->e_shstrndx  = TO_NATIVE(endian, 16, hdr->e_shstrndx);
-
-	sechdrs = RTE_PTR_ADD(hdr, hdr->e_shoff);
-	info->sechdrs = sechdrs;
-
-	/* Check if file offset is correct */
-	if (hdr->e_shoff > info->size) {
-		fprintf(stderr, "section header offset=%lu in file '%s' "
-		      "is bigger than filesize=%lu\n",
-		      (unsigned long)hdr->e_shoff,
-		      filename, info->size);
-		return 0;
-	}
-
-	if (hdr->e_shnum == SHN_UNDEF) {
-		/*
-		 * There are more than 64k sections,
-		 * read count from .sh_size.
-		 */
-		info->num_sections =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[0].sh_size);
-	} else {
-		info->num_sections = hdr->e_shnum;
-	}
-	if (hdr->e_shstrndx == SHN_XINDEX)
-		info->secindex_strings =
-			TO_NATIVE(endian, 32, sechdrs[0].sh_link);
-	else
-		info->secindex_strings = hdr->e_shstrndx;
-
-	/* Fix endianness in section headers */
-	for (i = 0; i < info->num_sections; i++) {
-		sechdrs[i].sh_name      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_name);
-		sechdrs[i].sh_type      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_type);
-		sechdrs[i].sh_flags     =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_flags);
-		sechdrs[i].sh_addr      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addr);
-		sechdrs[i].sh_offset    =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_offset);
-		sechdrs[i].sh_size      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_size);
-		sechdrs[i].sh_link      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_link);
-		sechdrs[i].sh_info      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_info);
-		sechdrs[i].sh_addralign =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addralign);
-		sechdrs[i].sh_entsize   =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_entsize);
-	}
-	/* Find symbol table. */
-	for (i = 1; i < info->num_sections; i++) {
-		int nobits = sechdrs[i].sh_type == SHT_NOBITS;
-
-		if (!nobits && sechdrs[i].sh_offset > info->size) {
-			fprintf(stderr, "%s is truncated. "
-			      "sechdrs[i].sh_offset=%lu > sizeof(*hrd)=%zu\n",
-			      filename, (unsigned long)sechdrs[i].sh_offset,
-			      sizeof(*hdr));
-			return 0;
-		}
-
-		if (sechdrs[i].sh_type == SHT_SYMTAB) {
-			unsigned int sh_link_idx;
-			symtab_idx = i;
-			info->symtab_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-			sh_link_idx = sechdrs[i].sh_link;
-			info->strtab       = RTE_PTR_ADD(hdr,
-				sechdrs[sh_link_idx].sh_offset);
-		}
-
-		/* 32bit section no. table? ("more than 64k sections") */
-		if (sechdrs[i].sh_type == SHT_SYMTAB_SHNDX) {
-			symtab_shndx_idx = i;
-			info->symtab_shndx_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_shndx_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-		}
-	}
-	if (!info->symtab_start)
-		fprintf(stderr, "%s has no symtab?\n", filename);
-	else {
-		/* Fix endianness in symbols */
-		for (sym = info->symtab_start; sym < info->symtab_stop; sym++) {
-			sym->st_shndx = TO_NATIVE(endian, 16, sym->st_shndx);
-			sym->st_name  = TO_NATIVE(endian, 32, sym->st_name);
-			sym->st_value = TO_NATIVE(endian, ADDR_SIZE, sym->st_value);
-			sym->st_size  = TO_NATIVE(endian, ADDR_SIZE, sym->st_size);
-		}
-	}
-
-	if (symtab_shndx_idx != ~0U) {
-		Elf32_Word *p;
-		if (symtab_idx != sechdrs[symtab_shndx_idx].sh_link)
-			fprintf(stderr,
-			      "%s: SYMTAB_SHNDX has bad sh_link: %u!=%u\n",
-			      filename, sechdrs[symtab_shndx_idx].sh_link,
-			      symtab_idx);
-		/* Fix endianness */
-		for (p = info->symtab_shndx_start; p < info->symtab_shndx_stop;
-		     p++)
-			*p = TO_NATIVE(endian, 32, *p);
-	}
-
-	return 1;
-}
-
-static void parse_elf_finish(struct elf_info *info)
-{
-	struct pmd_driver *tmp, *idx = info->drivers;
-	release_file(info->hdr, info->size);
-	while (idx) {
-		tmp = idx->next;
-		free(idx);
-		idx = tmp;
-	}
-}
-
-struct opt_tag {
-	const char *suffix;
-	const char *json_id;
-};
-
-static const struct opt_tag opt_tags[] = {
-	{"_param_string_export", "params"},
-	{"_kmod_dep_export", "kmod"},
-};
-
-static int complete_pmd_entry(struct elf_info *info, struct pmd_driver *drv)
-{
-	const char *tname;
-	int i;
-	char tmpsymname[128];
-	Elf_Sym *tmpsym;
-
-	drv->name = get_sym_value(info, drv->name_sym);
-
-	for (i = 0; i < PMD_OPT_MAX; i++) {
-		memset(tmpsymname, 0, 128);
-		sprintf(tmpsymname, "__%s%s", drv->name, opt_tags[i].suffix);
-		tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-		if (!tmpsym)
-			continue;
-		drv->opt_vals[i] = get_sym_value(info, tmpsym);
-	}
-
-	memset(tmpsymname, 0, 128);
-	sprintf(tmpsymname, "__%s_pci_tbl_export", drv->name);
-
-	tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-
-
-	/*
-	 * If this returns NULL, then this is a PMD_VDEV, because
-	 * it has no pci table reference
-	 */
-	if (!tmpsym) {
-		drv->pci_tbl = NULL;
-		return 0;
-	}
-
-	tname = get_sym_value(info, tmpsym);
-	tmpsym = find_sym_in_symtab(info, tname, NULL);
-	if (!tmpsym)
-		return -ENOENT;
-
-	drv->pci_tbl = (struct rte_pci_id *)get_sym_value(info, tmpsym);
-	if (!drv->pci_tbl)
-		return -ENOENT;
-
-	return 0;
-}
-
-static int locate_pmd_entries(struct elf_info *info)
-{
-	Elf_Sym *last = NULL;
-	struct pmd_driver *new;
-
-	info->drivers = NULL;
-
-	do {
-		new = calloc(sizeof(struct pmd_driver), 1);
-		if (new == NULL) {
-			fprintf(stderr, "Failed to calloc memory\n");
-			return -1;
-		}
-		new->name_sym = find_sym_in_symtab(info, "this_pmd_name", last);
-		last = new->name_sym;
-		if (!new->name_sym)
-			free(new);
-		else {
-			if (complete_pmd_entry(info, new)) {
-				fprintf(stderr,
-					"Failed to complete pmd entry\n");
-				free(new);
-			} else {
-				new->next = info->drivers;
-				info->drivers = new;
-			}
-		}
-	} while (last);
-
-	return 0;
-}
-
-static void output_pmd_info_string(struct elf_info *info, char *outfile)
-{
-	FILE *ofd;
-	struct pmd_driver *drv;
-	struct rte_pci_id *pci_ids;
-	int idx = 0;
-
-	if (use_stdout)
-		ofd = stdout;
-	else {
-		ofd = fopen(outfile, "w+");
-		if (!ofd) {
-			fprintf(stderr, "Unable to open output file\n");
-			return;
-		}
-	}
-
-	drv = info->drivers;
-
-	while (drv) {
-		fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
-			"\"PMD_INFO_STRING= {",
-			drv->name);
-		fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);
-
-		for (idx = 0; idx < PMD_OPT_MAX; idx++) {
-			if (drv->opt_vals[idx])
-				fprintf(ofd, "\\\"%s\\\" : \\\"%s\\\", ",
-					opt_tags[idx].json_id,
-					drv->opt_vals[idx]);
-		}
-
-		pci_ids = drv->pci_tbl;
-		fprintf(ofd, "\\\"pci_ids\\\" : [");
-
-		while (pci_ids && pci_ids->device_id) {
-			fprintf(ofd, "[%d, %d, %d, %d]",
-				pci_ids->vendor_id, pci_ids->device_id,
-				pci_ids->subsystem_vendor_id,
-				pci_ids->subsystem_device_id);
-			pci_ids++;
-			if (pci_ids->device_id)
-				fprintf(ofd, ",");
-			else
-				fprintf(ofd, " ");
-		}
-		fprintf(ofd, "]}\";\n");
-		drv = drv->next;
-	}
-
-	fclose(ofd);
-}
-
-int main(int argc, char **argv)
-{
-	struct elf_info info = {0};
-	int rc = 1;
-
-	if (argc < 3) {
-		fprintf(stderr,
-			"usage: %s <object file> <c output file>\n",
-			basename(argv[0]));
-		exit(127);
-	}
-	use_stdin = !strcmp(argv[1], "-");
-	use_stdout = !strcmp(argv[2], "-");
-	parse_elf(&info, argv[1]);
-
-	if (locate_pmd_entries(&info) < 0)
-		exit(1);
-
-	if (info.drivers) {
-		output_pmd_info_string(&info, argv[2]);
-		rc = 0;
-	} else {
-		fprintf(stderr, "No drivers registered\n");
-	}
-
-	parse_elf_finish(&info);
-	exit(rc);
-}
diff --git a/buildtools/pmdinfogen/pmdinfogen.h b/buildtools/pmdinfogen/pmdinfogen.h
deleted file mode 100644
index 38fd3ddcc..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.h
+++ /dev/null
@@ -1,119 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <stdarg.h>
-#include <string.h>
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#ifdef __linux__
-#include <endian.h>
-#else
-#include <sys/endian.h>
-#endif
-#include <fcntl.h>
-#include <unistd.h>
-#include <elf.h>
-#include <rte_pci.h>
-
-/* On BSD-alike OSes elf.h defines these according to host's word size */
-#undef ELF_ST_BIND
-#undef ELF_ST_TYPE
-#undef ELF_R_SYM
-#undef ELF_R_TYPE
-
-/*
- * Define ELF64_* to ELF_*, the latter being defined in both 32 and 64 bit
- * flavors in elf.h.  This makes our code a bit more generic between arches
- * and allows us to support 32 bit code in the future should we ever want to
- */
-#ifdef RTE_ARCH_64
-#define Elf_Ehdr    Elf64_Ehdr
-#define Elf_Shdr    Elf64_Shdr
-#define Elf_Sym     Elf64_Sym
-#define Elf_Addr    Elf64_Addr
-#define Elf_Sword   Elf64_Sxword
-#define Elf_Section Elf64_Half
-#define ELF_ST_BIND ELF64_ST_BIND
-#define ELF_ST_TYPE ELF64_ST_TYPE
-
-#define Elf_Rel     Elf64_Rel
-#define Elf_Rela    Elf64_Rela
-#define ELF_R_SYM   ELF64_R_SYM
-#define ELF_R_TYPE  ELF64_R_TYPE
-#else
-#define Elf_Ehdr    Elf32_Ehdr
-#define Elf_Shdr    Elf32_Shdr
-#define Elf_Sym     Elf32_Sym
-#define Elf_Addr    Elf32_Addr
-#define Elf_Sword   Elf32_Sxword
-#define Elf_Section Elf32_Half
-#define ELF_ST_BIND ELF32_ST_BIND
-#define ELF_ST_TYPE ELF32_ST_TYPE
-
-#define Elf_Rel     Elf32_Rel
-#define Elf_Rela    Elf32_Rela
-#define ELF_R_SYM   ELF32_R_SYM
-#define ELF_R_TYPE  ELF32_R_TYPE
-#endif
-
-
-/*
- * Note, it seems odd that we have both a CONVERT_NATIVE and a TO_NATIVE macro
- * below.  We do this because the values passed to TO_NATIVE may themselves be
- * macros and need both macros here to get expanded.  Specifically its the width
- * variable we are concerned with, because it needs to get expanded prior to
- * string concatenation
- */
-#define CONVERT_NATIVE(fend, width, x) ({ \
-typeof(x) ___x; \
-if ((fend) == ELFDATA2LSB) \
-	___x = le##width##toh(x); \
-else \
-	___x = be##width##toh(x); \
-___x; \
-})
-
-#define TO_NATIVE(fend, width, x) CONVERT_NATIVE(fend, width, x)
-
-enum opt_params {
-	PMD_PARAM_STRING = 0,
-	PMD_KMOD_DEP,
-	PMD_OPT_MAX
-};
-
-struct pmd_driver {
-	Elf_Sym *name_sym;
-	const char *name;
-	struct rte_pci_id *pci_tbl;
-	struct pmd_driver *next;
-
-	const char *opt_vals[PMD_OPT_MAX];
-};
-
-struct elf_info {
-	unsigned long size;
-	Elf_Ehdr     *hdr;
-	Elf_Shdr     *sechdrs;
-	Elf_Sym      *symtab_start;
-	Elf_Sym      *symtab_stop;
-	char         *strtab;
-
-	/* support for 32bit section numbers */
-
-	unsigned int num_sections; /* max_secindex + 1 */
-	unsigned int secindex_strings;
-	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
-	 * take shndx from symtab_shndx_start[N] instead
-	 */
-	Elf32_Word   *symtab_shndx_start;
-	Elf32_Word   *symtab_shndx_stop;
-
-	struct pmd_driver *drivers;
-};
-- 
2.29.2

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

[Thomas Monjalon]

22/01/2021 23:24, Dmitry Kozlyuk:
> On Fri, 22 Jan 2021 21:57:15 +0100, Thomas Monjalon wrote:
> > 22/01/2021 21:31, Dmitry Kozlyuk:
> > > On Wed, 20 Jan 2021 11:24:21 +0100, Thomas Monjalon wrote:  
> > > > 20/01/2021 08:23, Dmitry Kozlyuk:  
> > > > > On Wed, 20 Jan 2021 01:05:59 +0100, Thomas Monjalon wrote:    
> > > > > > This is now the right timeframe to introduce this change
> > > > > > with the new Python module dependency.
> > > > > > Unfortunately, the ABI check is returning an issue:
> > > > > > 
> > > > > > 'const char mlx5_common_pci_pmd_info[62]' was changed
> > > > > > to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c    
> > > > > 
> > > > > Will investigate and fix ASAP.  
> > > 
> > > Now that I think of it: strings like this change every time new PCI IDs are
> > > added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage,
> > > is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID")
> > > added 2020-07-08, i.e. clearly outside of ABI change window.  
> > 
> > You're right.
> > 
> > > "xxx_pmd_info" changes are due to JSON formatting (new is more canonical),
> > > which can be worked around easily, if the above is wrong.  
> > 
> > If the new format is better, please keep it.
> > What we need is an exception for the pmdinfo symbols
> > in the file devtools/libabigail.abignore.
> > You can probably use a regex for these symbols.
> 
> This would allow real breakages to pass ABI check, abidiff doesn't analyze
> variable content and it's not easy to compare. Maybe later a script can be
> added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of
> 5494 relevant commits between 19.11 and 20.11, though.
> 
> To verify there are no meaningful changes I ensured empty diff between
> results of the following command for "main" and the branch:
> 
> 	find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py

For now we cannot do such check as part of the ABI checker.
And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.

[dpdk-dev] [PATCH v10 0/3] pmdinfogen: rewrite in Python

[Dmitry Kozlyuk]

This patchset implements existing pmdinfogen logic in Python, replaces
and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.

Identity of generated PMD information is checked by comparing
output of pmdinfo before and after the patch:

    find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py

Acked-by: Neil Horman <nhorman@tuxdriver.com>
Tested-by: Jie Zhou <jizh@linux.microsoft.com>

---
Changes in v10:

    * Suppress ABI warnings for generated strings (Thomas).

Dmitry Kozlyuk (3):
  pmdinfogen: add Python implementation
  build: use Python pmdinfogen
  pmdinfogen: remove C implementation

 .github/workflows/build.yml           |   4 +-
 .travis.yml                           |   2 +-
 MAINTAINERS                           |   3 +-
 buildtools/gen-pmdinfo-cfile.sh       |   6 +-
 buildtools/meson.build                |  15 +
 buildtools/pmdinfogen.py              | 189 +++++++++++
 buildtools/pmdinfogen/meson.build     |  14 -
 buildtools/pmdinfogen/pmdinfogen.c    | 456 --------------------------
 buildtools/pmdinfogen/pmdinfogen.h    | 119 -------
 devtools/libabigail.abignore          |   4 +
 doc/guides/freebsd_gsg/build_dpdk.rst |   3 +-
 doc/guides/linux_gsg/sys_reqs.rst     |   6 +
 drivers/meson.build                   |   2 +-
 meson.build                           |   1 -
 14 files changed, 225 insertions(+), 599 deletions(-)
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

-- 
2.29.2

[dpdk-dev] [PATCH v10 1/3] pmdinfogen: add Python implementation

[Dmitry Kozlyuk]

Using a high-level, interpreted language simplifies maintenance and
build process. Furthermore, ELF handling is delegated to pyelftools
package. Original logic is kept, the copyright recognizes that.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 buildtools/pmdinfogen.py | 189 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 189 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py

diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
new file mode 100755
index 000000000..0cca47ff1
--- /dev/null
+++ b/buildtools/pmdinfogen.py
@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <nhorman@tuxdriver.com>
+# Copyright (c) 2020 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image.get_section_data(self._symbol["st_shndx"])
+        base = self._symbol["st_value"]
+        return data[base:base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        return value[:-1].decode() if value else ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def get_section_data(self, name):
+        return self._image.get_section(name).data()
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        return ELFSymbol(self, symbol[0]) if symbol else None
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver.pci_ids = cls._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    @staticmethod
+    def _load_pci_ids(image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        result = []
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            result.append([
+                pci_id.vendor_id,
+                pci_id.device_id,
+                pci_id.subsystem_vendor_id,
+                pci_id.subsystem_device_id,
+                ])
+        return result
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
-- 
2.29.2

[dpdk-dev] [PATCH v10 2/3] build: use Python pmdinfogen

[Dmitry Kozlyuk]

Use the same interpreter to run pmdinfogen as for other build scripts.
Adjust wrapper script accordingly and also don't suppress stderr from ar
and pmdinfogen. Add configure-time check for elftools Python module for
Unix hosts.

Add pyelftools to CI configuration and build requirements for Linux and
FreeBSD. Windows targets are not currently using pmdinfogen.

Suppress ABI warnings about generated PMD information strings.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 .github/workflows/build.yml           |  4 ++--
 .travis.yml                           |  2 +-
 buildtools/gen-pmdinfo-cfile.sh       |  6 +++---
 buildtools/meson.build                | 15 +++++++++++++++
 devtools/libabigail.abignore          |  4 ++++
 doc/guides/freebsd_gsg/build_dpdk.rst |  3 ++-
 doc/guides/linux_gsg/sys_reqs.rst     |  6 ++++++
 drivers/meson.build                   |  2 +-
 meson.build                           |  1 -
 9 files changed, 34 insertions(+), 9 deletions(-)

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 0b72df0eb..a5b579add 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -91,8 +91,8 @@ jobs:
       run: sudo apt update
     - name: Install packages
       run: sudo apt install -y ccache libnuma-dev python3-setuptools
-        python3-wheel python3-pip ninja-build libbsd-dev libpcap-dev
-        libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
+        python3-wheel python3-pip python3-pyelftools ninja-build libbsd-dev
+        libpcap-dev libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
     - name: Install libabigail build dependencies if no cache is available
       if: env.ABI_CHECKS == 'true' && steps.libabigail-cache.outputs.cache-hit != 'true'
       run: sudo apt install -y autoconf automake libtool pkg-config libxml2-dev
diff --git a/.travis.yml b/.travis.yml
index 5aa7ad49f..4391af1d5 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -14,7 +14,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
diff --git a/buildtools/gen-pmdinfo-cfile.sh b/buildtools/gen-pmdinfo-cfile.sh
index 43059cf36..109ee461e 100755
--- a/buildtools/gen-pmdinfo-cfile.sh
+++ b/buildtools/gen-pmdinfo-cfile.sh
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 04808dabc..dd4c0f640 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -17,3 +17,18 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
+
+# TODO: starting from Meson 0.51.0 use
+# 	python3 = import('python').find_installation('python',
+#		modules : python3_required_modules)
+python3_required_modules = []
+if host_machine.system() != 'windows'
+	python3_required_modules = ['elftools']
+endif
+foreach module : python3_required_modules
+	script = 'import importlib.util; import sys; exit(importlib.util.find_spec("@0@") is None)'
+	if run_command(py3, '-c', script.format(module)).returncode() != 0
+		error('missing python module: @0@'.format(module))
+	endif
+endforeach
diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore
index 1dc84fa74..05afccc1a 100644
--- a/devtools/libabigail.abignore
+++ b/devtools/libabigail.abignore
@@ -16,3 +16,7 @@
 [suppress_type]
         name = rte_cryptodev
         has_data_member_inserted_between = {0, 1023}
+
+; Ignore all changes in generated PMD information strings.
+[suppress_variable]
+        name_regex = _pmd_info$
diff --git a/doc/guides/freebsd_gsg/build_dpdk.rst b/doc/guides/freebsd_gsg/build_dpdk.rst
index e3005a7f3..bed353473 100644
--- a/doc/guides/freebsd_gsg/build_dpdk.rst
+++ b/doc/guides/freebsd_gsg/build_dpdk.rst
@@ -14,10 +14,11 @@ The following FreeBSD packages are required to build DPDK:
 * meson
 * ninja
 * pkgconf
+* py37-pyelftools
 
 These can be installed using (as root)::
 
-  pkg install meson pkgconf
+  pkg install meson pkgconf py37-pyelftools
 
 To compile the required kernel modules for memory management and working
 with physical NIC devices, the kernel sources for FreeBSD also
diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
index be714adf2..a05b5bd81 100644
--- a/doc/guides/linux_gsg/sys_reqs.rst
+++ b/doc/guides/linux_gsg/sys_reqs.rst
@@ -52,6 +52,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
diff --git a/drivers/meson.build b/drivers/meson.build
index 77f65fa90..ff5cdb952 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -132,7 +132,7 @@ foreach subpath:subdirs
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
diff --git a/meson.build b/meson.build
index 45d974cd2..2b9c37eb4 100644
--- a/meson.build
+++ b/meson.build
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
-- 
2.29.2

[dpdk-dev] [PATCH v10 3/3] pmdinfogen: remove C implementation

[Dmitry Kozlyuk]

Delete the files no longer used in build process.
Add myself as maintainer of new implementation.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 MAINTAINERS                        |   3 +-
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 4 files changed, 2 insertions(+), 590 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

diff --git a/MAINTAINERS b/MAINTAINERS
index aa973a396..65f6fffd1 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -133,7 +133,8 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <nhorman@tuxdriver.com>
-F: buildtools/pmdinfogen/
+M: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
diff --git a/buildtools/pmdinfogen/meson.build b/buildtools/pmdinfogen/meson.build
deleted file mode 100644
index 670528fac..000000000
--- a/buildtools/pmdinfogen/meson.build
+++ /dev/null
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
diff --git a/buildtools/pmdinfogen/pmdinfogen.c b/buildtools/pmdinfogen/pmdinfogen.c
deleted file mode 100644
index a68d1ea99..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.c
+++ /dev/null
@@ -1,456 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <ctype.h>
-#include <string.h>
-#include <limits.h>
-#include <stdbool.h>
-#include <errno.h>
-#include <libgen.h>
-
-#include <rte_common.h>
-#include "pmdinfogen.h"
-
-#ifdef RTE_ARCH_64
-#define ADDR_SIZE 64
-#else
-#define ADDR_SIZE 32
-#endif
-
-static int use_stdin, use_stdout;
-
-static const char *sym_name(struct elf_info *elf, Elf_Sym *sym)
-{
-	if (sym)
-		return elf->strtab + sym->st_name;
-	else
-		return "(unknown)";
-}
-
-static void *grab_file(const char *filename, unsigned long *size)
-{
-	struct stat st;
-	void *map = MAP_FAILED;
-	int fd = -1;
-
-	if (!use_stdin) {
-		fd = open(filename, O_RDONLY);
-		if (fd < 0)
-			return NULL;
-	} else {
-		/* from stdin, use a temporary file to mmap */
-		FILE *infile;
-		char buffer[1024];
-		int n;
-
-		infile = tmpfile();
-		if (infile == NULL) {
-			perror("tmpfile");
-			return NULL;
-		}
-		fd = dup(fileno(infile));
-		fclose(infile);
-		if (fd < 0)
-			return NULL;
-
-		n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		while (n > 0) {
-			if (write(fd, buffer, n) != n)
-				goto failed;
-			n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		}
-	}
-
-	if (fstat(fd, &st))
-		goto failed;
-
-	*size = st.st_size;
-	map = mmap(NULL, *size, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);
-
-failed:
-	close(fd);
-	if (map == MAP_FAILED)
-		return NULL;
-	return map;
-}
-
-/**
-  * Return a copy of the next line in a mmap'ed file.
-  * spaces in the beginning of the line is trimmed away.
-  * Return a pointer to a static buffer.
-  **/
-static void release_file(void *file, unsigned long size)
-{
-	munmap(file, size);
-}
-
-
-static void *get_sym_value(struct elf_info *info, const Elf_Sym *sym)
-{
-	return RTE_PTR_ADD(info->hdr,
-		info->sechdrs[sym->st_shndx].sh_offset + sym->st_value);
-}
-
-static Elf_Sym *find_sym_in_symtab(struct elf_info *info,
-				   const char *name, Elf_Sym *last)
-{
-	Elf_Sym *idx;
-	if (last)
-		idx = last+1;
-	else
-		idx = info->symtab_start;
-
-	for (; idx < info->symtab_stop; idx++) {
-		const char *n = sym_name(info, idx);
-		if (!strncmp(n, name, strlen(name)))
-			return idx;
-	}
-	return NULL;
-}
-
-static int parse_elf(struct elf_info *info, const char *filename)
-{
-	unsigned int i;
-	Elf_Ehdr *hdr;
-	Elf_Shdr *sechdrs;
-	Elf_Sym  *sym;
-	int endian;
-	unsigned int symtab_idx = ~0U, symtab_shndx_idx = ~0U;
-
-	hdr = grab_file(filename, &info->size);
-	if (!hdr) {
-		perror(filename);
-		exit(1);
-	}
-	info->hdr = hdr;
-	if (info->size < sizeof(*hdr)) {
-		/* file too small, assume this is an empty .o file */
-		return 0;
-	}
-	/* Is this a valid ELF file? */
-	if ((hdr->e_ident[EI_MAG0] != ELFMAG0) ||
-	    (hdr->e_ident[EI_MAG1] != ELFMAG1) ||
-	    (hdr->e_ident[EI_MAG2] != ELFMAG2) ||
-	    (hdr->e_ident[EI_MAG3] != ELFMAG3)) {
-		/* Not an ELF file - silently ignore it */
-		return 0;
-	}
-
-	if (!hdr->e_ident[EI_DATA]) {
-		/* Unknown endian */
-		return 0;
-	}
-
-	endian = hdr->e_ident[EI_DATA];
-
-	/* Fix endianness in ELF header */
-	hdr->e_type      = TO_NATIVE(endian, 16, hdr->e_type);
-	hdr->e_machine   = TO_NATIVE(endian, 16, hdr->e_machine);
-	hdr->e_version   = TO_NATIVE(endian, 32, hdr->e_version);
-	hdr->e_entry     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_entry);
-	hdr->e_phoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_phoff);
-	hdr->e_shoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_shoff);
-	hdr->e_flags     = TO_NATIVE(endian, 32, hdr->e_flags);
-	hdr->e_ehsize    = TO_NATIVE(endian, 16, hdr->e_ehsize);
-	hdr->e_phentsize = TO_NATIVE(endian, 16, hdr->e_phentsize);
-	hdr->e_phnum     = TO_NATIVE(endian, 16, hdr->e_phnum);
-	hdr->e_shentsize = TO_NATIVE(endian, 16, hdr->e_shentsize);
-	hdr->e_shnum     = TO_NATIVE(endian, 16, hdr->e_shnum);
-	hdr->e_shstrndx  = TO_NATIVE(endian, 16, hdr->e_shstrndx);
-
-	sechdrs = RTE_PTR_ADD(hdr, hdr->e_shoff);
-	info->sechdrs = sechdrs;
-
-	/* Check if file offset is correct */
-	if (hdr->e_shoff > info->size) {
-		fprintf(stderr, "section header offset=%lu in file '%s' "
-		      "is bigger than filesize=%lu\n",
-		      (unsigned long)hdr->e_shoff,
-		      filename, info->size);
-		return 0;
-	}
-
-	if (hdr->e_shnum == SHN_UNDEF) {
-		/*
-		 * There are more than 64k sections,
-		 * read count from .sh_size.
-		 */
-		info->num_sections =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[0].sh_size);
-	} else {
-		info->num_sections = hdr->e_shnum;
-	}
-	if (hdr->e_shstrndx == SHN_XINDEX)
-		info->secindex_strings =
-			TO_NATIVE(endian, 32, sechdrs[0].sh_link);
-	else
-		info->secindex_strings = hdr->e_shstrndx;
-
-	/* Fix endianness in section headers */
-	for (i = 0; i < info->num_sections; i++) {
-		sechdrs[i].sh_name      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_name);
-		sechdrs[i].sh_type      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_type);
-		sechdrs[i].sh_flags     =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_flags);
-		sechdrs[i].sh_addr      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addr);
-		sechdrs[i].sh_offset    =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_offset);
-		sechdrs[i].sh_size      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_size);
-		sechdrs[i].sh_link      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_link);
-		sechdrs[i].sh_info      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_info);
-		sechdrs[i].sh_addralign =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addralign);
-		sechdrs[i].sh_entsize   =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_entsize);
-	}
-	/* Find symbol table. */
-	for (i = 1; i < info->num_sections; i++) {
-		int nobits = sechdrs[i].sh_type == SHT_NOBITS;
-
-		if (!nobits && sechdrs[i].sh_offset > info->size) {
-			fprintf(stderr, "%s is truncated. "
-			      "sechdrs[i].sh_offset=%lu > sizeof(*hrd)=%zu\n",
-			      filename, (unsigned long)sechdrs[i].sh_offset,
-			      sizeof(*hdr));
-			return 0;
-		}
-
-		if (sechdrs[i].sh_type == SHT_SYMTAB) {
-			unsigned int sh_link_idx;
-			symtab_idx = i;
-			info->symtab_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-			sh_link_idx = sechdrs[i].sh_link;
-			info->strtab       = RTE_PTR_ADD(hdr,
-				sechdrs[sh_link_idx].sh_offset);
-		}
-
-		/* 32bit section no. table? ("more than 64k sections") */
-		if (sechdrs[i].sh_type == SHT_SYMTAB_SHNDX) {
-			symtab_shndx_idx = i;
-			info->symtab_shndx_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_shndx_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-		}
-	}
-	if (!info->symtab_start)
-		fprintf(stderr, "%s has no symtab?\n", filename);
-	else {
-		/* Fix endianness in symbols */
-		for (sym = info->symtab_start; sym < info->symtab_stop; sym++) {
-			sym->st_shndx = TO_NATIVE(endian, 16, sym->st_shndx);
-			sym->st_name  = TO_NATIVE(endian, 32, sym->st_name);
-			sym->st_value = TO_NATIVE(endian, ADDR_SIZE, sym->st_value);
-			sym->st_size  = TO_NATIVE(endian, ADDR_SIZE, sym->st_size);
-		}
-	}
-
-	if (symtab_shndx_idx != ~0U) {
-		Elf32_Word *p;
-		if (symtab_idx != sechdrs[symtab_shndx_idx].sh_link)
-			fprintf(stderr,
-			      "%s: SYMTAB_SHNDX has bad sh_link: %u!=%u\n",
-			      filename, sechdrs[symtab_shndx_idx].sh_link,
-			      symtab_idx);
-		/* Fix endianness */
-		for (p = info->symtab_shndx_start; p < info->symtab_shndx_stop;
-		     p++)
-			*p = TO_NATIVE(endian, 32, *p);
-	}
-
-	return 1;
-}
-
-static void parse_elf_finish(struct elf_info *info)
-{
-	struct pmd_driver *tmp, *idx = info->drivers;
-	release_file(info->hdr, info->size);
-	while (idx) {
-		tmp = idx->next;
-		free(idx);
-		idx = tmp;
-	}
-}
-
-struct opt_tag {
-	const char *suffix;
-	const char *json_id;
-};
-
-static const struct opt_tag opt_tags[] = {
-	{"_param_string_export", "params"},
-	{"_kmod_dep_export", "kmod"},
-};
-
-static int complete_pmd_entry(struct elf_info *info, struct pmd_driver *drv)
-{
-	const char *tname;
-	int i;
-	char tmpsymname[128];
-	Elf_Sym *tmpsym;
-
-	drv->name = get_sym_value(info, drv->name_sym);
-
-	for (i = 0; i < PMD_OPT_MAX; i++) {
-		memset(tmpsymname, 0, 128);
-		sprintf(tmpsymname, "__%s%s", drv->name, opt_tags[i].suffix);
-		tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-		if (!tmpsym)
-			continue;
-		drv->opt_vals[i] = get_sym_value(info, tmpsym);
-	}
-
-	memset(tmpsymname, 0, 128);
-	sprintf(tmpsymname, "__%s_pci_tbl_export", drv->name);
-
-	tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-
-
-	/*
-	 * If this returns NULL, then this is a PMD_VDEV, because
-	 * it has no pci table reference
-	 */
-	if (!tmpsym) {
-		drv->pci_tbl = NULL;
-		return 0;
-	}
-
-	tname = get_sym_value(info, tmpsym);
-	tmpsym = find_sym_in_symtab(info, tname, NULL);
-	if (!tmpsym)
-		return -ENOENT;
-
-	drv->pci_tbl = (struct rte_pci_id *)get_sym_value(info, tmpsym);
-	if (!drv->pci_tbl)
-		return -ENOENT;
-
-	return 0;
-}
-
-static int locate_pmd_entries(struct elf_info *info)
-{
-	Elf_Sym *last = NULL;
-	struct pmd_driver *new;
-
-	info->drivers = NULL;
-
-	do {
-		new = calloc(sizeof(struct pmd_driver), 1);
-		if (new == NULL) {
-			fprintf(stderr, "Failed to calloc memory\n");
-			return -1;
-		}
-		new->name_sym = find_sym_in_symtab(info, "this_pmd_name", last);
-		last = new->name_sym;
-		if (!new->name_sym)
-			free(new);
-		else {
-			if (complete_pmd_entry(info, new)) {
-				fprintf(stderr,
-					"Failed to complete pmd entry\n");
-				free(new);
-			} else {
-				new->next = info->drivers;
-				info->drivers = new;
-			}
-		}
-	} while (last);
-
-	return 0;
-}
-
-static void output_pmd_info_string(struct elf_info *info, char *outfile)
-{
-	FILE *ofd;
-	struct pmd_driver *drv;
-	struct rte_pci_id *pci_ids;
-	int idx = 0;
-
-	if (use_stdout)
-		ofd = stdout;
-	else {
-		ofd = fopen(outfile, "w+");
-		if (!ofd) {
-			fprintf(stderr, "Unable to open output file\n");
-			return;
-		}
-	}
-
-	drv = info->drivers;
-
-	while (drv) {
-		fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
-			"\"PMD_INFO_STRING= {",
-			drv->name);
-		fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);
-
-		for (idx = 0; idx < PMD_OPT_MAX; idx++) {
-			if (drv->opt_vals[idx])
-				fprintf(ofd, "\\\"%s\\\" : \\\"%s\\\", ",
-					opt_tags[idx].json_id,
-					drv->opt_vals[idx]);
-		}
-
-		pci_ids = drv->pci_tbl;
-		fprintf(ofd, "\\\"pci_ids\\\" : [");
-
-		while (pci_ids && pci_ids->device_id) {
-			fprintf(ofd, "[%d, %d, %d, %d]",
-				pci_ids->vendor_id, pci_ids->device_id,
-				pci_ids->subsystem_vendor_id,
-				pci_ids->subsystem_device_id);
-			pci_ids++;
-			if (pci_ids->device_id)
-				fprintf(ofd, ",");
-			else
-				fprintf(ofd, " ");
-		}
-		fprintf(ofd, "]}\";\n");
-		drv = drv->next;
-	}
-
-	fclose(ofd);
-}
-
-int main(int argc, char **argv)
-{
-	struct elf_info info = {0};
-	int rc = 1;
-
-	if (argc < 3) {
-		fprintf(stderr,
-			"usage: %s <object file> <c output file>\n",
-			basename(argv[0]));
-		exit(127);
-	}
-	use_stdin = !strcmp(argv[1], "-");
-	use_stdout = !strcmp(argv[2], "-");
-	parse_elf(&info, argv[1]);
-
-	if (locate_pmd_entries(&info) < 0)
-		exit(1);
-
-	if (info.drivers) {
-		output_pmd_info_string(&info, argv[2]);
-		rc = 0;
-	} else {
-		fprintf(stderr, "No drivers registered\n");
-	}
-
-	parse_elf_finish(&info);
-	exit(rc);
-}
diff --git a/buildtools/pmdinfogen/pmdinfogen.h b/buildtools/pmdinfogen/pmdinfogen.h
deleted file mode 100644
index 38fd3ddcc..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.h
+++ /dev/null
@@ -1,119 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <stdarg.h>
-#include <string.h>
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#ifdef __linux__
-#include <endian.h>
-#else
-#include <sys/endian.h>
-#endif
-#include <fcntl.h>
-#include <unistd.h>
-#include <elf.h>
-#include <rte_pci.h>
-
-/* On BSD-alike OSes elf.h defines these according to host's word size */
-#undef ELF_ST_BIND
-#undef ELF_ST_TYPE
-#undef ELF_R_SYM
-#undef ELF_R_TYPE
-
-/*
- * Define ELF64_* to ELF_*, the latter being defined in both 32 and 64 bit
- * flavors in elf.h.  This makes our code a bit more generic between arches
- * and allows us to support 32 bit code in the future should we ever want to
- */
-#ifdef RTE_ARCH_64
-#define Elf_Ehdr    Elf64_Ehdr
-#define Elf_Shdr    Elf64_Shdr
-#define Elf_Sym     Elf64_Sym
-#define Elf_Addr    Elf64_Addr
-#define Elf_Sword   Elf64_Sxword
-#define Elf_Section Elf64_Half
-#define ELF_ST_BIND ELF64_ST_BIND
-#define ELF_ST_TYPE ELF64_ST_TYPE
-
-#define Elf_Rel     Elf64_Rel
-#define Elf_Rela    Elf64_Rela
-#define ELF_R_SYM   ELF64_R_SYM
-#define ELF_R_TYPE  ELF64_R_TYPE
-#else
-#define Elf_Ehdr    Elf32_Ehdr
-#define Elf_Shdr    Elf32_Shdr
-#define Elf_Sym     Elf32_Sym
-#define Elf_Addr    Elf32_Addr
-#define Elf_Sword   Elf32_Sxword
-#define Elf_Section Elf32_Half
-#define ELF_ST_BIND ELF32_ST_BIND
-#define ELF_ST_TYPE ELF32_ST_TYPE
-
-#define Elf_Rel     Elf32_Rel
-#define Elf_Rela    Elf32_Rela
-#define ELF_R_SYM   ELF32_R_SYM
-#define ELF_R_TYPE  ELF32_R_TYPE
-#endif
-
-
-/*
- * Note, it seems odd that we have both a CONVERT_NATIVE and a TO_NATIVE macro
- * below.  We do this because the values passed to TO_NATIVE may themselves be
- * macros and need both macros here to get expanded.  Specifically its the width
- * variable we are concerned with, because it needs to get expanded prior to
- * string concatenation
- */
-#define CONVERT_NATIVE(fend, width, x) ({ \
-typeof(x) ___x; \
-if ((fend) == ELFDATA2LSB) \
-	___x = le##width##toh(x); \
-else \
-	___x = be##width##toh(x); \
-___x; \
-})
-
-#define TO_NATIVE(fend, width, x) CONVERT_NATIVE(fend, width, x)
-
-enum opt_params {
-	PMD_PARAM_STRING = 0,
-	PMD_KMOD_DEP,
-	PMD_OPT_MAX
-};
-
-struct pmd_driver {
-	Elf_Sym *name_sym;
-	const char *name;
-	struct rte_pci_id *pci_tbl;
-	struct pmd_driver *next;
-
-	const char *opt_vals[PMD_OPT_MAX];
-};
-
-struct elf_info {
-	unsigned long size;
-	Elf_Ehdr     *hdr;
-	Elf_Shdr     *sechdrs;
-	Elf_Sym      *symtab_start;
-	Elf_Sym      *symtab_stop;
-	char         *strtab;
-
-	/* support for 32bit section numbers */
-
-	unsigned int num_sections; /* max_secindex + 1 */
-	unsigned int secindex_strings;
-	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
-	 * take shndx from symtab_shndx_start[N] instead
-	 */
-	Elf32_Word   *symtab_shndx_start;
-	Elf32_Word   *symtab_shndx_stop;
-
-	struct pmd_driver *drivers;
-};
-- 
2.29.2

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

[Dmitry Kozlyuk]

On Sat, 23 Jan 2021 12:38:45 +0100, Thomas Monjalon wrote:
> 22/01/2021 23:24, Dmitry Kozlyuk:
> > On Fri, 22 Jan 2021 21:57:15 +0100, Thomas Monjalon wrote:  
> > > 22/01/2021 21:31, Dmitry Kozlyuk:  
> > > > On Wed, 20 Jan 2021 11:24:21 +0100, Thomas Monjalon wrote:    
> > > > > 20/01/2021 08:23, Dmitry Kozlyuk:    
> > > > > > On Wed, 20 Jan 2021 01:05:59 +0100, Thomas Monjalon wrote:      
> > > > > > > This is now the right timeframe to introduce this change
> > > > > > > with the new Python module dependency.
> > > > > > > Unfortunately, the ABI check is returning an issue:
> > > > > > > 
> > > > > > > 'const char mlx5_common_pci_pmd_info[62]' was changed
> > > > > > > to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c      
> > > > > > 
> > > > > > Will investigate and fix ASAP.    
> > > > 
> > > > Now that I think of it: strings like this change every time new PCI IDs are
> > > > added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage,
> > > > is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID")
> > > > added 2020-07-08, i.e. clearly outside of ABI change window.    
> > > 
> > > You're right.
> > >   
> > > > "xxx_pmd_info" changes are due to JSON formatting (new is more canonical),
> > > > which can be worked around easily, if the above is wrong.    
> > > 
> > > If the new format is better, please keep it.
> > > What we need is an exception for the pmdinfo symbols
> > > in the file devtools/libabigail.abignore.
> > > You can probably use a regex for these symbols.  
> > 
> > This would allow real breakages to pass ABI check, abidiff doesn't analyze
> > variable content and it's not easy to compare. Maybe later a script can be
> > added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of
> > 5494 relevant commits between 19.11 and 20.11, though.
> > 
> > To verify there are no meaningful changes I ensured empty diff between
> > results of the following command for "main" and the branch:
> > 
> > 	find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py  
> 
> For now we cannot do such check as part of the ABI checker.
> And we cannot merge this patch if the ABI check fails.
> I think the only solution is to allow any change in the pmdinfo variables.

Send v10 with suppression.

Such check, however, *can* be implemented: at ABI check stage we have two
install directories that dpdk-pmdinfo.py can inspect. Then a script can check
that diff contains only additions, i.e. no device support being removed.

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

[Kinsella, Ray]

On 23/01/2021 11:38, Thomas Monjalon wrote:
> 22/01/2021 23:24, Dmitry Kozlyuk:
>> On Fri, 22 Jan 2021 21:57:15 +0100, Thomas Monjalon wrote:
>>> 22/01/2021 21:31, Dmitry Kozlyuk:
>>>> On Wed, 20 Jan 2021 11:24:21 +0100, Thomas Monjalon wrote:  
>>>>> 20/01/2021 08:23, Dmitry Kozlyuk:  
>>>>>> On Wed, 20 Jan 2021 01:05:59 +0100, Thomas Monjalon wrote:    
>>>>>>> This is now the right timeframe to introduce this change
>>>>>>> with the new Python module dependency.
>>>>>>> Unfortunately, the ABI check is returning an issue:
>>>>>>>
>>>>>>> 'const char mlx5_common_pci_pmd_info[62]' was changed
>>>>>>> to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c    
>>>>>>
>>>>>> Will investigate and fix ASAP.  
>>>>
>>>> Now that I think of it: strings like this change every time new PCI IDs are
>>>> added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage,
>>>> is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID")
>>>> added 2020-07-08, i.e. clearly outside of ABI change window.  
>>>
>>> You're right.
>>>
>>>> "xxx_pmd_info" changes are due to JSON formatting (new is more canonical),
>>>> which can be worked around easily, if the above is wrong.  
>>>
>>> If the new format is better, please keep it.
>>> What we need is an exception for the pmdinfo symbols
>>> in the file devtools/libabigail.abignore.
>>> You can probably use a regex for these symbols.
>>
>> This would allow real breakages to pass ABI check, abidiff doesn't analyze
>> variable content and it's not easy to compare. Maybe later a script can be
>> added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of
>> 5494 relevant commits between 19.11 and 20.11, though.
>>
>> To verify there are no meaningful changes I ensured empty diff between
>> results of the following command for "main" and the branch:
>>
>> 	find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py
> 
> For now we cannot do such check as part of the ABI checker.
> And we cannot merge this patch if the ABI check fails.
> I think the only solution is to allow any change in the pmdinfo variables.
> 

So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI.
However we are going to end up carrying this rule in libabigail.ignore indefinitely.

Would it make sense to just fix the size of _pmd_info to some reasonably large value - 
say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?

Ray K

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

[Kinsella, Ray]

On 25/01/2021 09:25, Kinsella, Ray wrote:
> 
> 
> On 23/01/2021 11:38, Thomas Monjalon wrote:
>> 22/01/2021 23:24, Dmitry Kozlyuk:
>>> On Fri, 22 Jan 2021 21:57:15 +0100, Thomas Monjalon wrote:
>>>> 22/01/2021 21:31, Dmitry Kozlyuk:
>>>>> On Wed, 20 Jan 2021 11:24:21 +0100, Thomas Monjalon wrote:  
>>>>>> 20/01/2021 08:23, Dmitry Kozlyuk:  
>>>>>>> On Wed, 20 Jan 2021 01:05:59 +0100, Thomas Monjalon wrote:    
>>>>>>>> This is now the right timeframe to introduce this change
>>>>>>>> with the new Python module dependency.
>>>>>>>> Unfortunately, the ABI check is returning an issue:
>>>>>>>>
>>>>>>>> 'const char mlx5_common_pci_pmd_info[62]' was changed
>>>>>>>> to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c    
>>>>>>>
>>>>>>> Will investigate and fix ASAP.  
>>>>>
>>>>> Now that I think of it: strings like this change every time new PCI IDs are
>>>>> added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage,
>>>>> is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID")
>>>>> added 2020-07-08, i.e. clearly outside of ABI change window.  
>>>>
>>>> You're right.
>>>>
>>>>> "xxx_pmd_info" changes are due to JSON formatting (new is more canonical),
>>>>> which can be worked around easily, if the above is wrong.  
>>>>
>>>> If the new format is better, please keep it.
>>>> What we need is an exception for the pmdinfo symbols
>>>> in the file devtools/libabigail.abignore.
>>>> You can probably use a regex for these symbols.
>>>
>>> This would allow real breakages to pass ABI check, abidiff doesn't analyze
>>> variable content and it's not easy to compare. Maybe later a script can be
>>> added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of
>>> 5494 relevant commits between 19.11 and 20.11, though.
>>>
>>> To verify there are no meaningful changes I ensured empty diff between
>>> results of the following command for "main" and the branch:
>>>
>>> 	find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py
>>
>> For now we cannot do such check as part of the ABI checker.
>> And we cannot merge this patch if the ABI check fails.
>> I think the only solution is to allow any change in the pmdinfo variables.
>>
> 
> So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI.
> However we are going to end up carrying this rule in libabigail.ignore indefinitely.
> 
> Would it make sense to just fix the size of _pmd_info to some reasonably large value - 
> say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?
> 
> Ray K


Another point is - shouldn't _pmd_info probably live in "INTERNAL" is anycase?

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

[Dmitry Kozlyuk]

On Mon, 25 Jan 2021 09:25:51 +0000, Kinsella, Ray wrote:
> On 23/01/2021 11:38, Thomas Monjalon wrote:
> > 22/01/2021 23:24, Dmitry Kozlyuk:  
> >> On Fri, 22 Jan 2021 21:57:15 +0100, Thomas Monjalon wrote:  
> >>> 22/01/2021 21:31, Dmitry Kozlyuk:  
> >>>> On Wed, 20 Jan 2021 11:24:21 +0100, Thomas Monjalon wrote:    
> >>>>> 20/01/2021 08:23, Dmitry Kozlyuk:    
> >>>>>> On Wed, 20 Jan 2021 01:05:59 +0100, Thomas Monjalon wrote:      
> >>>>>>> This is now the right timeframe to introduce this change
> >>>>>>> with the new Python module dependency.
> >>>>>>> Unfortunately, the ABI check is returning an issue:
> >>>>>>>
> >>>>>>> 'const char mlx5_common_pci_pmd_info[62]' was changed
> >>>>>>> to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c      
> >>>>>>
> >>>>>> Will investigate and fix ASAP.    
> >>>>
> >>>> Now that I think of it: strings like this change every time new PCI IDs are
> >>>> added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage,
> >>>> is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID")
> >>>> added 2020-07-08, i.e. clearly outside of ABI change window.    
> >>>
> >>> You're right.
> >>>  
> >>>> "xxx_pmd_info" changes are due to JSON formatting (new is more canonical),
> >>>> which can be worked around easily, if the above is wrong.    
> >>>
> >>> If the new format is better, please keep it.
> >>> What we need is an exception for the pmdinfo symbols
> >>> in the file devtools/libabigail.abignore.
> >>> You can probably use a regex for these symbols.  
> >>
> >> This would allow real breakages to pass ABI check, abidiff doesn't analyze
> >> variable content and it's not easy to compare. Maybe later a script can be
> >> added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of
> >> 5494 relevant commits between 19.11 and 20.11, though.
> >>
> >> To verify there are no meaningful changes I ensured empty diff between
> >> results of the following command for "main" and the branch:
> >>
> >> 	find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py  
> > 
> > For now we cannot do such check as part of the ABI checker.
> > And we cannot merge this patch if the ABI check fails.
> > I think the only solution is to allow any change in the pmdinfo variables.
> >   
> 
> So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI.
> However we are going to end up carrying this rule in libabigail.ignore indefinitely.
> 
> Would it make sense to just fix the size of _pmd_info to some reasonably large value - 
> say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?

I don't think so. This is a JSON *string to be parsed;* considering its size
as part of application *binary* interface is wrong in the first place. As for
content, checking that no PCI IDs are removed is out of scope for libabigail
anyway. Technically we could fix _pmd_info size, but this still allows
breaking changes to pass the check with no benefit.

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

[Kinsella, Ray]

On 25/01/2021 10:05, Dmitry Kozlyuk wrote:
> On Mon, 25 Jan 2021 09:25:51 +0000, Kinsella, Ray wrote:
>> On 23/01/2021 11:38, Thomas Monjalon wrote:
>>> 22/01/2021 23:24, Dmitry Kozlyuk:  
>>>> On Fri, 22 Jan 2021 21:57:15 +0100, Thomas Monjalon wrote:  
>>>>> 22/01/2021 21:31, Dmitry Kozlyuk:  
>>>>>> On Wed, 20 Jan 2021 11:24:21 +0100, Thomas Monjalon wrote:    
>>>>>>> 20/01/2021 08:23, Dmitry Kozlyuk:    
>>>>>>>> On Wed, 20 Jan 2021 01:05:59 +0100, Thomas Monjalon wrote:      
>>>>>>>>> This is now the right timeframe to introduce this change
>>>>>>>>> with the new Python module dependency.
>>>>>>>>> Unfortunately, the ABI check is returning an issue:
>>>>>>>>>
>>>>>>>>> 'const char mlx5_common_pci_pmd_info[62]' was changed
>>>>>>>>> to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c      
>>>>>>>>
>>>>>>>> Will investigate and fix ASAP.    
>>>>>>
>>>>>> Now that I think of it: strings like this change every time new PCI IDs are
>>>>>> added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage,
>>>>>> is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID")
>>>>>> added 2020-07-08, i.e. clearly outside of ABI change window.    
>>>>>
>>>>> You're right.
>>>>>  
>>>>>> "xxx_pmd_info" changes are due to JSON formatting (new is more canonical),
>>>>>> which can be worked around easily, if the above is wrong.    
>>>>>
>>>>> If the new format is better, please keep it.
>>>>> What we need is an exception for the pmdinfo symbols
>>>>> in the file devtools/libabigail.abignore.
>>>>> You can probably use a regex for these symbols.  
>>>>
>>>> This would allow real breakages to pass ABI check, abidiff doesn't analyze
>>>> variable content and it's not easy to compare. Maybe later a script can be
>>>> added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of
>>>> 5494 relevant commits between 19.11 and 20.11, though.
>>>>
>>>> To verify there are no meaningful changes I ensured empty diff between
>>>> results of the following command for "main" and the branch:
>>>>
>>>> 	find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py  
>>>
>>> For now we cannot do such check as part of the ABI checker.
>>> And we cannot merge this patch if the ABI check fails.
>>> I think the only solution is to allow any change in the pmdinfo variables.
>>>   
>>
>> So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI.
>> However we are going to end up carrying this rule in libabigail.ignore indefinitely.
>>
>> Would it make sense to just fix the size of _pmd_info to some reasonably large value - 
>> say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?
> 
> I don't think so. This is a JSON *string to be parsed;* considering its size
> as part of application *binary* interface is wrong in the first place.

Right - then is belongs in INTERNAL, I would say. 

> As for
> content, checking that no PCI IDs are removed is out of scope for libabigail
> anyway. 

Lets be clear PCI IDs - are _nothing_ to do with ABI.

> Technically we could fix _pmd_info size, but this still allows
> breaking changes to pass the check with no benefit.

ABI changes or other, please explain?

Re: [dpdk-dev] [PATCH v10 2/3] build: use Python pmdinfogen

[Thomas Monjalon]

24/01/2021 21:51, Dmitry Kozlyuk:
> --- a/devtools/libabigail.abignore
> +++ b/devtools/libabigail.abignore
> +; Ignore all changes in generated PMD information strings.
> +[suppress_variable]
> +        name_regex = _pmd_info$

I will fix this rule as "name_regexp" so it works.

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

[David Marchand]

On Mon, Jan 25, 2021 at 11:01 AM Kinsella, Ray <mdr@ashroe.eu> wrote:
>
>
>
> On 25/01/2021 09:25, Kinsella, Ray wrote:
> >
> >
> > On 23/01/2021 11:38, Thomas Monjalon wrote:
> >> 22/01/2021 23:24, Dmitry Kozlyuk:
> >>> On Fri, 22 Jan 2021 21:57:15 +0100, Thomas Monjalon wrote:
> >>>> 22/01/2021 21:31, Dmitry Kozlyuk:
> >>>>> On Wed, 20 Jan 2021 11:24:21 +0100, Thomas Monjalon wrote:
> >>>>>> 20/01/2021 08:23, Dmitry Kozlyuk:
> >>>>>>> On Wed, 20 Jan 2021 01:05:59 +0100, Thomas Monjalon wrote:
> >>>>>>>> This is now the right timeframe to introduce this change
> >>>>>>>> with the new Python module dependency.
> >>>>>>>> Unfortunately, the ABI check is returning an issue:
> >>>>>>>>
> >>>>>>>> 'const char mlx5_common_pci_pmd_info[62]' was changed
> >>>>>>>> to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c
> >>>>>>>
> >>>>>>> Will investigate and fix ASAP.
> >>>>>
> >>>>> Now that I think of it: strings like this change every time new PCI IDs are
> >>>>> added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage,
> >>>>> is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID")
> >>>>> added 2020-07-08, i.e. clearly outside of ABI change window.
> >>>>
> >>>> You're right.
> >>>>
> >>>>> "xxx_pmd_info" changes are due to JSON formatting (new is more canonical),
> >>>>> which can be worked around easily, if the above is wrong.
> >>>>
> >>>> If the new format is better, please keep it.
> >>>> What we need is an exception for the pmdinfo symbols
> >>>> in the file devtools/libabigail.abignore.
> >>>> You can probably use a regex for these symbols.
> >>>
> >>> This would allow real breakages to pass ABI check, abidiff doesn't analyze
> >>> variable content and it's not easy to compare. Maybe later a script can be
> >>> added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of
> >>> 5494 relevant commits between 19.11 and 20.11, though.
> >>>
> >>> To verify there are no meaningful changes I ensured empty diff between
> >>> results of the following command for "main" and the branch:
> >>>
> >>>     find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py
> >>
> >> For now we cannot do such check as part of the ABI checker.
> >> And we cannot merge this patch if the ABI check fails.
> >> I think the only solution is to allow any change in the pmdinfo variables.
> >>
> >
> > So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI.
> > However we are going to end up carrying this rule in libabigail.ignore indefinitely.
> >
> > Would it make sense to just fix the size of _pmd_info to some reasonably large value -
> > say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?
> >
> > Ray K
>
>
> Another point is - shouldn't _pmd_info probably live in "INTERNAL" is anycase?

The symbol itself can be hidden from the ABeyes.
It is only a placeholder for the PMD_INFO_STRING= string used by
usertools/dpdk-pmdinfo.py and maybe some other parsing tool.

I guess a static symbol would be enough:

diff --git a/buildtools/pmdinfogen/pmdinfogen.c
b/buildtools/pmdinfogen/pmdinfogen.c
index a68d1ea999..14bf7d9f42 100644
--- a/buildtools/pmdinfogen/pmdinfogen.c
+++ b/buildtools/pmdinfogen/pmdinfogen.c
@@ -393,7 +393,7 @@ static void output_pmd_info_string(struct elf_info
*info, char *outfile)
        drv = info->drivers;

        while (drv) {
-               fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
+               fprintf(ofd, "static const char %s_pmd_info[]
__attribute__((used)) = "
                        "\"PMD_INFO_STRING= {",
                        drv->name);
                fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);


We will need an exception for the v21 ABI though.


-- 
David Marchand

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

[Dmitry Kozlyuk]

On Mon, 25 Jan 2021 10:11:07 +0000, Kinsella, Ray wrote:
> On 25/01/2021 10:05, Dmitry Kozlyuk wrote:
> > On Mon, 25 Jan 2021 09:25:51 +0000, Kinsella, Ray wrote:  
> >> On 23/01/2021 11:38, Thomas Monjalon wrote:  
> >>> 22/01/2021 23:24, Dmitry Kozlyuk:    
> >>>> On Fri, 22 Jan 2021 21:57:15 +0100, Thomas Monjalon wrote:    
> >>>>> 22/01/2021 21:31, Dmitry Kozlyuk:    
> >>>>>> On Wed, 20 Jan 2021 11:24:21 +0100, Thomas Monjalon wrote:      
> >>>>>>> 20/01/2021 08:23, Dmitry Kozlyuk:      
> >>>>>>>> On Wed, 20 Jan 2021 01:05:59 +0100, Thomas Monjalon wrote:        
> >>>>>>>>> This is now the right timeframe to introduce this change
> >>>>>>>>> with the new Python module dependency.
> >>>>>>>>> Unfortunately, the ABI check is returning an issue:
> >>>>>>>>>
> >>>>>>>>> 'const char mlx5_common_pci_pmd_info[62]' was changed
> >>>>>>>>> to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c        
> >>>>>>>>
> >>>>>>>> Will investigate and fix ASAP.      
> >>>>>>
> >>>>>> Now that I think of it: strings like this change every time new PCI IDs are
> >>>>>> added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage,
> >>>>>> is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID")
> >>>>>> added 2020-07-08, i.e. clearly outside of ABI change window.      
> >>>>>
> >>>>> You're right.
> >>>>>    
> >>>>>> "xxx_pmd_info" changes are due to JSON formatting (new is more canonical),
> >>>>>> which can be worked around easily, if the above is wrong.      
> >>>>>
> >>>>> If the new format is better, please keep it.
> >>>>> What we need is an exception for the pmdinfo symbols
> >>>>> in the file devtools/libabigail.abignore.
> >>>>> You can probably use a regex for these symbols.    
> >>>>
> >>>> This would allow real breakages to pass ABI check, abidiff doesn't analyze
> >>>> variable content and it's not easy to compare. Maybe later a script can be
> >>>> added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of
> >>>> 5494 relevant commits between 19.11 and 20.11, though.
> >>>>
> >>>> To verify there are no meaningful changes I ensured empty diff between
> >>>> results of the following command for "main" and the branch:
> >>>>
> >>>> 	find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py    
> >>>
> >>> For now we cannot do such check as part of the ABI checker.
> >>> And we cannot merge this patch if the ABI check fails.
> >>> I think the only solution is to allow any change in the pmdinfo variables.
> >>>     
> >>
> >> So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI.
> >> However we are going to end up carrying this rule in libabigail.ignore indefinitely.
> >>
> >> Would it make sense to just fix the size of _pmd_info to some reasonably large value - 
> >> say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?  
> > 
> > I don't think so. This is a JSON *string to be parsed;* considering its size
> > as part of application *binary* interface is wrong in the first place.  
> 
> Right - then is belongs in INTERNAL, I would say. 
>
> > As for
> > content, checking that no PCI IDs are removed is out of scope for libabigail
> > anyway.   
> 
> Lets be clear PCI IDs - are _nothing_ to do with ABI.

Technically, yes, but they're referred to in abi_policy.rst, because DPDK
behavior depends on them. Same issue as with as return values: no formats
change, yet compatibility is broken.

> > Technically we could fix _pmd_info size, but this still allows
> > breaking changes to pass the check with no benefit.  
> 
> ABI changes or other, please explain?

Behavioral changes via PCI ID removal, see above.

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

[Kinsella, Ray]

On 25/01/2021 10:29, David Marchand wrote:
> On Mon, Jan 25, 2021 at 11:01 AM Kinsella, Ray <mdr@ashroe.eu> wrote:
>>
>>
>>
>> On 25/01/2021 09:25, Kinsella, Ray wrote:
>>>
>>>
>>> On 23/01/2021 11:38, Thomas Monjalon wrote:
>>>> 22/01/2021 23:24, Dmitry Kozlyuk:
>>>>> On Fri, 22 Jan 2021 21:57:15 +0100, Thomas Monjalon wrote:
>>>>>> 22/01/2021 21:31, Dmitry Kozlyuk:
>>>>>>> On Wed, 20 Jan 2021 11:24:21 +0100, Thomas Monjalon wrote:
>>>>>>>> 20/01/2021 08:23, Dmitry Kozlyuk:
>>>>>>>>> On Wed, 20 Jan 2021 01:05:59 +0100, Thomas Monjalon wrote:
>>>>>>>>>> This is now the right timeframe to introduce this change
>>>>>>>>>> with the new Python module dependency.
>>>>>>>>>> Unfortunately, the ABI check is returning an issue:
>>>>>>>>>>
>>>>>>>>>> 'const char mlx5_common_pci_pmd_info[62]' was changed
>>>>>>>>>> to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c
>>>>>>>>>
>>>>>>>>> Will investigate and fix ASAP.
>>>>>>>
>>>>>>> Now that I think of it: strings like this change every time new PCI IDs are
>>>>>>> added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage,
>>>>>>> is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID")
>>>>>>> added 2020-07-08, i.e. clearly outside of ABI change window.
>>>>>>
>>>>>> You're right.
>>>>>>
>>>>>>> "xxx_pmd_info" changes are due to JSON formatting (new is more canonical),
>>>>>>> which can be worked around easily, if the above is wrong.
>>>>>>
>>>>>> If the new format is better, please keep it.
>>>>>> What we need is an exception for the pmdinfo symbols
>>>>>> in the file devtools/libabigail.abignore.
>>>>>> You can probably use a regex for these symbols.
>>>>>
>>>>> This would allow real breakages to pass ABI check, abidiff doesn't analyze
>>>>> variable content and it's not easy to compare. Maybe later a script can be
>>>>> added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of
>>>>> 5494 relevant commits between 19.11 and 20.11, though.
>>>>>
>>>>> To verify there are no meaningful changes I ensured empty diff between
>>>>> results of the following command for "main" and the branch:
>>>>>
>>>>>     find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py
>>>>
>>>> For now we cannot do such check as part of the ABI checker.
>>>> And we cannot merge this patch if the ABI check fails.
>>>> I think the only solution is to allow any change in the pmdinfo variables.
>>>>
>>>
>>> So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI.
>>> However we are going to end up carrying this rule in libabigail.ignore indefinitely.
>>>
>>> Would it make sense to just fix the size of _pmd_info to some reasonably large value -
>>> say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?
>>>
>>> Ray K
>>
>>
>> Another point is - shouldn't _pmd_info probably live in "INTERNAL" is anycase?
> 
> The symbol itself can be hidden from the ABeyes.
> It is only a placeholder for the PMD_INFO_STRING= string used by
> usertools/dpdk-pmdinfo.py and maybe some other parsing tool.
> 
> I guess a static symbol would be enough:
> 
> diff --git a/buildtools/pmdinfogen/pmdinfogen.c
> b/buildtools/pmdinfogen/pmdinfogen.c
> index a68d1ea999..14bf7d9f42 100644
> --- a/buildtools/pmdinfogen/pmdinfogen.c
> +++ b/buildtools/pmdinfogen/pmdinfogen.c
> @@ -393,7 +393,7 @@ static void output_pmd_info_string(struct elf_info
> *info, char *outfile)
>         drv = info->drivers;
> 
>         while (drv) {
> -               fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
> +               fprintf(ofd, "static const char %s_pmd_info[]
> __attribute__((used)) = "
>                         "\"PMD_INFO_STRING= {",
>                         drv->name);
>                 fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);
> 
> 
> We will need an exception for the v21 ABI though.
> 

Good suggestion +1

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

[Thomas Monjalon]

25/01/2021 11:46, Kinsella, Ray:
> On 25/01/2021 10:29, David Marchand wrote:
> > The symbol itself can be hidden from the ABeyes.
> > It is only a placeholder for the PMD_INFO_STRING= string used by
> > usertools/dpdk-pmdinfo.py and maybe some other parsing tool.
> > 
> > I guess a static symbol would be enough:
> > 
> > diff --git a/buildtools/pmdinfogen/pmdinfogen.c
> > b/buildtools/pmdinfogen/pmdinfogen.c
> > index a68d1ea999..14bf7d9f42 100644
> > --- a/buildtools/pmdinfogen/pmdinfogen.c
> > +++ b/buildtools/pmdinfogen/pmdinfogen.c
> > @@ -393,7 +393,7 @@ static void output_pmd_info_string(struct elf_info
> > *info, char *outfile)
> >         drv = info->drivers;
> > 
> >         while (drv) {
> > -               fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
> > +               fprintf(ofd, "static const char %s_pmd_info[]
> > __attribute__((used)) = "
> >                         "\"PMD_INFO_STRING= {",
> >                         drv->name);
> >                 fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);
> > 
> > 
> > We will need an exception for the v21 ABI though.
> > 
> 
> Good suggestion +1

Yes +1 for adding static on *_pmd_info

Re: [dpdk-dev] [PATCH v10 0/3] pmdinfogen: rewrite in Python

[Thomas Monjalon]

> Dmitry Kozlyuk (3):
>   pmdinfogen: add Python implementation
>   build: use Python pmdinfogen
>   pmdinfogen: remove C implementation

Applied, thanks.

Reminder: it is adding a NEW DEPENDENCY on pyelftools,
as highlighted already in CI and maintainers meetings:
	http://mails.dpdk.org/archives/ci/2021-January/000939.html
	http://mails.dpdk.org/archives/dev/2021-January/197814.html

If not already done, please UPDATE your environment! Thanks

Re: [dpdk-dev] [PATCH v10 0/3] pmdinfogen: rewrite in Python

[Brandon Lo]

Hi Thomas,

We have updated all UNH-IOL environments to include pyelftools.

Thanks,
Brandon

On Mon, Jan 25, 2021 at 8:13 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> > Dmitry Kozlyuk (3):
> >   pmdinfogen: add Python implementation
> >   build: use Python pmdinfogen
> >   pmdinfogen: remove C implementation
>
> Applied, thanks.
>
> Reminder: it is adding a NEW DEPENDENCY on pyelftools,
> as highlighted already in CI and maintainers meetings:
>         http://mails.dpdk.org/archives/ci/2021-January/000939.html
>         http://mails.dpdk.org/archives/dev/2021-January/197814.html
>
> If not already done, please UPDATE your environment! Thanks
>
>


-- 

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu

Re: [dpdk-dev] [PATCH v10 0/3] pmdinfogen: rewrite in Python

[Ali Alnubani]

Hi,

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Dmitry Kozlyuk
> Sent: Sunday, January 24, 2021 10:52 PM
> To: dev@dpdk.org
> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>; Bruce Richardson
> <bruce.richardson@intel.com>; NBU-Contact-Thomas Monjalon
> <thomas@monjalon.net>; Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>; Neil
> Horman <nhorman@tuxdriver.com>; Jie Zhou <jizh@linux.microsoft.com>
> Subject: [dpdk-dev] [PATCH v10 0/3] pmdinfogen: rewrite in Python
> 
> This patchset implements existing pmdinfogen logic in Python, replaces and
> removes the old code. The goals of rewriting are:
> 
> * easier maintenance by using a more high-level language,
> * simpler build process without host application and libelf,
> * foundation for adding Windows support.
> 
> Identity of generated PMD information is checked by comparing output of
> pmdinfo before and after the patch:
> 
>     find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py
> 
> Acked-by: Neil Horman <nhorman@tuxdriver.com>
> Tested-by: Jie Zhou <jizh@linux.microsoft.com>
> 
> ---

Please see the following build errors:
https://github.com/DPDK/dpdk/actions/runs/509771118
"""
FAILED: drivers/rte_common_cpt.pmd.c 
/usr/bin/python3 ../buildtools/gen-pmdinfo-cfile.py ar /home/runner/work/dpdk/dpdk/build/drivers/libtmp_rte_common_cpt.a drivers/rte_common_cpt.pmd.c /usr/bin/python3 ../buildtools/pmdinfogen.py elf
Traceback (most recent call last):
  File "../buildtools/gen-pmdinfo-cfile.py", line 14, in <module>
    [ar, "xv", os.path.abspath(archive)], capture_output=True, check=True, cwd=temp
  File "/usr/lib/python3.6/subprocess.py", line 423, in run
    with Popen(*popenargs, **kwargs) as process:
TypeError: __init__() got an unexpected keyword argument 'capture_output'
"""

I also see the exception in Ubuntu 16.04 (Python 3.5.2, pyelftools 0.27), CentOS 7 (Python 3.6.8, pyelftools 0.27), CentOS 8 (Python 3.6.8, pyelftools 0.27) and OpenSUSE Leap 15.2 (Python 3.6.12, pyelftools 0.26).

Thanks,
Ali

Re: [dpdk-dev] [PATCH v10 0/3] pmdinfogen: rewrite in Python

[Dmitry Kozlyuk]

Hi Ali,

> Please see the following build errors:
> https://github.com/DPDK/dpdk/actions/runs/509771118
> """
> FAILED: drivers/rte_common_cpt.pmd.c 
> /usr/bin/python3 ../buildtools/gen-pmdinfo-cfile.py ar /home/runner/work/dpdk/dpdk/build/drivers/libtmp_rte_common_cpt.a drivers/rte_common_cpt.pmd.c /usr/bin/python3 ../buildtools/pmdinfogen.py elf
> Traceback (most recent call last):
>   File "../buildtools/gen-pmdinfo-cfile.py", line 14, in <module>
>     [ar, "xv", os.path.abspath(archive)], capture_output=True, check=True, cwd=temp
>   File "/usr/lib/python3.6/subprocess.py", line 423, in run
>     with Popen(*popenargs, **kwargs) as process:
> TypeError: __init__() got an unexpected keyword argument 'capture_output'
> """
> 
> I also see the exception in Ubuntu 16.04 (Python 3.5.2, pyelftools 0.27), CentOS 7 (Python 3.6.8, pyelftools 0.27), CentOS 8 (Python 3.6.8, pyelftools 0.27) and OpenSUSE Leap 15.2 (Python 3.6.12, pyelftools 0.26).

If fixed the error you encountered and also discovered a bunch of other
issues, all of them hopefully fixed in the following series:

http://patchwork.dpdk.org/project/dpdk/list/?series=14938

Re: [dpdk-dev] [PATCH v10 0/3] pmdinfogen: rewrite in Python

[Tal Shnaiderman]

> Subject: Re: [dpdk-dev] [PATCH v10 0/3] pmdinfogen: rewrite in Python
> 
> External email: Use caution opening links or attachments
> 
> 
> Hi Thomas,
> 
> We have updated all UNH-IOL environments to include pyelftools.

FreeBSD compilation test is still failing on the missing dependency:

buildtools/meson.build:45:2: ERROR: Problem encountered: missing python module: elftools

> 
> Thanks,
> Brandon
> 
> On Mon, Jan 25, 2021 at 8:13 AM Thomas Monjalon
> <thomas@monjalon.net> wrote:
> >
> > > Dmitry Kozlyuk (3):
> > >   pmdinfogen: add Python implementation
> > >   build: use Python pmdinfogen
> > >   pmdinfogen: remove C implementation
> >
> > Applied, thanks.
> >
> > Reminder: it is adding a NEW DEPENDENCY on pyelftools, as highlighted
> > already in CI and maintainers meetings:
> >
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmails.
> dpdk.org%2Farchives%2Fci%2F2021-
> January%2F000939.html&amp;data=04%7C01%7Ctalshn%40nvidia.com%7C70
> 5c7ce391d1429bc94508d8c14b9880%7C43083d15727340c1b7db39efd9ccc17a
> %7C0%7C0%7C637471877725119466%7CUnknown%7CTWFpbGZsb3d8eyJWIj
> oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1
> 000&amp;sdata=Lei7o0Mtgsb3WZJ14lGolduVJK3fA7EbYuw8ojVCb3g%3D&a
> mp;reserved=0
> >
> >
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmails
> > .dpdk.org%2Farchives%2Fdev%2F2021-
> January%2F197814.html&amp;data=04%7C
> >
> 01%7Ctalshn%40nvidia.com%7C705c7ce391d1429bc94508d8c14b9880%7C430
> 83d15
> >
> 727340c1b7db39efd9ccc17a%7C0%7C0%7C637471877725119466%7CUnknow
> n%7CTWFp
> >
> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
> I6Mn
> >
> 0%3D%7C1000&amp;sdata=74Zp6%2B6YB8OCVfo%2F0Gb7C9C43QWAo1YD4
> AUANneBGdA%
> > 3D&amp;reserved=0
> >
> > If not already done, please UPDATE your environment! Thanks
> >
> >
> 
> 
> --
> 
> Brandon Lo
> 
> UNH InterOperability Laboratory
> 
> 21 Madbury Rd, Suite 100, Durham, NH 03824
> 
> blo@iol.unh.edu
> 
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww
> .iol.unh.edu%2F&amp;data=04%7C01%7Ctalshn%40nvidia.com%7C705c7ce3
> 91d1429bc94508d8c14b9880%7C43083d15727340c1b7db39efd9ccc17a%7C0%
> 7C0%7C637471877725119466%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4
> wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&a
> mp;sdata=2Fc4epSpG%2FhLFdxj9D1kVoafaxlfUhaZvpEm4xwHFd4%3D&amp;
> reserved=0