DPDK CI discussions
 help / color / mirror / Atom feed
* Re: [dpdk-ci] [dpdk-dev] [PATCH v5 0/3] pmdinfogen: rewrite in Python
       [not found] ` <20200927214732.12783-1-dmitry.kozliuk@gmail.com>
@ 2020-09-28  9:35   ` David Marchand
  2020-10-04  1:59   ` [dpdk-ci] [PATCH v6 " Dmitry Kozlyuk
  1 sibling, 0 replies; 21+ messages in thread
From: David Marchand @ 2020-09-28  9:35 UTC (permalink / raw)
  To: Dmitry Kozlyuk
  Cc: dev, Neil Horman, ci, Thomas Monjalon, Timothy Redaelli, Luca Boccassi

Hello Dmitry,

On Sun, Sep 27, 2020 at 11:48 PM Dmitry Kozlyuk
<dmitry.kozliuk@gmail.com> wrote:
>
> This patchset implements existing pmdinfogen logic in Python, replaces
> and removes the old code. The goals of rewriting are:
>

* support pmdinfo on Windows ? :-)

> * easier maintenance by using a more high-level language,
> * simpler build process without host application and libelf.
>
> Travis CI script is adjusted to install python3-pyelftools, but other CI
> systems may need similar tweaking. Particularly, testing on FreeBSD and
> big-endian targets is desired.

This new requirement will impact mainly distribution packagers,
developers and the CI.
While I think this is fine for the first two categories
(python-elftools is now available on most distributions, and you can
also pip install it), the last category is a concern.

If we want to move forward with this patch, this must be addressed in
a timely manner as we don't want to break the CI.


-- 
David Marchand


^ permalink raw reply	[flat|nested] 21+ messages in thread

* [dpdk-ci] [PATCH v6 0/3] pmdinfogen: rewrite in Python
       [not found] ` <20200927214732.12783-1-dmitry.kozliuk@gmail.com>
  2020-09-28  9:35   ` [dpdk-ci] [dpdk-dev] [PATCH v5 0/3] pmdinfogen: rewrite in Python David Marchand
@ 2020-10-04  1:59   ` Dmitry Kozlyuk
  2020-10-04  1:59     ` [dpdk-ci] [PATCH v6 1/3] pmdinfogen: add Python implementation Dmitry Kozlyuk
                       ` (4 more replies)
  1 sibling, 5 replies; 21+ messages in thread
From: Dmitry Kozlyuk @ 2020-10-04  1:59 UTC (permalink / raw)
  To: dev
  Cc: ci, Stephen Hemminger, David Marchand, Thomas Monjalon,
	Dmitry Kozlyuk, Neil Horman

This patchset implements existing pmdinfogen logic in Python, replaces
and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.

Travis CI script is adjusted to install python3-pyelftools, but other CI
systems may need similar tweaking. Particularly, testing on FreeBSD and
big-endian targets is desired.

Acked-by: Neil Horman <nhorman@tuxdriver.com>

---
Changes in v6:

1. Address yapf and pylint warnings (Stephen Hemminger).
2. Add myself as new implementation maintainer (Thomas Monjalon).

Dmitry Kozlyuk (3):
  pmdinfogen: add Python implementation
  build: use Python pmdinfogen
  pmdinfogen: remove C implementation

 .travis.yml                        |   2 +-
 MAINTAINERS                        |   3 +-
 buildtools/gen-pmdinfo-cfile.sh    |   6 +-
 buildtools/meson.build             |   1 +
 buildtools/pmdinfogen.py           | 189 ++++++++++++
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 doc/guides/linux_gsg/sys_reqs.rst  |   6 +
 drivers/meson.build                |   2 +-
 meson.build                        |   1 -
 11 files changed, 203 insertions(+), 596 deletions(-)
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

-- 
2.28.0


^ permalink raw reply	[flat|nested] 21+ messages in thread

* [dpdk-ci] [PATCH v6 1/3] pmdinfogen: add Python implementation
  2020-10-04  1:59   ` [dpdk-ci] [PATCH v6 " Dmitry Kozlyuk
@ 2020-10-04  1:59     ` Dmitry Kozlyuk
  2020-10-04  1:59     ` [dpdk-ci] [PATCH v6 2/3] build: use Python pmdinfogen Dmitry Kozlyuk
                       ` (3 subsequent siblings)
  4 siblings, 0 replies; 21+ messages in thread
From: Dmitry Kozlyuk @ 2020-10-04  1:59 UTC (permalink / raw)
  To: dev
  Cc: ci, Stephen Hemminger, David Marchand, Thomas Monjalon,
	Dmitry Kozlyuk, Neil Horman

Using a high-level, interpreted language simplifies maintenance and
build process. Furthermore, ELF handling is delegated to pyelftools
package. Original logic is kept, the copyright recognizes that.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 buildtools/pmdinfogen.py | 189 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 189 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py

diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
new file mode 100755
index 000000000..0cca47ff1
--- /dev/null
+++ b/buildtools/pmdinfogen.py
@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <nhorman@tuxdriver.com>
+# Copyright (c) 2020 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image.get_section_data(self._symbol["st_shndx"])
+        base = self._symbol["st_value"]
+        return data[base:base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        return value[:-1].decode() if value else ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def get_section_data(self, name):
+        return self._image.get_section(name).data()
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        return ELFSymbol(self, symbol[0]) if symbol else None
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver.pci_ids = cls._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    @staticmethod
+    def _load_pci_ids(image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        result = []
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            result.append([
+                pci_id.vendor_id,
+                pci_id.device_id,
+                pci_id.subsystem_vendor_id,
+                pci_id.subsystem_device_id,
+                ])
+        return result
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
-- 
2.28.0


^ permalink raw reply	[flat|nested] 21+ messages in thread

* [dpdk-ci] [PATCH v6 2/3] build: use Python pmdinfogen
  2020-10-04  1:59   ` [dpdk-ci] [PATCH v6 " Dmitry Kozlyuk
  2020-10-04  1:59     ` [dpdk-ci] [PATCH v6 1/3] pmdinfogen: add Python implementation Dmitry Kozlyuk
@ 2020-10-04  1:59     ` Dmitry Kozlyuk
  2020-10-04  1:59     ` [dpdk-ci] [PATCH v6 3/3] pmdinfogen: remove C implementation Dmitry Kozlyuk
                       ` (2 subsequent siblings)
  4 siblings, 0 replies; 21+ messages in thread
From: Dmitry Kozlyuk @ 2020-10-04  1:59 UTC (permalink / raw)
  To: dev
  Cc: ci, Stephen Hemminger, David Marchand, Thomas Monjalon,
	Dmitry Kozlyuk, Aaron Conole, Michael Santana, Bruce Richardson

Like for other build scripts, use Python interpreter to run pmdinfogen.
Adjust wrapper script accordingly and also don't suppress stderr from ar
and pmdinfogen.

Add python3-pyelftools to CI configuration. The package is available on
all major distributions. FreeBSD has no system requirements section in
its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE)
supported.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 .travis.yml                       | 2 +-
 buildtools/gen-pmdinfo-cfile.sh   | 6 +++---
 buildtools/meson.build            | 1 +
 doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
 drivers/meson.build               | 2 +-
 meson.build                       | 1 -
 6 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 5e12db23b..6744f6fc1 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -11,7 +11,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
diff --git a/buildtools/gen-pmdinfo-cfile.sh b/buildtools/gen-pmdinfo-cfile.sh
index 43059cf36..109ee461e 100755
--- a/buildtools/gen-pmdinfo-cfile.sh
+++ b/buildtools/gen-pmdinfo-cfile.sh
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 04808dabc..3a64b28b7 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -17,3 +17,4 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
index 6ecdc04aa..89c9f2570 100644
--- a/doc/guides/linux_gsg/sys_reqs.rst
+++ b/doc/guides/linux_gsg/sys_reqs.rst
@@ -52,6 +52,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
diff --git a/drivers/meson.build b/drivers/meson.build
index 5f9526557..88364259d 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -137,7 +137,7 @@ foreach subpath:subdirs
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
diff --git a/meson.build b/meson.build
index 61d9a4f5f..a6e8c09c2 100644
--- a/meson.build
+++ b/meson.build
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
-- 
2.28.0


^ permalink raw reply	[flat|nested] 21+ messages in thread

* [dpdk-ci] [PATCH v6 3/3] pmdinfogen: remove C implementation
  2020-10-04  1:59   ` [dpdk-ci] [PATCH v6 " Dmitry Kozlyuk
  2020-10-04  1:59     ` [dpdk-ci] [PATCH v6 1/3] pmdinfogen: add Python implementation Dmitry Kozlyuk
  2020-10-04  1:59     ` [dpdk-ci] [PATCH v6 2/3] build: use Python pmdinfogen Dmitry Kozlyuk
@ 2020-10-04  1:59     ` Dmitry Kozlyuk
  2020-10-14 14:37     ` [dpdk-ci] [dpdk-dev] [PATCH v6 0/3] pmdinfogen: rewrite in Python Maxime Coquelin
  2020-10-14 18:31     ` [dpdk-ci] [PATCH v7 " Dmitry Kozlyuk
  4 siblings, 0 replies; 21+ messages in thread
From: Dmitry Kozlyuk @ 2020-10-04  1:59 UTC (permalink / raw)
  To: dev
  Cc: ci, Stephen Hemminger, David Marchand, Thomas Monjalon, Dmitry Kozlyuk

Delete the files no longer used in build process.
Add myself as maintainer of new implementation.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 MAINTAINERS                        |   3 +-
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 4 files changed, 2 insertions(+), 590 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 681093d94..24a8a160c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -133,7 +133,8 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <nhorman@tuxdriver.com>
-F: buildtools/pmdinfogen/
+M: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
diff --git a/buildtools/pmdinfogen/meson.build b/buildtools/pmdinfogen/meson.build
deleted file mode 100644
index 670528fac..000000000
--- a/buildtools/pmdinfogen/meson.build
+++ /dev/null
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
diff --git a/buildtools/pmdinfogen/pmdinfogen.c b/buildtools/pmdinfogen/pmdinfogen.c
deleted file mode 100644
index a68d1ea99..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.c
+++ /dev/null
@@ -1,456 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <ctype.h>
-#include <string.h>
-#include <limits.h>
-#include <stdbool.h>
-#include <errno.h>
-#include <libgen.h>
-
-#include <rte_common.h>
-#include "pmdinfogen.h"
-
-#ifdef RTE_ARCH_64
-#define ADDR_SIZE 64
-#else
-#define ADDR_SIZE 32
-#endif
-
-static int use_stdin, use_stdout;
-
-static const char *sym_name(struct elf_info *elf, Elf_Sym *sym)
-{
-	if (sym)
-		return elf->strtab + sym->st_name;
-	else
-		return "(unknown)";
-}
-
-static void *grab_file(const char *filename, unsigned long *size)
-{
-	struct stat st;
-	void *map = MAP_FAILED;
-	int fd = -1;
-
-	if (!use_stdin) {
-		fd = open(filename, O_RDONLY);
-		if (fd < 0)
-			return NULL;
-	} else {
-		/* from stdin, use a temporary file to mmap */
-		FILE *infile;
-		char buffer[1024];
-		int n;
-
-		infile = tmpfile();
-		if (infile == NULL) {
-			perror("tmpfile");
-			return NULL;
-		}
-		fd = dup(fileno(infile));
-		fclose(infile);
-		if (fd < 0)
-			return NULL;
-
-		n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		while (n > 0) {
-			if (write(fd, buffer, n) != n)
-				goto failed;
-			n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		}
-	}
-
-	if (fstat(fd, &st))
-		goto failed;
-
-	*size = st.st_size;
-	map = mmap(NULL, *size, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);
-
-failed:
-	close(fd);
-	if (map == MAP_FAILED)
-		return NULL;
-	return map;
-}
-
-/**
-  * Return a copy of the next line in a mmap'ed file.
-  * spaces in the beginning of the line is trimmed away.
-  * Return a pointer to a static buffer.
-  **/
-static void release_file(void *file, unsigned long size)
-{
-	munmap(file, size);
-}
-
-
-static void *get_sym_value(struct elf_info *info, const Elf_Sym *sym)
-{
-	return RTE_PTR_ADD(info->hdr,
-		info->sechdrs[sym->st_shndx].sh_offset + sym->st_value);
-}
-
-static Elf_Sym *find_sym_in_symtab(struct elf_info *info,
-				   const char *name, Elf_Sym *last)
-{
-	Elf_Sym *idx;
-	if (last)
-		idx = last+1;
-	else
-		idx = info->symtab_start;
-
-	for (; idx < info->symtab_stop; idx++) {
-		const char *n = sym_name(info, idx);
-		if (!strncmp(n, name, strlen(name)))
-			return idx;
-	}
-	return NULL;
-}
-
-static int parse_elf(struct elf_info *info, const char *filename)
-{
-	unsigned int i;
-	Elf_Ehdr *hdr;
-	Elf_Shdr *sechdrs;
-	Elf_Sym  *sym;
-	int endian;
-	unsigned int symtab_idx = ~0U, symtab_shndx_idx = ~0U;
-
-	hdr = grab_file(filename, &info->size);
-	if (!hdr) {
-		perror(filename);
-		exit(1);
-	}
-	info->hdr = hdr;
-	if (info->size < sizeof(*hdr)) {
-		/* file too small, assume this is an empty .o file */
-		return 0;
-	}
-	/* Is this a valid ELF file? */
-	if ((hdr->e_ident[EI_MAG0] != ELFMAG0) ||
-	    (hdr->e_ident[EI_MAG1] != ELFMAG1) ||
-	    (hdr->e_ident[EI_MAG2] != ELFMAG2) ||
-	    (hdr->e_ident[EI_MAG3] != ELFMAG3)) {
-		/* Not an ELF file - silently ignore it */
-		return 0;
-	}
-
-	if (!hdr->e_ident[EI_DATA]) {
-		/* Unknown endian */
-		return 0;
-	}
-
-	endian = hdr->e_ident[EI_DATA];
-
-	/* Fix endianness in ELF header */
-	hdr->e_type      = TO_NATIVE(endian, 16, hdr->e_type);
-	hdr->e_machine   = TO_NATIVE(endian, 16, hdr->e_machine);
-	hdr->e_version   = TO_NATIVE(endian, 32, hdr->e_version);
-	hdr->e_entry     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_entry);
-	hdr->e_phoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_phoff);
-	hdr->e_shoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_shoff);
-	hdr->e_flags     = TO_NATIVE(endian, 32, hdr->e_flags);
-	hdr->e_ehsize    = TO_NATIVE(endian, 16, hdr->e_ehsize);
-	hdr->e_phentsize = TO_NATIVE(endian, 16, hdr->e_phentsize);
-	hdr->e_phnum     = TO_NATIVE(endian, 16, hdr->e_phnum);
-	hdr->e_shentsize = TO_NATIVE(endian, 16, hdr->e_shentsize);
-	hdr->e_shnum     = TO_NATIVE(endian, 16, hdr->e_shnum);
-	hdr->e_shstrndx  = TO_NATIVE(endian, 16, hdr->e_shstrndx);
-
-	sechdrs = RTE_PTR_ADD(hdr, hdr->e_shoff);
-	info->sechdrs = sechdrs;
-
-	/* Check if file offset is correct */
-	if (hdr->e_shoff > info->size) {
-		fprintf(stderr, "section header offset=%lu in file '%s' "
-		      "is bigger than filesize=%lu\n",
-		      (unsigned long)hdr->e_shoff,
-		      filename, info->size);
-		return 0;
-	}
-
-	if (hdr->e_shnum == SHN_UNDEF) {
-		/*
-		 * There are more than 64k sections,
-		 * read count from .sh_size.
-		 */
-		info->num_sections =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[0].sh_size);
-	} else {
-		info->num_sections = hdr->e_shnum;
-	}
-	if (hdr->e_shstrndx == SHN_XINDEX)
-		info->secindex_strings =
-			TO_NATIVE(endian, 32, sechdrs[0].sh_link);
-	else
-		info->secindex_strings = hdr->e_shstrndx;
-
-	/* Fix endianness in section headers */
-	for (i = 0; i < info->num_sections; i++) {
-		sechdrs[i].sh_name      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_name);
-		sechdrs[i].sh_type      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_type);
-		sechdrs[i].sh_flags     =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_flags);
-		sechdrs[i].sh_addr      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addr);
-		sechdrs[i].sh_offset    =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_offset);
-		sechdrs[i].sh_size      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_size);
-		sechdrs[i].sh_link      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_link);
-		sechdrs[i].sh_info      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_info);
-		sechdrs[i].sh_addralign =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addralign);
-		sechdrs[i].sh_entsize   =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_entsize);
-	}
-	/* Find symbol table. */
-	for (i = 1; i < info->num_sections; i++) {
-		int nobits = sechdrs[i].sh_type == SHT_NOBITS;
-
-		if (!nobits && sechdrs[i].sh_offset > info->size) {
-			fprintf(stderr, "%s is truncated. "
-			      "sechdrs[i].sh_offset=%lu > sizeof(*hrd)=%zu\n",
-			      filename, (unsigned long)sechdrs[i].sh_offset,
-			      sizeof(*hdr));
-			return 0;
-		}
-
-		if (sechdrs[i].sh_type == SHT_SYMTAB) {
-			unsigned int sh_link_idx;
-			symtab_idx = i;
-			info->symtab_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-			sh_link_idx = sechdrs[i].sh_link;
-			info->strtab       = RTE_PTR_ADD(hdr,
-				sechdrs[sh_link_idx].sh_offset);
-		}
-
-		/* 32bit section no. table? ("more than 64k sections") */
-		if (sechdrs[i].sh_type == SHT_SYMTAB_SHNDX) {
-			symtab_shndx_idx = i;
-			info->symtab_shndx_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_shndx_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-		}
-	}
-	if (!info->symtab_start)
-		fprintf(stderr, "%s has no symtab?\n", filename);
-	else {
-		/* Fix endianness in symbols */
-		for (sym = info->symtab_start; sym < info->symtab_stop; sym++) {
-			sym->st_shndx = TO_NATIVE(endian, 16, sym->st_shndx);
-			sym->st_name  = TO_NATIVE(endian, 32, sym->st_name);
-			sym->st_value = TO_NATIVE(endian, ADDR_SIZE, sym->st_value);
-			sym->st_size  = TO_NATIVE(endian, ADDR_SIZE, sym->st_size);
-		}
-	}
-
-	if (symtab_shndx_idx != ~0U) {
-		Elf32_Word *p;
-		if (symtab_idx != sechdrs[symtab_shndx_idx].sh_link)
-			fprintf(stderr,
-			      "%s: SYMTAB_SHNDX has bad sh_link: %u!=%u\n",
-			      filename, sechdrs[symtab_shndx_idx].sh_link,
-			      symtab_idx);
-		/* Fix endianness */
-		for (p = info->symtab_shndx_start; p < info->symtab_shndx_stop;
-		     p++)
-			*p = TO_NATIVE(endian, 32, *p);
-	}
-
-	return 1;
-}
-
-static void parse_elf_finish(struct elf_info *info)
-{
-	struct pmd_driver *tmp, *idx = info->drivers;
-	release_file(info->hdr, info->size);
-	while (idx) {
-		tmp = idx->next;
-		free(idx);
-		idx = tmp;
-	}
-}
-
-struct opt_tag {
-	const char *suffix;
-	const char *json_id;
-};
-
-static const struct opt_tag opt_tags[] = {
-	{"_param_string_export", "params"},
-	{"_kmod_dep_export", "kmod"},
-};
-
-static int complete_pmd_entry(struct elf_info *info, struct pmd_driver *drv)
-{
-	const char *tname;
-	int i;
-	char tmpsymname[128];
-	Elf_Sym *tmpsym;
-
-	drv->name = get_sym_value(info, drv->name_sym);
-
-	for (i = 0; i < PMD_OPT_MAX; i++) {
-		memset(tmpsymname, 0, 128);
-		sprintf(tmpsymname, "__%s%s", drv->name, opt_tags[i].suffix);
-		tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-		if (!tmpsym)
-			continue;
-		drv->opt_vals[i] = get_sym_value(info, tmpsym);
-	}
-
-	memset(tmpsymname, 0, 128);
-	sprintf(tmpsymname, "__%s_pci_tbl_export", drv->name);
-
-	tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-
-
-	/*
-	 * If this returns NULL, then this is a PMD_VDEV, because
-	 * it has no pci table reference
-	 */
-	if (!tmpsym) {
-		drv->pci_tbl = NULL;
-		return 0;
-	}
-
-	tname = get_sym_value(info, tmpsym);
-	tmpsym = find_sym_in_symtab(info, tname, NULL);
-	if (!tmpsym)
-		return -ENOENT;
-
-	drv->pci_tbl = (struct rte_pci_id *)get_sym_value(info, tmpsym);
-	if (!drv->pci_tbl)
-		return -ENOENT;
-
-	return 0;
-}
-
-static int locate_pmd_entries(struct elf_info *info)
-{
-	Elf_Sym *last = NULL;
-	struct pmd_driver *new;
-
-	info->drivers = NULL;
-
-	do {
-		new = calloc(sizeof(struct pmd_driver), 1);
-		if (new == NULL) {
-			fprintf(stderr, "Failed to calloc memory\n");
-			return -1;
-		}
-		new->name_sym = find_sym_in_symtab(info, "this_pmd_name", last);
-		last = new->name_sym;
-		if (!new->name_sym)
-			free(new);
-		else {
-			if (complete_pmd_entry(info, new)) {
-				fprintf(stderr,
-					"Failed to complete pmd entry\n");
-				free(new);
-			} else {
-				new->next = info->drivers;
-				info->drivers = new;
-			}
-		}
-	} while (last);
-
-	return 0;
-}
-
-static void output_pmd_info_string(struct elf_info *info, char *outfile)
-{
-	FILE *ofd;
-	struct pmd_driver *drv;
-	struct rte_pci_id *pci_ids;
-	int idx = 0;
-
-	if (use_stdout)
-		ofd = stdout;
-	else {
-		ofd = fopen(outfile, "w+");
-		if (!ofd) {
-			fprintf(stderr, "Unable to open output file\n");
-			return;
-		}
-	}
-
-	drv = info->drivers;
-
-	while (drv) {
-		fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
-			"\"PMD_INFO_STRING= {",
-			drv->name);
-		fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);
-
-		for (idx = 0; idx < PMD_OPT_MAX; idx++) {
-			if (drv->opt_vals[idx])
-				fprintf(ofd, "\\\"%s\\\" : \\\"%s\\\", ",
-					opt_tags[idx].json_id,
-					drv->opt_vals[idx]);
-		}
-
-		pci_ids = drv->pci_tbl;
-		fprintf(ofd, "\\\"pci_ids\\\" : [");
-
-		while (pci_ids && pci_ids->device_id) {
-			fprintf(ofd, "[%d, %d, %d, %d]",
-				pci_ids->vendor_id, pci_ids->device_id,
-				pci_ids->subsystem_vendor_id,
-				pci_ids->subsystem_device_id);
-			pci_ids++;
-			if (pci_ids->device_id)
-				fprintf(ofd, ",");
-			else
-				fprintf(ofd, " ");
-		}
-		fprintf(ofd, "]}\";\n");
-		drv = drv->next;
-	}
-
-	fclose(ofd);
-}
-
-int main(int argc, char **argv)
-{
-	struct elf_info info = {0};
-	int rc = 1;
-
-	if (argc < 3) {
-		fprintf(stderr,
-			"usage: %s <object file> <c output file>\n",
-			basename(argv[0]));
-		exit(127);
-	}
-	use_stdin = !strcmp(argv[1], "-");
-	use_stdout = !strcmp(argv[2], "-");
-	parse_elf(&info, argv[1]);
-
-	if (locate_pmd_entries(&info) < 0)
-		exit(1);
-
-	if (info.drivers) {
-		output_pmd_info_string(&info, argv[2]);
-		rc = 0;
-	} else {
-		fprintf(stderr, "No drivers registered\n");
-	}
-
-	parse_elf_finish(&info);
-	exit(rc);
-}
diff --git a/buildtools/pmdinfogen/pmdinfogen.h b/buildtools/pmdinfogen/pmdinfogen.h
deleted file mode 100644
index 93930e454..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.h
+++ /dev/null
@@ -1,119 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <stdarg.h>
-#include <string.h>
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#ifdef __linux__
-#include <endian.h>
-#else
-#include <sys/endian.h>
-#endif
-#include <fcntl.h>
-#include <unistd.h>
-#include <elf.h>
-#include <rte_pci.h>
-
-/* On BSD-alike OSes elf.h defines these according to host's word size */
-#undef ELF_ST_BIND
-#undef ELF_ST_TYPE
-#undef ELF_R_SYM
-#undef ELF_R_TYPE
-
-/*
- * Define ELF64_* to ELF_*, the latter being defined in both 32 and 64 bit
- * flavors in elf.h.  This makes our code a bit more generic between arches
- * and allows us to support 32 bit code in the future should we ever want to
- */
-#ifdef RTE_ARCH_64
-#define Elf_Ehdr    Elf64_Ehdr
-#define Elf_Shdr    Elf64_Shdr
-#define Elf_Sym     Elf64_Sym
-#define Elf_Addr    Elf64_Addr
-#define Elf_Sword   Elf64_Sxword
-#define Elf_Section Elf64_Half
-#define ELF_ST_BIND ELF64_ST_BIND
-#define ELF_ST_TYPE ELF64_ST_TYPE
-
-#define Elf_Rel     Elf64_Rel
-#define Elf_Rela    Elf64_Rela
-#define ELF_R_SYM   ELF64_R_SYM
-#define ELF_R_TYPE  ELF64_R_TYPE
-#else
-#define Elf_Ehdr    Elf32_Ehdr
-#define Elf_Shdr    Elf32_Shdr
-#define Elf_Sym     Elf32_Sym
-#define Elf_Addr    Elf32_Addr
-#define Elf_Sword   Elf32_Sxword
-#define Elf_Section Elf32_Half
-#define ELF_ST_BIND ELF32_ST_BIND
-#define ELF_ST_TYPE ELF32_ST_TYPE
-
-#define Elf_Rel     Elf32_Rel
-#define Elf_Rela    Elf32_Rela
-#define ELF_R_SYM   ELF32_R_SYM
-#define ELF_R_TYPE  ELF32_R_TYPE
-#endif
-
-
-/*
- * Note, it seems odd that we have both a CONVERT_NATIVE and a TO_NATIVE macro
- * below.  We do this because the values passed to TO_NATIVE may themselves be
- * macros and need both macros here to get expanded.  Specifically its the width
- * variable we are concerned with, because it needs to get expanded prior to
- * string concatenation
- */
-#define CONVERT_NATIVE(fend, width, x) ({ \
-typeof(x) ___x; \
-if ((fend) == ELFDATA2LSB) \
-	___x = le##width##toh(x); \
-else \
-	___x = be##width##toh(x); \
-	___x; \
-})
-
-#define TO_NATIVE(fend, width, x) CONVERT_NATIVE(fend, width, x)
-
-enum opt_params {
-	PMD_PARAM_STRING = 0,
-	PMD_KMOD_DEP,
-	PMD_OPT_MAX
-};
-
-struct pmd_driver {
-	Elf_Sym *name_sym;
-	const char *name;
-	struct rte_pci_id *pci_tbl;
-	struct pmd_driver *next;
-
-	const char *opt_vals[PMD_OPT_MAX];
-};
-
-struct elf_info {
-	unsigned long size;
-	Elf_Ehdr     *hdr;
-	Elf_Shdr     *sechdrs;
-	Elf_Sym      *symtab_start;
-	Elf_Sym      *symtab_stop;
-	char         *strtab;
-
-	/* support for 32bit section numbers */
-
-	unsigned int num_sections; /* max_secindex + 1 */
-	unsigned int secindex_strings;
-	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
-	 * take shndx from symtab_shndx_start[N] instead
-	 */
-	Elf32_Word   *symtab_shndx_start;
-	Elf32_Word   *symtab_shndx_stop;
-
-	struct pmd_driver *drivers;
-};
-- 
2.28.0


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [dpdk-ci] [dpdk-dev] [PATCH v6 0/3] pmdinfogen: rewrite in Python
  2020-10-04  1:59   ` [dpdk-ci] [PATCH v6 " Dmitry Kozlyuk
                       ` (2 preceding siblings ...)
  2020-10-04  1:59     ` [dpdk-ci] [PATCH v6 3/3] pmdinfogen: remove C implementation Dmitry Kozlyuk
@ 2020-10-14 14:37     ` Maxime Coquelin
  2020-10-14 15:40       ` Dmitry Kozlyuk
  2020-10-14 18:31     ` [dpdk-ci] [PATCH v7 " Dmitry Kozlyuk
  4 siblings, 1 reply; 21+ messages in thread
From: Maxime Coquelin @ 2020-10-14 14:37 UTC (permalink / raw)
  To: Dmitry Kozlyuk, dev
  Cc: ci, Stephen Hemminger, David Marchand, Thomas Monjalon, Neil Horman

Hi Dmitry,

On 10/4/20 3:59 AM, Dmitry Kozlyuk wrote:
> This patchset implements existing pmdinfogen logic in Python, replaces
> and removes the old code. The goals of rewriting are:
> 
> * easier maintenance by using a more high-level language,
> * simpler build process without host application and libelf,
> * foundation for adding Windows support.
> 
> Travis CI script is adjusted to install python3-pyelftools, but other CI
> systems may need similar tweaking. Particularly, testing on FreeBSD and
> big-endian targets is desired.
> 
> Acked-by: Neil Horman <nhorman@tuxdriver.com>
> 
> ---
> Changes in v6:
> 
> 1. Address yapf and pylint warnings (Stephen Hemminger).
> 2. Add myself as new implementation maintainer (Thomas Monjalon).
> 
> Dmitry Kozlyuk (3):
>   pmdinfogen: add Python implementation
>   build: use Python pmdinfogen
>   pmdinfogen: remove C implementation
> 
>  .travis.yml                        |   2 +-
>  MAINTAINERS                        |   3 +-
>  buildtools/gen-pmdinfo-cfile.sh    |   6 +-
>  buildtools/meson.build             |   1 +
>  buildtools/pmdinfogen.py           | 189 ++++++++++++
>  buildtools/pmdinfogen/meson.build  |  14 -
>  buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
>  buildtools/pmdinfogen/pmdinfogen.h | 119 --------
>  doc/guides/linux_gsg/sys_reqs.rst  |   6 +
>  drivers/meson.build                |   2 +-
>  meson.build                        |   1 -
>  11 files changed, 203 insertions(+), 596 deletions(-)
>  create mode 100755 buildtools/pmdinfogen.py
>  delete mode 100644 buildtools/pmdinfogen/meson.build
>  delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
>  delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
> 

Ninja fails when pyelftools is not installed.
Shouldn't meson detect and report it before build time?

Thanks,
Maxime


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [dpdk-ci] [dpdk-dev] [PATCH v6 0/3] pmdinfogen: rewrite in Python
  2020-10-14 14:37     ` [dpdk-ci] [dpdk-dev] [PATCH v6 0/3] pmdinfogen: rewrite in Python Maxime Coquelin
@ 2020-10-14 15:40       ` Dmitry Kozlyuk
  0 siblings, 0 replies; 21+ messages in thread
From: Dmitry Kozlyuk @ 2020-10-14 15:40 UTC (permalink / raw)
  To: Maxime Coquelin
  Cc: dev, ci, Stephen Hemminger, David Marchand, Thomas Monjalon, Neil Horman

Hi Maxime,

> Ninja fails when pyelftools is not installed.
> Shouldn't meson detect and report it before build time? 

Will add in v7. Meson 0.51 introduced a terse way to check for Python
dependencies, but it's not hard even with Meson 0.47.1.

https://github.com/mesonbuild/meson/issues/3551#issuecomment-518320640

^ permalink raw reply	[flat|nested] 21+ messages in thread

* [dpdk-ci] [PATCH v7 0/3] pmdinfogen: rewrite in Python
  2020-10-04  1:59   ` [dpdk-ci] [PATCH v6 " Dmitry Kozlyuk
                       ` (3 preceding siblings ...)
  2020-10-14 14:37     ` [dpdk-ci] [dpdk-dev] [PATCH v6 0/3] pmdinfogen: rewrite in Python Maxime Coquelin
@ 2020-10-14 18:31     ` Dmitry Kozlyuk
  2020-10-14 18:31       ` [dpdk-ci] [PATCH v7 1/3] pmdinfogen: add Python implementation Dmitry Kozlyuk
                         ` (4 more replies)
  4 siblings, 5 replies; 21+ messages in thread
From: Dmitry Kozlyuk @ 2020-10-14 18:31 UTC (permalink / raw)
  To: dev
  Cc: ci, Stephen Hemminger, David Marchand, Thomas Monjalon,
	Maxime Coquelin, Dmitry Kozlyuk, Neil Horman

This patchset implements existing pmdinfogen logic in Python, replaces
and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.

Travis CI script is adjusted to install python3-pyelftools, but other CI
systems may need similar tweaking. Particularly, testing on FreeBSD and
big-endian targets is desired.

Acked-by: Neil Horman <nhorman@tuxdriver.com>

---
Changes in v7:

    Add configure-time check for elftools module (Maxime Coquelin).

Dmitry Kozlyuk (3):
  pmdinfogen: add Python implementation
  build: use Python pmdinfogen
  pmdinfogen: remove C implementation

 .travis.yml                        |   2 +-
 MAINTAINERS                        |   3 +-
 buildtools/gen-pmdinfo-cfile.sh    |   6 +-
 buildtools/meson.build             |  15 +
 buildtools/pmdinfogen.py           | 189 ++++++++++++
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 doc/guides/linux_gsg/sys_reqs.rst  |   6 +
 drivers/meson.build                |   2 +-
 meson.build                        |   1 -
 11 files changed, 217 insertions(+), 596 deletions(-)
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

-- 
2.28.0


^ permalink raw reply	[flat|nested] 21+ messages in thread

* [dpdk-ci] [PATCH v7 1/3] pmdinfogen: add Python implementation
  2020-10-14 18:31     ` [dpdk-ci] [PATCH v7 " Dmitry Kozlyuk
@ 2020-10-14 18:31       ` Dmitry Kozlyuk
  2020-10-14 18:31       ` [dpdk-ci] [PATCH v7 2/3] build: use Python pmdinfogen Dmitry Kozlyuk
                         ` (3 subsequent siblings)
  4 siblings, 0 replies; 21+ messages in thread
From: Dmitry Kozlyuk @ 2020-10-14 18:31 UTC (permalink / raw)
  To: dev
  Cc: ci, Stephen Hemminger, David Marchand, Thomas Monjalon,
	Maxime Coquelin, Dmitry Kozlyuk, Neil Horman

Using a high-level, interpreted language simplifies maintenance and
build process. Furthermore, ELF handling is delegated to pyelftools
package. Original logic is kept, the copyright recognizes that.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 buildtools/pmdinfogen.py | 189 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 189 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py

diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
new file mode 100755
index 000000000..0cca47ff1
--- /dev/null
+++ b/buildtools/pmdinfogen.py
@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <nhorman@tuxdriver.com>
+# Copyright (c) 2020 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image.get_section_data(self._symbol["st_shndx"])
+        base = self._symbol["st_value"]
+        return data[base:base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        return value[:-1].decode() if value else ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def get_section_data(self, name):
+        return self._image.get_section(name).data()
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        return ELFSymbol(self, symbol[0]) if symbol else None
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver.pci_ids = cls._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    @staticmethod
+    def _load_pci_ids(image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        result = []
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            result.append([
+                pci_id.vendor_id,
+                pci_id.device_id,
+                pci_id.subsystem_vendor_id,
+                pci_id.subsystem_device_id,
+                ])
+        return result
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
-- 
2.28.0


^ permalink raw reply	[flat|nested] 21+ messages in thread

* [dpdk-ci] [PATCH v7 2/3] build: use Python pmdinfogen
  2020-10-14 18:31     ` [dpdk-ci] [PATCH v7 " Dmitry Kozlyuk
  2020-10-14 18:31       ` [dpdk-ci] [PATCH v7 1/3] pmdinfogen: add Python implementation Dmitry Kozlyuk
@ 2020-10-14 18:31       ` Dmitry Kozlyuk
  2020-10-14 18:31       ` [dpdk-ci] [PATCH v7 3/3] pmdinfogen: remove C implementation Dmitry Kozlyuk
                         ` (2 subsequent siblings)
  4 siblings, 0 replies; 21+ messages in thread
From: Dmitry Kozlyuk @ 2020-10-14 18:31 UTC (permalink / raw)
  To: dev
  Cc: ci, Stephen Hemminger, David Marchand, Thomas Monjalon,
	Maxime Coquelin, Dmitry Kozlyuk, Aaron Conole, Michael Santana,
	Bruce Richardson

Use the same interpreter to run pmdinfogen as for other build scripts.
Adjust wrapper script accordingly and also don't suppress stderr from ar
and pmdinfogen.

Add configure-time check for elftools Python module.

Add python3-pyelftools to CI configuration. The package is available on
all major distributions. FreeBSD has no system requirements section in
its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE)
supported.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 .travis.yml                       |  2 +-
 buildtools/gen-pmdinfo-cfile.sh   |  6 +++---
 buildtools/meson.build            | 15 +++++++++++++++
 doc/guides/linux_gsg/sys_reqs.rst |  6 ++++++
 drivers/meson.build               |  2 +-
 meson.build                       |  1 -
 6 files changed, 26 insertions(+), 6 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 5e12db23b..6744f6fc1 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -11,7 +11,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
diff --git a/buildtools/gen-pmdinfo-cfile.sh b/buildtools/gen-pmdinfo-cfile.sh
index 43059cf36..109ee461e 100755
--- a/buildtools/gen-pmdinfo-cfile.sh
+++ b/buildtools/gen-pmdinfo-cfile.sh
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 04808dabc..0c668eb46 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -17,3 +17,18 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
+
+# TODO: starting from Meson 0.51.0 use
+# 	python3 = import('python').find_installation('python',
+#		modules : python3_required_modules)
+python3_required_modules = ['elftools']
+foreach module : python3_required_modules
+	script = \
+		'import importlib.util;' + \
+		'import sys;' + \
+		'exit(importlib.util.find_spec("@0@") is None)'.format(module)
+	if run_command(py3, '-c', script).returncode() != 0
+		error('missing python module: @0@'.format(module))
+	endif
+endforeach
diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
index 6ecdc04aa..89c9f2570 100644
--- a/doc/guides/linux_gsg/sys_reqs.rst
+++ b/doc/guides/linux_gsg/sys_reqs.rst
@@ -52,6 +52,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
diff --git a/drivers/meson.build b/drivers/meson.build
index 5f9526557..88364259d 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -137,7 +137,7 @@ foreach subpath:subdirs
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
diff --git a/meson.build b/meson.build
index 61d9a4f5f..a6e8c09c2 100644
--- a/meson.build
+++ b/meson.build
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
-- 
2.28.0


^ permalink raw reply	[flat|nested] 21+ messages in thread

* [dpdk-ci] [PATCH v7 3/3] pmdinfogen: remove C implementation
  2020-10-14 18:31     ` [dpdk-ci] [PATCH v7 " Dmitry Kozlyuk
  2020-10-14 18:31       ` [dpdk-ci] [PATCH v7 1/3] pmdinfogen: add Python implementation Dmitry Kozlyuk
  2020-10-14 18:31       ` [dpdk-ci] [PATCH v7 2/3] build: use Python pmdinfogen Dmitry Kozlyuk
@ 2020-10-14 18:31       ` Dmitry Kozlyuk
  2020-10-20 16:02       ` [dpdk-ci] [PATCH v7 0/3] pmdinfogen: rewrite in Python David Marchand
  2020-10-20 17:44       ` [dpdk-ci] [PATCH v8 " Dmitry Kozlyuk
  4 siblings, 0 replies; 21+ messages in thread
From: Dmitry Kozlyuk @ 2020-10-14 18:31 UTC (permalink / raw)
  To: dev
  Cc: ci, Stephen Hemminger, David Marchand, Thomas Monjalon,
	Maxime Coquelin, Dmitry Kozlyuk

Delete the files no longer used in build process.
Add myself as maintainer of new implementation.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 MAINTAINERS                        |   3 +-
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 4 files changed, 2 insertions(+), 590 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 499c7d297..5f5c599d2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -133,7 +133,8 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <nhorman@tuxdriver.com>
-F: buildtools/pmdinfogen/
+M: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
diff --git a/buildtools/pmdinfogen/meson.build b/buildtools/pmdinfogen/meson.build
deleted file mode 100644
index 670528fac..000000000
--- a/buildtools/pmdinfogen/meson.build
+++ /dev/null
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
diff --git a/buildtools/pmdinfogen/pmdinfogen.c b/buildtools/pmdinfogen/pmdinfogen.c
deleted file mode 100644
index a68d1ea99..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.c
+++ /dev/null
@@ -1,456 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <ctype.h>
-#include <string.h>
-#include <limits.h>
-#include <stdbool.h>
-#include <errno.h>
-#include <libgen.h>
-
-#include <rte_common.h>
-#include "pmdinfogen.h"
-
-#ifdef RTE_ARCH_64
-#define ADDR_SIZE 64
-#else
-#define ADDR_SIZE 32
-#endif
-
-static int use_stdin, use_stdout;
-
-static const char *sym_name(struct elf_info *elf, Elf_Sym *sym)
-{
-	if (sym)
-		return elf->strtab + sym->st_name;
-	else
-		return "(unknown)";
-}
-
-static void *grab_file(const char *filename, unsigned long *size)
-{
-	struct stat st;
-	void *map = MAP_FAILED;
-	int fd = -1;
-
-	if (!use_stdin) {
-		fd = open(filename, O_RDONLY);
-		if (fd < 0)
-			return NULL;
-	} else {
-		/* from stdin, use a temporary file to mmap */
-		FILE *infile;
-		char buffer[1024];
-		int n;
-
-		infile = tmpfile();
-		if (infile == NULL) {
-			perror("tmpfile");
-			return NULL;
-		}
-		fd = dup(fileno(infile));
-		fclose(infile);
-		if (fd < 0)
-			return NULL;
-
-		n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		while (n > 0) {
-			if (write(fd, buffer, n) != n)
-				goto failed;
-			n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		}
-	}
-
-	if (fstat(fd, &st))
-		goto failed;
-
-	*size = st.st_size;
-	map = mmap(NULL, *size, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);
-
-failed:
-	close(fd);
-	if (map == MAP_FAILED)
-		return NULL;
-	return map;
-}
-
-/**
-  * Return a copy of the next line in a mmap'ed file.
-  * spaces in the beginning of the line is trimmed away.
-  * Return a pointer to a static buffer.
-  **/
-static void release_file(void *file, unsigned long size)
-{
-	munmap(file, size);
-}
-
-
-static void *get_sym_value(struct elf_info *info, const Elf_Sym *sym)
-{
-	return RTE_PTR_ADD(info->hdr,
-		info->sechdrs[sym->st_shndx].sh_offset + sym->st_value);
-}
-
-static Elf_Sym *find_sym_in_symtab(struct elf_info *info,
-				   const char *name, Elf_Sym *last)
-{
-	Elf_Sym *idx;
-	if (last)
-		idx = last+1;
-	else
-		idx = info->symtab_start;
-
-	for (; idx < info->symtab_stop; idx++) {
-		const char *n = sym_name(info, idx);
-		if (!strncmp(n, name, strlen(name)))
-			return idx;
-	}
-	return NULL;
-}
-
-static int parse_elf(struct elf_info *info, const char *filename)
-{
-	unsigned int i;
-	Elf_Ehdr *hdr;
-	Elf_Shdr *sechdrs;
-	Elf_Sym  *sym;
-	int endian;
-	unsigned int symtab_idx = ~0U, symtab_shndx_idx = ~0U;
-
-	hdr = grab_file(filename, &info->size);
-	if (!hdr) {
-		perror(filename);
-		exit(1);
-	}
-	info->hdr = hdr;
-	if (info->size < sizeof(*hdr)) {
-		/* file too small, assume this is an empty .o file */
-		return 0;
-	}
-	/* Is this a valid ELF file? */
-	if ((hdr->e_ident[EI_MAG0] != ELFMAG0) ||
-	    (hdr->e_ident[EI_MAG1] != ELFMAG1) ||
-	    (hdr->e_ident[EI_MAG2] != ELFMAG2) ||
-	    (hdr->e_ident[EI_MAG3] != ELFMAG3)) {
-		/* Not an ELF file - silently ignore it */
-		return 0;
-	}
-
-	if (!hdr->e_ident[EI_DATA]) {
-		/* Unknown endian */
-		return 0;
-	}
-
-	endian = hdr->e_ident[EI_DATA];
-
-	/* Fix endianness in ELF header */
-	hdr->e_type      = TO_NATIVE(endian, 16, hdr->e_type);
-	hdr->e_machine   = TO_NATIVE(endian, 16, hdr->e_machine);
-	hdr->e_version   = TO_NATIVE(endian, 32, hdr->e_version);
-	hdr->e_entry     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_entry);
-	hdr->e_phoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_phoff);
-	hdr->e_shoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_shoff);
-	hdr->e_flags     = TO_NATIVE(endian, 32, hdr->e_flags);
-	hdr->e_ehsize    = TO_NATIVE(endian, 16, hdr->e_ehsize);
-	hdr->e_phentsize = TO_NATIVE(endian, 16, hdr->e_phentsize);
-	hdr->e_phnum     = TO_NATIVE(endian, 16, hdr->e_phnum);
-	hdr->e_shentsize = TO_NATIVE(endian, 16, hdr->e_shentsize);
-	hdr->e_shnum     = TO_NATIVE(endian, 16, hdr->e_shnum);
-	hdr->e_shstrndx  = TO_NATIVE(endian, 16, hdr->e_shstrndx);
-
-	sechdrs = RTE_PTR_ADD(hdr, hdr->e_shoff);
-	info->sechdrs = sechdrs;
-
-	/* Check if file offset is correct */
-	if (hdr->e_shoff > info->size) {
-		fprintf(stderr, "section header offset=%lu in file '%s' "
-		      "is bigger than filesize=%lu\n",
-		      (unsigned long)hdr->e_shoff,
-		      filename, info->size);
-		return 0;
-	}
-
-	if (hdr->e_shnum == SHN_UNDEF) {
-		/*
-		 * There are more than 64k sections,
-		 * read count from .sh_size.
-		 */
-		info->num_sections =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[0].sh_size);
-	} else {
-		info->num_sections = hdr->e_shnum;
-	}
-	if (hdr->e_shstrndx == SHN_XINDEX)
-		info->secindex_strings =
-			TO_NATIVE(endian, 32, sechdrs[0].sh_link);
-	else
-		info->secindex_strings = hdr->e_shstrndx;
-
-	/* Fix endianness in section headers */
-	for (i = 0; i < info->num_sections; i++) {
-		sechdrs[i].sh_name      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_name);
-		sechdrs[i].sh_type      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_type);
-		sechdrs[i].sh_flags     =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_flags);
-		sechdrs[i].sh_addr      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addr);
-		sechdrs[i].sh_offset    =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_offset);
-		sechdrs[i].sh_size      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_size);
-		sechdrs[i].sh_link      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_link);
-		sechdrs[i].sh_info      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_info);
-		sechdrs[i].sh_addralign =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addralign);
-		sechdrs[i].sh_entsize   =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_entsize);
-	}
-	/* Find symbol table. */
-	for (i = 1; i < info->num_sections; i++) {
-		int nobits = sechdrs[i].sh_type == SHT_NOBITS;
-
-		if (!nobits && sechdrs[i].sh_offset > info->size) {
-			fprintf(stderr, "%s is truncated. "
-			      "sechdrs[i].sh_offset=%lu > sizeof(*hrd)=%zu\n",
-			      filename, (unsigned long)sechdrs[i].sh_offset,
-			      sizeof(*hdr));
-			return 0;
-		}
-
-		if (sechdrs[i].sh_type == SHT_SYMTAB) {
-			unsigned int sh_link_idx;
-			symtab_idx = i;
-			info->symtab_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-			sh_link_idx = sechdrs[i].sh_link;
-			info->strtab       = RTE_PTR_ADD(hdr,
-				sechdrs[sh_link_idx].sh_offset);
-		}
-
-		/* 32bit section no. table? ("more than 64k sections") */
-		if (sechdrs[i].sh_type == SHT_SYMTAB_SHNDX) {
-			symtab_shndx_idx = i;
-			info->symtab_shndx_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_shndx_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-		}
-	}
-	if (!info->symtab_start)
-		fprintf(stderr, "%s has no symtab?\n", filename);
-	else {
-		/* Fix endianness in symbols */
-		for (sym = info->symtab_start; sym < info->symtab_stop; sym++) {
-			sym->st_shndx = TO_NATIVE(endian, 16, sym->st_shndx);
-			sym->st_name  = TO_NATIVE(endian, 32, sym->st_name);
-			sym->st_value = TO_NATIVE(endian, ADDR_SIZE, sym->st_value);
-			sym->st_size  = TO_NATIVE(endian, ADDR_SIZE, sym->st_size);
-		}
-	}
-
-	if (symtab_shndx_idx != ~0U) {
-		Elf32_Word *p;
-		if (symtab_idx != sechdrs[symtab_shndx_idx].sh_link)
-			fprintf(stderr,
-			      "%s: SYMTAB_SHNDX has bad sh_link: %u!=%u\n",
-			      filename, sechdrs[symtab_shndx_idx].sh_link,
-			      symtab_idx);
-		/* Fix endianness */
-		for (p = info->symtab_shndx_start; p < info->symtab_shndx_stop;
-		     p++)
-			*p = TO_NATIVE(endian, 32, *p);
-	}
-
-	return 1;
-}
-
-static void parse_elf_finish(struct elf_info *info)
-{
-	struct pmd_driver *tmp, *idx = info->drivers;
-	release_file(info->hdr, info->size);
-	while (idx) {
-		tmp = idx->next;
-		free(idx);
-		idx = tmp;
-	}
-}
-
-struct opt_tag {
-	const char *suffix;
-	const char *json_id;
-};
-
-static const struct opt_tag opt_tags[] = {
-	{"_param_string_export", "params"},
-	{"_kmod_dep_export", "kmod"},
-};
-
-static int complete_pmd_entry(struct elf_info *info, struct pmd_driver *drv)
-{
-	const char *tname;
-	int i;
-	char tmpsymname[128];
-	Elf_Sym *tmpsym;
-
-	drv->name = get_sym_value(info, drv->name_sym);
-
-	for (i = 0; i < PMD_OPT_MAX; i++) {
-		memset(tmpsymname, 0, 128);
-		sprintf(tmpsymname, "__%s%s", drv->name, opt_tags[i].suffix);
-		tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-		if (!tmpsym)
-			continue;
-		drv->opt_vals[i] = get_sym_value(info, tmpsym);
-	}
-
-	memset(tmpsymname, 0, 128);
-	sprintf(tmpsymname, "__%s_pci_tbl_export", drv->name);
-
-	tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-
-
-	/*
-	 * If this returns NULL, then this is a PMD_VDEV, because
-	 * it has no pci table reference
-	 */
-	if (!tmpsym) {
-		drv->pci_tbl = NULL;
-		return 0;
-	}
-
-	tname = get_sym_value(info, tmpsym);
-	tmpsym = find_sym_in_symtab(info, tname, NULL);
-	if (!tmpsym)
-		return -ENOENT;
-
-	drv->pci_tbl = (struct rte_pci_id *)get_sym_value(info, tmpsym);
-	if (!drv->pci_tbl)
-		return -ENOENT;
-
-	return 0;
-}
-
-static int locate_pmd_entries(struct elf_info *info)
-{
-	Elf_Sym *last = NULL;
-	struct pmd_driver *new;
-
-	info->drivers = NULL;
-
-	do {
-		new = calloc(sizeof(struct pmd_driver), 1);
-		if (new == NULL) {
-			fprintf(stderr, "Failed to calloc memory\n");
-			return -1;
-		}
-		new->name_sym = find_sym_in_symtab(info, "this_pmd_name", last);
-		last = new->name_sym;
-		if (!new->name_sym)
-			free(new);
-		else {
-			if (complete_pmd_entry(info, new)) {
-				fprintf(stderr,
-					"Failed to complete pmd entry\n");
-				free(new);
-			} else {
-				new->next = info->drivers;
-				info->drivers = new;
-			}
-		}
-	} while (last);
-
-	return 0;
-}
-
-static void output_pmd_info_string(struct elf_info *info, char *outfile)
-{
-	FILE *ofd;
-	struct pmd_driver *drv;
-	struct rte_pci_id *pci_ids;
-	int idx = 0;
-
-	if (use_stdout)
-		ofd = stdout;
-	else {
-		ofd = fopen(outfile, "w+");
-		if (!ofd) {
-			fprintf(stderr, "Unable to open output file\n");
-			return;
-		}
-	}
-
-	drv = info->drivers;
-
-	while (drv) {
-		fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
-			"\"PMD_INFO_STRING= {",
-			drv->name);
-		fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);
-
-		for (idx = 0; idx < PMD_OPT_MAX; idx++) {
-			if (drv->opt_vals[idx])
-				fprintf(ofd, "\\\"%s\\\" : \\\"%s\\\", ",
-					opt_tags[idx].json_id,
-					drv->opt_vals[idx]);
-		}
-
-		pci_ids = drv->pci_tbl;
-		fprintf(ofd, "\\\"pci_ids\\\" : [");
-
-		while (pci_ids && pci_ids->device_id) {
-			fprintf(ofd, "[%d, %d, %d, %d]",
-				pci_ids->vendor_id, pci_ids->device_id,
-				pci_ids->subsystem_vendor_id,
-				pci_ids->subsystem_device_id);
-			pci_ids++;
-			if (pci_ids->device_id)
-				fprintf(ofd, ",");
-			else
-				fprintf(ofd, " ");
-		}
-		fprintf(ofd, "]}\";\n");
-		drv = drv->next;
-	}
-
-	fclose(ofd);
-}
-
-int main(int argc, char **argv)
-{
-	struct elf_info info = {0};
-	int rc = 1;
-
-	if (argc < 3) {
-		fprintf(stderr,
-			"usage: %s <object file> <c output file>\n",
-			basename(argv[0]));
-		exit(127);
-	}
-	use_stdin = !strcmp(argv[1], "-");
-	use_stdout = !strcmp(argv[2], "-");
-	parse_elf(&info, argv[1]);
-
-	if (locate_pmd_entries(&info) < 0)
-		exit(1);
-
-	if (info.drivers) {
-		output_pmd_info_string(&info, argv[2]);
-		rc = 0;
-	} else {
-		fprintf(stderr, "No drivers registered\n");
-	}
-
-	parse_elf_finish(&info);
-	exit(rc);
-}
diff --git a/buildtools/pmdinfogen/pmdinfogen.h b/buildtools/pmdinfogen/pmdinfogen.h
deleted file mode 100644
index 38fd3ddcc..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.h
+++ /dev/null
@@ -1,119 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <stdarg.h>
-#include <string.h>
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#ifdef __linux__
-#include <endian.h>
-#else
-#include <sys/endian.h>
-#endif
-#include <fcntl.h>
-#include <unistd.h>
-#include <elf.h>
-#include <rte_pci.h>
-
-/* On BSD-alike OSes elf.h defines these according to host's word size */
-#undef ELF_ST_BIND
-#undef ELF_ST_TYPE
-#undef ELF_R_SYM
-#undef ELF_R_TYPE
-
-/*
- * Define ELF64_* to ELF_*, the latter being defined in both 32 and 64 bit
- * flavors in elf.h.  This makes our code a bit more generic between arches
- * and allows us to support 32 bit code in the future should we ever want to
- */
-#ifdef RTE_ARCH_64
-#define Elf_Ehdr    Elf64_Ehdr
-#define Elf_Shdr    Elf64_Shdr
-#define Elf_Sym     Elf64_Sym
-#define Elf_Addr    Elf64_Addr
-#define Elf_Sword   Elf64_Sxword
-#define Elf_Section Elf64_Half
-#define ELF_ST_BIND ELF64_ST_BIND
-#define ELF_ST_TYPE ELF64_ST_TYPE
-
-#define Elf_Rel     Elf64_Rel
-#define Elf_Rela    Elf64_Rela
-#define ELF_R_SYM   ELF64_R_SYM
-#define ELF_R_TYPE  ELF64_R_TYPE
-#else
-#define Elf_Ehdr    Elf32_Ehdr
-#define Elf_Shdr    Elf32_Shdr
-#define Elf_Sym     Elf32_Sym
-#define Elf_Addr    Elf32_Addr
-#define Elf_Sword   Elf32_Sxword
-#define Elf_Section Elf32_Half
-#define ELF_ST_BIND ELF32_ST_BIND
-#define ELF_ST_TYPE ELF32_ST_TYPE
-
-#define Elf_Rel     Elf32_Rel
-#define Elf_Rela    Elf32_Rela
-#define ELF_R_SYM   ELF32_R_SYM
-#define ELF_R_TYPE  ELF32_R_TYPE
-#endif
-
-
-/*
- * Note, it seems odd that we have both a CONVERT_NATIVE and a TO_NATIVE macro
- * below.  We do this because the values passed to TO_NATIVE may themselves be
- * macros and need both macros here to get expanded.  Specifically its the width
- * variable we are concerned with, because it needs to get expanded prior to
- * string concatenation
- */
-#define CONVERT_NATIVE(fend, width, x) ({ \
-typeof(x) ___x; \
-if ((fend) == ELFDATA2LSB) \
-	___x = le##width##toh(x); \
-else \
-	___x = be##width##toh(x); \
-___x; \
-})
-
-#define TO_NATIVE(fend, width, x) CONVERT_NATIVE(fend, width, x)
-
-enum opt_params {
-	PMD_PARAM_STRING = 0,
-	PMD_KMOD_DEP,
-	PMD_OPT_MAX
-};
-
-struct pmd_driver {
-	Elf_Sym *name_sym;
-	const char *name;
-	struct rte_pci_id *pci_tbl;
-	struct pmd_driver *next;
-
-	const char *opt_vals[PMD_OPT_MAX];
-};
-
-struct elf_info {
-	unsigned long size;
-	Elf_Ehdr     *hdr;
-	Elf_Shdr     *sechdrs;
-	Elf_Sym      *symtab_start;
-	Elf_Sym      *symtab_stop;
-	char         *strtab;
-
-	/* support for 32bit section numbers */
-
-	unsigned int num_sections; /* max_secindex + 1 */
-	unsigned int secindex_strings;
-	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
-	 * take shndx from symtab_shndx_start[N] instead
-	 */
-	Elf32_Word   *symtab_shndx_start;
-	Elf32_Word   *symtab_shndx_stop;
-
-	struct pmd_driver *drivers;
-};
-- 
2.28.0


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [dpdk-ci] [PATCH v7 0/3] pmdinfogen: rewrite in Python
  2020-10-14 18:31     ` [dpdk-ci] [PATCH v7 " Dmitry Kozlyuk
                         ` (2 preceding siblings ...)
  2020-10-14 18:31       ` [dpdk-ci] [PATCH v7 3/3] pmdinfogen: remove C implementation Dmitry Kozlyuk
@ 2020-10-20 16:02       ` David Marchand
  2020-10-20 17:45         ` Dmitry Kozlyuk
  2020-10-20 22:09         ` Dmitry Kozlyuk
  2020-10-20 17:44       ` [dpdk-ci] [PATCH v8 " Dmitry Kozlyuk
  4 siblings, 2 replies; 21+ messages in thread
From: David Marchand @ 2020-10-20 16:02 UTC (permalink / raw)
  To: Dmitry Kozlyuk
  Cc: dev, ci, Chen, Zhaoyan, Stephen Hemminger, Thomas Monjalon,
	Maxime Coquelin, Neil Horman, Brandon Lo, Lincoln Lavoie

Hello Dmitry,

On Wed, Oct 14, 2020 at 8:31 PM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
>
> This patchset implements existing pmdinfogen logic in Python, replaces
> and removes the old code. The goals of rewriting are:
>
> * easier maintenance by using a more high-level language,
> * simpler build process without host application and libelf,
> * foundation for adding Windows support.
>
> Travis CI script is adjusted to install python3-pyelftools, but other CI
> systems may need similar tweaking. Particularly, testing on FreeBSD and
> big-endian targets is desired.
>

- Travis reported an error in meson:
https://travis-ci.com/github/ovsrobot/dpdk/jobs/399762460#L478

Build machine cpu: x86_64
buildtools/meson.build:27:10: ERROR:  Expecting endforeach got eol_cont.
script = \
          ^
For a block that started at 26,0
foreach module : python3_required_modules
^


- In a thread on ci@dpdk.org
http://inbox.dpdk.org/ci/CAOeXdvbosqRGouSX06_phVTCam57XUt0s=sf_kpe-VmiwFTgMA@mail.gmail.com/T/#m9bb193c74ca979c151f2dbfa35664f95093a9e71,
UNH people confirmed that python3-elftools should be available, but
the last test had an error too, which might be linked to incorrect
meson syntax:
http://mails.dpdk.org/archives/test-report/2020-October/159038.html

ninja: error: loading 'build.ninja': No such file or directory


- As far as Intel CI systems are concerned, there might be an issue
with the Windows system:
http://mails.dpdk.org/archives/test-report/2020-October/159123.html

OS: WIN10-64
Target: x86_64-windows-clang
Program binutils-avx512-check.sh found: YES
Program python3 found: YES (c:\program files\python37\python.exe)

buildtools\meson.build:32:2: ERROR: Problem encountered: missing
python module: elftools

A full log can be found at C:\windows-dpdk\dpdk\build\meson-logs\meson-log.txt
ninja: Entering directory `build'
DPDK STV team



-- 
David Marchand


^ permalink raw reply	[flat|nested] 21+ messages in thread

* [dpdk-ci] [PATCH v8 0/3] pmdinfogen: rewrite in Python
  2020-10-14 18:31     ` [dpdk-ci] [PATCH v7 " Dmitry Kozlyuk
                         ` (3 preceding siblings ...)
  2020-10-20 16:02       ` [dpdk-ci] [PATCH v7 0/3] pmdinfogen: rewrite in Python David Marchand
@ 2020-10-20 17:44       ` Dmitry Kozlyuk
  2020-10-20 17:44         ` [dpdk-ci] [PATCH v8 1/3] pmdinfogen: add Python implementation Dmitry Kozlyuk
                           ` (2 more replies)
  4 siblings, 3 replies; 21+ messages in thread
From: Dmitry Kozlyuk @ 2020-10-20 17:44 UTC (permalink / raw)
  To: dev
  Cc: ci, Stephen Hemminger, David Marchand, Thomas Monjalon,
	Maxime Coquelin, Dmitry Kozlyuk, Neil Horman

This patchset implements existing pmdinfogen logic in Python, replaces
and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.

Travis CI script is adjusted to install python3-pyelftools, but other CI
systems may need similar tweaking. Particularly, testing on FreeBSD and
big-endian targets is desired.

Acked-by: Neil Horman <nhorman@tuxdriver.com>

---
Changes in v8:

    * Fix syntax for meson 0.47.1 (CI).
    * Disable configure-time check for pyelftools for Windows (CI).

Dmitry Kozlyuk (3):
  pmdinfogen: add Python implementation
  build: use Python pmdinfogen
  pmdinfogen: remove C implementation

 .travis.yml                        |   2 +-
 MAINTAINERS                        |   3 +-
 buildtools/gen-pmdinfo-cfile.sh    |   6 +-
 buildtools/meson.build             |  15 +
 buildtools/pmdinfogen.py           | 189 ++++++++++++
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 doc/guides/linux_gsg/sys_reqs.rst  |   6 +
 drivers/meson.build                |   2 +-
 meson.build                        |   1 -
 11 files changed, 217 insertions(+), 596 deletions(-)
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

-- 
2.28.0


^ permalink raw reply	[flat|nested] 21+ messages in thread

* [dpdk-ci] [PATCH v8 1/3] pmdinfogen: add Python implementation
  2020-10-20 17:44       ` [dpdk-ci] [PATCH v8 " Dmitry Kozlyuk
@ 2020-10-20 17:44         ` Dmitry Kozlyuk
  2020-10-20 17:44         ` [dpdk-ci] [PATCH v8 2/3] build: use Python pmdinfogen Dmitry Kozlyuk
  2020-10-20 17:44         ` [dpdk-ci] [PATCH v8 3/3] pmdinfogen: remove C implementation Dmitry Kozlyuk
  2 siblings, 0 replies; 21+ messages in thread
From: Dmitry Kozlyuk @ 2020-10-20 17:44 UTC (permalink / raw)
  To: dev
  Cc: ci, Stephen Hemminger, David Marchand, Thomas Monjalon,
	Maxime Coquelin, Dmitry Kozlyuk, Neil Horman

Using a high-level, interpreted language simplifies maintenance and
build process. Furthermore, ELF handling is delegated to pyelftools
package. Original logic is kept, the copyright recognizes that.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 buildtools/pmdinfogen.py | 189 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 189 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py

diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
new file mode 100755
index 000000000..0cca47ff1
--- /dev/null
+++ b/buildtools/pmdinfogen.py
@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <nhorman@tuxdriver.com>
+# Copyright (c) 2020 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image.get_section_data(self._symbol["st_shndx"])
+        base = self._symbol["st_value"]
+        return data[base:base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        return value[:-1].decode() if value else ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def get_section_data(self, name):
+        return self._image.get_section(name).data()
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        return ELFSymbol(self, symbol[0]) if symbol else None
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver.pci_ids = cls._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    @staticmethod
+    def _load_pci_ids(image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        result = []
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            result.append([
+                pci_id.vendor_id,
+                pci_id.device_id,
+                pci_id.subsystem_vendor_id,
+                pci_id.subsystem_device_id,
+                ])
+        return result
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
-- 
2.28.0


^ permalink raw reply	[flat|nested] 21+ messages in thread

* [dpdk-ci] [PATCH v8 2/3] build: use Python pmdinfogen
  2020-10-20 17:44       ` [dpdk-ci] [PATCH v8 " Dmitry Kozlyuk
  2020-10-20 17:44         ` [dpdk-ci] [PATCH v8 1/3] pmdinfogen: add Python implementation Dmitry Kozlyuk
@ 2020-10-20 17:44         ` Dmitry Kozlyuk
  2020-10-21  9:00           ` Bruce Richardson
  2021-01-20  0:05           ` [dpdk-ci] [dpdk-dev] " Thomas Monjalon
  2020-10-20 17:44         ` [dpdk-ci] [PATCH v8 3/3] pmdinfogen: remove C implementation Dmitry Kozlyuk
  2 siblings, 2 replies; 21+ messages in thread
From: Dmitry Kozlyuk @ 2020-10-20 17:44 UTC (permalink / raw)
  To: dev
  Cc: ci, Stephen Hemminger, David Marchand, Thomas Monjalon,
	Maxime Coquelin, Dmitry Kozlyuk, Aaron Conole, Michael Santana,
	Bruce Richardson

Use the same interpreter to run pmdinfogen as for other build scripts.
Adjust wrapper script accordingly and also don't suppress stderr from ar
and pmdinfogen.

Add configure-time check for elftools Python module for Unix hosts.

Add python3-pyelftools to CI configuration. The package is available on
all major distributions. FreeBSD has no system requirements section in
its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE)
supported.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 .travis.yml                       |  2 +-
 buildtools/gen-pmdinfo-cfile.sh   |  6 +++---
 buildtools/meson.build            | 15 +++++++++++++++
 doc/guides/linux_gsg/sys_reqs.rst |  6 ++++++
 drivers/meson.build               |  2 +-
 meson.build                       |  1 -
 6 files changed, 26 insertions(+), 6 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 5e12db23b..6744f6fc1 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -11,7 +11,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
diff --git a/buildtools/gen-pmdinfo-cfile.sh b/buildtools/gen-pmdinfo-cfile.sh
index 43059cf36..109ee461e 100755
--- a/buildtools/gen-pmdinfo-cfile.sh
+++ b/buildtools/gen-pmdinfo-cfile.sh
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 04808dabc..dd4c0f640 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -17,3 +17,18 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
+
+# TODO: starting from Meson 0.51.0 use
+# 	python3 = import('python').find_installation('python',
+#		modules : python3_required_modules)
+python3_required_modules = []
+if host_machine.system() != 'windows'
+	python3_required_modules = ['elftools']
+endif
+foreach module : python3_required_modules
+	script = 'import importlib.util; import sys; exit(importlib.util.find_spec("@0@") is None)'
+	if run_command(py3, '-c', script.format(module)).returncode() != 0
+		error('missing python module: @0@'.format(module))
+	endif
+endforeach
diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
index 6ecdc04aa..89c9f2570 100644
--- a/doc/guides/linux_gsg/sys_reqs.rst
+++ b/doc/guides/linux_gsg/sys_reqs.rst
@@ -52,6 +52,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
diff --git a/drivers/meson.build b/drivers/meson.build
index a5a6fed06..94403f7ac 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -152,7 +152,7 @@ foreach subpath:subdirs
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
diff --git a/meson.build b/meson.build
index 61d9a4f5f..a6e8c09c2 100644
--- a/meson.build
+++ b/meson.build
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
-- 
2.28.0


^ permalink raw reply	[flat|nested] 21+ messages in thread

* [dpdk-ci] [PATCH v8 3/3] pmdinfogen: remove C implementation
  2020-10-20 17:44       ` [dpdk-ci] [PATCH v8 " Dmitry Kozlyuk
  2020-10-20 17:44         ` [dpdk-ci] [PATCH v8 1/3] pmdinfogen: add Python implementation Dmitry Kozlyuk
  2020-10-20 17:44         ` [dpdk-ci] [PATCH v8 2/3] build: use Python pmdinfogen Dmitry Kozlyuk
@ 2020-10-20 17:44         ` Dmitry Kozlyuk
  2 siblings, 0 replies; 21+ messages in thread
From: Dmitry Kozlyuk @ 2020-10-20 17:44 UTC (permalink / raw)
  To: dev
  Cc: ci, Stephen Hemminger, David Marchand, Thomas Monjalon,
	Maxime Coquelin, Dmitry Kozlyuk

Delete the files no longer used in build process.
Add myself as maintainer of new implementation.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 MAINTAINERS                        |   3 +-
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 4 files changed, 2 insertions(+), 590 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

diff --git a/MAINTAINERS b/MAINTAINERS
index a3d192747..996f732fd 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -133,7 +133,8 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <nhorman@tuxdriver.com>
-F: buildtools/pmdinfogen/
+M: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
diff --git a/buildtools/pmdinfogen/meson.build b/buildtools/pmdinfogen/meson.build
deleted file mode 100644
index 670528fac..000000000
--- a/buildtools/pmdinfogen/meson.build
+++ /dev/null
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
diff --git a/buildtools/pmdinfogen/pmdinfogen.c b/buildtools/pmdinfogen/pmdinfogen.c
deleted file mode 100644
index a68d1ea99..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.c
+++ /dev/null
@@ -1,456 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <ctype.h>
-#include <string.h>
-#include <limits.h>
-#include <stdbool.h>
-#include <errno.h>
-#include <libgen.h>
-
-#include <rte_common.h>
-#include "pmdinfogen.h"
-
-#ifdef RTE_ARCH_64
-#define ADDR_SIZE 64
-#else
-#define ADDR_SIZE 32
-#endif
-
-static int use_stdin, use_stdout;
-
-static const char *sym_name(struct elf_info *elf, Elf_Sym *sym)
-{
-	if (sym)
-		return elf->strtab + sym->st_name;
-	else
-		return "(unknown)";
-}
-
-static void *grab_file(const char *filename, unsigned long *size)
-{
-	struct stat st;
-	void *map = MAP_FAILED;
-	int fd = -1;
-
-	if (!use_stdin) {
-		fd = open(filename, O_RDONLY);
-		if (fd < 0)
-			return NULL;
-	} else {
-		/* from stdin, use a temporary file to mmap */
-		FILE *infile;
-		char buffer[1024];
-		int n;
-
-		infile = tmpfile();
-		if (infile == NULL) {
-			perror("tmpfile");
-			return NULL;
-		}
-		fd = dup(fileno(infile));
-		fclose(infile);
-		if (fd < 0)
-			return NULL;
-
-		n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		while (n > 0) {
-			if (write(fd, buffer, n) != n)
-				goto failed;
-			n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		}
-	}
-
-	if (fstat(fd, &st))
-		goto failed;
-
-	*size = st.st_size;
-	map = mmap(NULL, *size, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);
-
-failed:
-	close(fd);
-	if (map == MAP_FAILED)
-		return NULL;
-	return map;
-}
-
-/**
-  * Return a copy of the next line in a mmap'ed file.
-  * spaces in the beginning of the line is trimmed away.
-  * Return a pointer to a static buffer.
-  **/
-static void release_file(void *file, unsigned long size)
-{
-	munmap(file, size);
-}
-
-
-static void *get_sym_value(struct elf_info *info, const Elf_Sym *sym)
-{
-	return RTE_PTR_ADD(info->hdr,
-		info->sechdrs[sym->st_shndx].sh_offset + sym->st_value);
-}
-
-static Elf_Sym *find_sym_in_symtab(struct elf_info *info,
-				   const char *name, Elf_Sym *last)
-{
-	Elf_Sym *idx;
-	if (last)
-		idx = last+1;
-	else
-		idx = info->symtab_start;
-
-	for (; idx < info->symtab_stop; idx++) {
-		const char *n = sym_name(info, idx);
-		if (!strncmp(n, name, strlen(name)))
-			return idx;
-	}
-	return NULL;
-}
-
-static int parse_elf(struct elf_info *info, const char *filename)
-{
-	unsigned int i;
-	Elf_Ehdr *hdr;
-	Elf_Shdr *sechdrs;
-	Elf_Sym  *sym;
-	int endian;
-	unsigned int symtab_idx = ~0U, symtab_shndx_idx = ~0U;
-
-	hdr = grab_file(filename, &info->size);
-	if (!hdr) {
-		perror(filename);
-		exit(1);
-	}
-	info->hdr = hdr;
-	if (info->size < sizeof(*hdr)) {
-		/* file too small, assume this is an empty .o file */
-		return 0;
-	}
-	/* Is this a valid ELF file? */
-	if ((hdr->e_ident[EI_MAG0] != ELFMAG0) ||
-	    (hdr->e_ident[EI_MAG1] != ELFMAG1) ||
-	    (hdr->e_ident[EI_MAG2] != ELFMAG2) ||
-	    (hdr->e_ident[EI_MAG3] != ELFMAG3)) {
-		/* Not an ELF file - silently ignore it */
-		return 0;
-	}
-
-	if (!hdr->e_ident[EI_DATA]) {
-		/* Unknown endian */
-		return 0;
-	}
-
-	endian = hdr->e_ident[EI_DATA];
-
-	/* Fix endianness in ELF header */
-	hdr->e_type      = TO_NATIVE(endian, 16, hdr->e_type);
-	hdr->e_machine   = TO_NATIVE(endian, 16, hdr->e_machine);
-	hdr->e_version   = TO_NATIVE(endian, 32, hdr->e_version);
-	hdr->e_entry     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_entry);
-	hdr->e_phoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_phoff);
-	hdr->e_shoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_shoff);
-	hdr->e_flags     = TO_NATIVE(endian, 32, hdr->e_flags);
-	hdr->e_ehsize    = TO_NATIVE(endian, 16, hdr->e_ehsize);
-	hdr->e_phentsize = TO_NATIVE(endian, 16, hdr->e_phentsize);
-	hdr->e_phnum     = TO_NATIVE(endian, 16, hdr->e_phnum);
-	hdr->e_shentsize = TO_NATIVE(endian, 16, hdr->e_shentsize);
-	hdr->e_shnum     = TO_NATIVE(endian, 16, hdr->e_shnum);
-	hdr->e_shstrndx  = TO_NATIVE(endian, 16, hdr->e_shstrndx);
-
-	sechdrs = RTE_PTR_ADD(hdr, hdr->e_shoff);
-	info->sechdrs = sechdrs;
-
-	/* Check if file offset is correct */
-	if (hdr->e_shoff > info->size) {
-		fprintf(stderr, "section header offset=%lu in file '%s' "
-		      "is bigger than filesize=%lu\n",
-		      (unsigned long)hdr->e_shoff,
-		      filename, info->size);
-		return 0;
-	}
-
-	if (hdr->e_shnum == SHN_UNDEF) {
-		/*
-		 * There are more than 64k sections,
-		 * read count from .sh_size.
-		 */
-		info->num_sections =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[0].sh_size);
-	} else {
-		info->num_sections = hdr->e_shnum;
-	}
-	if (hdr->e_shstrndx == SHN_XINDEX)
-		info->secindex_strings =
-			TO_NATIVE(endian, 32, sechdrs[0].sh_link);
-	else
-		info->secindex_strings = hdr->e_shstrndx;
-
-	/* Fix endianness in section headers */
-	for (i = 0; i < info->num_sections; i++) {
-		sechdrs[i].sh_name      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_name);
-		sechdrs[i].sh_type      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_type);
-		sechdrs[i].sh_flags     =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_flags);
-		sechdrs[i].sh_addr      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addr);
-		sechdrs[i].sh_offset    =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_offset);
-		sechdrs[i].sh_size      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_size);
-		sechdrs[i].sh_link      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_link);
-		sechdrs[i].sh_info      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_info);
-		sechdrs[i].sh_addralign =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addralign);
-		sechdrs[i].sh_entsize   =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_entsize);
-	}
-	/* Find symbol table. */
-	for (i = 1; i < info->num_sections; i++) {
-		int nobits = sechdrs[i].sh_type == SHT_NOBITS;
-
-		if (!nobits && sechdrs[i].sh_offset > info->size) {
-			fprintf(stderr, "%s is truncated. "
-			      "sechdrs[i].sh_offset=%lu > sizeof(*hrd)=%zu\n",
-			      filename, (unsigned long)sechdrs[i].sh_offset,
-			      sizeof(*hdr));
-			return 0;
-		}
-
-		if (sechdrs[i].sh_type == SHT_SYMTAB) {
-			unsigned int sh_link_idx;
-			symtab_idx = i;
-			info->symtab_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-			sh_link_idx = sechdrs[i].sh_link;
-			info->strtab       = RTE_PTR_ADD(hdr,
-				sechdrs[sh_link_idx].sh_offset);
-		}
-
-		/* 32bit section no. table? ("more than 64k sections") */
-		if (sechdrs[i].sh_type == SHT_SYMTAB_SHNDX) {
-			symtab_shndx_idx = i;
-			info->symtab_shndx_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_shndx_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-		}
-	}
-	if (!info->symtab_start)
-		fprintf(stderr, "%s has no symtab?\n", filename);
-	else {
-		/* Fix endianness in symbols */
-		for (sym = info->symtab_start; sym < info->symtab_stop; sym++) {
-			sym->st_shndx = TO_NATIVE(endian, 16, sym->st_shndx);
-			sym->st_name  = TO_NATIVE(endian, 32, sym->st_name);
-			sym->st_value = TO_NATIVE(endian, ADDR_SIZE, sym->st_value);
-			sym->st_size  = TO_NATIVE(endian, ADDR_SIZE, sym->st_size);
-		}
-	}
-
-	if (symtab_shndx_idx != ~0U) {
-		Elf32_Word *p;
-		if (symtab_idx != sechdrs[symtab_shndx_idx].sh_link)
-			fprintf(stderr,
-			      "%s: SYMTAB_SHNDX has bad sh_link: %u!=%u\n",
-			      filename, sechdrs[symtab_shndx_idx].sh_link,
-			      symtab_idx);
-		/* Fix endianness */
-		for (p = info->symtab_shndx_start; p < info->symtab_shndx_stop;
-		     p++)
-			*p = TO_NATIVE(endian, 32, *p);
-	}
-
-	return 1;
-}
-
-static void parse_elf_finish(struct elf_info *info)
-{
-	struct pmd_driver *tmp, *idx = info->drivers;
-	release_file(info->hdr, info->size);
-	while (idx) {
-		tmp = idx->next;
-		free(idx);
-		idx = tmp;
-	}
-}
-
-struct opt_tag {
-	const char *suffix;
-	const char *json_id;
-};
-
-static const struct opt_tag opt_tags[] = {
-	{"_param_string_export", "params"},
-	{"_kmod_dep_export", "kmod"},
-};
-
-static int complete_pmd_entry(struct elf_info *info, struct pmd_driver *drv)
-{
-	const char *tname;
-	int i;
-	char tmpsymname[128];
-	Elf_Sym *tmpsym;
-
-	drv->name = get_sym_value(info, drv->name_sym);
-
-	for (i = 0; i < PMD_OPT_MAX; i++) {
-		memset(tmpsymname, 0, 128);
-		sprintf(tmpsymname, "__%s%s", drv->name, opt_tags[i].suffix);
-		tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-		if (!tmpsym)
-			continue;
-		drv->opt_vals[i] = get_sym_value(info, tmpsym);
-	}
-
-	memset(tmpsymname, 0, 128);
-	sprintf(tmpsymname, "__%s_pci_tbl_export", drv->name);
-
-	tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-
-
-	/*
-	 * If this returns NULL, then this is a PMD_VDEV, because
-	 * it has no pci table reference
-	 */
-	if (!tmpsym) {
-		drv->pci_tbl = NULL;
-		return 0;
-	}
-
-	tname = get_sym_value(info, tmpsym);
-	tmpsym = find_sym_in_symtab(info, tname, NULL);
-	if (!tmpsym)
-		return -ENOENT;
-
-	drv->pci_tbl = (struct rte_pci_id *)get_sym_value(info, tmpsym);
-	if (!drv->pci_tbl)
-		return -ENOENT;
-
-	return 0;
-}
-
-static int locate_pmd_entries(struct elf_info *info)
-{
-	Elf_Sym *last = NULL;
-	struct pmd_driver *new;
-
-	info->drivers = NULL;
-
-	do {
-		new = calloc(sizeof(struct pmd_driver), 1);
-		if (new == NULL) {
-			fprintf(stderr, "Failed to calloc memory\n");
-			return -1;
-		}
-		new->name_sym = find_sym_in_symtab(info, "this_pmd_name", last);
-		last = new->name_sym;
-		if (!new->name_sym)
-			free(new);
-		else {
-			if (complete_pmd_entry(info, new)) {
-				fprintf(stderr,
-					"Failed to complete pmd entry\n");
-				free(new);
-			} else {
-				new->next = info->drivers;
-				info->drivers = new;
-			}
-		}
-	} while (last);
-
-	return 0;
-}
-
-static void output_pmd_info_string(struct elf_info *info, char *outfile)
-{
-	FILE *ofd;
-	struct pmd_driver *drv;
-	struct rte_pci_id *pci_ids;
-	int idx = 0;
-
-	if (use_stdout)
-		ofd = stdout;
-	else {
-		ofd = fopen(outfile, "w+");
-		if (!ofd) {
-			fprintf(stderr, "Unable to open output file\n");
-			return;
-		}
-	}
-
-	drv = info->drivers;
-
-	while (drv) {
-		fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
-			"\"PMD_INFO_STRING= {",
-			drv->name);
-		fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);
-
-		for (idx = 0; idx < PMD_OPT_MAX; idx++) {
-			if (drv->opt_vals[idx])
-				fprintf(ofd, "\\\"%s\\\" : \\\"%s\\\", ",
-					opt_tags[idx].json_id,
-					drv->opt_vals[idx]);
-		}
-
-		pci_ids = drv->pci_tbl;
-		fprintf(ofd, "\\\"pci_ids\\\" : [");
-
-		while (pci_ids && pci_ids->device_id) {
-			fprintf(ofd, "[%d, %d, %d, %d]",
-				pci_ids->vendor_id, pci_ids->device_id,
-				pci_ids->subsystem_vendor_id,
-				pci_ids->subsystem_device_id);
-			pci_ids++;
-			if (pci_ids->device_id)
-				fprintf(ofd, ",");
-			else
-				fprintf(ofd, " ");
-		}
-		fprintf(ofd, "]}\";\n");
-		drv = drv->next;
-	}
-
-	fclose(ofd);
-}
-
-int main(int argc, char **argv)
-{
-	struct elf_info info = {0};
-	int rc = 1;
-
-	if (argc < 3) {
-		fprintf(stderr,
-			"usage: %s <object file> <c output file>\n",
-			basename(argv[0]));
-		exit(127);
-	}
-	use_stdin = !strcmp(argv[1], "-");
-	use_stdout = !strcmp(argv[2], "-");
-	parse_elf(&info, argv[1]);
-
-	if (locate_pmd_entries(&info) < 0)
-		exit(1);
-
-	if (info.drivers) {
-		output_pmd_info_string(&info, argv[2]);
-		rc = 0;
-	} else {
-		fprintf(stderr, "No drivers registered\n");
-	}
-
-	parse_elf_finish(&info);
-	exit(rc);
-}
diff --git a/buildtools/pmdinfogen/pmdinfogen.h b/buildtools/pmdinfogen/pmdinfogen.h
deleted file mode 100644
index 38fd3ddcc..000000000
--- a/buildtools/pmdinfogen/pmdinfogen.h
+++ /dev/null
@@ -1,119 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <nhorman@tuxdriver.com>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <stdarg.h>
-#include <string.h>
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#ifdef __linux__
-#include <endian.h>
-#else
-#include <sys/endian.h>
-#endif
-#include <fcntl.h>
-#include <unistd.h>
-#include <elf.h>
-#include <rte_pci.h>
-
-/* On BSD-alike OSes elf.h defines these according to host's word size */
-#undef ELF_ST_BIND
-#undef ELF_ST_TYPE
-#undef ELF_R_SYM
-#undef ELF_R_TYPE
-
-/*
- * Define ELF64_* to ELF_*, the latter being defined in both 32 and 64 bit
- * flavors in elf.h.  This makes our code a bit more generic between arches
- * and allows us to support 32 bit code in the future should we ever want to
- */
-#ifdef RTE_ARCH_64
-#define Elf_Ehdr    Elf64_Ehdr
-#define Elf_Shdr    Elf64_Shdr
-#define Elf_Sym     Elf64_Sym
-#define Elf_Addr    Elf64_Addr
-#define Elf_Sword   Elf64_Sxword
-#define Elf_Section Elf64_Half
-#define ELF_ST_BIND ELF64_ST_BIND
-#define ELF_ST_TYPE ELF64_ST_TYPE
-
-#define Elf_Rel     Elf64_Rel
-#define Elf_Rela    Elf64_Rela
-#define ELF_R_SYM   ELF64_R_SYM
-#define ELF_R_TYPE  ELF64_R_TYPE
-#else
-#define Elf_Ehdr    Elf32_Ehdr
-#define Elf_Shdr    Elf32_Shdr
-#define Elf_Sym     Elf32_Sym
-#define Elf_Addr    Elf32_Addr
-#define Elf_Sword   Elf32_Sxword
-#define Elf_Section Elf32_Half
-#define ELF_ST_BIND ELF32_ST_BIND
-#define ELF_ST_TYPE ELF32_ST_TYPE
-
-#define Elf_Rel     Elf32_Rel
-#define Elf_Rela    Elf32_Rela
-#define ELF_R_SYM   ELF32_R_SYM
-#define ELF_R_TYPE  ELF32_R_TYPE
-#endif
-
-
-/*
- * Note, it seems odd that we have both a CONVERT_NATIVE and a TO_NATIVE macro
- * below.  We do this because the values passed to TO_NATIVE may themselves be
- * macros and need both macros here to get expanded.  Specifically its the width
- * variable we are concerned with, because it needs to get expanded prior to
- * string concatenation
- */
-#define CONVERT_NATIVE(fend, width, x) ({ \
-typeof(x) ___x; \
-if ((fend) == ELFDATA2LSB) \
-	___x = le##width##toh(x); \
-else \
-	___x = be##width##toh(x); \
-___x; \
-})
-
-#define TO_NATIVE(fend, width, x) CONVERT_NATIVE(fend, width, x)
-
-enum opt_params {
-	PMD_PARAM_STRING = 0,
-	PMD_KMOD_DEP,
-	PMD_OPT_MAX
-};
-
-struct pmd_driver {
-	Elf_Sym *name_sym;
-	const char *name;
-	struct rte_pci_id *pci_tbl;
-	struct pmd_driver *next;
-
-	const char *opt_vals[PMD_OPT_MAX];
-};
-
-struct elf_info {
-	unsigned long size;
-	Elf_Ehdr     *hdr;
-	Elf_Shdr     *sechdrs;
-	Elf_Sym      *symtab_start;
-	Elf_Sym      *symtab_stop;
-	char         *strtab;
-
-	/* support for 32bit section numbers */
-
-	unsigned int num_sections; /* max_secindex + 1 */
-	unsigned int secindex_strings;
-	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
-	 * take shndx from symtab_shndx_start[N] instead
-	 */
-	Elf32_Word   *symtab_shndx_start;
-	Elf32_Word   *symtab_shndx_stop;
-
-	struct pmd_driver *drivers;
-};
-- 
2.28.0


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [dpdk-ci] [PATCH v7 0/3] pmdinfogen: rewrite in Python
  2020-10-20 16:02       ` [dpdk-ci] [PATCH v7 0/3] pmdinfogen: rewrite in Python David Marchand
@ 2020-10-20 17:45         ` Dmitry Kozlyuk
  2020-10-20 22:09         ` Dmitry Kozlyuk
  1 sibling, 0 replies; 21+ messages in thread
From: Dmitry Kozlyuk @ 2020-10-20 17:45 UTC (permalink / raw)
  To: David Marchand
  Cc: dev, ci, Chen, Zhaoyan, Stephen Hemminger, Thomas Monjalon,
	Maxime Coquelin, Neil Horman, Brandon Lo, Lincoln Lavoie

Hello David,

> - Travis reported an error in meson:
> https://travis-ci.com/github/ovsrobot/dpdk/jobs/399762460#L478
> 
> Build machine cpu: x86_64
> buildtools/meson.build:27:10: ERROR:  Expecting endforeach got eol_cont.
> script = \
>           ^
> For a block that started at 26,0
> foreach module : python3_required_modules
> ^

My bad, meson 0.47.1 doesn't understand this indeed, fixed in v8.

> - In a thread on ci@dpdk.org
> http://inbox.dpdk.org/ci/CAOeXdvbosqRGouSX06_phVTCam57XUt0s=sf_kpe-VmiwFTgMA@mail.gmail.com/T/#m9bb193c74ca979c151f2dbfa35664f95093a9e71,
> UNH people confirmed that python3-elftools should be available, but
> the last test had an error too, which might be linked to incorrect
> meson syntax:
> http://mails.dpdk.org/archives/test-report/2020-October/159038.html
> 
> ninja: error: loading 'build.ninja': No such file or directory

Makes sense.

> - As far as Intel CI systems are concerned, there might be an issue
> with the Windows system:
> http://mails.dpdk.org/archives/test-report/2020-October/159123.html
>
> OS: WIN10-64
> Target: x86_64-windows-clang
> Program binutils-avx512-check.sh found: YES
> Program python3 found: YES (c:\program files\python37\python.exe)
> 
> buildtools\meson.build:32:2: ERROR: Problem encountered: missing
> python module: elftools

This is the check added in the last revision per Maxime's suggestion.
Disabled the check on Windows in v8 until pmdinfogen supports COFF (and this
will probably introduce an alternative dependency).

I'll watch new CI results, haven't been doing so closely because failures were
expected for these patches. Tested v8 build on Linux->Linux, Linux->Windows,
Windows->Windows (GCC & clang, without pyelftools installed).

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [dpdk-ci] [PATCH v7 0/3] pmdinfogen: rewrite in Python
  2020-10-20 16:02       ` [dpdk-ci] [PATCH v7 0/3] pmdinfogen: rewrite in Python David Marchand
  2020-10-20 17:45         ` Dmitry Kozlyuk
@ 2020-10-20 22:09         ` Dmitry Kozlyuk
  1 sibling, 0 replies; 21+ messages in thread
From: Dmitry Kozlyuk @ 2020-10-20 22:09 UTC (permalink / raw)
  To: Brandon Lo, Lincoln Lavoie
  Cc: David Marchand, dev, ci, Chen, Zhaoyan, Stephen Hemminger,
	Thomas Monjalon, Maxime Coquelin, Neil Horman

Hi IOL team,

The following error persists in v8:

> - In a thread on ci@dpdk.org
> http://inbox.dpdk.org/ci/CAOeXdvbosqRGouSX06_phVTCam57XUt0s=sf_kpe-VmiwFTgMA@mail.gmail.com/T/#m9bb193c74ca979c151f2dbfa35664f95093a9e71,
> UNH people confirmed that python3-elftools should be available, but
> the last test had an error too, which might be linked to incorrect
> meson syntax:
> http://mails.dpdk.org/archives/test-report/2020-October/159038.html
> 
> ninja: error: loading 'build.ninja': No such file or directory

https://inbox.dpdk.org/test-report/20201020181426.1BC8788FC9@noxus.dpdklab.iol.unh.edu/T/#u

The message isn't very informative. I suspect meson failed to configure,
but there's no way to tell what went wrong. How can I get more information
about this failure or specification of your environment to reproduce it?

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [dpdk-ci] [PATCH v8 2/3] build: use Python pmdinfogen
  2020-10-20 17:44         ` [dpdk-ci] [PATCH v8 2/3] build: use Python pmdinfogen Dmitry Kozlyuk
@ 2020-10-21  9:00           ` Bruce Richardson
  2021-01-20  0:05           ` [dpdk-ci] [dpdk-dev] " Thomas Monjalon
  1 sibling, 0 replies; 21+ messages in thread
From: Bruce Richardson @ 2020-10-21  9:00 UTC (permalink / raw)
  To: Dmitry Kozlyuk
  Cc: dev, ci, Stephen Hemminger, David Marchand, Thomas Monjalon,
	Maxime Coquelin, Aaron Conole, Michael Santana

On Tue, Oct 20, 2020 at 08:44:02PM +0300, Dmitry Kozlyuk wrote:
> Use the same interpreter to run pmdinfogen as for other build scripts.
> Adjust wrapper script accordingly and also don't suppress stderr from ar
> and pmdinfogen.
> 
> Add configure-time check for elftools Python module for Unix hosts.
> 
> Add python3-pyelftools to CI configuration. The package is available on
> all major distributions. FreeBSD has no system requirements section in
> its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE)
> supported.
> 
> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> ---

From a meson viewpoint, this all looks ok. I'd hope in future that we could
merge the shell script wrapper into the python script directly, rather than
having two levels of scripting, but it's not something with any urgency.

Acked-by: Bruce Richardson <bruce.richardson@intel.com>

>  .travis.yml                       |  2 +-
>  buildtools/gen-pmdinfo-cfile.sh   |  6 +++---
>  buildtools/meson.build            | 15 +++++++++++++++
>  doc/guides/linux_gsg/sys_reqs.rst |  6 ++++++
>  drivers/meson.build               |  2 +-
>  meson.build                       |  1 -
>  6 files changed, 26 insertions(+), 6 deletions(-)
> 

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [dpdk-ci] [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
  2020-10-20 17:44         ` [dpdk-ci] [PATCH v8 2/3] build: use Python pmdinfogen Dmitry Kozlyuk
  2020-10-21  9:00           ` Bruce Richardson
@ 2021-01-20  0:05           ` Thomas Monjalon
  2021-01-20  7:23             ` Dmitry Kozlyuk
  1 sibling, 1 reply; 21+ messages in thread
From: Thomas Monjalon @ 2021-01-20  0:05 UTC (permalink / raw)
  To: Dmitry Kozlyuk
  Cc: dev, ci, Stephen Hemminger, David Marchand, Maxime Coquelin,
	Aaron Conole, Bruce Richardson, ferruh.yigit

This is now the right timeframe to introduce this change
with the new Python module dependency.
Unfortunately, the ABI check is returning an issue:

'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c


Few more comments below:

20/10/2020 19:44, Dmitry Kozlyuk:
> --- a/buildtools/meson.build
> +++ b/buildtools/meson.build
> +if host_machine.system() != 'windows'

You can use "is_windows".


> --- a/doc/guides/linux_gsg/sys_reqs.rst
> +++ b/doc/guides/linux_gsg/sys_reqs.rst
> +*   ``pyelftools`` (version 0.22+)

This requirement is missing in doc/guides/freebsd_gsg/build_dpdk.rst


> --- a/meson.build
> +++ b/meson.build
> -subdir('buildtools/pmdinfogen')

This could be in patch 3 (removing the code).





^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [dpdk-ci] [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
  2021-01-20  0:05           ` [dpdk-ci] [dpdk-dev] " Thomas Monjalon
@ 2021-01-20  7:23             ` Dmitry Kozlyuk
  0 siblings, 0 replies; 21+ messages in thread
From: Dmitry Kozlyuk @ 2021-01-20  7:23 UTC (permalink / raw)
  To: Thomas Monjalon
  Cc: dev, ci, Stephen Hemminger, David Marchand, Maxime Coquelin,
	Aaron Conole, Bruce Richardson, ferruh.yigit

On Wed, 20 Jan 2021 01:05:59 +0100, Thomas Monjalon wrote:
> This is now the right timeframe to introduce this change
> with the new Python module dependency.
> Unfortunately, the ABI check is returning an issue:
> 
> 'const char mlx5_common_pci_pmd_info[62]' was changed
> to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c

Will investigate and fix ASAP.
 
> Few more comments below:
> 
> 20/10/2020 19:44, Dmitry Kozlyuk:
> > --- a/buildtools/meson.build
> > +++ b/buildtools/meson.build
> > +if host_machine.system() != 'windows'  
> 
> You can use "is_windows".

It's defined by config/meson.build, which is processed after
buidtools/meson.build, because of the dependency, if swapped:

	config/x86/meson.build:6:1: ERROR: Unknown variable
	"binutils_avx512_check".

> > --- a/doc/guides/linux_gsg/sys_reqs.rst
> > +++ b/doc/guides/linux_gsg/sys_reqs.rst
> > +*   ``pyelftools`` (version 0.22+)  
> 
> This requirement is missing in doc/guides/freebsd_gsg/build_dpdk.rst

OK.

> > --- a/meson.build
> > +++ b/meson.build
> > -subdir('buildtools/pmdinfogen')  
> 
> This could be in patch 3 (removing the code).

It would redefine "pmdinfogen" variable to old pmdinfogen.
Besides, why build what's not used at this patch already?


^ permalink raw reply	[flat|nested] 21+ messages in thread

end of thread, other threads:[~2021-01-20  7:23 UTC | newest]

Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <20200708212335.25338-1-dmitry.kozliuk@gmail.com>
     [not found] ` <20200927214732.12783-1-dmitry.kozliuk@gmail.com>
2020-09-28  9:35   ` [dpdk-ci] [dpdk-dev] [PATCH v5 0/3] pmdinfogen: rewrite in Python David Marchand
2020-10-04  1:59   ` [dpdk-ci] [PATCH v6 " Dmitry Kozlyuk
2020-10-04  1:59     ` [dpdk-ci] [PATCH v6 1/3] pmdinfogen: add Python implementation Dmitry Kozlyuk
2020-10-04  1:59     ` [dpdk-ci] [PATCH v6 2/3] build: use Python pmdinfogen Dmitry Kozlyuk
2020-10-04  1:59     ` [dpdk-ci] [PATCH v6 3/3] pmdinfogen: remove C implementation Dmitry Kozlyuk
2020-10-14 14:37     ` [dpdk-ci] [dpdk-dev] [PATCH v6 0/3] pmdinfogen: rewrite in Python Maxime Coquelin
2020-10-14 15:40       ` Dmitry Kozlyuk
2020-10-14 18:31     ` [dpdk-ci] [PATCH v7 " Dmitry Kozlyuk
2020-10-14 18:31       ` [dpdk-ci] [PATCH v7 1/3] pmdinfogen: add Python implementation Dmitry Kozlyuk
2020-10-14 18:31       ` [dpdk-ci] [PATCH v7 2/3] build: use Python pmdinfogen Dmitry Kozlyuk
2020-10-14 18:31       ` [dpdk-ci] [PATCH v7 3/3] pmdinfogen: remove C implementation Dmitry Kozlyuk
2020-10-20 16:02       ` [dpdk-ci] [PATCH v7 0/3] pmdinfogen: rewrite in Python David Marchand
2020-10-20 17:45         ` Dmitry Kozlyuk
2020-10-20 22:09         ` Dmitry Kozlyuk
2020-10-20 17:44       ` [dpdk-ci] [PATCH v8 " Dmitry Kozlyuk
2020-10-20 17:44         ` [dpdk-ci] [PATCH v8 1/3] pmdinfogen: add Python implementation Dmitry Kozlyuk
2020-10-20 17:44         ` [dpdk-ci] [PATCH v8 2/3] build: use Python pmdinfogen Dmitry Kozlyuk
2020-10-21  9:00           ` Bruce Richardson
2021-01-20  0:05           ` [dpdk-ci] [dpdk-dev] " Thomas Monjalon
2021-01-20  7:23             ` Dmitry Kozlyuk
2020-10-20 17:44         ` [dpdk-ci] [PATCH v8 3/3] pmdinfogen: remove C implementation Dmitry Kozlyuk

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).