DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] [PATCH 0/4] pmdinfogen: support Windows
@ 2020-12-12 23:34 Dmitry Kozlyuk
  2020-12-12 23:34 ` [dpdk-dev] [PATCH 1/4] pmdinfogen: support COFF Dmitry Kozlyuk
                   ` (4 more replies)
  0 siblings, 5 replies; 13+ messages in thread
From: Dmitry Kozlyuk @ 2020-12-12 23:34 UTC (permalink / raw)
  To: dev
  Cc: Narcisa Ana Maria Vasile, Pallavi Kadam, Dmitry Malloy, Dmitry Kozlyuk

Based on Python version of pmdinfogen, deferred until 21.02:

    http://patchwork.dpdk.org/project/dpdk/list/?series=13153

There are a few Python libraries for PE/COFF, none of which suits the need,
so a custom COFF parser is used.  Advice is welcome, options considered:

* py-coff (https://github.com/jeppeter/py-coff): doesn't give access to
  symbol values, Python 2 code inside, not very popular.

* pefile (https://github.com/erocarrera/pefile): for PE (executables and
  libraries), not COFF (objects); most popular.

* pype32-py3 (https://github.com/crackinglandia/pype32): ditto, less popular.

A script to extract object files from library is still required. Meson has
extract_all_objects(), but they can't be passed as inputs to custom_target()
until 0.52.0 (commit f431cff809).

Depends-on: series-13153 ("pmdinfogen: rewrite in Python")

Dmitry Kozlyuk (4):
  pmdinfogen: support COFF
  pmdinfogen: allow multiple input files
  buildtools: support object file extraction for Windows
  build: enable pmdinfogen for Windows

 buildtools/coff.py              | 154 ++++++++++++++++++++++++++++++++
 buildtools/gen-pmdinfo-cfile.py |  19 ++++
 buildtools/gen-pmdinfo-cfile.sh |  14 ---
 buildtools/meson.build          |  15 +++-
 buildtools/pmdinfogen.py        | 126 +++++++++++++++++++-------
 drivers/meson.build             |  26 +++---
 6 files changed, 294 insertions(+), 60 deletions(-)
 create mode 100644 buildtools/coff.py
 create mode 100644 buildtools/gen-pmdinfo-cfile.py
 delete mode 100755 buildtools/gen-pmdinfo-cfile.sh

-- 
2.29.2


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [dpdk-dev] [PATCH 1/4] pmdinfogen: support COFF
  2020-12-12 23:34 [dpdk-dev] [PATCH 0/4] pmdinfogen: support Windows Dmitry Kozlyuk
@ 2020-12-12 23:34 ` Dmitry Kozlyuk
  2020-12-12 23:34 ` [dpdk-dev] [PATCH 2/4] pmdinfogen: allow multiple input files Dmitry Kozlyuk
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 13+ messages in thread
From: Dmitry Kozlyuk @ 2020-12-12 23:34 UTC (permalink / raw)
  To: dev
  Cc: Narcisa Ana Maria Vasile, Pallavi Kadam, Dmitry Malloy,
	Dmitry Kozlyuk, Neil Horman

Common Object File Format (COFF) is used on Windows in place of ELF.

Add COFF parser to pmdinfogen. Also add an argument to specify input
file format, which is selected at configure time based on the target.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 buildtools/coff.py       | 154 +++++++++++++++++++++++++++++++++++++++
 buildtools/meson.build   |   7 ++
 buildtools/pmdinfogen.py | 107 ++++++++++++++++++++-------
 3 files changed, 240 insertions(+), 28 deletions(-)
 create mode 100644 buildtools/coff.py

diff --git a/buildtools/coff.py b/buildtools/coff.py
new file mode 100644
index 000000000..86fb0602b
--- /dev/null
+++ b/buildtools/coff.py
@@ -0,0 +1,154 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2020 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import ctypes
+
+# x86_64 little-endian
+COFF_MAGIC = 0x8664
+
+# Names up to this length are stored immediately in symbol table entries.
+COFF_NAMELEN = 8
+
+# Special "section numbers" changing the meaning of symbol table entry.
+COFF_SN_UNDEFINED = 0
+COFF_SN_ABSOLUTE = -1
+COFF_SN_DEBUG = -2
+
+
+class CoffFileHeader(ctypes.LittleEndianStructure):
+    _pack_ = True
+    _fields_ = [
+        ("magic", ctypes.c_uint16),
+        ("section_count", ctypes.c_uint16),
+        ("timestamp", ctypes.c_uint32),
+        ("symbol_table_offset", ctypes.c_uint32),
+        ("symbol_count", ctypes.c_uint32),
+        ("optional_header_size", ctypes.c_uint16),
+        ("flags", ctypes.c_uint16),
+    ]
+
+
+class CoffName(ctypes.Union):
+    class Reference(ctypes.LittleEndianStructure):
+        _pack_ = True
+        _fields_ = [
+            ("zeroes", ctypes.c_uint32),
+            ("offset", ctypes.c_uint32),
+        ]
+
+    Immediate = ctypes.c_char * 8
+
+    _pack_ = True
+    _fields_ = [
+        ("immediate", Immediate),
+        ("reference", Reference),
+    ]
+
+
+class CoffSection(ctypes.LittleEndianStructure):
+    _pack_ = True
+    _fields_ = [
+        ("name", CoffName),
+        ("physical_address", ctypes.c_uint32),
+        ("physical_address", ctypes.c_uint32),
+        ("size", ctypes.c_uint32),
+        ("data_offset", ctypes.c_uint32),
+        ("relocations_offset", ctypes.c_uint32),
+        ("line_numbers_offset", ctypes.c_uint32),
+        ("relocation_count", ctypes.c_uint16),
+        ("line_number_count", ctypes.c_uint16),
+        ("flags", ctypes.c_uint32),
+    ]
+
+
+class CoffSymbol(ctypes.LittleEndianStructure):
+    _pack_ = True
+    _fields_ = [
+        ("name", CoffName),
+        ("value", ctypes.c_uint32),
+        ("section_number", ctypes.c_int16),
+        ("type", ctypes.c_uint16),
+        ("storage_class", ctypes.c_uint8),
+        ("auxiliary_count", ctypes.c_uint8),
+    ]
+
+
+class Symbol:
+    def __init__(self, image, symbol: CoffSymbol):
+        self._image = image
+        self._coff = symbol
+
+    @property
+    def name(self):
+        if self._coff.name.reference.zeroes:
+            return decode_asciiz(bytes(self._coff.name.immediate))
+
+        offset = self._coff.name.reference.offset
+        offset -= ctypes.sizeof(ctypes.c_uint32)
+        return self._image.get_string(offset)
+
+    def get_value(self, offset):
+        section_number = self._coff.section_number
+
+        if section_number == COFF_SN_UNDEFINED:
+            return None
+
+        if section_number == COFF_SN_DEBUG:
+            return None
+
+        if section_number == COFF_SN_ABSOLUTE:
+            return bytes(ctypes.c_uint32(self._coff.value))
+
+        section_data = self._image.get_section_data(section_number)
+        section_offset = self._coff.value + offset
+        return section_data[section_offset:]
+
+
+class Image:
+    def __init__(self, data):
+        header = CoffFileHeader.from_buffer_copy(data)
+        header_size = ctypes.sizeof(header) + header.optional_header_size
+
+        sections_desc = CoffSection * header.section_count
+        sections = sections_desc.from_buffer_copy(data, header_size)
+
+        symbols_desc = CoffSymbol * header.symbol_count
+        symbols = symbols_desc.from_buffer_copy(data, header.symbol_table_offset)
+
+        strings_offset = header.symbol_table_offset + ctypes.sizeof(symbols)
+        strings = Image._parse_strings(data[strings_offset:])
+
+        self._data = data
+        self._header = header
+        self._sections = sections
+        self._symbols = symbols
+        self._strings = strings
+
+    @staticmethod
+    def _parse_strings(data):
+        full_size = ctypes.c_uint32.from_buffer_copy(data)
+        header_size = ctypes.sizeof(full_size)
+        return data[header_size : full_size.value]
+
+    @property
+    def symbols(self):
+        i = 0
+        while i < self._header.symbol_count:
+            symbol = self._symbols[i]
+            yield Symbol(self, symbol)
+            i += symbol.auxiliary_count + 1
+
+    def get_section_data(self, number):
+        # section numbers are 1-based
+        section = self._sections[number - 1]
+        base = section.data_offset
+        return self._data[base : base + section.size]
+
+    def get_string(self, offset):
+        return decode_asciiz(self._strings[offset:])
+
+
+def decode_asciiz(data):
+    index = data.find(b'\x00')
+    end = index if index >= 0 else len(data)
+    return data[:end].decode()
diff --git a/buildtools/meson.build b/buildtools/meson.build
index dd4c0f640..23cefd4be 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -17,7 +17,14 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+
+# select object file format
 pmdinfogen = py3 + files('pmdinfogen.py')
+if host_machine.system() == 'windows'
+	pmdinfogen += 'coff'
+else
+	pmdinfogen += 'elf'
+endif
 
 # TODO: starting from Meson 0.51.0 use
 # 	python3 = import('python').find_installation('python',
diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
index 0cca47ff1..65d78b872 100755
--- a/buildtools/pmdinfogen.py
+++ b/buildtools/pmdinfogen.py
@@ -9,6 +9,8 @@
 import sys
 import tempfile
 
+import coff
+
 from elftools.elf.elffile import ELFFile
 from elftools.elf.sections import SymbolTableSection
 
@@ -18,21 +20,18 @@ def __init__(self, image, symbol):
         self._image = image
         self._symbol = symbol
 
-    @property
-    def size(self):
-        return self._symbol["st_size"]
-
-    @property
-    def value(self):
-        data = self._image.get_section_data(self._symbol["st_shndx"])
-        base = self._symbol["st_value"]
-        return data[base:base + self.size]
-
     @property
     def string_value(self):
-        value = self.value
+        size = self._symbol["st_size"]
+        value = self.get_value(0, size)
         return value[:-1].decode() if value else ""
 
+    def get_value(self, offset, size):
+        section = self._symbol["st_shndx"]
+        data = self._image.get_section(section).data()
+        base = self._symbol["st_value"] + offset
+        return data[base : base + size]
+
 
 class ELFImage:
     def __init__(self, data):
@@ -45,18 +44,50 @@ def __init__(self, data):
     def is_big_endian(self):
         return not self._image.little_endian
 
-    def get_section_data(self, name):
-        return self._image.get_section(name).data()
-
     def find_by_name(self, name):
         symbol = self._symtab.get_symbol_by_name(name)
-        return ELFSymbol(self, symbol[0]) if symbol else None
+        return ELFSymbol(self._image, symbol[0]) if symbol else None
 
     def find_by_prefix(self, prefix):
         for i in range(self._symtab.num_symbols()):
             symbol = self._symtab.get_symbol(i)
             if symbol.name.startswith(prefix):
-                yield ELFSymbol(self, symbol)
+                yield ELFSymbol(self._image, symbol)
+
+
+class COFFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    def get_value(self, offset, size):
+        value = self._symbol.get_value(offset)
+        return value[:size] if value else value
+
+    @property
+    def string_value(self):
+        value = self._symbol.get_value(0)
+        return coff.decode_asciiz(value) if value else ''
+
+
+class COFFImage:
+    def __init__(self, data):
+        self._image = coff.Image(data)
+
+    @property
+    def is_big_endian(self):
+        return False
+
+    def find_by_prefix(self, prefix):
+        for symbol in self._image.symbols:
+            if symbol.name.startswith(prefix):
+                yield COFFSymbol(self._image, symbol)
+
+    def find_by_name(self, name):
+        for symbol in self._image.symbols:
+            if symbol.name == name:
+                return COFFSymbol(self._image, symbol)
+        return None
 
 
 def define_rte_pci_id(is_big_endian):
@@ -117,19 +148,24 @@ def _load_pci_ids(image, table_name_symbol):
 
         rte_pci_id = define_rte_pci_id(image.is_big_endian)
 
-        pci_id_size = ctypes.sizeof(rte_pci_id)
-        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
-        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
         result = []
-        for pci_id in pci_ids:
+        while True:
+            size = ctypes.sizeof(rte_pci_id)
+            offset = size * len(result)
+            data = table_symbol.get_value(offset, size)
+            if not data:
+                break
+            pci_id = rte_pci_id.from_buffer_copy(data)
             if not pci_id.device_id:
                 break
-            result.append([
-                pci_id.vendor_id,
-                pci_id.device_id,
-                pci_id.subsystem_vendor_id,
-                pci_id.subsystem_device_id,
-                ])
+            result.append(
+                [
+                    pci_id.vendor_id,
+                    pci_id.device_id,
+                    pci_id.subsystem_vendor_id,
+                    pci_id.subsystem_device_id,
+                ]
+            )
         return result
 
     def dump(self, file):
@@ -157,6 +193,7 @@ def dump_drivers(drivers, file):
 
 def parse_args():
     parser = argparse.ArgumentParser()
+    parser.add_argument("format", help="object file format, 'elf' or 'coff'")
     parser.add_argument("input", help="input object file path or '-' for stdin")
     parser.add_argument("output", help="output C file path or '-' for stdout")
     return parser.parse_args()
@@ -170,6 +207,21 @@ def open_input(path):
     return open(path, "rb")
 
 
+def read_input(path):
+    if path == "-":
+        return sys.stdin.buffer.read()
+    with open(path, "rb") as file:
+        return file.read()
+
+
+def load_image(fmt, path):
+    if fmt == 'elf':
+        return ELFImage(open_input(path))
+    if fmt == 'coff':
+        return COFFImage(read_input(path))
+    raise Exception('unsupported object file format')
+
+
 def open_output(path):
     if path == "-":
         return sys.stdout
@@ -178,8 +230,7 @@ def open_output(path):
 
 def main():
     args = parse_args()
-    infile = open_input(args.input)
-    image = ELFImage(infile)
+    image = load_image(args.format, args.input)
     drivers = load_drivers(image)
     output = open_output(args.output)
     dump_drivers(drivers, output)
-- 
2.29.2


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [dpdk-dev] [PATCH 2/4] pmdinfogen: allow multiple input files
  2020-12-12 23:34 [dpdk-dev] [PATCH 0/4] pmdinfogen: support Windows Dmitry Kozlyuk
  2020-12-12 23:34 ` [dpdk-dev] [PATCH 1/4] pmdinfogen: support COFF Dmitry Kozlyuk
@ 2020-12-12 23:34 ` Dmitry Kozlyuk
  2020-12-12 23:34 ` [dpdk-dev] [PATCH 3/4] buildtools: support object file extraction for Windows Dmitry Kozlyuk
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 13+ messages in thread
From: Dmitry Kozlyuk @ 2020-12-12 23:34 UTC (permalink / raw)
  To: dev
  Cc: Narcisa Ana Maria Vasile, Pallavi Kadam, Dmitry Malloy,
	Dmitry Kozlyuk, Neil Horman

Process any number of input object files and write a unified output.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
Used in the next patch, separated for clarity.

 buildtools/pmdinfogen.py | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
index 65d78b872..3209510eb 100755
--- a/buildtools/pmdinfogen.py
+++ b/buildtools/pmdinfogen.py
@@ -194,7 +194,9 @@ def dump_drivers(drivers, file):
 def parse_args():
     parser = argparse.ArgumentParser()
     parser.add_argument("format", help="object file format, 'elf' or 'coff'")
-    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument(
+        "input", nargs='+', help="input object file path or '-' for stdin"
+    )
     parser.add_argument("output", help="output C file path or '-' for stdout")
     return parser.parse_args()
 
@@ -230,10 +232,14 @@ def open_output(path):
 
 def main():
     args = parse_args()
-    image = load_image(args.format, args.input)
-    drivers = load_drivers(image)
+    if args.input.count('-') > 1:
+        raise Exception("'-' input cannot be used multiple times")
+
     output = open_output(args.output)
-    dump_drivers(drivers, output)
+    for path in args.input:
+        image = load_image(args.format, path)
+        drivers = load_drivers(image)
+        dump_drivers(drivers, output)
 
 
 if __name__ == "__main__":
-- 
2.29.2


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [dpdk-dev] [PATCH 3/4] buildtools: support object file extraction for Windows
  2020-12-12 23:34 [dpdk-dev] [PATCH 0/4] pmdinfogen: support Windows Dmitry Kozlyuk
  2020-12-12 23:34 ` [dpdk-dev] [PATCH 1/4] pmdinfogen: support COFF Dmitry Kozlyuk
  2020-12-12 23:34 ` [dpdk-dev] [PATCH 2/4] pmdinfogen: allow multiple input files Dmitry Kozlyuk
@ 2020-12-12 23:34 ` Dmitry Kozlyuk
  2020-12-12 23:34 ` [dpdk-dev] [PATCH 4/4] build: enable pmdinfogen " Dmitry Kozlyuk
  2021-01-08  2:47 ` [dpdk-dev] [PATCH v2 0/4] pmdinfogen: support Windows Dmitry Kozlyuk
  4 siblings, 0 replies; 13+ messages in thread
From: Dmitry Kozlyuk @ 2020-12-12 23:34 UTC (permalink / raw)
  To: dev
  Cc: Narcisa Ana Maria Vasile, Pallavi Kadam, Dmitry Malloy,
	Dmitry Kozlyuk, Bruce Richardson, Neil Horman

clang archiver tool is llvm-ar on Windows and ar on other platforms.
MinGW always uses ar. Replace shell script (Unix-only) that calls ar
with a Python script (OS-independent) that calls an appropriate archiver
tool selected at configuration time. Move the logic not to generate
empty sources into pmdinfogen.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
Stdin and stdout are not longer used for input and output. Code to
handle that could be removed, but maybe it's useful for someone.

 buildtools/gen-pmdinfo-cfile.py | 19 +++++++++++++++++++
 buildtools/gen-pmdinfo-cfile.sh | 14 --------------
 buildtools/meson.build          | 10 ++++++++--
 buildtools/pmdinfogen.py        |  7 +++++++
 4 files changed, 34 insertions(+), 16 deletions(-)
 create mode 100644 buildtools/gen-pmdinfo-cfile.py
 delete mode 100755 buildtools/gen-pmdinfo-cfile.sh

diff --git a/buildtools/gen-pmdinfo-cfile.py b/buildtools/gen-pmdinfo-cfile.py
new file mode 100644
index 000000000..f1f289ffe
--- /dev/null
+++ b/buildtools/gen-pmdinfo-cfile.py
@@ -0,0 +1,19 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2020 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import os
+import subprocess
+import sys
+import tempfile
+
+_, ar, archive, output, *pmdinfogen = sys.argv
+with tempfile.TemporaryDirectory() as temp:
+    proc = subprocess.run(
+        # Don't use "ar p", because its output is corrupted on Windows.
+        [ar, "xv", os.path.abspath(archive)], capture_output=True, check=True, cwd=temp
+    )
+    lines = proc.stdout.decode().splitlines()
+    names = [line[len("x - ") :] for line in lines]
+    paths = [os.path.join(temp, name) for name in names]
+    subprocess.run(pmdinfogen + paths + [output], check=True)
diff --git a/buildtools/gen-pmdinfo-cfile.sh b/buildtools/gen-pmdinfo-cfile.sh
deleted file mode 100755
index 109ee461e..000000000
--- a/buildtools/gen-pmdinfo-cfile.sh
+++ /dev/null
@@ -1,14 +0,0 @@
-#! /bin/sh
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-arfile=$1
-output=$2
-shift 2
-pmdinfogen=$*
-
-# The generated file must not be empty if compiled in pedantic mode
-echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
-for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output
-done
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 23cefd4be..0a2e91a7b 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -2,7 +2,6 @@
 # Copyright(c) 2017-2019 Intel Corporation
 
 pkgconf = find_program('pkg-config', 'pkgconf', required: false)
-pmdinfo = find_program('gen-pmdinfo-cfile.sh')
 list_dir_globs = find_program('list-dir-globs.py')
 check_symbols = find_program('check-symbols.sh')
 ldflags_ibverbs_static = find_program('options-ibverbs-static.sh')
@@ -18,11 +17,18 @@ endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
 
-# select object file format
+# select library and object file format
+pmdinfo = py3 + files('gen-pmdinfo-cfile.py')
 pmdinfogen = py3 + files('pmdinfogen.py')
 if host_machine.system() == 'windows'
+	if cc.get_id() == 'gcc'
+		pmdinfo += 'ar'
+	else
+		pmdinfo += 'llvm-ar'
+	endif
 	pmdinfogen += 'coff'
 else
+	pmdinfo += 'ar'
 	pmdinfogen += 'elf'
 endif
 
diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
index 3209510eb..56f5f488c 100755
--- a/buildtools/pmdinfogen.py
+++ b/buildtools/pmdinfogen.py
@@ -230,12 +230,19 @@ def open_output(path):
     return open(path, "w")
 
 
+def write_header(output):
+    output.write(
+        "static __attribute__((unused)) const char *generator = \"%s\";\n" % sys.argv[0]
+    )
+
+
 def main():
     args = parse_args()
     if args.input.count('-') > 1:
         raise Exception("'-' input cannot be used multiple times")
 
     output = open_output(args.output)
+    write_header(output)
     for path in args.input:
         image = load_image(args.format, path)
         drivers = load_drivers(image)
-- 
2.29.2


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [dpdk-dev] [PATCH 4/4] build: enable pmdinfogen for Windows
  2020-12-12 23:34 [dpdk-dev] [PATCH 0/4] pmdinfogen: support Windows Dmitry Kozlyuk
                   ` (2 preceding siblings ...)
  2020-12-12 23:34 ` [dpdk-dev] [PATCH 3/4] buildtools: support object file extraction for Windows Dmitry Kozlyuk
@ 2020-12-12 23:34 ` Dmitry Kozlyuk
  2021-01-08  2:47 ` [dpdk-dev] [PATCH v2 0/4] pmdinfogen: support Windows Dmitry Kozlyuk
  4 siblings, 0 replies; 13+ messages in thread
From: Dmitry Kozlyuk @ 2020-12-12 23:34 UTC (permalink / raw)
  To: dev
  Cc: Narcisa Ana Maria Vasile, Pallavi Kadam, Dmitry Malloy, Dmitry Kozlyuk

Remove platform restriction when building drivers.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 drivers/meson.build | 26 ++++++++++++--------------
 1 file changed, 12 insertions(+), 14 deletions(-)

diff --git a/drivers/meson.build b/drivers/meson.build
index f49d4f79b..1dfa8738f 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -143,20 +143,18 @@ foreach subpath:subdirs
 			# lib and then running pmdinfogen on the contents of
 			# that lib. The final lib reuses the object files and
 			# adds in the new source file.
-			if not is_windows
-				out_filename = lib_name + '.pmd.c'
-				tmp_lib = static_library('tmp_' + lib_name,
-						sources,
-						include_directories: includes,
-						dependencies: static_deps,
-						c_args: cflags)
-				objs += tmp_lib.extract_all_objects()
-				sources = custom_target(out_filename,
-						command: [pmdinfo, tmp_lib.full_path(),
-							'@OUTPUT@', pmdinfogen],
-						output: out_filename,
-						depends: [tmp_lib])
-			endif
+			out_filename = lib_name + '.pmd.c'
+			tmp_lib = static_library('tmp_' + lib_name,
+					sources,
+					include_directories: includes,
+					dependencies: static_deps,
+					c_args: cflags)
+			objs += tmp_lib.extract_all_objects()
+			sources = custom_target(out_filename,
+					command: [pmdinfo, tmp_lib.full_path(),
+						'@OUTPUT@', pmdinfogen],
+					output: out_filename,
+					depends: [tmp_lib])
 
 			# now build the static driver
 			static_lib = static_library(lib_name,
-- 
2.29.2


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [dpdk-dev] [PATCH v2 0/4] pmdinfogen: support Windows
  2020-12-12 23:34 [dpdk-dev] [PATCH 0/4] pmdinfogen: support Windows Dmitry Kozlyuk
                   ` (3 preceding siblings ...)
  2020-12-12 23:34 ` [dpdk-dev] [PATCH 4/4] build: enable pmdinfogen " Dmitry Kozlyuk
@ 2021-01-08  2:47 ` Dmitry Kozlyuk
  2021-01-08  2:47   ` [dpdk-dev] [PATCH v2 1/4] pmdinfogen: support COFF Dmitry Kozlyuk
                     ` (4 more replies)
  4 siblings, 5 replies; 13+ messages in thread
From: Dmitry Kozlyuk @ 2021-01-08  2:47 UTC (permalink / raw)
  To: dev
  Cc: Narcisa Ana Maria Vasile, Pallavi Kadam, Dmitry Malloy,
	Bruce Richardson, Neil Horman, Dmitry Kozlyuk

Based on Python version of pmdinfogen, deferred until 21.02:

    http://patchwork.dpdk.org/project/dpdk/list/?series=13153

There are a few Python libraries for PE/COFF, none of which suits the need,
so a custom COFF parser is used.  Advice is welcome, options considered:

* py-coff (https://github.com/jeppeter/py-coff): doesn't give access to
  symbol values, Python 2 code inside, not very popular.

* pefile (https://github.com/erocarrera/pefile): for PE (executables and
  libraries), not COFF (objects); most popular.

* pype32-py3 (https://github.com/crackinglandia/pype32): ditto, less popular.

A script to extract object files from library is still required. Meson has
extract_all_objects(), but they can't be passed as inputs to custom_target()
until 0.52.0 (commit f431cff809).

v2:
    Don't require elftools on Windows (Jie Zhou).

Dmitry Kozlyuk (4):
  pmdinfogen: support COFF
  pmdinfogen: allow multiple input files
  buildtools: support object file extraction for Windows
  build: enable pmdinfogen for Windows

 buildtools/coff.py              | 154 ++++++++++++++++++++++++++++++++
 buildtools/gen-pmdinfo-cfile.py |  19 ++++
 buildtools/gen-pmdinfo-cfile.sh |  14 ---
 buildtools/meson.build          |  15 +++-
 buildtools/pmdinfogen.py        | 135 +++++++++++++++++++++-------
 drivers/meson.build             |  26 +++---
 6 files changed, 301 insertions(+), 62 deletions(-)
 create mode 100644 buildtools/coff.py
 create mode 100644 buildtools/gen-pmdinfo-cfile.py
 delete mode 100755 buildtools/gen-pmdinfo-cfile.sh

-- 
2.29.2


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [dpdk-dev] [PATCH v2 1/4] pmdinfogen: support COFF
  2021-01-08  2:47 ` [dpdk-dev] [PATCH v2 0/4] pmdinfogen: support Windows Dmitry Kozlyuk
@ 2021-01-08  2:47   ` Dmitry Kozlyuk
  2021-01-08  2:47   ` [dpdk-dev] [PATCH v2 2/4] pmdinfogen: allow multiple input files Dmitry Kozlyuk
                     ` (3 subsequent siblings)
  4 siblings, 0 replies; 13+ messages in thread
From: Dmitry Kozlyuk @ 2021-01-08  2:47 UTC (permalink / raw)
  To: dev
  Cc: Narcisa Ana Maria Vasile, Pallavi Kadam, Dmitry Malloy,
	Bruce Richardson, Neil Horman, Dmitry Kozlyuk

Common Object File Format (COFF) is used on Windows in place of ELF.

Add COFF parser to pmdinfogen. Also add an argument to specify input
file format, which is selected at configure time based on the target.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
Depends-on: series-13153 ("pmdinfogen: rewrite in Python")

 buildtools/coff.py       | 154 +++++++++++++++++++++++++++++++++++++++
 buildtools/meson.build   |   7 ++
 buildtools/pmdinfogen.py | 117 +++++++++++++++++++++--------
 3 files changed, 248 insertions(+), 30 deletions(-)
 create mode 100644 buildtools/coff.py

diff --git a/buildtools/coff.py b/buildtools/coff.py
new file mode 100644
index 000000000..86fb0602b
--- /dev/null
+++ b/buildtools/coff.py
@@ -0,0 +1,154 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2020 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import ctypes
+
+# x86_64 little-endian
+COFF_MAGIC = 0x8664
+
+# Names up to this length are stored immediately in symbol table entries.
+COFF_NAMELEN = 8
+
+# Special "section numbers" changing the meaning of symbol table entry.
+COFF_SN_UNDEFINED = 0
+COFF_SN_ABSOLUTE = -1
+COFF_SN_DEBUG = -2
+
+
+class CoffFileHeader(ctypes.LittleEndianStructure):
+    _pack_ = True
+    _fields_ = [
+        ("magic", ctypes.c_uint16),
+        ("section_count", ctypes.c_uint16),
+        ("timestamp", ctypes.c_uint32),
+        ("symbol_table_offset", ctypes.c_uint32),
+        ("symbol_count", ctypes.c_uint32),
+        ("optional_header_size", ctypes.c_uint16),
+        ("flags", ctypes.c_uint16),
+    ]
+
+
+class CoffName(ctypes.Union):
+    class Reference(ctypes.LittleEndianStructure):
+        _pack_ = True
+        _fields_ = [
+            ("zeroes", ctypes.c_uint32),
+            ("offset", ctypes.c_uint32),
+        ]
+
+    Immediate = ctypes.c_char * 8
+
+    _pack_ = True
+    _fields_ = [
+        ("immediate", Immediate),
+        ("reference", Reference),
+    ]
+
+
+class CoffSection(ctypes.LittleEndianStructure):
+    _pack_ = True
+    _fields_ = [
+        ("name", CoffName),
+        ("physical_address", ctypes.c_uint32),
+        ("physical_address", ctypes.c_uint32),
+        ("size", ctypes.c_uint32),
+        ("data_offset", ctypes.c_uint32),
+        ("relocations_offset", ctypes.c_uint32),
+        ("line_numbers_offset", ctypes.c_uint32),
+        ("relocation_count", ctypes.c_uint16),
+        ("line_number_count", ctypes.c_uint16),
+        ("flags", ctypes.c_uint32),
+    ]
+
+
+class CoffSymbol(ctypes.LittleEndianStructure):
+    _pack_ = True
+    _fields_ = [
+        ("name", CoffName),
+        ("value", ctypes.c_uint32),
+        ("section_number", ctypes.c_int16),
+        ("type", ctypes.c_uint16),
+        ("storage_class", ctypes.c_uint8),
+        ("auxiliary_count", ctypes.c_uint8),
+    ]
+
+
+class Symbol:
+    def __init__(self, image, symbol: CoffSymbol):
+        self._image = image
+        self._coff = symbol
+
+    @property
+    def name(self):
+        if self._coff.name.reference.zeroes:
+            return decode_asciiz(bytes(self._coff.name.immediate))
+
+        offset = self._coff.name.reference.offset
+        offset -= ctypes.sizeof(ctypes.c_uint32)
+        return self._image.get_string(offset)
+
+    def get_value(self, offset):
+        section_number = self._coff.section_number
+
+        if section_number == COFF_SN_UNDEFINED:
+            return None
+
+        if section_number == COFF_SN_DEBUG:
+            return None
+
+        if section_number == COFF_SN_ABSOLUTE:
+            return bytes(ctypes.c_uint32(self._coff.value))
+
+        section_data = self._image.get_section_data(section_number)
+        section_offset = self._coff.value + offset
+        return section_data[section_offset:]
+
+
+class Image:
+    def __init__(self, data):
+        header = CoffFileHeader.from_buffer_copy(data)
+        header_size = ctypes.sizeof(header) + header.optional_header_size
+
+        sections_desc = CoffSection * header.section_count
+        sections = sections_desc.from_buffer_copy(data, header_size)
+
+        symbols_desc = CoffSymbol * header.symbol_count
+        symbols = symbols_desc.from_buffer_copy(data, header.symbol_table_offset)
+
+        strings_offset = header.symbol_table_offset + ctypes.sizeof(symbols)
+        strings = Image._parse_strings(data[strings_offset:])
+
+        self._data = data
+        self._header = header
+        self._sections = sections
+        self._symbols = symbols
+        self._strings = strings
+
+    @staticmethod
+    def _parse_strings(data):
+        full_size = ctypes.c_uint32.from_buffer_copy(data)
+        header_size = ctypes.sizeof(full_size)
+        return data[header_size : full_size.value]
+
+    @property
+    def symbols(self):
+        i = 0
+        while i < self._header.symbol_count:
+            symbol = self._symbols[i]
+            yield Symbol(self, symbol)
+            i += symbol.auxiliary_count + 1
+
+    def get_section_data(self, number):
+        # section numbers are 1-based
+        section = self._sections[number - 1]
+        base = section.data_offset
+        return self._data[base : base + section.size]
+
+    def get_string(self, offset):
+        return decode_asciiz(self._strings[offset:])
+
+
+def decode_asciiz(data):
+    index = data.find(b'\x00')
+    end = index if index >= 0 else len(data)
+    return data[:end].decode()
diff --git a/buildtools/meson.build b/buildtools/meson.build
index dd4c0f640..23cefd4be 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -17,7 +17,14 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+
+# select object file format
 pmdinfogen = py3 + files('pmdinfogen.py')
+if host_machine.system() == 'windows'
+	pmdinfogen += 'coff'
+else
+	pmdinfogen += 'elf'
+endif
 
 # TODO: starting from Meson 0.51.0 use
 # 	python3 = import('python').find_installation('python',
diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
index 0cca47ff1..474168f21 100755
--- a/buildtools/pmdinfogen.py
+++ b/buildtools/pmdinfogen.py
@@ -9,8 +9,13 @@
 import sys
 import tempfile
 
-from elftools.elf.elffile import ELFFile
-from elftools.elf.sections import SymbolTableSection
+try:
+    from elftools.elf.elffile import ELFFile
+    from elftools.elf.sections import SymbolTableSection
+except ImportError:
+    pass
+
+import coff
 
 
 class ELFSymbol:
@@ -18,21 +23,18 @@ def __init__(self, image, symbol):
         self._image = image
         self._symbol = symbol
 
-    @property
-    def size(self):
-        return self._symbol["st_size"]
-
-    @property
-    def value(self):
-        data = self._image.get_section_data(self._symbol["st_shndx"])
-        base = self._symbol["st_value"]
-        return data[base:base + self.size]
-
     @property
     def string_value(self):
-        value = self.value
+        size = self._symbol["st_size"]
+        value = self.get_value(0, size)
         return value[:-1].decode() if value else ""
 
+    def get_value(self, offset, size):
+        section = self._symbol["st_shndx"]
+        data = self._image.get_section(section).data()
+        base = self._symbol["st_value"] + offset
+        return data[base : base + size]
+
 
 class ELFImage:
     def __init__(self, data):
@@ -45,18 +47,50 @@ def __init__(self, data):
     def is_big_endian(self):
         return not self._image.little_endian
 
-    def get_section_data(self, name):
-        return self._image.get_section(name).data()
-
     def find_by_name(self, name):
         symbol = self._symtab.get_symbol_by_name(name)
-        return ELFSymbol(self, symbol[0]) if symbol else None
+        return ELFSymbol(self._image, symbol[0]) if symbol else None
 
     def find_by_prefix(self, prefix):
         for i in range(self._symtab.num_symbols()):
             symbol = self._symtab.get_symbol(i)
             if symbol.name.startswith(prefix):
-                yield ELFSymbol(self, symbol)
+                yield ELFSymbol(self._image, symbol)
+
+
+class COFFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    def get_value(self, offset, size):
+        value = self._symbol.get_value(offset)
+        return value[:size] if value else value
+
+    @property
+    def string_value(self):
+        value = self._symbol.get_value(0)
+        return coff.decode_asciiz(value) if value else ''
+
+
+class COFFImage:
+    def __init__(self, data):
+        self._image = coff.Image(data)
+
+    @property
+    def is_big_endian(self):
+        return False
+
+    def find_by_prefix(self, prefix):
+        for symbol in self._image.symbols:
+            if symbol.name.startswith(prefix):
+                yield COFFSymbol(self._image, symbol)
+
+    def find_by_name(self, name):
+        for symbol in self._image.symbols:
+            if symbol.name == name:
+                return COFFSymbol(self._image, symbol)
+        return None
 
 
 def define_rte_pci_id(is_big_endian):
@@ -117,19 +151,24 @@ def _load_pci_ids(image, table_name_symbol):
 
         rte_pci_id = define_rte_pci_id(image.is_big_endian)
 
-        pci_id_size = ctypes.sizeof(rte_pci_id)
-        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
-        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
         result = []
-        for pci_id in pci_ids:
+        while True:
+            size = ctypes.sizeof(rte_pci_id)
+            offset = size * len(result)
+            data = table_symbol.get_value(offset, size)
+            if not data:
+                break
+            pci_id = rte_pci_id.from_buffer_copy(data)
             if not pci_id.device_id:
                 break
-            result.append([
-                pci_id.vendor_id,
-                pci_id.device_id,
-                pci_id.subsystem_vendor_id,
-                pci_id.subsystem_device_id,
-                ])
+            result.append(
+                [
+                    pci_id.vendor_id,
+                    pci_id.device_id,
+                    pci_id.subsystem_vendor_id,
+                    pci_id.subsystem_device_id,
+                ]
+            )
         return result
 
     def dump(self, file):
@@ -157,6 +196,7 @@ def dump_drivers(drivers, file):
 
 def parse_args():
     parser = argparse.ArgumentParser()
+    parser.add_argument("format", help="object file format, 'elf' or 'coff'")
     parser.add_argument("input", help="input object file path or '-' for stdin")
     parser.add_argument("output", help="output C file path or '-' for stdout")
     return parser.parse_args()
@@ -170,6 +210,21 @@ def open_input(path):
     return open(path, "rb")
 
 
+def read_input(path):
+    if path == "-":
+        return sys.stdin.buffer.read()
+    with open(path, "rb") as file:
+        return file.read()
+
+
+def load_image(fmt, path):
+    if fmt == "elf":
+        return ELFImage(open_input(path))
+    if fmt == "coff":
+        return COFFImage(read_input(path))
+    raise Exception("unsupported object file format")
+
+
 def open_output(path):
     if path == "-":
         return sys.stdout
@@ -178,8 +233,10 @@ def open_output(path):
 
 def main():
     args = parse_args()
-    infile = open_input(args.input)
-    image = ELFImage(infile)
+    if args.format == "elf" and "ELFFile" not in globals():
+        raise Exception("elftools module not found")
+
+    image = load_image(args.format, args.input)
     drivers = load_drivers(image)
     output = open_output(args.output)
     dump_drivers(drivers, output)
-- 
2.29.2


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [dpdk-dev] [PATCH v2 2/4] pmdinfogen: allow multiple input files
  2021-01-08  2:47 ` [dpdk-dev] [PATCH v2 0/4] pmdinfogen: support Windows Dmitry Kozlyuk
  2021-01-08  2:47   ` [dpdk-dev] [PATCH v2 1/4] pmdinfogen: support COFF Dmitry Kozlyuk
@ 2021-01-08  2:47   ` Dmitry Kozlyuk
  2021-01-08  2:47   ` [dpdk-dev] [PATCH v2 3/4] buildtools: support object file extraction for Windows Dmitry Kozlyuk
                     ` (2 subsequent siblings)
  4 siblings, 0 replies; 13+ messages in thread
From: Dmitry Kozlyuk @ 2021-01-08  2:47 UTC (permalink / raw)
  To: dev
  Cc: Narcisa Ana Maria Vasile, Pallavi Kadam, Dmitry Malloy,
	Bruce Richardson, Neil Horman, Dmitry Kozlyuk

Process any number of input object files and write a unified output.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 buildtools/pmdinfogen.py | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
index 474168f21..965c08945 100755
--- a/buildtools/pmdinfogen.py
+++ b/buildtools/pmdinfogen.py
@@ -197,7 +197,9 @@ def dump_drivers(drivers, file):
 def parse_args():
     parser = argparse.ArgumentParser()
     parser.add_argument("format", help="object file format, 'elf' or 'coff'")
-    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument(
+        "input", nargs='+', help="input object file path or '-' for stdin"
+    )
     parser.add_argument("output", help="output C file path or '-' for stdout")
     return parser.parse_args()
 
@@ -233,13 +235,16 @@ def open_output(path):
 
 def main():
     args = parse_args()
+    if args.input.count('-') > 1:
+        raise Exception("'-' input cannot be used multiple times")
     if args.format == "elf" and "ELFFile" not in globals():
         raise Exception("elftools module not found")
 
-    image = load_image(args.format, args.input)
-    drivers = load_drivers(image)
     output = open_output(args.output)
-    dump_drivers(drivers, output)
+    for path in args.input:
+        image = load_image(args.format, path)
+        drivers = load_drivers(image)
+        dump_drivers(drivers, output)
 
 
 if __name__ == "__main__":
-- 
2.29.2


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [dpdk-dev] [PATCH v2 3/4] buildtools: support object file extraction for Windows
  2021-01-08  2:47 ` [dpdk-dev] [PATCH v2 0/4] pmdinfogen: support Windows Dmitry Kozlyuk
  2021-01-08  2:47   ` [dpdk-dev] [PATCH v2 1/4] pmdinfogen: support COFF Dmitry Kozlyuk
  2021-01-08  2:47   ` [dpdk-dev] [PATCH v2 2/4] pmdinfogen: allow multiple input files Dmitry Kozlyuk
@ 2021-01-08  2:47   ` Dmitry Kozlyuk
  2021-01-25 15:57     ` Thomas Monjalon
  2021-01-08  2:47   ` [dpdk-dev] [PATCH v2 4/4] build: enable pmdinfogen " Dmitry Kozlyuk
  2021-01-25 15:59   ` [dpdk-dev] [PATCH v2 0/4] pmdinfogen: support Windows Thomas Monjalon
  4 siblings, 1 reply; 13+ messages in thread
From: Dmitry Kozlyuk @ 2021-01-08  2:47 UTC (permalink / raw)
  To: dev
  Cc: Narcisa Ana Maria Vasile, Pallavi Kadam, Dmitry Malloy,
	Bruce Richardson, Neil Horman, Dmitry Kozlyuk

clang archiver tool is llvm-ar on Windows and ar on other platforms.
MinGW always uses ar. Replace shell script (Unix-only) that calls ar
with a Python script (OS-independent) that calls an appropriate archiver
tool selected at configuration time. Move the logic not to generate
empty sources into pmdinfogen.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 buildtools/gen-pmdinfo-cfile.py | 19 +++++++++++++++++++
 buildtools/gen-pmdinfo-cfile.sh | 14 --------------
 buildtools/meson.build          | 10 ++++++++--
 buildtools/pmdinfogen.py        |  7 +++++++
 4 files changed, 34 insertions(+), 16 deletions(-)
 create mode 100644 buildtools/gen-pmdinfo-cfile.py
 delete mode 100755 buildtools/gen-pmdinfo-cfile.sh

diff --git a/buildtools/gen-pmdinfo-cfile.py b/buildtools/gen-pmdinfo-cfile.py
new file mode 100644
index 000000000..f1f289ffe
--- /dev/null
+++ b/buildtools/gen-pmdinfo-cfile.py
@@ -0,0 +1,19 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2020 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import os
+import subprocess
+import sys
+import tempfile
+
+_, ar, archive, output, *pmdinfogen = sys.argv
+with tempfile.TemporaryDirectory() as temp:
+    proc = subprocess.run(
+        # Don't use "ar p", because its output is corrupted on Windows.
+        [ar, "xv", os.path.abspath(archive)], capture_output=True, check=True, cwd=temp
+    )
+    lines = proc.stdout.decode().splitlines()
+    names = [line[len("x - ") :] for line in lines]
+    paths = [os.path.join(temp, name) for name in names]
+    subprocess.run(pmdinfogen + paths + [output], check=True)
diff --git a/buildtools/gen-pmdinfo-cfile.sh b/buildtools/gen-pmdinfo-cfile.sh
deleted file mode 100755
index 109ee461e..000000000
--- a/buildtools/gen-pmdinfo-cfile.sh
+++ /dev/null
@@ -1,14 +0,0 @@
-#! /bin/sh
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-arfile=$1
-output=$2
-shift 2
-pmdinfogen=$*
-
-# The generated file must not be empty if compiled in pedantic mode
-echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
-for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output
-done
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 23cefd4be..0a2e91a7b 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -2,7 +2,6 @@
 # Copyright(c) 2017-2019 Intel Corporation
 
 pkgconf = find_program('pkg-config', 'pkgconf', required: false)
-pmdinfo = find_program('gen-pmdinfo-cfile.sh')
 list_dir_globs = find_program('list-dir-globs.py')
 check_symbols = find_program('check-symbols.sh')
 ldflags_ibverbs_static = find_program('options-ibverbs-static.sh')
@@ -18,11 +17,18 @@ endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
 
-# select object file format
+# select library and object file format
+pmdinfo = py3 + files('gen-pmdinfo-cfile.py')
 pmdinfogen = py3 + files('pmdinfogen.py')
 if host_machine.system() == 'windows'
+	if cc.get_id() == 'gcc'
+		pmdinfo += 'ar'
+	else
+		pmdinfo += 'llvm-ar'
+	endif
 	pmdinfogen += 'coff'
 else
+	pmdinfo += 'ar'
 	pmdinfogen += 'elf'
 endif
 
diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
index 965c08945..893a6c571 100755
--- a/buildtools/pmdinfogen.py
+++ b/buildtools/pmdinfogen.py
@@ -233,6 +233,12 @@ def open_output(path):
     return open(path, "w")
 
 
+def write_header(output):
+    output.write(
+        "static __attribute__((unused)) const char *generator = \"%s\";\n" % sys.argv[0]
+    )
+
+
 def main():
     args = parse_args()
     if args.input.count('-') > 1:
@@ -241,6 +247,7 @@ def main():
         raise Exception("elftools module not found")
 
     output = open_output(args.output)
+    write_header(output)
     for path in args.input:
         image = load_image(args.format, path)
         drivers = load_drivers(image)
-- 
2.29.2


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [dpdk-dev] [PATCH v2 4/4] build: enable pmdinfogen for Windows
  2021-01-08  2:47 ` [dpdk-dev] [PATCH v2 0/4] pmdinfogen: support Windows Dmitry Kozlyuk
                     ` (2 preceding siblings ...)
  2021-01-08  2:47   ` [dpdk-dev] [PATCH v2 3/4] buildtools: support object file extraction for Windows Dmitry Kozlyuk
@ 2021-01-08  2:47   ` Dmitry Kozlyuk
  2021-01-08 18:29     ` Jie Zhou
  2021-01-25 15:59   ` [dpdk-dev] [PATCH v2 0/4] pmdinfogen: support Windows Thomas Monjalon
  4 siblings, 1 reply; 13+ messages in thread
From: Dmitry Kozlyuk @ 2021-01-08  2:47 UTC (permalink / raw)
  To: dev
  Cc: Narcisa Ana Maria Vasile, Pallavi Kadam, Dmitry Malloy,
	Bruce Richardson, Neil Horman, Dmitry Kozlyuk

Remove platform restriction when building drivers.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 drivers/meson.build | 26 ++++++++++++--------------
 1 file changed, 12 insertions(+), 14 deletions(-)

diff --git a/drivers/meson.build b/drivers/meson.build
index f49d4f79b..1dfa8738f 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -143,20 +143,18 @@ foreach subpath:subdirs
 			# lib and then running pmdinfogen on the contents of
 			# that lib. The final lib reuses the object files and
 			# adds in the new source file.
-			if not is_windows
-				out_filename = lib_name + '.pmd.c'
-				tmp_lib = static_library('tmp_' + lib_name,
-						sources,
-						include_directories: includes,
-						dependencies: static_deps,
-						c_args: cflags)
-				objs += tmp_lib.extract_all_objects()
-				sources = custom_target(out_filename,
-						command: [pmdinfo, tmp_lib.full_path(),
-							'@OUTPUT@', pmdinfogen],
-						output: out_filename,
-						depends: [tmp_lib])
-			endif
+			out_filename = lib_name + '.pmd.c'
+			tmp_lib = static_library('tmp_' + lib_name,
+					sources,
+					include_directories: includes,
+					dependencies: static_deps,
+					c_args: cflags)
+			objs += tmp_lib.extract_all_objects()
+			sources = custom_target(out_filename,
+					command: [pmdinfo, tmp_lib.full_path(),
+						'@OUTPUT@', pmdinfogen],
+					output: out_filename,
+					depends: [tmp_lib])
 
 			# now build the static driver
 			static_lib = static_library(lib_name,
-- 
2.29.2


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [dpdk-dev] [PATCH v2 4/4] build: enable pmdinfogen for Windows
  2021-01-08  2:47   ` [dpdk-dev] [PATCH v2 4/4] build: enable pmdinfogen " Dmitry Kozlyuk
@ 2021-01-08 18:29     ` Jie Zhou
  0 siblings, 0 replies; 13+ messages in thread
From: Jie Zhou @ 2021-01-08 18:29 UTC (permalink / raw)
  To: Dmitry Kozlyuk
  Cc: dev, Narcisa Ana Maria Vasile, Pallavi Kadam, Dmitry Malloy,
	Bruce Richardson, Neil Horman

On Fri, Jan 08, 2021 at 05:47:23AM +0300, Dmitry Kozlyuk wrote:
> Remove platform restriction when building drivers.
> 
> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> ---
>  drivers/meson.build | 26 ++++++++++++--------------
>  1 file changed, 12 insertions(+), 14 deletions(-)

Tested-by: Jie Zhou <jizh@linux.microsoft.com> 

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [dpdk-dev] [PATCH v2 3/4] buildtools: support object file extraction for Windows
  2021-01-08  2:47   ` [dpdk-dev] [PATCH v2 3/4] buildtools: support object file extraction for Windows Dmitry Kozlyuk
@ 2021-01-25 15:57     ` Thomas Monjalon
  0 siblings, 0 replies; 13+ messages in thread
From: Thomas Monjalon @ 2021-01-25 15:57 UTC (permalink / raw)
  To: Dmitry Kozlyuk
  Cc: dev, Narcisa Ana Maria Vasile, Pallavi Kadam, Dmitry Malloy,
	Bruce Richardson, david.marchand, talshn

08/01/2021 03:47, Dmitry Kozlyuk:
> --- /dev/null
> +++ b/buildtools/gen-pmdinfo-cfile.py
[...]
> +with tempfile.TemporaryDirectory() as temp:
> +    proc = subprocess.run(
> +        # Don't use "ar p", because its output is corrupted on Windows.
> +        [ar, "xv", os.path.abspath(archive)], capture_output=True, check=True, cwd=temp
> +    )

Not sure where is this temp file.
Please can you make sure it created inside the build directory?
It can be a follow-up patch, thanks.



^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [dpdk-dev] [PATCH v2 0/4] pmdinfogen: support Windows
  2021-01-08  2:47 ` [dpdk-dev] [PATCH v2 0/4] pmdinfogen: support Windows Dmitry Kozlyuk
                     ` (3 preceding siblings ...)
  2021-01-08  2:47   ` [dpdk-dev] [PATCH v2 4/4] build: enable pmdinfogen " Dmitry Kozlyuk
@ 2021-01-25 15:59   ` Thomas Monjalon
  4 siblings, 0 replies; 13+ messages in thread
From: Thomas Monjalon @ 2021-01-25 15:59 UTC (permalink / raw)
  To: Dmitry Kozlyuk
  Cc: dev, Narcisa Ana Maria Vasile, Pallavi Kadam, Dmitry Malloy,
	Bruce Richardson

08/01/2021 03:47, Dmitry Kozlyuk:
> Dmitry Kozlyuk (4):
>   pmdinfogen: support COFF
>   pmdinfogen: allow multiple input files
>   buildtools: support object file extraction for Windows
>   build: enable pmdinfogen for Windows

Applied, thanks.

Please follow-up on temporary file location.



^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2021-01-25 15:59 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-12 23:34 [dpdk-dev] [PATCH 0/4] pmdinfogen: support Windows Dmitry Kozlyuk
2020-12-12 23:34 ` [dpdk-dev] [PATCH 1/4] pmdinfogen: support COFF Dmitry Kozlyuk
2020-12-12 23:34 ` [dpdk-dev] [PATCH 2/4] pmdinfogen: allow multiple input files Dmitry Kozlyuk
2020-12-12 23:34 ` [dpdk-dev] [PATCH 3/4] buildtools: support object file extraction for Windows Dmitry Kozlyuk
2020-12-12 23:34 ` [dpdk-dev] [PATCH 4/4] build: enable pmdinfogen " Dmitry Kozlyuk
2021-01-08  2:47 ` [dpdk-dev] [PATCH v2 0/4] pmdinfogen: support Windows Dmitry Kozlyuk
2021-01-08  2:47   ` [dpdk-dev] [PATCH v2 1/4] pmdinfogen: support COFF Dmitry Kozlyuk
2021-01-08  2:47   ` [dpdk-dev] [PATCH v2 2/4] pmdinfogen: allow multiple input files Dmitry Kozlyuk
2021-01-08  2:47   ` [dpdk-dev] [PATCH v2 3/4] buildtools: support object file extraction for Windows Dmitry Kozlyuk
2021-01-25 15:57     ` Thomas Monjalon
2021-01-08  2:47   ` [dpdk-dev] [PATCH v2 4/4] build: enable pmdinfogen " Dmitry Kozlyuk
2021-01-08 18:29     ` Jie Zhou
2021-01-25 15:59   ` [dpdk-dev] [PATCH v2 0/4] pmdinfogen: support Windows Thomas Monjalon

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).