From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9395FA04F0 for ; Thu, 19 Dec 2019 15:40:24 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 88D481BFD5; Thu, 19 Dec 2019 15:40:24 +0100 (CET) Received: from mail-wm1-f49.google.com (mail-wm1-f49.google.com [209.85.128.49]) by dpdk.org (Postfix) with ESMTP id 2DCC71BFE1 for ; Thu, 19 Dec 2019 15:40:23 +0100 (CET) Received: by mail-wm1-f49.google.com with SMTP id t14so5850000wmi.5 for ; Thu, 19 Dec 2019 06:40:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5aejuS6n22vOrmA59xYoUdifaydkHw3E3ACu4VegqLg=; b=kR7lAGeqxJzoCa1ksUtDUOM0eueoLbmSvZr+MBJfMsiSuL20EWu0xEsp3YVKQhvW2/ fnbvwwIvRBmWI3dEF9M7xDvj9noWSHLTNmPB04mMGMdBdhX4dq+reDVZQh/9le6O7SHD fwB9ZH0kuD5ZAPlE9HGkhTQmVmAaSGRrDNsghoqhTFICqPwynoAAkP3LbOtdIne8LaaG ockXR2rC1noeYAMJqpFBkfbCz+FuiQpS9bVJAxCFJP2L6SlBp1LReEJdYY6I6UrFn63q JfhhV+dSWjw7o0IvjCwt8rGl4l7UIeeMt8QqEE6uZ41R5EPT+Vdj8gqxj2uZmHSyhpd8 UqCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5aejuS6n22vOrmA59xYoUdifaydkHw3E3ACu4VegqLg=; b=rxowAVoBX9mfAz0hUARxb6K1x40fy5O+INJA6D8h70VKv/08nL/eHbbCp+pGHdqZAb A6JJudXGKmzNe1ujdjM2+l2Q1gg7urXqRDFciuCjKUO+i2axotjJ36HDWEUZyhNF/Uu8 EWAUIxz4mTgKn3kbIElHyQD8pUiyWeweFNjbmyT04dWbE9VJ4TxpT6SUekID15qzMBW0 taNz+UUlLHKyA8e3NLoFw5u35WTsIh6jRkw0ESHDkLZwfGkmPgU/43CiX6d7SL5Ievh4 RBnL3Q4/FMBAhlLQzaLlJd/sozfhIJg0qhYXyo3j3BVEI7Euui7D6xVuPN2WuWvslGAC kH2w== X-Gm-Message-State: APjAAAWAuckwj/rOwBizyUb+kWnNuhUQlK/GlUkaeFjPBgMQdiDc1ok5 hafgyQnAIIOb+Aq0T1pzZu0= X-Google-Smtp-Source: APXvYqwD7uPNikcVEemkAX61iZ/MRj+UR0axiA4ihdTdHv9MOAIJWFqajoVjP9aJWLfmYRdF2MvXWg== X-Received: by 2002:a7b:c450:: with SMTP id l16mr10619553wmi.31.1576766422606; Thu, 19 Dec 2019 06:40:22 -0800 (PST) Received: from localhost ([88.98.246.218]) by smtp.gmail.com with ESMTPSA id i8sm6690600wro.47.2019.12.19.06.40.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Dec 2019 06:40:22 -0800 (PST) From: luca.boccassi@gmail.com To: Robin Jarry Cc: Olivier Matz , dpdk stable Date: Thu, 19 Dec 2019 14:34:04 +0000 Message-Id: <20191219143447.21506-97-luca.boccassi@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191219143447.21506-1-luca.boccassi@gmail.com> References: <20191219143447.21506-1-luca.boccassi@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-stable] patch 'usertools: fix pmdinfo with python 3 and pyelftools>=0.24' has been queued to LTS release 17.11.10 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Sender: "stable" Hi, FYI, your patch has been queued to LTS release 17.11.10 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 12/21/19. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Thanks. Luca Boccassi --- >From 4bb877c4bf4c583f5556865494ff69c3ae57a1b1 Mon Sep 17 00:00:00 2001 From: Robin Jarry Date: Tue, 15 Oct 2019 14:39:17 +0200 Subject: [PATCH] usertools: fix pmdinfo with python 3 and pyelftools>=0.24 [ upstream commit 4da069194ef4578aac7ae10cf05abd992c61723c ] Running dpdk-pmdinfo.py on Ubuntu 18.04 (bionic) with python 3 and pyelftools installed produces no output but no error is reported neither: ~$ python3 usertools/dpdk-pmdinfo.py -r build/app/testpmd ~$ echo $? 0 While with python 2, it works: ~# python2 usertools/dpdk-pmdinfo.py -r build/app/testpmd {"pci_ids": [], "name": "dpio"} {"pci_ids": [], "name": "dpbp"} {"pci_ids": [], "name": "dpaa2_qdma"} ..... On Ubuntu 18.04, pyelftools is version 0.24. The change log of pyelftools v0.24 says: - Symbol/section names are strings internally now, not bytestrings (this may affect API usage in Python 3) (#76). We cannot guess which version of pyelftools is actually being used. The elftools.__version__ symbol is not consistent with each distro's package version. For example, on Ubuntu 16.04 (xenial), the .deb package version is '0.23-2' but elftools.__version__ contains '0.25'. This is certainly due to partial backports. To have a more consistent behaviour of this script across all versions of python, add the unicode_literals future import so that literal strings are now always "unicode". Add 2 utility functions to force a string into bytes or bytes into an unicode string. Force pyelftools return values to unicode strings (will do nothing with recent version of pyelftools). If elffile.get_section_by_name returns None with a unicode section name, try with the same one encoded as bytes. Also, replace all open() calls by io.open() which behaves like the builtin open in python 3. The only non-binary opened file is /usr/share/hwdata/pci.ids which is UTF-8 encoded text. Explicitly specify that encoding. Link: https://github.com/eliben/pyelftools/blob/v0.24/CHANGES#L7 Link: https://github.com/eliben/pyelftools/commit/108eaea9e75a8b5a Fixes: 54ca545dce4b ("make python scripts python2/3 compliant") Signed-off-by: Robin Jarry Reviewed-by: Olivier Matz --- usertools/dpdk-pmdinfo.py | 65 +++++++++++++++++++++++++++------------ 1 file changed, 46 insertions(+), 19 deletions(-) diff --git a/usertools/dpdk-pmdinfo.py b/usertools/dpdk-pmdinfo.py index 46c1be081b..0fc51f0cbf 100755 --- a/usertools/dpdk-pmdinfo.py +++ b/usertools/dpdk-pmdinfo.py @@ -6,13 +6,15 @@ # # ------------------------------------------------------------------------- from __future__ import print_function +from __future__ import unicode_literals import json +import io import os import platform import string import sys from elftools.common.exceptions import ELFError -from elftools.common.py3compat import (byte2int, bytes2str, str2bytes) +from elftools.common.py3compat import byte2int from elftools.elf.elffile import ELFFile from optparse import OptionParser @@ -211,7 +213,8 @@ class PCIIds: """ Reads the local file """ - self.contents = open(filename).readlines() + with io.open(filename, 'r', encoding='utf-8') as f: + self.contents = f.readlines() self.date = self.findDate(self.contents) def loadLocal(self): @@ -265,7 +268,13 @@ class ReadElf(object): return None except ValueError: # Not a number. Must be a name then - return self.elffile.get_section_by_name(str2bytes(spec)) + section = self.elffile.get_section_by_name(force_unicode(spec)) + if section is None: + # No match with a unicode name. + # Some versions of pyelftools (<= 0.23) store internal strings + # as bytes. Try again with the name encoded as bytes. + section = self.elffile.get_section_by_name(force_bytes(spec)) + return section def pretty_print_pmdinfo(self, pmdinfo): global pcidb @@ -337,7 +346,8 @@ class ReadElf(object): while endptr < len(data) and byte2int(data[endptr]) != 0: endptr += 1 - mystring = bytes2str(data[dataptr:endptr]) + # pyelftools may return byte-strings, force decode them + mystring = force_unicode(data[dataptr:endptr]) rc = mystring.find("PMD_INFO_STRING") if (rc != -1): self.parse_pmd_info_string(mystring) @@ -346,9 +356,10 @@ class ReadElf(object): def find_librte_eal(self, section): for tag in section.iter_tags(): - if tag.entry.d_tag == 'DT_NEEDED': - if "librte_eal" in tag.needed: - return tag.needed + # pyelftools may return byte-strings, force decode them + if force_unicode(tag.entry.d_tag) == 'DT_NEEDED': + if "librte_eal" in force_unicode(tag.needed): + return force_unicode(tag.needed) return None def search_for_autoload_path(self): @@ -371,7 +382,7 @@ class ReadElf(object): return (None, None) if raw_output is False: print("Scanning for autoload path in %s" % library) - scanfile = open(library, 'rb') + scanfile = io.open(library, 'rb') scanelf = ReadElf(scanfile, sys.stdout) except AttributeError: # Not a dynamic binary @@ -401,7 +412,8 @@ class ReadElf(object): while endptr < len(data) and byte2int(data[endptr]) != 0: endptr += 1 - mystring = bytes2str(data[dataptr:endptr]) + # pyelftools may return byte-strings, force decode them + mystring = force_unicode(data[dataptr:endptr]) rc = mystring.find("DPDK_PLUGIN_PATH") if (rc != -1): rc = mystring.find("=") @@ -414,8 +426,9 @@ class ReadElf(object): def get_dt_runpath(self, dynsec): for tag in dynsec.iter_tags(): - if tag.entry.d_tag == 'DT_RUNPATH': - return tag.runpath + # pyelftools may return byte-strings, force decode them + if force_unicode(tag.entry.d_tag) == 'DT_RUNPATH': + return force_unicode(tag.runpath) return "" def process_dt_needed_entries(self): @@ -436,16 +449,16 @@ class ReadElf(object): return for tag in dynsec.iter_tags(): - if tag.entry.d_tag == 'DT_NEEDED': - rc = tag.needed.find(b"librte_pmd") - if (rc != -1): - library = search_file(tag.needed, + # pyelftools may return byte-strings, force decode them + if force_unicode(tag.entry.d_tag) == 'DT_NEEDED': + if 'librte_pmd' in force_unicode(tag.needed): + library = search_file(force_unicode(tag.needed), runpath + ":" + ldlibpath + ":/usr/lib64:/lib64:/usr/lib:/lib") if library is not None: if raw_output is False: print("Scanning %s for pmd information" % library) - with open(library, 'rb') as file: + with io.open(library, 'rb') as file: try: libelf = ReadElf(file, sys.stdout) except ELFError: @@ -456,6 +469,20 @@ class ReadElf(object): file.close() +# compat: remove force_unicode & force_bytes when pyelftools<=0.23 support is +# dropped. +def force_unicode(s): + if hasattr(s, 'decode') and callable(s.decode): + s = s.decode('latin-1') # same encoding used in pyelftools py3compat + return s + + +def force_bytes(s): + if hasattr(s, 'encode') and callable(s.encode): + s = s.encode('latin-1') # same encoding used in pyelftools py3compat + return s + + def scan_autoload_path(autoload_path): global raw_output @@ -474,7 +501,7 @@ def scan_autoload_path(autoload_path): scan_autoload_path(dpath) if os.path.isfile(dpath): try: - file = open(dpath, 'rb') + file = io.open(dpath, 'rb') readelf = ReadElf(file, sys.stdout) except ELFError: # this is likely not an elf file, skip it @@ -501,7 +528,7 @@ def scan_for_autoload_pmds(dpdk_path): print("Must specify a file name") return - file = open(dpdk_path, 'rb') + file = io.open(dpdk_path, 'rb') try: readelf = ReadElf(file, sys.stdout) except ElfError: @@ -593,7 +620,7 @@ def main(stream=None): print("File not found") sys.exit(1) - with open(myelffile, 'rb') as file: + with io.open(myelffile, 'rb') as file: try: readelf = ReadElf(file, sys.stdout) readelf.process_dt_needed_entries() -- 2.20.1 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2019-12-19 14:32:30.195909058 +0000 +++ 0097-usertools-fix-pmdinfo-with-python-3-and-pyelftools-0.patch 2019-12-19 14:32:26.225300205 +0000 @@ -1,8 +1,10 @@ -From 4da069194ef4578aac7ae10cf05abd992c61723c Mon Sep 17 00:00:00 2001 +From 4bb877c4bf4c583f5556865494ff69c3ae57a1b1 Mon Sep 17 00:00:00 2001 From: Robin Jarry Date: Tue, 15 Oct 2019 14:39:17 +0200 Subject: [PATCH] usertools: fix pmdinfo with python 3 and pyelftools>=0.24 +[ upstream commit 4da069194ef4578aac7ae10cf05abd992c61723c ] + Running dpdk-pmdinfo.py on Ubuntu 18.04 (bionic) with python 3 and pyelftools installed produces no output but no error is reported neither: @@ -53,7 +55,6 @@ Link: https://github.com/eliben/pyelftools/commit/108eaea9e75a8b5a Fixes: 54ca545dce4b ("make python scripts python2/3 compliant") -Cc: stable@dpdk.org Signed-off-by: Robin Jarry Reviewed-by: Olivier Matz @@ -62,10 +63,10 @@ 1 file changed, 46 insertions(+), 19 deletions(-) diff --git a/usertools/dpdk-pmdinfo.py b/usertools/dpdk-pmdinfo.py -index 03623d5b8b..069a3bf124 100755 +index 46c1be081b..0fc51f0cbf 100755 --- a/usertools/dpdk-pmdinfo.py +++ b/usertools/dpdk-pmdinfo.py -@@ -8,13 +8,15 @@ +@@ -6,13 +6,15 @@ # # ------------------------------------------------------------------------- from __future__ import print_function @@ -82,7 +83,7 @@ from elftools.elf.elffile import ELFFile from optparse import OptionParser -@@ -213,7 +215,8 @@ class PCIIds: +@@ -211,7 +213,8 @@ class PCIIds: """ Reads the local file """ @@ -92,7 +93,7 @@ self.date = self.findDate(self.contents) def loadLocal(self): -@@ -267,7 +270,13 @@ class ReadElf(object): +@@ -265,7 +268,13 @@ class ReadElf(object): return None except ValueError: # Not a number. Must be a name then @@ -107,7 +108,7 @@ def pretty_print_pmdinfo(self, pmdinfo): global pcidb -@@ -339,7 +348,8 @@ class ReadElf(object): +@@ -337,7 +346,8 @@ class ReadElf(object): while endptr < len(data) and byte2int(data[endptr]) != 0: endptr += 1 @@ -117,7 +118,7 @@ rc = mystring.find("PMD_INFO_STRING") if (rc != -1): self.parse_pmd_info_string(mystring) -@@ -348,9 +358,10 @@ class ReadElf(object): +@@ -346,9 +356,10 @@ class ReadElf(object): def find_librte_eal(self, section): for tag in section.iter_tags(): @@ -131,7 +132,7 @@ return None def search_for_autoload_path(self): -@@ -373,7 +384,7 @@ class ReadElf(object): +@@ -371,7 +382,7 @@ class ReadElf(object): return (None, None) if raw_output is False: print("Scanning for autoload path in %s" % library) @@ -140,7 +141,7 @@ scanelf = ReadElf(scanfile, sys.stdout) except AttributeError: # Not a dynamic binary -@@ -403,7 +414,8 @@ class ReadElf(object): +@@ -401,7 +412,8 @@ class ReadElf(object): while endptr < len(data) and byte2int(data[endptr]) != 0: endptr += 1 @@ -150,7 +151,7 @@ rc = mystring.find("DPDK_PLUGIN_PATH") if (rc != -1): rc = mystring.find("=") -@@ -416,8 +428,9 @@ class ReadElf(object): +@@ -414,8 +426,9 @@ class ReadElf(object): def get_dt_runpath(self, dynsec): for tag in dynsec.iter_tags(): @@ -162,7 +163,7 @@ return "" def process_dt_needed_entries(self): -@@ -438,16 +451,16 @@ class ReadElf(object): +@@ -436,16 +449,16 @@ class ReadElf(object): return for tag in dynsec.iter_tags(): @@ -184,7 +185,7 @@ try: libelf = ReadElf(file, sys.stdout) except ELFError: -@@ -458,6 +471,20 @@ class ReadElf(object): +@@ -456,6 +469,20 @@ class ReadElf(object): file.close() @@ -205,7 +206,7 @@ def scan_autoload_path(autoload_path): global raw_output -@@ -476,7 +503,7 @@ def scan_autoload_path(autoload_path): +@@ -474,7 +501,7 @@ def scan_autoload_path(autoload_path): scan_autoload_path(dpath) if os.path.isfile(dpath): try: @@ -214,7 +215,7 @@ readelf = ReadElf(file, sys.stdout) except ELFError: # this is likely not an elf file, skip it -@@ -503,7 +530,7 @@ def scan_for_autoload_pmds(dpdk_path): +@@ -501,7 +528,7 @@ def scan_for_autoload_pmds(dpdk_path): print("Must specify a file name") return @@ -223,7 +224,7 @@ try: readelf = ReadElf(file, sys.stdout) except ElfError: -@@ -595,7 +622,7 @@ def main(stream=None): +@@ -593,7 +620,7 @@ def main(stream=None): print("File not found") sys.exit(1)