From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8107EA2EFC for ; Tue, 15 Oct 2019 14:39:30 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 316B71E95A; Tue, 15 Oct 2019 14:39:29 +0200 (CEST) Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) by dpdk.org (Postfix) with ESMTP id 1F7F91E947 for ; Tue, 15 Oct 2019 14:39:28 +0200 (CEST) Received: by mail-wr1-f42.google.com with SMTP id b9so23698276wrs.0 for ; Tue, 15 Oct 2019 05:39:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/NnhKEOx472eSnR7xK6PODToM2r7ZCsRf6DlU9j7sNM=; b=QWGueZx9moyWe00UnExKdCwuPnfEfUL2OhJt0p44qVwZUnvCCzFM0Id1KxPPkZh50F UdIyQxcFgWOVBusr8RcoevZDPriPVnEIAT1Z+OpLGBH6t6vBQkVZyDotJ3EPYrIHv8CX BzWVvYO3SFa47peP7xQUzGt5fjrARv+ZGwuQTtGnRUUcxcPhd9CWDtKR3HDdNDTgHPLZ UgiBTmePR13zhzUrojje8gI9LzZ8Ila3rJ0JWhoXriUaS5vteGjFJUKqENueimuE3voi r85L8Na3GtHklomAewEKoCsPPOoUVnKFlEgSilwQgjJT6VEJB4lwBMkpiwflYsELGxx9 L9cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/NnhKEOx472eSnR7xK6PODToM2r7ZCsRf6DlU9j7sNM=; b=ppV8LERRAqRrBBwIg692GApD3uALtSDnbazGgKRUGh8AO9L+UOzY9iX2bbrOkEE4HY e4GNvcMJF7OcaefSOd5hNd17haldN9oiu/ONHR291iNT1eQYUuRiuWtdugQTofcnObsy 9mjMZIbnvCXHoYxaDwMh6B1a9jna7JkhmR4jsvkCgZoUd6woezw3Y50c5BbfX0L+75I8 IpWlDQ4gOsSmDICHTjtUdjW0H28TlyneiNMKSPB5gLZlJJ2kZXPdyVRd8gJibHZeew2M TIpUJNjaZxPPweXs4zmyIyc0vr/DmfghdSuWilnFJ3oUXB8gFvKwS79LaLuj3LCXq5J6 RkOQ== X-Gm-Message-State: APjAAAWIu+8LAJnq4nCle4Wr1gBuBeYnFYxSkC/fPRF95WMxevppJFJN NpbERtwW7iium9AEpFt1avOZQg== X-Google-Smtp-Source: APXvYqyr9EkxYSv7SMXzluUxnb7sYKATw/xdqcv19WTVFsJFBuJcVY/Gnax5pKLLUmA4k8Xzc+DDlw== X-Received: by 2002:a5d:6b0a:: with SMTP id v10mr28697501wrw.32.1571143167621; Tue, 15 Oct 2019 05:39:27 -0700 (PDT) Received: from tom.dev.6wind.com. (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id t6sm38626805wmf.8.2019.10.15.05.39.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Oct 2019 05:39:26 -0700 (PDT) From: Robin Jarry To: Neil Horman Cc: dev@dpdk.org, John McNamara , stable@dpdk.org, Olivier Matz Date: Tue, 15 Oct 2019 14:39:17 +0200 Message-Id: <20191015123918.10775-1-robin.jarry@6wind.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20190910071135.27546-1-robin.jarry@6wind.com> References: <20190910071135.27546-1-robin.jarry@6wind.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH v3] usertools: fix py3 support with pyelftools>=0.24 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Running dpdk-pmdinfo.py on Ubuntu 18.04 (bionic) with python 3 and pyelftools installed produces no output but no error is reported neither: ~$ python3 usertools/dpdk-pmdinfo.py -r build/app/testpmd ~$ echo $? 0 While with python 2, it works: ~# python2 usertools/dpdk-pmdinfo.py -r build/app/testpmd {"pci_ids": [], "name": "dpio"} {"pci_ids": [], "name": "dpbp"} {"pci_ids": [], "name": "dpaa2_qdma"} ..... On Ubuntu 18.04, pyelftools is version 0.24. The change log of pyelftools v0.24 says: - Symbol/section names are strings internally now, not bytestrings (this may affect API usage in Python 3) (#76). We cannot guess which version of pyelftools is actually being used. The elftools.__version__ symbol is not consistent with each distro's package version. For example, on Ubuntu 16.04 (xenial), the .deb package version is '0.23-2' but elftools.__version__ contains '0.25'. This is certainly due to partial backports. To have a more consistent behaviour of this script across all versions of python, add the unicode_literals future import so that literal strings are now always "unicode". Add 2 utility functions to force a string into bytes or bytes into an unicode string. Force pyelftools return values to unicode strings (will do nothing with recent version of pyelftools). If elffile.get_section_by_name returns None with a unicode section name, try with the same one encoded as bytes. Also, replace all open() calls by io.open() which behaves like the builtin open in python 3. The only non-binary opened file is /usr/share/hwdata/pci.ids which is UTF-8 encoded text. Explicitly specify that encoding. Link: https://github.com/eliben/pyelftools/blob/v0.24/CHANGES#L7 Link: https://github.com/eliben/pyelftools/commit/108eaea9e75a8b5a Fixes: 54ca545dce4b ("make python scripts python2/3 compliant") Cc: John McNamara Cc: stable@dpdk.org Signed-off-by: Robin Jarry Reviewed-by: Olivier Matz --- Change log: v2 * Behavior is different depending on pyelftools versions. Handle that properly. * More bytes/unicode fixes. v3 * Add missing Fixes line * Add Cc: stable@dpdk.org usertools/dpdk-pmdinfo.py | 65 +++++++++++++++++++++++++++------------ 1 file changed, 46 insertions(+), 19 deletions(-) diff --git a/usertools/dpdk-pmdinfo.py b/usertools/dpdk-pmdinfo.py index 03623d5b8b48..069a3bf124b2 100755 --- a/usertools/dpdk-pmdinfo.py +++ b/usertools/dpdk-pmdinfo.py @@ -8,13 +8,15 @@ # # ------------------------------------------------------------------------- from __future__ import print_function +from __future__ import unicode_literals import json +import io import os import platform import string import sys from elftools.common.exceptions import ELFError -from elftools.common.py3compat import (byte2int, bytes2str, str2bytes) +from elftools.common.py3compat import byte2int from elftools.elf.elffile import ELFFile from optparse import OptionParser @@ -213,7 +215,8 @@ def readLocal(self, filename): """ Reads the local file """ - self.contents = open(filename).readlines() + with io.open(filename, 'r', encoding='utf-8') as f: + self.contents = f.readlines() self.date = self.findDate(self.contents) def loadLocal(self): @@ -267,7 +270,13 @@ def _section_from_spec(self, spec): return None except ValueError: # Not a number. Must be a name then - return self.elffile.get_section_by_name(str2bytes(spec)) + section = self.elffile.get_section_by_name(force_unicode(spec)) + if section is None: + # No match with a unicode name. + # Some versions of pyelftools (<= 0.23) store internal strings + # as bytes. Try again with the name encoded as bytes. + section = self.elffile.get_section_by_name(force_bytes(spec)) + return section def pretty_print_pmdinfo(self, pmdinfo): global pcidb @@ -339,7 +348,8 @@ def display_pmd_info_strings(self, section_spec): while endptr < len(data) and byte2int(data[endptr]) != 0: endptr += 1 - mystring = bytes2str(data[dataptr:endptr]) + # pyelftools may return byte-strings, force decode them + mystring = force_unicode(data[dataptr:endptr]) rc = mystring.find("PMD_INFO_STRING") if (rc != -1): self.parse_pmd_info_string(mystring) @@ -348,9 +358,10 @@ def display_pmd_info_strings(self, section_spec): def find_librte_eal(self, section): for tag in section.iter_tags(): - if tag.entry.d_tag == 'DT_NEEDED': - if "librte_eal" in tag.needed: - return tag.needed + # pyelftools may return byte-strings, force decode them + if force_unicode(tag.entry.d_tag) == 'DT_NEEDED': + if "librte_eal" in force_unicode(tag.needed): + return force_unicode(tag.needed) return None def search_for_autoload_path(self): @@ -373,7 +384,7 @@ def search_for_autoload_path(self): return (None, None) if raw_output is False: print("Scanning for autoload path in %s" % library) - scanfile = open(library, 'rb') + scanfile = io.open(library, 'rb') scanelf = ReadElf(scanfile, sys.stdout) except AttributeError: # Not a dynamic binary @@ -403,7 +414,8 @@ def search_for_autoload_path(self): while endptr < len(data) and byte2int(data[endptr]) != 0: endptr += 1 - mystring = bytes2str(data[dataptr:endptr]) + # pyelftools may return byte-strings, force decode them + mystring = force_unicode(data[dataptr:endptr]) rc = mystring.find("DPDK_PLUGIN_PATH") if (rc != -1): rc = mystring.find("=") @@ -416,8 +428,9 @@ def search_for_autoload_path(self): def get_dt_runpath(self, dynsec): for tag in dynsec.iter_tags(): - if tag.entry.d_tag == 'DT_RUNPATH': - return tag.runpath + # pyelftools may return byte-strings, force decode them + if force_unicode(tag.entry.d_tag) == 'DT_RUNPATH': + return force_unicode(tag.runpath) return "" def process_dt_needed_entries(self): @@ -438,16 +451,16 @@ def process_dt_needed_entries(self): return for tag in dynsec.iter_tags(): - if tag.entry.d_tag == 'DT_NEEDED': - rc = tag.needed.find(b"librte_pmd") - if (rc != -1): - library = search_file(tag.needed, + # pyelftools may return byte-strings, force decode them + if force_unicode(tag.entry.d_tag) == 'DT_NEEDED': + if 'librte_pmd' in force_unicode(tag.needed): + library = search_file(force_unicode(tag.needed), runpath + ":" + ldlibpath + ":/usr/lib64:/lib64:/usr/lib:/lib") if library is not None: if raw_output is False: print("Scanning %s for pmd information" % library) - with open(library, 'rb') as file: + with io.open(library, 'rb') as file: try: libelf = ReadElf(file, sys.stdout) except ELFError: @@ -458,6 +471,20 @@ def process_dt_needed_entries(self): file.close() +# compat: remove force_unicode & force_bytes when pyelftools<=0.23 support is +# dropped. +def force_unicode(s): + if hasattr(s, 'decode') and callable(s.decode): + s = s.decode('latin-1') # same encoding used in pyelftools py3compat + return s + + +def force_bytes(s): + if hasattr(s, 'encode') and callable(s.encode): + s = s.encode('latin-1') # same encoding used in pyelftools py3compat + return s + + def scan_autoload_path(autoload_path): global raw_output @@ -476,7 +503,7 @@ def scan_autoload_path(autoload_path): scan_autoload_path(dpath) if os.path.isfile(dpath): try: - file = open(dpath, 'rb') + file = io.open(dpath, 'rb') readelf = ReadElf(file, sys.stdout) except ELFError: # this is likely not an elf file, skip it @@ -503,7 +530,7 @@ def scan_for_autoload_pmds(dpdk_path): print("Must specify a file name") return - file = open(dpdk_path, 'rb') + file = io.open(dpdk_path, 'rb') try: readelf = ReadElf(file, sys.stdout) except ElfError: @@ -595,7 +622,7 @@ def main(stream=None): print("File not found") sys.exit(1) - with open(myelffile, 'rb') as file: + with io.open(myelffile, 'rb') as file: try: readelf = ReadElf(file, sys.stdout) readelf.process_dt_needed_entries() -- 2.23.0