[dpdk-dev] [PATCH] pmdinfogen: allow padding after NUL terminator

[dpdk-dev] [PATCH] pmdinfogen: allow padding after NUL terminator

[Dmitry Kozlyuk]

Size of string constant symbol may be larger than its length
measured up to NUL terminator. In this case pmdinfogen included padding
bytes after NUL terminator in generated source, yielding incorrect code.

Always trim string data to NUL terminator while reading ELF.
It was already done for COFF because there's no symbol size.

Bugzilla ID: 720
Fixes: f0f93a7adfee ("buildtools: use Python pmdinfogen")

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 buildtools/coff.py       |  6 ------
 buildtools/pmdinfogen.py | 10 ++++++++--
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/buildtools/coff.py b/buildtools/coff.py
index 86fb0602b7..a7b6c37e32 100644
--- a/buildtools/coff.py
+++ b/buildtools/coff.py
@@ -146,9 +146,3 @@ def get_section_data(self, number):
 
     def get_string(self, offset):
         return decode_asciiz(self._strings[offset:])
-
-
-def decode_asciiz(data):
-    index = data.find(b'\x00')
-    end = index if index >= 0 else len(data)
-    return data[:end].decode()
diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
index 7a739ec7d4..8bb59699b8 100755
--- a/buildtools/pmdinfogen.py
+++ b/buildtools/pmdinfogen.py
@@ -19,6 +19,12 @@
 import coff
 
 
+def decode_asciiz(data):
+    index = data.find(b'\x00')
+    end = index if index >= 0 else len(data)
+    return data[:end].decode()
+
+
 class ELFSymbol:
     def __init__(self, image, symbol):
         self._image = image
@@ -28,7 +34,7 @@ def __init__(self, image, symbol):
     def string_value(self):
         size = self._symbol["st_size"]
         value = self.get_value(0, size)
-        return value[:-1].decode() if value else ""
+        return decode_asciiz(value) if value else ""
 
     def get_value(self, offset, size):
         section = self._symbol["st_shndx"]
@@ -86,7 +92,7 @@ def get_value(self, offset, size):
     @property
     def string_value(self):
         value = self._symbol.get_value(0)
-        return coff.decode_asciiz(value) if value else ''
+        return decode_asciiz(value) if value else ""
 
 
 class COFFImage:
-- 
2.29.3

Re: [dpdk-dev] [PATCH] pmdinfogen: allow padding after NUL terminator

[David Marchand]

On Wed, May 26, 2021 at 11:44 PM Dmitry Kozlyuk
<dmitry.kozliuk@gmail.com> wrote:
>
> Size of string constant symbol may be larger than its length
> measured up to NUL terminator. In this case pmdinfogen included padding
> bytes after NUL terminator in generated source, yielding incorrect code.
>
> Always trim string data to NUL terminator while reading ELF.
> It was already done for COFF because there's no symbol size.
>
> Bugzilla ID: 720
> Fixes: f0f93a7adfee ("buildtools: use Python pmdinfogen")
>
> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>

Reproduced the issue described in bz on my (old) RHEL7 with clang 3.4.2.

Reviewed-by: David Marchand <david.marchand@redhat.com>

Just to confirm my reading of the C version of pmdinfogen: the C
version formats those symbols fine with printf %s.
So there should be no need for a fix in stable branches, right?


-- 
David Marchand

Re: [dpdk-dev] [PATCH] pmdinfogen: allow padding after NUL terminator

[Dmitry Kozlyuk]

On Thu, May 27, 2021, 09:53 David Marchand <david.marchand@redhat.com>
wrote:

> Just to confirm my reading of the C version of pmdinfogen: the C version
> formats those symbols fine with printf %s.
> So there should be no need for a fix in stable branches, right?
>

Yes.

>

Re: [dpdk-dev] [PATCH] pmdinfogen: allow padding after NUL terminator

[David Marchand]

On Wed, May 26, 2021 at 11:44 PM Dmitry Kozlyuk
<dmitry.kozliuk@gmail.com> wrote:
>
> Size of string constant symbol may be larger than its length
> measured up to NUL terminator. In this case pmdinfogen included padding
> bytes after NUL terminator in generated source, yielding incorrect code.
>
> Always trim string data to NUL terminator while reading ELF.
> It was already done for COFF because there's no symbol size.
>
> Bugzilla ID: 720
> Fixes: f0f93a7adfee ("buildtools: use Python pmdinfogen")
>
> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> ---
>  buildtools/coff.py       |  6 ------
>  buildtools/pmdinfogen.py | 10 ++++++++--
>  2 files changed, 8 insertions(+), 8 deletions(-)
>
> diff --git a/buildtools/coff.py b/buildtools/coff.py
> index 86fb0602b7..a7b6c37e32 100644
> --- a/buildtools/coff.py
> +++ b/buildtools/coff.py
> @@ -146,9 +146,3 @@ def get_section_data(self, number):
>
>      def get_string(self, offset):
>          return decode_asciiz(self._strings[offset:])
                  ^^^^

coff.py still needs this helper.

Caught in the lab:
Traceback (most recent call last):

  File "../buildtools/pmdinfogen.py", line 280, in <module>
    main()
  File "../buildtools/pmdinfogen.py", line 275, in main
    drivers = load_drivers(image)
  File "../buildtools/pmdinfogen.py", line 208, in load_drivers
    for symbol in image.find_by_prefix("this_pmd_name"):
  File "../buildtools/pmdinfogen.py", line 108, in find_by_prefix
    if symbol.name.startswith(prefix):
  File "C:\Users\builder\jenkins\workspace\Windows-Compile-DPDK-Mingw64\dpdk\buildtools\coff.py",
line 84, in name
    return decode_asciiz(bytes(self._coff.name.immediate))
NameError: name 'decode_asciiz' is not defined
Traceback (most recent call last):
  File "../buildtools/gen-pmdinfo-cfile.py", line 20, in <module>
    subprocess.run(pmdinfogen + paths + [output], check=True)
  File "c:\python38\lib\subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,


-- 
David Marchand

[dpdk-dev] [PATCH v2] pmdinfogen: allow padding after NUL terminator

[Dmitry Kozlyuk]

Size of string constant symbol may be larger than its length
measured up to NUL terminator. In this case pmdinfogen included padding
bytes after NUL terminator in generated source, yielding incorrect code.

Always trim string data to NUL terminator while reading ELF.
It was already done for COFF because there's no symbol size.

Bugzilla ID: 720
Fixes: f0f93a7adfee ("buildtools: use Python pmdinfogen")

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
v2: return helper to coff.py, where it's needed (David Marchand).

 buildtools/pmdinfogen.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
index 7a739ec7d4..2a44f17bda 100755
--- a/buildtools/pmdinfogen.py
+++ b/buildtools/pmdinfogen.py
@@ -28,7 +28,7 @@ def __init__(self, image, symbol):
     def string_value(self):
         size = self._symbol["st_size"]
         value = self.get_value(0, size)
-        return value[:-1].decode() if value else ""
+        return coff.decode_asciiz(value)  # not COFF-specific
 
     def get_value(self, offset, size):
         section = self._symbol["st_shndx"]
-- 
2.29.3

Re: [dpdk-dev] [PATCH v2] pmdinfogen: allow padding after NUL terminator

[Dmitry Kozlyuk]

2021-05-28 00:24 (UTC+0300), Dmitry Kozlyuk:
> Size of string constant symbol may be larger than its length
> measured up to NUL terminator. In this case pmdinfogen included padding
> bytes after NUL terminator in generated source, yielding incorrect code.
> 
> Always trim string data to NUL terminator while reading ELF.
> It was already done for COFF because there's no symbol size.
> 
> Bugzilla ID: 720
> Fixes: f0f93a7adfee ("buildtools: use Python pmdinfogen")
> 
> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> ---
> v2: return helper to coff.py, where it's needed (David Marchand).
> 
>  buildtools/pmdinfogen.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
> index 7a739ec7d4..2a44f17bda 100755
> --- a/buildtools/pmdinfogen.py
> +++ b/buildtools/pmdinfogen.py
> @@ -28,7 +28,7 @@ def __init__(self, image, symbol):
>      def string_value(self):
>          size = self._symbol["st_size"]
>          value = self.get_value(0, size)
> -        return value[:-1].decode() if value else ""
> +        return coff.decode_asciiz(value)  # not COFF-specific
>  
>      def get_value(self, offset, size):
>          section = self._symbol["st_shndx"]

There are CI failures that seem unrelated to this patch:
some tests with NICs that I can't check
and an Arch Linux build failure that I failed to reproduce.
GitHub Actions are passing.
Are these known CI bugs or does this patch need any corrections?

Re: [dpdk-dev] [dpdk-ci] [PATCH v2] pmdinfogen: allow padding after NUL terminator

[Lincoln Lavoie]

Hi Dmitry,

If the failing test is the unit test func_reentrancy_autotest, that is
being looked into , as it seems like the test case does not reliably run.
It passes / fails between different systems running both on bare metal and
in virtual environments, on different OSes and architectures.

Do you have a link to your patch in patchworks or the lab dashboard?

Cheers,
Lincoln



On Wed, Jun 9, 2021 at 11:52 AM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
wrote:

> 2021-05-28 00:24 (UTC+0300), Dmitry Kozlyuk:
> > Size of string constant symbol may be larger than its length
> > measured up to NUL terminator. In this case pmdinfogen included padding
> > bytes after NUL terminator in generated source, yielding incorrect code.
> >
> > Always trim string data to NUL terminator while reading ELF.
> > It was already done for COFF because there's no symbol size.
> >
> > Bugzilla ID: 720
> > Fixes: f0f93a7adfee ("buildtools: use Python pmdinfogen")
> >
> > Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> > ---
> > v2: return helper to coff.py, where it's needed (David Marchand).
> >
> >  buildtools/pmdinfogen.py | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/buildtools/pmdinfogen.py b/buildtools/pmdinfogen.py
> > index 7a739ec7d4..2a44f17bda 100755
> > --- a/buildtools/pmdinfogen.py
> > +++ b/buildtools/pmdinfogen.py
> > @@ -28,7 +28,7 @@ def __init__(self, image, symbol):
> >      def string_value(self):
> >          size = self._symbol["st_size"]
> >          value = self.get_value(0, size)
> > -        return value[:-1].decode() if value else ""
> > +        return coff.decode_asciiz(value)  # not COFF-specific
> >
> >      def get_value(self, offset, size):
> >          section = self._symbol["st_shndx"]
>
> There are CI failures that seem unrelated to this patch:
> some tests with NICs that I can't check
> and an Arch Linux build failure that I failed to reproduce.
> GitHub Actions are passing.
> Are these known CI bugs or does this patch need any corrections?
>


-- 
*Lincoln Lavoie*
Principal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
<https://www.iol.unh.edu>

Re: [dpdk-dev] [dpdk-ci] [PATCH v2] pmdinfogen: allow padding after NUL terminator

[Dmitry Kozlyuk]

2021-06-09 12:01 (UTC-0400), Lincoln Lavoie:
> Hi Dmitry,
> 
> If the failing test is the unit test func_reentrancy_autotest, that is
> being looked into , as it seems like the test case does not reliably run.
> It passes / fails between different systems running both on bare metal and
> in virtual environments, on different OSes and architectures.
> 
> Do you have a link to your patch in patchworks or the lab dashboard?

Hi Lincoln,

Yes, this is func_reentrancy_autotest failure.
Dashboard link:
https://lab.dpdk.org/results/dashboard/patchsets/17240/
MTU and stats can hardly be affected by this patch.

Re: [dpdk-dev] [dpdk-ci] [PATCH v2] pmdinfogen: allow padding after NUL terminator

[Owen Hilyard]

Hi Dmitry,

Those failures are the result of a known issue with that machine. Those
tests were disabled on that machine shortly after you submitted that patch.
I've re-run the patch.

Owen Hilyard

On Wed, Jun 9, 2021 at 12:48 PM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
wrote:

> 2021-06-09 12:01 (UTC-0400), Lincoln Lavoie:
> > Hi Dmitry,
> >
> > If the failing test is the unit test func_reentrancy_autotest, that is
> > being looked into , as it seems like the test case does not reliably run.
> > It passes / fails between different systems running both on bare metal
> and
> > in virtual environments, on different OSes and architectures.
> >
> > Do you have a link to your patch in patchworks or the lab dashboard?
>
> Hi Lincoln,
>
> Yes, this is func_reentrancy_autotest failure.
> Dashboard link:
> https://lab.dpdk.org/results/dashboard/patchsets/17240/
> MTU and stats can hardly be affected by this patch.
>

Re: [dpdk-dev] [PATCH v2] pmdinfogen: allow padding after NUL terminator

[Thomas Monjalon]

27/05/2021 23:24, Dmitry Kozlyuk:
> Size of string constant symbol may be larger than its length
> measured up to NUL terminator. In this case pmdinfogen included padding
> bytes after NUL terminator in generated source, yielding incorrect code.
> 
> Always trim string data to NUL terminator while reading ELF.
> It was already done for COFF because there's no symbol size.
> 
> Bugzilla ID: 720
> Fixes: f0f93a7adfee ("buildtools: use Python pmdinfogen")
> 
> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>

Applied, thanks