[dpdk-ci] ABI test failing for openSUSE and Arch Linux

[dpdk-ci] ABI test failing for openSUSE and Arch Linux

[David Marchand]

Hello,

Looking at the dashboard, we have ABI checks reporting failures.
Example: https://lab.dpdk.org/results/dashboard/patchsets/17309/

I can't debug this, but I suspect that the reference dumps are not
generated the same way the patches are tested.


-- 
David Marchand

Re: [dpdk-ci] [dpdklab] ABI test failing for openSUSE and Arch Linux

[Lincoln Lavoie]

[-- Attachment #1: Type: text/plain, Size: 3157 bytes --]

All,

The ABI references for all systems were updated this week to the 21.05
release code.  The two failures look like places where the interfaces
didn't actually change. We do see the failures across multiple patches,
which might imply something got merged that caused these changes / failures.

---------------------------------------------------------------------------------------------
OpenSUSE
2 Removed function symbols not referenced by debug info:

  [D] _fini
  [D] _init

---------------------------------------------------------------------------------------------
Arch Linux (one example from the output)

Functions changes summary: 0 Removed, 0 Changed, 0 Added function
Variables changes summary: 0 Removed, 1 Changed (13 filtered out), 0 Added
variables

1 Changed variable:

  [C] 'rte_table_ops rte_table_acl_ops' was changed at rte_table_acl.h:60:1:
    type of variable changed:
      type size hasn't changed
      1 data member change:
        type of 'rte_table_op_lookup f_lookup' changed:
          underlying type 'int (void*, rte_mbuf**, typedef uint64_t,
uint64_t*, void**)*' changed:
            in pointed to type 'function type int (void*, rte_mbuf**,
typedef uint64_t, uint64_t*, void**)':
              parameter 2 of type 'rte_mbuf**' has sub-type changes:
                in pointed to type 'rte_mbuf*':
                  in pointed to type 'struct rte_mbuf' at
rte_mbuf_core.h:484:1:
                    type size hasn't changed
                    1 data member changes (1 filtered):
                      type of 'anonymous data member union {uint32_t
packet_type; struct {uint8_t l2_type; uint8_t l3_type; uint8_t l4_type;
uint8_t tun_type; union {uint8_t inner_esp_next_proto; struct {uint8_t
inner_l2_type; uint8_t inner_l3_type;};}; uint8_t inner_l4_type;};}'
changed:
                        type size hasn't changed
                        1 data member change:
                          type of 'anonymous data member struct {uint8_t
l2_type; uint8_t l3_type; uint8_t l4_type; uint8_t tun_type; union {uint8_t
inner_esp_next_proto; struct {uint8_t inner_l2_type; uint8_t
inner_l3_type;};}; uint8_t inner_l4_type;}' changed:
                            type size hasn't changed
                            3 data member changes:
                              'uint8_t inner_l4_type' offset changed from 0
to 24 (in bits) (by +24 bits)
                              'uint8_t l4_type' offset changed from 0 to 8
(in bits) (by +8 bits)
                              'uint8_t tun_type' offset changed from 4 to
12 (in bits) (by +8 bits)




On Fri, Jun 4, 2021 at 3:59 AM David Marchand <david.marchand@redhat.com>
wrote:

> Hello,
>
> Looking at the dashboard, we have ABI checks reporting failures.
> Example: https://lab.dpdk.org/results/dashboard/patchsets/17309/
>
> I can't debug this, but I suspect that the reference dumps are not
> generated the same way the patches are tested.
>
>
> --
> David Marchand
>
>

-- 
*Lincoln Lavoie*
Principal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
<https://www.iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 5555 bytes --]

Re: [dpdk-ci] [dpdklab] ABI test failing for openSUSE and Arch Linux

[David Marchand]

On Fri, Jun 4, 2021 at 3:53 PM Lincoln Lavoie <lylavoie@iol.unh.edu> wrote:
>
> All,
>
> The ABI references for all systems were updated this week to the 21.05 release code.  The two failures look like places where the interfaces didn't actually change. We do see the failures across multiple patches, which might imply something got merged that caused these changes / failures.

I checked ABI for 64307fad7d2b ("telemetry: remove static limit on
callbacks count") against v21.05, yesterday.
There is no issue.

The report I mentioned
https://lab.dpdk.org/results/dashboard/patchsets/17309/ is for a patch
added on top of 64307fad7d2b.
And the patch for this patchset won't break ABI.


Can you confirm the ABI check runs fine against the main repo?
I can't find a report for it in lab.dpdk.org.
https://lab.dpdk.org/results/dashboard/tarballs/


-- 
David Marchand

Re: [dpdk-ci] [dpdklab] ABI test failing for openSUSE and Arch Linux

[Owen Hilyard]

[-- Attachment #1: Type: text/plain, Size: 1191 bytes --]

All,

The ABI reference seems to have been generated incorrectly. After
re-generating the ABI reference and re-running all of the patches, ABI
passed for all affected patches.

Owen Hilyard

On Fri, Jun 4, 2021 at 10:02 AM David Marchand <david.marchand@redhat.com>
wrote:

> On Fri, Jun 4, 2021 at 3:53 PM Lincoln Lavoie <lylavoie@iol.unh.edu>
> wrote:
> >
> > All,
> >
> > The ABI references for all systems were updated this week to the 21.05
> release code.  The two failures look like places where the interfaces
> didn't actually change. We do see the failures across multiple patches,
> which might imply something got merged that caused these changes / failures.
>
> I checked ABI for 64307fad7d2b ("telemetry: remove static limit on
> callbacks count") against v21.05, yesterday.
> There is no issue.
>
> The report I mentioned
> https://lab.dpdk.org/results/dashboard/patchsets/17309/ is for a patch
> added on top of 64307fad7d2b.
> And the patch for this patchset won't break ABI.
>
>
> Can you confirm the ABI check runs fine against the main repo?
> I can't find a report for it in lab.dpdk.org.
> https://lab.dpdk.org/results/dashboard/tarballs/
>
>
> --
> David Marchand
>
>

[-- Attachment #2: Type: text/html, Size: 1959 bytes --]

Re: [dpdk-ci] [dpdklab] ABI test failing for openSUSE and Arch Linux

[David Marchand]

Trying to do a post mortem.


On Fri, Jun 4, 2021 at 3:53 PM Lincoln Lavoie <lylavoie@iol.unh.edu> wrote:
> The ABI references for all systems were updated this week to the 21.05 release code.  The two failures look like places where the interfaces didn't actually change. We do see the failures across multiple patches, which might imply something got merged that caused these changes / failures.
>
> ---------------------------------------------------------------------------------------------
> OpenSUSE
> 2 Removed function symbols not referenced by debug info:
>
>   [D] _fini
>   [D] _init

This one, I am not sure what was wrong.
This is not a symbol from DPDK itself.
This comes from a libc / toolchain and/or linker change.


>
> ---------------------------------------------------------------------------------------------
> Arch Linux (one example from the output)
>
> Functions changes summary: 0 Removed, 0 Changed, 0 Added function
> Variables changes summary: 0 Removed, 1 Changed (13 filtered out), 0 Added variables
>
> 1 Changed variable:
>
>   [C] 'rte_table_ops rte_table_acl_ops' was changed at rte_table_acl.h:60:1:
>     type of variable changed:
>       type size hasn't changed
>       1 data member change:
>         type of 'rte_table_op_lookup f_lookup' changed:
>           underlying type 'int (void*, rte_mbuf**, typedef uint64_t, uint64_t*, void**)*' changed:
>             in pointed to type 'function type int (void*, rte_mbuf**, typedef uint64_t, uint64_t*, void**)':
>               parameter 2 of type 'rte_mbuf**' has sub-type changes:
>                 in pointed to type 'rte_mbuf*':
>                   in pointed to type 'struct rte_mbuf' at rte_mbuf_core.h:484:1:
>                     type size hasn't changed
>                     1 data member changes (1 filtered):
>                       type of 'anonymous data member union {uint32_t packet_type; struct {uint8_t l2_type; uint8_t l3_type; uint8_t l4_type; uint8_t tun_type; union {uint8_t inner_esp_next_proto; struct {uint8_t inner_l2_type; uint8_t inner_l3_type;};}; uint8_t inner_l4_type;};}' changed:
>                         type size hasn't changed
>                         1 data member change:
>                           type of 'anonymous data member struct {uint8_t l2_type; uint8_t l3_type; uint8_t l4_type; uint8_t tun_type; union {uint8_t inner_esp_next_proto; struct {uint8_t inner_l2_type; uint8_t inner_l3_type;};}; uint8_t inner_l4_type;}' changed:
>                             type size hasn't changed
>                             3 data member changes:
>                               'uint8_t inner_l4_type' offset changed from 0 to 24 (in bits) (by +24 bits)
>                               'uint8_t l4_type' offset changed from 0 to 8 (in bits) (by +8 bits)
>                               'uint8_t tun_type' offset changed from 4 to 12 (in bits) (by +8 bits)

For this one, this is probably due to
https://sourceware.org/bugzilla/show_bug.cgi?id=26684

I had a discussion with Dodji (libabigail maintainer).

Going from dwarf 4 to dwarf 5 is something that is decided at gcc /
binutils level.

Arch Linux recently adopted gcc 11.
https://archlinux.org/packages/core/x86_64/gcc/

If we compare gcc 10 and gcc 11:
- https://gcc.gnu.org/onlinedocs/gcc-10.1.0/gcc/Debugging-Options.html
"Produce debugging information in DWARF format (if that is supported).
The value of version may be either 2, 3, 4 or 5; the default version
for most targets is 4. DWARF Version 5 is only experimental. "
- https://gcc.gnu.org/onlinedocs/gcc-11.1.0/gcc/Debugging-Options.html
"Produce debugging information in DWARF format (if that is supported).
The value of version may be either 2, 3, 4 or 5; the default version
for most targets is 5 (with the exception of VxWorks, TPF and
Darwin/Mac OS X, which default to version 2, and AIX, which defaults
to version 4)."

So the reason, for the issue reported above, could be that the
reference had been generated before upgrading gcc to 11.


I have some trouble finding the actual date for the Arch Linux switch
to gcc 11 (maybe 2021/05/17).
Are you able to correlate this with the ABI reference previous generation?


-- 
David Marchand

Re: [dpdk-ci] [dpdklab] ABI test failing for openSUSE and Arch Linux

[Brandon Lo]

Hi David,

For the Arch container, the gcc version did update to 11.1 which
contributed to this error.
The OpenSUSE container also underwent a similar update process where
various packages related to the toolchain were updated. However, it
appears that OpenSUSE remained on gcc version 7.5.0.
This is, in part, also caused by the fact that we generate the ABI
references once and store them. This is simply to reduce the amount of
time per ABI test.

To streamline this entire process, we are working on a job or pipeline
to automate refreshing all of the images and recreate the ABI
references.

Thanks,
Brandon

On Mon, Jun 7, 2021 at 4:34 AM David Marchand <david.marchand@redhat.com> wrote:
>
> Trying to do a post mortem.
>
>
> On Fri, Jun 4, 2021 at 3:53 PM Lincoln Lavoie <lylavoie@iol.unh.edu> wrote:
> > The ABI references for all systems were updated this week to the 21.05 release code.  The two failures look like places where the interfaces didn't actually change. We do see the failures across multiple patches, which might imply something got merged that caused these changes / failures.
> >
> > ---------------------------------------------------------------------------------------------
> > OpenSUSE
> > 2 Removed function symbols not referenced by debug info:
> >
> >   [D] _fini
> >   [D] _init
>
> This one, I am not sure what was wrong.
> This is not a symbol from DPDK itself.
> This comes from a libc / toolchain and/or linker change.
>
>
> >
> > ---------------------------------------------------------------------------------------------
> > Arch Linux (one example from the output)
> >
> > Functions changes summary: 0 Removed, 0 Changed, 0 Added function
> > Variables changes summary: 0 Removed, 1 Changed (13 filtered out), 0 Added variables
> >
> > 1 Changed variable:
> >
> >   [C] 'rte_table_ops rte_table_acl_ops' was changed at rte_table_acl.h:60:1:
> >     type of variable changed:
> >       type size hasn't changed
> >       1 data member change:
> >         type of 'rte_table_op_lookup f_lookup' changed:
> >           underlying type 'int (void*, rte_mbuf**, typedef uint64_t, uint64_t*, void**)*' changed:
> >             in pointed to type 'function type int (void*, rte_mbuf**, typedef uint64_t, uint64_t*, void**)':
> >               parameter 2 of type 'rte_mbuf**' has sub-type changes:
> >                 in pointed to type 'rte_mbuf*':
> >                   in pointed to type 'struct rte_mbuf' at rte_mbuf_core.h:484:1:
> >                     type size hasn't changed
> >                     1 data member changes (1 filtered):
> >                       type of 'anonymous data member union {uint32_t packet_type; struct {uint8_t l2_type; uint8_t l3_type; uint8_t l4_type; uint8_t tun_type; union {uint8_t inner_esp_next_proto; struct {uint8_t inner_l2_type; uint8_t inner_l3_type;};}; uint8_t inner_l4_type;};}' changed:
> >                         type size hasn't changed
> >                         1 data member change:
> >                           type of 'anonymous data member struct {uint8_t l2_type; uint8_t l3_type; uint8_t l4_type; uint8_t tun_type; union {uint8_t inner_esp_next_proto; struct {uint8_t inner_l2_type; uint8_t inner_l3_type;};}; uint8_t inner_l4_type;}' changed:
> >                             type size hasn't changed
> >                             3 data member changes:
> >                               'uint8_t inner_l4_type' offset changed from 0 to 24 (in bits) (by +24 bits)
> >                               'uint8_t l4_type' offset changed from 0 to 8 (in bits) (by +8 bits)
> >                               'uint8_t tun_type' offset changed from 4 to 12 (in bits) (by +8 bits)
>
> For this one, this is probably due to
> https://sourceware.org/bugzilla/show_bug.cgi?id=26684
>
> I had a discussion with Dodji (libabigail maintainer).
>
> Going from dwarf 4 to dwarf 5 is something that is decided at gcc /
> binutils level.
>
> Arch Linux recently adopted gcc 11.
> https://archlinux.org/packages/core/x86_64/gcc/
>
> If we compare gcc 10 and gcc 11:
> - https://gcc.gnu.org/onlinedocs/gcc-10.1.0/gcc/Debugging-Options.html
> "Produce debugging information in DWARF format (if that is supported).
> The value of version may be either 2, 3, 4 or 5; the default version
> for most targets is 4. DWARF Version 5 is only experimental. "
> - https://gcc.gnu.org/onlinedocs/gcc-11.1.0/gcc/Debugging-Options.html
> "Produce debugging information in DWARF format (if that is supported).
> The value of version may be either 2, 3, 4 or 5; the default version
> for most targets is 5 (with the exception of VxWorks, TPF and
> Darwin/Mac OS X, which default to version 2, and AIX, which defaults
> to version 4)."
>
> So the reason, for the issue reported above, could be that the
> reference had been generated before upgrading gcc to 11.
>
>
> I have some trouble finding the actual date for the Arch Linux switch
> to gcc 11 (maybe 2021/05/17).
> Are you able to correlate this with the ABI reference previous generation?
>
>
> --
> David Marchand
>


-- 

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu

Re: [dpdk-ci] [dpdklab] ABI test failing for openSUSE and Arch Linux

[David Marchand]

On Wed, Jun 9, 2021 at 5:07 PM Brandon Lo <blo@iol.unh.edu> wrote:
> To streamline this entire process, we are working on a job or pipeline
> to automate refreshing all of the images and recreate the ABI
> references.

I understand the motivation, but will we have a clear idea of which
ABI reference has been used and how to reproduce its generation (sha1,
toolchain, libc, libabigail and such packages versions, version of the
script generating the reference) ?
If something breaks later and we don't know clearly how/if a reference
changed, it will be a pain to analyse.


-- 
David Marchand

Re: [dpdk-ci] [dpdklab] ABI test failing for openSUSE and Arch Linux

[Lincoln Lavoie]

[-- Attachment #1: Type: text/plain, Size: 1583 bytes --]

Hi David,

I think yes.  What Brandon was referring to is linking the process we use
to refresh the container images and the rebuild of the ABI references, so
one triggers the other.  What happened with the failure was the
container images got rebuilt, and that pulled in updates that change the
ABI output (in valid ways), which then "look like" a failure or change from
the reference that was previously saved off.

We save off older versions of the container images (i.e. things are
tagged), so we can always roll back if need to.  ABI reference generation
should be deterministic on that container image, so we don't save
"versions" of those references.

Cheers,
Lincoln

On Thu, Jun 10, 2021 at 4:02 AM David Marchand <david.marchand@redhat.com>
wrote:

> On Wed, Jun 9, 2021 at 5:07 PM Brandon Lo <blo@iol.unh.edu> wrote:
> > To streamline this entire process, we are working on a job or pipeline
> > to automate refreshing all of the images and recreate the ABI
> > references.
>
> I understand the motivation, but will we have a clear idea of which
> ABI reference has been used and how to reproduce its generation (sha1,
> toolchain, libc, libabigail and such packages versions, version of the
> script generating the reference) ?
> If something breaks later and we don't know clearly how/if a reference
> changed, it will be a pain to analyse.
>
>
> --
> David Marchand
>
>

-- 
*Lincoln Lavoie*
Principal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
<https://www.iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 3106 bytes --]