Re: [dpdk-stable] [dpdk-dev] [PATCH 20.11] config/arm: replace native machine args

[Jerin Jacob <jerinjacobk@gmail.com>]

On Mon, Mar 8, 2021 at 5:38 PM Luca Boccassi <bluca@debian.org> wrote:
>
> On Mon, 2021-03-08 at 03:23 +0000, Ruifeng Wang wrote:
> > > -----Original Message-----
> > > From: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> > > Sent: Sunday, March 7, 2021 9:35 PM
> > > To: Ruifeng Wang <Ruifeng.Wang@arm.com>; Juraj Linkeš
> > > <juraj.linkes@pantheon.tech>; Luca Boccassi <bluca@debian.org>;
> > > stable@dpdk.org; dev@dpdk.org; thomas@monjalon.net; Ashwin Sekhar
> > > Thalakalath Kottilveetil <asekhar@marvell.com>; Andrew Pinski
> > > <apinski@marvell.com>
> > > Cc: david.marchand@redhat.com; nd <nd@arm.com>; nd <nd@arm.com>
> > > Subject: RE: [PATCH 20.11] config/arm: replace native machine args
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Ruifeng Wang <Ruifeng.Wang@arm.com>
> > > > Sent: Monday, March 1, 2021 11:10 AM
> > > > To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Juraj Linkeš
> > > > <juraj.linkes@pantheon.tech>; Luca Boccassi <bluca@debian.org>;
> > > > stable@dpdk.org
> > > > Cc: david.marchand@redhat.com; nd <nd@arm.com>; nd <nd@arm.com>
> > > > Subject: [EXT] RE: [PATCH 20.11] config/arm: replace native machine
> > > > args
> > > >
> > > > External Email
> > > >
> > > > ----------------------------------------------------------------------
> > > > > -----Original Message-----
> > > > > From: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> > > > > Sent: Thursday, February 25, 2021 8:15 PM
> > > > > To: Ruifeng Wang <Ruifeng.Wang@arm.com>; Juraj Linkeš
> > > > > <juraj.linkes@pantheon.tech>; Luca Boccassi <bluca@debian.org>;
> > > > > stable@dpdk.org
> > > > > Cc: david.marchand@redhat.com; nd <nd@arm.com>
> > > > > Subject: RE: [PATCH 20.11] config/arm: replace native machine args
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Ruifeng Wang <Ruifeng.Wang@arm.com>
> > > > > > Sent: Saturday, February 20, 2021 9:13 AM
> > > > > > To: Juraj Linkeš <juraj.linkes@pantheon.tech>; Luca Boccassi
> > > > > > <bluca@debian.org>; stable@dpdk.org; Jerin Jacob Kollanukkaran
> > > > > > <jerinj@marvell.com>
> > > > > > Cc: david.marchand@redhat.com; nd <nd@arm.com>
> > > > > > Subject: [EXT] RE: [PATCH 20.11] config/arm: replace native
> > > > > > machine args
> > > > > >
> > > > > > External Email
> > > > > >
> > > > > > ------------------------------------------------------------------
> > > > > > --
> > > > > > --
> > > > > > > -----Original Message-----
> > > > > > > From: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > > > > > Sent: Friday, February 19, 2021 8:10 PM
> > > > > > > To: Luca Boccassi <bluca@debian.org>; stable@dpdk.org
> > > > > > > Cc: jerinj@marvell.com; Ruifeng Wang <Ruifeng.Wang@arm.com>;
> > > > > > > david.marchand@redhat.com
> > > > > > > Subject: RE: [PATCH 20.11] config/arm: replace native machine
> > > > > > > args
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Luca Boccassi <bluca@debian.org>
> > > > > > > > Sent: Friday, February 19, 2021 12:33 PM
> > > > > > > > To: Juraj Linkeš <juraj.linkes@pantheon.tech>; stable@dpdk.org
> > > > > > > > Cc: jerinj@marvell.com; ruifeng.wang@arm.com;
> > > > > > > > david.marchand@redhat.com
> > > > > > > > Subject: Re: [PATCH 20.11] config/arm: replace native machine
> > > > > > > > args
> > > > > > > >
> > > > > > > > On Fri, 2021-02-19 at 11:06 +0000, Juraj Linkeš wrote:
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: luca.boccassi@gmail.com <luca.boccassi@gmail.com>
> > > > > > > > > > Sent: Friday, February 19, 2021 11:58 AM
> > > > > > > > > > To: stable@dpdk.org
> > > > > > > > > > Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>;
> > > > > > > > > > jerinj@marvell.com; ruifeng.wang@arm.com;
> > > > > > > > > > david.marchand@redhat.com
> > > > > > > > > > Subject: [PATCH 20.11] config/arm: replace native machine
> > > > > > > > > > args
> > > > > > > > > >
> > > > > > > > > > From: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > > > > > > > >
> > > > > > > > > > [ backported from upstream commit
> > > > > > > > > > 9186e5a07f35ae74a1f7fa2d89671b5f77eae407 ]
> > > > > > > > > >
> > > > > > > > > > There are compiler issues when building with -mcpu=native
> > > > > > > > > > with popular compilers, such as GCC-8.4:
> > > > > > > > > > In file included from ../lib/librte_eal/arm/include/rte_vect.h:11,
> > > > > > > > > >                  from ../lib/librte_net/net_crc_neon.c:10:
> > > > > > > > > > ../lib/librte_net/net_crc_neon.c: In function
> > > ‘crcr32_folding_round’:
> > > > > > > > > > /usr/lib/gcc/aarch64-linux-gnu/8/include/arm_neon.h:26094:1:
> > > > > error:
> > > > > > > > > > inlining failed in call to always_inline ‘vmull_p64’:
> > > > > > > > > > target specific option mismatch
> > > > > > > > > >  vmull_p64 (poly64_t a, poly64_t b)
> > > > > > > > > > ../lib/librte_net/net_crc_neon.c:50:20: note: called from here
> > > > > > > > > >   uint64x2_t tmp1 = vreinterpretq_u64_p128(vmull_p64(
> > > > > > > > > >     vgetq_lane_p64(vreinterpretq_p64_u64(fold), 0),
> > > > > > > > > >     vgetq_lane_p64(vreinterpretq_p64_u64(precomp), 1)));
> > > > > > > > > >
> > > > > > > > > > and clang:
> > > > > > > > > > gcc -E -dM -mcpu="native" - < /dev/null | grep
> > > > > > > > > > __ARM_FEATURE_ATOMICS
> > > > > > > > > > clang-9 -E -dM -mcpu="native" - < /dev/null | grep
> > > > > > > > > > __ARM_FEATURE_ATOMICS <no output> # no clang support
> > > > > > > > > >
> > > > > > > > > > Fix this by always specifying the proper machine args and
> > > > > > > > > > never using the native flags.
> > > > > > > > > >
> > > > > > > > > > Fixes: 78ac8eac7e8a ("config/arm: use native machine build
> > > > > > > > > > arguments")
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > > > > > > > > Signed-off-by: Luca Boccassi <luca.boccassi@microsoft.com>
> > > > > > > > > > ---
> > > > > > > > > > This is a crude backport, but it fixes the build for arm64.
> > > > > > > > > > It's a release blocker for 20.11.1, so I would appreciate
> > > > > > > > > > a quick
> > > > > review.
> > > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > > > What does this fix? With or without the below change, the
> > > > > > > > > native machine
> > > > > > > > args are not used. The patch shoudn't actually change the
> > > > > > > > configuration of the build at all, so I'm a bit confused.
> > > > > > > >
> > > > > > > > It fixes the build on some build workers with thunderx
> > > > > > > > hardware
> > > > > > > > - without this I get failures like:
> > > > > > > >
> > > > > > > > arm_neon.h:26647:1: error: inlining failed in call to 'always_inline'
> > > > > > > > 'vmull_p64': target specific option mismatch
> > > > > > > >
> > > > > > >
> > > > > > > I tried the patch and I'm seeing the same errors on a ThunderX
> > > > > > > server (with and without the patch). Is this actually the right patch?
> > > > > > >
> > > > > > > One of the four failures looks like this:
> > > > > > > In file included from ../lib/librte_eal/arm/include/rte_vect.h:11,
> > > > > > >                  from ../lib/librte_net/net_crc_neon.c:10:
> > > > > > > ../lib/librte_net/net_crc_neon.c: In function 'crcr32_folding_round':
> > > > > > > /usr/lib/gcc/aarch64-linux-gnu/8/include/arm_neon.h:26094:1: error:
> > > > > > > inlining failed in call to always_inline 'vmull_p64': target
> > > > > > > specific option mismatch
> > > > > > >  vmull_p64 (poly64_t a, poly64_t b)  ^~~~~~~~~
> > > > > > > ../lib/librte_net/net_crc_neon.c:50:20: note: called from here
> > > > > > >   uint64x2_t tmp1 = vreinterpretq_u64_p128(vmull_p64(
> > > > > > >                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > > > > >     vgetq_lane_p64(vreinterpretq_p64_u64(fold), 0),
> > > > > > >     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > > > > >     vgetq_lane_p64(vreinterpretq_p64_u64(precomp), 1)));
> > > > > > >     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > > > > >
> > > > > > > Ruifeng, any ideas on how to fix this?
> > > > > >
> > > > > > Gcc build on ThunderX platform is broken. Issue can be seen in
> > > > > > some
> > > > > > CentOS-8 OBS builds.
> > > > > > https://urldefense.proofpoint.com/v2/url?u=https-
> > > > > > 3A__mails.dpdk.org_archives_dev_2020-
> > > > > >
> > > 2DNovember_192909.html&d=DwIGaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=1
> > > > > DG
> > > ob4H4rxz6H8uITozGOCa0s5f4wCNtTa4UUKvcsvI&m=mgzJ6z43dsDFwI6rdgKC
> > > > > Uj
> > > > > > 0GCMNjEKQAa7dfRZxvrdU&s=UWUJTFdGC2mD2x-rcuRnH1I7-
> > > > > > 1jKFC40Bh5hFanzu0A&e=
> > > > > > I tried tuning compiler flags used, but could not resolve the issue.
> > > > > >
> > > > > > Need help from Marvell to look at this.
> > > > > > Hi Jerin, do you have any thoughts on this?
> > > > >
> > > > > Ruifeng, If you are able to reproduce this issue, Could you add "-
> > > > > march=armv8.1-a+crc+crypto" In ThunderX config  and check is this
> > > > > Fixing the issue?
> > > > >
> > > > > [main] [dpdk.org] $ git diff
> > > > > diff --git a/config/arm/meson.build b/config/arm/meson.build index
> > > > > 00bc4610a..ef65b4bb6 100644
> > > > > --- a/config/arm/meson.build
> > > > > +++ b/config/arm/meson.build
> > > > > @@ -96,15 +96,18 @@ implementer_cavium = {
> > > > >         ],
> > > > >         'part_number_config': {
> > > > >                 '0xa1': {
> > > > > -                       'machine_args': ['-mcpu=thunderxt88'],
> > > > > +                       'machine_args': ['-march=armv8.1-a+crc+crypto+lse',
> > > > > +                                        '-mcpu=thunderxt88'],
> > > > >                         'flags': flags_part_number_thunderx
> > > > >                 },
> > > > >                 '0xa2': {
> > > > > -                       'machine_args': ['-mcpu=thunderxt81'],
> > > > > +                       'machine_args': ['-march=armv8.1-a+crc+crypto+lse',
> > > > > +                                        '-mcpu=thunderxt81'],
> > > > >                         'flags': flags_part_number_thunderx
> > > > >                 },
> > > > >                 '0xa3': {
> > > > > -                       'machine_args': ['-mcpu=thunderxt83'],
> > > > > +                       'machine_args': ['-march=armv8.1-a+crc+crypto+lse',
> > > > > +                                        '-mcpu=thunderxt83'],
> > > > >                         'flags': flags_part_number_thunderx
> > > > >                 },
> > > > >                 '0xaf': {
> > > > >
> > > >
> > > > Hi Jerin,
> > > >
> > > > The patch doesn't work. Build failed at link stage.
> > > > I used gcc 8.4 and tried build on thunderxt88.
> > >
> > >
> > > Hi Ruifeng,
> > >
> > > I talked to compiler experts here in Marvell. It looks like compiler issue, As a
> > > workaround couple of these could try:
> > > 1) Reduce the external libraries linked to the application like mlx5 etc
> >
> > I tried building with lots of drivers disabled. Not yet able to get a successful build.
> >
> > > 2) Add -mcmodel=large flag will fix "relocation truncated to fit" issue as
> > > testing purpose as we are not sure about the implication of this flag.
> >
> > Looks like this flag is not supported by gcc 8.4 that I am using.
> >
> > One thing we can do to overcome the build failure is to switch to default / release build in OBS CI.
> > OBS CI is running native build, so it could hit this issue when CI job is scheduled to thunderxt88 infrastructure.
> > I think we should change to do release build (-Dmachine=default) which is more suitable for generic CI verification.
> > As I checked, release build can pass on my thunderxt88 platform.
> >
> > What do you think?
>
> Hi,
>
> I've done as suggested, and it seems to fare better, thank you.
>
> There is still a build error on CentOS 7, not sure if it is related


That ICE is a known issue from the compiler. Please use gcc 4.8 at
least for arm64 work.
Similar issue[1] present in some 5.2.1, 6.0 version too.

[1]
 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=67143


> though:
>
> [  555s] ../drivers/event/octeontx2/otx2_tim_worker.c: In function 'otx2_tim_arm_tmo_tick_burst_mod':
> [  555s] ../drivers/event/octeontx2/otx2_tim_worker.c:154:18: error: could not split insn
> [  555s]            struct rte_event_timer **tim, \
> [  555s]                   ^
> [  555s] ../drivers/event/octeontx2/otx2_tim_evdev.h:208:1: note: in expansion of macro 'FP'
> [  555s]  FP(mod,   0, 0, 0, OTX2_TIM_BKT_MOD | OTX2_TIM_ENA_DFB)  \
> [  555s]  ^
> [  555s] ../drivers/event/octeontx2/otx2_tim_worker.c:161:1: note: in expansion of macro 'TIM_ARM_TMO_FASTPATH_MODES'
> [  555s]  TIM_ARM_TMO_FASTPATH_MODES
> [  555s]  ^
> [  555s] (insn 252 250 255 (parallel [
> [  555s]             (set (reg:DI 1 x1 [orig:230 D.17092 ] [230])
> [  555s]                 (mem/v:DI (reg/f:DI 10 x10 [orig:229 D.17094 ] [229]) [-1  S8 A64]))
> [  555s]             (set (mem/v:DI (reg/f:DI 10 x10 [orig:229 D.17094 ] [229]) [-1  S8 A64])
> [  555s]                 (unspec_volatile:DI [
> [  555s]                         (plus:DI (mem/v:DI (reg/f:DI 10 x10 [orig:229 D.17094 ] [229]) [-1  S8 A64])
> [  555s]                             (const_int 1099511627776 [0x10000000000]))
> [  555s]                         (const_int 2 [0x2])
> [  555s]                     ] UNSPECV_ATOMIC_OP))
> [  555s]             (clobber (reg:CC 66 cc))
> [  555s]             (clobber (reg:DI 4 x4))
> [  555s]             (clobber (reg:SI 3 x3))
> [  555s]         ]) ../drivers/event/octeontx2/otx2_tim_worker.h:81 1832 {atomic_fetch_adddi}
> [  555s]      (expr_list:REG_UNUSED (reg:CC 66 cc)
> [  555s]         (expr_list:REG_UNUSED (reg:DI 4 x4)
> [  555s]             (expr_list:REG_UNUSED (reg:SI 3 x3)
> [  555s]                 (nil)))))
> [  555s] ../drivers/event/octeontx2/otx2_tim_worker.c:154:18: internal compiler error: in final_scan_insn, at final.c:2897
> [  555s]            struct rte_event_timer **tim, \
> [  555s]                   ^
> [  555s] ../drivers/event/octeontx2/otx2_tim_evdev.h:208:1: note: in expansion of macro 'FP'
> [  555s]  FP(mod,   0, 0, 0, OTX2_TIM_BKT_MOD | OTX2_TIM_ENA_DFB)  \
> [  555s]  ^
> [  555s] ../drivers/event/octeontx2/otx2_tim_worker.c:161:1: note: in expansion of macro 'TIM_ARM_TMO_FASTPATH_MODES'
> [  555s]  TIM_ARM_TMO_FASTPATH_MODES
> [  555s]  ^
> [  555s] Please submit a full bug report,
> [  555s] with preprocessed source if appropriate.
> [  555s] See <http://bugzilla.redhat.com/bugzilla> for instructions.
> [  555s] {standard input}: Assembler messages:
> [  555s] {standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive
> [  555s] Preprocessed source stored into /tmp/ccpVQUdT.out file, please attach this to your bugreport.
>
> --
> Kind regards,
> Luca Boccassi