[dpdk-dev] RTM instruction compile failure for XABORT when AVX is active

[dpdk-dev] RTM instruction compile failure for XABORT when AVX is active

[Matthew Hall]

Hi all,

I am getting a strange error compiling some RTM instructions when I upgraded my VM environment from VirtualBox 4 to VirtualBox 5 and AVX instructions from the host CPU became available. However when I am reading the opcode description for XABORT it supposedly allows an immediate operand for the argument so I can't understand why this code would not compile. Any advice?

  CC test_hash_scaling.o
In file included from /vagrant/external/dpdk/app/test/test_hash_scaling.c:37:
In file included from /vagrant/external/dpdk/build/include/rte_spinlock.h:42:
/vagrant/external/dpdk/build/include/rte_rtm.h:56:15: error: invalid operand for inline asm constraint 'i'
        asm volatile(".byte 0xc6,0xf8,%P0" :: "i" (status) : "memory");

Matthew.

Re: [dpdk-dev] RTM instruction compile failure for XABORT when AVX is active

[Matthew Hall]

It would appear there is some bug in the new lock elision patches that is preventing it from compiling with clang. Any suggestions?

On Jun 28, 2015, at 10:16 AM, Matthew Hall <mhall@mhcomputing.net> wrote:

> Hi all,
> 
> I am getting a strange error compiling some RTM instructions when I upgraded my VM environment from VirtualBox 4 to VirtualBox 5 and AVX instructions from the host CPU became available. However when I am reading the opcode description for XABORT it supposedly allows an immediate operand for the argument so I can't understand why this code would not compile. Any advice?
> 
>  CC test_hash_scaling.o
> In file included from /vagrant/external/dpdk/app/test/test_hash_scaling.c:37:
> In file included from /vagrant/external/dpdk/build/include/rte_spinlock.h:42:
> /vagrant/external/dpdk/build/include/rte_rtm.h:56:15: error: invalid operand for inline asm constraint 'i'
>        asm volatile(".byte 0xc6,0xf8,%P0" :: "i" (status) : "memory");
> 
> Matthew.

Re: [dpdk-dev] RTM instruction compile failure for XABORT when AVX is active

[Thomas Monjalon]

2015-06-28 10:59, Matthew Hall:
> It would appear there is some bug in the new lock elision patches that is preventing it from compiling with clang. Any suggestions?

It builds with clang.
My suggestion is to add the patch author in recipient's list ;)


> On Jun 28, 2015, at 10:16 AM, Matthew Hall <mhall@mhcomputing.net> wrote:
> 
> > Hi all,
> > 
> > I am getting a strange error compiling some RTM instructions when I upgraded my VM environment from VirtualBox 4 to VirtualBox 5 and AVX instructions from the host CPU became available. However when I am reading the opcode description for XABORT it supposedly allows an immediate operand for the argument so I can't understand why this code would not compile. Any advice?
> > 
> >  CC test_hash_scaling.o
> > In file included from /vagrant/external/dpdk/app/test/test_hash_scaling.c:37:
> > In file included from /vagrant/external/dpdk/build/include/rte_spinlock.h:42:
> > /vagrant/external/dpdk/build/include/rte_rtm.h:56:15: error: invalid operand for inline asm constraint 'i'
> >        asm volatile(".byte 0xc6,0xf8,%P0" :: "i" (status) : "memory");
> > 
> > Matthew.
> 

Re: [dpdk-dev] RTM instruction compile failure for XABORT when AVX is active

[Roman Dementiev]

Hello Thomas,

Sunday, June 28, 2015, 9:38:14 PM, you wrote:

> 2015-06-28 10:59, Matthew Hall:
>> It would appear there is some bug in the new lock elision patches that is preventing it from compiling with clang. Any suggestions?

> It builds with clang.
> My suggestion is to add the patch author in recipient's list ;)


>> On Jun 28, 2015, at 10:16 AM, Matthew Hall <mhall@mhcomputing.net> wrote:
>> 
>> > Hi all,
>> > 
>> > I am getting a strange error compiling some RTM instructions when I upgraded my VM environment from VirtualBox 4 to VirtualBox 5 and AVX instructions from the host CPU became available. However when I am reading the opcode description for XABORT it supposedly allows an immediate operand for the argument so I can't understand why this code would not compile. Any advice?
>> > 
>> >  CC test_hash_scaling.o
>> > In file included from /vagrant/external/dpdk/app/test/test_hash_scaling.c:37:
>> > In file included from /vagrant/external/dpdk/build/include/rte_spinlock.h:42:
>> > /vagrant/external/dpdk/build/include/rte_rtm.h:56:15: error: invalid operand for inline asm constraint 'i'
>> >        asm volatile(".byte 0xc6,0xf8,%P0" :: "i" (status) : "memory");
>> > 
>> > Matthew.
>> 

It  looks  like  a  compiler  bug for me. In the meantime Clang should
have native TSX intrinsics (but    most   compilers   fail   to   implement
them correctly  on  a first attempt allowing instruction reordering). Could
you try this workaround:

diff --git a/lib/librte_eal/common/include/arch/x86/rte_rtm.h b/lib/librte_eal/common/include/arch/x86/rte_rtm.h
index d935641..ee73dd4 100644
--- a/lib/librte_eal/common/include/arch/x86/rte_rtm.h
+++ b/lib/librte_eal/common/include/arch/x86/rte_rtm.h
@@ -50,10 +50,14 @@ void rte_xend(void)
         asm volatile(".byte 0x0f,0x01,0xd5" ::: "memory");
 }

+#include <immintrin.h>
+
 static __attribute__((__always_inline__)) inline
 void rte_xabort(const unsigned int status)
 {
-       asm volatile(".byte 0xc6,0xf8,%P0" :: "i" (status) : "memory");
+       asm volatile(::: "memory");
+       _xabort(status);
+       asm volatile(::: "memory");
 }

 static __attribute__((__always_inline__)) inline


-- 
Best regards,
 Roman                            mailto:roman.dementiev@intel.com

Intel GmbH
Dornacher Strasse 1
85622 Feldkirchen/Muenchen, Deutschland
Sitz der Gesellschaft: Feldkirchen bei Muenchen
Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas Lusk
Registergericht: Muenchen HRB 47456
Ust.-IdNr./VAT Registration No.: DE129385895
Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052

Re: [dpdk-dev] RTM instruction compile failure for XABORT when AVX is active

[Thomas Monjalon]

2015-06-29 12:11, Roman Dementiev:
> >> On Jun 28, 2015, at 10:16 AM, Matthew Hall <mhall@mhcomputing.net> wrote:
> >> > I am getting a strange error compiling some RTM instructions when
> >> > I upgraded my VM environment from VirtualBox 4 to VirtualBox 5 and
> >> > AVX instructions from the host CPU became available. However when
> >> > I am reading the opcode description for XABORT it supposedly allows
> >> > an immediate operand for the argument so I can't understand why this
> >> > code would not compile. Any advice?
> >> > 
> >> >  CC test_hash_scaling.o
> >> > In file included from /vagrant/external/dpdk/app/test/test_hash_scaling.c:37:
> >> > In file included from /vagrant/external/dpdk/build/include/rte_spinlock.h:42:
> >> > /vagrant/external/dpdk/build/include/rte_rtm.h:56:15: error: invalid operand for inline asm constraint 'i'
> >> >        asm volatile(".byte 0xc6,0xf8,%P0" :: "i" (status) : "memory");

There is no such bug with my compiler:
	clang version 3.6.1 (tags/RELEASE_361/final)
	Target: x86_64-unknown-linux-gnu

Matthew, which version are you using?

> It  looks  like  a  compiler  bug for me. In the meantime Clang should
> have native TSX intrinsics (but    most   compilers   fail   to   implement
> them correctly  on  a first attempt allowing instruction reordering). Could
> you try this workaround:
> 
> diff --git a/lib/librte_eal/common/include/arch/x86/rte_rtm.h b/lib/librte_eal/common/include/arch/x86/rte_rtm.h
> index d935641..ee73dd4 100644
> --- a/lib/librte_eal/common/include/arch/x86/rte_rtm.h
> +++ b/lib/librte_eal/common/include/arch/x86/rte_rtm.h
> @@ -50,10 +50,14 @@ void rte_xend(void)
>          asm volatile(".byte 0x0f,0x01,0xd5" ::: "memory");
>  }
> 
> +#include <immintrin.h>
> +
>  static __attribute__((__always_inline__)) inline
>  void rte_xabort(const unsigned int status)
>  {
> -       asm volatile(".byte 0xc6,0xf8,%P0" :: "i" (status) : "memory");
> +       asm volatile(::: "memory");
> +       _xabort(status);
> +       asm volatile(::: "memory");
>  }
> 
>  static __attribute__((__always_inline__)) inline

Re: [dpdk-dev] RTM instruction compile failure for XABORT when AVX is active

[Matthew Hall]

On Jun 29, 2015, at 3:19 AM, Thomas Monjalon <thomas.monjalon@6wind.com> wrote:
> There is no such bug with my compiler:
> 	clang version 3.6.1 (tags/RELEASE_361/final)
> 	Target: x86_64-unknown-linux-gnu
> 
> Matthew, which version are you using?

Hi Thomas and Roman,

It seems to happen if I have set -mavx in CFLAGS with clang 1:3.4-1ubuntu3.

I get a different issue that only shows up at runtime in clang 3.6.2-svn240577-1~exp1:

ERROR: This system does not support "FSGSBASE".
Please check that RTE_MACHINE is set correctly.

It appears I probably need to learn how to do a better job on my EXTRA_CFLAGS. Do we have some recommendations what should be used on the different Intel CPUs to avoid build issues but still get the best performance? This would help a lot.

Matthew.

Re: [dpdk-dev] RTM instruction compile failure for XABORT when AVX is active

[Matthew Hall]

To be a bit more specific, this is what I had to do to fix it for clang 3.6 SVN snapshot release.

I am not sure if there is a better way of handling this situation. I'd love to know where I could improve it.

Matthew.

diff --git a/mk/rte.cpuflags.mk b/mk/rte.cpuflags.mk
index f595cd0..8c883ee 100644
--- a/mk/rte.cpuflags.mk
+++ b/mk/rte.cpuflags.mk
@@ -77,13 +77,13 @@ ifneq ($(filter $(AUTO_CPUFLAGS),__RDRND__),)
 CPUFLAGS += RDRAND
 endif

-ifneq ($(filter $(AUTO_CPUFLAGS),__FSGSBASE__),)
-CPUFLAGS += FSGSBASE
-endif
+#ifneq ($(filter $(AUTO_CPUFLAGS),__FSGSBASE__),)
+#CPUFLAGS += FSGSBASE
+#endif

-ifneq ($(filter $(AUTO_CPUFLAGS),__F16C__),)
-CPUFLAGS += F16C
-endif
+#ifneq ($(filter $(AUTO_CPUFLAGS),__F16C__),)
+#CPUFLAGS += F16C
+#endif

 ifneq ($(filter $(AUTO_CPUFLAGS),__AVX2__),)
 CPUFLAGS += AVX2

Re: [dpdk-dev] RTM instruction compile failure for XABORT when AVX is active

[Matthew Hall]

With those two items commented out, and these CFLAGS:

"-g -O0 -fPIC -msse4.2"

it looks like I can reproduce the issue in clang 2.6 series:

/vagrant/external/dpdk/build/include/rte_rtm.h:56:15: error: invalid operand for inline asm constraint 'i'
        asm volatile(".byte 0xc6,0xf8,%P0" :: "i" (status) : "memory");

So there are definitely some corner cases that seem to be able to trigger it.

On Jun 30, 2015, at 10:17 PM, Matthew Hall <mhall@mhcomputing.net> wrote:

> To be a bit more specific, this is what I had to do to fix it for clang 3.6 SVN snapshot release.
> 
> I am not sure if there is a better way of handling this situation. I'd love to know where I could improve it.
> 
> Matthew.
> 
> diff --git a/mk/rte.cpuflags.mk b/mk/rte.cpuflags.mk
> index f595cd0..8c883ee 100644
> --- a/mk/rte.cpuflags.mk
> +++ b/mk/rte.cpuflags.mk
> @@ -77,13 +77,13 @@ ifneq ($(filter $(AUTO_CPUFLAGS),__RDRND__),)
> CPUFLAGS += RDRAND
> endif
> 
> -ifneq ($(filter $(AUTO_CPUFLAGS),__FSGSBASE__),)
> -CPUFLAGS += FSGSBASE
> -endif
> +#ifneq ($(filter $(AUTO_CPUFLAGS),__FSGSBASE__),)
> +#CPUFLAGS += FSGSBASE
> +#endif
> 
> -ifneq ($(filter $(AUTO_CPUFLAGS),__F16C__),)
> -CPUFLAGS += F16C
> -endif
> +#ifneq ($(filter $(AUTO_CPUFLAGS),__F16C__),)
> +#CPUFLAGS += F16C
> +#endif
> 
> ifneq ($(filter $(AUTO_CPUFLAGS),__AVX2__),)
> CPUFLAGS += AVX2

Re: [dpdk-dev] RTM instruction compile failure for XABORT when AVX is active

[Bruce Richardson]

On Tue, Jun 30, 2015 at 10:49:26PM -0700, Matthew Hall wrote:
> With those two items commented out, and these CFLAGS:
> 
> "-g -O0 -fPIC -msse4.2"
> 

The recommended way of specifying a particular instruction set is via the
RTE_MACHINE setting in your build time config. Can you perhaps reproduce the
issue using a setting there?

/Bruce

> it looks like I can reproduce the issue in clang 2.6 series:
> 
> /vagrant/external/dpdk/build/include/rte_rtm.h:56:15: error: invalid operand for inline asm constraint 'i'
>         asm volatile(".byte 0xc6,0xf8,%P0" :: "i" (status) : "memory");
> 
> So there are definitely some corner cases that seem to be able to trigger it.
> 
> On Jun 30, 2015, at 10:17 PM, Matthew Hall <mhall@mhcomputing.net> wrote:
> 
> > To be a bit more specific, this is what I had to do to fix it for clang 3.6 SVN snapshot release.
> > 
> > I am not sure if there is a better way of handling this situation. I'd love to know where I could improve it.
> > 
> > Matthew.
> > 
> > diff --git a/mk/rte.cpuflags.mk b/mk/rte.cpuflags.mk
> > index f595cd0..8c883ee 100644
> > --- a/mk/rte.cpuflags.mk
> > +++ b/mk/rte.cpuflags.mk
> > @@ -77,13 +77,13 @@ ifneq ($(filter $(AUTO_CPUFLAGS),__RDRND__),)
> > CPUFLAGS += RDRAND
> > endif
> > 
> > -ifneq ($(filter $(AUTO_CPUFLAGS),__FSGSBASE__),)
> > -CPUFLAGS += FSGSBASE
> > -endif
> > +#ifneq ($(filter $(AUTO_CPUFLAGS),__FSGSBASE__),)
> > +#CPUFLAGS += FSGSBASE
> > +#endif
> > 
> > -ifneq ($(filter $(AUTO_CPUFLAGS),__F16C__),)
> > -CPUFLAGS += F16C
> > -endif
> > +#ifneq ($(filter $(AUTO_CPUFLAGS),__F16C__),)
> > +#CPUFLAGS += F16C
> > +#endif
> > 
> > ifneq ($(filter $(AUTO_CPUFLAGS),__AVX2__),)
> > CPUFLAGS += AVX2
> 

Re: [dpdk-dev] RTM instruction compile failure for XABORT when AVX is active

[Matthew Hall]

Previously, with the -msse4.2 flag removed, the build failed for a different reason.

I can retry without it and see if it's the case in the new DPDK.

On Jul 1, 2015, at 4:10 AM, Bruce Richardson <bruce.richardson@intel.com> wrote:

> On Tue, Jun 30, 2015 at 10:49:26PM -0700, Matthew Hall wrote:
>> With those two items commented out, and these CFLAGS:
>> 
>> "-g -O0 -fPIC -msse4.2"
>> 
> 
> The recommended way of specifying a particular instruction set is via the
> RTE_MACHINE setting in your build time config. Can you perhaps reproduce the
> issue using a setting there?
> 
> /Bruce