From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <ferruh.yigit@intel.com>
Received: from mga07.intel.com (mga07.intel.com [134.134.136.100])
 by dpdk.org (Postfix) with ESMTP id 7E0EF28FD;
 Fri,  9 Nov 2018 21:06:23 +0100 (CET)
X-Amp-Result: SKIPPED(no attachment in message)
X-Amp-File-Uploaded: False
Received: from fmsmga002.fm.intel.com ([10.253.24.26])
 by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;
 09 Nov 2018 12:06:21 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.54,484,1534834800"; d="scan'208";a="103086400"
Received: from fyigit-mobl.ger.corp.intel.com (HELO [10.237.221.79])
 ([10.237.221.79])
 by fmsmga002.fm.intel.com with ESMTP; 09 Nov 2018 12:06:18 -0800
To: Thomas Monjalon <thomas@monjalon.net>, dev@dpdk.org
Cc: stable@dpdk.org, Yongseok Koh <yskoh@mellanox.com>,
 keith.wiles@intel.com, bruce.richardson@intel.com,
 Shahaf Shuler <shahafs@mellanox.com>, konstantin.ananyev@intel.com,
 anatoly.burakov@intel.com, justin.parus@microsoft.com,
 "christian.ehrhardt@canonical.com" <christian.ehrhardt@canonical.com>,
 "david.coronel@canonical.com" <david.coronel@canonical.com>,
 "josh.powers@canonical.com" <josh.powers@canonical.com>,
 "jay.vosburgh@canonical.com" <jay.vosburgh@canonical.com>,
 "dan.streetman@canonical.com" <dan.streetman@canonical.com>
References: <20181023212318.43082-1-yskoh@mellanox.com>
 <9891a520-492a-2d18-65e7-c4fcd96a5e3f@intel.com> <2236912.nQJHooqEX5@xps>
 <1672099.GfbVsIzoma@xps>
From: Ferruh Yigit <ferruh.yigit@intel.com>
Openpgp: preference=signencrypt
Autocrypt: addr=ferruh.yigit@intel.com; prefer-encrypt=mutual; keydata=
 xsFNBFXZCFABEADCujshBOAaqPZpwShdkzkyGpJ15lmxiSr3jVMqOtQS/sB3FYLT0/d3+bvy
 qbL9YnlbPyRvZfnP3pXiKwkRoR1RJwEo2BOf6hxdzTmLRtGtwWzI9MwrUPj6n/ldiD58VAGQ
 +iR1I/z9UBUN/ZMksElA2D7Jgg7vZ78iKwNnd+vLBD6I61kVrZ45Vjo3r+pPOByUBXOUlxp9
 GWEKKIrJ4eogqkVNSixN16VYK7xR+5OUkBYUO+sE6etSxCr7BahMPKxH+XPlZZjKrxciaWQb
 +dElz3Ab4Opl+ZT/bK2huX+W+NJBEBVzjTkhjSTjcyRdxvS1gwWRuXqAml/sh+KQjPV1PPHF
 YK5LcqLkle+OKTCa82OvUb7cr+ALxATIZXQkgmn+zFT8UzSS3aiBBohg3BtbTIWy51jNlYdy
 ezUZ4UxKSsFuUTPt+JjHQBvF7WKbmNGS3fCid5Iag4tWOfZoqiCNzxApkVugltxoc6rG2TyX
 CmI2rP0mQ0GOsGXA3+3c1MCdQFzdIn/5tLBZyKy4F54UFo35eOX8/g7OaE+xrgY/4bZjpxC1
 1pd66AAtKb3aNXpHvIfkVV6NYloo52H+FUE5ZDPNCGD0/btFGPWmWRmkPybzColTy7fmPaGz
 cBcEEqHK4T0aY4UJmE7Ylvg255Kz7s6wGZe6IR3N0cKNv++O7QARAQABzSVGZXJydWggWWln
 aXQgPGZlcnJ1aC55aWdpdEBpbnRlbC5jb20+wsGVBBMBAgA/AhsDBgsJCAcDAgYVCAIJCgsE
 FgIDAQIeAQIXgBYhBNI2U4dCLsKE45mBx/kz60PfE2EfBQJbughWBQkHwjOGAAoJEPkz60Pf
 E2Eft84QAIbKWqhgqRfoiw/BbXbA1+qm2o4UgkCRQ0yJgt9QsnbpOmPKydHH0ixCliNz1J8e
 mRXCkMini1bTpnzp7spOjQGLeAFkNFz6BMq8YF2mVWbGEDE9WgnAxZdi0eLY7ZQnHbE6AxKL
 SXmpe9INb6z3ztseFt7mqje/W/6DWYIMnH3Yz9KzxujFWDcq8UCAvPkxVQXLTMpauhFgYeEx
 Nub5HbvhxTfUkapLwRQsSd/HbywzqZ3s/bbYMjj5JO3tgMiM9g9HOjv1G2f1dQjHi5YQiTZl
 1eIIqQ3pTic6ROaiZqNmQFXPsoOOFfXF8nN2zg8kl/sSdoXWHhama5hbwwtl1vdaygQYlmdK
 H2ueiFh/UvT3WG3waNv2eZiEbHV8Rk52Xyn2w1G90lV0fYC6Ket1Xjoch7kjwbx793Kz/RfQ
 rmBY8/S4DTGn3oq3dMdQY+b6+7VMUeLMMh2CXYO9ErkOq+qNTD1IY+cBAkXnaDbQfz0zbste
 ZGWH74FAZ9nCpDOqbRTrBL42aMGhfOWEyeA1x7+hl6JZfabBWAuf4nnCXuorKHzBXTrf7u7p
 fXsKQClWRW77PF1VmzrtKNVSytQAmlCWApQIw20AarFipXmVdIjHmJPU611WoyxZPb4JTOxx
 5cv9B+nr/RIB+v5dcStyHCCwO1be7nBDdCgd4F6kTQPLzsFNBFfWTL4BEACnNA29e8TarUsB
 L5n6eLZHXcFvVwNLVlirWOClHXf44o2KnN3ww+eBEmKVfEFo9MSuGDNHS8Zw1NiGMYxLIUgd
 U6gGrVVs/VrQWL82pbMk6jCj98N+BXIri+6K1z+AImz7ax7iF1kDgRAnFWU0znWWBgM2mM8Y
 gDjcxfXk4sCKnvf6Gjo08Ey5zmqx7dekAKU2EEp8Q1EJY3jbymLdZWRP4AFFMTS1rGMk0/tt
 v71NBg1GobCcbNfn9chK/jhqxYhAJqq86RdJQkt3/9x1U1Oq0vXCt4JVVHmkxePtUiuWTTt+
 aYlUAsKYZsWvncExvw77x2ArYDmaK0yfjh37wp0lY7DOJHFxoyT8tyWZlLci/VMRG2Ja33xj
 0CN4C1yBg+QDeV3QFxQo42iA/ykdXPUR3ezmsND3XKvVLTC4DNb3V/EZQ7jBj64+bEK0VW4G
 B31VP00ApNQvSoczsIOAKdk97RNbpmPw6q10ILIB+9T1xbnFYzshzGF17oC0/GENIHATx8vZ
 masOZoDiOZQpeneLgnFE9JfzhLTxv6wNZcc/HLXRQVTkDsQr8ERtkAoHCf1E5+b5Yr7pfnE4
 YuhET746o25S53ELUYPIs49qoJsEJL34/oexMfPGyPIlrbufiNyty5jc/1MRwUlhJlJ5IOHy
 ZUa+6CLR7GdImusFkPJUJwARAQABwsF8BBgBAgAmAhsMFiEE0jZTh0IuwoTjmYHH+TPrQ98T
 YR8FAlu6CHAFCQXE7zIACgkQ+TPrQ98TYR9nXxAAqNBgkYNyGuWUuy0GwDQCbu3iiMyH1+D7
 llafPcK4NYy1Z4AYuVwC9nmLaoj+ozdqS3ncRo57ncRsKEJC46nDJJZYZ5LSJVn63Y3NBF86
 lxQAgjj2oyZEwaLKtKbAFsXL43jv1pUGgSvWwYtDwHITXXFQto9rZEuUDRFSx4sg9OR+Q6/6
 LY+nQQ3OdHlBkflzYMPcWgDcvcTAO6yasLEUf7UcYoSWTyMYjLB4QuNlXzTswzGVMssJF/vo
 V8lD1eqqaSUWG3STF6GVLQOr1NLvN5+kUBiEStHFxBpgSCvYY9sNV8FS6N24CAWMBl+10W+D
 2h1yiiP5dOdPcBDYKsgqDD91/sP0WdyMJkwdQJtD49f9f+lYloxHnSAxMleOpyscg1pldw+i
 mPaUY1bmIknLhhkqfMmjywQOXpac5LRMibAAYkcB8v7y3kwELnt8mhqqZy6LUsqcWygNbH/W
 K3GGt5tRpeIXeJ25x8gg5EBQ0Jnvp/IbBYQfPLtXH0Myq2QuAhk/1q2yEIbVjS+7iowEZNyE
 56K63WBJxsJPB2mvmLgn98GqB4G6GufP1ndS0XDti/2K0o8rep9xoY/JDGi0n0L0tk9BHyoP
 Y7kaEpu7UyY3nVdRLe5H1/MnFG8hdJ97WqnPS0buYZlrbTV0nRFL/NI2VABl18vEEXvNQiO+ vM8=
Message-ID: <38714ff6-371d-5fd2-4da7-0384a9bc8615@intel.com>
Date: Fri, 9 Nov 2018 20:06:17 +0000
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101
 Thunderbird/60.3.0
MIME-Version: 1.0
In-Reply-To: <1672099.GfbVsIzoma@xps>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Subject: Re: [dpdk-dev] [dpdk-stable] AVX512 bug on SkyLake
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Fri, 09 Nov 2018 20:06:24 -0000

On 11/9/2018 2:27 PM, Thomas Monjalon wrote:
> 09/11/2018 14:17, Thomas Monjalon:
>> 09/11/2018 11:03, Ferruh Yigit:
>>> On 11/8/2018 11:01 PM, Yongseok Koh wrote:
>>>>
>>>>> On Nov 8, 2018, at 9:21 AM, Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>>>>>
>>>>> On 11/8/2018 3:59 PM, Thomas Monjalon wrote:
>>>>>> Hi,
>>>>>>
>>>>>> We need to gather more information about this bug.
>>>>>> More below.
>>>>>>
>>>>>> 07/11/2018 10:04, Wiles, Keith:
>>>>>>>> On Nov 6, 2018, at 9:30 PM, Yongseok Koh <yskoh@mellanox.com> wrote:
>>>>>>>>> On Nov 5, 2018, at 6:06 AM, Wiles, Keith <keith.wiles@intel.com> wrote:
>>>>>>>>>> On Nov 2, 2018, at 9:04 PM, Yongseok Koh <yskoh@mellanox.com> wrote:
>>>>>>>>>>
>>>>>>>>>> This is a workaround to prevent a crash, which might be caused by
>>>>>>>>>> optimization of newer gcc (7.3.0) on Intel Skylake.
>>>>>>>>>
>>>>>>>>> Should the code below not also test for the gcc version and
>>>>>>>>> the Sky Lake processor, maybe I am wrong but it seems it is
>>>>>>>>> turning AVX512 for all GCC builds
>>>>>>>>
>>>>>>>> I didn't want to check gcc version as 7.3.0 is very new. Only gcc 8 is newly up since then (gcc 8.2).
>>>>>>>> Also, I wasn't able to test every gcc versions and I wanted to be a bit conservative for this crash.
>>>>>>>> Performance drop (if any) by disabling a new (experimental) feature would be less risky than unaccountable crash.
>>>>>>>> And, it does disable the feature only if CONFIG_RTE_ENABLE_AVX512=n. Please refer to v3.
>>>>>>>
>>>>>>> Are you not turning off all of the GCC versions for AVX512.
>>>>>>> And you can test for range or greater then GCC version and
>>>>>>> it just seems like we are turning off every gcc version, is that true?
>>>>>>
>>>>>> Do we know exactly which GCC versions are affected?
>>>>>>
>>>>>>>>> Also bug 97 seems a bit obscure reference, maybe you know
>>>>>>>>> the bug report, but more details would be good?
>>>>>>>>
>>>>>>>> I sent out the report to dev list two month ago.
>>>>>>>> And I created the Bug 97 in order to reference it
>>>>>>>> in the commit message.
>>>>>>>> I didn't want to repeat same message here and there,
>>>>>>>> but it would've been better to have some sort of summary
>>>>>>>> of the Bug, although v3 has a few more words.
>>>>>>>> However, v3 has been merged.
>>>>>>>
>>>>>>> Still this is too obscure if nothing else give a link to
>>>>>>> a specific bug not just 97.
>>>>>>
>>>>>> The URL is
>>>>>> 	https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.dpdk.org%2Fshow_bug.cgi%3Fid%3D97&amp;data=02%7C01%7Cyskoh%40mellanox.com%7C90ff6c361faf422b976108d6459eb490%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636772945282345908&amp;sdata=2o%2Fg203aWrKCYg16S6oI4BcS41igpLu1DloS%2FrRnknc%3D&amp;reserved=0
>>>>>> The bug is also pointing to an email:
>>>>>> 	https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmails.dpdk.org%2Farchives%2Fdev%2F2018-September%2F111522.html&amp;data=02%7C01%7Cyskoh%40mellanox.com%7C90ff6c361faf422b976108d6459eb490%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636772945282345908&amp;sdata=NCFKxaREd69iZ8eyFKg%2FWBP73CLTXkxrNQQeii%2Bbsao%3D&amp;reserved=0
>>>>>>
>>>>>> Summary:
>>>>>> 	- CPU: Intel Skylake
>>>>>> 	- Linux environment: Ubuntu 18.04
>>>>>> 	- Compiler: gcc-7.3 (Ubuntu 7.3.0-16ubuntu3)
>>>>>
>>>>> Is it possible to test a few other gcc versions to check if the issue is
>>>>> specific to this compiler version?
>>>>
>>>> Nothing's impossible but even with my quick search in gcc.gnu.org,
>>>> I could find the following documents mention mavx512f support:
>>>>
>>>> GCC 4.9.0
>>>> April 22, 2014 (changes, documentation)
>>>>  
>>>> GCC 5.1
>>>> April 22, 2015 (changes, documentation)
>>>>  
>>>> GCC 6.4
>>>> July 4, 2017 (changes, documentation)
>>>>  
>>>> GCC 7.1
>>>> May 2, 2017 (changes, documentation)
>>>>  
>>>> GCC 8.1
>>>> May 2, 2018 (changes, documentation)
>>>>
>>>> We altogether have to put quite large resource to verify all of the versions.
>>>>  
>>>> I assumed older than gcc 7 would have the same issue. I know it was a speculation
>>>> but like I mentioned I wanted to be more conservative. I didn't mean this is a permanent fix.
>>>> For two months, we couldn't have any tangible solution (actually nobody cared including myself),
>>>> so I submitted the patch to temporarily disable mavx512f.
>>>>
>>>> I'm still not sure what the best option is...
>>>
>>> For permanent fix we need more information, currently we can't re-produce this
>>> defect. Since you can reproduce it we need your support.
>>>
>>> Right now we don't know if this is compiler issue or code defect in rte_memcpy()
>>> or something else.
>>>
>>> It is easy to disable mavx512f as temporarily solution but it is coming with the
>>> cost of the performance drop, also without knowing the actual root cause I
>>> wouldn't say this is being conservative, actual issue may be just hidden with
>>> this change.
>>>
>>> I think as first thing we need to find a way to reproduce this issue in any
>>> other way than using mlx5 PMD. So that we can put more organized effort to fix this.
>>> I attached a simple unit test for rte_memcpy(), if this is a rte_memcpy() with
>>> avx512f defect as claimed, you should be able to see the issue with that, right?
>>> Did you able to find a chance to test it? Do you observer any crash there?
>>
>> I am able to connect to a machine where the issue is reproduced.
>> So I have tested replacing rte_memcpy with memcpy,
>> and the crash disappears when using memcpy.
>> So it confirms that the issue is in rte_memcpy.
> 
> One workaround is to disable CONFIG_RTE_ENABLE_AVX,
> but it is disabling AVX and AVX2 for all DPDK code.
> 
> A more limited fix (tested) can be to disable AVX2 version of rte_memcpy
> and rely on the AVX version (which is not crashing):
> 
> --- a/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
> +++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
> -#elif defined RTE_MACHINE_CPUFLAG_AVX2
> +#elif defined RTE_MACHINE_CPUFLAG_AVX2_disable

I put a patch into bugzilla:
https://bugs.dpdk.org/attachment.cgi?id=18&action=diff

Can you please check if this workaround prevents the crash without performance drop.

Also there is another suggestion from Yongseok, that looks simpler, but not
covering CONFIG_RTE_ENABLE_AVX512=y case.