From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by dpdk.org (Postfix) with ESMTP id 0E5D94C77; Fri, 9 Nov 2018 10:49:25 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 09 Nov 2018 01:49:24 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,483,1534834800"; d="scan'208";a="106823398" Received: from fyigit-mobl.ger.corp.intel.com (HELO [10.237.221.79]) ([10.237.221.79]) by orsmga001.jf.intel.com with ESMTP; 09 Nov 2018 01:49:21 -0800 To: Christian Ehrhardt , yskoh@mellanox.com Cc: Thomas Monjalon , keith.wiles@intel.com, dev , Bruce Richardson , Shahaf Shuler , "Ananyev, Konstantin" , anatoly.burakov@intel.com, stable@dpdk.org, justin.parus@microsoft.com, David Coronel , Josh Powers , Jay Vosburgh , Dan Streetman References: <20181023212318.43082-1-yskoh@mellanox.com> <432F92CE-5714-45DC-B72F-CD8771DAFC89@intel.com> <1612642.At0RDolh7h@xps> <9d3f48fc-5a47-c813-1da8-7e1cab6bdd9e@intel.com> From: Ferruh Yigit Openpgp: preference=signencrypt Autocrypt: addr=ferruh.yigit@intel.com; prefer-encrypt=mutual; keydata= xsFNBFXZCFABEADCujshBOAaqPZpwShdkzkyGpJ15lmxiSr3jVMqOtQS/sB3FYLT0/d3+bvy qbL9YnlbPyRvZfnP3pXiKwkRoR1RJwEo2BOf6hxdzTmLRtGtwWzI9MwrUPj6n/ldiD58VAGQ +iR1I/z9UBUN/ZMksElA2D7Jgg7vZ78iKwNnd+vLBD6I61kVrZ45Vjo3r+pPOByUBXOUlxp9 GWEKKIrJ4eogqkVNSixN16VYK7xR+5OUkBYUO+sE6etSxCr7BahMPKxH+XPlZZjKrxciaWQb +dElz3Ab4Opl+ZT/bK2huX+W+NJBEBVzjTkhjSTjcyRdxvS1gwWRuXqAml/sh+KQjPV1PPHF YK5LcqLkle+OKTCa82OvUb7cr+ALxATIZXQkgmn+zFT8UzSS3aiBBohg3BtbTIWy51jNlYdy ezUZ4UxKSsFuUTPt+JjHQBvF7WKbmNGS3fCid5Iag4tWOfZoqiCNzxApkVugltxoc6rG2TyX CmI2rP0mQ0GOsGXA3+3c1MCdQFzdIn/5tLBZyKy4F54UFo35eOX8/g7OaE+xrgY/4bZjpxC1 1pd66AAtKb3aNXpHvIfkVV6NYloo52H+FUE5ZDPNCGD0/btFGPWmWRmkPybzColTy7fmPaGz cBcEEqHK4T0aY4UJmE7Ylvg255Kz7s6wGZe6IR3N0cKNv++O7QARAQABzSVGZXJydWggWWln aXQgPGZlcnJ1aC55aWdpdEBpbnRlbC5jb20+wsGVBBMBAgA/AhsDBgsJCAcDAgYVCAIJCgsE FgIDAQIeAQIXgBYhBNI2U4dCLsKE45mBx/kz60PfE2EfBQJbughWBQkHwjOGAAoJEPkz60Pf E2Eft84QAIbKWqhgqRfoiw/BbXbA1+qm2o4UgkCRQ0yJgt9QsnbpOmPKydHH0ixCliNz1J8e mRXCkMini1bTpnzp7spOjQGLeAFkNFz6BMq8YF2mVWbGEDE9WgnAxZdi0eLY7ZQnHbE6AxKL SXmpe9INb6z3ztseFt7mqje/W/6DWYIMnH3Yz9KzxujFWDcq8UCAvPkxVQXLTMpauhFgYeEx Nub5HbvhxTfUkapLwRQsSd/HbywzqZ3s/bbYMjj5JO3tgMiM9g9HOjv1G2f1dQjHi5YQiTZl 1eIIqQ3pTic6ROaiZqNmQFXPsoOOFfXF8nN2zg8kl/sSdoXWHhama5hbwwtl1vdaygQYlmdK H2ueiFh/UvT3WG3waNv2eZiEbHV8Rk52Xyn2w1G90lV0fYC6Ket1Xjoch7kjwbx793Kz/RfQ rmBY8/S4DTGn3oq3dMdQY+b6+7VMUeLMMh2CXYO9ErkOq+qNTD1IY+cBAkXnaDbQfz0zbste ZGWH74FAZ9nCpDOqbRTrBL42aMGhfOWEyeA1x7+hl6JZfabBWAuf4nnCXuorKHzBXTrf7u7p fXsKQClWRW77PF1VmzrtKNVSytQAmlCWApQIw20AarFipXmVdIjHmJPU611WoyxZPb4JTOxx 5cv9B+nr/RIB+v5dcStyHCCwO1be7nBDdCgd4F6kTQPLzsFNBFfWTL4BEACnNA29e8TarUsB L5n6eLZHXcFvVwNLVlirWOClHXf44o2KnN3ww+eBEmKVfEFo9MSuGDNHS8Zw1NiGMYxLIUgd U6gGrVVs/VrQWL82pbMk6jCj98N+BXIri+6K1z+AImz7ax7iF1kDgRAnFWU0znWWBgM2mM8Y gDjcxfXk4sCKnvf6Gjo08Ey5zmqx7dekAKU2EEp8Q1EJY3jbymLdZWRP4AFFMTS1rGMk0/tt v71NBg1GobCcbNfn9chK/jhqxYhAJqq86RdJQkt3/9x1U1Oq0vXCt4JVVHmkxePtUiuWTTt+ aYlUAsKYZsWvncExvw77x2ArYDmaK0yfjh37wp0lY7DOJHFxoyT8tyWZlLci/VMRG2Ja33xj 0CN4C1yBg+QDeV3QFxQo42iA/ykdXPUR3ezmsND3XKvVLTC4DNb3V/EZQ7jBj64+bEK0VW4G B31VP00ApNQvSoczsIOAKdk97RNbpmPw6q10ILIB+9T1xbnFYzshzGF17oC0/GENIHATx8vZ masOZoDiOZQpeneLgnFE9JfzhLTxv6wNZcc/HLXRQVTkDsQr8ERtkAoHCf1E5+b5Yr7pfnE4 YuhET746o25S53ELUYPIs49qoJsEJL34/oexMfPGyPIlrbufiNyty5jc/1MRwUlhJlJ5IOHy ZUa+6CLR7GdImusFkPJUJwARAQABwsF8BBgBAgAmAhsMFiEE0jZTh0IuwoTjmYHH+TPrQ98T YR8FAlu6CHAFCQXE7zIACgkQ+TPrQ98TYR9nXxAAqNBgkYNyGuWUuy0GwDQCbu3iiMyH1+D7 llafPcK4NYy1Z4AYuVwC9nmLaoj+ozdqS3ncRo57ncRsKEJC46nDJJZYZ5LSJVn63Y3NBF86 lxQAgjj2oyZEwaLKtKbAFsXL43jv1pUGgSvWwYtDwHITXXFQto9rZEuUDRFSx4sg9OR+Q6/6 LY+nQQ3OdHlBkflzYMPcWgDcvcTAO6yasLEUf7UcYoSWTyMYjLB4QuNlXzTswzGVMssJF/vo V8lD1eqqaSUWG3STF6GVLQOr1NLvN5+kUBiEStHFxBpgSCvYY9sNV8FS6N24CAWMBl+10W+D 2h1yiiP5dOdPcBDYKsgqDD91/sP0WdyMJkwdQJtD49f9f+lYloxHnSAxMleOpyscg1pldw+i mPaUY1bmIknLhhkqfMmjywQOXpac5LRMibAAYkcB8v7y3kwELnt8mhqqZy6LUsqcWygNbH/W K3GGt5tRpeIXeJ25x8gg5EBQ0Jnvp/IbBYQfPLtXH0Myq2QuAhk/1q2yEIbVjS+7iowEZNyE 56K63WBJxsJPB2mvmLgn98GqB4G6GufP1ndS0XDti/2K0o8rep9xoY/JDGi0n0L0tk9BHyoP Y7kaEpu7UyY3nVdRLe5H1/MnFG8hdJ97WqnPS0buYZlrbTV0nRFL/NI2VABl18vEEXvNQiO+ vM8= Message-ID: Date: Fri, 9 Nov 2018 09:49:17 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] AVX512 bug on SkyLake X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Nov 2018 09:49:27 -0000 On 11/9/2018 6:27 AM, Christian Ehrhardt wrote: > On Fri, Nov 9, 2018 at 12:01 AM Yongseok Koh wrote: >> >> >>> On Nov 8, 2018, at 9:21 AM, Ferruh Yigit wrote: >>> >>> On 11/8/2018 3:59 PM, Thomas Monjalon wrote: >>>> Hi, >>>> >>>> We need to gather more information about this bug. >>>> More below. >>>> > > Thanks Thomas for looping us in! > >>>> 07/11/2018 10:04, Wiles, Keith: >>>>>> On Nov 6, 2018, at 9:30 PM, Yongseok Koh wrote: >>>>>>> On Nov 5, 2018, at 6:06 AM, Wiles, Keith wrote: >>>>>>>> On Nov 2, 2018, at 9:04 PM, Yongseok Koh wrote: >>>>>>>> >>>>>>>> This is a workaround to prevent a crash, which might be caused by >>>>>>>> optimization of newer gcc (7.3.0) on Intel Skylake. >>>>>>> >>>>>>> Should the code below not also test for the gcc version and >>>>>>> the Sky Lake processor, maybe I am wrong but it seems it is >>>>>>> turning AVX512 for all GCC builds >>>>>> >>>>>> I didn't want to check gcc version as 7.3.0 is very new. Only gcc 8 is newly up since then (gcc 8.2). >>>>>> Also, I wasn't able to test every gcc versions and I wanted to be a bit conservative for this crash. >>>>>> Performance drop (if any) by disabling a new (experimental) feature would be less risky than unaccountable crash. >>>>>> And, it does disable the feature only if CONFIG_RTE_ENABLE_AVX512=n. Please refer to v3. >>>>> >>>>> Are you not turning off all of the GCC versions for AVX512. >>>>> And you can test for range or greater then GCC version and >>>>> it just seems like we are turning off every gcc version, is that true? >>>> >>>> Do we know exactly which GCC versions are affected? >>>> >>>>>>> Also bug 97 seems a bit obscure reference, maybe you know >>>>>>> the bug report, but more details would be good? >>>>>> >>>>>> I sent out the report to dev list two month ago. >>>>>> And I created the Bug 97 in order to reference it >>>>>> in the commit message. >>>>>> I didn't want to repeat same message here and there, >>>>>> but it would've been better to have some sort of summary >>>>>> of the Bug, although v3 has a few more words. >>>>>> However, v3 has been merged. >>>>> >>>>> Still this is too obscure if nothing else give a link to >>>>> a specific bug not just 97. >>>> >>>> The URL is >>>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.dpdk.org%2Fshow_bug.cgi%3Fid%3D97&data=02%7C01%7Cyskoh%40mellanox.com%7C90ff6c361faf422b976108d6459eb490%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636772945282345908&sdata=2o%2Fg203aWrKCYg16S6oI4BcS41igpLu1DloS%2FrRnknc%3D&reserved=0 >>>> The bug is also pointing to an email: >>>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmails.dpdk.org%2Farchives%2Fdev%2F2018-September%2F111522.html&data=02%7C01%7Cyskoh%40mellanox.com%7C90ff6c361faf422b976108d6459eb490%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636772945282345908&sdata=NCFKxaREd69iZ8eyFKg%2FWBP73CLTXkxrNQQeii%2Bbsao%3D&reserved=0 >>>> >>>> Summary: >>>> - CPU: Intel Skylake >>>> - Linux environment: Ubuntu 18.04 >>>> - Compiler: gcc-7.3 (Ubuntu 7.3.0-16ubuntu3) >>> >>> Is it possible to test a few other gcc versions to check if the issue is >>> specific to this compiler version? >> >> Nothing's impossible but even with my quick search in gcc.gnu.org, >> I could find the following documents mention mavx512f support: >> >> GCC 4.9.0 >> April 22, 2014 (changes, documentation) >> >> GCC 5.1 >> April 22, 2015 (changes, documentation) >> >> GCC 6.4 >> July 4, 2017 (changes, documentation) >> >> GCC 7.1 >> May 2, 2017 (changes, documentation) >> >> GCC 8.1 >> May 2, 2018 (changes, documentation) >> >> We altogether have to put quite large resource to verify all of the versions. >> >> I assumed older than gcc 7 would have the same issue. I know it was a speculation >> but like I mentioned I wanted to be more conservative. I didn't mean this is a permanent fix. >> For two months, we couldn't have any tangible solution (actually nobody cared including myself), >> so I submitted the patch to temporarily disable mavx512f. >> >> I'm still not sure what the best option is... >> > > What I wonder in all of this as I don't understand that part of it yet is this. > I assume you are building on Ubuntu as that is your gcc reference. > FYI: as people asked for bug references, there also is [1] which seems > pretty much the same issue. > > It builds with mostly defaults, that means per > mk/machine/default/rte.vars.mk and similar it sets -march=corei7 > > But when I look at what that implies all avx512 is disabled > $ gcc -Q --help=target -m64 -march=corei7 | grep avx512f > -mavx512f [disabled] This is output is the "corei7" architecture. Defect is for "native" build on Skylake. For "native" arch: gcc -Q --help=target -m64 -march=native | grep avx512f -mavx512f [enabled] Also I can confirm from disassembly output that avx512 instructions are used without '-mno-avx512f' provided. > > So I wonder what/why -mno-avx512f should help at all. > I used the full list of gcc args we have for the build (e.g. [2] of a > 18.05 build), but that doesn't change that (mostly -W, -I and -D). > So I wonder, did people do a custom build and bump up march or enable > -mavx512f on their own to hit that? > Or are we facing a real gcc issue where " -mavx512f [disabled]" is not > the same as -mno-avx512f ? > Maybe someone who hit the bug could clarify that please? > > BTW: per reports I've seen it also seems to apply to the latest > compiler update of the same series - at least it was said to be fully > updated, that would be 7.3.0-27ubuntu1~18.04 > But this is 2nd grade information as I don't have a system with the > right combo MLX5+Skylake available atm, so I can't confirm for sure > :-/ > > [1]: https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1799397 > [2]: https://launchpadlibrarian.net/373589345/buildlog_ubuntu-bionic-amd64.dpdk_18.05-1~ubuntu0.18.04.1_BUILDING.txt.gz >