From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by dpdk.org (Postfix) with ESMTP id 439D94C94; Fri, 9 Nov 2018 14:17:36 +0100 (CET) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id C63EE22011; Fri, 9 Nov 2018 08:17:35 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Fri, 09 Nov 2018 08:17:35 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s=mesmtp; bh=+8TNoxOHC82TPPpTkeJrakglAES7zfY6caoD7hCEsVE=; b=a2K0QBctezFd XGR/wkcnMsLHv4jSCLTxFKEXnBe4Li6XyGDV6PEOqEvuJ/8A5RdwzzTxs8cyyf3R gEjAAG6z2Lj2/lPbv47FVu2QdbCsV2eh8thYGELfSlla7/JEJZxrZ7dPPBzj32/C LViX94b0MgOz5xTmSYSvWkHb9GaRzkw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=+8TNoxOHC82TPPpTkeJrakglAES7zfY6caoD7hCEs VE=; b=suh3dm2bHf3+pmBQrPtRZvDUgW76+2aTvhIFCeXnnmf/n+zj3VBrQPBxo 7T32uyAWngk3fJM8sf4054BRY+9vnvKd3vNmjICRbV54uRnrav5M/1/ekZUds0PK BiAoHP1T+iZRJs5kFL07f4xM7BAn4juvsYyx/gBEh+kd/nwavvnmk3HWjrM8m/di BDK5pp1dYwwmKGP7BDYPHWaoTpiKKjWsoK4CtTPhJLbKK4GpwVPHNHXAFLc/toD2 uOD2VcCuwo17FoIKTOO1XnD/94mOrFhZKafr+9cRjWD6+kKgPjt/y412DD/z/MBq 4Dr0ireqUOpN8qmBQr+aqvWE3oHZw== X-ME-Sender: X-ME-Proxy: Received: from xps.localnet (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id 6E207E4788; Fri, 9 Nov 2018 08:17:33 -0500 (EST) From: Thomas Monjalon To: Ferruh Yigit Cc: stable@dpdk.org, Yongseok Koh , keith.wiles@intel.com, dev@dpdk.org, bruce.richardson@intel.com, Shahaf Shuler , konstantin.ananyev@intel.com, anatoly.burakov@intel.com, justin.parus@microsoft.com, "christian.ehrhardt@canonical.com" , "david.coronel@canonical.com" , "josh.powers@canonical.com" , "jay.vosburgh@canonical.com" , "dan.streetman@canonical.com" Date: Fri, 09 Nov 2018 14:17:31 +0100 Message-ID: <2236912.nQJHooqEX5@xps> In-Reply-To: <9891a520-492a-2d18-65e7-c4fcd96a5e3f@intel.com> References: <20181023212318.43082-1-yskoh@mellanox.com> <9891a520-492a-2d18-65e7-c4fcd96a5e3f@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Subject: Re: [dpdk-dev] [dpdk-stable] AVX512 bug on SkyLake X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Nov 2018 13:17:38 -0000 09/11/2018 11:03, Ferruh Yigit: > On 11/8/2018 11:01 PM, Yongseok Koh wrote: > > > >> On Nov 8, 2018, at 9:21 AM, Ferruh Yigit wrote: > >> > >> On 11/8/2018 3:59 PM, Thomas Monjalon wrote: > >>> Hi, > >>> > >>> We need to gather more information about this bug. > >>> More below. > >>> > >>> 07/11/2018 10:04, Wiles, Keith: > >>>>> On Nov 6, 2018, at 9:30 PM, Yongseok Koh wrote: > >>>>>> On Nov 5, 2018, at 6:06 AM, Wiles, Keith wrote: > >>>>>>> On Nov 2, 2018, at 9:04 PM, Yongseok Koh wrote: > >>>>>>> > >>>>>>> This is a workaround to prevent a crash, which might be caused by > >>>>>>> optimization of newer gcc (7.3.0) on Intel Skylake. > >>>>>> > >>>>>> Should the code below not also test for the gcc version and > >>>>>> the Sky Lake processor, maybe I am wrong but it seems it is > >>>>>> turning AVX512 for all GCC builds > >>>>> > >>>>> I didn't want to check gcc version as 7.3.0 is very new. Only gcc 8 is newly up since then (gcc 8.2). > >>>>> Also, I wasn't able to test every gcc versions and I wanted to be a bit conservative for this crash. > >>>>> Performance drop (if any) by disabling a new (experimental) feature would be less risky than unaccountable crash. > >>>>> And, it does disable the feature only if CONFIG_RTE_ENABLE_AVX512=n. Please refer to v3. > >>>> > >>>> Are you not turning off all of the GCC versions for AVX512. > >>>> And you can test for range or greater then GCC version and > >>>> it just seems like we are turning off every gcc version, is that true? > >>> > >>> Do we know exactly which GCC versions are affected? > >>> > >>>>>> Also bug 97 seems a bit obscure reference, maybe you know > >>>>>> the bug report, but more details would be good? > >>>>> > >>>>> I sent out the report to dev list two month ago. > >>>>> And I created the Bug 97 in order to reference it > >>>>> in the commit message. > >>>>> I didn't want to repeat same message here and there, > >>>>> but it would've been better to have some sort of summary > >>>>> of the Bug, although v3 has a few more words. > >>>>> However, v3 has been merged. > >>>> > >>>> Still this is too obscure if nothing else give a link to > >>>> a specific bug not just 97. > >>> > >>> The URL is > >>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.dpdk.org%2Fshow_bug.cgi%3Fid%3D97&data=02%7C01%7Cyskoh%40mellanox.com%7C90ff6c361faf422b976108d6459eb490%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636772945282345908&sdata=2o%2Fg203aWrKCYg16S6oI4BcS41igpLu1DloS%2FrRnknc%3D&reserved=0 > >>> The bug is also pointing to an email: > >>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmails.dpdk.org%2Farchives%2Fdev%2F2018-September%2F111522.html&data=02%7C01%7Cyskoh%40mellanox.com%7C90ff6c361faf422b976108d6459eb490%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636772945282345908&sdata=NCFKxaREd69iZ8eyFKg%2FWBP73CLTXkxrNQQeii%2Bbsao%3D&reserved=0 > >>> > >>> Summary: > >>> - CPU: Intel Skylake > >>> - Linux environment: Ubuntu 18.04 > >>> - Compiler: gcc-7.3 (Ubuntu 7.3.0-16ubuntu3) > >> > >> Is it possible to test a few other gcc versions to check if the issue is > >> specific to this compiler version? > > > > Nothing's impossible but even with my quick search in gcc.gnu.org, > > I could find the following documents mention mavx512f support: > > > > GCC 4.9.0 > > April 22, 2014 (changes, documentation) > > > > GCC 5.1 > > April 22, 2015 (changes, documentation) > > > > GCC 6.4 > > July 4, 2017 (changes, documentation) > > > > GCC 7.1 > > May 2, 2017 (changes, documentation) > > > > GCC 8.1 > > May 2, 2018 (changes, documentation) > > > > We altogether have to put quite large resource to verify all of the versions. > > > > I assumed older than gcc 7 would have the same issue. I know it was a speculation > > but like I mentioned I wanted to be more conservative. I didn't mean this is a permanent fix. > > For two months, we couldn't have any tangible solution (actually nobody cared including myself), > > so I submitted the patch to temporarily disable mavx512f. > > > > I'm still not sure what the best option is... > > For permanent fix we need more information, currently we can't re-produce this > defect. Since you can reproduce it we need your support. > > Right now we don't know if this is compiler issue or code defect in rte_memcpy() > or something else. > > It is easy to disable mavx512f as temporarily solution but it is coming with the > cost of the performance drop, also without knowing the actual root cause I > wouldn't say this is being conservative, actual issue may be just hidden with > this change. > > I think as first thing we need to find a way to reproduce this issue in any > other way than using mlx5 PMD. So that we can put more organized effort to fix this. > I attached a simple unit test for rte_memcpy(), if this is a rte_memcpy() with > avx512f defect as claimed, you should be able to see the issue with that, right? > Did you able to find a chance to test it? Do you observer any crash there? I am able to connect to a machine where the issue is reproduced. So I have tested replacing rte_memcpy with memcpy, and the crash disappears when using memcpy. So it confirms that the issue is in rte_memcpy. About the unit test you attached in bugzilla: https://bugs.dpdk.org/attachment.cgi?id=15 It does not reproduce the bug: RTE>>rte_memcpy_autotest ................................................................. Test OK