From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-eopbgr10067.outbound.protection.outlook.com [40.107.1.67]) by dpdk.org (Postfix) with ESMTP id A537A4CB5 for ; Thu, 13 Sep 2018 01:37:33 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/LzAO1XK9kGsUdj8mZJGxEv7E2JbcS2qh2GSjPG8br8=; b=Q7CciKHgxpCoCb31s7M6nDjSfKanN10VuoimnYqCDnaQI8DQSmqiP/RNCaXxDFLrmDhf/sxaYa4a86xdQuZ+5lJJ/YHdTs3ilfrH65WRcnYtTISnMYWOrj7sA/sIJWC7EjhaE93ZLT/rH4MGHAB4lOrwz0VyF6n0XS5x62p7oxs= Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by DB3PR0502MB3994.eurprd05.prod.outlook.com (52.134.72.29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1122.15; Wed, 12 Sep 2018 23:37:32 +0000 Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::452:cfe7:8363:61c1]) by DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::452:cfe7:8363:61c1%2]) with mapi id 15.20.1122.020; Wed, 12 Sep 2018 23:37:32 +0000 From: Yongseok Koh To: Christian Ehrhardt CC: Shahaf Shuler , Olga Shern , Thomas Monjalon , Talat Batheesh , Noa Spanier , dev Thread-Topic: rte_memcpy() moves data incorrectly on Ubuntu 18.04 on Intel Skylake. Thread-Index: AQHUStsHHNuDvhsmCkWCnVEBsPvkLqTtTW+A Date: Wed, 12 Sep 2018 23:37:32 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; x-originating-ip: [209.116.155.178] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; DB3PR0502MB3994; 6:ShgzP+UMmWhoBNaBmk5mZhxjSNLQfURJSNwjJXKuWBwlhLtXCgTS/uySAWdT3CbBt+npu14Hs34td0oODa7l8cJzLdpLjTXFFCrdXmb/0p6h5CHeG6UiH2Nf0IRxWKuZ9oVAXe8X4Go4tksqf/81WY1tbzbZr9iL+M9LmjjcT7OYuD5C3E5+SPX+kEd4dPjoaE4sQTVLvVkzld7eq1SoW7Mmw0bf/PoIAMaNBk/gV44SeUsz1a1EjRLjuAgsbB7diP+uro7fdaeRo/0xDp8/sT/vfKE2+12+qvGOnFTNq8nwCbe+L0Vnl8goTH483+Eh6Kn0MdaCCYwW73iDb+7//YBiNdMNovM97TqhvkELSO5Wp9JIhQDrEbz0aoc7xf37qYGEQDs9zBSBGABk3RbG0w2GFHp4dhKb5tvhrcey9McGyoSlT5OVFc8qpKvohgtdQrChZ4lR6eqvIZwSpnAmeg==; 5:It/MsvN44V+YTPK8qCT3SaTDs6RN8AmgkZo++wpkGhLReC12lmXTojlT73YP7DgiCy2NWfdnQQKFFPjQJ3bphYj9nkfBft+3EVodZYuZ2Vji56631arZSimHt3zqiUCZ/PDxqiktg3CG8R7VmvsL/BzbY4KHPnb/xzglAam8g3c=; 7:7mAw59VRV9vvgMSNGAsrfwVzzajJp0zGVW2VD3v2y4kQkchZmEmOqbOCiPIntohaquxoJgkoDWHIcdxlZIqLvdjDAdOT3PO0pROH8owWS0AzW/sqPXHTi1bzbiO0thMVgkVNDsYBjM0UtkppPyb5kaPzm6o60KXJZw5dgH98ISFuupjIS2iaEfoCS2qXP1PABSup+omUfYxOLENCZklaTujg4LPjywhRghFCdLwR55f9ZSmO3fB4S0yy6YD7BNft x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: f41a1381-3cbc-414f-575b-08d61908b619 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989137)(4534165)(4627221)(201703031133081)(201702281549075)(8990107)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020); SRVR:DB3PR0502MB3994; x-ms-traffictypediagnostic: DB3PR0502MB3994: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(131327999870524)(155532106045638); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(823301075)(3002001)(93006095)(93001095)(3231311)(944501410)(52105095)(10201501046)(6055026)(149027)(150027)(6041310)(20161123564045)(20161123562045)(20161123558120)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(201708071742011)(7699050); SRVR:DB3PR0502MB3994; BCL:0; PCL:0; RULEID:; SRVR:DB3PR0502MB3994; x-forefront-prvs: 07935ACF08 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39860400002)(376002)(366004)(136003)(396003)(346002)(189003)(199004)(45954006)(229853002)(26005)(106356001)(446003)(11346002)(105586002)(6486002)(5660300001)(6116002)(54906003)(3846002)(25786009)(2906002)(6916009)(33656002)(2900100001)(316002)(6436002)(6246003)(8676002)(186003)(66066001)(83716003)(4326008)(6512007)(256004)(14444005)(68736007)(76176011)(8936002)(99286004)(7736002)(36756003)(14454004)(5024004)(476003)(53546011)(6506007)(53936002)(486006)(102836004)(97736004)(478600001)(2616005)(86362001)(81156014)(81166006)(5250100002)(305945005)(82746002)(19627235002); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB3994; H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: 7nubkvtZ+4xg6yjVFU6hvi8SqUXlK9iPYYf8j46sm3nbCd//Ofx3oPX4VVb9J6+4YvVv9a0ZFVjiwWoew6B3ruCjp4B1Oc7b4WrzfnKMPBbraCgkR6E4zPUR6ElIFrjJVSW+D/0o0Ro/GrGk2nRChAZosoxOGaglpn4Ptu48ZAZQGCXytSjFqw1hVKJpVyg+7UNfR4NAgetzfQ3bOFyN0Hw0ah8ehF63RkA0emrC4DrQ+Ei1zHoj4+wzOeE9g7NOhtNUuPUGfdm3mYvfkgHAy14T86LgP/CKfyMWHGbzXHXZN0kAQ1TfGnJ+k8E+ZnbGVbbOB2aLnTQTbuifBYmi9b+eq9vn+Rg8ByL+1M1HGck= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-ID: <04B70A66C827214C8D10577A57AA1A48@eurprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: f41a1381-3cbc-414f-575b-08d61908b619 X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Sep 2018 23:37:32.4113 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB3994 Subject: Re: [dpdk-dev] rte_memcpy() moves data incorrectly on Ubuntu 18.04 on Intel Skylake. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Sep 2018 23:37:35 -0000 > On Sep 12, 2018, at 1:56 PM, Yongseok Koh wrote: >=20 > Hi, Christian >=20 > We've recently encountered a weird issue with Ubuntu 18.04 on the Skylake > server. I can always reproduce this crash and I could narrowed it down. I= guess > it could be a GCC issue. >=20 >=20 > [1] How to reproduce > - ConnectX-4Lx/ConnectX-5 with mlx5 PMD in DPDK 18.02.1 > - Ubuntu 18.04 on Intel Skylake server > - gcc (Ubuntu 7.3.0-16ubuntu3) 7.3.0 > - Testpmd crashes when it starts to forward traffic. Easy to reproduce. > - Only happens on the Skylake server. > - DPDK 18.05 and later don't have such issue. git-bisect gives no clue. This is because I enabled MEMPOOL_DEBUG and MLX5_DEBUG. As mempool/rte_memc= py is inlined function, it should be affected. Now I can see the crash regardless= ly - 18.02, 18.05 and 18.08. Thanks, Yongseok. >=20 >=20 > [2] Failure point >=20 > The attached patch gives an insight of why it crashes. The following is t= he > result of the patch and the GDB commands. >=20 > In summary, rte_memcpy() doesn't work as expected. In __mempool_generic_p= ut(), > there's rte_memcpy() to move the array of objects to the lcore cache. If = I run > memcmp() right after rte_memcpy(dst, src, n), data in dst differs from da= ta in > src. And it looks like some of data got shifted by a few bytes as you can= see > below. >=20 > [GDB command] > $dst =3D 0x7ffff4e09ea8 > $src =3D 0x7fffce3fb970 > $n =3D 256 > x/32gx 0x7ffff4e09ea8 > x/32gx 0x7fffce3fb970 > testpmd: /home/mlnxtest/dpdk/build/include/rte_mempool.h:1140: __mempool= _generic_put: Assertion `0' failed. >=20 > Thread 4 "lcore-slave-1" received signal SIGABRT, Aborted. > [Switching to Thread 0x7fffce3ff700 (LWP 69913)] > (gdb) x/32gx 0x7ffff4e09ea8 > 0x7ffff4e09ea8: 0x00007fffaac38ec0 0x00007fffaac38500 > 0x7ffff4e09eb8: 0x00007fffaac37b40 0x00007fffaac37180 > 0x7ffff4e09ec8: 0x850000007fffaac3 0x7b4000007fffaac3 > 0x7ffff4e09ed8: 0x00007fffaac35440 0x00007fffaac34a80 > 0x7ffff4e09ee8: 0xaac3850000007fff 0xaac37b4000007fff > 0x7ffff4e09ef8: 0x00007fffaac32d40 0x00007fffaac32380 > 0x7ffff4e09f08: 0x7fffaac385000000 0x7fffaac37b400000 > 0x7ffff4e09f18: 0x00007fffaac30640 0x00007fffaac2fc80 > 0x7ffff4e09f28: 0x00007fffaac2f2c0 0x00007fffaac2e900 > 0x7ffff4e09f38: 0x00007fffaac2df40 0x00007fffaac2d580 > 0x7ffff4e09f48: 0x00007fffaac2cbc0 0x00007fffaac2c200 > 0x7ffff4e09f58: 0x00007fffaac2b840 0x00007fffaac2ae80 > 0x7ffff4e09f68: 0x00007fffaac2a4c0 0x00007fffaac29b00 > 0x7ffff4e09f78: 0x00007fffaac29140 0x00007fffaac28780 > 0x7ffff4e09f88: 0x00007fffaac27dc0 0x00007fffaac27400 > 0x7ffff4e09f98: 0x00007fffaac26a40 0x00007fffaac26080 > (gdb) x/32gx 0x7fffce3fb970 > 0x7fffce3fb970: 0x00007fffaac38ec0 0x00007fffaac38500 > 0x7fffce3fb980: 0x00007fffaac37b40 0x00007fffaac37180 > 0x7fffce3fb990: 0x00007fffaac367c0 0x00007fffaac35e00 > 0x7fffce3fb9a0: 0x00007fffaac35440 0x00007fffaac34a80 > 0x7fffce3fb9b0: 0x00007fffaac340c0 0x00007fffaac33700 > 0x7fffce3fb9c0: 0x00007fffaac32d40 0x00007fffaac32380 > 0x7fffce3fb9d0: 0x00007fffaac319c0 0x00007fffaac31000 > 0x7fffce3fb9e0: 0x00007fffaac30640 0x00007fffaac2fc80 > 0x7fffce3fb9f0: 0x00007fffaac2f2c0 0x00007fffaac2e900 > 0x7fffce3fba00: 0x00007fffaac2df40 0x00007fffaac2d580 > 0x7fffce3fba10: 0x00007fffaac2cbc0 0x00007fffaac2c200 > 0x7fffce3fba20: 0x00007fffaac2b840 0x00007fffaac2ae80 > 0x7fffce3fba30: 0x00007fffaac2a4c0 0x00007fffaac29b00 > 0x7fffce3fba40: 0x00007fffaac29140 0x00007fffaac28780 > 0x7fffce3fba50: 0x00007fffaac27dc0 0x00007fffaac27400 > 0x7fffce3fba60: 0x00007fffaac26a40 0x00007fffaac26080 >=20 >=20 > AFAIK, AVX512F support is disabled by default in DPDK as it is still > experimental (CONFIG_RTE_ENABLE_AVX512=3Dn). But with gcc optimization, A= VX2 > version of rte_memcpy() seems to be optimized with 512b instructions. If = I > disable it by adding EXTRA_CFLAGS=3D"-mno-avx512f", then it works fine an= d doesn't > crash. >=20 > Do you have any idea regarding this issue or are you already aware of it? >=20 >=20 > Thanks, > Yongseok >=20 >=20 > $ git diff > diff --git a/config/common_base b/config/common_base > index ad03cf433..f512b5a88 100644 > --- a/config/common_base > +++ b/config/common_base > @@ -275,8 +275,8 @@ CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE=3D8 > # > # Compile burst-oriented Mellanox ConnectX-4 & ConnectX-5 (MLX5) PMD > # > -CONFIG_RTE_LIBRTE_MLX5_PMD=3Dn > -CONFIG_RTE_LIBRTE_MLX5_DEBUG=3Dn > +CONFIG_RTE_LIBRTE_MLX5_PMD=3Dy > +CONFIG_RTE_LIBRTE_MLX5_DEBUG=3Dy > CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS=3Dn > CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE=3D8 >=20 > @@ -597,7 +597,7 @@ CONFIG_RTE_RING_USE_C11_MEM_MODEL=3Dn > # > CONFIG_RTE_LIBRTE_MEMPOOL=3Dy > CONFIG_RTE_MEMPOOL_CACHE_MAX_SIZE=3D512 > -CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG=3Dn > +CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG=3Dy >=20 > # > # Compile Mempool drivers > diff --git a/lib/librte_mempool/rte_mempool.h b/lib/librte_mempool/rte_me= mpool.h > index 8b1b7f7ed..9f48028d9 100644 > --- a/lib/librte_mempool/rte_mempool.h > +++ b/lib/librte_mempool/rte_mempool.h > @@ -39,6 +39,7 @@ > #include > #include > #include > +#include >=20 > #include > #include > @@ -1123,6 +1124,22 @@ __mempool_generic_put(struct rte_mempool *mp, void= * const *obj_table, > /* Add elements back into the cache */ > rte_memcpy(&cache_objs[0], obj_table, sizeof(void *) * n); >=20 > + if(memcmp(&cache_objs[0], obj_table, sizeof(void *) * n)) { > + printf("[GDB command] \n" > + "$dst =3D %p\n" > + "$src =3D %p\n" > + "$n =3D %ld\n" > + "x/%ldgx %p\n" > + "x/%ldgx %p\n", > + (void *)&cache_objs[0], > + (const void *)obj_table, > + sizeof(void *) * n, > + sizeof(void *) * n / 8, (void *)&cache_objs[0], > + sizeof(void *) * n / 8, (const void *)obj_table > + ); > + assert(0); > + } > + > cache->len +=3D n; >=20 > if (cache->len >=3D cache->flushthresh) { >=20 >=20