From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-VE1-obe.outbound.protection.outlook.com (mail-eopbgr20085.outbound.protection.outlook.com [40.107.2.85]) by dpdk.org (Postfix) with ESMTP id 8FACC4CBB for ; Wed, 12 Sep 2018 22:56:10 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=65cGAGyAKqNjuPuThJoP0zBuoKqHVG/hl6Sne74WWaA=; b=hv2DMCyZyMl9amLUHqKVN9sDf4txT9wpC/PLZVt2NHz4keUSW470KBn9t+USUvdwkeAzLR2eyG2DjrCVRK02nMC7vP2kDsdIBHX6Izd8s0v65/KlzQD8FS5p/UXaz7oJ2r/7aNIg5E34Usl+uwFRLUffGydYmklzGh0Yp1MZEJg= Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by DB3PR0502MB4012.eurprd05.prod.outlook.com (52.134.72.141) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1122.17; Wed, 12 Sep 2018 20:56:08 +0000 Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::452:cfe7:8363:61c1]) by DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::452:cfe7:8363:61c1%2]) with mapi id 15.20.1122.020; Wed, 12 Sep 2018 20:56:08 +0000 From: Yongseok Koh To: Christian Ehrhardt CC: Shahaf Shuler , Olga Shern , Thomas Monjalon , Talat Batheesh , Noa Spanier , dev Thread-Topic: rte_memcpy() moves data incorrectly on Ubuntu 18.04 on Intel Skylake. Thread-Index: AQHUStsHwxJwBzukgE2t66MTnwb4hQ== Date: Wed, 12 Sep 2018 20:56:07 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; x-originating-ip: [209.116.155.178] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; DB3PR0502MB4012; 6:AyFHBun4ALeZjGHrJpwDmp/7MjQhTBlO9faqBvG7D3CNe45TftNZncIZT2vT07rvZ7OWtHeBJsD95pBbP+TUrLvgDsOq/C8Cd9fu6xIlnpW5mT1DGpgqEOoAnCXr70CwoLHkL1Smafw+G6NIJMOsuLANBli9T3C48D6tQKquyjA0E13YoUVRrY/+a13rCoEq+fpTDtt8gPzX9AMfSTQolfKWDJG4nXhOID0TxVizb9Nz/FYUJnfod9v5R1oCQyaMXqSWeivhEsx8GAIQ5g1kbaB1afqRUA3Flu6dgXgRICSh/sXgygCtcANZNc4N3htltbfLgvvhljiIpGoWXffo45FaSRjJiYsIxgDUV9rVMOQnPw2fn+Kx2e/fh4izN4rT7AazVsp9G4k3R4Tq8tBJtDhV8uzS5q4Ins2572ZjeVs6Wm4SK4IW5KSmb2uN/yaDU3nmO9by8coT1d6eZRx1Qw==; 5:wtg5pOP5s1URQ+/2w0hKp3EOy+nYwosKRuVl6bJytpza5nV/dp4dU6zKUmaq2JjwqAg8CwcRsU+/vya5qqrMHqApJ2IY+OGzdnxh5YadrEkkUbB2HhTZk6yGQnABvExj6X8TmbpFYeuF/tL674j+GUSR5XV76G87vS4Fbe4Xzk4=; 7:nSMJTRWpAfUkDVu7K5ax74fMh7WoBh/2UDtRqrY3CLpaEamOFTA6OIZq5yrSAhbGLsPrMXS1x6Rjh8DnhFB1OuidiRIeu8easwtnq2CQOwXZAdTnD9e9BDpT817J6aGmExWRNXm2fiJXcuQEp6hBGblRltv0MruqrJTI6GVLHZ258DKqGaBRcNwHiicPAQVFEC+AwQMKE1Nbk7Zxx19Kl++bvbm/0f4xYnxubagUHqeCmv8iebIhrzitKJfAFCup x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: cfa43b9e-d475-4a97-6e6e-08d618f229c7 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989137)(4534165)(4627221)(201703031133081)(201702281549075)(8990107)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020); SRVR:DB3PR0502MB4012; x-ms-traffictypediagnostic: DB3PR0502MB4012: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(131327999870524)(155532106045638); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(823301075)(3002001)(10201501046)(3231311)(944501410)(52105095)(93006095)(93001095)(6055026)(149027)(150027)(6041310)(20161123562045)(20161123564045)(20161123560045)(20161123558120)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(201708071742011)(7699050); SRVR:DB3PR0502MB4012; BCL:0; PCL:0; RULEID:; SRVR:DB3PR0502MB4012; x-forefront-prvs: 07935ACF08 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(136003)(346002)(366004)(39860400002)(376002)(396003)(199004)(189003)(99286004)(6436002)(25786009)(14444005)(5024004)(5660300001)(105586002)(5250100002)(6916009)(86362001)(6512007)(81156014)(4326008)(476003)(8676002)(81166006)(6116002)(2616005)(3846002)(68736007)(8936002)(45954006)(256004)(106356001)(486006)(6486002)(53936002)(33656002)(6506007)(186003)(102836004)(82746002)(14454004)(2906002)(26005)(19627235002)(66066001)(97736004)(54906003)(478600001)(7736002)(305945005)(36756003)(316002)(83716003)(2900100001); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB4012; H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: sZpa109o+GR2lVKHT02JH2WHyO20/l7QC8piL9jtxFgMhwvyZO2OohgTCDgjWLIK3BelOfVoyMXk+WxxWeI3xlsYB/xDaVDkb/2BChsivzc2aeTc6d7Hx2OfcjdzBHkq07ieiHP7LaY9Scmven/AH6bO3jsRnpIpMq3rOpZP+vkR8DOoffX0czX6f/WCAOannMJGZvyXe018aUBjDwaXoyHy3a9dXP6BmB6bR5ws7ToK1D5x23YlEgpj9sZoKCX9F7gnLZNd65gYLnO1fKn2NAq3VpLOyBdqtUu0iy6vyR+C79v41RMWXEHR9+lCa4EzU//CxTnQ6T5dExyYA8P7FWku4rJ5NUZUzKAI09NIwVo= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-ID: <098D0F53CF9FE54A8CEC181BBCCB4131@eurprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: cfa43b9e-d475-4a97-6e6e-08d618f229c7 X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Sep 2018 20:56:08.0185 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB4012 Subject: [dpdk-dev] rte_memcpy() moves data incorrectly on Ubuntu 18.04 on Intel Skylake. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Sep 2018 20:56:10 -0000 Hi, Christian We've recently encountered a weird issue with Ubuntu 18.04 on the Skylake server. I can always reproduce this crash and I could narrowed it down. I g= uess it could be a GCC issue. [1] How to reproduce - ConnectX-4Lx/ConnectX-5 with mlx5 PMD in DPDK 18.02.1 - Ubuntu 18.04 on Intel Skylake server - gcc (Ubuntu 7.3.0-16ubuntu3) 7.3.0 - Testpmd crashes when it starts to forward traffic. Easy to reproduce. - Only happens on the Skylake server. - DPDK 18.05 and later don't have such issue. git-bisect gives no clue. [2] Failure point The attached patch gives an insight of why it crashes. The following is the result of the patch and the GDB commands. In summary, rte_memcpy() doesn't work as expected. In __mempool_generic_put= (), there's rte_memcpy() to move the array of objects to the lcore cache. If I = run memcmp() right after rte_memcpy(dst, src, n), data in dst differs from data= in src. And it looks like some of data got shifted by a few bytes as you can s= ee below. [GDB command] $dst =3D 0x7ffff4e09ea8 $src =3D 0x7fffce3fb970 $n =3D 256 x/32gx 0x7ffff4e09ea8 x/32gx 0x7fffce3fb970 testpmd: /home/mlnxtest/dpdk/build/include/rte_mempool.h:1140: __mempool_g= eneric_put: Assertion `0' failed. Thread 4 "lcore-slave-1" received signal SIGABRT, Aborted. [Switching to Thread 0x7fffce3ff700 (LWP 69913)] (gdb) x/32gx 0x7ffff4e09ea8 0x7ffff4e09ea8: 0x00007fffaac38ec0 0x00007fffaac38500 0x7ffff4e09eb8: 0x00007fffaac37b40 0x00007fffaac37180 0x7ffff4e09ec8: 0x850000007fffaac3 0x7b4000007fffaac3 0x7ffff4e09ed8: 0x00007fffaac35440 0x00007fffaac34a80 0x7ffff4e09ee8: 0xaac3850000007fff 0xaac37b4000007fff 0x7ffff4e09ef8: 0x00007fffaac32d40 0x00007fffaac32380 0x7ffff4e09f08: 0x7fffaac385000000 0x7fffaac37b400000 0x7ffff4e09f18: 0x00007fffaac30640 0x00007fffaac2fc80 0x7ffff4e09f28: 0x00007fffaac2f2c0 0x00007fffaac2e900 0x7ffff4e09f38: 0x00007fffaac2df40 0x00007fffaac2d580 0x7ffff4e09f48: 0x00007fffaac2cbc0 0x00007fffaac2c200 0x7ffff4e09f58: 0x00007fffaac2b840 0x00007fffaac2ae80 0x7ffff4e09f68: 0x00007fffaac2a4c0 0x00007fffaac29b00 0x7ffff4e09f78: 0x00007fffaac29140 0x00007fffaac28780 0x7ffff4e09f88: 0x00007fffaac27dc0 0x00007fffaac27400 0x7ffff4e09f98: 0x00007fffaac26a40 0x00007fffaac26080 (gdb) x/32gx 0x7fffce3fb970 0x7fffce3fb970: 0x00007fffaac38ec0 0x00007fffaac38500 0x7fffce3fb980: 0x00007fffaac37b40 0x00007fffaac37180 0x7fffce3fb990: 0x00007fffaac367c0 0x00007fffaac35e00 0x7fffce3fb9a0: 0x00007fffaac35440 0x00007fffaac34a80 0x7fffce3fb9b0: 0x00007fffaac340c0 0x00007fffaac33700 0x7fffce3fb9c0: 0x00007fffaac32d40 0x00007fffaac32380 0x7fffce3fb9d0: 0x00007fffaac319c0 0x00007fffaac31000 0x7fffce3fb9e0: 0x00007fffaac30640 0x00007fffaac2fc80 0x7fffce3fb9f0: 0x00007fffaac2f2c0 0x00007fffaac2e900 0x7fffce3fba00: 0x00007fffaac2df40 0x00007fffaac2d580 0x7fffce3fba10: 0x00007fffaac2cbc0 0x00007fffaac2c200 0x7fffce3fba20: 0x00007fffaac2b840 0x00007fffaac2ae80 0x7fffce3fba30: 0x00007fffaac2a4c0 0x00007fffaac29b00 0x7fffce3fba40: 0x00007fffaac29140 0x00007fffaac28780 0x7fffce3fba50: 0x00007fffaac27dc0 0x00007fffaac27400 0x7fffce3fba60: 0x00007fffaac26a40 0x00007fffaac26080 AFAIK, AVX512F support is disabled by default in DPDK as it is still experimental (CONFIG_RTE_ENABLE_AVX512=3Dn). But with gcc optimization, AVX= 2 version of rte_memcpy() seems to be optimized with 512b instructions. If I disable it by adding EXTRA_CFLAGS=3D"-mno-avx512f", then it works fine and = doesn't crash. Do you have any idea regarding this issue or are you already aware of it? Thanks, Yongseok $ git diff diff --git a/config/common_base b/config/common_base index ad03cf433..f512b5a88 100644 --- a/config/common_base +++ b/config/common_base @@ -275,8 +275,8 @@ CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE=3D8 # # Compile burst-oriented Mellanox ConnectX-4 & ConnectX-5 (MLX5) PMD # -CONFIG_RTE_LIBRTE_MLX5_PMD=3Dn -CONFIG_RTE_LIBRTE_MLX5_DEBUG=3Dn +CONFIG_RTE_LIBRTE_MLX5_PMD=3Dy +CONFIG_RTE_LIBRTE_MLX5_DEBUG=3Dy CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS=3Dn CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE=3D8 @@ -597,7 +597,7 @@ CONFIG_RTE_RING_USE_C11_MEM_MODEL=3Dn # CONFIG_RTE_LIBRTE_MEMPOOL=3Dy CONFIG_RTE_MEMPOOL_CACHE_MAX_SIZE=3D512 -CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG=3Dn +CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG=3Dy # # Compile Mempool drivers diff --git a/lib/librte_mempool/rte_mempool.h b/lib/librte_mempool/rte_memp= ool.h index 8b1b7f7ed..9f48028d9 100644 --- a/lib/librte_mempool/rte_mempool.h +++ b/lib/librte_mempool/rte_mempool.h @@ -39,6 +39,7 @@ #include #include #include +#include #include #include @@ -1123,6 +1124,22 @@ __mempool_generic_put(struct rte_mempool *mp, void *= const *obj_table, /* Add elements back into the cache */ rte_memcpy(&cache_objs[0], obj_table, sizeof(void *) * n); + if(memcmp(&cache_objs[0], obj_table, sizeof(void *) * n)) { + printf("[GDB command] \n" + "$dst =3D %p\n" + "$src =3D %p\n" + "$n =3D %ld\n" + "x/%ldgx %p\n" + "x/%ldgx %p\n", + (void *)&cache_objs[0], + (const void *)obj_table, + sizeof(void *) * n, + sizeof(void *) * n / 8, (void *)&cache_objs[0], + sizeof(void *) * n / 8, (const void *)obj_table + ); + assert(0); + } + cache->len +=3D n; if (cache->len >=3D cache->flushthresh) {