From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 317EBA0471 for ; Fri, 19 Jul 2019 05:53:59 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id DCD092BE5; Fri, 19 Jul 2019 05:53:57 +0200 (CEST) Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-eopbgr60069.outbound.protection.outlook.com [40.107.6.69]) by dpdk.org (Postfix) with ESMTP id 1D65B1B53 for ; Fri, 19 Jul 2019 05:53:56 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ote/YwYp/7Y28bolulfmvCAEJMU9ha1U5g+rRZQzjzFMiZR0O5Le1TympNK47QuIVY4pmrUojU7JbR6NxWK9UFKwO8SNU0QYZxwgLk+961gJxoJle9PONzq1XQkMvda6sscM0Z7Ny5mjD/d+QzaqH1jSAfmKxnc1JycUBDDoD7+8F5dmMvKmh/k8cZ/UURxJRtLtXwU9N1QnGX5lpiIb65fok7KqqCyBI2w7Ayy0vAQqIeOqTPYAurL/1OrXpuE8B8KYc0hJNOf2Zqqz6yvNYSu0u0mtl8TNnXHIWiEOBSbtlquSDBYJKhEAkmm/CG1JK8TwV3qwY7XNOI1Ydg+ugw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dQzqh+AoYSPPmTRcVWZFEl/NZJ10xjYw8ZvoPfkHXAM=; b=FwZoHJOWBWHz+o1V4EIqWUofwRBLva4jmd4Hhkcux/1n5fbsuCmPK9kr/GmTONexlBVd5SYyeQkef/pP8MqRmMGkjdPeJNcNmVvPlQJKYmsMQ9mskUzgVwxcT4DunhG7/MWM4/cNaJRQZn3st2hqFqOBKRLv2JoRA7B8aR4DkEDYru41nluR4eJM8cN6keYyMH6HhCZnoyqzU5cOPzhEmIQjxYNZCvPR8pXjBTa6Zfo6RzipQEXbMOhQ74Ya8CKxuQnbfyvg0yBZPrgvcBEHEwqAOjR6aLydbFlxsGOox1bjW8j9aQuJLxQ8GCKy1T6ZZmCo4x/isQTzCX6qRJWiBQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1;spf=pass smtp.mailfrom=mellanox.com;dmarc=pass action=none header.from=mellanox.com;dkim=pass header.d=mellanox.com;arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dQzqh+AoYSPPmTRcVWZFEl/NZJ10xjYw8ZvoPfkHXAM=; b=cwpq8WkNoHE/KEh0hAMeYdEP6mQ/BSRxQnJ5oK/Lbz3kiZmdwkwNTKnzACkgmCGHIy0d9ZIIoxkQ0mwqS52Krai6mQCNdkCUeTrESjheL8ke/sxAI7bX3pPqZ/JrylV9hYZowGBXm2sRdTODqCIuyhVYA897oT8GWNIhGK+aCUU= Received: from VI1PR05MB3278.eurprd05.prod.outlook.com (10.170.238.23) by VI1PR05MB5918.eurprd05.prod.outlook.com (20.178.126.27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2094.11; Fri, 19 Jul 2019 03:53:55 +0000 Received: from VI1PR05MB3278.eurprd05.prod.outlook.com ([fe80::75a7:b54e:c238:5c21]) by VI1PR05MB3278.eurprd05.prod.outlook.com ([fe80::75a7:b54e:c238:5c21%5]) with mapi id 15.20.2094.011; Fri, 19 Jul 2019 03:53:55 +0000 From: Slava Ovsiienko To: Yasuhiro Ohara , "dev@dpdk.org" Thread-Topic: [dpdk-dev] ConnectX-4/mlx5 crashes around rxq_cqe_comp_en? Thread-Index: AQHVONBXqU+rn8w1rU++NGWyCCSX7qbRPzuAgAAYwkA= Date: Fri, 19 Jul 2019 03:53:54 +0000 Message-ID: References: <20190713.013853.751044529514409504.yasu@nttv6.jp> <20190719.111945.2086809368117346464.yasu@nttv6.jp> In-Reply-To: <20190719.111945.2086809368117346464.yasu@nttv6.jp> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=viacheslavo@mellanox.com; x-originating-ip: [95.67.35.250] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 439c5890-ffec-4959-908a-08d70bfcb877 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600148)(711020)(4605104)(1401327)(4618075)(2017052603328)(7193020); SRVR:VI1PR05MB5918; x-ms-traffictypediagnostic: VI1PR05MB5918: x-ms-exchange-purlcount: 1 x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 01039C93E4 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(136003)(366004)(376002)(39860400002)(346002)(396003)(199004)(189003)(13464003)(76116006)(66476007)(110136005)(66946007)(5660300002)(26005)(11346002)(76176011)(102836004)(476003)(186003)(86362001)(71190400001)(7696005)(71200400001)(446003)(68736007)(256004)(14444005)(53936002)(99286004)(6506007)(53546011)(81156014)(486006)(2501003)(64756008)(74316002)(66446008)(8936002)(316002)(14454004)(478600001)(52536014)(81166006)(66556008)(33656002)(6306002)(55016002)(6246003)(229853002)(305945005)(6116002)(7736002)(25786009)(66066001)(6436002)(8676002)(2906002)(3846002)(9686003); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR05MB5918; H:VI1PR05MB3278.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: asFZpQndMf2DS0lhMub2MJX9Rop6sB/9E4PyqQ20+QrRDLxcImvldiqpP+likW3+n3FcVwlVoIdyFVJElXX+HLjkgygEjow4IfDiyCRM02c65LtvbyBBihpCbmSoY7lllIzSRUogrGalDL1B/hTZL5AU44P9j6vaaPQWglxnX0ehSjRp0/Eg3qaX/KV1tqaX1pAFRKIno2tKGk84xiDpBmYpCp6UnQYsB5dN7CI8Ymk3bbDnzdAOHn0avv1mUUlX5+ANfe/zTHlDyzrVAPxv/JT7c03WrPxMtRXaN/Y6PBMSajd6ViLsOtZSQWskERwVq6Xjlx1MgEVulKCCMjjXLLsGu0sN+Auk8NmDjlKOEAegGHtDVLsAbuHieOyc4jPwAX6KPWcM8aAeVq5uynqVmFYuJbsacLbK1UmcuIFByBg= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 439c5890-ffec-4959-908a-08d70bfcb877 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Jul 2019 03:53:54.9342 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: viacheslavo@mellanox.com X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR05MB5918 Subject: Re: [dpdk-dev] ConnectX-4/mlx5 crashes around rxq_cqe_comp_en? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi, Yasuhiro Could you, please, create the ticket in Bugzilla.dpdk.org to store the deta= ils? The Rx CQE compression can be disabled with specifying "rxq_cqe_comp_en=3D0= ". WBR, Slava > -----Original Message----- > From: dev On Behalf Of Yasuhiro Ohara > Sent: Friday, July 19, 2019 5:20 > To: dev@dpdk.org > Subject: Re: [dpdk-dev] ConnectX-4/mlx5 crashes around rxq_cqe_comp_en? >=20 >=20 > The same goes with DPDK-19.05 too. >=20 > When crash happens, > mcqe_n =3D=3D t_pkt->data_len =3D=3D 124. >=20 > struct rte_mbuf **elts (which seems to be prepared somewhere) looks like > it's supposed to contain valid mbufs, but (when under a significant load?= ) it > doesn't. >=20 > (gdb) p/x (void*[124])elts[0] > $31 =3D {0x1d0bd0d80, 0x1d1cfef80, 0x1d28f6a40, 0x1d22eb100, > 0x1d195a8c0, > 0x1d2137200, 0x1d1eb5540, 0x1d1d0fec0, 0x1d28ecf40, 0x1d19b1bc0, > 0x1cec8a200, 0x1d02e2980, 0x1d085cdc0, 0x1d04e8e00, 0x1ccb4e140, > 0x1d1e17e80, 0x1d17a1c40, 0x1d14a6e00, 0x1d2871700, 0x1d20b6c40, > 0x1d29831c0, 0x1d04941c0, 0x1d0921080, 0x1d070ea40, 0x1d148ea80, > 0x1cee100c0, 0x1d1a47e40, 0x1d0ee6600, 0x1d02f1200, 0x1d24bc100, > 0x1d1e84e40, 0x1d1e1f2c0, 0x1d28b7ac0, 0x1d2195940, 0x1d21bc540, > 0x1d228f080, 0x1d1026100, 0x1d285e100, 0x1d211c7c0, 0x1d2128980, > 0x1d1787200, 0x1d170e080, 0x1d1e0e380, 0x1ce638500, 0x1d21a6880, > 0x1d20d8ac0, 0x1d25e8600, 0x1d2377880, 0x1d0e13ac0, 0x1c0c07100, > 0x1c0c07100, 0x1c0c07100, 0x1c0c07100, 0x0, 0x0, 0x0, 0x0, > 0x7ffff7ff487c, > 0x1c0c06f00, 0x1c0c08b00, 0x0, 0x0, 0x7ffff7ff207c, 0x1, 0x1480, > 0x140000000, 0x100000000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, > 0x40000000ffffffff, 0x4000000000000001, 0xcd64010000000002, > 0x0 } >=20 > (gdb) p elts[48] > $38 =3D (struct rte_mbuf *) 0x1d0e13ac0 > (gdb) p elts[49] > $39 =3D (struct rte_mbuf *) 0x1c0c07100 > (gdb) p elts[50] > $40 =3D (struct rte_mbuf *) 0x1c0c07100 > (gdb) p elts[51] > $41 =3D (struct rte_mbuf *) 0x1c0c07100 > (gdb) p elts[52] > $42 =3D (struct rte_mbuf *) 0x1c0c07100 > (gdb) p elts[53] > $43 =3D (struct rte_mbuf *) 0x0 >=20 > Any thoughts? >=20 > regards, > Yasu >=20 > From: Yasuhiro Ohara > Subject: [dpdk-dev] ConnectX-4/mlx5 crashes around rxq_cqe_comp_en? > Date: Sat, 13 Jul 2019 01:38:53 +0900 (JST) > Message-ID: <20190713.013853.751044529514409504.yasu@nttv6.jp> >=20 > > > > Hi, > > > > I get a crash when I put a significant amount of load on > > ConnectX-4/mlx5, i.e., 50Gbps for 100GbE port. > > > > Thread 22 "lcore-slave-19" received signal SIGSEGV, Segmentation fault. > > [Switching to Thread 0x7fffe77ee700 (LWP 33519)] > > 0x0000555555f010a3 in _mm_storeu_si128 (__B=3D..., __P=3D0x10) > > at /usr/lib/gcc/x86_64-linux-gnu/7/include/emmintrin.h:721 > > 721 *__P =3D __B; > > (gdb) bt > > #0 0x0000555555f010a3 in _mm_storeu_si128 (__B=3D..., __P=3D0x10) > > at /usr/lib/gcc/x86_64-linux-gnu/7/include/emmintrin.h:721 > > #1 rxq_cq_decompress_v (rxq=3D0x22c910ccc0, cq=3D0x22c8fd1800, > elts=3D0x22c910d240) > > at > > /usr/local/dpdk-stable-18.11.2/drivers/net/mlx5/mlx5_rxtx_vec_sse.h:42 > > 1 > > #2 0x0000555555f04b42 in rxq_burst_v (rxq=3D0x22c910ccc0, > pkts=3D0x7fffe77eba40, > > pkts_n=3D32, err=3D0x7fffe77dc978) > > at > > /usr/local/dpdk-stable-18.11.2/drivers/net/mlx5/mlx5_rxtx_vec_sse.h:95 > > 6 > > #3 0x0000555555f055ea in mlx5_rx_burst_vec (dpdk_rxq=3D0x22c910ccc0, > > pkts=3D0x7fffe77eba40, pkts_n=3D32) > > at > > /usr/local/dpdk-stable-18.11.2/drivers/net/mlx5/mlx5_rxtx_vec.c:238 > > #4 0x0000555555632772 in rte_eth_rx_burst (port_id=3D4, queue_id=3D5, > > rx_pkts=3D0x7fffe77eba40, nb_pkts=3D32) > > at > > /usr/local/dpdk-18.11/x86_64-native-linuxapp-gcc/include/rte_ethdev.h: > > 3879 > > > > My environments are: > > > > Ubuntu 18.04.2 LTS 4.15.0-50-generic > > MLNX_OFED_LINUX-4.5-1.0.1.0-ubuntu18.04-x86_64 > > fw_ver: 12.17.2020 > > vendor_id: 0x02c9 > > vendor_part_id: 4115 > > hw_ver: 0x0 > > board_id: LNR3270110033 > > DPDK 18.11.2 > > > > It looks like the CQE compression is the crashing place. > > > > dpdk-stable-18.11.2/drivers/net/mlx5/mlx5_rxtx_vec_sse.h:956 > > 953 /* Decompress the last CQE if compressed. */ > > 954 if (comp_idx < MLX5_VPMD_DESCS_PER_LOOP && comp_idx =3D=3D = n) > { > > 955 assert(comp_idx =3D=3D (nocmp_n % > MLX5_VPMD_DESCS_PER_LOOP)); > > 956 rxq_cq_decompress_v(rxq, &cq[nocmp_n], &elts[nocmp_= n]); > > > > And I'm wondering how I can disable rxq_cqe_comp_en devargs. > > > > > > .dpdk.org%2Fguides- > 18.02%2Fnics%2Fmlx5.html&data=3D02%7C01%7Cviaches > > > lavo%40mellanox.com%7C346e0b13aba945bce40808d70befa6e9%7Ca6529 > 71c7d2e4 > > > d9ba6a4d149256f461b%7C0%7C0%7C636990996240384705&sdata=3Did > %2FJY%2BM > > PhzsUBmFP9YoKtFkcu%2FbJtO6Ntb1QghmnSdQ%3D&reserved=3D0> > > 22.5.3. Run-time configuration > > rxq_cqe_comp_en parameter [int] > > > > Any information or guesses are appreciated. > > > > Best regards, > > Yasu > >