From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7C7D7A0C45 for ; Wed, 4 Aug 2021 09:14:22 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2667740696; Wed, 4 Aug 2021 09:14:22 +0200 (CEST) Received: from CNSHJSMIN03.NOKIA-SBELL.COM (cnshjsmin03.app.nokia-sbell.com [116.246.26.71]) by mails.dpdk.org (Postfix) with ESMTP id 12F384014F for ; Wed, 4 Aug 2021 09:14:19 +0200 (CEST) X-AuditID: ac189297-2dfff70000021385-6e-610a3e49dfbf Received: from CNSHPPEXCH1605.nsn-intra.net (Unknown_Domain [135.251.51.105]) (using TLS with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by CNSHJSMIN03.NOKIA-SBELL.COM (Symantec Messaging Gateway) with SMTP id B1.04.04997.94E3A016; Wed, 4 Aug 2021 15:14:17 +0800 (HKT) Received: from CNSHPPEXCH1601.nsn-intra.net (135.251.51.101) by CNSHPPEXCH1605.nsn-intra.net (135.251.51.105) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Wed, 4 Aug 2021 15:14:16 +0800 Received: from CNSHPPEXCH1601.nsn-intra.net ([135.251.51.101]) by CNSHPPEXCH1601.nsn-intra.net ([135.251.51.101]) with mapi id 15.01.2176.012; Wed, 4 Aug 2021 15:14:16 +0800 From: "Yan, Xiaoping (NSB - CN/Hangzhou)" To: "users@dpdk.org" Thread-Topic: dpdk-pdump prints "EAL: Error: Invalid memory" Thread-Index: AdeI/DA4vQDCrUQrTEeqWKgRv8gPiQ== Date: Wed, 4 Aug 2021 07:14:16 +0000 Message-ID: Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.251.51.115] MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrDLMWRmVeSWpSXmKPExsXS/ts4U9fTjivRYO96PYvmg+tZHBg9fi1Y yhrAGMVlk5Kak1mWWqRvl8CV0bP7AlvBteWMFTtuOjYwnpvI2MXIySEhYCKx4f19ti5GLg4h gUNMEi3HjjBDOH8YJXZ2z2QFqRIS2Mgo8e2qOIjNJuAh8eLEVmYQW0RAVeLLrL0sILawgLnE viOz2SHiNhI/Nq1igrD1JJovrgerZxFQkbjRdx9sM6+AncTe6TPA4owCshLTHt0Hq2cWEJe4 9WQ+E8R1AhJL9pxnhrBFJV4+/gd0DweQrSTRtwGqPFVi+s+3bBAjBSVOznzCMoFRaBaSSbOQ lM1CUgYR15FYsPsTG4StLbFs4WtmGPvMgcdMyOILGNlXMUo7+wV7eAX7evoZGOv5+Xt7OuoG O7n6+Og5+/tuYgRGyBqJSdN3MJ6b+03vECMTB+MhRgkOZiUR3tAbHIlCvCmJlVWpRfnxRaU5 qcWHGKU5WJTEeVUMuBKFBNITS1KzU1MLUotgskwcnFINTB6rz5df6Pj2+2/9ro5aPYXDompK rxi3rt85bdYSfr6IYzO69tY2ylVkHeY7WXxkUat53/95nu92xcxXqvts7qu+4flKb71FOg0h hue5hFXLrj7ZLJGeP9XSLfqRrW7JFGGudSe2ZV7yuq3svnBGbuyFCB/e5p9BP6rPW37L/vTD f93VzMe6JT6P4rOFHKZd9t++3tb/0klOng1bvA7rZjxgF6hxmL7Qv3921tYbGVuUJe8VNe/d P+3wTq3iJvNHbbF63Es1rLS0t3NvDLzE+GZNqf2qBX+e8M6cmMLS8fHUx/edZybe63gg4PPB aUOb6by9rD/2K+wx4irK4X+1ZbH0juk9e2bbeqac5InXY1RiKc5INNRiLipOBABiNt8O/wIA AA== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: [dpdk-users] dpdk-pdump prints "EAL: Error: Invalid memory" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi, After updating dpdk version from 19.11 to 20.11 dpdk-pdump prints such error: EAL: Error: Invalid memory Port 7 MAC: 02 70 63 61 70 03 core (2), capture for (1) tuples - port 0 device ((null)) queue 65535 ^C --legacy-mem is used for both primary primary and dpdk-pdump. With some debug, I find that mlx5_mem_is_rte incorrectly consider this add= ress from os memory ((addr=3D0x4482b80)) as rte address, so mlx5_free calls rte_free() to free it and caused error. And this seems to because len of some unused memsegs is not set to 0 (so rt= e_mem_virt2memseg_list(0x4482b80) returns a memseg). Here is memsegs: (gdb) p mcfg->memsegs $3 =3D {{{base_va =3D 0x2aac0000000, addr_64 =3D 2932388921344}, page_sz = =3D 1073741824, socket_id =3D 0, version =3D 0, len =3D 34359738368, external =3D 0, heap =3D 1, memseg_= arr =3D { name =3D "memseg-1048576k-0-0", '\000' , count =3D = 5, len =3D 32, elt_sz =3D 48, data =3D 0x2aaa302e000, rwlock =3D {cnt =3D 0}}}, {{ba= se_va =3D 0x0, addr_64 =3D 0}, page_sz =3D 1073741824, socket_id =3D 0, version =3D 0, len =3D 3435973= 8368, external =3D 0, heap =3D 0, memseg_arr =3D {name =3D '\000' , count = =3D 0, len =3D 0, elt_sz =3D 0, data =3D 0x0, rwlock =3D {cnt =3D 0}}}, {{base_va =3D 0x0, addr_64 = =3D 0}, page_sz =3D 1073741824, socket_id =3D 1, version =3D 0, len =3D 34359738368, external =3D 0, he= ap =3D 0, memseg_arr =3D { name =3D '\000' , count =3D 0, len =3D 0, elt_sz = =3D 0, data =3D 0x0, rwlock =3D { cnt =3D 0}}}, {{base_va =3D 0x0, addr_64 =3D 0}, page_sz =3D 107374= 1824, socket_id =3D 1, version =3D 0, len =3D 34359738368, external =3D 0, heap =3D 0, memseg_= arr =3D { name =3D '\000' , count =3D 0, len =3D 0, elt_sz = =3D 0, data =3D 0x0, rwlock =3D { cnt =3D 0}}}, {{base_va =3D 0x0, addr_64 =3D 0}, page_sz =3D 0, soc= ket_id =3D 0, version =3D 0, len =3D 0, external =3D 0, heap =3D 0, memseg_arr =3D {name =3D '\000' = , count =3D 0, len =3D 0, elt_sz =3D 0, data =3D 0x0, rwlock =3D {cnt =3D 0}}} } Here is the stack trace (gdb) bt #0 mlx5_free (addr=3D0x4482b80) at ../dpdk-20.11/drivers/common/mlx5/mlx5_= malloc.c:260 #1 0x0000000000706f5c in mlx5_mp_req_verbs_cmd_fd (mp_id=3Dmp_id@entry=3D0= x7ffcdb6d9e50) at ../dpdk-20.11/drivers/common/mlx5/mlx5_common_mp.c:140 #2 0x000000000050496f in mlx5_dev_spawn (config=3D0x7ffcdb6d9d70, spawn=3D= 0x2ab753799c0, dpdk_dev=3D0x4491400) at ../dpdk-20.11/drivers/net/mlx5/linux/mlx5_os.c= :774 #3 mlx5_os_pci_probe (pci_drv=3D, pci_dev=3D= ) at ../dpdk-20.11/drivers/net/mlx5/linux/mlx5_os.c:2154 #4 0x0000000000708b5a in drivers_probe (user_classes=3D1, pci_dev=3D0x4491= 3f0, pci_drv=3D0xe01800 , dev=3D0x2ab75379a80) at ../dpdk-20.11/drivers/common/mlx5/mlx5_common_pci.c:246 #5 mlx5_common_pci_probe (pci_drv=3D0xe01800 , pci_dev=3D= 0x44913f0) at ../dpdk-20.11/drivers/common/mlx5/mlx5_common_pci.c:308 #6 0x00000000004268f9 in rte_pci_probe_one_driver (dev=3D0x44913f0, dr=3D0xe01800 ) at ../dpdk-20.11/drivers/bus/pci/pci_c= ommon.c:243 #7 pci_probe_all_drivers (dev=3D0x44913f0) at ../dpdk-20.11/drivers/bus/pc= i/pci_common.c:318 #8 pci_probe () at ../dpdk-20.11/drivers/bus/pci/pci_common.c:345 #9 0x00000000006bc4d3 in rte_bus_probe () at ../dpdk-20.11/lib/librte_eal/common/eal_common_bus.c:72 #10 0x0000000000422304 in rte_eal_init (argc=3Dargc@entry=3D16, argv=3Dargv= @entry=3D0x7ffcdb6da530) at ../dpdk-20.11/lib/librte_eal/linux/eal.c:1210 #11 0x000000000056fee9 in main (argc=3D16, argv=3D0x7ffcdb6da748) at ../dpdk-20.11/app/pdump/main.c:1118 (gdb) c Continuing. Thread 1 "dpdk-pdump" hit Breakpoint 5, rte_free (addr=3D0x4482b80) It seems to me that below code piece from eal_legacy_hugepage_init should a= lso set len to 0? /* we're not going to allocate more pages, so release VA spac= e for * unused memseg lists */ for (i =3D 0; i < RTE_MAX_MEMSEG_LISTS; i++) { struct rte_memseg_list *msl =3D &mcfg->memsegs= [i]; size_t mem_sz; /* skip inactive lists */ if (msl->base_va =3D=3D NULL) continue; /* skip lists where there is at least one page= allocated */ if (msl->memseg_arr.count > 0) continue; /* this is an unused list, deallocate it */ mem_sz =3D msl->len; munmap(msl->base_va, mem_sz); msl->base_va =3D NULL; // here, we should add msl->len =3D 0; ? msl->heap =3D 0; /* destroy backing fbarray */ rte_fbarray_destroy(&msl->memseg_arr); } Any comment? Thank you. Best regards Yan Xiaoping