From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2E59FA0C41; Wed, 17 Nov 2021 16:32:40 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9C51D40687; Wed, 17 Nov 2021 16:32:39 +0100 (CET) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2101.outbound.protection.outlook.com [40.107.244.101]) by mails.dpdk.org (Postfix) with ESMTP id DEB7B40040 for ; Wed, 17 Nov 2021 15:42:27 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HrwL0AansDYZFtLU0yEYXdIrgc5bzANUjYv4fROBDmO2AxpbAlyBIt2b8ole1b0lK4oibSxXN2h+/IPYA+QGZtusPtE8w2+Rcl+/6cguOvMxDLNJ4ZT00F7WF2P9HG0XxwSIOL5qSjDCq4zTDBsdoI3gmIT+B5UD4x2FeFN+vhOzQp3UZOBrGX70TgT2ENQ9FzD2aA/sqFKsP0uRRDl5MK/WvWjVdSnaBe0vYYG73qoHwdaAPMo8g5vz+wQG+Y3IZpleUhfb7hKdK1OB3X9SUSCC9t5JM4LDTmprmFQySlElEjP6m1YVdHwQFP85PcgtOKkEGiI9ymZHa1Ks5ytEFA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=aHCoePNpioZjfNfXoKNLf6c1n2qsB+ZB0Paz9mJNpeM=; b=oBcJP+8WWRum9e/NuuVoS9j6nqOFGQXcmwJ4bgue+vTM8Y2WJdCehKreKm6G3sDHsEmVm+atBWY/WwU95f+/90sSkQ6yG57HE/DWfTZQxI6zK4RujPJwEdD3zU/ac0QQiT5cvxwweOsqf5mhyY8hbN4Bk5uAFJCuYfLOea94Gl6DbVG2udFsJWtFKjjF3MCddFBdR8s7IC84aiSB5fxe7ZDWsy2fYEbBCaW7+oHa/OyOdTffY6qM3On61Sgw8nLQNywqzBAyjvVRh+tFpc06oEU78Ll82maIUOdbu4FU6FYnacY4vhTIAMAoUWQCB20HVkCrFy9rRS1B4cGajZE+dw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nokia.com; dmarc=pass action=none header.from=nokia.com; dkim=pass header.d=nokia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nokia.onmicrosoft.com; s=selector1-nokia-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=aHCoePNpioZjfNfXoKNLf6c1n2qsB+ZB0Paz9mJNpeM=; b=bOwyxlBrAEeMoqCJxsy3KfjwOGAL4NabxQPp0qb0WtWsPC90CCRIc8NrguOH9S24epFURy3Ui8nEG3lwrvfo+yLaYiFR7umC1sqINpZ44g1a5bgAwtKOO8IosObQKp9rt+r/5uDOtr/fckbSTH+lCBqXR7oUyclBRNR4Fq0ZS2I= Received: from SA1PR08MB7133.namprd08.prod.outlook.com (2603:10b6:806:187::8) by SN4PR0801MB3759.namprd08.prod.outlook.com (2603:10b6:803:4d::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4690.27; Wed, 17 Nov 2021 14:42:24 +0000 Received: from SA1PR08MB7133.namprd08.prod.outlook.com ([fe80::f478:be2f:9aa2:88b9]) by SA1PR08MB7133.namprd08.prod.outlook.com ([fe80::f478:be2f:9aa2:88b9%2]) with mapi id 15.20.4669.016; Wed, 17 Nov 2021 14:42:24 +0000 From: "Bendror, Eran (Nokia - US)" To: "ktraynor@redhat.com" CC: "ayeh@cisco.com" , "chenbo.xia@intel.com" , "dev@dpdk.org" , "ian.stokes@intel.com" , "maxime.coquelin@redhat.com" , "yega@cisco.com" Subject: Re: [dpdk-dev] [ovs-dev] ovs-vswitchd with DPDK crashed when guest VM restarts network service Thread-Topic: Re: [dpdk-dev] [ovs-dev] ovs-vswitchd with DPDK crashed when guest VM restarts network service Thread-Index: AdfbwCZ8VehLonaZTjivER47b5OHEQ== Date: Wed, 17 Nov 2021 14:42:24 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nokia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 52b9ac9c-3ae0-405d-b3fa-08d9a9d87873 x-ms-traffictypediagnostic: SN4PR0801MB3759: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Jju5vuPDCw+XXfnEszJCWdErTwd8BZ+jtnJgsvTEtxL9hXf00NtllAk/26xspdbuKVVF+BwMntft5eDZL8DmuGjhdvBX8q/Z6rglp+XX86BAY8qV/6/J/mP+VUMvMJgHJlgj5sbi13wRl806Iau52spcbv9UWqVkJPGIxH8/w6zuVZqQvNF1h49fkbVH+aZouR4nnLe34BHzlRHP6KAFOhsn5KAiKTnBbt2sF1r1wHjAHtZoNW0kiW5PUZoQ1UIM54HeEZg+ng4ju/Cx+yd1dT2QYiwbxZQ+vvRCvBv6Q0fwNR6DoT4SQkkpqA04ElqU8H/xPPv+HCdqwYeT2HIHdeD/873RlJ+CNxHtx9ne6AIqSdxPmRTeeBoNaFfhX2XvAbNKtd20E+lKdIT2YatGqcaL1R0WijDO/c3Ym6M/F25/5T+3LaDlBceGyYr+hiXPvzKwiWRpBH3ZC/Z9OD9gxBRWAXIrbUS0vHR+QPXkfdydlmCuRb+OH/+DgLeKQxF9F9Ez/ffhzwdR38chXtpMcywwt9Oq/dy6eOxaS6FMtjEmrkjDwFyKk7byYKjAFgtP3hLgMgdZL5wZJyDpS5efsDf6K+esYFuBniQrHNCeS4K6rVutbrB1QINyvHxLdJa6mlzanRJJBuW2+VR87UZFdBKmAMvJ4iOSQMwZwwrwBz4MFBiEUJ4CEGCIXW2fJKEGl3zzFSfLUsIieGOxx6rdcA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SA1PR08MB7133.namprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(38100700002)(66556008)(2906002)(66476007)(86362001)(38070700005)(66946007)(71200400001)(26005)(4326008)(33656002)(122000001)(76116006)(8676002)(64756008)(508600001)(82960400001)(55016002)(186003)(66446008)(6506007)(9326002)(6916009)(316002)(54906003)(8936002)(5660300002)(7696005)(52536014)(83380400001)(9686003); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?4g/nJLN1+XdwftXExrA8W1fMNpaCqup1yngCKrl9avHINtEuwCPB3W8lNuRU?= =?us-ascii?Q?NG0s+LcnXFRlSaWGexiMrms0IPI/G1+7R6hTMQLO0tg8wDgO879N4+tS4rsZ?= =?us-ascii?Q?OUZAFvVtlX7oSHAKtcfJLFdjG5COOMj24g/FtP3SWw3NSlVFrEpgdny8mVHw?= =?us-ascii?Q?hWCivpPeCo0/r+CZdVviQ/x860h72pLfq0el2b2ZbDRps8GaqotltvNBub5H?= =?us-ascii?Q?lrMnEYdN1sH/S8XMtuSRs+RyYBRhpGJv+QFyGaiU7WPpCu5ci0dwOCGZQZRv?= =?us-ascii?Q?QRvYXVjNG+Oti6nm7frIQQ272hKxUA5tMX+/vPrbOUcwPVdFF8o9+LzgoP1w?= =?us-ascii?Q?u/vfWdk3AGahI6Ea8rVTkQg1cYza+nx/C1vpfKZvV/URg1gx7vl50U94No4T?= =?us-ascii?Q?EufgshNdE4jE2b+uaAIzqnbt2U8hRqsAa95I5MkLUZEuX3Fbu+7YI0Z47KjZ?= =?us-ascii?Q?AFhzOfRhYoAn+szqNWuB/GPZJAdS6aonmxzaRC+0RkFtRKaWb4/hszAllXj1?= =?us-ascii?Q?aeWp8rXQhNVsS/w8dLjhjz3SwYptP5Licf9oTyRGv4o19Fr2Grxl3skKlVGB?= =?us-ascii?Q?3PJXBYkFbkupd/3E1NYydRVpB0qw9XtgIW4fU0zsYwB6Kc7Oi7EYfIId6eMM?= =?us-ascii?Q?MvAUKlU8DqKxK3JEm4EJeJtC1otoQO+6cudsOKmH9ztT1C3TdVVaGBAxIkyv?= =?us-ascii?Q?mDQMmCr1VMtqU7vWQZiQN99JmcIftrVMGqKq4whDQ6rHhEnjBd74cCT9m4l+?= =?us-ascii?Q?aeS8mukM/89vDziHvXhIVhCt5Oho1UdWXiw6mkTLMYlUYUrlHPdjZ+fgUXj2?= =?us-ascii?Q?aJ0x3mISuElqulLXk/LoXPH3s4UDnVXybi2q7ndOAO/Ux/gDHIXRDAVjTs0H?= =?us-ascii?Q?zlf5aGWWGb68uVCBTu5AUXBrgEzjiE5JL7VSMxmbrG6mOoJ6oOxtCA+iQQ/s?= =?us-ascii?Q?MvluZeH7YJqDhvrSD5UWaiSVeRtU3OI3AilO2vianLah926D2pXeVgA4CdMJ?= =?us-ascii?Q?EoKpmLx/pfyEf87QO2hFHy42yyleD2gW7jCKMrGhmpaI3/zT1P2mq9PqGx5N?= =?us-ascii?Q?PVishh3Rkfwxpskb+g16VTZHSoBgOsEh3Vzg12McI7Qbj87aCBaLFJ2MEgFx?= =?us-ascii?Q?DbzRsC2Oxh88gCqm6flg4wd6VAZMJmPY4yr3ilWFcP/t8f1odz1jWJ3u+Zql?= =?us-ascii?Q?vkELNg9wTDTHvI/SZCQ6tN2vUo+bwQYRxQBqKDQkNexk8ZcrbutnVkUReF9l?= =?us-ascii?Q?KjEWbvUJfUV74p3S/+YE112UYQJARC4mm0soLLat4rGtPYajcvDraLM0jnBK?= =?us-ascii?Q?7OyPR66SorysK9ZLRWWz01TzOjSe0nQWLkwgoywshSey3dSaIniqimYsbXzP?= =?us-ascii?Q?mmxmjeGJBYsHC172hE5Qug6o1M+ffcBZjhDpuhQNmFtzfeBSiSd7EnTs7O9h?= =?us-ascii?Q?zdvVpviOHq0AN3pvXTWMcMyrpMndIxGtj/KvcBgqpFwMo13G7vtex3soCSO0?= =?us-ascii?Q?N1R8FZMugS0x+4AKpLL7MjfRZv5COKTMRHCe8EXzZ8jBJUE2y/Qf9eL539KB?= =?us-ascii?Q?sa41BGDRU3M7HC/Kvjs876bQolTpIlHe4qCoyHFot6QRuUSCm9i9Q1QhMHz+?= =?us-ascii?Q?WD54Yxq9lGVplM+fpvTPWPI=3D?= Content-Type: multipart/alternative; boundary="_000_SA1PR08MB713373B0D19329C38C7527BB839A9SA1PR08MB7133namp_" MIME-Version: 1.0 X-OriginatorOrg: nokia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SA1PR08MB7133.namprd08.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 52b9ac9c-3ae0-405d-b3fa-08d9a9d87873 X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Nov 2021 14:42:24.6483 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 5d471751-9675-428d-917b-70f44f9630b0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 0rKxm5WOlV1aqdvZoowERGZQVexqK/P6oVgn1WAxn2PnUaHjolA87K3VeraN54tjT0B7zZ5F6LgUr+oYi0dOEg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN4PR0801MB3759 X-Mailman-Approved-At: Wed, 17 Nov 2021 16:32:38 +0100 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --_000_SA1PR08MB713373B0D19329C38C7527BB839A9SA1PR08MB7133namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello, I am wondering if there was any progress in this topic, we are seeing a ver= y similar issue, where a VM level application restart triggers segmentation= fault and failed to allocate MBuf on the host level CentOS Linux release 7.8.2003 (Core) dpdk-18.11.5-1.el7_8.x86_64 openvswitch-2.11.0-4.el7.x86_64 libvirt 4.5.0 QEMU 4.5.0 (API) QEMU 2.12.0 3.10.0-1127.13.1.el7.x86_64 And we get the same crash #0 0x00007f96cb72e7ee in rte_memcpy_generic () from /lib64/librte_vhost.so= .4 #1 0x00007f96cb7350f2 in rte_vhost_dequeue_burst () from /lib64/librte_vho= st.so.4 #2 0x00007f96caf97f03 in netdev_dpdk_vhost_rxq_recv () from /lib64/libopen= vswitch-2.11.so.0 #3 0x00007f96caed21e6 in netdev_rxq_recv () from /lib64/libopenvswitch-2.1= 1.so.0 #4 0x00007f96caea07ca in dp_netdev_process_rxq_port () from /lib64/libopen= vswitch-2.11.so.0 #5 0x00007f96caea0ca5 in pmd_thread_main () from /lib64/libopenvswitch-2.1= 1.so.0 #6 0x00007f96caf2da3f in ovsthread_wrapper () from /lib64/libopenvswitch-2= .11.so.0 #7 0x00007f96c9ef3ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f96c94118dd in clone () from /lib64/libc.so.6 We have tried upgrading host level artifacts: dpdk-20.11.3-1.el7.x86_64 openvswitch-2.16.1-1.el7.x86_64 With backtrace: #0 0x00007f6b8b49748c in virtio_dev_tx_split_legacy () from /lib64/librte_= vhost.so.21 #1 0x00007f6b8b4c0fdb in rte_vhost_dequeue_burst () from /lib64/librte_vho= st.so.21 #2 0x000055bd714c2802 in netdev_dpdk_vhost_rxq_recv () #3 0x000055bd713f8e51 in netdev_rxq_recv () #4 0x000055bd713c9d2a in dp_netdev_process_rxq_port () #5 0x000055bd713ca1f9 in pmd_thread_main () #6 0x000055bd71455cdf in ovsthread_wrapper () #7 0x00007f6b8a6a9ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6b89bc78dd in clone () from /lib64/libc.so.6 Regards, Eran --_000_SA1PR08MB713373B0D19329C38C7527BB839A9SA1PR08MB7133namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hello,

 

I am wondering if there was any progress in this top= ic, we are seeing a very similar issue, where a VM level application restar= t triggers segmentation fault and failed to allocate MBuf on the host level=

 

CentOS Linux release 7.8.2003 (Core)

dpdk-18.11.5-1.el7_8.x86_64

openvswitch-2.11.0-4.el7.x86_64

libvirt 4.5.0

QEMU 4.5.0 (API)

QEMU 2.12.0

3.10.0-1127.13.1.el7.x86_64

 

And we get the same crash

 

#0  0x00007f96cb72e7ee in rte_memcpy_generic ()= from /lib64/librte_vhost.so.4

#1  0x00007f96cb7350f2 in rte_vhost_dequeue_bur= st () from /lib64/librte_vhost.so.4

#2  0x00007f96caf97f03 in netdev_dpdk_vhost_rxq= _recv () from /lib64/libopenvswitch-2.11.so.0

#3  0x00007f96caed21e6 in netdev_rxq_recv () fr= om /lib64/libopenvswitch-2.11.so.0

#4  0x00007f96caea07ca in dp_netdev_process_rxq= _port () from /lib64/libopenvswitch-2.11.so.0

#5  0x00007f96caea0ca5 in pmd_thread_main () fr= om /lib64/libopenvswitch-2.11.so.0

#6  0x00007f96caf2da3f in ovsthread_wrapper () = from /lib64/libopenvswitch-2.11.so.0

#7  0x00007f96c9ef3ea5 in start_thread () from = /lib64/libpthread.so.0

#8  0x00007f96c94118dd in clone () from /lib64/= libc.so.6

 

We have tried upgrading host level artifacts:

 

dpdk-20.11.3-1.el7.x86_64

openvswitch-2.16.1-1.el7.x86_64

 

With backtrace:

 

#0  0x00007f6b8b49748c in virtio_dev_tx_split_l= egacy () from /lib64/librte_vhost.so.21

#1  0x00007f6b8b4c0fdb in rte_vhost_dequeue_bur= st () from /lib64/librte_vhost.so.21

#2  0x000055bd714c2802 in netdev_dpdk_vhost_rxq= _recv ()

#3  0x000055bd713f8e51 in netdev_rxq_recv ()

#4  0x000055bd713c9d2a in dp_netdev_process_rxq= _port ()

#5  0x000055bd713ca1f9 in pmd_thread_main ()

#6  0x000055bd71455cdf in ovsthread_wrapper ()<= o:p>

#7  0x00007f6b8a6a9ea5 in start_thread () from = /lib64/libpthread.so.0

#8  0x00007f6b89bc78dd in clone () from /lib64/= libc.so.6

 

Regards,

Eran

 

--_000_SA1PR08MB713373B0D19329C38C7527BB839A9SA1PR08MB7133namp_--