From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9EC9CA034E; Mon, 21 Feb 2022 15:19:58 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6AA884068C; Mon, 21 Feb 2022 15:19:58 +0100 (CET) Received: from de-smtp-delivery-102.mimecast.com (de-smtp-delivery-102.mimecast.com [194.104.111.102]) by mails.dpdk.org (Postfix) with ESMTP id 6BD6C4013F for ; Mon, 21 Feb 2022 15:19:57 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=mimecast20200619; t=1645453197; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UmPzqH8/DToZb5bUoJaC3msxZt6+lrwYtVNfPVbYrIA=; b=kVP56/lFPyJ4NK1sap7HeI/HOg+gJqi6YtqhP8GQzowHLGzFkXL3dFb06GYv11BHfeHpFM GkCrgvYMKFN0QXHqrf6Y8pmd4NkBj7LX0/IaDbDkrrym6grf0ZoGmuQoJtX7NCHRDU08jL TRTsAdF7hPnUPFAqz/B+NfpxjjK3aYw= Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05lp2105.outbound.protection.outlook.com [104.47.17.105]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id de-mta-30-o9yCpJvLNAWDr48GBLTnBg-1; Mon, 21 Feb 2022 15:19:55 +0100 X-MC-Unique: o9yCpJvLNAWDr48GBLTnBg-1 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FeM5RydOC0+2ZlgpF/JHHX0PBuMz/3lOMohKFkVeXJQOS7sYmJ+4nCvjD5AoY9JXUT+0A2JkLQLvNFp6SGzkjOb5cPzDnBqEuiCghCuF3pR2ndXKd/V3T1OPYyyaXVNAfJ08jb/qRev15WgOyM6RS5sqoLiCyZtzaCRF3DY2Cbqm5thYgvWZQxMdRGVfYxpHwf0zaW4xH8VzBtYrKDTtdi8/pYTXbapBYdTKgdQIGxhZuxBjtm7Ol0SzKlevMbxwoSZiJF63yf913ZPnWCZy+rNmhQsQdc6Ty4ou8K4E+bhd2+6ZIlSe3/VO60wgHt/MUsGLjMHaKJUXgemcOMItRQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Lce08Rs33rf7GBH7rEoQ4lX55jPWautyM116sPtKF9o=; b=nepCwFFSL/tyMHTWFjc37Y5HlAGdKAbJBTeZ2hw+Vo/bKvNfbkBG7FBN9HHyXOEP/W+WsTexoB+EVYXoCrBRRm81/UFplnTduu0/7XK1frcbXE8K1v13kdPe4qLsIuEaal3VRQvLZfQY+WR6uIYJ0xtpjxpLTYAr4x5D3ATbb2dPyJF+ih175zQu6pH3d+Xtk07gOAt+O7MYAtdISsSwk3doGBtS7A2q3fC5ywhZdFjvt88eSGjpOzgJlQYWvUeWF+TCFRetWaxKVcV4lucbXRRhIasDEBBetyZbe3QRgH5z6zO342LDv3M8jbG9g+mIRHhxJw2uUzBDbne9wjMkOw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=suse.com; dmarc=pass action=none header.from=suse.com; dkim=pass header.d=suse.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=suse.com; Received: from VI1PR04MB3085.eurprd04.prod.outlook.com (2603:10a6:802:6::17) by DB7PR04MB4331.eurprd04.prod.outlook.com (2603:10a6:5:25::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4995.27; Mon, 21 Feb 2022 14:19:53 +0000 Received: from VI1PR04MB3085.eurprd04.prod.outlook.com ([fe80::b00a:7157:4eae:567a]) by VI1PR04MB3085.eurprd04.prod.outlook.com ([fe80::b00a:7157:4eae:567a%3]) with mapi id 15.20.4975.018; Mon, 21 Feb 2022 14:19:53 +0000 Message-ID: <85935ffc-aa30-6f08-df02-94dccf394872@suse.com> Date: Mon, 21 Feb 2022 15:19:50 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.5.1 Subject: Re: [dpdk-dev] [ovs-dev] ovs-vswitchd with DPDK crashed when guest VM restarts network service Content-Language: en-US To: "Xia, Chenbo" , "ktraynor@redhat.com" , "maxime.coquelin@redhat.com" , "dev@dpdk.org" CC: "ayeh@cisco.com" , "Stokes, Ian" , "yega@cisco.com" , "Bendror, Eran (Nokia - US)" References: From: Marco Varlese In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: ZR0P278CA0158.CHEP278.PROD.OUTLOOK.COM (2603:10a6:910:41::17) To VI1PR04MB3085.eurprd04.prod.outlook.com (2603:10a6:802:6::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 51c2c6c4-5a61-484d-e8ac-08d9f5453a5e X-MS-TrafficTypeDiagnostic: DB7PR04MB4331:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: DJPHnPKr5dIsjbpVU0KTQ5G1MX8/pejvE+HYYd9qi1lLtY43VXCH195AaXVExwQp1k4KNZRmju2aGIEdV0w5PD0ZI6/0hpg4Ovp/9BEV1ihYndoI6ZHCS7PA/f0WUX3KERkoZ4C2+vqLgipUcpuUlrrBrjmG2OuelSvuDzdPwe8lw8UeIr0Mg0lGEb9RQKCffUBUk7nKh1e3f6s00wAfY39ywS0IBEdMEx5MyfFY/TxGe2Wwlx7/eLGtEAm2kJvzbOeEpp/WyGK+H23jvYrzE3rc+Sj6vcOZSDyiTsa41Q8MdctjXkIaVlfIapXFzArW3JAE4U2OSGjaaHL2T0mZSMxq1WdqSMFa7/dp3z11HztaIOmBDkQMDmZzjTdKF3U98UCMxWw0xUR/A5kNJIorQSqN1l2yPZ0W+LDj2cfjMNK+xRUpM7GGsNB/M/fPk5an2hyLh7FA9fcmHSKsnQASFsv0Jl4VoBLF6w4k0qtAXtNAbnSXCV05VWjWzyz/g0xTCJbLsXy+dLHu7ZICuMELfue2tDXiB3cN9+FiDncWM1hgpvxgjYRbcMnu8sBx9WGzuRbOpQsS/Ir6FO/NqV4HJGPA6N5BnIVCjD8U1tSZUBr3ylXRgA3osjVlRU6DOzsKGqI6XrMUtoI3lke+srbuo4DU08epmUVxIVurdmupYWiRB8TFoVoCt0N4X0IQUQeS4+AWA+B7p7zmdQJrtuDbZcUFVB5CA1sNQlkqjD/zqXw= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:VI1PR04MB3085.eurprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(5660300002)(66476007)(44832011)(26005)(8936002)(2616005)(6486002)(186003)(6512007)(54906003)(6506007)(110136005)(38100700002)(316002)(508600001)(66946007)(66556008)(8676002)(4326008)(86362001)(31696002)(53546011)(36756003)(2906002)(31686004)(83380400001)(43740500002)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?4ZMHoy/X/Vo+onJSghTHvNT1Kax530hhbsuSWrPkwRH4VpU7vNBsQUMHoagX?= =?us-ascii?Q?IWDQCek6SYCMfnq01ePsrfAXajQdQQXBDS9buwcVDRNiDyeuJZQddd1Dew3W?= =?us-ascii?Q?d6q0Pa8AxQNoXAUaq0hIFgLp0CL84AbQdM4ZKNVCFFQNCVceSqCLCLGZ2PdL?= =?us-ascii?Q?73SFDjDEl3O88z7ZEUsQlLJPJbTZcjDffOWov0WvEUPRVSXnOlEeoFAhGxEX?= =?us-ascii?Q?u7Hg3OvpszktK7G1uD56aQovvZjD1i3NexkEm3ZokmsteRUoQKUpkWUAvUcI?= =?us-ascii?Q?orrh0IyW3NKRnKzqWcY8dZ8V/zNe2LWDU9JbuNXl1Y0lmzvfP97WHfyei/0j?= =?us-ascii?Q?D7UKmVJ5Sqx5ueDnvEb0KdnFbE3fCuvCfbkhYDTt7SRx6nAWBbx5YhJ80Sgt?= =?us-ascii?Q?bbabHV31mi/t7tsgg71L7pAfJVasVkmkUoltcd1hIxGt5r6DMKsCo5UqTvXI?= =?us-ascii?Q?gCerU0GYmJ4dMsnD1Sm+WrLy1qXgEqPfZDbIqkLAAxSmLGtA+uOX2EI4lP8n?= =?us-ascii?Q?r45kCPg5jd8KbHgj5UyLfpq4OwL4mVAzlgpiG3HyaL+ct8YbG8GraPUYWuGV?= =?us-ascii?Q?sDimWHSuD+bpzf9EWejbLxDosblAihHSm3oHNVKdJ8Pe9kIsB0iPtBMqypUl?= =?us-ascii?Q?t0z9V+vM7lFEMm2DsxQtJZ8dJnE5C59OEdrIGKcZakVLLSBa7WZskezEjptS?= =?us-ascii?Q?GSnPX1AY7ztePenkXW6ejDolJymCdBlG35GdkTEEfcoqUCBfrS9gQ89+ek8T?= =?us-ascii?Q?/VeDZACaC3rfgZWEr0ol4KhFs27mfi/40t0zkG/V0UjPPbSMVleQDhT1cMla?= =?us-ascii?Q?DwMQg8HpDcoU5KAWkJysAIHsTzA/l3ZL0i3Z+OGm/XjLxvq/22d30XodswPb?= =?us-ascii?Q?cG1e4zEn1Xjw+qpTNfxu0b4RqK7Afa2TMYQkedWMVJvNukZoGqBmhh8Aj7oN?= =?us-ascii?Q?M5laWFpt4DqDBk0l1M7OK13j9rrMF9M1YX27FmEOqoqV6BRNisDspWyV6qk+?= =?us-ascii?Q?55IiGmS45z5Hk5YIA4McQ4DwXIFBMlbz+PHfty+ax3Osf43mrIACuUy8Ou17?= =?us-ascii?Q?jCR5mIbi4bFAUvyby0OAXE88UGSSt/S754KoX8Yo4fMqZYJmtERVbH6BG4Do?= =?us-ascii?Q?bocoa17FayArVruJjfqAGK4SJw3mdQ/HNgaYKEosKwegJfi4gcWOLavkVCpV?= =?us-ascii?Q?Qa+hCkEsTlAfl/YuLw3SQHDyClLqIcdnJ4oLWO0/xPm+FpL0sDkxGpd2FD4r?= =?us-ascii?Q?Kw5wq4ylTwVrQtTc+lXqm2WsdaV/YxaS//wQb8KqL4nSCTJInahYkWRdKvtD?= =?us-ascii?Q?v7pbooEFXAIrUKo/KS7ifR1XBUWA2/3NsFEgtZAk4Tye5UTPBepKuR3nhB0t?= =?us-ascii?Q?jQOtIXoO7j7uTmJTl67nqSLwyp5WrDFzEFxiesACYCS1KpyesG2MQQ7SIkF5?= =?us-ascii?Q?uhC9Z2dqYcuucZkYdD5LU35Q1uh33I68/cgt8cslo4D0HTxUHFOuxGMOF2+c?= =?us-ascii?Q?ZAnAiP2ZplQNu2Mkbnk4dh6gHV6bC5ap+bVUfEvr/u9Ah6j3hspYHYWvm6Pi?= =?us-ascii?Q?g6kpUQGFkR0B8gUeBzTliBDe2XnZe31HyPorDbPFb4ISqNJ6e8/HNt6hEKEk?= =?us-ascii?Q?I1XGdjrQ80g4MygbeKJ2Qi4=3D?= X-OriginatorOrg: suse.com X-MS-Exchange-CrossTenant-Network-Message-Id: 51c2c6c4-5a61-484d-e8ac-08d9f5453a5e X-MS-Exchange-CrossTenant-AuthSource: VI1PR04MB3085.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Feb 2022 14:19:53.2670 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: f7a17af6-1c5c-4a36-aa8b-f5be247aa4ba X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wLZVzsw2EArbMTvI78bYMrE8RTaP39tjME9ze2hWrdzqRVlUpbKSJDth2ZeSUPhBP2eMaQ5J3MODE3QwIcG2Pw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB7PR04MB4331 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hello, I have been seeing the same issue with several different DPDK-OVS=20 versions as well as QEMU versions. It looks like an issue with handling the VHOST_USER_GET_VRING_BASE once=20 the application in the guest is restarted. It might probably have to do=20 with QEMU asynchronous message passing... I am not an expert on the vhost/virtio so trying to have your help with=20 this. Has anybody had the chance to look into this issue and found a=20 solution or workaround? Cheers, Marco On 11/26/21 15:09, Bendror, Eran (Nokia - US) wrote: > Hi, >=20 > Internally the VM is using DPDK 17.05, on Centos7.9 =E2=80=93 but this se= ems to=20 > be reproducing with guest level 18.11 as well. >=20 > The issue is when the DPDK PMDs get started at guest, so the assumption=20 > is that that presents bad / inaccessible memory towards the host. >=20 > We did notice some mis-use at the guest of selinux permissions, and=20 > removing that helped reducing the frequency significantly. >=20 > Is there a way to map the shared memory between VM and host to see where= =20 > is the segmentation fault coming from? >=20 > I will see if I can upload the VM xml, but it is a multi-queue 4 port VM. >=20 > Thanks for the assistance, >=20 > Eran >=20 > *From:* Xia, Chenbo > *Sent:* Friday, November 26, 2021 4:25 AM > *To:* Bendror, Eran (Nokia - US) ;=20 > ktraynor@redhat.com > *Cc:* ayeh@cisco.com; dev@dpdk.org; Stokes, Ian ;=20 > maxime.coquelin@redhat.com; yega@cisco.com; Marco Varlese=20 > > *Subject:* RE: [dpdk-dev] [ovs-dev] ovs-vswitchd with DPDK crashed when=20 > guest VM restarts network service >=20 > Hi, >=20 > Is it possible that you can provide more info about this isuee. I mean:=20 > qemu cmdline/libvirt xml, ovs cmdline, guest driver version and etc=E2=80= =A6 Or=20 > it=E2=80=99s hard to reproduce the issue. >=20 > Thanks, >=20 > Chenbo >=20 > *From:* Bendror, Eran (Nokia - US) > > *Sent:* Wednesday, November 17, 2021 10:42 PM > *To:* ktraynor@redhat.com > *Cc:* ayeh@cisco.com ; Xia, Chenbo=20 > >; dev@dpdk.org=20 > ; Stokes, Ian >; maxime.coquelin@redhat.com=20 > ; yega@cisco.com > *Subject:* Re: [dpdk-dev] [ovs-dev] ovs-vswitchd with DPDK crashed when=20 > guest VM restarts network service >=20 > Hello, >=20 > I am wondering if there was any progress in this topic, we are seeing a=20 > very similar issue, where a VM level application restart triggers=20 > segmentation fault and failed to allocate MBuf on the host level >=20 > CentOS Linux release 7.8.2003 (Core) >=20 > dpdk-18.11.5-1.el7_8.x86_64 >=20 > openvswitch-2.11.0-4.el7.x86_64 >=20 > libvirt 4.5.0 >=20 > QEMU 4.5.0 (API) >=20 > QEMU 2.12.0 >=20 > 3.10.0-1127.13.1.el7.x86_64 >=20 > And we get the same crash >=20 > #0=C2=A0 0x00007f96cb72e7ee in rte_memcpy_generic () from=20 > /lib64/librte_vhost.so.4 >=20 > #1=C2=A0 0x00007f96cb7350f2 in rte_vhost_dequeue_burst () from=20 > /lib64/librte_vhost.so.4 >=20 > #2=C2=A0 0x00007f96caf97f03 in netdev_dpdk_vhost_rxq_recv () from=20 > /lib64/libopenvswitch-2.11.so.0 >=20 > #3=C2=A0 0x00007f96caed21e6 in netdev_rxq_recv () from=20 > /lib64/libopenvswitch-2.11.so.0 >=20 > #4=C2=A0 0x00007f96caea07ca in dp_netdev_process_rxq_port () from=20 > /lib64/libopenvswitch-2.11.so.0 >=20 > #5=C2=A0 0x00007f96caea0ca5 in pmd_thread_main () from=20 > /lib64/libopenvswitch-2.11.so.0 >=20 > #6=C2=A0 0x00007f96caf2da3f in ovsthread_wrapper () from=20 > /lib64/libopenvswitch-2.11.so.0 >=20 > #7=C2=A0 0x00007f96c9ef3ea5 in start_thread () from /lib64/libpthread.so.= 0 >=20 > #8=C2=A0 0x00007f96c94118dd in clone () from /lib64/libc.so.6 >=20 > We have tried upgrading host level artifacts: >=20 > dpdk-20.11.3-1.el7.x86_64 >=20 > openvswitch-2.16.1-1.el7.x86_64 >=20 > With backtrace: >=20 > #0=C2=A0 0x00007f6b8b49748c in virtio_dev_tx_split_legacy () from=20 > /lib64/librte_vhost.so.21 >=20 > #1=C2=A0 0x00007f6b8b4c0fdb in rte_vhost_dequeue_burst () from=20 > /lib64/librte_vhost.so.21 >=20 > #2=C2=A0 0x000055bd714c2802 in netdev_dpdk_vhost_rxq_recv () >=20 > #3=C2=A0 0x000055bd713f8e51 in netdev_rxq_recv () >=20 > #4=C2=A0 0x000055bd713c9d2a in dp_netdev_process_rxq_port () >=20 > #5=C2=A0 0x000055bd713ca1f9 in pmd_thread_main () >=20 > #6=C2=A0 0x000055bd71455cdf in ovsthread_wrapper () >=20 > #7=C2=A0 0x00007f6b8a6a9ea5 in start_thread () from /lib64/libpthread.so.= 0 >=20 > #8=C2=A0 0x00007f6b89bc78dd in clone () from /lib64/libc.so.6 >=20 > Regards, >=20 > Eran >=20