From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from sesbmg22.ericsson.net (sesbmg22.ericsson.net [193.180.251.48]) by dpdk.org (Postfix) with ESMTP id DC6722BDA for ; Thu, 17 Mar 2016 07:37:17 +0100 (CET) X-AuditID: c1b4fb30-f79246d00000788a-8e-56ea509cfdfb Received: from ESESSHC020.ericsson.se (Unknown_Domain [153.88.183.78]) by sesbmg22.ericsson.net (Symantec Mail Security) with SMTP id FA.7C.30858.C905AE65; Thu, 17 Mar 2016 07:37:17 +0100 (CET) Received: from [147.214.67.148] (153.88.183.153) by smtp.internal.ericsson.com (153.88.183.80) with Microsoft SMTP Server id 14.3.248.2; Thu, 17 Mar 2016 07:37:16 +0100 To: "Xie, Huawei" , "dev@dpdk.org" References: <56E956F5.6080606@ericsson.com> From: Patrik Andersson R Organization: Ericsson AB Message-ID: <56EA509C.8080905@ericsson.com> Date: Thu, 17 Mar 2016 07:37:16 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrPLMWRmVeSWpSXmKPExsUyM2K7n+7cgFdhBk1bmC3efdrOZNE+8yyT A5PHrwVLWT0W73nJFMAUxWWTkpqTWZZapG+XwJWxZ/IstoIL8hUfOyaxNTCuEe9i5OSQEDCR uDh9KiOELSZx4d56ti5GLg4hgcOMEmdv/mOGcNYwSjxf38QCUiUskC9xaesZMFtEwF1i0sE5 YLaQQKnEua+fwSaxCVhJzNu2jAnE5heQlNjQsJsZxOYV0JbY1N8IVsMioCqxrPcuWI2oQITE k7knGSFqBCVOznwCNpNTIETi1bYfQL0cHMwC9hIPtpaBhJkF5CW2v53DDLFWR+LVmbdsExgF ZyHpnoXQMQtJxwJG5lWMosWpxUm56UZGeqlFmcnFxfl5enmpJZsYgcF6cMtvgx2ML587HmIU 4GBU4uEtePoyTIg1say4MvcQowQHs5IIb4nwqzAh3pTEyqrUovz4otKc1OJDjNIcLErivKyf LocJCaQnlqRmp6YWpBbBZJk4OKUaGHfsfL7sdY7ZVMudOlWdmwR3edwo52/+Y2NQ4rTaaoPL x+gV0cJdVVvEXaa3aqQkbRX7cfI4z7/Hb9kPvmTt+8H04ORRj8lPTmq5peqe3NqbErpgtX6L bU2t7Dz5KK0JU/XaF60xmtB64K/PpWfr7vi8eM/nH9K2YNGmuY///pnamOXjsE7cy1iJpTgj 0VCLuag4EQCZq6juUgIAAA== Subject: Re: [dpdk-dev] vhost: no protection against malformed queue descriptors in rte_vhost_dequeue_burst() X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Mar 2016 06:37:18 -0000 Hi Huawei, thank you for the quick response and for the pointer to the 16.04-rc1 version. Nice! I think it would be great also to have a sanity check on the gpa_to_vva(). Although nothing recent has hit it we had some problems in that area in the past. Regards, Patrik On 03/17/2016 02:35 AM, Xie, Huawei wrote: > On 3/16/2016 8:53 PM, Patrik Andersson R wrote: >> Hello, >> >> When taking a snapshot of a running VM instance, using OpenStack >> "nova image-create", I noticed that one OVS pmd-thread eventually >> failed in DPDK rte_vhost_dequeue_burst() with repeating log entries: >> >> compute-0-6 ovs-vswitchd[38172]: VHOST_DATA: Failed to allocate >> memory for mbuf. >> >> >> Debugging (data included further down) this issue lead to the >> observation that there is no protection against malformed vhost >> queue descriptors, thus tenant separation might be violated as a >> single faulty VM might bring down the connectivity of all VMs >> connected to the same virtual switch. >> >> To avoid this, validation would be needed at some points in the >> rte_vhost_dequeue_burst() code: >> >> 1) when the queue descriptor is picked up for processing, >> desc->flags and desc->len might both be 0 >> >> ... >> desc = &vq->desc[head[entry_success]]; >> ... >> /* Discard first buffer as it is the virtio header */ >> if (desc->flags & VRING_DESC_F_NEXT) { >> desc = &vq->desc[desc->next]; >> vb_offset = 0; >> vb_avail = desc->len; >> } else { >> vb_offset = vq->vhost_hlen; >> vb_avail = desc->len - vb_offset; >> } >> .... >> >> 2) at buffer address translation gpa_to_vva(), might fail >> returning NULL as indication >> >> vb_addr = gpa_to_vva(dev, desc->addr); >> ... >> while (cpy_len != 0) { >> rte_memcpy(rte_pktmbuf_mtod_offset(cur, void *, seg_offset), >> (void *)((uintptr_t)(vb_addr + vb_offset)), >> cpy_len); >> ... >> } >> ... >> >> >> Wondering if there are any plans of adding any kind of validation in >> DPDK, or if it would be useful to suggest specific implementation of >> such validations in the DPDK code? >> >> Or is there some mechanism that gives us the confidence to trust >> the vhost queue content absolutely? >> >> >> >> Debugging data: >> >> For my scenario the problem occurs in DPDK rte_vhost_dequeue_burst() >> due to use of a vhost queue descriptor that has all fields 0: >> >> (gdb) print *desc >> {addr = 0, len = 0, flags = 0, next = 0} >> >> >> Subsequent use of desc->len to compute vb_avail = desc->len - vb_offset, >> leads to the problem observed. What happens is that the packet needs to >> be segmented -- on my system it fails roughly at segment 122000 when >> memory available for mbufs run out. >> >> The relevant local variables for rte_vhost_dequeue_burst() when breaking >> on the condition desc->len == 0: >> >> vb_avail = 4294967284 (0xfffffff4) >> seg_avail = 2608 >> vb_offset = 12 >> cpy_len = 2608 >> seg_num = 1 >> desc = 0x2aadb6e5c000 >> vb_addr = 46928960159744 >> entry_success = 0 >> >> Note also that there is no crash despite to the desc->addr being zero, >> it is a valid address in the regions mapped to the device. Although, the >> 3 regions mapped does not seem to be correct either at this stage. >> >> >> The versions that I'm running are OVS 2.4.0, with corrections from the >> 2.4 branch, and DPDK 2.1.0. QEMU emulator version 2.2.0 and >> libvirt version 1.2.12. >> >> >> Regards, >> >> Patrik > Thanks Patrik. You are right. We had planned to enhance the robustness > of vhost so that neither malicious nor buggy guest virtio driver could > corrupt vhost. Actually the 16.04 RC1 has fixed some issues (the return > of gpa_to_vva isn't checked). >