From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from sesbmg22.ericsson.net (sesbmg22.ericsson.net [193.180.251.48]) by dpdk.org (Postfix) with ESMTP id 6EF4C5597 for ; Wed, 16 Mar 2016 13:52:34 +0100 (CET) X-AuditID: c1b4fb30-f79246d00000788a-b7-56e9570e4032 Received: from ESESSHC012.ericsson.se (Unknown_Domain [153.88.183.54]) by sesbmg22.ericsson.net (Symantec Mail Security) with SMTP id 62.0D.30858.E0759E65; Wed, 16 Mar 2016 13:52:31 +0100 (CET) Received: from [147.214.49.111] (153.88.183.153) by smtp.internal.ericsson.com (153.88.183.56) with Microsoft SMTP Server id 14.3.248.2; Wed, 16 Mar 2016 13:52:05 +0100 From: Patrik Andersson R Organization: Ericsson AB To: Message-ID: <56E956F5.6080606@ericsson.com> Date: Wed, 16 Mar 2016 13:52:05 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprPLMWRmVeSWpSXmKPExsUyM2K7mS5/+MswgynHmS3efdrO5MDo8WvB UtYAxigum5TUnMyy1CJ9uwSujHk/DjEVHJGo2Hj9BUsD43vBLkZODgkBE4mTJ6awQthiEhfu rWfrYuTiEBI4zChx4PZSFghnDaPE0RmX2UGq2ASsJOZtW8YEYgsLJEhM3TKHDcTmF5CU2NCw mxnEFhEQklj6EaKeV0Bb4tvfNWAbWARUJTZteMACYosKREg8mXuSEaJGUOLkzCdgcWYBC4mZ 888zQtjyEtvfzgGbKSSgI/HqzFu2CYz8s5C0zELSMgtJywJG5lWMosWpxUm56UZGeqlFmcnF xfl5enmpJZsYgcF2cMtvgx2ML587HmIU4GBU4uH9MPlFmBBrYllxZe4hRgkOZiURXsuwl2FC vCmJlVWpRfnxRaU5qcWHGKU5WJTEeVk/XQ4TEkhPLEnNTk0tSC2CyTJxcEo1MHZvY4/elj/5 ybZGS5HPpXufhqyr4mDxZFg0YZp2ec7BNJYz06ctMtE8XR024cpr36C1k/9aRxV9d2Xscwry 1yv/WHrhKOd8ubKw7Vnqn/0nm7W639uzdJqL8z3FLc05M5QE94c33DBzOF7+LvK7/ewNL+7G tt05/bLCrMd3lUJoj62YtmFgpBJLcUaioRZzUXEiAAfKQxwyAgAA Subject: [dpdk-dev] vhost: no protection against malformed queue descriptors in rte_vhost_dequeue_burst() X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Mar 2016 12:52:34 -0000 Hello, When taking a snapshot of a running VM instance, using OpenStack "nova image-create", I noticed that one OVS pmd-thread eventually failed in DPDK rte_vhost_dequeue_burst() with repeating log entries: compute-0-6 ovs-vswitchd[38172]: VHOST_DATA: Failed to allocate memory for mbuf. Debugging (data included further down) this issue lead to the observation that there is no protection against malformed vhost queue descriptors, thus tenant separation might be violated as a single faulty VM might bring down the connectivity of all VMs connected to the same virtual switch. To avoid this, validation would be needed at some points in the rte_vhost_dequeue_burst() code: 1) when the queue descriptor is picked up for processing, desc->flags and desc->len might both be 0 ... desc = &vq->desc[head[entry_success]]; ... /* Discard first buffer as it is the virtio header */ if (desc->flags & VRING_DESC_F_NEXT) { desc = &vq->desc[desc->next]; vb_offset = 0; vb_avail = desc->len; } else { vb_offset = vq->vhost_hlen; vb_avail = desc->len - vb_offset; } .... 2) at buffer address translation gpa_to_vva(), might fail returning NULL as indication vb_addr = gpa_to_vva(dev, desc->addr); ... while (cpy_len != 0) { rte_memcpy(rte_pktmbuf_mtod_offset(cur, void *, seg_offset), (void *)((uintptr_t)(vb_addr + vb_offset)), cpy_len); ... } ... Wondering if there are any plans of adding any kind of validation in DPDK, or if it would be useful to suggest specific implementation of such validations in the DPDK code? Or is there some mechanism that gives us the confidence to trust the vhost queue content absolutely? Debugging data: For my scenario the problem occurs in DPDK rte_vhost_dequeue_burst() due to use of a vhost queue descriptor that has all fields 0: (gdb) print *desc {addr = 0, len = 0, flags = 0, next = 0} Subsequent use of desc->len to compute vb_avail = desc->len - vb_offset, leads to the problem observed. What happens is that the packet needs to be segmented -- on my system it fails roughly at segment 122000 when memory available for mbufs run out. The relevant local variables for rte_vhost_dequeue_burst() when breaking on the condition desc->len == 0: vb_avail = 4294967284 (0xfffffff4) seg_avail = 2608 vb_offset = 12 cpy_len = 2608 seg_num = 1 desc = 0x2aadb6e5c000 vb_addr = 46928960159744 entry_success = 0 Note also that there is no crash despite to the desc->addr being zero, it is a valid address in the regions mapped to the device. Although, the 3 regions mapped does not seem to be correct either at this stage. The versions that I'm running are OVS 2.4.0, with corrections from the 2.4 branch, and DPDK 2.1.0. QEMU emulator version 2.2.0 and libvirt version 1.2.12. Regards, Patrik