From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io0-f182.google.com (mail-io0-f182.google.com [209.85.223.182]) by dpdk.org (Postfix) with ESMTP id E14508DB1 for ; Fri, 2 Oct 2015 18:57:20 +0200 (CEST) Received: by iow1 with SMTP id 1so89470021iow.1 for ; Fri, 02 Oct 2015 09:57:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-type:content-transfer-encoding; bh=aWXo2o6VSzgrrBdMQgXoBkKiIMmmyZgVhYBjh0WmdqE=; b=DQD3oyzyge3E1UhlZjWxcYuOUc/Xei/WImos0y3eHe8D/7zSZ5B3cfmepMkCxSDMrR za+eMd5DWQYy6G9qjU3kFJWqgUjCMxlJbEbuTMLSVopV949nHiJG8vEMyJFZ+9Qvfh0E vSnmxSKB0lHIt2WzEdp8+W33K+BEcJCbxds6utTiwTE/i6+SqogiUW+ds6mqx8rnbunl oEK7UtuGhk5J8f7FGnZ0P+9eiucqiO1mgMDEbbmJTNHeHC0vqFyLqvnnPgA7OcDqtETi f5N/P+ChCl1wXp18FZjUP9wWH6/MfhQMa8lD/UuLLqhNcBnp4VsKBDiUAjrgjDDyq5CZ OcOQ== X-Received: by 10.107.41.138 with SMTP id p132mr17279470iop.188.1443805040350; Fri, 02 Oct 2015 09:57:20 -0700 (PDT) Received: from [192.168.1.188] (static-50-53-21-5.bvtn.or.frontiernet.net. [50.53.21.5]) by smtp.googlemail.com with ESMTPSA id w10sm3875642igz.13.2015.10.02.09.57.19 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 02 Oct 2015 09:57:19 -0700 (PDT) To: "Michael S. Tsirkin" References: <20151001120027-mutt-send-email-mst@redhat.com> <560CFB66.5050904@scylladb.com> <20151001124211-mutt-send-email-mst@redhat.com> <560D0413.5080401@scylladb.com> <20151001131754-mutt-send-email-mst@redhat.com> <20151001110806.GA16248@bricha3-MOBL3> <20151001141124-mutt-send-email-mst@redhat.com> <20151001120713.GA11504@bricha3-MOBL3> <20151001155943-mutt-send-email-mst@redhat.com> <560D9F60.6040907@gmail.com> <20151002165033-mutt-send-email-mst@redhat.com> From: Alexander Duyck Message-ID: <560EB76E.2090301@gmail.com> Date: Fri, 2 Oct 2015 09:57:18 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <20151002165033-mutt-send-email-mst@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: "dev@dpdk.org" , Avi Kivity Subject: Re: [dpdk-dev] Having troubles binding an SR-IOV VF to uio_pci_generic on Amazon instance X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 16:57:21 -0000 On 10/02/2015 07:00 AM, Michael S. Tsirkin wrote: > On Thu, Oct 01, 2015 at 02:02:24PM -0700, Alexander Duyck wrote: >> validation and translation would add 10s if not 100s of nanoseconds to the >> time needed to process each packet. In addition we are talking about doing >> this in kernel space which means we wouldn't really be able to take >> advantage of things like SSE or AVX instructions. > Yes. But the nice thing is that it's rearming so it can happen on > a separate core, in parallel with packet processing. > It does not need to add to latency. Moving it to another core is automatically going to add extra latency. You will have to evict the data out of the L1 cache for one core and into the L1 cache for another when you update it, and then reading it will force it to have to transition back out. If you are lucky it is only evicted to L2, if not then to L3, or possibly even back to memory. Odds are that alone will add tens of nanoseconds to the process, and you would need three or more cores to do the same workload as running the process over multiple threads means having to add synchronization primitives to the whole mess. Then there is the NUMA factor on top of that. > You will burn up more CPU, but again, all this for boxes/hypervisors > without an IOMMU. There are use cases this will completely make useless. If for example you are running a workload that needs three cores with DPDK bumping it to nine or more will likely push you out of being able to do the workload on some systems. > I'm sure people can come up with even better approaches, once enough > people get it that kernel absolutely needs to be protected from > userspace. I don't see that happening. Many people don't care about kernel security that much. If they did something like DPDK wouldn't have gotten off of the ground. Once someone has the ability to load kernel modules any protection of the kernel from userspace pretty much goes right out the window. You are just as much at risk from a buggy driver in userspace as you are from one that can be added to the kernel. > Long term, the right thing to do is to focus on IOMMU support. This > gives you hardware-based memory protection without need to burn up CPU > cycles. We have a solution that makes use of IOMMU support with vfio. The problem is there are multiple cases where that support is either not available, or using the IOMMU provides excess overhead. - Alex