From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f177.google.com (mail-wi0-f177.google.com [209.85.212.177]) by dpdk.org (Postfix) with ESMTP id 3BC5D8D37 for ; Thu, 1 Oct 2015 12:50:13 +0200 (CEST) Received: by wicfx3 with SMTP id fx3so26786319wic.1 for ; Thu, 01 Oct 2015 03:50:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=dwzxEOucqlcwk1x99WHPwwuqCYZpJYoCtJsE8F81gWA=; b=ZJgfZXd1Y7l2jsotlXgMGZ429BrOjD8OsiilNWykT7mDycPDqRM8iJMhGuf+/7tD+o nVbUgQN/KXXgm12Pdk/ZZGVCc5C0xgrSfwPlmLDEsbMSGgDwD0/1T7Hvb478NdVOmsld ZOMvL/ulOq97yfCmIAH5iw5wq29ZEQkiRj2ddkdxdxUdqX3J/kU4jnApg07q3rW7XOkq OvrbRp+rEriHXhMOxB2slNoGIsthrBZCJToLqfnE1Y7V0TSnoJ6HY/eiiB5Ojxrow5k3 GwhOF1qoRqcPDpp3/z49qamfRCtZL4shXlZI2RamkVtrvKztL+KVIV9sMVqZqI+/D5VA w2Iw== X-Gm-Message-State: ALoCoQknR1hh4Pt1dynZRXb3Eql7CgFSOs7qrS+T65j/hPkvRcPTs3GniYLcmsEJX6F0tq/eNQG/ X-Received: by 10.194.112.162 with SMTP id ir2mr10930567wjb.40.1443696613056; Thu, 01 Oct 2015 03:50:13 -0700 (PDT) Received: from avi.cloudius ([37.142.229.250]) by smtp.googlemail.com with ESMTPSA id z2sm2558675wij.1.2015.10.01.03.50.11 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 01 Oct 2015 03:50:11 -0700 (PDT) To: "Michael S. Tsirkin" References: <560BF782.4070308@scylladb.com> <20150930175848-mutt-send-email-mst@redhat.com> <560C0171.7080507@scylladb.com> <20150930204016.GA29975@redhat.com> <20151001113828-mutt-send-email-mst@redhat.com> <560CF44A.60102@scylladb.com> <20151001120027-mutt-send-email-mst@redhat.com> <560CFB66.5050904@scylladb.com> <20151001124211-mutt-send-email-mst@redhat.com> <560D0413.5080401@scylladb.com> <20151001131754-mutt-send-email-mst@redhat.com> From: Avi Kivity Message-ID: <560D0FE2.7010905@scylladb.com> Date: Thu, 1 Oct 2015 13:50:10 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <20151001131754-mutt-send-email-mst@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] Having troubles binding an SR-IOV VF to uio_pci_generic on Amazon instance X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2015 10:50:13 -0000 On 10/01/2015 01:38 PM, Michael S. Tsirkin wrote: > On Thu, Oct 01, 2015 at 12:59:47PM +0300, Avi Kivity wrote: >> >> On 10/01/2015 12:55 PM, Michael S. Tsirkin wrote: >>> On Thu, Oct 01, 2015 at 12:22:46PM +0300, Avi Kivity wrote: >>>> It's easy to claim that >>>> a solution is around the corner, only no one was looking for it, but the >>>> reality is that kernel bypass has been a solution for years for high >>>> performance users, >>> I never said that it's trivial. >>> >>> It's probably a lot of work. It's definitely more work than just abusing >>> sysfs. >>> >>> But it looks like a write system call into an eventfd is about 1.5 >>> microseconds on my laptop. Even with a system call per packet, system >>> call overhead is not what makes DPDK drivers outperform Linux ones. >>> >> 1.5 us = 0.6 Mpps per core limit. > Oh, I calculated it incorrectly. It's 0.15 us. So 6Mpps. You also trimmed the extra work that needs to be done, that I mentioned. Maybe your ring proxy can work, maybe it can't. In any case it's a hefty chunk of work. Should this work block users from using their VFs, if they happen to need interrupt support? > But for RX, you can batch a lot of packets. > > You can see by now I'm not that good at benchmarking. > Here's what I wrote: > > > #include > #include > #include > #include > > > int main(int argc, char **argv) > { > int e = eventfd(0, 0); > uint64_t v = 1; > > int i; > > for (i = 0; i < 10000000; ++i) { > write(e, &v, sizeof v); > } > } > > > This takes 1.5 seconds to run on my laptop: > > $ time ./a.out > > real 0m1.507s > user 0m0.179s > sys 0m1.328s > > >> dpdk performance is in the tens of >> millions of packets per system. > I think that's with a bunch of batching though. Yes, it's also with their application code running as well. They didn't reach this kind of performance by spending cycles unnecessarily. I'm not saying that the ring proxy is not workable; just that we don't know whether it is or not, and I don't think that a patch that enables _existing functionality_ for VFs should be blocked in favor of a new and unproven approach. > >> It's not just the lack of system calls, of course, the architecture is >> completely different. > Absolutely - I'm not saying move all of DPDK into kernel. > We just need to protect the RX rings so hardware does > not corrupt kernel memory. > > > Thinking about it some more, many devices > have separate rings for DMA: TX (device reads memory) > and RX (device writes memory). > With such devices, a mode where userspace can write TX ring > but not RX ring might make sense. I'm sure you can cause havoc just by reading, if you read from I/O memory. > > This will mean userspace might read kernel memory > through the device, but can not corrupt it. > > That's already a big win! > > And RX buffers do not have to be added one at a time. > If we assume 0.2usec per system call, batching some 100 buffers per > system call gives you 2 nano seconds overhead. That seems quite > reasonable. You're ignoring the page table walk and other per-descriptor processing. Again^2, maybe this can work. But it shouldn't block a patch enabling interrupt support of VFs. After the ring proxy is available and proven for a few years, we can deprecate bus mastering from uio, and after a few more years remove it.