From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <mst@redhat.com>
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28])
 by dpdk.org (Postfix) with ESMTP id F3DD08E9E
 for <dev@dpdk.org>; Thu, 21 Jan 2016 14:39:30 +0100 (CET)
Received: from int-mx10.intmail.prod.int.phx2.redhat.com
 (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23])
 by mx1.redhat.com (Postfix) with ESMTPS id 481638DFEC;
 Thu, 21 Jan 2016 13:39:30 +0000 (UTC)
Received: from redhat.com (vpn1-6-206.ams2.redhat.com [10.36.6.206])
 by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with SMTP id
 u0LDdQ7G021501; Thu, 21 Jan 2016 08:39:27 -0500
Date: Thu, 21 Jan 2016 15:39:26 +0200
From: "Michael S. Tsirkin" <mst@redhat.com>
To: virtio@lists.oasis-open.org
Message-ID: <20160121145418-mutt-send-email-mst@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23
Cc: virtio-dev@lists.oasis-open.org, kvm@vger.kernel.org, dev@dpdk.org,
 linux-kernel@vger.kernel.org, qemu-devel@nongnu.org,
 virtualization@lists.linux-foundation.org
Subject: [dpdk-dev] virtio ring layout changes for optimal single-stream
	performance
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Thu, 21 Jan 2016 13:39:31 -0000

Hi all!
I have been experimenting with alternative virtio ring layouts,
in order to speed up single stream performance.

I have just posted a benchmark I wrote for the purpose, and a (partial)
alternative layout implementation.  This achieves 20-40% reduction in
virtio overhead in the (default) polling mode.

http://article.gmane.org/gmane.linux.kernel.virtualization/26889

The layout is trying to be as simple as possible, to reduce
the number of cache lines bouncing between CPUs.

For benchmarking, the idea is to emulate virtio in user-space,
artificially adding overhead for e.g. signalling to match what happens
in case of a VM.

I'd be very curious to get feedback on this, in particular, some people
discussed using vectored operations to format virtio ring - would it
conflict with this work?

You are all welcome to post enhancements or more layout alternatives as
patches.

TODO:
- documentation+discussion of interaction with CPU caching
- thorough benchmarking of different configurations/hosts
- experiment with event index replacements
- better emulate vmexit/vmentry cost overhead
- virtio spec proposal

Thanks!
-- 
MST