From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by dpdk.org (Postfix) with ESMTP id 9FAA9374E for ; Thu, 7 Sep 2017 07:07:44 +0200 (CEST) Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga104.jf.intel.com with ESMTP; 06 Sep 2017 22:07:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.42,356,1500966000"; d="scan'208";a="897904423" Received: from debian-zgviawfucg.sh.intel.com (HELO debian-ZGViaWFuCg) ([10.67.104.160]) by FMSMGA003.fm.intel.com with ESMTP; 06 Sep 2017 22:07:41 -0700 Date: Thu, 7 Sep 2017 13:08:06 +0800 From: Tiwei Bie To: Maxime Coquelin Cc: dev@dpdk.org, yliu@fridaylinux.org, jfreiman@redhat.com, mst@redhat.com, vkaplans@redhat.com, jasowang@redhat.com, lei.a.yao@intel.com, cunming.liang@intel.com Message-ID: <20170907050806.GA16503@debian-ZGViaWFuCg> References: <20170831095023.21037-1-maxime.coquelin@redhat.com> <20170831095023.21037-4-maxime.coquelin@redhat.com> <20170905044516.GC31895@debian-ZGViaWFuCg> <68468145-5b45-5875-b37f-35df3482379a@redhat.com> <20170905100751.GA7290@debian-ZGViaWFuCg> <0362ed01-211f-d4fc-d4ae-11ea81ad5df1@redhat.com> <20170906011459.GA3965@debian-ZGViaWFuCg> <06b50f42-9151-9b94-646e-e2e2b153a957@redhat.com> <20170906073021.GA15842@debian-ZGViaWFuCg> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.7.2 (2016-11-26) Subject: Re: [dpdk-dev] [PATCH 03/21] vhost: protect virtio_net device struct X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Sep 2017 05:07:45 -0000 On Wed, Sep 06, 2017 at 10:02:29PM +0200, Maxime Coquelin wrote: > On 09/06/2017 09:30 AM, Tiwei Bie wrote: > > On Wed, Sep 06, 2017 at 09:15:47AM +0200, Maxime Coquelin wrote: > > > Hi Tiwei, > > > > > > On 09/06/2017 03:15 AM, Tiwei Bie wrote: > > > > On Tue, Sep 05, 2017 at 01:00:42PM +0200, Maxime Coquelin wrote: > > > > > On 09/05/2017 12:07 PM, Tiwei Bie wrote: > > > > > > On Tue, Sep 05, 2017 at 11:24:14AM +0200, Maxime Coquelin wrote: > > > > > > > On 09/05/2017 06:45 AM, Tiwei Bie wrote: > > > > > > > > On Thu, Aug 31, 2017 at 11:50:05AM +0200, Maxime Coquelin wrote: > > > > > > > > > virtio_net device might be accessed while being reallocated > > > > > > > > > in case of NUMA awareness. This case might be theoretical, > > > > > > > > > but it will be needed anyway to protect vrings pages against > > > > > > > > > invalidation. > > > > > > > > > > > > > > > > > > The virtio_net devs are now protected with a readers/writers > > > > > > > > > lock, so that before reallocating the device, it is ensured > > > > > > > > > that it is not being referenced by the processing threads. > > > > > > > > > > > > > > > > > [...] > > > > > > > > > +struct virtio_net * > > > > > > > > > +get_device(int vid) > > > > > > > > > +{ > > > > > > > > > + struct virtio_net *dev; > > > > > > > > > + > > > > > > > > > + rte_rwlock_read_lock(&vhost_devices[vid].lock); > > > > > > > > > + > > > > > > > > > + dev = __get_device(vid); > > > > > > > > > + if (unlikely(!dev)) > > > > > > > > > + rte_rwlock_read_unlock(&vhost_devices[vid].lock); > > > > > > > > > + > > > > > > > > > + return dev; > > > > > > > > > +} > > > > > > > > > + > > > > > > > > > +void > > > > > > > > > +put_device(int vid) > > > > > > > > > +{ > > > > > > > > > + rte_rwlock_read_unlock(&vhost_devices[vid].lock); > > > > > > > > > +} > > > > > > > > > + > > > > > > > > > > > > > > > > This patch introduced a per-device rwlock which needs to be acquired > > > > > > > > unconditionally in the data path. So for each vhost device, the IO > > > > > > > > threads of different queues will need to acquire/release this lock > > > > > > > > during each enqueue and dequeue operation, which will cause cache > > > > > > > > contention when multiple queues are enabled and handled by different > > > > > > > > cores. With this patch alone, I saw ~7% performance drop when enabling > > > > > > > > 6 queues to do 64bytes iofwd loopback test. Is there any way to avoid > > > > > > > > introducing this lock to the data path? > > > > > > > > > > > > > > First, I'd like to thank you for running the MQ test. > > > > > > > I agree it may have a performance impact in this case. > > > > > > > > > > > > > > This lock has currently two purposes: > > > > > > > 1. Prevent referencing freed virtio_dev struct in case of numa_realloc. > > > > > > > 2. Protect vring pages against invalidation. > > > > > > > > > > > > > > For 2., it can be fixed by using the per-vq IOTLB lock (it was not the > > > > > > > case in my early prototypes that had per device IOTLB cache). > > > > > > > > > > > > > > For 1., this is an existing problem, so we might consider it is > > > > > > > acceptable to keep current state. Maybe it could be improved by only > > > > > > > reallocating in case VQ0 is not on the right NUMA node, the other VQs > > > > > > > not being initialized at this point. > > > > > > > > > > > > > > If we do this we might be able to get rid of this lock, I need some more > > > > > > > time though to ensure I'm not missing something. > > > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > > > > > Cool. So it's possible that the lock in the data path will be > > > > > > acquired only when the IOMMU feature is enabled. It will be > > > > > > great! > > > > > > > > > > > > Besides, I just did a very simple MQ test to verify my thoughts. > > > > > > Lei (CC'ed in this mail) may do a thorough performance test for > > > > > > this patch set to evaluate the performance impacts. > > > > > > > > > > I'll try to post v2 this week including the proposed change. > > > > > Maybe it'll be better Lei waits for the v2. > > > > > > > > > > > > > Cool. Sure. Thank you! :) > > > > > > I have done the changes, you can find the v2 on my gitlab repo: > > > https://gitlab.com/mcoquelin/dpdk-next-virtio/commits/vhost_iotlb_v2 > > > > > > I'm testing it right now, but if you'd like to run some early benchmark > > > before I post the series, there it is! > > > > > > > Got it. Thanks! :) > > Just to let you know that I have updated my branch to remove another > regression with iommu=off by inlining the noiommu part of > vhost_iova_to_vva call (See below for the patch, that is squashed into > my branch). > > Without this, when running microbenchmarks (txonly, rxonly, ...) I > noticed a 4% perf degradation. > Nice work! Best regards, Tiwei Bie