From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 969882BA1; Fri, 17 Mar 2017 06:49:13 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=intel.com; i=@intel.com; q=dns/txt; s=intel; t=1489729754; x=1521265754; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=JR/HbctRkfpKnF540SJOCivVWcg5l2gevj/IXqtXavs=; b=sCA0A2RVPeowHr3dXC0vv9zuAp+vbD5iCzcK9YX+JsqUwEWIK0U/7COE pR8+PoElSqceLsIeXdeHHnoh07Iw/Q==; Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Mar 2017 22:49:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.36,175,1486454400"; d="scan'208";a="237378647" Received: from yliu-dev.sh.intel.com (HELO yliu-dev) ([10.239.67.162]) by fmsmga004.fm.intel.com with ESMTP; 16 Mar 2017 22:49:11 -0700 Date: Fri, 17 Mar 2017 13:47:25 +0800 From: Yuanhan Liu To: Kevin Traynor Cc: maxime.coquelin@redhat.com, dev@dpdk.org, stable@dpdk.org Message-ID: <20170317054725.GC18844@yliu-dev.sh.intel.com> References: <1489605049-18686-1-git-send-email-ktraynor@redhat.com> <20170316062122.GN18844@yliu-dev.sh.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Subject: Re: [dpdk-stable] [PATCH] vhost: fix virtio_net cache sharing of broadcast_rarp X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Mar 2017 05:49:14 -0000 On Thu, Mar 16, 2017 at 10:10:05AM +0000, Kevin Traynor wrote: > On 03/16/2017 06:21 AM, Yuanhan Liu wrote: > > On Wed, Mar 15, 2017 at 07:10:49PM +0000, Kevin Traynor wrote: > >> The virtio_net structure is used in both enqueue and dequeue datapaths. > >> broadcast_rarp is checked with cmpset in the dequeue datapath regardless > >> of whether descriptors are available or not. > >> > >> It is observed in some cases where dequeue and enqueue are performed by > >> different cores and no packets are available on the dequeue datapath > >> (i.e. uni-directional traffic), the frequent checking of broadcast_rarp > >> in dequeue causes performance degradation for the enqueue datapath. > >> > >> In OVS the issue can cause a uni-directional performance drop of up to 15%. > >> > >> Fix that by moving broadcast_rarp to a different cache line in > >> virtio_net struct. > > > > Thanks, but I'm a bit confused. The drop looks like being caused by > > cache false sharing, but I don't see anything would lead to a false > > sharing. I mean, there is no write in the same cache line where the > > broadcast_rarp belongs. Or, the "volatile" type is the culprit here? > > > > Yes, the cmpset code uses cmpxchg and that performs a write regardless > of the result - it either writes the new value or back the old value. Oh, right, I missed this part! > > Talking about that, I had actually considered to turn "broadcast_rarp" > > to a simple "int" or "uint16_t" type, to make it more light weight. > > The reason I used atomic type is to exactly send one broadcast RARP > > packet once SEND_RARP request is recieved. Otherwise, we may send more > > than one RARP packet when MQ is invovled. But I think we don't have > > to be that accurate: it's tolerable when more RARP are sent. I saw 4 > > SEND_RARP requests (aka 4 RARP packets) in the last time I tried > > vhost-user live migration after all. I don't quite remember why > > it was 4 though. > > > > That said, I think it also would resolve the performance issue if you > > change "rte_atomic16_t" to "uint16_t", without moving the place? > > > > Yes, that should work fine, with the side effect you mentioned of > possibly some more rarps - no big deal. > > I tested another solution also - as it is unlikely we would need to send > the broadcast_rarp, you can first read and only do the cmpset if it is > likely to succeed. This resolved the issue too. > > --- a/lib/librte_vhost/virtio_net.c > +++ b/lib/librte_vhost/virtio_net.c > @@ -1057,7 +1057,8 @@ static inline bool __attribute__((always_inline)) > * > * Check user_send_rarp() for more information. > */ > - if (unlikely(rte_atomic16_cmpset((volatile uint16_t *) > + if (unlikely(rte_atomic16_read(&dev->broadcast_rarp) && > + rte_atomic16_cmpset((volatile uint16_t *) > &dev->broadcast_rarp.cnt, 1, 0))) { > rarp_mbuf = rte_pktmbuf_alloc(mbuf_pool); > if (rarp_mbuf == NULL) { I'm okay with this one. It's simple and clean enough, that it could be picked to a stable release. Later, I'd like to send another patch to turn it to "uint16_t". Since it changes the behaviour a bit, it is not a good candidate for stable release. BTW, would you please include the root cause (false sharing) into your commit log? --yliu > > I choose changing the struct because the 'read && cmpset' code is > non-obvious and someone might not think to do that in the future. I did > a PVP test with testpmd and didn't see any degradation with the struct > change, so I thought it can be a good solution. > > I tested the struct change with several combinations of DPDK > 16.11.1/17.02/master combined with OVS 2.6/2.7/master. If you prefer one > of the other solutions, let me know and I'll perform some additional > testing. > > Kevin. > > > --yliu > >