From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <nhorman@tuxdriver.com>
Received: from smtp.tuxdriver.com (charlotte.tuxdriver.com [70.61.120.58])
by dpdk.org (Postfix) with ESMTP id C9B883B5
for <dev@dpdk.org>; Wed, 1 Oct 2014 13:07:09 +0200 (CEST)
Received: from [2001:470:8:a08:9833:6894:f2b2:43a] (helo=localhost)
by smtp.tuxdriver.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.63)
(envelope-from <nhorman@tuxdriver.com>)
id 1XZHqt-0007R6-AO; Wed, 01 Oct 2014 07:13:52 -0400
Date: Wed, 1 Oct 2014 07:13:46 -0400
From: Neil Horman <nhorman@tuxdriver.com>
To: Hiroshi Shimamoto <h-shimamoto@ct.jp.nec.com>
Message-ID: <20141001111346.GD21151@hmsreliant.think-freely.org>
References: <7F861DC0615E0C47A872E6F3C5FCDDBD02AE26C5@BPXM14GP.gisp.nec.co.jp>
<20140930143242.GI2193@hmsreliant.think-freely.org>
<7F861DC0615E0C47A872E6F3C5FCDDBD02AE2D37@BPXM14GP.gisp.nec.co.jp>
<20141001084445.GC1204@BRICHA3-MOBL>
<7F861DC0615E0C47A872E6F3C5FCDDBD02AE3BAF@BPXM14GP.gisp.nec.co.jp>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <7F861DC0615E0C47A872E6F3C5FCDDBD02AE3BAF@BPXM14GP.gisp.nec.co.jp>
User-Agent: Mutt/1.5.23 (2014-03-12)
X-Spam-Score: -2.9 (--)
X-Spam-Status: No
Cc: "dev@dpdk.org" <dev@dpdk.org>, Hayato Momma <h-momma@ce.jp.nec.com>
Subject: Re: [dpdk-dev] [memnic PATCH v2 6/7] pmd: add branch hint in
recv/xmit
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Wed, 01 Oct 2014 11:07:10 -0000
On Wed, Oct 01, 2014 at 09:12:44AM +0000, Hiroshi Shimamoto wrote:
> > Subject: Re: [dpdk-dev] [memnic PATCH v2 6/7] pmd: add branch hint in recv/xmit
> >
> > On Tue, Sep 30, 2014 at 11:52:00PM +0000, Hiroshi Shimamoto wrote:
> > > Hi,
> > >
> > > > Subject: Re: [dpdk-dev] [memnic PATCH v2 6/7] pmd: add branch hint in recv/xmit
> > > >
> > > > On Tue, Sep 30, 2014 at 11:14:40AM +0000, Hiroshi Shimamoto wrote:
> > > > > From: Hiroshi Shimamoto <h-shimamoto@ct.jp.nec.com>
> > > > >
> > > > > To reduce instruction cache miss, add branch condition hints into
> > > > > recv/xmit functions. This improves a bit performance.
> > > > >
> > > > > We can see performance improvements with memnic-tester.
> > > > > Using Xeon E5-2697 v2 @ 2.70GHz, 4 vCPU.
> > > > > size | before | after
> > > > > 64 | 5.54Mpps | 5.55Mpps
> > > > > 128 | 5.46Mpps | 5.44Mpps
> > > > > 256 | 5.21Mpps | 5.22Mpps
> > > > > 512 | 4.50Mpps | 4.52Mpps
> > > > > 1024 | 3.71Mpps | 3.73Mpps
> > > > > 1280 | 3.21Mpps | 3.22Mpps
> > > > > 1518 | 2.92Mpps | 2.93Mpps
> > > > >
> > > > > Signed-off-by: Hiroshi Shimamoto <h-shimamoto@ct.jp.nec.com>
> > > > > Reviewed-by: Hayato Momma <h-momma@ce.jp.nec.com>
> > > > > ---
> > > > > pmd/pmd_memnic.c | 18 +++++++++---------
> > > > > 1 file changed, 9 insertions(+), 9 deletions(-)
> > > > >
> > > > > diff --git a/pmd/pmd_memnic.c b/pmd/pmd_memnic.c
> > > > > index 7fc3093..875d3ea 100644
> > > > > --- a/pmd/pmd_memnic.c
> > > > > +++ b/pmd/pmd_memnic.c
> > > > > @@ -289,26 +289,26 @@ static uint16_t memnic_recv_pkts(void *rx_queue,
> > > > > int idx, next;
> > > > > struct rte_eth_stats *st = &adapter->stats[rte_lcore_id()];
> > > > >
> > > > > - if (!adapter->nic->hdr.valid)
> > > > > + if (unlikely(!adapter->nic->hdr.valid))
> > > > > return 0;
> > > > >
> > > > > pkts = bytes = errs = 0;
> > > > > idx = adapter->up_idx;
> > > > > for (nr = 0; nr < nb_pkts; nr++) {
> > > > > p = &data->packets[idx];
> > > > > - if (p->status != MEMNIC_PKT_ST_FILLED)
> > > > > + if (unlikely(p->status != MEMNIC_PKT_ST_FILLED))
> > > > > break;
> > > > > /* prefetch the next area */
> > > > > next = idx;
> > > > > - if (++next >= MEMNIC_NR_PACKET)
> > > > > + if (unlikely(++next >= MEMNIC_NR_PACKET))
> > > > > next = 0;
> > > > > rte_prefetch0(&data->packets[next]);
> > > > > - if (p->len > framesz) {
> > > > > + if (unlikely(p->len > framesz)) {
> > > > > errs++;
> > > > > goto drop;
> > > > > }
> > > > > mb = rte_pktmbuf_alloc(adapter->mp);
> > > > > - if (!mb)
> > > > > + if (unlikely(!mb))
> > > > > break;
> > > > >
> > > > > rte_memcpy(rte_pktmbuf_mtod(mb, void *), p->data, p->len);
> > > > > @@ -350,7 +350,7 @@ static uint16_t memnic_xmit_pkts(void *tx_queue,
> > > > > uint64_t pkts, bytes, errs;
> > > > > uint32_t framesz = adapter->framesz;
> > > > >
> > > > > - if (!adapter->nic->hdr.valid)
> > > > > + if (unlikely(!adapter->nic->hdr.valid))
> > > > > return 0;
> > > > >
> > > > > pkts = bytes = errs = 0;
> > > > > @@ -360,7 +360,7 @@ static uint16_t memnic_xmit_pkts(void *tx_queue,
> > > > > struct rte_mbuf *sg;
> > > > > void *ptr;
> > > > >
> > > > > - if (pkt_len > framesz) {
> > > > > + if (unlikely(pkt_len > framesz)) {
> > > > > errs++;
> > > > > break;
> > > > > }
> > > > > @@ -379,7 +379,7 @@ retry:
> > > > > goto retry;
> > > > > }
> > > > >
> > > > > - if (idx != ACCESS_ONCE(adapter->down_idx)) {
> > > > > + if (unlikely(idx != ACCESS_ONCE(adapter->down_idx))) {
> > > > Why are you using ACCESS_ONCE here? Or for that matter, anywhere else in this
> > > > PMD? The whole idea of the ACCESS_ONCE macro is to assign a value to a variable
> > > > once and prevent it from getting reloaded from memory at a later time, this is
> > > > exactly contrary to that, both in the sense that you're explicitly reloading the
> > > > same variable multiple times, and that you're using it as part of a comparison
> > > > operation, rather than an asignment operation
> > >
> > > ACCESS_ONCE prevents compiler optimization and ensures load from memory.
> > > There could be multiple threads which read/write that index.
> > > We should compare the value previous and the current value in memory.
> > > In that reason, I use ACCESS_ONCE macro to get value in the memory.
> >
> > Should you not just make the variable volatile? That's the normal way to
> > guarantee reads from memory and prevent the compiler caching things in
> > registers.
>
> We don't want always accessing to memory, it could cause performance degradation.
> Like linux kernel, I use it in the place only we really load from memory.
>
Thats not true at all. Every single read of adapter->down_idx in
memnic_xmit_pkts() is wrapped in a ACCESS_ONCE call. Theres no difference in
doing that and just declaring a volitile variable and pointing it to
&adapter->down_idx (save for the increased legibility of the code)
Neil
> thanks,
> Hiroshi
>
> >
> > /Bruce
> >
> > >
> > > thanks,
> > > Hiroshi
> > >
> > > >
> > > > Neil
> > >
>