From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from tyo202.gate.nec.co.jp (TYO202.gate.nec.co.jp [210.143.35.52]) by dpdk.org (Postfix) with ESMTP id A3310212 for ; Thu, 2 Oct 2014 04:00:43 +0200 (CEST) Received: from mailgate3.nec.co.jp ([10.7.69.195]) by tyo202.gate.nec.co.jp (8.13.8/8.13.4) with ESMTP id s9227T2I006359; Thu, 2 Oct 2014 11:07:29 +0900 (JST) Received: from mailsv3.nec.co.jp (imss61.nec.co.jp [10.7.69.156]) by mailgate3.nec.co.jp (8.11.7/3.7W-MAILGATE-NEC) with ESMTP id s9227Sx27989; Thu, 2 Oct 2014 11:07:28 +0900 (JST) Received: from mail03.kamome.nec.co.jp (mail03.kamome.nec.co.jp [10.25.43.7]) by mailsv3.nec.co.jp (8.13.8/8.13.4) with ESMTP id s9227SW6022846; Thu, 2 Oct 2014 11:07:28 +0900 (JST) Received: from bpxc99gp.gisp.nec.co.jp ([10.38.151.143] [10.38.151.143]) by mail01b.kamome.nec.co.jp with ESMTP id BT-MMP-2274340; Thu, 2 Oct 2014 11:07:11 +0900 Received: from BPXM14GP.gisp.nec.co.jp ([169.254.1.136]) by BPXC15GP.gisp.nec.co.jp ([10.38.151.143]) with mapi id 14.03.0174.002; Thu, 2 Oct 2014 11:07:10 +0900 From: Hiroshi Shimamoto To: Neil Horman Thread-Topic: [dpdk-dev] [memnic PATCH v2 6/7] pmd: add branch hint in recv/xmit Thread-Index: Ac/cn7cr+0i1sUkPRO2iopF1mOwIlf//oHoA//7TFSCAAl4JgP//ZtzAgADCxgD//p8jQABLHBYA//9oqXA= Date: Thu, 2 Oct 2014 02:07:09 +0000 Message-ID: <7F861DC0615E0C47A872E6F3C5FCDDBD02AE4668@BPXM14GP.gisp.nec.co.jp> References: <7F861DC0615E0C47A872E6F3C5FCDDBD02AE26C5@BPXM14GP.gisp.nec.co.jp> <20140930143242.GI2193@hmsreliant.think-freely.org> <7F861DC0615E0C47A872E6F3C5FCDDBD02AE2D37@BPXM14GP.gisp.nec.co.jp> <20141001084445.GC1204@BRICHA3-MOBL> <7F861DC0615E0C47A872E6F3C5FCDDBD02AE3BAF@BPXM14GP.gisp.nec.co.jp> <20141001111346.GD21151@hmsreliant.think-freely.org> <7F861DC0615E0C47A872E6F3C5FCDDBD02AE422F@BPXM14GP.gisp.nec.co.jp> <20141002020126.GA9757@localhost.localdomain> In-Reply-To: <20141002020126.GA9757@localhost.localdomain> Accept-Language: ja-JP, en-US Content-Language: ja-JP X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.205.5.123] Content-Type: text/plain; charset="iso-2022-jp" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "dev@dpdk.org" , Hayato Momma Subject: Re: [dpdk-dev] [memnic PATCH v2 6/7] pmd: add branch hint in recv/xmit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Oct 2014 02:00:44 -0000 > Subject: Re: [dpdk-dev] [memnic PATCH v2 6/7] pmd: add branch hint in rec= v/xmit >=20 > On Wed, Oct 01, 2014 at 11:33:23PM +0000, Hiroshi Shimamoto wrote: > > > Subject: Re: [dpdk-dev] [memnic PATCH v2 6/7] pmd: add branch hint in= recv/xmit > > > > > > On Wed, Oct 01, 2014 at 09:12:44AM +0000, Hiroshi Shimamoto wrote: > > > > > Subject: Re: [dpdk-dev] [memnic PATCH v2 6/7] pmd: add branch hin= t in recv/xmit > > > > > > > > > > On Tue, Sep 30, 2014 at 11:52:00PM +0000, Hiroshi Shimamoto wrote= : > > > > > > Hi, > > > > > > > > > > > > > Subject: Re: [dpdk-dev] [memnic PATCH v2 6/7] pmd: add branch= hint in recv/xmit > > > > > > > > > > > > > > On Tue, Sep 30, 2014 at 11:14:40AM +0000, Hiroshi Shimamoto w= rote: > > > > > > > > From: Hiroshi Shimamoto > > > > > > > > > > > > > > > > To reduce instruction cache miss, add branch condition hint= s into > > > > > > > > recv/xmit functions. This improves a bit performance. > > > > > > > > > > > > > > > > We can see performance improvements with memnic-tester. > > > > > > > > Using Xeon E5-2697 v2 @ 2.70GHz, 4 vCPU. > > > > > > > > size | before | after > > > > > > > > 64 | 5.54Mpps | 5.55Mpps > > > > > > > > 128 | 5.46Mpps | 5.44Mpps > > > > > > > > 256 | 5.21Mpps | 5.22Mpps > > > > > > > > 512 | 4.50Mpps | 4.52Mpps > > > > > > > > 1024 | 3.71Mpps | 3.73Mpps > > > > > > > > 1280 | 3.21Mpps | 3.22Mpps > > > > > > > > 1518 | 2.92Mpps | 2.93Mpps > > > > > > > > > > > > > > > > Signed-off-by: Hiroshi Shimamoto > > > > > > > > Reviewed-by: Hayato Momma > > > > > > > > --- > > > > > > > > pmd/pmd_memnic.c | 18 +++++++++--------- > > > > > > > > 1 file changed, 9 insertions(+), 9 deletions(-) > > > > > > > > > > > > > > > > diff --git a/pmd/pmd_memnic.c b/pmd/pmd_memnic.c > > > > > > > > index 7fc3093..875d3ea 100644 > > > > > > > > --- a/pmd/pmd_memnic.c > > > > > > > > +++ b/pmd/pmd_memnic.c > > > > > > > > @@ -289,26 +289,26 @@ static uint16_t memnic_recv_pkts(void= *rx_queue, > > > > > > > > int idx, next; > > > > > > > > struct rte_eth_stats *st =3D &adapter->stats[rte_lcore_id= ()]; > > > > > > > > > > > > > > > > - if (!adapter->nic->hdr.valid) > > > > > > > > + if (unlikely(!adapter->nic->hdr.valid)) > > > > > > > > return 0; > > > > > > > > > > > > > > > > pkts =3D bytes =3D errs =3D 0; > > > > > > > > idx =3D adapter->up_idx; > > > > > > > > for (nr =3D 0; nr < nb_pkts; nr++) { > > > > > > > > p =3D &data->packets[idx]; > > > > > > > > - if (p->status !=3D MEMNIC_PKT_ST_FILLED) > > > > > > > > + if (unlikely(p->status !=3D MEMNIC_PKT_ST_FILLED)) > > > > > > > > break; > > > > > > > > /* prefetch the next area */ > > > > > > > > next =3D idx; > > > > > > > > - if (++next >=3D MEMNIC_NR_PACKET) > > > > > > > > + if (unlikely(++next >=3D MEMNIC_NR_PACKET)) > > > > > > > > next =3D 0; > > > > > > > > rte_prefetch0(&data->packets[next]); > > > > > > > > - if (p->len > framesz) { > > > > > > > > + if (unlikely(p->len > framesz)) { > > > > > > > > errs++; > > > > > > > > goto drop; > > > > > > > > } > > > > > > > > mb =3D rte_pktmbuf_alloc(adapter->mp); > > > > > > > > - if (!mb) > > > > > > > > + if (unlikely(!mb)) > > > > > > > > break; > > > > > > > > > > > > > > > > rte_memcpy(rte_pktmbuf_mtod(mb, void *), p->data, p->len= ); > > > > > > > > @@ -350,7 +350,7 @@ static uint16_t memnic_xmit_pkts(void *= tx_queue, > > > > > > > > uint64_t pkts, bytes, errs; > > > > > > > > uint32_t framesz =3D adapter->framesz; > > > > > > > > > > > > > > > > - if (!adapter->nic->hdr.valid) > > > > > > > > + if (unlikely(!adapter->nic->hdr.valid)) > > > > > > > > return 0; > > > > > > > > > > > > > > > > pkts =3D bytes =3D errs =3D 0; > > > > > > > > @@ -360,7 +360,7 @@ static uint16_t memnic_xmit_pkts(void *= tx_queue, > > > > > > > > struct rte_mbuf *sg; > > > > > > > > void *ptr; > > > > > > > > > > > > > > > > - if (pkt_len > framesz) { > > > > > > > > + if (unlikely(pkt_len > framesz)) { > > > > > > > > errs++; > > > > > > > > break; > > > > > > > > } > > > > > > > > @@ -379,7 +379,7 @@ retry: > > > > > > > > goto retry; > > > > > > > > } > > > > > > > > > > > > > > > > - if (idx !=3D ACCESS_ONCE(adapter->down_idx)) { > > > > > > > > + if (unlikely(idx !=3D ACCESS_ONCE(adapter->down_idx))) { > > > > > > > Why are you using ACCESS_ONCE here? Or for that matter, anyw= here else in this > > > > > > > PMD? The whole idea of the ACCESS_ONCE macro is to assign a = value to a variable > > > > > > > once and prevent it from getting reloaded from memory at a la= ter time, this is > > > > > > > exactly contrary to that, both in the sense that you're expli= citly reloading the > > > > > > > same variable multiple times, and that you're using it as par= t of a comparison > > > > > > > operation, rather than an asignment operation > > > > > > > > > > > > ACCESS_ONCE prevents compiler optimization and ensures load fro= m memory. > > > > > > There could be multiple threads which read/write that index. > > > > > > We should compare the value previous and the current value in m= emory. > > > > > > In that reason, I use ACCESS_ONCE macro to get value in the mem= ory. > > > > > > > > > > Should you not just make the variable volatile? That's the normal= way to > > > > > guarantee reads from memory and prevent the compiler caching thin= gs in > > > > > registers. > > > > > > > > We don't want always accessing to memory, it could cause performanc= e degradation. > > > > Like linux kernel, I use it in the place only we really load from m= emory. > > > > > > > Thats not true at all. Every single read of adapter->down_idx in > > > memnic_xmit_pkts() is wrapped in a ACCESS_ONCE call. Theres no diffe= rence in > > > doing that and just declaring a volitile variable and pointing it to > > > &adapter->down_idx (save for the increased legibility of the code) > > > > You're right, at this moment there is no reference without ACCESS_ONCE. > > I'm not sure adding code to access that variable in the future, but > > would like to avoid accidentally a code which causes a performance issu= e, > > I think keeping the declaration in structure without volatile. > > As you mentioned, using local variable which points down_idx will be fi= ne. > So you would like to continue using a macro incorrectly to avoid a possib= le > performance issue with code that hasn't been written yet? Thats nonsensi= cal. No, I will replace ACCESS_ONCE macro with local volatile variable, then ACCESS_ONCE macro will disappear. thanks, Hiroshi > What performance issue to see occuring if you created a volatile variable= and > then used it in conjunction with ACCESS_ONCE? >=20 > Neil >=20 > > > > I will submit a cleanup patch before starting the next development for = DPDK v1.8. > > > > thanks, > > Hiroshi > > > > > > > > Neil > > > > > > > thanks, > > > > Hiroshi > > > > > > > > > > > > > > /Bruce > > > > > > > > > > > > > > > > > thanks, > > > > > > Hiroshi > > > > > > > > > > > > > > > > > > > > Neil > > > > > > > > > > > >