From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 7E1BCAF87 for ; Tue, 9 Sep 2014 13:37:40 +0200 (CEST) Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga102.fm.intel.com with ESMTP; 09 Sep 2014 04:42:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.04,491,1406617200"; d="scan'208";a="596775744" Received: from irsmsx101.ger.corp.intel.com ([163.33.3.153]) by fmsmga002.fm.intel.com with ESMTP; 09 Sep 2014 04:42:41 -0700 Received: from irsmsx109.ger.corp.intel.com (163.33.3.23) by IRSMSX101.ger.corp.intel.com (163.33.3.153) with Microsoft SMTP Server (TLS) id 14.3.195.1; Tue, 9 Sep 2014 12:42:41 +0100 Received: from irsmsx108.ger.corp.intel.com ([169.254.11.157]) by IRSMSX109.ger.corp.intel.com ([169.254.13.200]) with mapi id 14.03.0195.001; Tue, 9 Sep 2014 12:42:40 +0100 From: "De Lara Guarch, Pablo" To: "Richardson, Bruce" , Matthew Hall , "dev@dpdk.org" Thread-Topic: [dpdk-dev] Defaults for rte_hash Thread-Index: AQHPzBl3DCRFmUi6tEq7jcRQIsiXg5v4jV6AgAAgd/A= Date: Tue, 9 Sep 2014 11:42:40 +0000 Message-ID: References: <20140909103153.GA7969@mhcomputing.net> <59AF69C657FD0841A61C55336867B5B0343EFBBD@IRSMSX103.ger.corp.intel.com> In-Reply-To: <59AF69C657FD0841A61C55336867B5B0343EFBBD@IRSMSX103.ger.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [163.33.239.181] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] Defaults for rte_hash X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Sep 2014 11:37:41 -0000 > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Richardson, Bruce > Sent: Tuesday, September 09, 2014 11:45 AM > To: Matthew Hall; dev@dpdk.org > Subject: Re: [dpdk-dev] Defaults for rte_hash >=20 > > -----Original Message----- > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Matthew Hall > > Sent: Tuesday, September 09, 2014 11:32 AM > > To: dev@dpdk.org > > Subject: [dpdk-dev] Defaults for rte_hash > > > > Hello, > > > > I was looking at the code which inits rte_hash objects in examples/l3fw= d. > It's > > using approx. 1M to 4M hash 'entries' depending on 32-bit vs 64-bit, bu= t it's > > setting the 'bucket_entries' to just 4. > > > > Normally I'm used to using somewhat deeper hash buckets than that... it > seems > > like having a zillion little tiny hash buckets would cause more TLB pre= ssure > > and memory overhead... or does 4 get shifted / exponentiated into 2**4 = ? > > That 4 is not shifted, so it is actually 4 entries/bucket. Actually, the ma= ximum number of entries you can use is 16, as bucket will be as big as a ca= che line. However, regardless the number of entries, memory size will remain the same= , but using 4 entries/bucket, with 16-byte key, all keys stored for a bucke= t will fit in a cache line,=20 so performance looks to be better in this case (although a non-optimal hash= function could lead not to be able to store all keys, as chances to fill a= bucket are higher). Anyway, for this example, 4 entries/bucket looks a good number to me. > > The documentation in > > http://dpdk.org/doc/api/structrte__hash__parameters.html > > and http://dpdk.org/doc/api/rte__hash_8h.html isn't that clear... is th= ere a > > better place to look for this? > > > > In my case I'm looking to create a table of 4M or 8M entries, containin= g > > tables of security threat IPs / domains, to be detected in the traffic.= So it > > would be good to have some understanding how not to waste a ton of > memory > > on a > > table this huge without making it run super slow either. > > > > Did anybody have some experience with how to get this right? >=20 > It might be worth looking too at the hash table structures in the librte_= table > directory for packet framework. These should give better scalability acro= ss > millions of flows than the existing rte_hash implementation. [We're looki= ng > here to provide in the future a similar, more scalable, hash table > implementation with an API like that of rte_hash, but that is still under > development here at the moment.] >=20 > > > > Another thing... the LPM table uses 16-bit Hop IDs. But I would probabl= y > have > > more than 64K CIDR blocks of badness on the Internet available to me fo= r > > analysis. How would I cope with this, besides just letting some attacke= rs > > escape unnoticed? ;) >=20 > Actually, I think the next hop field in the lpm implementation is only 8-= bits, > not 16 :-). Each lpm entry is only 16-bits in total. >=20 > > > > Have we got some kind of structure which allows a greater number of > CIDRs > > even > > if it's not quite as fast? > > > > Thanks, > > Matthew.