From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id E34921B72D for ; Thu, 10 May 2018 01:13:02 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 09 May 2018 16:13:01 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.49,383,1520924400"; d="scan'208";a="227258069" Received: from fyigit-mobl.ger.corp.intel.com (HELO [10.237.221.77]) ([10.237.221.77]) by fmsmga006.fm.intel.com with ESMTP; 09 May 2018 16:13:00 -0700 To: Yongseok Koh , adrien.mazarguil@6wind.com, nelio.laranjeiro@6wind.com Cc: dev@dpdk.org References: <20180502231654.7596-1-yskoh@mellanox.com> <20180509110906.19462-1-yskoh@mellanox.com> <20180509110906.19462-5-yskoh@mellanox.com> From: Ferruh Yigit Openpgp: preference=signencrypt Autocrypt: addr=ferruh.yigit@intel.com; prefer-encrypt=mutual; keydata= xsFNBFXZCFABEADCujshBOAaqPZpwShdkzkyGpJ15lmxiSr3jVMqOtQS/sB3FYLT0/d3+bvy qbL9YnlbPyRvZfnP3pXiKwkRoR1RJwEo2BOf6hxdzTmLRtGtwWzI9MwrUPj6n/ldiD58VAGQ +iR1I/z9UBUN/ZMksElA2D7Jgg7vZ78iKwNnd+vLBD6I61kVrZ45Vjo3r+pPOByUBXOUlxp9 GWEKKIrJ4eogqkVNSixN16VYK7xR+5OUkBYUO+sE6etSxCr7BahMPKxH+XPlZZjKrxciaWQb +dElz3Ab4Opl+ZT/bK2huX+W+NJBEBVzjTkhjSTjcyRdxvS1gwWRuXqAml/sh+KQjPV1PPHF YK5LcqLkle+OKTCa82OvUb7cr+ALxATIZXQkgmn+zFT8UzSS3aiBBohg3BtbTIWy51jNlYdy ezUZ4UxKSsFuUTPt+JjHQBvF7WKbmNGS3fCid5Iag4tWOfZoqiCNzxApkVugltxoc6rG2TyX CmI2rP0mQ0GOsGXA3+3c1MCdQFzdIn/5tLBZyKy4F54UFo35eOX8/g7OaE+xrgY/4bZjpxC1 1pd66AAtKb3aNXpHvIfkVV6NYloo52H+FUE5ZDPNCGD0/btFGPWmWRmkPybzColTy7fmPaGz cBcEEqHK4T0aY4UJmE7Ylvg255Kz7s6wGZe6IR3N0cKNv++O7QARAQABzSVGZXJydWggWWln aXQgPGZlcnJ1aC55aWdpdEBpbnRlbC5jb20+wsF+BBMBAgAoAhsDBgsJCAcDAgYVCAIJCgsE FgIDAQIeAQIXgAUCWZR3VQUJB33WBQAKCRD5M+tD3xNhH6DWEACVhEb8q1epPwZrUDoxzu7E TS1b8tmabOmnjXZRs6+EXgUVHkp2xxkCfDmL3pa5bC0G/74aJnWjNsdvE05V1cb4YK4kRQ62 FwDQ+hlrFrwFB3PtDZk1tpkzCRHvJgnIil+0MuEh32Y57ig6hy8yO8ql7Lohyrnpfk/nNpm4 jQGEF5qEeHcEFe1AZQlPHN/STno8NZSz2nl0b2cw+cujN1krmvB52Ah/2KugQ6pprVyrGrzB c34ZQO9OsmSjJlETCZk6EZzuhfe16iqBFbOSadi9sPcJRwaUQBid+xdFWl7GQ8qC3zNPibSF HmU43yBZUqJDZlhIcl6/cFpOSjv2sDWdtjEXTDn5y/0FsuY0mFE78ItC4kCTIVk17VZoywcd fmbbnwOSWzDq7hiUYuQGkIudJw5k/A1CMsyLkoUEGN3sLfsw6KASgS4XrrmPO4UVr3mH5bP1 yC7i1OVNpzvOxtahmzm481ID8sk72GC2RktTOHb0cX+qdoiMMfYgo3wRRDYCBt6YoGYUxF1p msjocXyqToKhhnFbXLaZlVfnQ9i2i8jsj9SKig+ewC2p3lkPj6ncye9q95bzhmUeJO6sFhJg Hiz6syOMg8yCcq60j07airybAuHIDNFWk0gaWAmtHZxLObZx2PVn2nv9kLYGohFekw0AOsIW ta++5m48dnCoAc7BTQRX1ky+ARAApzQNvXvE2q1LAS+Z+ni2R13Bb1cDS1ZYq1jgpR13+OKN ipzd8MPngRJilXxBaPTErhgzR0vGcNTYhjGMSyFIHVOoBq1VbP1a0Fi/NqWzJOowo/fDfgVy K4vuitc/gCJs+2se4hdZA4EQJxVlNM51lgYDNpjPGIA43MX15OLAip73+ho6NPBMuc5qse3X pAClNhBKfENRCWN428pi3WVkT+ABRTE0taxjJNP7bb+9TQYNRqGwnGzX5/XISv44asWIQCaq vOkXSUJLd//cdVNTqtL1wreCVVR5pMXj7VIrlk07fmmJVALCmGbFr53BMb8O+8dgK2A5mitM n44d+8KdJWOwziRxcaMk/LclmZS3Iv1TERtiWt98Y9AjeAtcgYPkA3ld0BcUKONogP8pHVz1 Ed3s5rDQ91yr1S0wuAzW91fxGUO4wY+uPmxCtFVuBgd9VT9NAKTUL0qHM7CDgCnZPe0TW6Zj 8OqtdCCyAfvU9cW5xWM7Icxhde6AtPxhDSBwE8fL2ZmrDmaA4jmUKXp3i4JxRPSX84S08b+s DWXHPxy10UFU5A7EK/BEbZAKBwn9ROfm+WK+6X5xOGLoRE++OqNuUudxC1GDyLOPaqCbBCS9 +P6HsTHzxsjyJa27n4jcrcuY3P9TEcFJYSZSeSDh8mVGvugi0exnSJrrBZDyVCcAEQEAAcLB ZQQYAQIADwIbDAUCWZR1ZwUJA59cIQAKCRD5M+tD3xNhH5b+D/9XG44Ci6STdcA5RO/ur05J EE3Ux1DCHZ5V7vNAtX/8Wg4l4GZfweauXwuJ1w7Sp7fklwcNC6wsceI+EmNjGMqfIaukGetG +jBGqsQ7moOZodfXUoCK98gblKgt/BPYMVidzlGC8Q/+lZg1+o29sPnwImW+MXt/Z5az/Z17 Qc265g+p5cqJHzq6bpQdnF7Fu6btKU/kv6wJghENvgMXBuyThqsyFReJWFh2wfaKyuix3Zyj ccq7/blkhzIKmtFWgDcgaSc2UAuJU+x9nuYjihW6WobpKP/nlUDu3BIsbIq09UEke+uE/QK+ FJ8PTJkAsXOf1Bc2C0XbW4Y2hf103+YY6L8weUCBsWC5VH5VtVmeuh26ENURclwfeXhWQ9Og 77yzpTXWr5g1Z0oLpYpWPv745J4bE7pv+dzxOrFdM1xNkzY2pvXph/A8OjxZNQklDkHQ7PIB Lki5L2F4XkEOddUUQchJwzMqTPsggPDmGjgLZrqgO+s4ECZK5+nLD3HEpAbPa3JLDaScy+90 Nu1lAqPUHSnP3vYZVw85ZYm6UCxHE4VLMnnJsN09ZhsOSVR+GyP5Nyw9rT1V3lcsuH7M5Naa 2Xobn9m7l9bRCD/Ji8kG15eV1WTxx1HXVQGjdUYDI7UwegBNbwMLh17XDy+3sn/6SgcqtECA Q6pZKA2mTQxEKMLBZQQYAQIADwIbDAUCWZR3hQUJA59eRwAKCRD5M+tD3xNhH4a/D/4jLAZu UhvU1swWcNEVVCELZ0D3LOV14XcY2MXa3QOpeZ9Bgq7YYJ4S5YXK+SBQS0FkRZdjGNvlGZoG ZdpU+NsQmQFhqHGwX0IT9MeTFM8uvKgxNKGwMVcV9g0IOqwBhGHne+BFboRA9362fgGW5AYQ zT0mzzRKEoOh4r3AQvbM6kLISxo0k1ujdYiI5nj/5WoKDqxTwwfuN1uDUHsWo3tzenRmpMyU NyW3Dc+1ajvXLyo09sRRq7BnM99Rix1EGL8Qhwy+j0YAv+FuspWxUX9FxXYho5PvGLHLsHfK FYQ7x/RRbpMjkJWVfIe/xVnfvn4kz+MTA5yhvsuNi678fLwY9hBP0y4lO8Ob2IhEPdfnTuIs tFVxXuelJ9xAe5TyqP0f+fQjf1ixsBZkqOohsBXDfje0iaUpYa/OQ/BBeej0dUdg2JEu4jAC x41HpVCnP9ipLpD0fYz1d/dX0F/VY2ovW6Eba/y/ngOSAR6C+u881m7oH2l0G47MTwkaQCBA bLGXPj4TCdX3lftqt4bcBPBJ+rFAnJmRHtUuyyaewBnZ81ZU2YAptqFM1kTh+aSvMvGhfVsQ qZL2rk2OPN1hg+KXhErlbTZ6oPtLCFhSHQmuxQ4oc4U147wBTUuOdwNjtnNatUhRCp8POc+3 XphVR5G70mnca1E2vzC77z+XSlTyRA== Message-ID: <274843e9-ed82-9576-baf8-a704babf64c5@intel.com> Date: Thu, 10 May 2018 00:12:59 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <20180509110906.19462-5-yskoh@mellanox.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH v2 4/4] net/mlx4: add new Memory Region support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 May 2018 23:13:03 -0000 On 5/9/2018 12:09 PM, Yongseok Koh wrote: > This is the new design of Memory Region (MR) for mlx PMD, in order to: > - Accommodate the new memory hotplug model. > - Support non-contiguous Mempool. > > There are multiple layers for MR search. > > L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most > Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized > array by linear search. L0/L1 is in an inline function - > mlx4_mr_lookup_cache(). > > If L1 misses, the bottom-half function is called to look up the address > from the bigger local cache of the queue. This is L2 - mlx4_mr_addr2mr_bh() > and it is not an inline function. Data structure for L2 is the Binary Tree. > > If L2 misses, the search falls into the slowest path which takes locks in > order to access global device cache (priv->mr.cache) which is also a B-tree > and caches the original MR list (priv->mr.mr_list) of the device. Unless > the global cache is overflowed, it is all-inclusive of the MR list. This is > L3 - mlx4_mr_lookup_dev(). The size of the L3 cache table is limited and > can't be expanded on the fly due to deadlock. Refer to the comments in the > code for the details - mr_lookup_dev(). If L3 is overflowed, the list will > have to be searched directly bypassing the cache although it is slower. > > If L3 misses, a new MR for the address should be created - > mlx4_mr_create(). When it creates a new MR, it tries to register adjacent > memsegs as much as possible which are virtually contiguous around the > address. This must take two locks - memory_hotplug_lock and > priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any > allocation/free of memory inside. > > In the free callback of the memory hotplug event, freed space is searched > from the MR list and corresponding bits are cleared from the bitmap of MRs. > This can fragment a MR and the MR will have multiple search entries in the > caches. Once there's a change by the event, the global cache must be > rebuilt and all the per-queue caches will be flushed as well. If memory is > frequently freed in run-time, that may cause jitter on dataplane processing > in the worst case by incurring MR cache flush and rebuild. But, it would be > the least probable scenario. > > To guarantee the most optimal performance, it is highly recommended to use > an EAL option - '--socket-mem'. Then, the reserved memory will be pinned > and won't be freed dynamically. And it is also recommended to configure > per-lcore cache of Mempool. Even though there're many MRs for a device or > MRs are highly fragmented, the cache of Mempool will be much helpful to > reduce misses on per-queue caches anyway. > > '--legacy-mem' is also supported. > > Signed-off-by: Yongseok Koh <...> > +/** > + * Insert an entry to B-tree lookup table. > + * > + * @param bt > + * Pointer to B-tree structure. > + * @param entry > + * Pointer to new entry to insert. > + * > + * @return > + * 0 on success, -1 on failure. > + */ > +static int > +mr_btree_insert(struct mlx4_mr_btree *bt, struct mlx4_mr_cache *entry) > +{ > + struct mlx4_mr_cache *lkp_tbl; > + uint16_t idx = 0; > + size_t shift; > + > + assert(bt != NULL); > + assert(bt->len <= bt->size); > + assert(bt->len > 0); > + lkp_tbl = *bt->table; > + /* Find out the slot for insertion. */ > + if (mr_btree_lookup(bt, &idx, entry->start) != UINT32_MAX) { > + DEBUG("abort insertion to B-tree(%p):" > + " already exist at idx=%u [0x%lx, 0x%lx) lkey=0x%x", > + (void *)bt, idx, entry->start, entry->end, entry->lkey); This and various other logs causing 32bits build error because of %lx usage. Can you please check them? I am feeling sad to complain a patch like this just because of log format issue, we should find a solution to this issue as community, either checkpatch checks or automated 32bit builds, I don't know.