From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id EA7F3A04FD;
	Wed, 26 Oct 2022 21:44:26 +0200 (CEST)
Received: from [217.70.189.124] (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 879C1400D5;
	Wed, 26 Oct 2022 21:44:26 +0200 (CEST)
Received: from shelob.oktetlabs.ru (shelob.oktetlabs.ru [91.220.146.113])
 by mails.dpdk.org (Postfix) with ESMTP id 2F2AD40041
 for <dev@dpdk.org>; Wed, 26 Oct 2022 21:44:25 +0200 (CEST)
Received: from [192.168.1.126] (unknown [188.242.181.57])
 (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
 (No client certificate requested)
 by shelob.oktetlabs.ru (Postfix) with ESMTPSA id 84D6C5A;
 Wed, 26 Oct 2022 22:44:24 +0300 (MSK)
DKIM-Filter: OpenDKIM Filter v2.11.0 shelob.oktetlabs.ru 84D6C5A
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=oktetlabs.ru;
 s=default; t=1666813464;
 bh=hEpsuD8WQdA/2iLJ3EFGNSFbWU3K38BXwJcj3zs37bs=;
 h=Date:Subject:To:Cc:References:From:In-Reply-To:From;
 b=BdRAl1St1k5UDBy5DgeUsveFDU1g0+KHLlRkQ1qg+LWOUBqMdL6ekBlctitcphdkt
 vuGexQaFk1Wi+dA1SCs+bAmMyugFWigHnvKFlvgcDEPEX6d/v+r4M0FqxYmfA8wXnD
 7XAdxoU17k7xGn55bsRUKUoLio0fGWQ0Ab1/zfN4=
Message-ID: <0736e03c-f4e6-2e09-1f8a-122c2e2c6a06@oktetlabs.ru>
Date: Wed, 26 Oct 2022 22:44:24 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.3.0
Subject: Re: [PATCH] mempool: cache align mempool cache objects
Content-Language: en-US
To: =?UTF-8?Q?Morten_Br=c3=b8rup?= <mb@smartsharesystems.com>,
 olivier.matz@6wind.com, jerinj@marvell.com, thomas@monjalon.net
Cc: bruce.richardson@intel.com, dev@dpdk.org
References: <98CBD80474FA8B44BF855DF32C47DC35D8744E@smartserver.smartshare.dk>
 <20221026144436.71068-1-mb@smartsharesystems.com>
From: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
In-Reply-To: <20221026144436.71068-1-mb@smartsharesystems.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org

On 10/26/22 17:44, Morten Brørup wrote:
> Add __rte_cache_aligned to the objs array.
> 
> It makes no difference in the general case, but if get/put operations are
> always 32 objects, it will reduce the number of memory (or last level
> cache) accesses from five to four 64 B cache lines for every get/put
> operation.
> 
> For readability reasons, an example using 16 objects follows:
> 
> Currently, with 16 objects (128B), we access to 3
> cache lines:
> 
>        ┌────────┐
>        │len     │
> cache │********│---
> line0 │********│ ^
>        │********│ |
>        ├────────┤ | 16 objects
>        │********│ | 128B
> cache │********│ |
> line1 │********│ |
>        │********│ |
>        ├────────┤ |
>        │********│_v_
> cache │        │
> line2 │        │
>        │        │
>        └────────┘
> 
> With the alignment, it is also 3 cache lines:
> 
>        ┌────────┐
>        │len     │
> cache │        │
> line0 │        │
>        │        │
>        ├────────┤---
>        │********│ ^
> cache │********│ |
> line1 │********│ |
>        │********│ |
>        ├────────┤ | 16 objects
>        │********│ | 128B
> cache │********│ |
> line2 │********│ |
>        │********│ v
>        └────────┘---
> 
> However, accessing the objects at the bottom of the mempool cache is a
> special case, where cache line0 is also used for objects.
> 
> Consider the next burst (and any following bursts):
> 
> Current:
>        ┌────────┐
>        │len     │
> cache │        │
> line0 │        │
>        │        │
>        ├────────┤
>        │        │
> cache │        │
> line1 │        │
>        │        │
>        ├────────┤
>        │        │
> cache │********│---
> line2 │********│ ^
>        │********│ |
>        ├────────┤ | 16 objects
>        │********│ | 128B
> cache │********│ |
> line3 │********│ |
>        │********│ |
>        ├────────┤ |
>        │********│_v_
> cache │        │
> line4 │        │
>        │        │
>        └────────┘
> 4 cache lines touched, incl. line0 for len.
> 
> With the proposed alignment:
>        ┌────────┐
>        │len     │
> cache │        │
> line0 │        │
>        │        │
>        ├────────┤
>        │        │
> cache │        │
> line1 │        │
>        │        │
>        ├────────┤
>        │        │
> cache │        │
> line2 │        │
>        │        │
>        ├────────┤
>        │********│---
> cache │********│ ^
> line3 │********│ |
>        │********│ | 16 objects
>        ├────────┤ | 128B
>        │********│ |
> cache │********│ |
> line4 │********│ |
>        │********│_v_
>        └────────┘
> Only 3 cache lines touched, incl. line0 for len.
> 
> Credits go to Olivier Matz for the nice ASCII graphics.
> 
> Signed-off-by: Morten Brørup <mb@smartsharesystems.com>

Reviewed-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>