From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1DA2DA04B5; Thu, 29 Oct 2020 15:43:01 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 30820CE6A; Thu, 29 Oct 2020 15:42:59 +0100 (CET) Received: from qrelay176.mxroute.com (qrelay176.mxroute.com [172.82.139.176]) by dpdk.org (Postfix) with ESMTP id 6687ECE50 for ; Thu, 29 Oct 2020 15:42:56 +0100 (CET) Received: from filter003.mxroute.com ([168.235.111.26] 168-235-111-26.cloud.ramnode.com) (Authenticated sender: mN4UYu2MZsgR) by qrelay176.mxroute.com (ZoneMTA) with ESMTPA id 17574cfb69b0004441.002 for ; Thu, 29 Oct 2020 14:42:51 +0000 X-Zone-Loop: 20a58d83902e71c650f302fd10a4d1309473c9bdc18c X-Originating-IP: [168.235.111.26] Received: from echo.mxrouting.net (echo.mxrouting.net [116.202.222.109]) by filter003.mxroute.com (Postfix) with ESMTPS id 6ACA86004E; Thu, 29 Oct 2020 14:42:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ashroe.eu; s=x; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=E8NtjQCACtYoXgMZ4VA5uDxb80zF2VJ5Q1ifhC8rB4M=; b=TkdcgXuSa9xVjEFQ909UQIwCuS dchngyKRBzC0LaImFvB2YWjXM48V4uETXR6RLOzUL/V8N6N5ClD2PbJi1aCTITxXkkE5oosFRN298 T1Z1BhjpLersRLp67nQRd6CnVUx6g8+z4kyExuDgrs38ZfmBpx3MmrJkdF0xXaFuj3QuwzAIcNPt+ YfSEaK5laDDGdFG0SUk9UAmuF4EePoogO1S5orX9g0sHT4ATa/jWSOJHhPYMQ54RVpljxy7L2w6Tf fkbUl4Cfr6QXADiDHX3o7cvDaOZ8BiN5JlZOK9QsV8nDN0toFtGyoXZkVWb34iBIt+BqBceeLwfJB vVfhZOnQ==; To: Thomas Monjalon , dev@dpdk.org Cc: ferruh.yigit@intel.com, david.marchand@redhat.com, bruce.richardson@intel.com, olivier.matz@6wind.com, andrew.rybchenko@oktetlabs.ru, jerinj@marvell.com, viacheslavo@nvidia.com, Neil Horman References: <20201029092751.3837177-1-thomas@monjalon.net> <20201029092751.3837177-16-thomas@monjalon.net> From: "Kinsella, Ray" Autocrypt: addr=mdr@ashroe.eu; keydata= mQINBFv8B3wBEAC+5ImcgbIvadt3axrTnt7Sxch3FsmWTTomXfB8YiuHT8KL8L/bFRQSL1f6 ASCHu3M89EjYazlY+vJUWLr0BhK5t/YI7bQzrOuYrl9K94vlLwzD19s/zB/g5YGGR5plJr0s JtJsFGEvF9LL3e+FKMRXveQxBB8A51nAHfwG0WSyx53d61DYz7lp4/Y4RagxaJoHp9lakn8j HV2N6rrnF+qt5ukj5SbbKWSzGg5HQF2t0QQ5tzWhCAKTfcPlnP0GymTBfNMGOReWivi3Qqzr S51Xo7hoGujUgNAM41sxpxmhx8xSwcQ5WzmxgAhJ/StNV9cb3HWIoE5StCwQ4uXOLplZNGnS uxNdegvKB95NHZjRVRChg/uMTGpg9PqYbTIFoPXjuk27sxZLRJRrueg4tLbb3HM39CJwSB++ YICcqf2N+GVD48STfcIlpp12/HI+EcDSThzfWFhaHDC0hyirHxJyHXjnZ8bUexI/5zATn/ux TpMbc/vicJxeN+qfaVqPkCbkS71cHKuPluM3jE8aNCIBNQY1/j87k5ELzg3qaesLo2n1krBH bKvFfAmQuUuJT84/IqfdVtrSCTabvDuNBDpYBV0dGbTwaRfE7i+LiJJclUr8lOvHUpJ4Y6a5 0cxEPxm498G12Z3NoY/mP5soItPIPtLR0rA0fage44zSPwp6cQARAQABtBxSYXkgS2luc2Vs bGEgPG1kckBhc2hyb2UuZXU+iQJUBBMBCAA+FiEEcDUDlKDJaDuJlfZfdJdaH/sCCpsFAlv8 B3wCGyMFCQlmAYAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQdJdaH/sCCptdtRAAl0oE msa+djBVYLIsax+0f8acidtWg2l9f7kc2hEjp9h9aZCpPchQvhhemtew/nKavik3RSnLTAyn B3C/0GNlmvI1l5PFROOgPZwz4xhJKGN7jOsRrbkJa23a8ly5UXwF3Vqnlny7D3z+7cu1qq/f VRK8qFyWkAb+xgqeZ/hTcbJUWtW+l5Zb+68WGEp8hB7TuJLEWb4+VKgHTpQ4vElYj8H3Z94a 04s2PJMbLIZSgmKDASnyrKY0CzTpPXx5rSJ1q+B1FCsfepHLqt3vKSALa3ld6bJ8fSJtDUJ7 JLiU8dFZrywgDIVme01jPbjJtUScW6jONLvhI8Z2sheR71UoKqGomMHNQpZ03ViVWBEALzEt TcjWgJFn8yAmxqM4nBnZ+hE3LbMo34KCHJD4eg18ojDt3s9VrDLa+V9fNxUHPSib9FD9UX/1 +nGfU/ZABmiTuUDM7WZdXri7HaMpzDRJUKI6b+/uunF8xH/h/MHW16VuMzgI5dkOKKv1LejD dT5mA4R+2zBS+GsM0oa2hUeX9E5WwjaDzXtVDg6kYq8YvEd+m0z3M4e6diFeLS77/sAOgaYL 92UcoKD+Beym/fVuC6/55a0e12ksTmgk5/ZoEdoNQLlVgd2INtvnO+0k5BJcn66ZjKn3GbEC VqFbrnv1GnA58nEInRCTzR1k26h9nmS5Ag0EW/wHfAEQAMth1vHr3fOZkVOPfod3M6DkQir5 xJvUW5EHgYUjYCPIa2qzgIVVuLDqZgSCCinyooG5dUJONVHj3nCbITCpJp4eB3PI84RPfDcC hf/V34N/Gx5mTeoymSZDBmXT8YtvV/uJvn+LvHLO4ZJdvq5ZxmDyxfXFmkm3/lLw0+rrNdK5 pt6OnVlCqEU9tcDBezjUwDtOahyV20XqxtUttN4kQWbDRkhT+HrA9WN9l2HX91yEYC+zmF1S OhBqRoTPLrR6g4sCWgFywqztpvZWhyIicJipnjac7qL/wRS+wrWfsYy6qWLIV80beN7yoa6v ccnuy4pu2uiuhk9/edtlmFE4dNdoRf7843CV9k1yRASTlmPkU59n0TJbw+okTa9fbbQgbIb1 pWsAuicRHyLUIUz4f6kPgdgty2FgTKuPuIzJd1s8s6p2aC1qo+Obm2gnBTduB+/n1Jw+vKpt 07d+CKEKu4CWwvZZ8ktJJLeofi4hMupTYiq+oMzqH+V1k6QgNm0Da489gXllU+3EFC6W1qKj tkvQzg2rYoWeYD1Qn8iXcO4Fpk6wzylclvatBMddVlQ6qrYeTmSbCsk+m2KVrz5vIyja0o5Y yfeN29s9emXnikmNfv/dA5fpi8XCANNnz3zOfA93DOB9DBf0TQ2/OrSPGjB3op7RCfoPBZ7u AjJ9dM7VABEBAAGJAjwEGAEIACYWIQRwNQOUoMloO4mV9l90l1of+wIKmwUCW/wHfAIbDAUJ CWYBgAAKCRB0l1of+wIKm3KlD/9w/LOG5rtgtCUWPl4B3pZvGpNym6XdK8cop9saOnE85zWf u+sKWCrxNgYkYP7aZrYMPwqDvilxhbTsIJl5HhPgpTO1b0i+c0n1Tij3EElj5UCg3q8mEc17 c+5jRrY3oz77g7E3oPftAjaq1ybbXjY4K32o3JHFR6I8wX3m9wJZJe1+Y+UVrrjY65gZFxcA thNVnWKErarVQGjeNgHV4N1uF3pIx3kT1N4GSnxhoz4Bki91kvkbBhUgYfNflGURfZT3wIKK +d50jd7kqRouXUCzTdzmDh7jnYrcEFM4nvyaYu0JjSS5R672d9SK5LVIfWmoUGzqD4AVmUW8 pcv461+PXchuS8+zpltR9zajl72Q3ymlT4BTAQOlCWkD0snBoKNUB5d2EXPNV13nA0qlm4U2 GpROfJMQXjV6fyYRvttKYfM5xYKgRgtP0z5lTAbsjg9WFKq0Fndh7kUlmHjuAIwKIV4Tzo75 QO2zC0/NTaTjmrtiXhP+vkC4pcrOGNsbHuaqvsc/ZZ0siXyYsqbctj/sCd8ka2r94u+c7o4l BGaAm+FtwAfEAkXHu4y5Phuv2IRR+x1wTey1U1RaEPgN8xq0LQ1OitX4t2mQwjdPihZQBCnZ wzOrkbzlJMNrMKJpEgulmxAHmYJKgvZHXZXtLJSejFjR0GdHJcL5rwVOMWB8cg== Message-ID: <29547082-92a3-2383-6476-1643f5eb07c2@ashroe.eu> Date: Thu, 29 Oct 2020 14:42:46 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.12.1 MIME-Version: 1.0 In-Reply-To: <20201029092751.3837177-16-thomas@monjalon.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-AuthUser: mdr@ashroe.eu Subject: Re: [dpdk-dev] [PATCH 15/15] mbuf: move pool pointer in hotter first half X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 29/10/2020 09:27, Thomas Monjalon wrote: > The mempool pointer in the mbuf struct is moved > from the second to the first half. > It should increase performance on most systems having 64-byte cache line, > i.e. mbuf is split in two cache lines. > On such system, the first half (also called first cache line) is hotter > than the second one where the pool pointer was. > > Moving this field gives more space to dynfield1. > > This is how the mbuf layout looks like (pahole-style): > > word type name byte size > 0 void * buf_addr; /* 0 + 8 */ > 1 rte_iova_t buf_iova /* 8 + 8 */ > /* --- RTE_MARKER64 rearm_data; */ > 2 uint16_t data_off; /* 16 + 2 */ > uint16_t refcnt; /* 18 + 2 */ > uint16_t nb_segs; /* 20 + 2 */ > uint16_t port; /* 22 + 2 */ > 3 uint64_t ol_flags; /* 24 + 8 */ > /* --- RTE_MARKER rx_descriptor_fields1; */ > 4 uint32_t union packet_type; /* 32 + 4 */ > uint32_t pkt_len; /* 36 + 4 */ > 5 uint16_t data_len; /* 40 + 2 */ > uint16_t vlan_tci; /* 42 + 2 */ > 5.5 uint64_t union hash; /* 44 + 8 */ > 6.5 uint16_t vlan_tci_outer; /* 52 + 2 */ > uint16_t buf_len; /* 54 + 2 */ > 7 struct rte_mempool * pool; /* 56 + 8 */ > /* --- RTE_MARKER cacheline1; */ > 8 struct rte_mbuf * next; /* 64 + 8 */ > 9 uint64_t union tx_offload; /* 72 + 8 */ > 10 uint16_t priv_size; /* 80 + 2 */ > uint16_t timesync; /* 82 + 2 */ > uint32_t seqn; /* 84 + 4 */ > 11 struct rte_mbuf_ext_shared_info * shinfo; /* 88 + 8 */ > 12 uint64_t dynfield1[4]; /* 96 + 32 */ > 16 /* --- END 128 */ > > Signed-off-by: Thomas Monjalon > --- > doc/guides/rel_notes/deprecation.rst | 5 ----- > lib/librte_kni/rte_kni_common.h | 3 ++- > lib/librte_mbuf/rte_mbuf_core.h | 5 ++--- > 3 files changed, 4 insertions(+), 9 deletions(-) > > diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst > index 72dbb25b83..07ca1dcbb2 100644 > --- a/doc/guides/rel_notes/deprecation.rst > +++ b/doc/guides/rel_notes/deprecation.rst > @@ -88,11 +88,6 @@ Deprecation Notices > > - ``seqn`` > > - As a consequence, the layout of the ``struct rte_mbuf`` will be re-arranged, > - avoiding impact on vectorized implementation of the driver datapaths, > - while evaluating performance gains of a better use of the first cache line. > - > - > * ethdev: the legacy filter API, including > ``rte_eth_dev_filter_supported()``, ``rte_eth_dev_filter_ctrl()`` as well > as filter types MACVLAN, ETHERTYPE, FLEXIBLE, SYN, NTUPLE, TUNNEL, FDIR, > diff --git a/lib/librte_kni/rte_kni_common.h b/lib/librte_kni/rte_kni_common.h > index 36d66e2ffa..ffb3182731 100644 > --- a/lib/librte_kni/rte_kni_common.h > +++ b/lib/librte_kni/rte_kni_common.h > @@ -84,10 +84,11 @@ struct rte_kni_mbuf { > char pad2[4]; > uint32_t pkt_len; /**< Total pkt len: sum of all segment data_len. */ > uint16_t data_len; /**< Amount of data in segment buffer. */ > + char pad3[14]; > + void *pool; > > /* fields on second cache line */ > __attribute__((__aligned__(RTE_CACHE_LINE_MIN_SIZE))) > - void *pool; > void *next; /**< Physical address of next mbuf in kernel. */ > }; > > diff --git a/lib/librte_mbuf/rte_mbuf_core.h b/lib/librte_mbuf/rte_mbuf_core.h > index 52ca1c842f..ee185fa32b 100644 > --- a/lib/librte_mbuf/rte_mbuf_core.h > +++ b/lib/librte_mbuf/rte_mbuf_core.h > @@ -584,12 +584,11 @@ struct rte_mbuf { > > uint16_t buf_len; /**< Length of segment buffer. */ > > - uint64_t unused; > + struct rte_mempool *pool; /**< Pool from which mbuf was allocated. */ > > /* second cache line - fields only used in slow path or on TX */ > RTE_MARKER cacheline1 __rte_cache_min_aligned; > > - struct rte_mempool *pool; /**< Pool from which mbuf was allocated. */ > struct rte_mbuf *next; /**< Next segment of scattered packet. */ > > /* fields to support TX offloads */ > @@ -646,7 +645,7 @@ struct rte_mbuf { > */ > struct rte_mbuf_ext_shared_info *shinfo; > > - uint64_t dynfield1[3]; /**< Reserved for dynamic fields. */ > + uint64_t dynfield1[4]; /**< Reserved for dynamic fields. */ > } __rte_cache_aligned; > > /** > I will let other chime in on the merits of positioning cache alignment of the mempool pointer. >From the ABI PoV, depreciate notice has been observed and since mbuf effects everything doing it outside of a ABI Breakage window is impossible, so it now or never. Ray K