From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8237AA04B5; Thu, 29 Oct 2020 11:50:41 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E550CC9B2; Thu, 29 Oct 2020 11:50:38 +0100 (CET) Received: from shelob.oktetlabs.ru (shelob.oktetlabs.ru [91.220.146.113]) by dpdk.org (Postfix) with ESMTP id 1C5C1C9AC for ; Thu, 29 Oct 2020 11:50:36 +0100 (CET) Received: from [192.168.38.17] (aros.oktetlabs.ru [192.168.38.17]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by shelob.oktetlabs.ru (Postfix) with ESMTPSA id 9C0287F4F0; Thu, 29 Oct 2020 13:50:34 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 shelob.oktetlabs.ru 9C0287F4F0 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=oktetlabs.ru; s=default; t=1603968634; bh=l5SUZkfIxHuJht2g9qsKfa1tH+yl6hyLpqR1NsACpAg=; h=Subject:To:Cc:References:From:Date:In-Reply-To; b=wqc3BfdKWvxixsoHCOEbNH65Cx4yvQKN4tI88mlR6owrp7PZiunri8VPxDDqBQNdi 1Mwi1lc1g7z+o7FdK5azb581Ag1XmSkpT4N/u20BZjTZkvMkEkbRm+4BLk7i95ADLk 4vOzj+EYt0HI05ZGrL5tr/npIw6ChdK2fgT5G/Nc= To: Thomas Monjalon , dev@dpdk.org Cc: ferruh.yigit@intel.com, david.marchand@redhat.com, bruce.richardson@intel.com, olivier.matz@6wind.com, jerinj@marvell.com, viacheslavo@nvidia.com, Ray Kinsella , Neil Horman References: <20201029092751.3837177-1-thomas@monjalon.net> <20201029092751.3837177-16-thomas@monjalon.net> From: Andrew Rybchenko Organization: OKTET Labs Message-ID: <293d3128-7215-c47c-de20-6c510bc8acb3@oktetlabs.ru> Date: Thu, 29 Oct 2020 13:50:34 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0 MIME-Version: 1.0 In-Reply-To: <20201029092751.3837177-16-thomas@monjalon.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH 15/15] mbuf: move pool pointer in hotter first half X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 10/29/20 12:27 PM, Thomas Monjalon wrote: > The mempool pointer in the mbuf struct is moved > from the second to the first half. > It should increase performance on most systems having 64-byte cache line, > i.e. mbuf is split in two cache lines. > On such system, the first half (also called first cache line) is hotter > than the second one where the pool pointer was. > > Moving this field gives more space to dynfield1. > > This is how the mbuf layout looks like (pahole-style): > > word type name byte size > 0 void * buf_addr; /* 0 + 8 */ > 1 rte_iova_t buf_iova /* 8 + 8 */ > /* --- RTE_MARKER64 rearm_data; */ > 2 uint16_t data_off; /* 16 + 2 */ > uint16_t refcnt; /* 18 + 2 */ > uint16_t nb_segs; /* 20 + 2 */ > uint16_t port; /* 22 + 2 */ > 3 uint64_t ol_flags; /* 24 + 8 */ > /* --- RTE_MARKER rx_descriptor_fields1; */ > 4 uint32_t union packet_type; /* 32 + 4 */ > uint32_t pkt_len; /* 36 + 4 */ > 5 uint16_t data_len; /* 40 + 2 */ > uint16_t vlan_tci; /* 42 + 2 */ > 5.5 uint64_t union hash; /* 44 + 8 */ > 6.5 uint16_t vlan_tci_outer; /* 52 + 2 */ > uint16_t buf_len; /* 54 + 2 */ > 7 struct rte_mempool * pool; /* 56 + 8 */ > /* --- RTE_MARKER cacheline1; */ > 8 struct rte_mbuf * next; /* 64 + 8 */ > 9 uint64_t union tx_offload; /* 72 + 8 */ > 10 uint16_t priv_size; /* 80 + 2 */ > uint16_t timesync; /* 82 + 2 */ > uint32_t seqn; /* 84 + 4 */ > 11 struct rte_mbuf_ext_shared_info * shinfo; /* 88 + 8 */ > 12 uint64_t dynfield1[4]; /* 96 + 32 */ > 16 /* --- END 128 */ > > Signed-off-by: Thomas Monjalon I'd like to understand why pool is chosen instead of, for example, next pointer. Pool is used on housekeeping when driver refills Rx ring or free completed Tx mbufs. Free thresholds try to avoid it on every Rx/Tx burst (if possible). Next is used for multi-segment Tx and scattered (and buffer split) Rx. IMHO the key question here is we consider these use cases as common and priority to optimize. If yes, I'd vote to have next on the first cacheline. I'm not sure. Just trying to hear a bit more about it.