From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 019ACA04B5; Thu, 29 Oct 2020 11:56:58 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 714E5C9CE; Thu, 29 Oct 2020 11:56:57 +0100 (CET) Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com [66.111.4.230]) by dpdk.org (Postfix) with ESMTP id 6448BC9CA for ; Thu, 29 Oct 2020 11:56:55 +0100 (CET) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailnew.nyi.internal (Postfix) with ESMTP id D9EF5580475; Thu, 29 Oct 2020 06:56:54 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Thu, 29 Oct 2020 06:56:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s=fm2; bh= Q316Z/87mHnpUjtusDrqYU/u3kY6tXB98EXOgMNEJyk=; b=R/2dXGHACIb7hbK9 kGvUr9rawAIgC+9bZv56zPFFXsGSandV2CmYgF7trOhcWFlU4VE95dsfEkFmXR4g vkQXOQSnxoEbUiG7uyuRUix1YX8SQWKO5H8hDS9d1D5wetdfdAQaX5t08Ms/nlcY ZfrtyO12LNdRcH1ido4S5jCmvpGDP3xFo1DzShYWE+k8XOxWEDv7A8jOJydUGLQk F7lFaR6SPK9x13wvUE/t8yoB2Ibp7MsV1eRZWz/nFNI0C8p/ijokFLCl5J+xu0fz yLk+HPYV8i4WN/BTVY/y3HG0LY/+inci6Ul0PE218S24Uzr1K7qJ8l/b21ZPc9id YBgjIQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=Q316Z/87mHnpUjtusDrqYU/u3kY6tXB98EXOgMNEJ yk=; b=Fv74p24SV/Adjma3dw9iFuA7csJocZQmouz1cs46jzQRLF5Gn8LZBiZAm Bqx3WZbQ2w2bWbrJ8NMCrdTbWHdR6AjuFDyf76EJLJm5+53/jzTqf38bbOXPMF/F lHKfiN71FMktd3ayrRmjkFqzKtHN6dNLbfLtsg7z74DU+79stZG6R4uB77cuA3Y0 UhQurVjpGjddQEc9aNp0TeKpZSBfAW/dX69J7r/CagEKNTn4SMVjpyH5lOhkDPRV Glg0UxOgYCmphxkTPo/4ExW6MEqWGG8utOA+OZ890Jv5gCfPIyoGHXSt6r+as3P4 myNUmu6Wcr5k4PeD1DTxZwgXHNpww== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedrleefgddvfecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkjghfggfgtgesthfuredttddtvdenucfhrhhomhepvfhhohhmrghs ucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenucggtf frrghtthgvrhhnpedugefgvdefudfftdefgeelgffhueekgfffhfeujedtteeutdejueei iedvffegheenucfkphepjeejrddufeegrddvtdefrddukeegnecuvehluhhsthgvrhfuih iivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepthhhohhmrghssehmohhnjhgrlhho nhdrnhgvth X-ME-Proxy: Received: from xps.localnet (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id 15BFC328005D; Thu, 29 Oct 2020 06:56:52 -0400 (EDT) From: Thomas Monjalon To: Andrew Rybchenko Cc: dev@dpdk.org, ferruh.yigit@intel.com, david.marchand@redhat.com, bruce.richardson@intel.com, olivier.matz@6wind.com, jerinj@marvell.com, viacheslavo@nvidia.com, ajit.khaparde@broadcom.com, konstantin.ananyev@intel.com, honnappa.nagarahalli@arm.com, maxime.coquelin@redhat.com, stephen@networkplumber.org, hemant.agrawal@nxp.com Date: Thu, 29 Oct 2020 11:56:51 +0100 Message-ID: <2815114.5jsDjZKEVh@thomas> In-Reply-To: <293d3128-7215-c47c-de20-6c510bc8acb3@oktetlabs.ru> References: <20201029092751.3837177-1-thomas@monjalon.net> <20201029092751.3837177-16-thomas@monjalon.net> <293d3128-7215-c47c-de20-6c510bc8acb3@oktetlabs.ru> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Subject: Re: [dpdk-dev] [PATCH 15/15] mbuf: move pool pointer in hotter first half X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" 29/10/2020 11:50, Andrew Rybchenko: > On 10/29/20 12:27 PM, Thomas Monjalon wrote: > > The mempool pointer in the mbuf struct is moved > > from the second to the first half. > > It should increase performance on most systems having 64-byte cache line, > > i.e. mbuf is split in two cache lines. > > On such system, the first half (also called first cache line) is hotter > > than the second one where the pool pointer was. > > > > Moving this field gives more space to dynfield1. > > > > This is how the mbuf layout looks like (pahole-style): > > > > word type name byte size > > 0 void * buf_addr; /* 0 + 8 */ > > 1 rte_iova_t buf_iova /* 8 + 8 */ > > /* --- RTE_MARKER64 rearm_data; */ > > 2 uint16_t data_off; /* 16 + 2 */ > > uint16_t refcnt; /* 18 + 2 */ > > uint16_t nb_segs; /* 20 + 2 */ > > uint16_t port; /* 22 + 2 */ > > 3 uint64_t ol_flags; /* 24 + 8 */ > > /* --- RTE_MARKER rx_descriptor_fields1; */ > > 4 uint32_t union packet_type; /* 32 + 4 */ > > uint32_t pkt_len; /* 36 + 4 */ > > 5 uint16_t data_len; /* 40 + 2 */ > > uint16_t vlan_tci; /* 42 + 2 */ > > 5.5 uint64_t union hash; /* 44 + 8 */ > > 6.5 uint16_t vlan_tci_outer; /* 52 + 2 */ > > uint16_t buf_len; /* 54 + 2 */ > > 7 struct rte_mempool * pool; /* 56 + 8 */ > > /* --- RTE_MARKER cacheline1; */ > > 8 struct rte_mbuf * next; /* 64 + 8 */ > > 9 uint64_t union tx_offload; /* 72 + 8 */ > > 10 uint16_t priv_size; /* 80 + 2 */ > > uint16_t timesync; /* 82 + 2 */ > > uint32_t seqn; /* 84 + 4 */ > > 11 struct rte_mbuf_ext_shared_info * shinfo; /* 88 + 8 */ > > 12 uint64_t dynfield1[4]; /* 96 + 32 */ > > 16 /* --- END 128 */ > > > > Signed-off-by: Thomas Monjalon > > I'd like to understand why pool is chosen instead of, for > example, next pointer. > > Pool is used on housekeeping when driver refills Rx ring or > free completed Tx mbufs. Free thresholds try to avoid it on > every Rx/Tx burst (if possible). > > Next is used for multi-segment Tx and scattered (and buffer > split) Rx. IMHO the key question here is we consider these > use cases as common and priority to optimize. If yes, I'd > vote to have next on the first cacheline. > > I'm not sure. Just trying to hear a bit more about it. That's a good question. Clearly pool and next are good options. The best would be to have some benchmarks. If one use case shows no benefit, the decision is easier. If you prefer, we can leave this last patch for -rc3.