From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 80857A04DC; Sat, 31 Oct 2020 21:41:01 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id DC75C54AE; Sat, 31 Oct 2020 21:40:59 +0100 (CET) Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com [66.111.4.221]) by dpdk.org (Postfix) with ESMTP id 076B65323 for ; Sat, 31 Oct 2020 21:40:58 +0100 (CET) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailnew.nyi.internal (Postfix) with ESMTP id 444585800D0; Sat, 31 Oct 2020 16:40:57 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Sat, 31 Oct 2020 16:40:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s=fm2; bh= Z5REPXarmSu1ZFkyofsvv+oc5RsLDS7SuaMZXty/1yM=; b=UJeu7yo29MUNBMaE ASSNjIZK4pJ2pfo7UrjULrG6xqXc5pU6rQiyreS0CJ7GgpD3bTBXIDkr2tE7t/wi aJPvB/p7cnLv+IiYMeY90HEK9vmDi5edn+SnHExg0L42xMqJaF+KHNA0WzqM4qlM l2jNHQdCDdPgDrxY0ZU25XGZ5Ye/+c9sun3rvd0e8lO33d5+EfihLelzscuFdnZU xzjOfS6kr+Sej3ezvS61b9pIz/JBDg9bQh1OvezD+h6wlFCvOnbFmUzRGLnUFM5m DAX+F6aBMASxQDwQwSU/g8WwFniapNuhSD3pVk3/t538UyUS29xIi3k9MYE1cndn vHp5iA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=Z5REPXarmSu1ZFkyofsvv+oc5RsLDS7SuaMZXty/1 yM=; b=ViwJv3V5CUAIheBOpU6tZN4CHGw3MbA9GwpL/RijP/q0jxMPRv2DFdanl gDnn7Ks0usnryJG/2oYMcAobeW6D5j3iH/xWyZ5jlkM0wdJ60vHm4smOQVVnRYpO WAfCU0TKgJD1unNn9iHFiQZmYzMNdDnsNmIaGXlK6d+Wy7vzn58+B2Dc/e2qHgST zp9HrYR007P6YQioas6mVwrN0amyjRmkh9kWM5+yUJFDwVH2aK0QI3nLV2OqNGWo VTy6zsVFFBmtbOPrZ/+3DFuclFUf0BeCE+J4kS6ft7LWEcYQ6+AaZHhcQTeswwJm J2ryAlPKjXOTQOypIrJlcgcQuXuPA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedrleejgddugedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkfgjfhgggfgtsehtqhertddttddunecuhfhrohhmpefvhhhomhgr shcuofhonhhjrghlohhnuceothhhohhmrghssehmohhnjhgrlhhonhdrnhgvtheqnecugg ftrfgrthhtvghrnhepfeegffeihfeftedthfdvgfetkeffffdukeevtdevtddvgfevuedu veegvdeggedtnecukfhppeejjedrudefgedrvddtfedrudekgeenucevlhhushhtvghruf hiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehthhhomhgrshesmhhonhhjrghl ohhnrdhnvght X-ME-Proxy: Received: from xps.localnet (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id C4D00328005A; Sat, 31 Oct 2020 16:40:53 -0400 (EDT) From: Thomas Monjalon To: Morten =?ISO-8859-1?Q?Br=F8rup?= Cc: Ajit Khaparde , "Ananyev, Konstantin" , Andrew Rybchenko , dev@dpdk.org, "Yigit, Ferruh" , david.marchand@redhat.com, "Richardson, Bruce" , olivier.matz@6wind.com, jerinj@marvell.com, viacheslavo@nvidia.com, honnappa.nagarahalli@arm.com, maxime.coquelin@redhat.com, stephen@networkplumber.org, hemant.agrawal@nxp.com, viacheslavo@nvidia.com Date: Sat, 31 Oct 2020 21:40:46 +0100 Message-ID: <3458411.u7HY7OY5Un@thomas> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35C613CA@smartserver.smartshare.dk> References: <20201029092751.3837177-1-thomas@monjalon.net> <98CBD80474FA8B44BF855DF32C47DC35C613CA@smartserver.smartshare.dk> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="iso-8859-1" Subject: Re: [dpdk-dev] [PATCH 15/15] mbuf: move pool pointer in hotterfirst half X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Thanks for the thoughts Morten. I believe we need benchmarks of different scenarios with different drivers. 31/10/2020 19:20, Morten Br=F8rup: > Thomas, >=20 > Adding my thoughts to the already detailed feedback on this important pat= ch... >=20 > The first cache line is not inherently "hotter" than the second. The hotn= ess depends on their usage. >=20 > The mbuf cacheline1 marker has the following comment: > /* second cache line - fields only used in slow path or on TX */ >=20 > In other words, the second cache line is intended not to be touched in fa= st path RX. >=20 > I do not think this is true anymore. Not even with simple non-scattered R= X. And regression testing probably didn't catch this, because the tests per= form TX after RX, so the cache miss moved from TX to RX and became a cache = hit in TX instead. (I may be wrong about this claim, but it's not important= for the discussion.) >=20 > I think the right question for this patch is: Can we achieve this - not u= sing the second cache line for fast path RX - again by putting the right fi= elds in the first cache line? >=20 > Probably not in all cases, but perhaps for some... >=20 > Consider the application scenarios. >=20 > When a packet is received, one of three things happens to it: > 1. It is immediately transmitted on one or more ports. > 2. It is immediately discarded, e.g. by a firewall rule. > 3. It is put in some sort of queue, e.g. a ring for the next pipeline sta= ge, or in a QoS queue. >=20 > 1. If the packet is immediately transmitted, the m->tx_offload field in t= he second cache line will be touched by the application and TX function any= way, so we don't need to optimize the mbuf layout for this scenario. >=20 > 2. The second scenario touches m->pool no matter how it is implemented. T= he application can avoid touching m->next by using rte_mbuf_raw_free(), kno= wing that the mbuf came directly from RX and thus no other fields have been= touched. In this scenario, we want m->pool in the first cache line. >=20 > 3. Now, let's consider the third scenario, where RX is followed by enqueu= e into a ring. If the application does nothing but put the packet into a ri= ng, we don't need to move anything into the first cache line. But applicati= ons usually does more... So it is application specific what would be good t= o move to the first cache line: >=20 > A. If the application does not use segmented mbufs, and performs analysis= and preparation for transmission in the initial pipeline stages, and only = the last pipeline stage performs TX, we could move m->tx_offload to the fir= st cache line, which would keep the second cache line cold until the actual= TX happens in the last pipeline stage - maybe even after the packet has wa= ited in a QoS queue for a long time, and its cache lines have gone cold. >=20 > B. If the application uses segmented mbufs on RX, it might make sense mov= ing m->next to the first cache line. (We don't use segmented mbufs, so I'm = not sure about this.) >=20 >=20 > However, reality perhaps beats theory: >=20 > Looking at the E1000 PMD, it seems like even its non-scattered RX functio= n, eth_igb_recv_pkts(), sets m->next. If it only kept its own free pool pre= =2Dinitialized instead... I haven't investigated other PMDs, except briefly= looking at the mlx5 PMD, and it seems like it doesn't touch m->next in RX. >=20 > I haven't looked deeper into how m->pool is being used by RX in PMDs, but= I suppose that it isn't touched in RX. >=20 > > If only we had a performance test where RX was not immediately followed b= y TX, but the packets were passed through a large queue in-between, so RX c= ache misses were not free of charge because they transform TX cache misses = into cache hits instead... > >=20 > Whatever you choose, I am sure that most applications will find it more u= seful than the timestamp. :-)