From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from dpdk.org (dpdk.org [92.243.14.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 019ACA04B5;
	Thu, 29 Oct 2020 11:56:58 +0100 (CET)
Received: from [92.243.14.124] (localhost [127.0.0.1])
	by dpdk.org (Postfix) with ESMTP id 714E5C9CE;
	Thu, 29 Oct 2020 11:56:57 +0100 (CET)
Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com
 [66.111.4.230]) by dpdk.org (Postfix) with ESMTP id 6448BC9CA
 for <dev@dpdk.org>; Thu, 29 Oct 2020 11:56:55 +0100 (CET)
Received: from compute2.internal (compute2.nyi.internal [10.202.2.42])
 by mailnew.nyi.internal (Postfix) with ESMTP id D9EF5580475;
 Thu, 29 Oct 2020 06:56:54 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute2.internal (MEProxy); Thu, 29 Oct 2020 06:56:54 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h=
 from:to:cc:subject:date:message-id:in-reply-to:references
 :mime-version:content-transfer-encoding:content-type; s=fm2; bh=
 Q316Z/87mHnpUjtusDrqYU/u3kY6tXB98EXOgMNEJyk=; b=R/2dXGHACIb7hbK9
 kGvUr9rawAIgC+9bZv56zPFFXsGSandV2CmYgF7trOhcWFlU4VE95dsfEkFmXR4g
 vkQXOQSnxoEbUiG7uyuRUix1YX8SQWKO5H8hDS9d1D5wetdfdAQaX5t08Ms/nlcY
 ZfrtyO12LNdRcH1ido4S5jCmvpGDP3xFo1DzShYWE+k8XOxWEDv7A8jOJydUGLQk
 F7lFaR6SPK9x13wvUE/t8yoB2Ibp7MsV1eRZWz/nFNI0C8p/ijokFLCl5J+xu0fz
 yLk+HPYV8i4WN/BTVY/y3HG0LY/+inci6Ul0PE218S24Uzr1K7qJ8l/b21ZPc9id
 YBgjIQ==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-transfer-encoding:content-type
 :date:from:in-reply-to:message-id:mime-version:references
 :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender
 :x-sasl-enc; s=fm1; bh=Q316Z/87mHnpUjtusDrqYU/u3kY6tXB98EXOgMNEJ
 yk=; b=Fv74p24SV/Adjma3dw9iFuA7csJocZQmouz1cs46jzQRLF5Gn8LZBiZAm
 Bqx3WZbQ2w2bWbrJ8NMCrdTbWHdR6AjuFDyf76EJLJm5+53/jzTqf38bbOXPMF/F
 lHKfiN71FMktd3ayrRmjkFqzKtHN6dNLbfLtsg7z74DU+79stZG6R4uB77cuA3Y0
 UhQurVjpGjddQEc9aNp0TeKpZSBfAW/dX69J7r/CagEKNTn4SMVjpyH5lOhkDPRV
 Glg0UxOgYCmphxkTPo/4ExW6MEqWGG8utOA+OZ890Jv5gCfPIyoGHXSt6r+as3P4
 myNUmu6Wcr5k4PeD1DTxZwgXHNpww==
X-ME-Sender: <xms:9Z-aX2yAn2ZljFjHD7qUIp0qNhmfWcvUp9wnPTl8wlZ4rU5Kb60AZQ>
 <xme:9Z-aXyTS0FJ9DJgJZbZau6EIbbhxJEH1JjnI99EOeFBXwOCwFFuUdKqb-kUkfUSIo
 W1eZBJ5oRyir1bihg>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedrleefgddvfecutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefhvffufffkjghfggfgtgesthfuredttddtvdenucfhrhhomhepvfhhohhmrghs
 ucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenucggtf
 frrghtthgvrhhnpedugefgvdefudfftdefgeelgffhueekgfffhfeujedtteeutdejueei
 iedvffegheenucfkphepjeejrddufeegrddvtdefrddukeegnecuvehluhhsthgvrhfuih
 iivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepthhhohhmrghssehmohhnjhgrlhho
 nhdrnhgvth
X-ME-Proxy: <xmx:9Z-aX4UpRsh65u_RV3v99YOraIBVjvWbhaSPhvfcihQ1C2OPeXGrAw>
 <xmx:9Z-aX8iXrDAsfSa9e-LySpd7EcDXr_54mbhR_6t9RJCesqYACw4zfA>
 <xmx:9Z-aX4AUfmIjOoS83n71WqPUk86zms7oESQ7_02GoKgomNmClLK_8A>
 <xmx:9p-aXxLCgTPaQGUsQc__4C7ZzMLL4I_hLFSz0OFIWeRb7d1s0q5Yvw>
Received: from xps.localnet (184.203.134.77.rev.sfr.net [77.134.203.184])
 by mail.messagingengine.com (Postfix) with ESMTPA id 15BFC328005D;
 Thu, 29 Oct 2020 06:56:52 -0400 (EDT)
From: Thomas Monjalon <thomas@monjalon.net>
To: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Cc: dev@dpdk.org, ferruh.yigit@intel.com, david.marchand@redhat.com,
 bruce.richardson@intel.com, olivier.matz@6wind.com, jerinj@marvell.com,
 viacheslavo@nvidia.com, ajit.khaparde@broadcom.com,
 konstantin.ananyev@intel.com, honnappa.nagarahalli@arm.com,
 maxime.coquelin@redhat.com, stephen@networkplumber.org, hemant.agrawal@nxp.com
Date: Thu, 29 Oct 2020 11:56:51 +0100
Message-ID: <2815114.5jsDjZKEVh@thomas>
In-Reply-To: <293d3128-7215-c47c-de20-6c510bc8acb3@oktetlabs.ru>
References: <20201029092751.3837177-1-thomas@monjalon.net>
 <20201029092751.3837177-16-thomas@monjalon.net>
 <293d3128-7215-c47c-de20-6c510bc8acb3@oktetlabs.ru>
MIME-Version: 1.0
Content-Transfer-Encoding: 7Bit
Content-Type: text/plain; charset="us-ascii"
Subject: Re: [dpdk-dev] [PATCH 15/15] mbuf: move pool pointer in hotter
	first half
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>

29/10/2020 11:50, Andrew Rybchenko:
> On 10/29/20 12:27 PM, Thomas Monjalon wrote:
> > The mempool pointer in the mbuf struct is moved
> > from the second to the first half.
> > It should increase performance on most systems having 64-byte cache line,
> > i.e. mbuf is split in two cache lines.
> > On such system, the first half (also called first cache line) is hotter
> > than the second one where the pool pointer was.
> > 
> > Moving this field gives more space to dynfield1.
> > 
> > This is how the mbuf layout looks like (pahole-style):
> > 
> > word  type                              name                byte  size
> >  0    void *                            buf_addr;         /*   0 +  8 */
> >  1    rte_iova_t                        buf_iova          /*   8 +  8 */
> >       /* --- RTE_MARKER64               rearm_data;                   */
> >  2    uint16_t                          data_off;         /*  16 +  2 */
> >       uint16_t                          refcnt;           /*  18 +  2 */
> >       uint16_t                          nb_segs;          /*  20 +  2 */
> >       uint16_t                          port;             /*  22 +  2 */
> >  3    uint64_t                          ol_flags;         /*  24 +  8 */
> >       /* --- RTE_MARKER                 rx_descriptor_fields1;        */
> >  4    uint32_t             union        packet_type;      /*  32 +  4 */
> >       uint32_t                          pkt_len;          /*  36 +  4 */
> >  5    uint16_t                          data_len;         /*  40 +  2 */
> >       uint16_t                          vlan_tci;         /*  42 +  2 */
> >  5.5  uint64_t             union        hash;             /*  44 +  8 */
> >  6.5  uint16_t                          vlan_tci_outer;   /*  52 +  2 */
> >       uint16_t                          buf_len;          /*  54 +  2 */
> >  7    struct rte_mempool *              pool;             /*  56 +  8 */
> >       /* --- RTE_MARKER                 cacheline1;                   */
> >  8    struct rte_mbuf *                 next;             /*  64 +  8 */
> >  9    uint64_t             union        tx_offload;       /*  72 +  8 */
> > 10    uint16_t                          priv_size;        /*  80 +  2 */
> >       uint16_t                          timesync;         /*  82 +  2 */
> >       uint32_t                          seqn;             /*  84 +  4 */
> > 11    struct rte_mbuf_ext_shared_info * shinfo;           /*  88 +  8 */
> > 12    uint64_t                          dynfield1[4];     /*  96 + 32 */
> > 16    /* --- END                                             128      */
> > 
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> 
> I'd like to understand why pool is chosen instead of, for
> example, next pointer.
> 
> Pool is used on housekeeping when driver refills Rx ring or
> free completed Tx mbufs. Free thresholds try to avoid it on
> every Rx/Tx burst (if possible).
> 
> Next is used for multi-segment Tx and scattered (and buffer
> split) Rx. IMHO the key question here is we consider these
> use cases as common and priority to optimize. If yes, I'd
> vote to have next on the first cacheline.
> 
> I'm not sure. Just trying to hear a bit more about it.

That's a good question.
Clearly pool and next are good options.
The best would be to have some benchmarks.
If one use case shows no benefit, the decision is easier.

If you prefer, we can leave this last patch for -rc3.