From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 99282A04B5; Thu, 29 Oct 2020 19:45:35 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B405AC90A; Thu, 29 Oct 2020 19:45:33 +0100 (CET) Received: from mail-oo1-f66.google.com (mail-oo1-f66.google.com [209.85.161.66]) by dpdk.org (Postfix) with ESMTP id D0177C8FC for ; Thu, 29 Oct 2020 19:45:32 +0100 (CET) Received: by mail-oo1-f66.google.com with SMTP id j6so978290oot.3 for ; Thu, 29 Oct 2020 11:45:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=QwxvT8RvaKSVoj3sVSAzy6PipkIaPZ7dqKegXIisINA=; b=UPsi7g3fsRIdudRg40GnKuIWPYVIdyYkSrjMprY+HejrMwT2YmEdFRmCQly5aJelGd +68ny6DueVDyU+eICp70IrN0FsYuOE8CsPtzLbOBh7wQBTs47WjgCN515MgzjbcXvrIG 6F/jrYWvBkks8CGioPqjlSX41SbRLnWgMSVEY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=QwxvT8RvaKSVoj3sVSAzy6PipkIaPZ7dqKegXIisINA=; b=F0Pfp4Bk2Sn4OC75e/GREpr7TJVvVTqNnimjYS2EjO52cvKt1PLiYNIbDzPd87Sxa4 c31tfTOe0pKoYqRLxjwBTc4SvOOlizuz22/ocVf1g4klzIrUy73o7D8cL5EKXfeitTC6 32MIWDi5FmKaYLMVM/FpWwTSNK6FJdBi9rtyo/V3B0N2alR+RyP39ZGXrG8vUPGKRKY2 WoqRLyRCFAZQoCGXJqZUmPEgDiJmi4n/buMt1toeq5yWEwkpPqwO8T20Dw6TTwiEMnQc LQNCCJq+E0g3zKtariGRqaDYhOmz6bHkmJ82OkWn7WY6TnDf3Jh4y5TA56Gg1HwdWnRt S9kg== X-Gm-Message-State: AOAM530LgQu0TR8rfQwSpP4l8DUfDepN+KLuXY0eZh24J6+CF9zSpRjG KtAVhWnn+hgjf11vguEadbh3FK7uUQgWz7yhs17oXw== X-Google-Smtp-Source: ABdhPJwOcBthXs9Gi9AIrAaz3YUc3b0bGLLFQyC88+af4MwHWO8oIn+Q/C4ltwDIhEyyHhmlcmTuY0aRsa0/VYPH86k= X-Received: by 2002:a4a:d554:: with SMTP id q20mr4324642oos.23.1603997130957; Thu, 29 Oct 2020 11:45:30 -0700 (PDT) MIME-Version: 1.0 References: <20201029092751.3837177-1-thomas@monjalon.net> <20201029092751.3837177-16-thomas@monjalon.net> <293d3128-7215-c47c-de20-6c510bc8acb3@oktetlabs.ru> <2815114.5jsDjZKEVh@thomas> In-Reply-To: From: Ajit Khaparde Date: Thu, 29 Oct 2020 11:45:14 -0700 Message-ID: To: "Ananyev, Konstantin" Cc: Thomas Monjalon , Andrew Rybchenko , "dev@dpdk.org" , "Yigit, Ferruh" , "david.marchand@redhat.com" , "Richardson, Bruce" , "olivier.matz@6wind.com" , "jerinj@marvell.com" , "viacheslavo@nvidia.com" , "honnappa.nagarahalli@arm.com" , "maxime.coquelin@redhat.com" , "stephen@networkplumber.org" , "hemant.agrawal@nxp.com" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH 15/15] mbuf: move pool pointer in hotter first half X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Thu, Oct 29, 2020 at 7:15 AM Ananyev, Konstantin wrote: > > > > > > > 29/10/2020 11:50, Andrew Rybchenko: > > > On 10/29/20 12:27 PM, Thomas Monjalon wrote: > > > > The mempool pointer in the mbuf struct is moved > > > > from the second to the first half. > > > > It should increase performance on most systems having 64-byte cache line, > > > > i.e. mbuf is split in two cache lines. > > > > On such system, the first half (also called first cache line) is hotter > > > > than the second one where the pool pointer was. > > > > > > > > Moving this field gives more space to dynfield1. > > > > > > > > This is how the mbuf layout looks like (pahole-style): > > > > > > > > word type name byte size > > > > 0 void * buf_addr; /* 0 + 8 */ > > > > 1 rte_iova_t buf_iova /* 8 + 8 */ > > > > /* --- RTE_MARKER64 rearm_data; */ > > > > 2 uint16_t data_off; /* 16 + 2 */ > > > > uint16_t refcnt; /* 18 + 2 */ > > > > uint16_t nb_segs; /* 20 + 2 */ > > > > uint16_t port; /* 22 + 2 */ > > > > 3 uint64_t ol_flags; /* 24 + 8 */ > > > > /* --- RTE_MARKER rx_descriptor_fields1; */ > > > > 4 uint32_t union packet_type; /* 32 + 4 */ > > > > uint32_t pkt_len; /* 36 + 4 */ > > > > 5 uint16_t data_len; /* 40 + 2 */ > > > > uint16_t vlan_tci; /* 42 + 2 */ > > > > 5.5 uint64_t union hash; /* 44 + 8 */ > > > > 6.5 uint16_t vlan_tci_outer; /* 52 + 2 */ > > > > uint16_t buf_len; /* 54 + 2 */ > > > > 7 struct rte_mempool * pool; /* 56 + 8 */ > > > > /* --- RTE_MARKER cacheline1; */ > > > > 8 struct rte_mbuf * next; /* 64 + 8 */ > > > > 9 uint64_t union tx_offload; /* 72 + 8 */ > > > > 10 uint16_t priv_size; /* 80 + 2 */ > > > > uint16_t timesync; /* 82 + 2 */ > > > > uint32_t seqn; /* 84 + 4 */ > > > > 11 struct rte_mbuf_ext_shared_info * shinfo; /* 88 + 8 */ > > > > 12 uint64_t dynfield1[4]; /* 96 + 32 */ > > > > 16 /* --- END 128 */ > > > > > > > > Signed-off-by: Thomas Monjalon > > > > > > I'd like to understand why pool is chosen instead of, for > > > example, next pointer. > > > > > > Pool is used on housekeeping when driver refills Rx ring or > > > free completed Tx mbufs. Free thresholds try to avoid it on > > > every Rx/Tx burst (if possible). > > > > > > Next is used for multi-segment Tx and scattered (and buffer > > > split) Rx. IMHO the key question here is we consider these > > > use cases as common and priority to optimize. If yes, I'd > > > vote to have next on the first cacheline. > > Between these two I also would probably lean towards *next* > (after all _free_ also has to access/update next). +1 > As another alternative to consider: tx_offload. > It is also used quite widely. > > > > > > > I'm not sure. Just trying to hear a bit more about it. > > . > > That's a good question. > > Clearly pool and next are good options. > > The best would be to have some benchmarks. > > If one use case shows no benefit, the decision is easier. > > > > If you prefer, we can leave this last patch for -rc3. > > >