From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 04DEBA04B5; Thu, 29 Oct 2020 15:15:52 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 37119CDDD; Thu, 29 Oct 2020 15:15:50 +0100 (CET) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id E836ACD78 for ; Thu, 29 Oct 2020 15:15:48 +0100 (CET) IronPort-SDR: zQWbDO5lORd0C5HoFFqo7XZI6w/z3LRlC+uX7CPCOCiS6ZcUnC9Imdb5o7pKLbjK0Nym7vO2Bu Q6+steypqNdQ== X-IronPort-AV: E=McAfee;i="6000,8403,9788"; a="155408103" X-IronPort-AV: E=Sophos;i="5.77,430,1596524400"; d="scan'208";a="155408103" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Oct 2020 07:15:46 -0700 IronPort-SDR: OIbpanaqyrQ9fSm/gJ421CdoRY8bLceFjGJed5V3e1MWjzQPOf0AmDyG4Zu6qte9sRCVJwb57c nuXFtRkUzJpQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,430,1596524400"; d="scan'208";a="361504322" Received: from orsmsx606.amr.corp.intel.com ([10.22.229.19]) by FMSMGA003.fm.intel.com with ESMTP; 29 Oct 2020 07:15:45 -0700 Received: from orsmsx601.amr.corp.intel.com (10.22.229.14) by ORSMSX606.amr.corp.intel.com (10.22.229.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 29 Oct 2020 07:15:45 -0700 Received: from ORSEDG601.ED.cps.intel.com (10.7.248.6) by orsmsx601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5 via Frontend Transport; Thu, 29 Oct 2020 07:15:45 -0700 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.107) by edgegateway.intel.com (134.134.137.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.1713.5; Thu, 29 Oct 2020 07:15:44 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ME33lFsjRX8gc4FkxYRYD8XwKA/Ue5MAYyRVHETSULh9yP7FDk/H5hI3nUjjhD2ba9SU1/+O1cyRImjKbFs6s1UWKV/fdCHGhqsoJeWeWfBXpMfQNmdmLl48P8n9dc8hLe8JMex6R6oYxXVFmaM1nDIO4+lcQcrqBshB7MmIjdMtZ1BUWC1Qy+ZsNZIXET3L0/hSEGRVYsavLQqYhRzQnxybTv5DSPxc20wT5fThtzunuUjs1zKXxLxqYMwv9YR9B9TdROinsc0NTHwbsuCT+Yr6Izf8Xx83CUjRgtIwkH/tgArP6CqSRJgDB886i45Q2C/5WKhmNx2qajk/Po5PGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kK/UFsSZTqCztApBMy9BJ8PQ5+nP+0lljYH4wUoOpQk=; b=dSGh+6gL9E42YF8MIfIw/76li4+kffPI7Zi9steRQbjx+5yp55EXWELJYYf2SFAKE6HL8JXaNgIg5Tej0zbq+Kc+WymIhEeBe7FMY+xKTdrfGfzPJh/KlmAlKvNQKlG5KMFS6Ue+HQi2EaUVIRM5GuEmgrbozANdP3BITcwHZzOLhmkEgYHt+czbHqoQTRSLFmRMAgtFYkHWHJtzLisR5agPIRfGiTmwZ1NPLPnjxP+B5O39/UZK7g4MBplBIE35/oslgBww4yIx8cs2Dyj03GtVee6iM+H5gXZcL9Eqg3Gk5T1rLI3tinGVTfq6/eGgE0B6p3lq9wOEqCZPlNWGaA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kK/UFsSZTqCztApBMy9BJ8PQ5+nP+0lljYH4wUoOpQk=; b=Ht7Z3WaqPBXEQAkl9+WIjdIjUB4nab7Ileoaqj+T0sOp0ZND9mOU1qOVs/QOjtirmNTaT5k+FllTJUOsKFqeE39jIQO5VHRv4YIthsysYDUfwpmHMoc/+FSxje+4Yc5D0MLvhFAH/ZhjEoMLjQa5FGHIwiFN//vcgxzwt/tOXac= Received: from BYAPR11MB3301.namprd11.prod.outlook.com (2603:10b6:a03:7f::26) by BYAPR11MB3622.namprd11.prod.outlook.com (2603:10b6:a03:fe::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3477.20; Thu, 29 Oct 2020 14:15:42 +0000 Received: from BYAPR11MB3301.namprd11.prod.outlook.com ([fe80::f5a4:3f6b:ade3:296b]) by BYAPR11MB3301.namprd11.prod.outlook.com ([fe80::f5a4:3f6b:ade3:296b%3]) with mapi id 15.20.3499.027; Thu, 29 Oct 2020 14:15:42 +0000 From: "Ananyev, Konstantin" To: Thomas Monjalon , Andrew Rybchenko CC: "dev@dpdk.org" , "Yigit, Ferruh" , "david.marchand@redhat.com" , "Richardson, Bruce" , "olivier.matz@6wind.com" , "jerinj@marvell.com" , "viacheslavo@nvidia.com" , "ajit.khaparde@broadcom.com" , "honnappa.nagarahalli@arm.com" , "maxime.coquelin@redhat.com" , "stephen@networkplumber.org" , "hemant.agrawal@nxp.com" Thread-Topic: [PATCH 15/15] mbuf: move pool pointer in hotter first half Thread-Index: AQHWreJZcpHbRVJaK0e1jLzrQ/pkJamundEQ Date: Thu, 29 Oct 2020 14:15:41 +0000 Message-ID: References: <20201029092751.3837177-1-thomas@monjalon.net> <20201029092751.3837177-16-thomas@monjalon.net> <293d3128-7215-c47c-de20-6c510bc8acb3@oktetlabs.ru> <2815114.5jsDjZKEVh@thomas> In-Reply-To: <2815114.5jsDjZKEVh@thomas> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-reaction: no-action dlp-version: 11.5.1.3 authentication-results: monjalon.net; dkim=none (message not signed) header.d=none;monjalon.net; dmarc=none action=none header.from=intel.com; x-originating-ip: [46.7.39.127] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 5a81b890-d207-45df-e4cb-08d87c151e74 x-ms-traffictypediagnostic: BYAPR11MB3622: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: zRTz1Pf/ZlYdCenLl1pBl1lwraLA1VajtPoJSRj8VapxXFdQsa8NEfeaMxJ6W0wP/OTLrAuUb/Bv7HJGVNypyNaPOM7nLyrHHex16v0a11PtvyWLacEFrMMiR1wA6gtTS+7vHtgm4dumD+S3ztYYF34JkpgZXeUsJyESx8pdEdriYBt4djJknUjNOdokSyQURDUK0Pzs93TzQoTLdE4bjTm7NSQb3RNcTymfN1srwOOX+fhxk4kxUSF6KKWCoQ2puoJTe6r2Ffz6WwYLhPwpvowzH8GqO0C+vc/xRXHbJNbnwZmHNjMqw3THFL7wk90VSzJMd/8JR3baWyjV6a7ytQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR11MB3301.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(136003)(39860400002)(396003)(376002)(346002)(366004)(8936002)(33656002)(5660300002)(53546011)(2906002)(6506007)(83380400001)(7696005)(478600001)(316002)(7416002)(86362001)(52536014)(66946007)(66476007)(76116006)(186003)(64756008)(66556008)(26005)(55016002)(8676002)(71200400001)(9686003)(4326008)(66446008)(54906003)(110136005); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: hIMugHXPlrOm7VRnzxFqfPYkGJ/HqVctvX/L4y26AAhvSQ6oCQv9Shxk/tybba6prCqwgZNWUAU+3zyFasQ2JLcnpWEZz36Xvu9iw1B+nHrvMUN6PkeqirSHFliHrycgwndnxbmzjNWi2xdyBBrXa8tZuG8ZNrnjdLvmNYL+2dDMjTiE2fD0Uc1HWXA03b/rXcDxKxNs0KTdlqQvLTQ4hCs3gucbnwI8e927gd2iyUK1nan3EEUw214YfCw+dFrbm2EsNsjgGim0qLWSxqy5oz2q/ulxmzHTTLXv1VUZFLKek+VhIB/CEezrpoHAJ76IvQoYdMv3GUsPAurHu58B6KtMXKTsvCvmgpi2mkit7cS46mOZUcy0szhCK0XTpgL+nazUKh+Cf72MUpr6YnFWr/VsYTEipN0mcUcNlRJor2o5R3jTJpu7Ig96VtaHLN4mGGvGTyy1pOaRQA3lzEgBEL2k6oYO5J92U3kicnUOh0qLUo90hrqWN8+rVdpnsUYdULS20tEaf0Anqzj1L2YY4vuIV4ZBlHkas9flB4W6RF0PYh4vO5LBeR3OAhmKMrz4CRbVkP8mFtUs6yLsXDSs3pAQTeLC3vTTuzF1nb/cXMX7th+XQ/8hjuEXzlwYsIlt054rUStNT4o+i0ry5dt/sQ== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3301.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5a81b890-d207-45df-e4cb-08d87c151e74 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Oct 2020 14:15:41.9150 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Iv4pAHYFw0n6T5JqVCRCDYOjhGP4pqXY57IwgWVX8X0Cggyj2duNjV2FJ4HWDgkAu5lGtgNp65ss9lRvNiAOlAF2693uGqJXOH0gNuhwisU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB3622 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH 15/15] mbuf: move pool pointer in hotter first half X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" >=20 > 29/10/2020 11:50, Andrew Rybchenko: > > On 10/29/20 12:27 PM, Thomas Monjalon wrote: > > > The mempool pointer in the mbuf struct is moved > > > from the second to the first half. > > > It should increase performance on most systems having 64-byte cache l= ine, > > > i.e. mbuf is split in two cache lines. > > > On such system, the first half (also called first cache line) is hott= er > > > than the second one where the pool pointer was. > > > > > > Moving this field gives more space to dynfield1. > > > > > > This is how the mbuf layout looks like (pahole-style): > > > > > > word type name byte siz= e > > > 0 void * buf_addr; /* 0 + 8= */ > > > 1 rte_iova_t buf_iova /* 8 + 8= */ > > > /* --- RTE_MARKER64 rearm_data; = */ > > > 2 uint16_t data_off; /* 16 + 2= */ > > > uint16_t refcnt; /* 18 + 2= */ > > > uint16_t nb_segs; /* 20 + 2= */ > > > uint16_t port; /* 22 + 2= */ > > > 3 uint64_t ol_flags; /* 24 + 8= */ > > > /* --- RTE_MARKER rx_descriptor_fields1; = */ > > > 4 uint32_t union packet_type; /* 32 + 4= */ > > > uint32_t pkt_len; /* 36 + 4= */ > > > 5 uint16_t data_len; /* 40 + 2= */ > > > uint16_t vlan_tci; /* 42 + 2= */ > > > 5.5 uint64_t union hash; /* 44 + 8= */ > > > 6.5 uint16_t vlan_tci_outer; /* 52 + 2= */ > > > uint16_t buf_len; /* 54 + 2= */ > > > 7 struct rte_mempool * pool; /* 56 + 8= */ > > > /* --- RTE_MARKER cacheline1; = */ > > > 8 struct rte_mbuf * next; /* 64 + 8= */ > > > 9 uint64_t union tx_offload; /* 72 + 8= */ > > > 10 uint16_t priv_size; /* 80 + 2= */ > > > uint16_t timesync; /* 82 + 2= */ > > > uint32_t seqn; /* 84 + 4= */ > > > 11 struct rte_mbuf_ext_shared_info * shinfo; /* 88 + 8= */ > > > 12 uint64_t dynfield1[4]; /* 96 + 32= */ > > > 16 /* --- END 128 = */ > > > > > > Signed-off-by: Thomas Monjalon > > > > I'd like to understand why pool is chosen instead of, for > > example, next pointer. > > > > Pool is used on housekeeping when driver refills Rx ring or > > free completed Tx mbufs. Free thresholds try to avoid it on > > every Rx/Tx burst (if possible). > > > > Next is used for multi-segment Tx and scattered (and buffer > > split) Rx. IMHO the key question here is we consider these > > use cases as common and priority to optimize. If yes, I'd > > vote to have next on the first cacheline. Between these two I also would probably lean towards *next* (after all _free_ also has to access/update next). As another alternative to consider: tx_offload. It is also used quite widely.=20 > > > > I'm not sure. Just trying to hear a bit more about it. > . > That's a good question. > Clearly pool and next are good options. > The best would be to have some benchmarks. > If one use case shows no benefit, the decision is easier. >=20 > If you prefer, we can leave this last patch for -rc3. >=20