From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E2961A0C44; Mon, 14 Jun 2021 16:59:52 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 80C474067E; Mon, 14 Jun 2021 16:59:52 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id AF2D44067A for ; Mon, 14 Jun 2021 16:59:50 +0200 (CEST) IronPort-SDR: fUtsAuJxCaAiwUU/ZYy+CYOKmuEcu7ywiCTHZTtbHaYmZyT/8F2qRPr5XWuAV/nR1MXrA2HUTJ qpozwyH52b4A== X-IronPort-AV: E=McAfee;i="6200,9189,10015"; a="186197063" X-IronPort-AV: E=Sophos;i="5.83,273,1616482800"; d="scan'208";a="186197063" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2021 07:59:49 -0700 IronPort-SDR: f/vvWVWBZxUeIEGx/kT9+ks+yIjlifhnaOcE8C533nVG7Xl2laD7/6YgJeC+GEX3HdbPlqpVHk idSovUkxyfvA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.83,273,1616482800"; d="scan'208";a="449916196" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by orsmga008.jf.intel.com with ESMTP; 14 Jun 2021 07:59:48 -0700 Received: from fmsmsx612.amr.corp.intel.com (10.18.126.92) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.4; Mon, 14 Jun 2021 07:59:47 -0700 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx612.amr.corp.intel.com (10.18.126.92) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.4 via Frontend Transport; Mon, 14 Jun 2021 07:59:47 -0700 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (104.47.59.169) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2242.4; Mon, 14 Jun 2021 07:59:45 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=X9FtB5vdEyo72rRTTri/LDuQnOt36Gr2WsgLHoLCWrPy5fSbUbFhXFhdrcvOOItkSsaru7YfsFVt2ACnV80e7qPqN/ZFkhRWs+ExsGv0AEc3s++QJyvYsD1K07BrGcVMoL1L5aU+DC6hW0E+pekCNVi5SLGZMRMlfExBUCYSu5SPyDFPkkJHDbqrdih+QfffM7sPMJbRiiC3jopBcu3NsoG5dbu6w9UMT5TSTE7vCyI5znXT1occaYhvbl4QuJd2BBby6gLrLBDDei0y0W3MJXQ6A32UejpzOJGVmM9C3TauYm6sLJnChc9AUWERT3sIdxeBy94O7BcmIsY/m0a2JQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7b7bPR3lYl2CeozU+cF51hQZyMI0/PcYCzXGWPqv11c=; b=A1SAxAKv6rm3C91dLHitNdQ5fZJvN3Tvcx5cUdX/fP25ov9+oFm0sQZoJ4rgjxfzLSDMm/l0VuECw1zcOsdUs73eB71FMzPprMb2mtp/K8l6x2jumVuBDkmYMXnHxsABkUXnbeHyJtxAWX5u7dUL6GCRtCzta2RRVhTNFc926wXMot7XwrYuGNcZ+ozSm5ZNJrT7qyZNozsDrYC2/3FpRp0X/rWk9zDgj//zUK+2SxnRAD8SEExa4srHwB4tnx4Ox1Y4cvxnfCFrWvGK/+JnU+FJ6SACLGPLOCX/fQbGFAB1sNvn8bO+vB1n2zlMxPJ77kZ0rz+lxnQVaQI3ng1bXw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7b7bPR3lYl2CeozU+cF51hQZyMI0/PcYCzXGWPqv11c=; b=dEYj70cFJQPnoDgG7oP4ZtjVE/d6SKuUeJye9nq9yqEpli3OGGhYa2KU8xwlXjCMIpXAy6RoVyf74DptDWkLohJ5/lRd8OK36yW0+sHriXf4M+dNWLxYspiP5PiBuI6PscmNYGPVnVKOaMTT+r7osvDFs7Z9Tbc3ZzorTlt1CZQ= Received: from DM6PR11MB4491.namprd11.prod.outlook.com (2603:10b6:5:204::19) by DM4PR11MB5536.namprd11.prod.outlook.com (2603:10b6:5:39b::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.24; Mon, 14 Jun 2021 14:59:43 +0000 Received: from DM6PR11MB4491.namprd11.prod.outlook.com ([fe80::7dc4:66b0:f76b:6d48]) by DM6PR11MB4491.namprd11.prod.outlook.com ([fe80::7dc4:66b0:f76b:6d48%7]) with mapi id 15.20.4219.025; Mon, 14 Jun 2021 14:59:43 +0000 From: "Ananyev, Konstantin" To: Thomas Monjalon , "Richardson, Bruce" CC: =?iso-8859-1?Q?Morten_Br=F8rup?= , "dev@dpdk.org" , "olivier.matz@6wind.com" , "andrew.rybchenko@oktetlabs.ru" , "honnappa.nagarahalli@arm.com" , "Yigit, Ferruh" , "jerinj@marvell.com" , "gakhil@marvell.com" Thread-Topic: [dpdk-dev] [PATCH] parray: introduce internal API for dynamic arrays Thread-Index: AQHXYQxT3nKj3OToOE+g04oypCLg2asTbdkAgAAOoICAAATCgIAAFeZg Date: Mon, 14 Jun 2021 14:59:43 +0000 Message-ID: References: <20210614105839.3379790-1-thomas@monjalon.net> <98CBD80474FA8B44BF855DF32C47DC35C6184E@smartserver.smartshare.dk> <2004320.XGyPsaEoyj@thomas> In-Reply-To: <2004320.XGyPsaEoyj@thomas> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-reaction: no-action dlp-version: 11.5.1.3 authentication-results: monjalon.net; dkim=none (message not signed) header.d=none;monjalon.net; dmarc=none action=none header.from=intel.com; x-originating-ip: [109.255.184.192] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 4f4f4d09-29ee-4739-44d9-08d92f450af5 x-ms-traffictypediagnostic: DM4PR11MB5536: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 9uDKuly6klFmSCcJ7bC1WMHHN0X0i5A77bhIE0zgyQqjiQIFgoUrjjywoc4Py8Oz0WMZDYakOZn/L++NgadMYXizqaH0A355fAa5Efc+E5MBsVHKysDvKqm/yt3mmDQhgvDHN1y4TBNwDr1WX+PovD/CMLOjZVaIySA24el7uUWZSPr0bf790Q3bvXUpdgCdnidq4eQYUw7n9lu9uawQ4vCPCF6gcVLWVboDk/A7JTUOv7yXdHJLuA8PzKdn4BCa03ic4nBHdeFpFoJvAYQCo5apzq2LDtX0eW1JcUxgfV+ers8SVW+5lcjz2IiZu7IV2UYoDfKeG/BMhGyu/7iy9/d+Dgr1+052Z3Lts3KRbEvmpXZqxLFOmilzIaOYAeFe07ODR1xpfZVXGo2HmYFF++suJgbeezwuhQLyZvshTy/s7C65n3sbSb59yT6STxvETwHUiZZLxe+tzD1KNcNgJfPbkLCSrUn6jTQL1KpEp0eFzytEc3tPDCMa4iuBADvtoDtTFXp3XCFT1++622UN6Y09Hxt6wqPt1x8l6ZkdG9v3YGOGUwVm/EK1rweImk1Ozz4LMMCQhEqOZ97OfvNsy72eMxiPhbujEiYVZOWRKC8= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR11MB4491.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(39860400002)(346002)(366004)(376002)(396003)(136003)(110136005)(5660300002)(186003)(52536014)(54906003)(9686003)(55016002)(7696005)(8676002)(66556008)(316002)(26005)(6636002)(6506007)(8936002)(66476007)(66446008)(64756008)(55236004)(76116006)(66946007)(86362001)(478600001)(2906002)(4326008)(38100700002)(122000001)(71200400001)(66574015)(83380400001)(33656002); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?n1DJ9EpNCBxLQ7pp6ItcK8FPUfh3p7PeGB5S7LbgL5DRt6vNs1MtCqRn65?= =?iso-8859-1?Q?C+lK9tdYflae4ivukJbdnPpoOYSm/xqHJjq687DA2caqF2s3uveqsfgeOa?= =?iso-8859-1?Q?QwqfDI6JroGfBtW2eoSQBC4egy9ePjYsz6Eg57Jq5SppXvgM3CRxtFJdqk?= =?iso-8859-1?Q?oTth4l9n/UgH97t5B5/gKxtfC1aOKzRQN9vg2szE30Qrk53lrj6bfySl5E?= =?iso-8859-1?Q?aYpOHGlJSk6GUD+nhYz2Egju8iEsJJ6o6gxevvkuf3WcHmqd4UAnDpjvNQ?= =?iso-8859-1?Q?YsW13WelmjUOfc/BaE7jR97V/6YWRTmzpghTdcnyIGZxg0Tm+2KC2ULou1?= =?iso-8859-1?Q?eNnRTODpqEu3at2nQnN8h37cBPXJbNVFUP6qH70DLpDZcpdxwY7Kxk3jaO?= =?iso-8859-1?Q?XVDwQCE6jTR5EWycZVPc6zEb72HhOudh1mfqSe3mKkIGqMX6eyIzi8ioEx?= =?iso-8859-1?Q?pRm20UnDFJkdrr14v/nz6yXVW6xEui3lAsV7U/dAaxMe4XUYcRISR/8RzN?= =?iso-8859-1?Q?Zharls9e+9Ayeimj3tS+43eY+HI7c6PoDDc1RzNeApPHZkeAIlJQAUqaow?= =?iso-8859-1?Q?llH4ThWmyzjW36nHAK91dJZyG3+oMJ0H7A+4FiT6FhdpFn/q0hQhW9g5QY?= =?iso-8859-1?Q?W3VVdIkQbUPl8CHU5sHmZXVE6ZTjS7eCmkJiUq2BrD5OF6/hc3OJaFnaj+?= =?iso-8859-1?Q?ShOO/Z3/XrdXlo3TL+Rfdo+1v/zgImtLtcRhacH6qcCcX0DKtA9/ZHwnnl?= =?iso-8859-1?Q?iyHqwgc785WRq+p938xO0DRoQgZN9VNIC8DxE8XWTCRr0n+oCFJmOKjVSS?= =?iso-8859-1?Q?t6am9694amJGtzAjtwl9MuRZLkIsVtFVg6kgMl2R4Z+QvJqWp5N2xwRsJ4?= =?iso-8859-1?Q?b926wbf5qo5XfwcxhH2RVqePP9lyzhQI6Nb/2+vMY7ebHB76HecEpYGxJ9?= =?iso-8859-1?Q?qdoHAI/bd80EazHvMN2goIVzN34FTTbfOZv24kczkhtI7yR3iDEasQF9YJ?= =?iso-8859-1?Q?YyBDDSLVJNyHP1Em6CqxmbwNv85DYjOdk/4kAMW8G/WSmmk3SB8axvDcds?= =?iso-8859-1?Q?Wa4Z80aO5ieDhghpPXFe0eSJf3/51WiO8K2RUTrEEqO+BrANPDcHSmPUFm?= =?iso-8859-1?Q?+quv1sFz++XVAW7Wq7V3gLHO9uOR6ljxJLwFLNA0EBNjaTQm2JBjEyTDMm?= =?iso-8859-1?Q?cusGjTjiCra+E6LgohCBV4vwRNFUZu+s+K2ME+OLO4HQo3n0Oqppzh/5gy?= =?iso-8859-1?Q?LIKjXPRlTWhIXG8VuAqIXBSYsshkPgDYIvNZP9iFT5MU0QX7HvCufC7P1Z?= =?iso-8859-1?Q?S7pMdHPP/gXm9CW6Dx+9nu4sLtzAxHmQPHLTWXtFaEGiEET7RHTv4COzGD?= =?iso-8859-1?Q?rrjD9evyF0?= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB4491.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4f4f4d09-29ee-4739-44d9-08d92f450af5 X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Jun 2021 14:59:43.2181 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: VdOKOlAzW2I2vNEUhA+vfp7dWlqXtthGX0ahvpyOMfEoMwQTN6rgrOcL6JYjLT6oWLFnqPR65TJc0qgobhOK+ZdeawPGdOPe3FMbX6GbxUs= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB5536 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH] parray: introduce internal API for dynamic arrays X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" >=20 > 14/06/2021 15:15, Bruce Richardson: > > On Mon, Jun 14, 2021 at 02:22:42PM +0200, Morten Br=F8rup wrote: > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas Monjalo= n > > > > Sent: Monday, 14 June 2021 12.59 > > > > > > > > Performance of access in a fixed-size array is very good > > > > because of cache locality > > > > and because there is a single pointer to dereference. > > > > The only drawback is the lack of flexibility: > > > > the size of such an array cannot be increase at runtime. > > > > > > > > An approach to this problem is to allocate the array at runtime, > > > > being as efficient as static arrays, but still limited to a maximum= . > > > > > > > > That's why the API rte_parray is introduced, > > > > allowing to declare an array of pointer which can be resized > > > > dynamically > > > > and automatically at runtime while keeping a good read performance. > > > > > > > > After resize, the previous array is kept until the next resize > > > > to avoid crashs during a read without any lock. > > > > > > > > Each element is a pointer to a memory chunk dynamically allocated. > > > > This is not good for cache locality but it allows to keep the same > > > > memory per element, no matter how the array is resized. > > > > Cache locality could be improved with mempools. > > > > The other drawback is having to dereference one more pointer > > > > to read an element. > > > > > > > > There is not much locks, so the API is for internal use only. > > > > This API may be used to completely remove some compilation-time > > > > maximums. > > > > > > I get the purpose and overall intention of this library. > > > > > > I probably already mentioned that I prefer "embedded style programmin= g" with fixed size arrays, rather than runtime configurability. It's > my personal opinion, and the DPDK Tech Board clearly prefers reducing the= amount of compile time configurability, so there is no way for > me to stop this progress, and I do not intend to oppose to this library. = :-) > > > > > > This library is likely to become a core library of DPDK, so I think i= t is important getting it right. Could you please mention a few examples > where you think this internal library should be used, and where it should= not be used. Then it is easier to discuss if the border line between > control path and data plane is correct. E.g. this library is not intended= to be used for dynamically sized packet queues that grow and shrink in > the fast path. > > > > > > If the library becomes a core DPDK library, it should probably be pub= lic instead of internal. E.g. if the library is used to make > RTE_MAX_ETHPORTS dynamic instead of compile time fixed, then some applica= tions might also need dynamically sized arrays for their > application specific per-port runtime data, and this library could serve = that purpose too. > > > > > > > Thanks Thomas for starting this discussion and Morten for follow-up. > > > > My thinking is as follows, and I'm particularly keeping in mind the cas= es > > of e.g. RTE_MAX_ETHPORTS, as a leading candidate here. > > > > While I dislike the hard-coded limits in DPDK, I'm also not convinced t= hat > > we should switch away from the flat arrays or that we need fully dynami= c > > arrays that grow/shrink at runtime for ethdevs. I would suggest a half-= way > > house here, where we keep the ethdevs as an array, but one allocated/si= zed > > at runtime rather than statically. This would allow us to have a > > compile-time default value, but, for use cases that need it, allow use = of a > > flag e.g. "max-ethdevs" to change the size of the parameter given to t= he > > malloc call for the array. This max limit could then be provided to ap= ps > > too if they want to match any array sizes. [Alternatively those apps co= uld > > check the provided size and error out if the size has been increased be= yond > > what the app is designed to use?]. There would be no extra dereferences= per > > rx/tx burst call in this scenario so performance should be the same as > > before (potentially better if array is in hugepage memory, I suppose). >=20 > I think we need some benchmarks to decide what is the best tradeoff. > I spent time on this implementation, but sorry I won't have time for benc= hmarks. > Volunteers? =20 I had only a quick look at your approach so far. But from what I can read, in MT environment your suggestion will require extra synchronization for each read-write access to such parray element (lo= ck, rcu, ...). I think what Bruce suggests will be much ligther, easier to implement and l= ess error prone. At least for rte_ethdevs[] and friends. Konstantin =20