From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5A7ACA034F; Fri, 8 Oct 2021 17:57:30 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A0F44407FF; Fri, 8 Oct 2021 17:57:28 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id A307740143 for ; Fri, 8 Oct 2021 17:57:24 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10131"; a="207343622" X-IronPort-AV: E=Sophos;i="5.85,358,1624345200"; d="scan'208";a="207343622" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Oct 2021 08:57:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,358,1624345200"; d="scan'208";a="439975348" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by orsmga006.jf.intel.com with ESMTP; 08 Oct 2021 08:57:23 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.12; Fri, 8 Oct 2021 08:57:22 -0700 Received: from fmsmsx601.amr.corp.intel.com (10.18.126.81) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.12; Fri, 8 Oct 2021 08:57:22 -0700 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.12 via Frontend Transport; Fri, 8 Oct 2021 08:57:22 -0700 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.168) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2242.12; Fri, 8 Oct 2021 08:57:22 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IRHuA4IGlPIor2paxHGe4NMwNdqd8z5oGTxL00cKJ+ctzucLl6VUeHcI+Bp2a4erUnQ3KZZABXXz8oJcn/seu/12lQsD7p/xCOd/4bzrDSfY7XpzRCn2hijN9+vHKtiYl98vPffcdLHrQALtlgCHp5vNyjNAwVC4qXK10dwo3DPHqvgDP6xnGY+N153O/6A7DkrOswXsv+pwVlW8Cy16ThLqA64nRrRqWFSmNmBG+ncPMEG382r3Go3DjezVVXpEe8j2AHKO1QdZEa22YR4ay7gHKxyM5xQDBoTtl2XNgQD/VXOTJud4ypRXhdYX1ZcTIR1aM9RkHxuDgCegeSHQDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZPx7cjEy/48IBW+Ca7tZ3t2/p5tw0s41wfPkFy49SCE=; b=P8ac3N/355wuJly55vZMfdvH6Ur5TMnH6HQWaT7vkyGdLzYwYkK4z0tShJpkRiehChcnROesHD5GLXpX4AXN0qOwASftUUVv3p9uWXD4FfZ/L3/IugoAXjDNiY8mQO9/UtBSyhUbh1f7ydwNUnUynkU2U5kzOCrfbvsfZMtvxtvpnpvldThEbWIrhcW3O6ywfBGxszUsBcKOS9coyv0jEIc7o0QP4F4A4C5AiIjBV7z/TBz3O89myNGtP7GEjfa7kDWRmuIy32/sgD0lNgWJTJAgeh2CCFUXtiTkZZxe/Z7Qz+YinEmg29cD4TRPmR0Dt/c+f8wCZkd+PMwidBagJQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZPx7cjEy/48IBW+Ca7tZ3t2/p5tw0s41wfPkFy49SCE=; b=No1HftVJPnpOON7DZC3JXc3WsQg1fKRi9jOf6QvBpCyNus90GPyGnQKryPx5kajkkA6YGVejWCW3e7hdCY/SCTQaTRfffGqW4sO0IDKNr1V0Jy1FYI5OQOa06xDKFAPj+4+1YSoqX6Labew5LO76bFkcY+s1XrzJ1ZK95dj/rms= Received: from SA2PR11MB5196.namprd11.prod.outlook.com (2603:10b6:806:119::18) by SN6PR11MB2592.namprd11.prod.outlook.com (2603:10b6:805:57::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.19; Fri, 8 Oct 2021 15:57:21 +0000 Received: from SA2PR11MB5196.namprd11.prod.outlook.com ([fe80::b8ae:106d:fe92:1996]) by SA2PR11MB5196.namprd11.prod.outlook.com ([fe80::b8ae:106d:fe92:1996%9]) with mapi id 15.20.4587.022; Fri, 8 Oct 2021 15:57:21 +0000 From: "Carrillo, Erik G" To: Pavan Nikhilesh Bhagavatula , "Jerin Jacob Kollanukkaran" CC: "dev@dpdk.org" Thread-Topic: [dpdk-dev] [PATCH v3 11/14] eventdev: move timer adapters memory to hugepage Thread-Index: AQHXun6aUDZahjbJtUm2J+HQ1hXTDKvH+J2QgACf0oCAAKtioA== Date: Fri, 8 Oct 2021 15:57:20 +0000 Message-ID: References: <20211003082710.8398-1-pbhagavatula@marvell.com> <20211006065012.16508-1-pbhagavatula@marvell.com> <20211006065012.16508-11-pbhagavatula@marvell.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: marvell.com; dkim=none (message not signed) header.d=none;marvell.com; dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 8a509281-8687-4cd2-9da7-08d98a744fd8 x-ms-traffictypediagnostic: SN6PR11MB2592: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: X2EgD/a1LC6bUZG5F5R/IaVh+Tnn2vkElGdwCLHCz5CRLhLJ3t2Vzalzu5Kv7ReafynR/i961Z53yR2oEfkzt9B/pKKX8nnkLmjBzf3FtyGaWep7vreg9sXlQiYG6EwX98H3J8XcRYuZn/WT6DP80cnmvdEygtpZi38ag21/OoI72r7FvVbYxC//MdgR/hwNvZD0Hbxnx74SEXXExKBYZ2MCEUBxnvKiCByu+sMGNU36qANoFcdD3+ab1rzxVlAdi9YFR7f6MqZNjWf487Oa0MaoxKOm51XnkPYeBz560VIkLEuJW/7EYKN99Fsa7Yppl9jwOLHAnuHeQEOT3u2GMO/NFrPdngAZ5tVEugG6W3HIO76e30syoUXheD9metmPYRNt+C15/kwa51toN4vwe3aOrbFHm3pVu083T0AyX5vYOJEG8DhWy+TLKqh0yf8dtfRChPiPKQdY6pjiKZx+KWBMFbFn8f7CPA0yTgSZrqXzwBGFrsSeL4ZTyxRUm1Kjdhd8nGIodH5WcFH/YFp3MD/OOefX0U4SrHjOpdatD6SA9M6VQPrZwV9yblm5Fl6yKudrG5hnUhykR7uRGIicywhN69sjqw1ILUDDm7FhutYA/hjgQrwo4hrztWWvARfnQQJdeCHA9mpE/Wb3sE+8GbebBVyRx9+yWqBCNQDOijQLbYMCKxltB7ttI2KHWLj9qRxpcxsSx89x2E1u0UhoIg== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SA2PR11MB5196.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(83380400001)(6506007)(66946007)(2906002)(53546011)(186003)(38100700002)(5660300002)(52536014)(508600001)(7696005)(110136005)(316002)(55016002)(122000001)(4326008)(8936002)(9686003)(76116006)(86362001)(66556008)(66476007)(26005)(71200400001)(33656002)(66446008)(64756008)(8676002)(38070700005); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?h+0KlU8f4io/PUzlgNgJRrD+hXrpy94F4MvnzU+3THIQegz1yVLnExaL2TCY?= =?us-ascii?Q?rfZxVZE4NASXR3mnmcVikvTsV8qHcjJml+dJIXAOBCBKCXxiKwioGKmgRkHe?= =?us-ascii?Q?jQ3qcRWwraTivcBlUbHiguEo6l5X2cf34LqCdCq3RtjwUoK5idh7gMbHrCUt?= =?us-ascii?Q?WdVk4jZe4LAd0fjlyAo1q/pKJtCMsa6H/itbHeiSM3TFxeFnNubGSOKkvMhW?= =?us-ascii?Q?4JgVjaNovhb/SaTFohihdE4CMig6OFn6SO3dgjZW5nXVFlfMKJXkxadk20Ur?= =?us-ascii?Q?fIvgE9RK8T2+qEpUjmgWzbI3o2mBvGcCx4qtXXQA/IqeBpwa/7+yIQIbVkhg?= =?us-ascii?Q?z1hnz0J5Vw9JvVQrnbQjBF4bA6yW29YDS2QCjKM6exuNkE4JTPwtOvTzI6s9?= =?us-ascii?Q?RpqpyUpMYf1dyxWytIbXQesb3oTN4DqY7xEFuDx2YxkK7g4xyMsiBaExG8m+?= =?us-ascii?Q?R2NWMXbr9YUePfql+2VwmE1qpfs2Ay1lR7Sy9D0gqA6QIwm+1huw9tat1Rkq?= =?us-ascii?Q?M3cdArfcSb2YzXSegWB6Rhoob+q+epUWJbd2UF4/f+mlA1OxV6vx1lX4fi9u?= =?us-ascii?Q?7JjpLHb9QhTmI6ihG04jbTfrsDtQf0b1JCVqXFO8S4GhFu1VKy8BVNyRzb0W?= =?us-ascii?Q?ZnRUOQR1fgMcrD4piNJ0hLe3jPWQQvToAPTmtcbb90WZmbBInHsnZIQxylTk?= =?us-ascii?Q?Ae91/HihyGist3c1N/mTaJX2KkDnyPdJb1e+84Gdn8O9zN/F5JE2tu4xsKQx?= =?us-ascii?Q?Y9IvvgeFCcN075gTV6DE0LjQwDp80eIpfIrMKWg6eK+Dz7DfQvZTUfThOdOf?= =?us-ascii?Q?g5JK/GLjQdExmvDILSQ84yatTJn3FMP0ZcqlxrHZlZ3hbJ/lpsiVctlJIXYR?= =?us-ascii?Q?OJeIb5PcdhTFtiToP0KvCPUNjduUjcXZhUE0/ta2MqIB0lc17nekF8dySSfE?= =?us-ascii?Q?4BMP2hQ2o4gksuVKd7JIWLBMksjylnZyUrrw7QH65rilRAI62SVHyfuQSftf?= =?us-ascii?Q?y09NMl7S0qdcLiTAU1d8qw6yDygyDAwgkwAcOojsxMaADgKx0mvW+cle7bJ5?= =?us-ascii?Q?33NefnmraxmCjuvX1X9sKCyG00PCv/VjVQhx8mY1kW69AEV7Nq0Hsvlsz4oA?= =?us-ascii?Q?7I7liYTUP/tsFsliy8HgX6c4s9QKueIQbENcdNY6MbOGUwQ/dKJwlHlZCyqO?= =?us-ascii?Q?3lsNxYH1PePLLuAhbDCxXuSso1eXXlHN9RQIIljqX5z/cwg2sRoC2n2d1acz?= =?us-ascii?Q?M6V5D8ktVX7+XQLmxf+nIcUwspktwhjTwIL/FcHW/wwv5vhe0oqYJrXIsv1p?= =?us-ascii?Q?NwAYm0+7Rz5MdJuen95I5pmo?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SA2PR11MB5196.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8a509281-8687-4cd2-9da7-08d98a744fd8 X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Oct 2021 15:57:20.7751 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: qbyCUZfyjaEUEFxsZyQmNnAafj/z9wlGdrb/pvZzxaxgHHVxcbDs+mts2KAUBkCzdNITLtN0d75iMGAtssHiaxsLq0Z0Er6NLA/hJSrYiu0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR11MB2592 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH v3 11/14] eventdev: move timer adapters memory to hugepage X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Pavan Nikhilesh Bhagavatula > Sent: Friday, October 8, 2021 12:38 AM > To: Carrillo, Erik G ; Jerin Jacob Kollanukkar= an > > Cc: dev@dpdk.org > Subject: RE: [dpdk-dev] [PATCH v3 11/14] eventdev: move timer adapters > memory to hugepage >=20 > Hi Erik, >=20 > >Hi Pavan, > > > >Some comments below: > > > >> -----Original Message----- > >> From: pbhagavatula@marvell.com > >> Sent: Wednesday, October 6, 2021 1:50 AM > >> To: jerinj@marvell.com; Carrillo, Erik G > >> Cc: dev@dpdk.org; Pavan Nikhilesh > >> Subject: [dpdk-dev] [PATCH v3 11/14] eventdev: move timer adapters > >> memory to hugepage > >> > >> From: Pavan Nikhilesh > >> > >> Move memory used by timer adapters to hugepage. > >> Allocate memory on the first adapter create or lookup to address both > >> primary and secondary process usecases. > >> This will prevent TLB misses if any and aligns to memory structure of > >other > >> subsystems. > >> > >> Signed-off-by: Pavan Nikhilesh > >> --- > >> lib/eventdev/rte_event_timer_adapter.c | 24 > >> +++++++++++++++++++++++- > >> 1 file changed, 23 insertions(+), 1 deletion(-) > >> > >> diff --git a/lib/eventdev/rte_event_timer_adapter.c > >> b/lib/eventdev/rte_event_timer_adapter.c > >> index ae55407042..c4dc7a5fd4 100644 > >> --- a/lib/eventdev/rte_event_timer_adapter.c > >> +++ b/lib/eventdev/rte_event_timer_adapter.c > >> @@ -33,7 +33,7 @@ RTE_LOG_REGISTER_SUFFIX(evtim_logtype, > >> adapter.timer, NOTICE); > >> RTE_LOG_REGISTER_SUFFIX(evtim_buffer_logtype, adapter.timer, > >NOTICE); > >> RTE_LOG_REGISTER_SUFFIX(evtim_svc_logtype, adapter.timer.svc, > >> NOTICE); > >> > >> -static struct rte_event_timer_adapter > >> adapters[RTE_EVENT_TIMER_ADAPTER_NUM_MAX]; > >> +static struct rte_event_timer_adapter *adapters; > >> > >> static const struct event_timer_adapter_ops swtim_ops; > >> > >> @@ -138,6 +138,17 @@ rte_event_timer_adapter_create_ext( > >> int n, ret; > >> struct rte_eventdev *dev; > >> > >> + if (adapters =3D=3D NULL) { > >> + adapters =3D rte_zmalloc("Eventdev", > >> + sizeof(struct > >rte_event_timer_adapter) * > >> + > >> RTE_EVENT_TIMER_ADAPTER_NUM_MAX, > >> + RTE_CACHE_LINE_SIZE); > >> + if (adapters =3D=3D NULL) { > >> + rte_errno =3D ENOMEM; > >> + return NULL; > >> + } > >> + } > >> + > >> if (conf =3D=3D NULL) { > >> rte_errno =3D EINVAL; > >> return NULL; > >> @@ -312,6 +323,17 @@ rte_event_timer_adapter_lookup(uint16_t > >> adapter_id) > >> int ret; > >> struct rte_eventdev *dev; > >> > >> + if (adapters =3D=3D NULL) { > >> + adapters =3D rte_zmalloc("Eventdev", > >> + sizeof(struct > >rte_event_timer_adapter) * > >> + > >> RTE_EVENT_TIMER_ADAPTER_NUM_MAX, > >> + RTE_CACHE_LINE_SIZE); > >> + if (adapters =3D=3D NULL) { > >> + rte_errno =3D ENOMEM; > >> + return NULL; > >> + } > >> + } > >> + > >> if (adapters[adapter_id].allocated) > >> return &adapters[adapter_id]; /* Adapter is already > >loaded > >> */ > >> > >> -- > >> 2.17.1 > > > >The rte_event_timer_adapter struct has several fields that have per- > >process values. > > > >For example, there are three fast path function pointers and each will > >be assigned distinct addresses for each process in a multi-process > >scenario. The "allocated" field is also per-process. With the changes > >above, if a secondary process did a lookup() after a primary process > >did a create(), the secondary would get a reference to an object with > >function pointers that are invalid in the secondary process. > > >=20 > I understand, the current patch doesn't unify the memory between > processes. > Instead, we zmalloc the per-process 'array' that holds the adapter object= s > when ever the process calls either create or lookup and initialize the pe= r- > process data structure. >=20 > The pointer to the adapter array is static, so when ever a process is ini= tialized > it will be NULL. >=20 Ah, right - I missed that. This looks OK to me now. One other thing: we never do a free of the array we zmalloc'd, but it looks= like we could in rte_event_timer_adapter_free(), if we were freeing the la= st adapter instance. =20 >=20 > >To fully move the adapter object table into shared hugepage memory, > >those "per-process" members would need to be collected into a per- > >process data structure that could be independently allocated for each > >process. However, that would add one more pointer dereference to get > >to the fast path functions, and avoiding that was the original reason > >to put those pointers there. This is similar to the rte_eventdev struct= . > > > >Thanks, > >Erik