From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5713242D3D; Sat, 24 Jun 2023 13:52:45 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D012240691; Sat, 24 Jun 2023 13:52:44 +0200 (CEST) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by mails.dpdk.org (Postfix) with ESMTP id 9B72B40395 for ; Sat, 24 Jun 2023 13:52:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1687607562; x=1719143562; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=D9Ko21KAFTxTfqQew2bHeF/3dkxvybGGuDJoepnYIJA=; b=UnxH8qryuQRcqzv5Pq78RFfryxT/UcLyE1LVxK8zENOG1bQrlofwI4bU +gWejgrniP2CrSYjMTiUHOyTy2O1TZpJ1Oek6yYcnU3CNWat+G4kkVtpZ CbpWnIPbN/ttqLdXK2o5MveMT2774tYwb6DUS4tEEo86UqvClMrmUn46U Prpv7JlO7u+0emgMB11/M7YD3rJ5vEGxQ6qqlYHInLNDhVLMuczO6xdMl h4454uAbUVAJsn44ZQFQVGBjvjbkrrSUHdbzkfG2zO8s0psklZ9TnOVVu eOmJa7pWCBe6UPFDgR3MEkws4ovhoo2j+mveCu1BQSPS80c4WTAxJaY0X w==; X-IronPort-AV: E=McAfee;i="6600,9927,10750"; a="426926671" X-IronPort-AV: E=Sophos;i="6.01,155,1684825200"; d="scan'208";a="426926671" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jun 2023 04:52:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10750"; a="828666899" X-IronPort-AV: E=Sophos;i="6.01,155,1684825200"; d="scan'208";a="828666899" Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16]) by fmsmga002.fm.intel.com with ESMTP; 24 Jun 2023 04:52:40 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Sat, 24 Jun 2023 04:52:39 -0700 Received: from orsmsx601.amr.corp.intel.com (10.22.229.14) by ORSMSX610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Sat, 24 Jun 2023 04:52:39 -0700 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23 via Frontend Transport; Sat, 24 Jun 2023 04:52:39 -0700 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (104.47.56.173) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.23; Sat, 24 Jun 2023 04:52:38 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TCI9pnkXTY9/giYzH+F60ALEZPi1Rq1c0yFvWH0a1asVfpmHSwX8+TBhcb8qKwhD2n+ftIMlwc9H6wBSKw3t607l1ByI7KkfkMjxTIy+Qg54GFWqteyhM+euX83YnDgo82CTr2Pp+HvD+fZwxQWu9MkYzkOYKmN610oryHzlyyrgOR+sl99nz1NiX//mcZv9ntuDLdWehi4jouyDe9ueUINCTvWDJrmzfrDS2IjF8Yedw/C6eIjklp6Gmebj467ZapJwQkJFNuSMuAj01ZwxPeyHinj2sN9hWL/KTXXBRuDCZcqgYOsI/RBlgk/ruC88K2P8XCHzd7hJuH9IvlyxAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=l3d1h106G6CMX28MDRwaZYaT3rtI3pO0pir+szFRZoI=; b=BXpCU89x6FpZmzTytSsNxfvUuDUbHcTxnon2bBoA36SdNl28F9bOEM2BjwKGFyKWcyckPFpYD/tN6BA3M1Oxf9DGvWseZsUexf4oFu1QeN8YUDuAyXxC8a2rYrEXuOnYSMjC7EgxoyVSdc33ntfudYOtqlTGYw8CRSeC6/7ty5fpEBvhC2xrnoropJZFXiuGXMKN71KbodN3eWZIrzwo73/Bhed4ZPHEN0IZyqp1vsQT1salBQ5vwk1EKDPidsOq05a7GjHzqwk4KYVxn/dRGiOExbSUY355iq1VDHP/PoEIZo7ZMBPDiQI/mnPmAyN/WFp5LSIVWS8K5FXnxEZyhA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from SN7PR11MB7019.namprd11.prod.outlook.com (2603:10b6:806:2ae::22) by IA1PR11MB6098.namprd11.prod.outlook.com (2603:10b6:208:3d6::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.26; Sat, 24 Jun 2023 11:52:36 +0000 Received: from SN7PR11MB7019.namprd11.prod.outlook.com ([fe80::2d39:426d:f529:6088]) by SN7PR11MB7019.namprd11.prod.outlook.com ([fe80::2d39:426d:f529:6088%4]) with mapi id 15.20.6521.023; Sat, 24 Jun 2023 11:52:35 +0000 From: "Jiang, Cheng1" To: Anoob Joseph , "thomas@monjalon.net" , "Richardson, Bruce" , "mb@smartsharesystems.com" , "Xia, Chenbo" , Amit Prakash Shukla , "huangdengdui@huawei.com" , "Laatz, Kevin" , "fengchengwen@huawei.com" , Jerin Jacob Kollanukkaran CC: "dev@dpdk.org" , "Hu, Jiayu" , "Ding, Xuan" , "Ma, WenwuX" , "Wang, YuanX" , "He, Xingguang" , "Ling, WeiX" Subject: RE: [EXT] [PATCH v8] app/dma-perf: introduce dma-perf application Thread-Topic: [EXT] [PATCH v8] app/dma-perf: introduce dma-perf application Thread-Index: AQHZo0kNpe8Gy+tjUE+NMkL9JTEUOK+X+ByAgAHScEA= Date: Sat, 24 Jun 2023 11:52:34 +0000 Message-ID: References: <20230420072215.19069-1-cheng1.jiang@intel.com> <20230620065330.9999-1-cheng1.jiang@intel.com> In-Reply-To: Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SN7PR11MB7019:EE_|IA1PR11MB6098:EE_ x-ms-office365-filtering-correlation-id: e4ccba5d-b5b0-4c7f-9b78-08db74a97ffd x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: +y8iwnBbEmhqsJLbbl4AAUsyg6OMNFcQFZDGV89Ah6qWlzQeF90utg1j84+If94HoK58Wu00ILsqKz7k/ieoH/fY6myx+nD2KQd7dIHtxO+aP7xlQtePErId+ojxi/hGCD6rpBdgxjE2bc7+yZsev88EH8Tj2WbCDPibF88Mb7mpznqvt7W1vz6t663Ya2bp8w9E9QK8D/zr3mAukU3sjUDDkOAyOAqS4BcWREvEst0FRE5Pg2YXYy+OOWZA2bDptqNB13zJtlg9CvBAU0uhn2fffrTKtXKu9tSpd0TzL4X4jpukjksfidWiLOvKsjVrVCunRrsmjvXXf3P5f61e0JH/5utjsRNklFNTtQsJHlC6NqSsG372EKlghjgGd7xC2PGrCXz5Z4nzXzKR0M1AUtDHtV70ghbjT717Sf/3YGYDjp3+SD0eCbE7+XiC7MeO7+ouHvUV8Vm3gW8qo3800txI3Hk/6kFdRfD5PnvWVBmmGTVjTps74kdP+HkhP5E/eu8Kpj4yQMlOxNqPFPZ+Zwu2Fa354N4ItTsUudT6FtY2H2IMW4iIAZ16zlYfZIu36IpFomOtkxNl3Lhr0Tn8twN+X3a6k6FLdHHATg88puFqWB6d1VF9NlAeKh0U5FXVQng99i9JH6Nl1gMk4Wr98g== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN7PR11MB7019.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(39860400002)(136003)(346002)(366004)(396003)(376002)(451199021)(921005)(82960400001)(38100700002)(122000001)(38070700005)(66574015)(83380400001)(33656002)(86362001)(478600001)(54906003)(110136005)(7696005)(71200400001)(41300700001)(55016003)(66946007)(66556008)(66476007)(66446008)(64756008)(316002)(8676002)(76116006)(8936002)(53546011)(4326008)(6506007)(26005)(186003)(107886003)(9686003)(66899021)(2906002)(30864003)(5660300002)(52536014); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?ljIsmCHByU3jmzshaYQwFdJqvl651OFLrgRl9hLYD31HVXHGMhzi+UGiaY?= =?iso-8859-1?Q?N+zjiX0rPq9oBEjci4fTEsWrENSX1Ja9wqmXIuKL1ciqLSM4psSxv/A2QG?= =?iso-8859-1?Q?38xB/i7vDq6RNisHJhAWHSJY+Vp/gfNhCdxEMNi6I+88kBcOt692H7KECF?= =?iso-8859-1?Q?pERENmYKGm/0CJjl60Q7vygg5YCs7Nr8W2uIqKIhVNealVx07pF2h6s2tq?= =?iso-8859-1?Q?1wT53EBhuwvPe2BK3WrGL36RtKTAZkmvX6y1tJHjpOFRe4WmbLxzTjr1M/?= =?iso-8859-1?Q?TJ8qc89ynV3fIAg4t3+XAdm+FfY3dny3AKbWhl47r+rFGugZiJWH49xv+Y?= =?iso-8859-1?Q?JZR+UlEvxA6K5qaR4t/1cJ6TN+MBgGGiwuSGP98pBUTekBjRcXGjQtqCis?= =?iso-8859-1?Q?1z3a+KxTTGQ67DcDhnS8rxTyU26PrxSZ9W5VQh40fsvLIKI6W3KiIi73y8?= =?iso-8859-1?Q?mO+jc9Sb1MQHduiuJX65fUM1qlpG9jGj1+zWe91eXu89sbrPNUMEY1v7R7?= =?iso-8859-1?Q?6+GQO9ZptE6i2ZTatNS5qQ6UVpCyYFlqe7JlTNIztuzPipYqauY9ba/qrs?= =?iso-8859-1?Q?SBWELaM9NCE8mZmlSLTp4abLN5mEwUITsoPBN/3B/9MmWUaOnE3+rdIxYe?= =?iso-8859-1?Q?UtMiX3wT+aIsMa8pmCSRPfg26iejV623Fk348b2MYllvSz6bAtDNv70zzX?= =?iso-8859-1?Q?lotWCe+QdBEZ3/FxXH9Vns2FMu5CvjUGEEt5d7DmRoMwx/yai85OrnBbQd?= =?iso-8859-1?Q?StKATc0KHi5Ka7hkd/nlRJ60tT2hu7rBHteH/Zj7PDCI4DaBzO70czMz+V?= =?iso-8859-1?Q?4QnoiT9EVAWw+C2OGEfIQRcCv+Sc2c1pRyRmpLdbmn5qEfv6/T1VS7vZEQ?= =?iso-8859-1?Q?gOFs7CCYVJNALmcW5CkJlXisKYOjfQ9+NOkjb26wdLQ2QLQi6ZhBnwhYkP?= =?iso-8859-1?Q?csJ3oG1hwoIYhG3rx6M7W9LJNoSseWlmZEbS55Vvc1Pkf+1vByz40FradM?= =?iso-8859-1?Q?nFYRh6mKGOJKzomHN0aHdSuPPTczcT2dOyU3RKafRc7KFT2Qp4tiU3mVsN?= =?iso-8859-1?Q?PUKf3CX5Jz0ML357weX8LqRN/VVKVhEe6be60R13gi/iCh4sXnEfV2TNqa?= =?iso-8859-1?Q?FbbT0YSNnAQ1RD0q43tWDr7cnokpa9KVMv1+40pIvhad9ecHr5IoaE/m5P?= =?iso-8859-1?Q?e9zZANn2Ce/9msPz1a0/sSq/eft5ZKFgqKIw9HvmY+rfB2tr3n07D+mTcO?= =?iso-8859-1?Q?4e+F97dMKmijO0UQxa8HKXsSdV100EQUwPImT+knbj2FvH7uJbWDUrmki7?= =?iso-8859-1?Q?wmAKPb/ekm3CMJWdoYWeaC3Um1rbzrpUPzgOO1LRsnM7M3FIeLm2ai9LN8?= =?iso-8859-1?Q?xTaLZWFbMOv+b1XuLTnEOkDqyWOYtD5moBUtacJsq+5VwKBageJ39qjkxV?= =?iso-8859-1?Q?lBrMorOt/RHckH/LYUnlxMUE4y9dL/JrHKhbXPTeH7U6a/837VfvYJmTvb?= =?iso-8859-1?Q?MA5K8HcWfenaUtAJiQeHDS3nYVfOwIbBX3LeXuFL9mk5pyC8bf4fTIMN3J?= =?iso-8859-1?Q?kXEwotZNoVFiZ/70X8ZCxMi1r55NufrH+PEJYcbaB3/JMgKvqQCmc0z5CB?= =?iso-8859-1?Q?sEZbZG6eNvCPsMjrClcbVk+Y9L+oeFWFkR?= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN7PR11MB7019.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: e4ccba5d-b5b0-4c7f-9b78-08db74a97ffd X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Jun 2023 11:52:34.8634 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 2oMB5CBctAHx/ZfN1D59hbKTR/HriCb7rwXZevj+MY6VCrK6YHtFKWIJx8h1JeW7TEe8sVbA/kt6PTi8SCGy6Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB6098 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Anoob, Replies are inline. Thanks, Cheng > -----Original Message----- > From: Anoob Joseph > Sent: Friday, June 23, 2023 2:53 PM > To: Jiang, Cheng1 ; thomas@monjalon.net; > Richardson, Bruce ; > mb@smartsharesystems.com; Xia, Chenbo ; Amit > Prakash Shukla ; huangdengdui@huawei.com; > Laatz, Kevin ; fengchengwen@huawei.com; Jerin > Jacob Kollanukkaran > Cc: dev@dpdk.org; Hu, Jiayu ; Ding, Xuan > ; Ma, WenwuX ; Wang, > YuanX ; He, Xingguang ; > Ling, WeiX > Subject: RE: [EXT] [PATCH v8] app/dma-perf: introduce dma-perf applicatio= n >=20 > Hi Cheng, >=20 > Thanks for the new version. Please see inline. >=20 > Thanks, > Anoob >=20 > > -----Original Message----- > > From: Cheng Jiang > > Sent: Tuesday, June 20, 2023 12:24 PM > > To: thomas@monjalon.net; bruce.richardson@intel.com; > > mb@smartsharesystems.com; chenbo.xia@intel.com; Amit Prakash Shukla > > ; Anoob Joseph ; > > huangdengdui@huawei.com; kevin.laatz@intel.com; > > fengchengwen@huawei.com; Jerin Jacob Kollanukkaran > > > > Cc: dev@dpdk.org; jiayu.hu@intel.com; xuan.ding@intel.com; > > wenwux.ma@intel.com; yuanx.wang@intel.com; xingguang.he@intel.com; > > weix.ling@intel.com; Cheng Jiang > > Subject: [EXT] [PATCH v8] app/dma-perf: introduce dma-perf application > > > > External Email > > > > ---------------------------------------------------------------------- > > There are many high-performance DMA devices supported in DPDK now, > and > > these DMA devices can also be integrated into other modules of DPDK as > > accelerators, such as Vhost. Before integrating DMA into applications, > > developers need to know the performance of these DMA devices in > > various scenarios and the performance of CPUs in the same scenario, > > such as different buffer lengths. Only in this way can we know the > > target performance of the application accelerated by using them. This > > patch introduces a high-performance testing tool, which supports > > comparing the performance of CPU and DMA in different scenarios > > automatically with a pre- set config file. Memory Copy performance test > are supported for now. > > > > Signed-off-by: Cheng Jiang > > Signed-off-by: Jiayu Hu > > Signed-off-by: Yuan Wang > > Acked-by: Morten Br=F8rup > > Acked-by: Chenbo Xia > > --- > > v8: > > fixed string copy issue in parse_lcore(); > > improved some data display format; > > added doc in doc/guides/tools; > > updated release notes; > > > > v7: > > fixed some strcpy issues; > > removed cache setup in calling rte_pktmbuf_pool_create(); > > fixed some typos; > > added some memory free and null set operations; > > improved result calculation; > > v6: > > improved code based on Anoob's comments; > > fixed some code structure issues; > > v5: > > fixed some LONG_LINE warnings; > > v4: > > fixed inaccuracy of the memory footprint display; > > v3: > > fixed some typos; > > v2: > > added lcore/dmadev designation; > > added error case process; > > removed worker_threads parameter from config.ini; > > improved the logs; > > improved config file; > > > > app/meson.build | 1 + > > app/test-dma-perf/benchmark.c | 498 +++++++++++++++++++++ > > app/test-dma-perf/config.ini | 61 +++ > > app/test-dma-perf/main.c | 594 +++++++++++++++++++++++++ > > app/test-dma-perf/main.h | 69 +++ > > app/test-dma-perf/meson.build | 17 + > > doc/guides/rel_notes/release_23_07.rst | 6 + > > doc/guides/tools/dmaperf.rst | 103 +++++ > > doc/guides/tools/index.rst | 1 + > > 9 files changed, 1350 insertions(+) > > create mode 100644 app/test-dma-perf/benchmark.c create mode 100644 > > app/test-dma-perf/config.ini create mode 100644 app/test-dma- > > perf/main.c create mode 100644 app/test-dma-perf/main.h create mode > > 100644 app/test-dma-perf/meson.build create mode 100644 > > doc/guides/tools/dmaperf.rst > > >=20 > >=20 > > +/* Configuration of device. */ > > +static void > > +configure_dmadev_queue(uint32_t dev_id, uint32_t ring_size) { > > + uint16_t vchan =3D 0; > > + struct rte_dma_info info; > > + struct rte_dma_conf dev_config =3D { .nb_vchans =3D 1 }; > > + struct rte_dma_vchan_conf qconf =3D { > > + .direction =3D RTE_DMA_DIR_MEM_TO_MEM, > > + .nb_desc =3D ring_size > > + }; > > + > > + if (rte_dma_configure(dev_id, &dev_config) !=3D 0) > > + rte_exit(EXIT_FAILURE, "Error with dma configure.\n"); > > + > > + if (rte_dma_vchan_setup(dev_id, vchan, &qconf) !=3D 0) > > + rte_exit(EXIT_FAILURE, "Error with queue configuration.\n"); > > + > > + rte_dma_info_get(dev_id, &info); >=20 > [Anoob] This API can return errors. Better to add handling. [Cheng] Sure, I'll fix it in the next version. >=20 > > + if (info.nb_vchans !=3D 1) > > + rte_exit(EXIT_FAILURE, "Error, no configured queues > > reported on device id. %u\n", > > + dev_id); > > + > > + if (rte_dma_start(dev_id) !=3D 0) > > + rte_exit(EXIT_FAILURE, "Error with dma start.\n"); } > > + > > +static int > > +config_dmadevs(struct test_configure *cfg) { > > + uint32_t ring_size =3D cfg->ring_size.cur; > > + struct lcore_dma_map_t *ldm =3D &cfg->lcore_dma_map; > > + uint32_t nb_workers =3D ldm->cnt; > > + uint32_t i; > > + int dev_id; > > + uint16_t nb_dmadevs =3D 0; > > + char *dma_name; > > + > > + for (i =3D 0; i < ldm->cnt; i++) { > > + dma_name =3D ldm->dma_names[i]; > > + dev_id =3D rte_dma_get_dev_id_by_name(dma_name); > > + if (dev_id =3D=3D -1) { >=20 > [Anoob] Can you check the above API definition? I think it returns not ju= st -1 > in case of errors. [Cheng] Yes, you are right, I'll fix it in the next version. Thanks a lot. >=20 > > + fprintf(stderr, "Error: Fail to find DMA %s.\n", > > dma_name); > > + goto end; > > + } > > + > > + ldm->dma_ids[i] =3D dev_id; > > + configure_dmadev_queue(dev_id, ring_size); > > + ++nb_dmadevs; > > + } > > + > > +end: > > + if (nb_dmadevs < nb_workers) { > > + printf("Not enough dmadevs (%u) for all workers (%u).\n", > > nb_dmadevs, nb_workers); > > + return -1; > > + } > > + > > + printf("Number of used dmadevs: %u.\n", nb_dmadevs); > > + > > + return 0; > > +} > > + > > +static inline void > > +do_dma_submit_and_poll(uint16_t dev_id, uint64_t *async_cnt, > > + volatile struct worker_info *worker_info) { > > + int ret; > > + uint16_t nr_cpl; > > + > > + ret =3D rte_dma_submit(dev_id, 0); > > + if (ret < 0) { > > + rte_dma_stop(dev_id); > > + rte_dma_close(dev_id); > > + rte_exit(EXIT_FAILURE, "Error with dma submit.\n"); > > + } > > + > > + nr_cpl =3D rte_dma_completed(dev_id, 0, MAX_DMA_CPL_NB, NULL, > > NULL); > > + *async_cnt -=3D nr_cpl; > > + worker_info->total_cpl +=3D nr_cpl; > > +} > > + > > +static inline int > > +do_dma_mem_copy(void *p) >=20 > [Anoob] Just curious, why not pass struct lcore_params *para itself? Is i= t > because the pointer is volatile? If yes, then we can take an AI to split = the > struct into volatile and non-volatile parts. [Cheng] The reason I did it this way is because I want to launch this funct= ion on another core by spawning a new thread, and rte_eal_remote_launch() t= akes a void * as the parameter. That's why I passed void *p. Your suggesti= on to split the struct into volatile and non-volatile parts is quite reason= able. I am thinking about the best way to implement it. Thanks. >=20 > > +{ > > + const uint16_t *para_idx =3D (uint16_t *)p; > > + volatile struct lcore_params *para =3D lcores_p[*para_idx].v_ptr; > > + volatile struct worker_info *worker_info =3D &(para->worker_info); > > + const uint16_t dev_id =3D para->dev_id; > > + const uint32_t nr_buf =3D para->nr_buf; > > + const uint16_t kick_batch =3D para->kick_batch; > > + const uint32_t buf_size =3D para->buf_size; > > + struct rte_mbuf **srcs =3D para->srcs; > > + struct rte_mbuf **dsts =3D para->dsts; > > + uint16_t nr_cpl; > > + uint64_t async_cnt =3D 0; > > + uint32_t i; > > + uint32_t poll_cnt =3D 0; > > + int ret; > > + > > + worker_info->stop_flag =3D false; > > + worker_info->ready_flag =3D true; > > + > > + while (!worker_info->start_flag) > > + ; > > + > > + while (1) { > > + for (i =3D 0; i < nr_buf; i++) { > > +dma_copy: > > + ret =3D rte_dma_copy(dev_id, 0, > > rte_pktmbuf_iova(srcs[i]), > > + rte_pktmbuf_iova(dsts[i]), buf_size, 0); >=20 > [Anoob] Do we need to use ' rte_mbuf_data_iova' here instead of > 'rte_pktmbuf_iova'? [Cheng] yes rte_mbuf_data_iova is more appropriate, I'll fix it in the next= version. Thanks. >=20 > > + if (unlikely(ret < 0)) { > > + if (ret =3D=3D -ENOSPC) { > > + do_dma_submit_and_poll(dev_id, > > &async_cnt, worker_info); > > + goto dma_copy; > > + } else { > > + /* Error exit */ > > + rte_dma_stop(dev_id); >=20 > [Anoob] Missing rte_dma_close() here. Also, may be introduce a static voi= d > function so that rte_exit call etc won't be part of the fastpath loop. >=20 > May be something like below and you can call it from here and > "do_dma_submit_and_poll". >=20 > static void > error_exit(int dev_id) > { > /* Error exit */ > rte_dma_stop(dev_id); > rte_dma_close(dev_id); > rte_exit(EXIT_FAILURE, "DMA enqueue failed\n"); } >=20 [Cheng] I'm not so sure here. rte_dma_close() is called in the rte_exit(). = Do we still call it explicitly before rte_exit()? > > + rte_exit(EXIT_FAILURE, "DMA > > enqueue failed\n"); > > + } > > + } > > + async_cnt++; > > + > > + if ((async_cnt % kick_batch) =3D=3D 0) > > + do_dma_submit_and_poll(dev_id, > > &async_cnt, worker_info); > > + } > > + > > + if (worker_info->stop_flag) > > + break; > > + } > > + > > + rte_dma_submit(dev_id, 0); > > + while ((async_cnt > 0) && (poll_cnt++ < POLL_MAX)) { > > + nr_cpl =3D rte_dma_completed(dev_id, 0, > > MAX_DMA_CPL_NB, NULL, NULL); > > + async_cnt -=3D nr_cpl; > > + } > > + > > + return 0; > > +} > > + >=20 > >=20 > > +static int > > +setup_memory_env(struct test_configure *cfg, struct rte_mbuf ***srcs, > > + struct rte_mbuf ***dsts) > > +{ > > + unsigned int buf_size =3D cfg->buf_size.cur; > > + unsigned int nr_sockets; > > + uint32_t nr_buf =3D cfg->nr_buf; > > + > > + nr_sockets =3D rte_socket_count(); > > + if (cfg->src_numa_node >=3D nr_sockets || > > + cfg->dst_numa_node >=3D nr_sockets) { > > + printf("Error: Source or destination numa exceeds the acture > > numa nodes.\n"); > > + return -1; > > + } > > + > > + src_pool =3D rte_pktmbuf_pool_create("Benchmark_DMA_SRC", > > + nr_buf, > > + 0, > > + 0, > > + buf_size + RTE_PKTMBUF_HEADROOM, > > + cfg->src_numa_node); > > + if (src_pool =3D=3D NULL) { > > + PRINT_ERR("Error with source mempool creation.\n"); > > + return -1; > > + } > > + > > + dst_pool =3D rte_pktmbuf_pool_create("Benchmark_DMA_DST", > > + nr_buf, > > + 0, > > + 0, > > + buf_size + RTE_PKTMBUF_HEADROOM, > > + cfg->dst_numa_node); > > + if (dst_pool =3D=3D NULL) { > > + PRINT_ERR("Error with destination mempool creation.\n"); > > + return -1; > > + } > > + > > + *srcs =3D rte_malloc(NULL, nr_buf * sizeof(struct rte_mbuf *), 0); > > + if (*srcs =3D=3D NULL) { > > + printf("Error: srcs malloc failed.\n"); > > + return -1; > > + } > > + > > + *dsts =3D rte_malloc(NULL, nr_buf * sizeof(struct rte_mbuf *), 0); > > + if (*dsts =3D=3D NULL) { > > + printf("Error: dsts malloc failed.\n"); > > + return -1; > > + } > > + > > + if (rte_mempool_get_bulk(src_pool, (void **)*srcs, nr_buf) !=3D 0) { >=20 > [Anoob] Might be better to use 'rte_pktmbuf_alloc_bulk' since we use ' > rte_mbuf_data_iova' in the datapath and it is desirable to initialize it = properly > as an mbuf. Same comment below as well. >=20 [Cheng] sure, I'll fix it in the next version. > >=20 > > + > > + for (i =3D 0; i < nb_workers; i++) { > > + calc_result(buf_size, nr_buf, nb_workers, test_secs, > > + lcores_p[i].v_ptr->worker_info.test_cpl, > > + &memory, &avg_cycles, &bandwidth, &mops); > > + output_result(cfg->scenario_id, lcores_p[i].v_ptr->lcore_id, > > + lcores_p[i].v_ptr->dma_name, > > avg_cycles, buf_size, > > + nr_buf / nb_workers, memory, > > bandwidth, mops, is_dma); > > + } >=20 > [Anoob] Can you also print total_bandwidth & total_mops? It can be a simp= le > aggregation in the above loop. Would help when we are dealing with larger > number of queues but single hardware block. [Cheng] sure, good point. I'll add it in the next version, thanks. >=20 > > + > > +out: > > + /* free mbufs used in the test */ > > + if (srcs) >=20 > [Anoob] DPDK coding guidelines recommend a usage like below, > (if (srcs !=3D NULL) >=20 [Cheng] sure, thanks. I'll fix it in the next version. > > + rte_pktmbuf_free_bulk(srcs, nr_buf); > > + if (dsts) > > + rte_pktmbuf_free_bulk(dsts, nr_buf); > > + > > + /* free the points for the mbufs */ > > + rte_free(srcs); > > + srcs =3D NULL; > > + rte_free(dsts); > > + dsts =3D NULL; > > + > > + if (src_pool) { > > + rte_mempool_free(src_pool); > > + src_pool =3D NULL; > > + } > > + if (dst_pool) { > > + rte_mempool_free(dst_pool); > > + src_pool =3D NULL; >=20 > [Anoob] Should be dst_pool, right? [Cheng] yes, sorry for the miss. I'll fix it in the next version. >=20 > >=20 > > diff --git a/app/test-dma-perf/main.h b/app/test-dma-perf/main.h new > > file mode 100644 index 0000000000..215ac42673 > > --- /dev/null > > +++ b/app/test-dma-perf/main.h > > @@ -0,0 +1,69 @@ > > +/* SPDX-License-Identifier: BSD-3-Clause > > + * Copyright(c) 2023 Intel Corporation */ > > + > > +#ifndef _MAIN_H_ > > +#define _MAIN_H_ > > + > > + > > +#include > > +#include > > +#include > > +#include >=20 > [Anoob] Is the above include (rte_dmadev.h) required? [Cheng] you are right. It's not required. I'll remove it in the next versio= n. >=20 > > + > > +#ifndef __maybe_unused > > +#define __maybe_unused __rte_unused > > +#endif >=20 > [Anoob] Can you try to avoid this and use rte_unused or RTE_SET_USED in > the cache_flush_buf() function? [Cheng] sure, I'll fix it in the next version.