From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 93D4142CC4; Thu, 15 Jun 2023 10:01:13 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 831A341104; Thu, 15 Jun 2023 10:01:13 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 7FB4640A84 for ; Thu, 15 Jun 2023 10:01:10 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1686816070; x=1718352070; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=42tv1tI1WjDjjwIdZHaEDDQWBpWD5a8zIV+NUBcxXIs=; b=IW1hb9hn5YmFnFpnNr8abKlKHiEPg4bfpHlUYZmF8mFH+MXmEcGSMwUZ lWlSpmcm5s1H64GFsjPP+H67c0Yyxhsboo/qWgona3OrSPmxj8790Xqxc 5dLDpidal6N1ksaEXegkzHfA7eCg8B45H1+mCs+Rel++W7kt4llyMxbLJ keo3kJD6qepgwCprm5Z0dDun5/sD4u63nNCiwFEJfkAzGQOnhU+Am+0Wy N357cwpPXFGWJNz6shd8I+ZRYJglA4GM7L7H3rtfN4pkPcy/NSnvr6v8n j78kfPKMgNna4IoatwndP2a+26ejpWenqNyto/uEUFQB1L25GwBkU+qW2 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10741"; a="358838253" X-IronPort-AV: E=Sophos;i="6.00,244,1681196400"; d="scan'208";a="358838253" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Jun 2023 01:01:09 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10741"; a="1042549889" X-IronPort-AV: E=Sophos;i="6.00,244,1681196400"; d="scan'208";a="1042549889" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by fmsmga005.fm.intel.com with ESMTP; 15 Jun 2023 01:01:09 -0700 Received: from fmsmsx611.amr.corp.intel.com (10.18.126.91) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Thu, 15 Jun 2023 01:01:08 -0700 Received: from fmsmsx601.amr.corp.intel.com (10.18.126.81) by fmsmsx611.amr.corp.intel.com (10.18.126.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Thu, 15 Jun 2023 01:01:07 -0700 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23 via Frontend Transport; Thu, 15 Jun 2023 01:01:07 -0700 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (104.47.70.102) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.23; Thu, 15 Jun 2023 01:01:06 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BCA76opL9nCiKMRkJNSBS6TF0sNoduG7BiPzv4BBOupXssL5vSVL3Z5uNpVoqztCFOg9tOgLgEMnwcQJvUUj84q27Zz2+RJFj7wkCeSgjOzoj1YVx98IWE4O1SN9I4mL81ve7AWZSsWjkUdHHiZ+i5dZQhhTqicBgJLYPanmTssflbHIqkXme+Y/GmPdv0DgtJ/ico1rgIVG1zcJBWgVTnwuex4Crw0v/eUHwEz/4EOnc3IHgsciRZSMEF4M3krcPyTgq9gn1qYjEg0ncktZbM8OkBdqUnYteiKXj5hW+9Xz91gmHU+1RM6s8kS9DhAE7P/hH95jIjv5qmXAXTlgMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jVfbST21ZsvP0Wqqkg1ie6WnYs7EhthlsWLUPw42Ylo=; b=fjtBE2j7NR4O/Q4r4Mt9pFt90rUFydBZHOKzKcPzI1ewz+u9rG2jHq06ZPmB2LhXobei6/klXeUPtZGpRNfrDcSS7INkO6F0gf4DoUZRatHj6595xbgw5Zchp4QB8u3IvPFT4NKRtYt8EWLrQuqwmrQi3K9lmvH95LmZGS6eJPrwK1PDrLKIxN5to5Y/tALQSGinzznfK7pa5qPgR0d6T1PkmlUR1OJZqGQxFlkKK4uMdDeejO1NXjZRDUj6fAmjoNqR4bpaj8hZUhZn81PzjmhNBAN6Y9xEA4qR135gJHbBjCRw16/Mj3NQ3dDPLDVbQnJQEQng+e2PZyDdLpS36A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from SN7PR11MB7019.namprd11.prod.outlook.com (2603:10b6:806:2ae::22) by CH0PR11MB8144.namprd11.prod.outlook.com (2603:10b6:610:18d::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6455.37; Thu, 15 Jun 2023 08:01:03 +0000 Received: from SN7PR11MB7019.namprd11.prod.outlook.com ([fe80::2d39:426d:f529:6088]) by SN7PR11MB7019.namprd11.prod.outlook.com ([fe80::2d39:426d:f529:6088%4]) with mapi id 15.20.6477.037; Thu, 15 Jun 2023 08:01:03 +0000 From: "Jiang, Cheng1" To: Anoob Joseph , "thomas@monjalon.net" , "Richardson, Bruce" , "mb@smartsharesystems.com" , "Xia, Chenbo" , Amit Prakash Shukla CC: "dev@dpdk.org" , "Hu, Jiayu" , "Ding, Xuan" , "Ma, WenwuX" , "Wang, YuanX" , "He, Xingguang" Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application Thread-Topic: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application Thread-Index: AQHZnbUBtaI3/jfMaEW1JJ15Pu3zk6+LVxgAgAAWYWA= Date: Thu, 15 Jun 2023 08:01:03 +0000 Message-ID: References: <20230420072215.19069-1-cheng1.jiang@intel.com> <20230613043140.18101-1-cheng1.jiang@intel.com> In-Reply-To: Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SN7PR11MB7019:EE_|CH0PR11MB8144:EE_ x-ms-office365-filtering-correlation-id: 5ff2bb75-7b1c-46b3-5451-08db6d76aa50 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: +73ZhDjvCJTxzjSCz0SSQJ4NrKXhxa7tL77ik/H6YPUJ1VQngzlyx+kB+DGTVsMrMXgv+q3Xze4p2KafvVeWFcJ/tl1Ptu6Fp1j2gFekeH30z5IwHEP0KSleVjwOUHTtdwbp7eU2wiez8j/CiF2qr1hXe1vcdqmwfiLNCwjPBV1mg4Z35oyRYUjcduu05sa1QtW2Z+1nH5Zy2g6+BzHr2o9BQS9LgYk4W4Zd/eUFksn9TtU9d9QgGpdJU0RWxCPLu8jpjwtwaOaBiGFM74Q4908weUBkFKmd4ouZF/pFTBYndgJUWTaW93icAIME1cZsLhsU1phk3iweXyRwS5A39SQoZj7hPERA0hk/JBUuV4JHDfaV6QeevvPENGGBlTs7R3nT8f/DIQU3C0CM1Q/Weo6o1LqQfWTDO0M0mmC55BjfmvrhApQcRrZVM8pJAsMqJH9BfRQVCvJtrPLeM4nCNZ+0txJdPH+b53uAp1D8tWcWAlgUX86ixDA5rLZ4/psf0rFjHnaGcixkZRUgXOeAxH8837hwpxvziV3mm1OX52xb8ewf5YL/6U32/MvWLwgeOqKKUmwzbu3tEjzAlMXUJjQIW6TFf2ifCA8zIpRpC3WlJxcMl9k0Aqxw8rJLxMrC x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN7PR11MB7019.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(366004)(376002)(396003)(39860400002)(136003)(346002)(451199021)(8936002)(8676002)(52536014)(5660300002)(66446008)(64756008)(66946007)(76116006)(66556008)(4326008)(66476007)(316002)(54906003)(41300700001)(110136005)(2906002)(30864003)(71200400001)(478600001)(7696005)(6506007)(55016003)(122000001)(9686003)(107886003)(83380400001)(186003)(33656002)(26005)(66574015)(82960400001)(53546011)(38100700002)(86362001)(38070700005)(559001)(579004); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?b/BGusI+Uc6Kt7TUm41mwkQDCXGb3gU+UuTbPJzRWZVWA4VbJi44gh5h2b?= =?iso-8859-1?Q?Mz+nJPAbtUdkBxH0LsKvwP767ZqWOX89Ipg7gadb2wFgjuU7aV/MZRUKX2?= =?iso-8859-1?Q?gUlnYvpeVRZjIX/74jLWDDT/WbFBbbCLPCL+weU2jNiHhFihoJsueR+lSx?= =?iso-8859-1?Q?6y2Nm6bkaxv7GjqctsOCKBZ6dFBG2Uzq++tVWMG+0qC9A3/aQE5zVDlulg?= =?iso-8859-1?Q?YpIOlXOZNq4/wuKfcPfE/5v3N9vm297YLW+LUPZh1BxS5h9A0/AJT9sOb3?= =?iso-8859-1?Q?FAxmzXgVScjbDohuVpf7YDDK49ABo5efe4aXbusCDlEtvy6t1CmtPCA5rE?= =?iso-8859-1?Q?er/rfsdm5k2RH/nJ5ozafib50jg5OyjkeqxwyCGQUsyVRd8Z/Np0FZDccc?= =?iso-8859-1?Q?wwKhlRE4evtMi+BGZsbzMUs55lfBeoL3Ex214fG1woQVd0t+ASa6eNu9sB?= =?iso-8859-1?Q?K+ibFFy2ZeRiq0cxPhmtv6LNWo5ZIRWAl4XeoJyC1lL5LCGg2TOaQDKrWe?= =?iso-8859-1?Q?8FSssvXxveO79xVWoByxr0itGCJwjBmKn5Tir+XZjm6PGnk2KZgsfBH8b0?= =?iso-8859-1?Q?CXQ75Iq1k9iKTKXTF4xZj93bb/x7Jk0c2YaYtJN2ambm5xWAnYUGfyb4GA?= =?iso-8859-1?Q?lUa8EXSOoV670WGwk5Ohcb+x0hPi5l8YaakDr2zZaBh67PdP4+CWVuBgyf?= =?iso-8859-1?Q?CKpECPBG7hxrKQXfTuxgnK+0JKa6kkcQVwHVEF2dyNpVToxU5Y7mldRUQY?= =?iso-8859-1?Q?SN+2hsu5Ca6aSsq3g9ERQs3+O77G07L8ycFtOQ3IoCKR3RrJumrIx83aMo?= =?iso-8859-1?Q?UwspNqXYsw2sGbSTzdGG+A0D2Uv0viK/sRzaFPdI13WH7NK0tcB9SV8Wds?= =?iso-8859-1?Q?ttvD20c5oerWUzm1rlZ/PZYRToo1dbb/irBV6SPXeTtQ9P0qv/5OrTKBeB?= =?iso-8859-1?Q?uU7eaYtH5RxrQJgFMm0Km6zPAYrxcrd9yJuGFNwQER/UV7CT1pBfLIAqlV?= =?iso-8859-1?Q?2zkC1LYOFvxh4sQQyWuz3kBIJ+iiVNkAZnzK6WSiCV2cxewL0U5SXMM5Aw?= =?iso-8859-1?Q?1rsQcTsrVf3qJFY/WYUucfRqPwcQIg8fe23ofprI7woeAydI2mGiOLRlp9?= =?iso-8859-1?Q?vv0r7GSlrVztwnYaTRsSp30z31nwx9tSb772RAr+T8XccZDzAXfNr5nYZY?= =?iso-8859-1?Q?tEUse9a54QufNLYRa2wwhUtIn1vLJVj0qqjlvV/hugRaEwlhrLiwa2XedO?= =?iso-8859-1?Q?IBZuz1ugz8qYPb+Wo2Drx09rL8FjgGRalrFT68B6Of4HCShVEY/tmK963C?= =?iso-8859-1?Q?QEPnCbT7mEu37m3jM8mk22RIvnu6BG4GEiO+G2TP1p7ABdLoE3rgfbYtwI?= =?iso-8859-1?Q?u+nXC7Jf7pgQGeKcBswJdzqeOF0JP5py3Wg94Z5sbPVYo+YggNfWf3kzxF?= =?iso-8859-1?Q?m9nPexVcHUkkjz9ukfScFCBVMfSIaMBHc8joLf0S68tON2UigJpiT5/7MO?= =?iso-8859-1?Q?9FOjtrjnnCzoDCErQK6Fy9p0PAYK/SK5WQhz/SwXQZ0hIeBm3i6bvTcdqY?= =?iso-8859-1?Q?mCkzT7y0KlHO59E/ReskJzmjxw9/rDrvFIr9De6hDDAK8lr/IRe/q+3q9O?= =?iso-8859-1?Q?RhqISZk4QAT6mEbun8kn0kaDB1GW6JIXDU?= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN7PR11MB7019.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5ff2bb75-7b1c-46b3-5451-08db6d76aa50 X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Jun 2023 08:01:03.3768 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ZK5n9wrIrlS5bpk2GIIQwI/5JSASsccgF4Gew7yGzPCH02jWxaTjnbkUIr7mg8te+FWu5dNeobkOt2lZO0gUFg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR11MB8144 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi, Thanks for your comments, the replies are inline. Thanks, Cheng > -----Original Message----- > From: Anoob Joseph > Sent: Thursday, June 15, 2023 1:22 PM > To: Jiang, Cheng1 ; thomas@monjalon.net; > Richardson, Bruce ; mb@smartsharesystems.com; > Xia, Chenbo ; Amit Prakash Shukla > > Cc: dev@dpdk.org; Hu, Jiayu ; Ding, Xuan > ; Ma, WenwuX ; Wang, YuanX > ; He, Xingguang > Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf applicatio= n >=20 > Hi, >=20 > Thanks for working on the comments. Few more top level comment inline. >=20 > Thanks, > Anoob >=20 > > -----Original Message----- > > From: Cheng Jiang > > Sent: Tuesday, June 13, 2023 10:02 AM > > To: thomas@monjalon.net; bruce.richardson@intel.com; > > mb@smartsharesystems.com; chenbo.xia@intel.com; Amit Prakash Shukla > > ; Anoob Joseph > > Cc: dev@dpdk.org; jiayu.hu@intel.com; xuan.ding@intel.com; > > wenwux.ma@intel.com; yuanx.wang@intel.com; xingguang.he@intel.com; > > Cheng Jiang > > Subject: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application > > > > External Email > > > > ---------------------------------------------------------------------- > > There are many high-performance DMA devices supported in DPDK now, and > > these DMA devices can also be integrated into other modules of DPDK as > > accelerators, such as Vhost. Before integrating DMA into applications, > > developers need to know the performance of these DMA devices in > > various scenarios and the performance of CPUs in the same scenario, > > such as different buffer lengths. Only in this way can we know the > > target performance of the application accelerated by using them. This > > patch introduces a high-performance testing tool, which supports > > comparing the performance of CPU and DMA in different scenarios > > automatically with a pre- set config file. Memory Copy performance test= are > supported for now. > > > > Signed-off-by: Cheng Jiang > > Signed-off-by: Jiayu Hu > > Signed-off-by: Yuan Wang > > Acked-by: Morten Br=F8rup > > Acked-by: Chenbo Xia > > --- > > v6: > > improved code based on Anoob's comments; > > fixed some code structure issues; > > v5: > > fixed some LONG_LINE warnings; > > v4: > > fixed inaccuracy of the memory footprint display; > > v3: > > fixed some typos; > > v2: > > added lcore/dmadev designation; > > added error case process; > > removed worker_threads parameter from config.ini; > > improved the logs; > > improved config file; > > > > app/meson.build | 1 + > > app/test-dma-perf/benchmark.c | 477 ++++++++++++++++++++++++++++ > > app/test-dma-perf/config.ini | 59 ++++ > > app/test-dma-perf/main.c | 569 > > ++++++++++++++++++++++++++++++++++ > > app/test-dma-perf/main.h | 69 +++++ > > app/test-dma-perf/meson.build | 17 + > > 6 files changed, 1192 insertions(+) > > create mode 100644 app/test-dma-perf/benchmark.c create mode 100644 > > app/test-dma-perf/config.ini create mode 100644 app/test-dma- > > perf/main.c create mode 100644 app/test-dma-perf/main.h create mode > > 100644 app/test-dma-perf/meson.build > > > > diff --git a/app/meson.build b/app/meson.build index > > 74d2420f67..4fc1a83eba 100644 > > --- a/app/meson.build > > +++ b/app/meson.build > > @@ -19,6 +19,7 @@ apps =3D [ > > 'test-cmdline', > > 'test-compress-perf', > > 'test-crypto-perf', > > + 'test-dma-perf', > > 'test-eventdev', > > 'test-fib', > > 'test-flow-perf', > > diff --git a/app/test-dma-perf/benchmark.c b/app/test-dma- > > perf/benchmark.c new file mode 100644 index 0000000000..bc1ca82297 > > --- /dev/null > > +++ b/app/test-dma-perf/benchmark.c > > @@ -0,0 +1,477 @@ > > +/* SPDX-License-Identifier: BSD-3-Clause > > + * Copyright(c) 2023 Intel Corporation */ > > + > > +#include > > +#include > > +#include > > +#include > > + > > +#include > > +#include > > +#include > > +#include > > +#include > > + > > +#include "main.h" > > + > > +#define MAX_DMA_CPL_NB 255 > > + > > +#define TEST_WAIT_U_SECOND 10000 > > + > > +#define CSV_LINE_DMA_FMT "Scenario %u,%u,%s,%u,%u,%.2lf,%" PRIu64 > > ",%.3lf,%.3lf\n" > > +#define CSV_LINE_CPU_FMT "Scenario %u,%u,NA,%u,%u,%.2lf,%" PRIu64 > > ",%.3lf,%.3lf\n" > > + > > +struct worker_info { > > + bool ready_flag; > > + bool start_flag; > > + bool stop_flag; > > + uint32_t total_cpl; > > + uint32_t test_cpl; > > +}; > > + > > +struct lcore_params { > > + uint8_t scenario_id; > > + unsigned int lcore_id; > > + char *dma_name; > > + uint16_t worker_id; > > + uint16_t dev_id; > > + uint32_t nr_buf; > > + uint16_t kick_batch; > > + uint32_t buf_size; > > + uint16_t test_secs; > > + struct rte_mbuf **srcs; > > + struct rte_mbuf **dsts; > > + struct worker_info worker_info; > > +}; > > + > > +static struct rte_mempool *src_pool; > > +static struct rte_mempool *dst_pool; > > + > > +static volatile struct lcore_params *worker_params[MAX_WORKER_NB]; > > + > > +#define PRINT_ERR(...) print_err(__func__, __LINE__, __VA_ARGS__) > > + > > +static inline int > > +__rte_format_printf(3, 4) > > +print_err(const char *func, int lineno, const char *format, ...) { > > + va_list ap; > > + int ret; > > + > > + ret =3D fprintf(stderr, "In %s:%d - ", func, lineno); > > + va_start(ap, format); > > + ret +=3D vfprintf(stderr, format, ap); > > + va_end(ap); > > + > > + return ret; > > +} > > + > > +static inline void > > +calc_result(uint32_t buf_size, uint32_t nr_buf, uint16_t nb_workers, > > uint16_t test_secs, > > + uint32_t total_cnt, float *memory, uint32_t > > *ave_cycle, > > + float *bandwidth, float *mops) > > +{ > > + *memory =3D (float)(buf_size * (nr_buf / nb_workers) * 2) / (1024 * > > 1024); > > + *ave_cycle =3D test_secs * rte_get_timer_hz() / total_cnt; > > + *bandwidth =3D (buf_size * 8 * (rte_get_timer_hz() / > > (float)*ave_cycle)) / 1000000000; > > + *mops =3D (float)rte_get_timer_hz() / *ave_cycle / 1000000; } > > + > > +static void > > +output_result(uint8_t scenario_id, uint32_t lcore_id, char *dma_name, > > uint64_t ave_cycle, > > + uint32_t buf_size, uint32_t nr_buf, float memory, > > + float bandwidth, float mops, bool is_dma) { > > + if (is_dma) > > + printf("lcore %u, DMA %s:\n", lcore_id, dma_name); > > + else > > + printf("lcore %u\n", lcore_id); > > + > > + printf("average cycles/op: %" PRIu64 ", buffer size: %u, nr_buf: %u, > > memory: %.2lfMB, frequency: %" PRIu64 ".\n", > > + ave_cycle, buf_size, nr_buf, memory, > > rte_get_timer_hz()); > > + printf("Average bandwidth: %.3lfGbps, MOps: %.3lf\n", bandwidth, > > +mops); > > + > > + if (is_dma) > > + snprintf(output_str[lcore_id], MAX_OUTPUT_STR_LEN, > > CSV_LINE_DMA_FMT, > > + scenario_id, lcore_id, dma_name, buf_size, > > + nr_buf, memory, ave_cycle, bandwidth, mops); > > + else > > + snprintf(output_str[lcore_id], MAX_OUTPUT_STR_LEN, > > CSV_LINE_CPU_FMT, > > + scenario_id, lcore_id, buf_size, > > + nr_buf, memory, ave_cycle, bandwidth, mops); } > > + > > +static inline void > > +cache_flush_buf(__maybe_unused struct rte_mbuf **array, > > + __maybe_unused uint32_t buf_size, > > + __maybe_unused uint32_t nr_buf) > > +{ > > +#ifdef RTE_ARCH_X86_64 > > + char *data; > > + struct rte_mbuf **srcs =3D array; > > + uint32_t i, offset; > > + > > + for (i =3D 0; i < nr_buf; i++) { > > + data =3D rte_pktmbuf_mtod(srcs[i], char *); > > + for (offset =3D 0; offset < buf_size; offset +=3D 64) > > + __builtin_ia32_clflush(data + offset); > > + } > > +#endif > > +} > > + > > +/* Configuration of device. */ > > +static void > > +configure_dmadev_queue(uint32_t dev_id, uint32_t ring_size) { > > + uint16_t vchan =3D 0; > > + struct rte_dma_info info; > > + struct rte_dma_conf dev_config =3D { .nb_vchans =3D 1 }; > > + struct rte_dma_vchan_conf qconf =3D { > > + .direction =3D RTE_DMA_DIR_MEM_TO_MEM, > > + .nb_desc =3D ring_size > > + }; > > + > > + if (rte_dma_configure(dev_id, &dev_config) !=3D 0) > > + rte_exit(EXIT_FAILURE, "Error with dma configure.\n"); > > + > > + if (rte_dma_vchan_setup(dev_id, vchan, &qconf) !=3D 0) > > + rte_exit(EXIT_FAILURE, "Error with queue configuration.\n"); > > + > > + rte_dma_info_get(dev_id, &info); > > + if (info.nb_vchans !=3D 1) > > + rte_exit(EXIT_FAILURE, "Error, no configured queues > > reported on device id. %u\n", > > + dev_id); > > + > > + if (rte_dma_start(dev_id) !=3D 0) > > + rte_exit(EXIT_FAILURE, "Error with dma start.\n"); } > > + > > +static int > > +config_dmadevs(struct test_configure *cfg) { > > + uint32_t ring_size =3D cfg->ring_size.cur; > > + struct lcore_dma_map_t *ldm =3D &cfg->lcore_dma_map; > > + uint32_t nb_workers =3D ldm->cnt; > > + uint32_t i; > > + int dev_id; > > + uint16_t nb_dmadevs =3D 0; > > + char *dma_name; > > + > > + for (i =3D 0; i < ldm->cnt; i++) { > > + dma_name =3D ldm->dma_names[i]; > > + dev_id =3D rte_dma_get_dev_id_by_name(dma_name); > > + if (dev_id =3D=3D -1) { > > + fprintf(stderr, "Error: Fail to find DMA %s.\n", > > dma_name); > > + goto end; > > + } > > + > > + ldm->dma_ids[i] =3D dev_id; > > + configure_dmadev_queue(dev_id, ring_size); > > + ++nb_dmadevs; > > + } > > + > > +end: > > + if (nb_dmadevs < nb_workers) { > > + printf("Not enough dmadevs (%u) for all workers (%u).\n", > > nb_dmadevs, nb_workers); > > + return -1; > > + } > > + > > + printf("Number of used dmadevs: %u.\n", nb_dmadevs); > > + > > + return 0; > > +} > > + > > +#define POLL_MAX 1000 > > + > > + >=20 > [Anoob] Extra blank line. You can consider removing. [Cheng] sure, sorry for the miss. >=20 > > +static inline void > > +do_dma_submit_and_poll(uint16_t dev_id, uint64_t *async_cnt, > > + volatile struct worker_info *worker_info) { > > + int ret; > > + uint16_t nr_cpl; > > + > > + ret =3D rte_dma_submit(dev_id, 0); > > + if (ret < 0) { > > + rte_dma_stop(dev_id); > > + rte_dma_close(dev_id); > > + rte_exit(EXIT_FAILURE, "Error with dma submit.\n"); > > + } > > + > > + nr_cpl =3D rte_dma_completed(dev_id, 0, MAX_DMA_CPL_NB, NULL, > > NULL); > > + *async_cnt -=3D nr_cpl; > > + worker_info->total_cpl +=3D nr_cpl; > > +} > > + > > +static inline int > > +do_dma_mem_copy(void *p) > > +{ > > + const uint16_t *para_idx =3D (uint16_t *)p; > > + volatile struct lcore_params *para =3D worker_params[*para_idx]; > > + volatile struct worker_info *worker_info =3D &(para->worker_info); > > + const uint16_t dev_id =3D para->dev_id; > > + const uint32_t nr_buf =3D para->nr_buf; > > + const uint16_t kick_batch =3D para->kick_batch; > > + const uint32_t buf_size =3D para->buf_size; > > + struct rte_mbuf **srcs =3D para->srcs; > > + struct rte_mbuf **dsts =3D para->dsts; > > + uint16_t nr_cpl; > > + uint64_t async_cnt =3D 0; > > + uint32_t i; > > + uint32_t poll_cnt =3D 0; > > + int ret; > > + > > + worker_info->stop_flag =3D false; > > + worker_info->ready_flag =3D true; > > + > > + while (!worker_info->start_flag) > > + ; > > + > > + while (1) { > > + for (i =3D 0; i < nr_buf; i++) { > > +dma_copy: > > + ret =3D rte_dma_copy(dev_id, 0, > > rte_pktmbuf_iova(srcs[i]), > > + rte_pktmbuf_iova(dsts[i]), buf_size, 0); > > + if (unlikely(ret < 0)) { > > + if (ret =3D=3D -ENOSPC) { > > + do_dma_submit_and_poll(dev_id, > > &async_cnt, worker_info); > > + goto dma_copy; > > + } else { > > + /* Error exit */ > > + rte_dma_stop(dev_id); > > + rte_exit(EXIT_FAILURE, "DMA > > enqueue failed\n"); > > + } > > + } > > + async_cnt++; > > + > > + if ((async_cnt % kick_batch) =3D=3D 0) > > + do_dma_submit_and_poll(dev_id, > > &async_cnt, worker_info); > > + } > > + > > + if (worker_info->stop_flag) > > + break; > > + } > > + > > + rte_dma_submit(dev_id, 0); > > + while ((async_cnt > 0) && (poll_cnt++ < POLL_MAX)) { > > + nr_cpl =3D rte_dma_completed(dev_id, 0, > > MAX_DMA_CPL_NB, NULL, NULL); > > + async_cnt -=3D nr_cpl; > > + } > > + > > + return 0; > > +} > > + > > +static inline int > > +do_cpu_mem_copy(void *p) > > +{ > > + const uint16_t *para_idx =3D (uint16_t *)p; > > + volatile struct lcore_params *para =3D worker_params[*para_idx]; > > + volatile struct worker_info *worker_info =3D &(para->worker_info); > > + const uint32_t nr_buf =3D para->nr_buf; > > + const uint32_t buf_size =3D para->buf_size; > > + struct rte_mbuf **srcs =3D para->srcs; > > + struct rte_mbuf **dsts =3D para->dsts; > > + uint32_t i; > > + > > + worker_info->stop_flag =3D false; > > + worker_info->ready_flag =3D true; > > + > > + while (!worker_info->start_flag) > > + ; > > + > > + while (1) { > > + for (i =3D 0; i < nr_buf; i++) { > > + /* copy buffer form src to dst */ > > + rte_memcpy((void > > *)(uintptr_t)rte_mbuf_data_iova(dsts[i]), > > + (void > > *)(uintptr_t)rte_mbuf_data_iova(srcs[i]), > > + (size_t)buf_size); > > + worker_info->total_cpl++; > > + } > > + if (worker_info->stop_flag) > > + break; > > + } > > + > > + return 0; > > +} > > + > > +static int > > +setup_memory_env(struct test_configure *cfg, struct rte_mbuf ***srcs, > > + struct rte_mbuf ***dsts) > > +{ > > + unsigned int buf_size =3D cfg->buf_size.cur; > > + unsigned int nr_sockets; > > + uint32_t nr_buf =3D cfg->nr_buf; > > + > > + nr_sockets =3D rte_socket_count(); > > + if (cfg->src_numa_node >=3D nr_sockets || > > + cfg->dst_numa_node >=3D nr_sockets) { > > + printf("Error: Source or destination numa exceeds the acture > > numa nodes.\n"); > > + return -1; > > + } > > + > > + src_pool =3D rte_pktmbuf_pool_create("Benchmark_DMA_SRC", > > + nr_buf, /* n =3D=3D num elements */ > > + 64, /* cache size */ > > + 0, /* priv size */ > > + buf_size + RTE_PKTMBUF_HEADROOM, > > + cfg->src_numa_node); > > + if (src_pool =3D=3D NULL) { > > + PRINT_ERR("Error with source mempool creation.\n"); > > + return -1; > > + } > > + > > + dst_pool =3D rte_pktmbuf_pool_create("Benchmark_DMA_DST", > > + nr_buf, /* n =3D=3D num elements */ > > + 64, /* cache size */ >=20 > [Anoob] We do not alloc or free pointers in the datapath, right? So why b= other > with cache? [Cheng] Yes, you are right, the cache size is not necessary here, I'll fix = it in the next version. >=20 > > + 0, /* priv size */ > > + buf_size + RTE_PKTMBUF_HEADROOM, > > + cfg->dst_numa_node); > > + if (dst_pool =3D=3D NULL) { > > + PRINT_ERR("Error with destination mempool creation.\n"); > > + return -1; > > + } > > + > > + *srcs =3D rte_malloc(NULL, nr_buf * sizeof(struct rte_mbuf *), 0); > > + if (*srcs =3D=3D NULL) { > > + printf("Error: srcs malloc failed.\n"); > > + return -1; > > + } >=20 > [Anoob] Are we freeing these memory? The ones allocated with rte_malloc. [Cheng] yes, we freed the memory in the end of mem_copy_benchmark() when we= finished the test. >=20 > > + > > + *dsts =3D rte_malloc(NULL, nr_buf * sizeof(struct rte_mbuf *), 0); > > + if (*dsts =3D=3D NULL) { > > + printf("Error: dsts malloc failed.\n"); > > + return -1; > > + } > > + > > + if (rte_mempool_get_bulk(src_pool, (void **)*srcs, nr_buf) !=3D 0) { > > + printf("get src mbufs failed.\n"); > > + return -1; > > + } > > + if (rte_mempool_get_bulk(dst_pool, (void **)*dsts, nr_buf) !=3D 0) { > > + printf("get dst mbufs failed.\n"); > > + return -1; > > + } > > + > > + return 0; > > +} > > + > > +void > > +mem_copy_benchmark(struct test_configure *cfg, bool is_dma) { > > + uint16_t i; > > + uint32_t offset; > > + unsigned int lcore_id =3D 0; > > + struct rte_mbuf **srcs =3D NULL, **dsts =3D NULL; > > + struct lcore_dma_map_t *ldm =3D &cfg->lcore_dma_map; > > + unsigned int buf_size =3D cfg->buf_size.cur; > > + uint16_t kick_batch =3D cfg->kick_batch.cur; > > + uint32_t nr_buf =3D cfg->nr_buf =3D (cfg->mem_size.cur * 1024 * 1024)= / > > (cfg->buf_size.cur * 2); > > + uint16_t nb_workers =3D ldm->cnt; > > + uint16_t test_secs =3D cfg->test_secs; > > + float memory; > > + uint32_t avg_cycles =3D 0; > > + float mops; > > + float bandwidth; > > + > > + if (setup_memory_env(cfg, &srcs, &dsts) < 0) > > + goto out; > > + > > + if (is_dma) > > + if (config_dmadevs(cfg) < 0) > > + goto out; > > + > > + if (cfg->cache_flush) { > > + cache_flush_buf(srcs, buf_size, nr_buf); > > + cache_flush_buf(dsts, buf_size, nr_buf); > > + rte_mb(); > > + } > > + > > + printf("Start testing....\n"); > > + > > + for (i =3D 0; i < nb_workers; i++) { > > + lcore_id =3D ldm->lcores[i]; > > + offset =3D nr_buf / nb_workers * i; > > + > > + worker_params[i] =3D rte_malloc(NULL, sizeof(struct > > lcore_params), 0); > > + if (!worker_params[i]) { > > + printf("lcore parameters malloc failure for lcore > > %d\n", lcore_id); > > + break; > > + } >=20 > [Anoob] Are we freeing the above memory? [Cheng] sorry, I missed that, I'll add worker_params memory free in the nex= t version, thanks. >=20 > > + if (is_dma) { > > + worker_params[i]->dma_name =3D ldm- > > >dma_names[i]; > > + worker_params[i]->dev_id =3D ldm->dma_ids[i]; > > + worker_params[i]->kick_batch =3D kick_batch; > > + } > > + worker_params[i]->worker_id =3D i; > > + worker_params[i]->nr_buf =3D (uint32_t)(nr_buf / > > nb_workers); > > + worker_params[i]->buf_size =3D buf_size; > > + worker_params[i]->test_secs =3D test_secs; > > + worker_params[i]->srcs =3D srcs + offset; > > + worker_params[i]->dsts =3D dsts + offset; > > + worker_params[i]->scenario_id =3D cfg->scenario_id; > > + worker_params[i]->lcore_id =3D lcore_id; > > + > > + if (is_dma) > > + rte_eal_remote_launch(do_dma_mem_copy, (void > > *)(&i), lcore_id); > > + else > > + rte_eal_remote_launch(do_cpu_mem_copy, (void > > *)(&i), lcore_id); > > + } > > + > > + while (1) { > > + bool ready =3D true; > > + for (i =3D 0; i < nb_workers; i++) { > > + if (worker_params[i]->worker_info.ready_flag =3D=3D > > false) { > > + ready =3D 0; > > + break; > > + } > > + } > > + if (ready) > > + break; > > + } > > + > > + for (i =3D 0; i < nb_workers; i++) > > + worker_params[i]->worker_info.start_flag =3D true; > > + > > + usleep(TEST_WAIT_U_SECOND); > > + for (i =3D 0; i < nb_workers; i++) > > + worker_params[i]->worker_info.test_cpl =3D > > +worker_params[i]->worker_info.total_cpl; > > + > > + usleep(test_secs * 1000 * 1000); > > + for (i =3D 0; i < nb_workers; i++) > > + worker_params[i]->worker_info.test_cpl =3D > > worker_params[i]->worker_info.total_cpl - > > + worker_params[i]- > > >worker_info.test_cpl; > > + > > + for (i =3D 0; i < nb_workers; i++) > > + worker_params[i]->worker_info.stop_flag =3D true; > > + > > + rte_eal_mp_wait_lcore(); > > + > > + for (i =3D 0; i < nb_workers; i++) { > > + calc_result(buf_size, nr_buf, nb_workers, test_secs, > > + worker_params[i]->worker_info.test_cpl, > > + &memory, &avg_cycles, &bandwidth, &mops); > > + output_result(cfg->scenario_id, worker_params[i]->lcore_id, > > + worker_params[i]->dma_name, > > avg_cycles, buf_size, > > + nr_buf / nb_workers, memory, > > bandwidth, mops, is_dma); > > + } > > + > > +out: > > + /* free env */ > > + if (srcs) > > + rte_pktmbuf_free_bulk(srcs, nr_buf); > > + if (dsts) > > + rte_pktmbuf_free_bulk(dsts, nr_buf); > > + > > + if (src_pool) > > + rte_mempool_free(src_pool); > > + if (dst_pool) > > + rte_mempool_free(dst_pool); > > + > > + if (is_dma) { > > + for (i =3D 0; i < nb_workers; i++) { > > + printf("Stopping dmadev %d\n", ldm->dma_ids[i]); > > + rte_dma_stop(ldm->dma_ids[i]); > > + } > > + } > > +} > > diff --git a/app/test-dma-perf/config.ini > > b/app/test-dma-perf/config.ini new file mode 100644 index > > 0000000000..2fd9c3c387 > > --- /dev/null > > +++ b/app/test-dma-perf/config.ini > > @@ -0,0 +1,59 @@ > > + > > +; This is an example configuration file for dma-perf, which details > > +the meanings of each parameter ; and instructions on how to use dma-pe= rf. > > + > > +; Supported test types are DMA_MEM_COPY and CPU_MEM_COPY. > > + > > +; Parameters: > > +; "mem_size" denotes the size of the memory footprint. > > +; "buf_size" denotes the memory size of a single operation. > > +; "dma_ring_size" denotes the dma ring buffer size. It should be > > +greater > > than 64 normally. > > +; "kick_batch" denotes the dma operation batch size, and should be > > +greater > > than 1 normally. > > + > > +; The format for variables is variable=3Dfirst,last,increment,ADD|MUL. > > + > > +; src_numa_node is used to control the numa node where the source > > memory is allocated. > > +; dst_numa_node is used to control the numa node where the > > +destination > > memory is allocated. > > + > > +; cache_flush is used to determine whether or not the cache should be > > +flushed, with 1 indicating to ; flush and 0 indicating to not flush. > > + > > +; test_seconds controls the test time of the whole case. > > + > > +; To use DMA for a test, please specify the "lcore_dma" parameter. > > +; If you have already set the "-l" and "-a" parameters using EAL, ; > > +make sure that the value of "lcore_dma" falls within their range of > > +the > > values. > > + > > +; To use CPU for a test, please specify the "lcore" parameter. > > +; If you have already set the "-l" and "-a" parameters using EAL, ; > > +make sure that the value of "lcore" falls within their range of values= . > > + > > +; To specify a configuration file, use the "--config" flag followed > > +by the path > > to the file. > > + > > +; To specify a result file, use the "--result" flag followed by the > > +path to the > > file. > > +; If you do not specify a result file, one will be generated with the > > +same name as the configuration ; file, with the addition of > > +"_result.csv" at > > the end. > > + > > +[case1] > > +type=3DDMA_MEM_COPY > > +mem_size=3D10 > > +buf_size=3D64,8192,2,MUL > > +dma_ring_size=3D1024 > > +kick_batch=3D32 > > +src_numa_node=3D0 > > +dst_numa_node=3D0 > > +cache_flush=3D0 > > +test_seconds=3D2 > > +lcore_dma=3Dlcore10@0000:00:04.2, lcore11@0000:00:04.3 >=20 > [Anoob] Isn't it better if we allow user to specify DMA dev ID rather tha= n the PCI > DBDF? >=20 > In the long run, I would expect config file to provide {core, dma_dev_id, > queue_id} >=20 > Another thought is why to expose this at all? If we can restrict this per= f > application to have one thread only use one vchan, then application can e= asily > create this mapping in run time. Unless you want one thread to use 2 diff= erent > vchans which may not be desirable since this is a standalone perf app. [Cheng] Thank you for the feedback. Here are my thoughts: Firstly, the user may not know which device the DMA dev ID corresponds to, = or which NUMA node it is on. In my example, I used the CBDMA environment, s= o I did not specify the work queue ID. When using DSA, the configuration wo= uld be something like lcore10@0000:00:04.2-q0 which contains core, dma and = work queue id. The reason for exposing these options is that we want the us= er to fully understand which cores and devices are being used so that they = know exactly where the performance data is coming from. For example, perfor= mance when cores and DMA devices are not on the same NUMA node, etc. This a= llows the testing scenario to be precise and flexible. If the application h= andles the mapping itself, the user loses control over the mapping and may = not get the performance data they want. We believe control should be given = to the user rather than the application. =20 >=20 > > +eal_args=3D--in-memory --file-prefix=3Dtest > > + > > +[case2] > > +type=3DCPU_MEM_COPY > > +mem_size=3D10 > > +buf_size=3D64,8192,2,MUL > > +src_numa_node=3D0 > > +dst_numa_node=3D1 > > +cache_flush=3D0 > > +test_seconds=3D2 > > +lcore =3D 3, 4 > > +eal_args=3D--in-memory --no-pci > > diff --git a/app/test-dma-perf/main.c b/app/test-dma-perf/main.c new > > file mode 100644 index 0000000000..d65655b87b > > --- /dev/null > > +++ b/app/test-dma-perf/main.c > > @@ -0,0 +1,569 @@ > > +/* SPDX-License-Identifier: BSD-3-Clause > > + * Copyright(c) 2023 Intel Corporation */ > > + > > +#include > > +#include > > +#include > > +#include > > +#include > > +#include > > +#include > > +#include > > +#include > > + > > +#include > > +#include > > +#include > > +#include > > + > > +#include "main.h" > > + > > +#define CSV_HDR_FMT "Case %u : %s,lcore,DMA,buffer > > size,nr_buf,memory(MB),cycle,bandwidth(Gbps),MOps\n" > > + > > +#define MAX_EAL_PARAM_NB 100 > > +#define MAX_EAL_PARAM_LEN 1024 > > + > > +#define DMA_MEM_COPY "DMA_MEM_COPY" > > +#define CPU_MEM_COPY "CPU_MEM_COPY" > > + > > +#define CMDLINE_CONFIG_ARG "--config" > > +#define CMDLINE_RESULT_ARG "--result" > > + > > +#define MAX_PARAMS_PER_ENTRY 4 > > + > > +#define MAX_LONG_OPT_SZ 64 > > + > > +enum { > > + TEST_TYPE_NONE =3D 0, > > + TEST_TYPE_DMA_MEM_COPY, > > + TEST_TYPE_CPU_MEM_COPY > > +}; > > + > > +#define MAX_TEST_CASES 16 > > +static struct test_configure test_cases[MAX_TEST_CASES]; > > + > > +char output_str[MAX_WORKER_NB][MAX_OUTPUT_STR_LEN]; > > + > > +static FILE *fd; > > + > > +static void > > +output_csv(bool need_blankline) > > +{ > > + uint32_t i; > > + > > + if (need_blankline) { > > + fprintf(fd, ",,,,,,,,\n"); > > + fprintf(fd, ",,,,,,,,\n"); > > + } > > + > > + for (i =3D 0; i < RTE_DIM(output_str); i++) { > > + if (output_str[i][0]) { > > + fprintf(fd, "%s", output_str[i]); > > + output_str[i][0] =3D '\0'; > > + } > > + } > > + > > + fflush(fd); > > +} > > + > > +static void > > +output_env_info(void) > > +{ > > + snprintf(output_str[0], MAX_OUTPUT_STR_LEN, "test > > environment:\n"); > > + snprintf(output_str[1], MAX_OUTPUT_STR_LEN, "CPU frequency,%" > > + PRIu64 "\n", rte_get_timer_hz()); > > + > > + output_csv(true); > > +} > > + > > +static void > > +output_header(uint32_t case_id, struct test_configure *case_cfg) { > > + snprintf(output_str[0], MAX_OUTPUT_STR_LEN, > > + CSV_HDR_FMT, case_id, case_cfg->test_type_str); > > + > > + output_csv(true); > > +} > > + > > +static void > > +run_test_case(struct test_configure *case_cfg) { > > + switch (case_cfg->test_type) { > > + case TEST_TYPE_DMA_MEM_COPY: > > + mem_copy_benchmark(case_cfg, true); > > + break; > > + case TEST_TYPE_CPU_MEM_COPY: > > + mem_copy_benchmark(case_cfg, false); > > + break; > > + default: > > + printf("Unknown test type. %s\n", case_cfg->test_type_str); > > + break; > > + } > > +} > > + > > +static void > > +run_test(uint32_t case_id, struct test_configure *case_cfg) { > > + uint32_t i; > > + uint32_t nb_lcores =3D rte_lcore_count(); > > + struct test_configure_entry *mem_size =3D &case_cfg->mem_size; > > + struct test_configure_entry *buf_size =3D &case_cfg->buf_size; > > + struct test_configure_entry *ring_size =3D &case_cfg->ring_size; > > + struct test_configure_entry *kick_batch =3D &case_cfg->kick_batch; > > + struct test_configure_entry dummy =3D { 0 }; > > + struct test_configure_entry *var_entry =3D &dummy; > > + > > + for (i =3D 0; i < RTE_DIM(output_str); i++) > > + memset(output_str[i], 0, MAX_OUTPUT_STR_LEN); > > + > > + if (nb_lcores <=3D case_cfg->lcore_dma_map.cnt) { > > + printf("Case %u: Not enough lcores.\n", case_id); > > + return; > > + } > > + > > + printf("Number of used lcores: %u.\n", nb_lcores); > > + > > + if (mem_size->incr !=3D 0) > > + var_entry =3D mem_size; > > + > > + if (buf_size->incr !=3D 0) > > + var_entry =3D buf_size; > > + > > + if (ring_size->incr !=3D 0) > > + var_entry =3D ring_size; > > + > > + if (kick_batch->incr !=3D 0) > > + var_entry =3D kick_batch; > > + > > + case_cfg->scenario_id =3D 0; > > + > > + output_header(case_id, case_cfg); > > + > > + for (var_entry->cur =3D var_entry->first; var_entry->cur <=3D var_ent= ry- > > >last;) { > > + case_cfg->scenario_id++; > > + printf("\nRunning scenario %d\n", case_cfg->scenario_id); > > + > > + run_test_case(case_cfg); > > + output_csv(false); > > + > > + if (var_entry->op =3D=3D OP_ADD) > > + var_entry->cur +=3D var_entry->incr; > > + else if (var_entry->op =3D=3D OP_MUL) > > + var_entry->cur *=3D var_entry->incr; > > + else > > + break; > > + } > > +} > > + > > +static int > > +parse_lcore(struct test_configure *test_case, const char *value) { > > + size_t len =3D strlen(value); > > + char *input =3D (char *) malloc((len + 1) * sizeof(char)); > > + strcpy(input, value); > > + struct lcore_dma_map_t *lcore_dma_map =3D &(test_case- > > >lcore_dma_map); > > + > > + if (test_case =3D=3D NULL || value =3D=3D NULL) > > + return -1; > > + > > + memset(lcore_dma_map, 0, sizeof(struct lcore_dma_map_t)); > > + > > + char *token =3D strtok(input, ", "); > > + while (token !=3D NULL) { > > + if (lcore_dma_map->cnt >=3D MAX_LCORE_NB) { > > + free(input); > > + return -1; > > + } > > + > > + uint16_t lcore_id =3D atoi(token); > > + lcore_dma_map->lcores[lcore_dma_map->cnt++] =3D lcore_id; > > + > > + token =3D strtok(NULL, ", "); > > + } > > + > > + free(input); > > + return 0; > > +} > > + > > +static int > > +parse_lcore_dma(struct test_configure *test_case, const char *value) { > > + struct lcore_dma_map_t *lcore_dma_map; > > + char *input =3D strndup(value, strlen(value) + 1); > > + char *addrs =3D input; > > + char *ptrs[2]; > > + char *start, *end, *substr; > > + uint16_t lcore_id; > > + int ret =3D 0; > > + > > + while (*addrs =3D=3D '\0') > > + addrs++; > > + if (*addrs =3D=3D '\0') { > > + fprintf(stderr, "No input DMA addresses\n"); > > + ret =3D -1; > > + goto out; > > + } > > + > > + substr =3D strtok(addrs, ","); > > + if (substr =3D=3D NULL) { > > + fprintf(stderr, "No input DMA address\n"); > > + ret =3D -1; > > + goto out; > > + } > > + > > + memset(&test_case->lcore_dma_map, 0, sizeof(struct > > lcore_dma_map_t)); > > + > > + do { > > + rte_strsplit(substr, strlen(substr), ptrs, 2, '@'); > > + > > + start =3D strstr(ptrs[0], "lcore"); > > + if (start =3D=3D NULL) { > > + fprintf(stderr, "Illegal lcore\n"); > > + ret =3D -1; > > + break; > > + } > > + > > + start +=3D 5; > > + lcore_id =3D strtol(start, &end, 0); > > + if (end =3D=3D start) { > > + fprintf(stderr, "No input lcore ID or ID %d is > > wrong\n", lcore_id); > > + ret =3D -1; > > + break; > > + } > > + > > + lcore_dma_map =3D &test_case->lcore_dma_map; > > + lcore_dma_map->lcores[lcore_dma_map->cnt] =3D lcore_id; > > + strcpy(lcore_dma_map->dma_names[lcore_dma_map- > > >cnt], ptrs[1]); > > + lcore_dma_map->cnt++; > > + substr =3D strtok(NULL, ","); > > + } while (substr !=3D NULL); > > + > > +out: > > + free(input); > > + return ret; > > +} > > + > > +static int > > +parse_entry(const char *value, struct test_configure_entry *entry) { > > + char input[255] =3D {0}; > > + char *args[MAX_PARAMS_PER_ENTRY]; > > + int args_nr =3D -1; > > + > > + if (value =3D=3D NULL || entry =3D=3D NULL) > > + goto out; > > + > > + strncpy(input, value, 254); > > + if (*input =3D=3D '\0') > > + goto out; > > + > > + args_nr =3D rte_strsplit(input, strlen(input), args, > > MAX_PARAMS_PER_ENTRY, ','); > > + if (args_nr !=3D 1 && args_nr !=3D 4) > > + goto out; > > + > > + entry->cur =3D entry->first =3D (uint32_t)atoi(args[0]); > > + > > + if (args_nr =3D=3D 4) { > > + entry->last =3D (uint32_t)atoi(args[1]); > > + entry->incr =3D (uint32_t)atoi(args[2]); > > + if (!strcmp(args[3], "MUL")) > > + entry->op =3D OP_MUL; > > + else if (!strcmp(args[3], "ADD")) > > + entry->op =3D OP_ADD; > > + else { > > + printf("Invalid op %s.\n", args[3]); > > + args_nr =3D -1; > > + } > > + } else { > > + entry->op =3D OP_NONE; > > + entry->last =3D 0; > > + entry->incr =3D 0; > > + } > > +out: > > + return args_nr; > > +} > > + > > +static uint16_t > > +load_configs(const char *path) > > +{ > > + struct rte_cfgfile *cfgfile; > > + int nb_sections, i; > > + struct test_configure *test_case; > > + char section_name[CFG_NAME_LEN]; > > + const char *case_type; > > + const char *lcore_dma; > > + const char *mem_size_str, *buf_size_str, *ring_size_str, > > *kick_batch_str; > > + int args_nr, nb_vp; > > + bool is_dma; > > + > > + printf("config file parsing...\n"); > > + cfgfile =3D rte_cfgfile_load(path, 0); > > + if (!cfgfile) { > > + printf("Open configure file error.\n"); > > + exit(1); > > + } > > + > > + nb_sections =3D rte_cfgfile_num_sections(cfgfile, NULL, 0); > > + if (nb_sections > MAX_TEST_CASES) { > > + printf("Error: The maximum number of cases is %d.\n", > > MAX_TEST_CASES); > > + exit(1); > > + } > > + > > + for (i =3D 0; i < nb_sections; i++) { > > + snprintf(section_name, CFG_NAME_LEN, "case%d", i + 1); > > + test_case =3D &test_cases[i]; > > + case_type =3D rte_cfgfile_get_entry(cfgfile, section_name, > > "type"); > > + if (!case_type) { > > + printf("Error: No case type in case %d, the test will be > > finished here.\n", > > + i + 1); > > + test_case->is_valid =3D false; > > + continue; > > + } > > + > > + if (strcmp(case_type, DMA_MEM_COPY) =3D=3D 0) { > > + test_case->test_type =3D > > TEST_TYPE_DMA_MEM_COPY; > > + test_case->test_type_str =3D DMA_MEM_COPY; > > + is_dma =3D true; > > + } else if (strcmp(case_type, CPU_MEM_COPY) =3D=3D 0) { > > + test_case->test_type =3D > > TEST_TYPE_CPU_MEM_COPY; > > + test_case->test_type_str =3D CPU_MEM_COPY; > > + is_dma =3D false; > > + } else { > > + printf("Error: Cannot find case type %s in case%d.\n", > > case_type, i + 1); > > + test_case->is_valid =3D false; > > + continue; > > + } > > + > > + nb_vp =3D 0; > > + > > + test_case->src_numa_node =3D > > (int)atoi(rte_cfgfile_get_entry(cfgfile, > > + > > section_name, "src_numa_node")); > > + test_case->dst_numa_node =3D > > (int)atoi(rte_cfgfile_get_entry(cfgfile, > > + > > section_name, "dst_numa_node")); > > + > > + mem_size_str =3D rte_cfgfile_get_entry(cfgfile, section_name, > > "mem_size"); > > + args_nr =3D parse_entry(mem_size_str, &test_case- > > >mem_size); > > + if (args_nr < 0) { > > + printf("parse error in case %d.\n", i + 1); > > + test_case->is_valid =3D false; > > + continue; > > + } else if (args_nr > 1) > > + nb_vp++; > > + > > + buf_size_str =3D rte_cfgfile_get_entry(cfgfile, section_name, > > "buf_size"); > > + args_nr =3D parse_entry(buf_size_str, &test_case->buf_size); > > + if (args_nr < 0) { > > + printf("parse error in case %d.\n", i + 1); > > + test_case->is_valid =3D false; > > + continue; > > + } else if (args_nr > 1) > > + nb_vp++; > > + > > + if (is_dma) { > > + ring_size_str =3D rte_cfgfile_get_entry(cfgfile, > > section_name, > > + > > "dma_ring_size"); > > + args_nr =3D parse_entry(ring_size_str, &test_case- > > >ring_size); > > + if (args_nr < 0) { > > + printf("parse error in case %d.\n", i + 1); > > + test_case->is_valid =3D false; > > + continue; > > + } else if (args_nr > 1) > > + nb_vp++; > > + > > + kick_batch_str =3D rte_cfgfile_get_entry(cfgfile, > > section_name, "kick_batch"); > > + args_nr =3D parse_entry(kick_batch_str, &test_case- > > >kick_batch); > > + if (args_nr < 0) { > > + printf("parse error in case %d.\n", i + 1); > > + test_case->is_valid =3D false; > > + continue; > > + } else if (args_nr > 1) > > + nb_vp++; > > + > > + lcore_dma =3D rte_cfgfile_get_entry(cfgfile, > > section_name, "lcore_dma"); > > + int lcore_ret =3D parse_lcore_dma(test_case, > > lcore_dma); > > + if (lcore_ret < 0) { > > + printf("parse lcore dma error in case %d.\n", i > 1); > > + test_case->is_valid =3D false; > > + continue; > > + } > > + } else { > > + lcore_dma =3D rte_cfgfile_get_entry(cfgfile, > > section_name, "lcore"); > > + int lcore_ret =3D parse_lcore(test_case, lcore_dma); > > + if (lcore_ret < 0) { > > + printf("parse lcore error in case %d.\n", i + 1); > > + test_case->is_valid =3D false; > > + continue; > > + } > > + } > > + > > + if (nb_vp > 1) { > > + printf("Error, each section can only have a single > > variable parameter.\n"); > > + test_case->is_valid =3D false; > > + continue; > > + } > > + > > + test_case->cache_flush =3D > > + (int)atoi(rte_cfgfile_get_entry(cfgfile, section_name, > > "cache_flush")); > > + test_case->test_secs =3D > > (uint16_t)atoi(rte_cfgfile_get_entry(cfgfile, > > + section_name, "test_seconds")); > > + > > + test_case->eal_args =3D rte_cfgfile_get_entry(cfgfile, > > section_name, "eal_args"); > > + test_case->is_valid =3D true; > > + } > > + > > + rte_cfgfile_close(cfgfile); > > + printf("config file parsing complete.\n\n"); > > + return i; > > +} > > + > > +/* Parse the argument given in the command line of the application */ > > +static int append_eal_args(int argc, char **argv, const char > > +*eal_args, char **new_argv) { > > + int i; > > + char *tokens[MAX_EAL_PARAM_NB]; > > + char args[MAX_EAL_PARAM_LEN] =3D {0}; > > + int token_nb, new_argc =3D 0; > > + > > + for (i =3D 0; i < argc; i++) { > > + if ((strcmp(argv[i], CMDLINE_CONFIG_ARG) =3D=3D 0) || > > + (strcmp(argv[i], CMDLINE_RESULT_ARG) =3D=3D > > 0)) { > > + i++; > > + continue; > > + } > > + strlcpy(new_argv[new_argc], argv[i], > > sizeof(new_argv[new_argc])); > > + new_argc++; > > + } > > + > > + if (eal_args) { > > + strlcpy(args, eal_args, sizeof(args)); > > + token_nb =3D rte_strsplit(args, strlen(args), > > + tokens, MAX_EAL_PARAM_NB, ' '); > > + for (i =3D 0; i < token_nb; i++) > > + strcpy(new_argv[new_argc++], tokens[i]); > > + } > > + > > + return new_argc; > > +} > > + > > +int > > +main(int argc, char *argv[]) > > +{ > > + int ret; > > + uint16_t case_nb; > > + uint32_t i, nb_lcores; > > + pid_t cpid, wpid; > > + int wstatus; > > + char args[MAX_EAL_PARAM_NB][MAX_EAL_PARAM_LEN]; > > + char *pargs[MAX_EAL_PARAM_NB]; > > + char *cfg_path_ptr =3D NULL; > > + char *rst_path_ptr =3D NULL; > > + char rst_path[PATH_MAX]; > > + int new_argc; > > + bool is_first_case =3D true; > > + > > + memset(args, 0, sizeof(args)); > > + > > + for (i =3D 0; i < RTE_DIM(pargs); i++) > > + pargs[i] =3D args[i]; > > + > > + for (i =3D 0; i < (uint32_t)argc; i++) { > > + if (strncmp(argv[i], CMDLINE_CONFIG_ARG, > > MAX_LONG_OPT_SZ) =3D=3D 0) > > + cfg_path_ptr =3D argv[i + 1]; > > + if (strncmp(argv[i], CMDLINE_RESULT_ARG, > > MAX_LONG_OPT_SZ) =3D=3D 0) > > + rst_path_ptr =3D argv[i + 1]; > > + } > > + if (cfg_path_ptr =3D=3D NULL) { > > + printf("Config file not assigned.\n"); > > + return -1; > > + } > > + if (rst_path_ptr =3D=3D NULL) { > > + strlcpy(rst_path, cfg_path_ptr, PATH_MAX); > > + strcat(strtok(basename(rst_path), "."), "_result.csv"); > > + rst_path_ptr =3D rst_path; > > + } > > + > > + case_nb =3D load_configs(cfg_path_ptr); > > + fd =3D fopen(rst_path_ptr, "w"); > > + if (fd =3D=3D NULL) { > > + printf("Open output CSV file error.\n"); > > + return -1; > > + } > > + fclose(fd); > > + > > + for (i =3D 0; i < case_nb; i++) { > > + if (test_cases[i].test_type =3D=3D TEST_TYPE_NONE) { > > + printf("No test type in test case %d.\n\n", i + 1); > > + continue; > > + } > > + if (!test_cases[i].is_valid) { > > + printf("Invalid test case %d.\n\n", i + 1); > > + continue; > > + } > > + > > + cpid =3D fork(); >=20 > [Anoob] Do we really need fork()? Can't we use code like, >=20 > RTE_LCORE_FOREACH_WORKER(lcore_id) { > ret |=3D rte_eal_wait_lcore(lcore_id); > } >=20 > to wait for all threads to exit? [Cheng] Good question. Fork() is used here to establish a new process for t= he new test case. In order for each test case to have a new EAL environment= (for the flexibility), the EAL must be reinitialized for each case. Howeve= r, the EAL parameters can only be initialized once per process. Therefore, = we use a new process to run each new test case. Moreover, each test case ru= ns sequentially and does not affect the others, ensuring the accuracy of th= e performance data. Your code would wait for all threads to exit in the sam= e process. However, it would not provide a "clean" environment for each tes= t case like fork() does. Fork() allows us to have a fully reinitialized env= ironment, with no impact or side effects from previous test cases. This res= ults in clean, precise performance data for each case. Please let me know your thoughts on this. And please let me know if you hav= e any other questions or require any clarification. Thanks, Cheng >=20 > > + if (cpid < 0) { > > + printf("Fork case %d failed.\n", i + 1); > > + exit(EXIT_FAILURE); > > + } else if (cpid =3D=3D 0) { > > + printf("\nRunning case %u\n\n", i + 1); > > + > > + new_argc =3D append_eal_args(argc, argv, > > test_cases[i].eal_args, pargs); > > + ret =3D rte_eal_init(new_argc, pargs); > > + if (ret < 0) > > + rte_exit(EXIT_FAILURE, "Invalid EAL > > arguments\n"); > > + > > + /* Check lcores. */ > > + nb_lcores =3D rte_lcore_count(); > > + if (nb_lcores < 2) > > + rte_exit(EXIT_FAILURE, > > + "There should be at least 2 worker > > lcores.\n"); > > + > > + fd =3D fopen(rst_path_ptr, "a"); > > + if (!fd) { > > + printf("Open output CSV file error.\n"); > > + return 0; > > + } > > + > > + if (is_first_case) { > > + output_env_info(); > > + is_first_case =3D false; > > + } > > + run_test(i + 1, &test_cases[i]); > > + > > + /* clean up the EAL */ > > + rte_eal_cleanup(); > > + > > + fclose(fd); > > + > > + printf("\nCase %u completed.\n\n", i + 1); > > + > > + exit(EXIT_SUCCESS); > > + } else { > > + wpid =3D waitpid(cpid, &wstatus, 0); > > + if (wpid =3D=3D -1) { > > + printf("waitpid error.\n"); > > + exit(EXIT_FAILURE); > > + } > > + > > + if (WIFEXITED(wstatus)) > > + printf("Case process exited. status %d\n\n", > > + WEXITSTATUS(wstatus)); > > + else if (WIFSIGNALED(wstatus)) > > + printf("Case process killed by signal %d\n\n", > > + WTERMSIG(wstatus)); > > + else if (WIFSTOPPED(wstatus)) > > + printf("Case process stopped by signal > > %d\n\n", > > + WSTOPSIG(wstatus)); > > + else if (WIFCONTINUED(wstatus)) > > + printf("Case process continued.\n\n"); > > + else > > + printf("Case process unknown > > terminated.\n\n"); > > + } > > + } > > + > > + printf("Bye...\n"); > > + return 0; > > +} > > + > > diff --git a/app/test-dma-perf/main.h b/app/test-dma-perf/main.h new > > file mode 100644 index 0000000000..215ac42673 > > --- /dev/null > > +++ b/app/test-dma-perf/main.h > > @@ -0,0 +1,69 @@ > > +/* SPDX-License-Identifier: BSD-3-Clause > > + * Copyright(c) 2023 Intel Corporation */ > > + > > +#ifndef _MAIN_H_ > > +#define _MAIN_H_ > > + > > + > > +#include > > +#include > > +#include > > +#include > > + > > +#ifndef __maybe_unused > > +#define __maybe_unused __rte_unused > > +#endif > > + > > +#define MAX_WORKER_NB 128 > > +#define MAX_OUTPUT_STR_LEN 512 > > + > > +#define MAX_DMA_NB 128 > > +#define MAX_LCORE_NB 256 > > + > > +extern char output_str[MAX_WORKER_NB][MAX_OUTPUT_STR_LEN]; > > + > > +typedef enum { > > + OP_NONE =3D 0, > > + OP_ADD, > > + OP_MUL > > +} alg_op_type; > > + > > +struct test_configure_entry { > > + uint32_t first; > > + uint32_t last; > > + uint32_t incr; > > + alg_op_type op; > > + uint32_t cur; > > +}; > > + > > +struct lcore_dma_map_t { > > + uint32_t lcores[MAX_WORKER_NB]; > > + char dma_names[MAX_WORKER_NB][RTE_DEV_NAME_MAX_LEN]; > > + int16_t dma_ids[MAX_WORKER_NB]; > > + uint16_t cnt; > > +}; > > + > > +struct test_configure { > > + bool is_valid; > > + uint8_t test_type; > > + const char *test_type_str; > > + uint16_t src_numa_node; > > + uint16_t dst_numa_node; > > + uint16_t opcode; > > + bool is_dma; > > + struct lcore_dma_map_t lcore_dma_map; > > + struct test_configure_entry mem_size; > > + struct test_configure_entry buf_size; > > + struct test_configure_entry ring_size; > > + struct test_configure_entry kick_batch; > > + uint32_t cache_flush; > > + uint32_t nr_buf; > > + uint16_t test_secs; > > + const char *eal_args; > > + uint8_t scenario_id; > > +}; > > + > > +void mem_copy_benchmark(struct test_configure *cfg, bool is_dma); > > + > > +#endif /* _MAIN_H_ */ > > diff --git a/app/test-dma-perf/meson.build b/app/test-dma- > > perf/meson.build new file mode 100644 index 0000000000..bd6c264002 > > --- /dev/null > > +++ b/app/test-dma-perf/meson.build > > @@ -0,0 +1,17 @@ > > +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2019-2023 > > +Intel Corporation > > + > > +# meson file, for building this app as part of a main DPDK build. > > + > > +if is_windows > > + build =3D false > > + reason =3D 'not supported on Windows' > > + subdir_done() > > +endif > > + > > +deps +=3D ['dmadev', 'mbuf', 'cfgfile'] > > + > > +sources =3D files( > > + 'main.c', > > + 'benchmark.c', > > +) > > -- > > 2.40.1