From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 46358A0A02; Tue, 18 May 2021 16:30:42 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B60044068E; Tue, 18 May 2021 16:30:41 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id AABBC40041 for ; Tue, 18 May 2021 16:30:38 +0200 (CEST) IronPort-SDR: SCCNTqKvOxJ1dSR6vprDkTOg4mDn9MPZxqAFjrS3qWlxuWxRPwCUUZOg1Y2egmEG6cyzMLCT4b BpRb7A2St/7w== X-IronPort-AV: E=McAfee;i="6200,9189,9988"; a="188135018" X-IronPort-AV: E=Sophos;i="5.82,310,1613462400"; d="scan'208";a="188135018" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 May 2021 07:30:37 -0700 IronPort-SDR: sCPS7lJSFKDd6wuVaDCXMbLqT0lNmhoRCnqVnIZk1a7plw9rMtGA8c137kiyZGZEkyqNFoa/J9 e+6hGozbO4RQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,310,1613462400"; d="scan'208";a="439457076" Received: from orsmsx606.amr.corp.intel.com ([10.22.229.19]) by orsmga008.jf.intel.com with ESMTP; 18 May 2021 07:30:37 -0700 Received: from orsmsx607.amr.corp.intel.com (10.22.229.20) by ORSMSX606.amr.corp.intel.com (10.22.229.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.4; Tue, 18 May 2021 07:30:36 -0700 Received: from orsmsx607.amr.corp.intel.com (10.22.229.20) by ORSMSX607.amr.corp.intel.com (10.22.229.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.4; Tue, 18 May 2021 07:30:36 -0700 Received: from ORSEDG601.ED.cps.intel.com (10.7.248.6) by orsmsx607.amr.corp.intel.com (10.22.229.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.4 via Frontend Transport; Tue, 18 May 2021 07:30:36 -0700 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (104.47.55.103) by edgegateway.intel.com (134.134.137.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2106.2; Tue, 18 May 2021 07:30:36 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=c74wG+pLJgN0MkknijbmqIPHcY78YEEjPUDVaF4xgqPTEOoazjL3D6/vpNldiLPMe/Cazh0yoPLcIB3Avi6VBpqn/Rjy9ORMHjaaM3DfnHIPK5ycPaKpv0I6FYo5TI98E45OqMJlYhwjwa/d2LUCIxIe4FsarC4kzhDlU1sbutSOrYeIxZT5ypwT+I+kPdsTy4oAEOlotms6fRlD4pO5BKs5i1P75gCgEKxEOtiFu9z/z67WXRB1d6D+BqwcuO5giDPfRYHtCSAbolainHcVHhUS5ffB7NL1DUu+LVfN+d/tkIYoYcxZDIkw4Fa9+spXM5ByxZ45VchJquPibFEdQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DsW7Yh3JJUkRiGWIBAKMO3dSY8MGZoK4St0DUlx4ts8=; b=gOVyW9OXhZFEPCwxKsI6GI7dJC6xNApbTc6ul6MZy6xLRvCozYu7p4bdOb5UlCq6q2g4N8ry/xrK9MsqRB52J6vi+xE0k4XqW6uOmIiH5LQh/aXZhkETv+eqCkr42DVWil3u1WrhEf1JZqas5rEgf2Lxuh6PYQAKXLGVBKJHeeSf9YED11H0ov5fSHa8gU2CMA+yIHD6sMT3IpBo+fb3r1V93NqG16pdcrp3udGukieJrWJ+gnPtRHuNEtCXX3UdsdwO91lxFFNvypbZY3GNCrovw/T5FB6uN9dv8dN40rza5WnK+OgmP8ES5u9e5TiCXgkVG5v0jpcWjNxgv3uluA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DsW7Yh3JJUkRiGWIBAKMO3dSY8MGZoK4St0DUlx4ts8=; b=llgxc+mj0HUMVBnswgR1OIzhnmIKfNkXqF59EKbzbGSxkG0tUneoRKPqLxS+DWNj87fcwvhTBKaJslTtnVHMZFs+vdToETa+GWEThOuAamjPJy+S/VGfQWVe6246SE0isaorYiGd2qtqFxyFaAHHSdMcXU4Zku8jK32wcbaoMEQ= Received: from MN2PR11MB3821.namprd11.prod.outlook.com (2603:10b6:208:f7::24) by MN2PR11MB4517.namprd11.prod.outlook.com (2603:10b6:208:24e::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4129.26; Tue, 18 May 2021 14:30:35 +0000 Received: from MN2PR11MB3821.namprd11.prod.outlook.com ([fe80::9900:c6fc:a30f:7933]) by MN2PR11MB3821.namprd11.prod.outlook.com ([fe80::9900:c6fc:a30f:7933%7]) with mapi id 15.20.4129.031; Tue, 18 May 2021 14:30:35 +0000 From: "Power, Ciara" To: Aaron Conole CC: "dev@dpdk.org" , "Zhang, Roy Fan" , "Doherty, Declan" Thread-Topic: [PATCH] doc/guides: add details for new test structure Thread-Index: AQHXSzUexXptfhMCM0S+5dKyK6QDNqrn9xHmgAFUZtA= Date: Tue, 18 May 2021 14:30:35 +0000 Message-ID: References: <20210517155517.806841-1-ciara.power@intel.com> In-Reply-To: Accept-Language: en-IE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-version: 11.5.1.3 dlp-product: dlpe-windows dlp-reaction: no-action authentication-results: redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [37.228.239.233] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 269e5e30-0e65-48af-af38-08d91a097ffa x-ms-traffictypediagnostic: MN2PR11MB4517: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: /1wy8IA28ax7k5PHgsDtXVyzN5MrU4ywwLF9Fjtegd7InRedbqflZWOFrX2i8mBC2DaRdDN4eYl80OUzhePKDh9x2ynwi7S1NTqJtXKaUm+N8K6vdziBIUBehYyYfdLKqXnyDwJd0sU5f+hg4CfI3ycyPg/m13QpWzVaD9iIp0/jtsEopDifnFBHwbTxNxJYSy7solJwDvOJejN2a92aaHNLECBppSVqaMbuvyxzdc8cqaF+EO7G24ccY9EGllA8WnUGOGlvin6SOu3YvUOtqISlRz9XtEglZZsaj/ceJWgpv1z+B2oU6aTjHdYY6UcWvk8JdiCNlGJa7TU2o2AmqOEEQBecDrjcv1yjrCt6Ehqh6dFerdT1gtt15O2jI6H/nKolXXR2zTbDM3w/HxcFcZjQ4MLWMwUtsKpuPY1vRYn17EBoPncIr873Jq010Dn9J3izBtm/rsGCwS0/0Mopx3AdRSS6ZSqH4KDyCVPA1MBxSweobfiA0jzymsf63xZzIKD4ZyPO8ZWrbG293JFF6AAFtAhLVDLyuhzYO3Om9ZftBGDP2dKMkqFHGxOoJZBr89eiNzdvoCFAY+qRbzFbdZ2f/U9IxLdV3i6njK12p+s= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MN2PR11MB3821.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(39860400002)(346002)(366004)(136003)(376002)(9686003)(52536014)(66946007)(66556008)(83380400001)(55016002)(8936002)(76116006)(66446008)(107886003)(8676002)(6916009)(38100700002)(186003)(54906003)(33656002)(478600001)(7696005)(122000001)(6506007)(26005)(316002)(64756008)(2906002)(66476007)(4326008)(71200400001)(5660300002)(86362001); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?onD8qQpvl1t7ok6dItgqScI/IzBxekFGNAjshZGE13hIocZ8DEGSt5yXX9eR?= =?us-ascii?Q?nbpCcEF1fU4pzQjK6L/KOtOpYbjBhsjfX2wSD+BNTutKmCrcLH6EUau2NycK?= =?us-ascii?Q?FvIg0LdR4BQWVtUhwyhuAJj+l4vHji6Jk6dubhzPavKceuy2Q6AfkOdDOM+s?= =?us-ascii?Q?vzqf6YBMWT8HMRzBnHVwKb9ZNrYtqDj5Ha2/COp/u45c8AZR3w1wa1DVOQEN?= =?us-ascii?Q?JKXNs4KkG4juXdMmCiINEkVG5f532IOc+gnnUIq69foIHxmzhvEcpIFbd7cx?= =?us-ascii?Q?28ZBzcKFfgR55QhjhZIMUrYAHXlCZxprBWJQGHlwsZNJUvuK4nQsZbbm6g0g?= =?us-ascii?Q?1U244ap7pZkQtXVZ5IuZz3D2taq6NKfzLg+9QXg9yDu4kIr6ybIwlvKS45VX?= =?us-ascii?Q?W8Pf6f3/mk0wHctiUwOi1PXrEfwDWLG7qDQYZB3FAe3osdTWWCPAeTF6ntMG?= =?us-ascii?Q?fub+sd1g3Z3GCWbbwltzLs6V7WWZdKdl5qwChKgsCzUYl5xfhVzxveY/1gAb?= =?us-ascii?Q?Alf7ha0MLlljw5Xzb+up2PusAy/frasYoaxGoOjhDA5M/8JxtjlcKC3gulhE?= =?us-ascii?Q?lLENylYkx3i2VcHeF4MJiY6QjbxNYvDnGAXAmoHMlG74NIuFS6lRGVXPkJP8?= =?us-ascii?Q?FzWN4lubnBm6340B8nUhhKz8ncdBVH8f5yB/Ja9VPhMCFC03YQ9LsUg/mBoj?= =?us-ascii?Q?MV63SzxuTW11j594hqLbRf1tEtc+yRa4l4Yc3DjgsYIPsU92MttX7Gcoxfn+?= =?us-ascii?Q?uwlSQDwuryyU+KgfFlu6E31SYN98i8yT4rtNNmLXoRzaPY3dhyelCX6w+FXc?= =?us-ascii?Q?8AJylPstYzJMJdtjab+1BgttsNChIQgB+5FEBJBN0IR3bLAL8J9sRQ1slzLh?= =?us-ascii?Q?8G3GyNVViNlvSmz5dgLApf6Qz+DjfghzYJfCAIvqr5kwLzZuzd9dhiz3dAp4?= =?us-ascii?Q?q3Jf/lLbr+AywpTdw62l6O8Z1IYI4UI2QyhQAB1OPNoVgO6oOw1PC8cTl4ob?= =?us-ascii?Q?At3xRulgXF2KzquAOyn/JYc6G5occkB83yi2FCVEPGgmpFOEdJCPxiT03Mo0?= =?us-ascii?Q?KktMEDEFJTyWqFHnHgHG7HwmFmAHAAA66QlYHVLG24/CLt6J9aw+yLja29L5?= =?us-ascii?Q?sNd1YZ0eaBFBNLBLNcjMUxS8io3Y8d2e7XrjtBPXTVTtnofCU3Y+7GHE5bbO?= =?us-ascii?Q?XxA2al+/YLUvzz1a82UjYXQHrXBRQK+OJ1qlIvGA8WM4CuN1ahnMEFAkX7vq?= =?us-ascii?Q?DLSa10tenNYuoLzV7vfxajyf/C/iwWE+aAerI0D4fS/UJDOkKiAJlNlIJT8y?= =?us-ascii?Q?6kZEElgmrk4Sxetk+Vl0BDZx?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR11MB3821.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 269e5e30-0e65-48af-af38-08d91a097ffa X-MS-Exchange-CrossTenant-originalarrivaltime: 18 May 2021 14:30:35.2979 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 1/SHUQQIF+AXuGUejISo+cdfTY0JvMkK+OSkCBOMp0qEQO40Z4zaOGSAnKBgYTueIfJb+NWotaOBC27mR0Ui3A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR11MB4517 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH] doc/guides: add details for new test structure X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Aaron, >-----Original Message----- >From: Aaron Conole >Sent: Monday 17 May 2021 19:02 >To: Power, Ciara >Cc: dev@dpdk.org; Zhang, Roy Fan ; Doherty, >Declan >Subject: Re: [PATCH] doc/guides: add details for new test structure > >Ciara Power writes: > >> The testing guide is now updated to include details about using >> sub-testsuites. Some example code is given to demonstrate how they can >> be used. >> >> A note is also added to highlight the need for using vdev EAL args >> when running cryptodev tests. >> >> Depends-on: patch-88751 ("guides: add a guide for developing unit >> tests") >> >> Signed-off-by: Ciara Power >> --- >> doc/guides/contributing/testing.rst | 89 >> ++++++++++++++++++++++++++++- >> 1 file changed, 87 insertions(+), 2 deletions(-) >> >> diff --git a/doc/guides/contributing/testing.rst >> b/doc/guides/contributing/testing.rst >> index 0757d71ad0..1188d63a97 100644 >> --- a/doc/guides/contributing/testing.rst >> +++ b/doc/guides/contributing/testing.rst >> @@ -114,13 +114,21 @@ for interacting with the test harness: >> >> 2. unit_test_suite_runner(struct unit_test_suite \*) >> Returns a runner for a full test suite object, which contains >> - a test suite name, setup, tear down, and vector of unit test >> - cases. >> + a test suite name, setup, tear down, a pointer to a list of >> + sub-testsuites, and vector of unit test cases. >> >> Each test suite has a setup and tear down function that runs at the >> beginning and end of the test suite execution. Each unit test has a >> similar function for test case setup and tear down. >> >> +Each testsuite may use a nested list of sub-testsuites, which are >> +iterated by the unit_test_suite_runner. >> +This support allows for better granularity when designing test suites. >> +The sub-testsuites list can also be used in parallel with the vector >> +of testcases, in this case the testcases will be run, and then each >> +sub-testsuite is executed. To see an example of a testsuite using >> +sub-testsuites, see *app/test/test_cryptodev.c*. >> + >> Test cases are added to the `.unit_test_cases` element of the unit >> test suite structure. Ex: >> >> @@ -165,6 +173,70 @@ test suite structure. Ex: >> The above code block is a small example that can be used to create a >> complete test suite with test case. >> >> +Sub-testsuites can be added to the `.unit_test_suites` element of the >> +unit test suite structure. Ex: >> + >> +.. code-block:: c >> + :linenos: >> + >> + static int testsuite_setup(void) { return TEST_SUCCESS; } >> + static void testsuite_teardown(void) { } >> + >> + static int ut_setup(void) { return TEST_SUCCESS; } >> + static void ut_teardown(void) { } >> + >> + static int test_case_first(void) { return TEST_SUCCESS; } >> + >> + static struct unit_test_suite example_parent_testsuite =3D { >> + .suite_name =3D "EXAMPLE PARENT TEST SUITE", >> + .setup =3D testsuite_setup, >> + .teardown =3D testsuite_teardown, >> + .unit_test_cases =3D {TEST_CASES_END()} >> + }; >> + >> + static int sub_testsuite_setup(void) { return TEST_SUCCESS; } >> + static void sub_testsuite_teardown(void) { } >> + >> + static struct unit_test_suite example_sub_testsuite =3D { >> + .suite_name =3D "EXAMPLE SUB TEST SUITE", >> + .setup =3D sub_testsuite_setup, >> + .teardown =3D sub_testsuite_teardown, >> + .unit_test_cases =3D { >> + TEST_CASE_ST(ut_setup, ut_teardown, test_case_first), >> + >> + TEST_CASES_END(), /**< NULL terminate unit test array */ >> + }, >> + }; >> + >> + static struct unit_test_suite end_testsuite =3D { >> + .suite_name =3D NULL, >> + .setup =3D NULL, >> + .teardown =3D NULL, >> + .unit_test_suites =3D NULL >> + }; >> + >> + static int example_tests() >> + { >> + uint8_t ret, i =3D 0; >> + struct unit_test_suite *sub_suites[] =3D { >> + &example_sub_testsuite, >> + &end_testsuite /**< NULL test suite to indicate end of li= st */ >> + }; >> + >> + example_parent_testsuite.unit_test_suites =3D >> + malloc(sizeof(struct unit_test_suite *) * >> + RTE_DIM(sub_suites)); >> + >> + for (i =3D 0; i < RTE_DIM(sub_suites); i++) >> + example_parent_testsuite.unit_test_suites[i] =3D >> + sub_suites[i]; >> + >> + ret =3D unit_test_suite_runner(&example_parent_testsuite); >> + free(example_parent_testsuite.unit_test_suites); >> + >> + return ret; >> + } >> + >> + REGISTER_TEST_COMMAND(example_autotest, example_tests); >> + >> >> Designing a test >> ---------------- >> @@ -241,3 +313,16 @@ be run as part of the appropriate class (fast, perf= , >driver, etc.). >> Some of these default test suites are run during continuous >> integration tests, making regression checking automatic for new >> patches submitted to the project. >> + >> + >> +Running Cryptodev Tests >> +----------------------- >> + >> +When running cryptodev tests, the user must create any required >> +virtual device via EAL args, as this is not automatically done by the t= est. > >How does this differ from just running the tests? Will I get 'SKIP' or 'F= AIL' if I >don't do this? Should the cryptodev tests stay as part of the driver test= suite >or be moved out? > Before the refactor, the vdev creation was done by the test app when needed= . For example, with DPDK_TEST set to "cryptodev_aesni_mb_autotest", which nee= ds a crypto_aesni_mb vdev, the user would just need to run: ./dpdk-test But now with this refactor, this vdev isn't created by the test app any lon= ger. For the same autotest, the user must now run: ./dpdk-test --vdev crypto_aesni_mb Without passing the vdev argument, the test is skipped with log: "USER1: No= crypto_aesni_mb devices found?" With regards the driver test suite, if the suite is run without any extra a= rgs, for example: meson test --suite=3DDPDK:driver-tests -Cbuild The vdevs will not be created, and the autotests that don't have the vdev r= equirements met will be skipped. But we can pass extra test args to create the vdevs we want to test: meson test --suite=3DDPDK:driver-tests -Cbuild --test-args=3D"--vdev crypto= _aesni_mb --vdev crypto_aesni_gcm" I think it is still ok to include these in the driver suite, and allow the = user to choose the vdevs they need and can support on their system. Thanks, Ciara >> + >> +.. note:: >> + >> + The cryptodev_scheduler_autotest is the only exception to this. >> + This vdev will be created automatically by the test app, >> + as it requires a more complex setup than other vdevs.