From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 938F542651; Wed, 27 Sep 2023 11:56:16 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 624B5402DC; Wed, 27 Sep 2023 11:56:16 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id 3900E402C8 for ; Wed, 27 Sep 2023 11:56:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1695808574; x=1727344574; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=B+eCQ992ITYVnB+Gc/FA7gk/LLgRXy9tEK7lojop5MI=; b=hnxQTKbhjJU0CfSwj7HEKfTvMOR0/8n0uXICURfYljzmTa61elLIrB64 YFK0Y2sIbEXQchecyT6S30/k6HwEsXQ5Mj3MzDELHW/8cMGBXudPsFS3q VU9EVAsv/dboTek9Zs40Ovwu/szaFFUCfI9DSLOFSpogEYCSaYs37IEvb T0YMR1uzEIfKee+fRYluCkcIz+UgZI3cdYz3yfX7i0rC1DaeKmIaSGHsP fLOE3vDemVG/OnSZXeqybqrPhuV91SGrL7MPZZFlN6Et4BMzSq6su1eM3 bLan8QsBMeCoo6BCYlqj1CNyuZpIcSYT5NPNHf4b9DhTPQzB8ITIDyWoZ w==; X-IronPort-AV: E=McAfee;i="6600,9927,10845"; a="380655966" X-IronPort-AV: E=Sophos;i="6.03,179,1694761200"; d="scan'208";a="380655966" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Sep 2023 02:56:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10845"; a="892543852" X-IronPort-AV: E=Sophos;i="6.03,179,1694761200"; d="scan'208";a="892543852" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by fmsmga001.fm.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 27 Sep 2023 02:55:05 -0700 Received: from fmsmsx611.amr.corp.intel.com (10.18.126.91) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.32; Wed, 27 Sep 2023 02:56:12 -0700 Received: from fmsmsx603.amr.corp.intel.com (10.18.126.83) by fmsmsx611.amr.corp.intel.com (10.18.126.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.32; Wed, 27 Sep 2023 02:56:11 -0700 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.32 via Frontend Transport; Wed, 27 Sep 2023 02:56:11 -0700 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (104.47.55.101) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.32; Wed, 27 Sep 2023 02:56:11 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gaxThhKNP8WqHBST35lLPEtKycZ0X43ZRpk8RE6AQ/qznBoPBtiFI4W0gjgY4ZxEoVS50FfndWGTXWmiB0i1fKt1JimiHLn4g/maQR2fKdWffhqqnAbZvkSA2AO+QuRIE99JqAfeHJb+qjCmPknsvbFKD3u4pGKFGJmnXlTLVO43gCU5UlPaLU8V0gyC3P0RdD+Si6Uf2ZIL60GUViN6FkVzU8K/JR3Ns75PZ7BvRByrwe7MLGmyEPqBgiUdhoBKUMacRZ0EE7NNymgxYkO4QUwOyE+vEUc6VP8l3RzAB083dMNF3U3kVQGDZlaOVU/mUaSfFAOHsllpmZBnnwf2QA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fkxdXAoqSCWJQkFfb5vbmoYjQDmVpues8Vm/FUBT6ts=; b=aMc4cOw7f1TdZ2SKzTohOzmITms/P+yxYZD2O4AkkWWMtbOgphm16LZrUsA4WI3QXkgRB7ic5HljFgd4BwZ01zyVW0cNkj6ZpfT8bB6fntRPjTd0MjlOH+AxRJ3ROi3kBe+U9jr2zLSdgJLUDKV5mlKfWPKsp7tGOQKFCmcBNRFuty4cgXZqLU4r/y+NhpqUpDEDk0P6TyKf7XPvjibWsG45bndyYEYZ3MlTYEY5gMvUfvY013Dl+yisdIbAaUSZ7fsjQ4lLXATrM6yEiCsXkrwXwhjMcBJoARapsih7eA3RShDZf1v36ugOfAFr9XImYjmHqjT+3H2iJ02GYO7d1Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by DS0PR11MB8761.namprd11.prod.outlook.com (2603:10b6:8:1a1::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6813.25; Wed, 27 Sep 2023 09:56:10 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::43d1:af60:464:347]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::43d1:af60:464:347%5]) with mapi id 15.20.6768.029; Wed, 27 Sep 2023 09:56:10 +0000 Date: Wed, 27 Sep 2023 10:56:03 +0100 From: Bruce Richardson To: David Marchand CC: Aaron Conole , , Thomas Monjalon , Ferruh Yigit , "Jerin Jacob Kollanukkaran" , Akhil Goyal , Maxime Coquelin Subject: Re: [PATCH 0/2] add checks for tests not in a suite Message-ID: References: <20230915115206.132198-1-bruce.richardson@intel.com> Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-ClientProxiedBy: DB8P191CA0012.EURP191.PROD.OUTLOOK.COM (2603:10a6:10:130::22) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|DS0PR11MB8761:EE_ X-MS-Office365-Filtering-Correlation-Id: 2cc0abdc-3c0b-43f6-eb3d-08dbbf3ff9f2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: N5oS5nbtWJvR5hCas/xlFWi2ly+ktPspVyRF8KbSuF+FfVtUbEeM57LCMVlCSzjFfwnV5FOus9jVPP8M0imSyKfdUVUh84ca0PgrQW6+W7XuPZdqjUtakKTF4E1ZbXEAX2KHKMIKagnhPTHmrOa4ygccSTCJ7vduReBthQhJNgiMr8EPgYfamhFobSO9MNFz5QDqPtUgtUS06LxZBZGSvwskYefXbzz+eO8CbpoFUZBBLSJiw5rT82RjMCfhc6G85G7wimWqMPJ/VCTrgMjcadQN9AUNWSqMHkNuQX1ZK7ulQ7DSKAyXNL+rWFGZ4SAoKZMxhVGXqMM/2CBEflWrkpSr42oCfo7kKGGR39cGguZeqxLG8NvKUF0dZ+OWFZm0KTHdbYJSfFAppYU32pfH5UqZ4U+65guHkw26BSeX9B7yh5khZwDzYJiy4zfI8iqsFx1SXqDUnPXiWjZGw85o3vobHwM+grtf84DQbSA9KTs/N3R9m7d3eptb0KYqteg4njSO3YnzAm78334j8K5WqEqseMUo19aNJZVn91i2AJA= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(366004)(136003)(39860400002)(376002)(396003)(346002)(230922051799003)(451199024)(186009)(1800799009)(6486002)(53546011)(6506007)(6512007)(38100700002)(66946007)(26005)(66556008)(966005)(6666004)(66476007)(86362001)(5660300002)(54906003)(2906002)(82960400001)(41300700001)(6916009)(8676002)(8936002)(478600001)(316002)(44832011)(4326008)(83380400001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?VTRNSDhMRUJwbzhUaXpGK2FoVVRpM3dPNXllSC9jelVySHVrOTBjbU1WT0JW?= =?utf-8?B?dzZDU0Uwb2pKWmx4dHFleXlDVW5ncVBGVVYyay9YMkYxejd0WUNOWHF0dEgv?= =?utf-8?B?RHFwanlVZC9aaGhpb3ptMzRGTkdSekd2d0lYYm9NM1REdFA1Z3VWc0xLSEw5?= =?utf-8?B?QWgvU0trRDJ5c1Y0QmNnWVNHb1pDYmJ4MVhlSnlSOGtmUEJoeEFMc1F0bmR1?= =?utf-8?B?TnJKRUFxd1lSYUViWEZmZkRmWUcraFl5UTIza01uZmh4b2FPRmxJcUR6RDdj?= =?utf-8?B?TW9QUEQ0VkFiMlZDSklYK0FkOU1CbER5MDNPOHJtSXlNd29FMlpJeThERmxZ?= =?utf-8?B?dDVRZEpINFBaMXNEb3FPd2FqeUNqNmQvZkM0QzRHZFN2ZUpOaEdKTXNTWEw1?= =?utf-8?B?bnRrN3hYK0llRWlYZ3Y4YXNSOHFxZU5kRUdJdml5WVUvbk9RY1FDV2FYaE51?= =?utf-8?B?eitEQ3hzSmdjQ0J0STd3QURobXUyVmpGNCt3dml6MExkd05xTlp0RWxUdU5y?= =?utf-8?B?WGlHWHpOSzVGQkxOL1VIa3FtY1I5cUlleGs4Z0tIMEF6OTN5ZFdML2ZHc0xk?= =?utf-8?B?ajN4WjA3dWkwOWtIRWp1STlGc0hjeGFwUEw4MGhKb25nL041dW9kaE1vcklZ?= =?utf-8?B?NHcvcGJ3YUVLMzBlTFBKeS8rM2MyUnFwM3JqRlZPRGt0MCszNGtVSGpsSUJ0?= =?utf-8?B?NkpmZWNLUjNnVlNCYWNRWmg0MXljZ2phRkdycWwyNXJDY1hSOEFLK0VSVnhi?= =?utf-8?B?WWNCVmpwNkN0b0x2ZzFUWnFDbXR4SDZ6bm51dDEweGZSZWk5VFlPU0NCeWg5?= =?utf-8?B?aThqRVVPTndRYys0dnhFVU1EeEM0TGN2T05EZ2VQZCtTdXczS0l5TlFwU1Jk?= =?utf-8?B?V0xYV2UvKzd6anJtT0xCM3hHM2UwN0dLMUtUTjVETFI0UUMxN2xNWjhxRnFD?= =?utf-8?B?QVVUdWc4bW9NYm9EZ2YrRExucmJHVWRYM0VZYldQMzY1bzFsbGNMdVRXT25C?= =?utf-8?B?L3dDU3lGUFdVRHBiRzRET0g2TWk0RmVWY1IzUUlKRW5NcTNwclFoQU9rdlJr?= =?utf-8?B?bXVVY204RGNVVWVYQmRpbzlTKzdUZFNqa09QVHB1ZFQ3QU04NDFTM0t1aWVU?= =?utf-8?B?VWJaL0hndmp3dXRmSU9wVGNTaEJSb3ZKMVJPNkdpWDFnMjhZZFdzR015akpV?= =?utf-8?B?U0hsWmJzVUh6VlZickI4Y1hQWWtQWmJFeXBkOHB5YnVMNU03R2NuMFB5eTdo?= =?utf-8?B?ZnNtWGNiVmQ1TEVXSE1PM1ZtS0NwMnFzZHh3Rlp5UE5VSFVyeEFjS1E5c3JJ?= =?utf-8?B?UzluVkRPQmx0R2svd3JxOVFyTjZOMHhTOU04V2d2SzdBNlMrSVlvQWFjSjZX?= =?utf-8?B?dG55eTZpY0YvczA0TFF0bUo4TjhZRUxTdDRlbm4vSklNOXRiN0gxdHoxK2ox?= =?utf-8?B?ZUlRQjB2bkFnNGVQTi9wUVpKU0d3R1ZlS1lyUXJmU2pMbW5oVnBFeFlFU2JK?= =?utf-8?B?TjROb2Y4NUl5MVA1SGY4eXJPbVZzOWlxMDhrWG5KdXpEbFZCNS9yOHVMbWlS?= =?utf-8?B?KzEwOU5TWktBam5wKzdEQW82WU5mRjNFWkpzQnVEdnNXUURIQlJnNWVOWXFO?= =?utf-8?B?VVdIV0lzbm5MN1UwQ1ZnaFk3aFFuakI1M0lubWhOcFBUMmVzN1cwVUVBRzMr?= =?utf-8?B?VnRDYy9MdTk1MkNBMmVYN2cxWnREdzQvdWZ2bkFCV0k2OVR1RHBUWlhWTXlT?= =?utf-8?B?UmRiVi9VcFN0U1I2ZUNTL3ZwdEVpNFVnck1QUlFMck9MTUJGdWpUc05oUXIy?= =?utf-8?B?bnJkR1ZWek9MZUVCT2swTEdsVk4yU09nS2NwcFdaR1Y1NG5XUWtpcDB5RGkx?= =?utf-8?B?VzdDdUY3SThuTVJGOGo5NnJ2SzlUVEh5UHRTSGZHNFp1WHJBSkZFaWNLMHlx?= =?utf-8?B?Q2pQbGtxcjVkMjVvRjBWUndKNjlhcmE4cHBSRmplbTcyeHFzdDlqQmRxM2c1?= =?utf-8?B?djBWcHIwSVRIcHFOK0k3ckw4NlgwRjVPcE9FS2FFc0lZMjFGVS90Q2hmVmZQ?= =?utf-8?B?eERFdzlSWElhVjNsVkpKUzhETnZQVEZBMjV4WHVDdVhpSWw2UjErSTVoY0lW?= =?utf-8?B?eVAyWUsxTEltWU5lWlhDM3p2QXJJbzBUZE5ZUjI1Mi9yaExXalE0cTZ5NWtZ?= =?utf-8?B?ckE9PQ==?= X-MS-Exchange-CrossTenant-Network-Message-Id: 2cc0abdc-3c0b-43f6-eb3d-08dbbf3ff9f2 X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Sep 2023 09:56:10.2124 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: f5O4Xa4VFZNis+mt5/ykFHtFhZFY1mIcAsdUnW/F4ArIpa3VoivE3pvwWU7vjFhuMrqDaQsOLQqeYVDSiP9Tk0NNXMJ7qxC4ljoEz4LsTDo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB8761 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Wed, Sep 27, 2023 at 11:31:07AM +0200, David Marchand wrote: > On Wed, Sep 27, 2023 at 10:27 AM Bruce Richardson > wrote: > > On Wed, Sep 27, 2023 at 08:30:05AM +0200, David Marchand wrote: > > > On Tue, Sep 26, 2023 at 5:01 PM Aaron Conole wrote: > > > > David Marchand writes: > > > > > On Tue, Sep 19, 2023 at 10:36 AM Bruce Richardson > > > > > wrote: > > > > >> > > To help ensure that we don't have "orphaned" tests not in any test > > > > >> > > suites we can add the following checks: > > > > >> > > > > > > >> > > * In developer-mode builds, emit a warning for each test defined using > > > > >> > > REGISTER_TEST_COMMAND > > > > >> > > * In checkpatches, add a check to prevent the addition of new tests > > > > >> > > using the REGISTER_TEST_COMMAND macro > > > > >> > > > > > > >> > > Bruce Richardson (2): > > > > >> > > app/test: emit warning for tests not in a test suite > > > > >> > > devtools: check for tests added without a test suite > > > > >> > > > > > > >> > > app/test/suites/meson.build | 13 ++++++++++++- > > > > >> > > buildtools/get-test-suites.py | 12 +++++++++--- > > > > >> > > devtools/checkpatches.sh | 8 ++++++++ > > > > >> > > 3 files changed, 29 insertions(+), 4 deletions(-) > > > > >> > > > > > >> > The "non_suite_tests" testsuite returned by > > > > >> > buildtools/get-test-suites.py is a bit strange, as it is not a > > > > >> > testsuite from meson pov. > > > > >> > > > > >> Yeah, it is a bit strange, and I'm open to new ideas on other solutions. I > > > > >> did it that way to avoid having yet another script to scan the files - I > > > > >> figured it was faster (in terms of runtime, not dev time) to do the > > > > > > > > > > I had figured it was "faster dev time" that won :-). > > > > > I am fine with it, I don't expect more complications in this area in the future. > > > > > > > > > > > > > > >> scanning when the files are already being opened and processed by this one. > > > > >> > > > > >> Of course, if we can get the un-suitened [:-)] test cases down to zero, we > > > > >> can theoretically drop this check in future, and just use the checkpatch > > > > >> one. > > > > > > > > > > Well, that's still a question that nobody seems to comment on. > > > > > > > > > > What should we do with tests that don't enter one of those testsuites, > > > > > and are not invoked by the CI? > > > > > > > > > > Though we may be removing some level of coverage, I am for > > > > > cleaning/unused dead code. > > > > > > > > I guess it does require actually looking at these tests and classifying > > > > them to either put them into the proper suites. As of now, we aren't > > > > really removing coverage if they aren't being run - but are any > > > > maintainers or developers actually running them? > > > > > > Could we go a step further than Bruce runtime warning (which is at the > > > meson level and does not impact running the test)? > > > Perhaps have those orphaned tests fail unless their test names are > > > provided in a env variable like > > > DPDK_TRUST_ME_I_WILL_SUBMIT_A_PATCH_FOR_THIS_TEST (naming is hard > > > ;-))? > > > > > > With a systematic failure, there is less chance that > > > developers/maintainers miss the situation. > > > If those developers/maintainers simply waive the warning with the env > > > variable and don't send a patch, well.. too bad. > > > > > > After a release or two, if we don't hear from anyone, we can start > > > removing the unused one. > > > > > I think that seems a littel severe at this point. > > > > The one gap we have right now, as far as I can see, is actually explaining > > what the various suite types are, so that developers can choose the right > > one for their test(s). We may even need a couple more if some tests do not > > fit into the existing categories. Once that is done, we should then start > > looking for tests that are obsolete and can be removed. > > > > If we do already have documentation on the various suites and how to use > > them, apologies for my ignorance, and perhaps someone could post a link > > here. > > We have some guidelines but I agree the testsuites are not described. > https://doc.dpdk.org/guides/contributing/unit_test.html > I also see I missed updating the reference to REGISTER_TEST_COMMAND in the docs. I'll do a patch for a doc update for that. /Bruce