From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 209D742651; Wed, 27 Sep 2023 10:27:17 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BE778402E7; Wed, 27 Sep 2023 10:27:16 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id 3B851402CA for ; Wed, 27 Sep 2023 10:27:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1695803235; x=1727339235; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=g7qKbbL64ERo72t/tfIr9auyuOCi1LBUpqSADf+uSJ0=; b=JgKdeTgLpcEJl4kKaqPy/FXl9isRbkPaBqcDfzpXHWpdF9nPobPjHjEn qjj3GEXLbLoPqpNvcBNyKO1pFGDrCm7ttCc5xOWao3piZP0SrLRQjp0dt Url/W0ZjJHwrvtXFueVDhLfVXpE5gE1lxxxiKPHF6OVXlLsm6m1+vcRON vnHJMsb+TGTOR3e6ClIUdHAciWbHbXUQxel7sjRs7LfJRNyc+hhJu/rIR 3ynw5atsZ2W/2x85JEtXA44R+l4VGv8i9JkEDblKKCxmC+3AMQqFkso91 Q5w7ecipBFAOFst1J/GMdU6G+5Gk7h2MWVwSf+k16XUtWk+CQQP4ovSfG g==; X-IronPort-AV: E=McAfee;i="6600,9927,10845"; a="361152774" X-IronPort-AV: E=Sophos;i="6.03,179,1694761200"; d="scan'208";a="361152774" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Sep 2023 01:26:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10845"; a="864719368" X-IronPort-AV: E=Sophos;i="6.03,179,1694761200"; d="scan'208";a="864719368" Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16]) by fmsmga002.fm.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 27 Sep 2023 01:26:49 -0700 Received: from orsmsx612.amr.corp.intel.com (10.22.229.25) by ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.32; Wed, 27 Sep 2023 01:26:47 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX612.amr.corp.intel.com (10.22.229.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.32; Wed, 27 Sep 2023 01:26:45 -0700 Received: from ORSEDG602.ED.cps.intel.com (10.7.248.7) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.32 via Frontend Transport; Wed, 27 Sep 2023 01:26:45 -0700 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (104.47.70.107) by edgegateway.intel.com (134.134.137.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.32; Wed, 27 Sep 2023 01:26:45 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=J9M39KYYw6cfx6cAHIe3IrOe+FPw9IeG1bqdJjv0h60NvgIYN7XgLMLZ69CpIgR8QsxO0nSciaHgvP3HZ9+AkDkaC7pTdKF3braBdoq2dU1P+gzCpMgNV05N3aATSzBuNMB/RbJhl/Ahx5wtfnrlqUUIpjzINMQCLTCUYwYsOkXbFm26ogAzm/Ud1NYKbcWPKcwiNCLFKDKlk4f32hYU0BcQc4/jC48tQYoVtLIV3dwG+Db8xdK8a+FsJIQFW8BJENfG07rjjrTG16sJURJVM+thiGE7P+Br+B7UByGY2ZHXVhODUEagAg0Msg1PUHSAHd3/UNKTerOexsOuI9y3Tg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dxKi2R1NsxEpNr4in/q8UiiY2pOzIWZuMcsioRG/MoU=; b=MNTpDRetEhUEMU/n2AtIFkFL0YKre321ACJkRJA2jrdnTxg/ibvV88+im5oo5FHNx3rwZ+2gsKL3rOgd9wmNyeO3sKtu7ymQAEAyEGMrPBGtnVWRh4gO08YYeYrkpH4I6Qjj6g4+H2KzwsWUB8HDEALwIMZmUTolqsKSvGxSiVyUY3QVcJSVwb6pJG9/dYm1JPtfXHx2KG7V65O/bDlVt+tA3mo/jBqT5wsjpYNJR39ajRYb72IgG0YL1Gg15wlIu1ncWNV3jI5z5t/svHXcQ8V45vu0XkDuPzfFjkcLVn1EMG8TALl2ZOPzsmjdLshtMeu+wLHddJM4sJB4KmrA1Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by DS0PR11MB7621.namprd11.prod.outlook.com (2603:10b6:8:143::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6813.28; Wed, 27 Sep 2023 08:26:43 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::43d1:af60:464:347]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::43d1:af60:464:347%5]) with mapi id 15.20.6768.029; Wed, 27 Sep 2023 08:26:43 +0000 Date: Wed, 27 Sep 2023 09:26:36 +0100 From: Bruce Richardson To: David Marchand CC: Aaron Conole , , Thomas Monjalon , Ferruh Yigit , "Jerin Jacob Kollanukkaran" , Akhil Goyal , Maxime Coquelin Subject: Re: [PATCH 0/2] add checks for tests not in a suite Message-ID: References: <20230915115206.132198-1-bruce.richardson@intel.com> Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-ClientProxiedBy: DB8PR09CA0028.eurprd09.prod.outlook.com (2603:10a6:10:a0::41) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|DS0PR11MB7621:EE_ X-MS-Office365-Filtering-Correlation-Id: 4abdece5-4457-46ea-96bd-08dbbf337af6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7UcqMY22DpCV42R2gGHgetUOHyjCu1CcoczWxqoMKi//erIc3AStrkHBHrUv6kpnDsxpZ9auKBsSqg/ET4R2TalMlo6n/1snzIDvtneAtTK3gvai2CHkVwltcWZ2UVEBq+jlM/x8tdP2cx11A38QNSVXTCjseAgjyGhxeLyzKBgJS3yfroGwtYU6XmXxxfSVXn3Bk881PMKD7Mlhvv0YNz+rJesAWp0hb4EB6Q3jOHIhPs3nVts7eOKKzGxMbJzAcAocUbhnHaFZc/pdAjKBsPDSJgNmKBqK/+42LlH5IG1xqwPp6JCXDpXGnsPp07NHHyIrvZXRF7+3wNrStCd9uc4wGs3W5EOmntg0TY8RnVeJZtVsyn8zvYjko7bGgDF6AK+m5hbCv8HA9ZlKCDxy+OwfDB6Vo2jBKdUo7kcfJmnZOZ0hQ//YTeTvhUcWJy7HpzmcGy1uxhk6xrxxYU3Ve44/ePZxzEtBrgNN1cW8CrY73fw3VWq/IJsuGMFFgrQFSFg+RcxIZ3oK9StaggG1QEIY8QReGk9XErwkp6wAqDUdgBEZxc65DdHCFXgObfkm X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(136003)(346002)(376002)(366004)(396003)(39860400002)(230922051799003)(451199024)(186009)(1800799009)(54906003)(41300700001)(316002)(66476007)(66556008)(66946007)(6916009)(8676002)(6512007)(4326008)(2906002)(8936002)(44832011)(478600001)(26005)(6666004)(6486002)(6506007)(83380400001)(38100700002)(5660300002)(53546011)(86362001)(82960400001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZXRRRU9iYWR2L01SNXRtSFNCV01pNit0d01jT09YVG9Xc0lXQnBzbDh3cXdL?= =?utf-8?B?Y3VLWCtjU0FNQ3FNVTN1RmxQMU91NWNtcE15SW1TRHFaWUVHc3kzQ0lUVXNr?= =?utf-8?B?YmlHZjA2bG41M2JZb29zNDdQQVVMNU5qYlVYV0JEK01yb0ZacHhMelhHdXdO?= =?utf-8?B?MnBZWXBYUmZNN2k2ODUraVUrUDN0blg5WDlCQmpSTmNCYTNBOFNkdmxqcGVz?= =?utf-8?B?WHl0SG11QXFERDVVZDBaM0RDaWd6UU94TVBMMzEwMVk0UnZ1eGI5RkZRLzc2?= =?utf-8?B?MVUzeGEyZ3d0MHZNekxPR0dHR0NjdGhmN1M1cTFVQnQ2ZnpwWTdvRVA0TzhY?= =?utf-8?B?ZThLTnZGMThqbFpQZjgrRnhLbm1RTCtraE50Z0pYMld6eGlvRnJRL2hCTnE1?= =?utf-8?B?TzB2N2NTcEZXeDMrMERDUmFkQkpBWXpyT3RJTjJkVDEvaXFBOHpTWlJtR0dv?= =?utf-8?B?ZkZPTVFYSXJ6TTBkTmQyUXB1R0x4OU5jdEZlZm44SHFLZHFsM2E5eDY4RlVB?= =?utf-8?B?dkk5ZFM5ZDREWXV6Z0xLbjN2bjExWlBBK25JRzEyK1JqemV2aFNQQzZueWJM?= =?utf-8?B?Tm44K2N3aFAvWUpiNVFNNDNIQUlPTXQzUlhld1VoREV0NmVYRE1GUFI3Q2I5?= =?utf-8?B?cVh3UDFvcXVDeHBkTklqVThvb1NXZVFJREs2ZnEzQjVUZ2tHNXBCTmFCaDV5?= =?utf-8?B?UlR2ODVBYk5rQ1BZT255c01mdDViYWlnR3pjMDYyeGExdTVwUVlqWlhpSDhG?= =?utf-8?B?ZHZJTGhXc241bHRGb2cyY3EwS3ZJb1Fhb0t0M1YwSXVlMGNqYm1kdzBPQWE4?= =?utf-8?B?RVRmdVY2cUlxR0FDdHVIcmQrR3pkTFlvaFphQmJQMFljd2RqdDNQMmZPcnFY?= =?utf-8?B?ckhIeU8rOHRoeWtid240VDdqTHIyZXFob241c05tYXpCbmx0OFJtTzBQYU9N?= =?utf-8?B?UE5JUzh3TE5LMTY3c0ZQUVpzR21ZQWVvNnNNVG5KcHl1WHRGdjEwN2ZVUHMr?= =?utf-8?B?aWxtTGMwRFZWKzNOMHd2TVZyN3EzbDZhUTh6UEdINEFsUm9NbjU5WGQzTi9U?= =?utf-8?B?SDZkd2FQUnlMbU82V3IzTGU3TTd6QzV1clF2NmdJYjYxMHlGRE5ncWxxZ3dP?= =?utf-8?B?VmVHZ3g0ZUZJNlV6M0pKM1RmaGZnL1JTa21HaGxRczljb2J6bVpjb2JIcFVq?= =?utf-8?B?NVlkRGdLYTdMbTl1Z2NFTHNxU0tRcFIzamxLbktOYWxkbFUxTjBUcjhFb09z?= =?utf-8?B?cTQ3bE9QOUhXeGR1eUZ2L2VjbGM5OGtkTDYzT2luME5CelN2K1lzUURhRENS?= =?utf-8?B?MldyaytpVi9HWCs4TjRZMmR0MExXRjhDaERQN3p3UjdQSURpSUVGb2c4djBq?= =?utf-8?B?bUp2eFJrNWg4Mk15bnZGRFJWank2WGtaNG11aGtBYS9YZU52dHJ3N2puZzIy?= =?utf-8?B?TUhISlA0cHloVm9ad1hzYnlKOEFoZFBPNHBjODJGQlp1UXBzZHJ6bVdlbzFv?= =?utf-8?B?VCtHM3F3OGhTZFBleVIvZXoydUxzUWZzT2VkQXl3Zkg2QjlwcGRPMHN3dmlI?= =?utf-8?B?ME5PSjBESXQvekZYbC8vajM4bVYzbUIwbUZ4RVFQUW9FNmZ3eTUyRkhCNjhF?= =?utf-8?B?TVVOamtKaEZnOGtiSDZ5Zk91amlMQTRselAwNHMycG9ZcktLK3FIa3Z6TzBD?= =?utf-8?B?TzBiNldrY0xuNFRQYWIxNlJYblgzQWQzMk1hcXN3c25DU0pRa0pmOXRZVUkx?= =?utf-8?B?alRCcUZJMlM1TXVSUzJLVlRkbG1iaFlhWHNabkttOTZKSmRyeHdualZtK3lG?= =?utf-8?B?MHJuVStEVS9EeVdNdmRjVVB1cmtFWUxhQ3A2RVZ3SHJSTXpkVThYTjZoeHRH?= =?utf-8?B?WTNBbFhqNFFhNXIyd1YvM1V1Z0ZCWW1WQ0V1TnFwUS9lSWczVWpBWVBnT1Vq?= =?utf-8?B?OFQ3dkU5WEFGdWw3MUluWGxHOHhVK2s5S1VpUC85OHFDRTRHeGV1a2hUdG0w?= =?utf-8?B?MFpKeTdjTGRuUk9NSzFka3ljSWJELytTSDA0MVVzU1RXU3FkWWZRQStpaE9P?= =?utf-8?B?Nm10a2lTRzZVVWI3RXpxVEFqQVRYejJqWXFnMnZHdlFyQTZ0NHBXV3l2NldL?= =?utf-8?B?R1pwckFLYmxtRUg2VWQzaXlGd0hYeFlGZURUTDkxMHVmWWpnKzdUNVhCdXph?= =?utf-8?B?NVE9PQ==?= X-MS-Exchange-CrossTenant-Network-Message-Id: 4abdece5-4457-46ea-96bd-08dbbf337af6 X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Sep 2023 08:26:43.2989 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cBeszdCR6eNthcv4caHK+TvxliL45eMu3o2uauOUcCEKLILksJnfU95o8FoiO2vKfJ5amqR6tQ7Dnyim8khrKI1yb90L6PUTMwiSQqaumnA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB7621 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Wed, Sep 27, 2023 at 08:30:05AM +0200, David Marchand wrote: > On Tue, Sep 26, 2023 at 5:01 PM Aaron Conole wrote: > > David Marchand writes: > > > On Tue, Sep 19, 2023 at 10:36 AM Bruce Richardson > > > wrote: > > >> > > To help ensure that we don't have "orphaned" tests not in any test > > >> > > suites we can add the following checks: > > >> > > > > >> > > * In developer-mode builds, emit a warning for each test defined using > > >> > > REGISTER_TEST_COMMAND > > >> > > * In checkpatches, add a check to prevent the addition of new tests > > >> > > using the REGISTER_TEST_COMMAND macro > > >> > > > > >> > > Bruce Richardson (2): > > >> > > app/test: emit warning for tests not in a test suite > > >> > > devtools: check for tests added without a test suite > > >> > > > > >> > > app/test/suites/meson.build | 13 ++++++++++++- > > >> > > buildtools/get-test-suites.py | 12 +++++++++--- > > >> > > devtools/checkpatches.sh | 8 ++++++++ > > >> > > 3 files changed, 29 insertions(+), 4 deletions(-) > > >> > > > >> > The "non_suite_tests" testsuite returned by > > >> > buildtools/get-test-suites.py is a bit strange, as it is not a > > >> > testsuite from meson pov. > > >> > > >> Yeah, it is a bit strange, and I'm open to new ideas on other solutions. I > > >> did it that way to avoid having yet another script to scan the files - I > > >> figured it was faster (in terms of runtime, not dev time) to do the > > > > > > I had figured it was "faster dev time" that won :-). > > > I am fine with it, I don't expect more complications in this area in the future. > > > > > > > > >> scanning when the files are already being opened and processed by this one. > > >> > > >> Of course, if we can get the un-suitened [:-)] test cases down to zero, we > > >> can theoretically drop this check in future, and just use the checkpatch > > >> one. > > > > > > Well, that's still a question that nobody seems to comment on. > > > > > > What should we do with tests that don't enter one of those testsuites, > > > and are not invoked by the CI? > > > > > > Though we may be removing some level of coverage, I am for > > > cleaning/unused dead code. > > > > I guess it does require actually looking at these tests and classifying > > them to either put them into the proper suites. As of now, we aren't > > really removing coverage if they aren't being run - but are any > > maintainers or developers actually running them? > > Could we go a step further than Bruce runtime warning (which is at the > meson level and does not impact running the test)? > Perhaps have those orphaned tests fail unless their test names are > provided in a env variable like > DPDK_TRUST_ME_I_WILL_SUBMIT_A_PATCH_FOR_THIS_TEST (naming is hard > ;-))? > > With a systematic failure, there is less chance that > developers/maintainers miss the situation. > If those developers/maintainers simply waive the warning with the env > variable and don't send a patch, well.. too bad. > > After a release or two, if we don't hear from anyone, we can start > removing the unused one. > I think that seems a littel severe at this point. The one gap we have right now, as far as I can see, is actually explaining what the various suite types are, so that developers can choose the right one for their test(s). We may even need a couple more if some tests do not fit into the existing categories. Once that is done, we should then start looking for tests that are obsolete and can be removed. If we do already have documentation on the various suites and how to use them, apologies for my ignorance, and perhaps someone could post a link here. /Bruce