From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A6B3042644; Tue, 26 Sep 2023 17:01:52 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4BCB840277; Tue, 26 Sep 2023 17:01:52 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id D1BB040269 for ; Tue, 26 Sep 2023 17:01:50 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1695740510; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rRAEPPiPpXoQVGMXu23S77LD7fURSgNnQ/Itt+OCK90=; b=eFX+NPMPsiaKOYCVqNpIc+cdwVESDjLovPR9h1F6GncZnpfAwAmnM3F94CuOKHUOJVo3Qj xpG97GRh+nW5ODAU6xADZyGYire5/JDPfHd5QMuwJ4Bfrf07OoM4VSgL324985FQIdiN4p PFE02X/40KxFgNIKt0SMG6twOkPGJ8c= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-331-03nJ7dXdNZ6i3R1FiM_LJA-1; Tue, 26 Sep 2023 11:01:42 -0400 X-MC-Unique: 03nJ7dXdNZ6i3R1FiM_LJA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0C87418811BB; Tue, 26 Sep 2023 15:01:42 +0000 (UTC) Received: from RHTPC1VM0NT (unknown [10.22.8.239]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 83B861005B96; Tue, 26 Sep 2023 15:01:41 +0000 (UTC) From: Aaron Conole To: David Marchand Cc: Bruce Richardson , dev@dpdk.org, Thomas Monjalon , Ferruh Yigit , Jerin Jacob Kollanukkaran , Akhil Goyal , Maxime Coquelin Subject: Re: [PATCH 0/2] add checks for tests not in a suite References: <20230915115206.132198-1-bruce.richardson@intel.com> Date: Tue, 26 Sep 2023 11:01:41 -0400 In-Reply-To: (David Marchand's message of "Tue, 19 Sep 2023 11:07:13 +0200") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org David Marchand writes: > On Tue, Sep 19, 2023 at 10:36=E2=80=AFAM Bruce Richardson > wrote: >> >> On Tue, Sep 19, 2023 at 10:29:07AM +0200, David Marchand wrote: >> > On Fri, Sep 15, 2023 at 1:52=E2=80=AFPM Bruce Richardson >> > wrote: >> > > >> > > To help ensure that we don't have "orphaned" tests not in any test >> > > suites we can add the following checks: >> > > >> > > * In developer-mode builds, emit a warning for each test defined usi= ng >> > > REGISTER_TEST_COMMAND >> > > * In checkpatches, add a check to prevent the addition of new tests >> > > using the REGISTER_TEST_COMMAND macro >> > > >> > > Bruce Richardson (2): >> > > app/test: emit warning for tests not in a test suite >> > > devtools: check for tests added without a test suite >> > > >> > > app/test/suites/meson.build | 13 ++++++++++++- >> > > buildtools/get-test-suites.py | 12 +++++++++--- >> > > devtools/checkpatches.sh | 8 ++++++++ >> > > 3 files changed, 29 insertions(+), 4 deletions(-) >> > >> > The "non_suite_tests" testsuite returned by >> > buildtools/get-test-suites.py is a bit strange, as it is not a >> > testsuite from meson pov. >> >> Yeah, it is a bit strange, and I'm open to new ideas on other solutions.= I >> did it that way to avoid having yet another script to scan the files - I >> figured it was faster (in terms of runtime, not dev time) to do the > > I had figured it was "faster dev time" that won :-). > I am fine with it, I don't expect more complications in this area in the = future. > > >> scanning when the files are already being opened and processed by this o= ne. >> >> Of course, if we can get the un-suitened [:-)] test cases down to zero, = we >> can theoretically drop this check in future, and just use the checkpatc= h >> one. > > Well, that's still a question that nobody seems to comment on. > > What should we do with tests that don't enter one of those testsuites, > and are not invoked by the CI? > > Though we may be removing some level of coverage, I am for > cleaning/unused dead code. I guess it does require actually looking at these tests and classifying them to either put them into the proper suites. As of now, we aren't really removing coverage if they aren't being run - but are any maintainers or developers actually running them?