From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2B46C41F55; Tue, 29 Aug 2023 15:39:50 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BBC05402B5; Tue, 29 Aug 2023 15:39:49 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id 7089E40279 for ; Tue, 29 Aug 2023 15:39:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1693316387; x=1724852387; h=date:from:to:cc:subject:message-id:references: in-reply-to:mime-version; bh=OYAfAaqkvd+qGoFUTPbnfIzSVA9S+Si8NpgEZdqTvsM=; b=idHRZy86pywha92WtsSxK1GNAm2d8RqguPKsbwjB4hN2BwCC5xQ44uNt 4thiMOb2UUfDAQz9SiMsSwIvR2q9sZ3POWyEwgn3zubCMdEhw7ZCJ1tUw h2Z3NRwk6iwHuRd7AV/oZupC22pHcm4k2D7RQnH2ffKldZ3fh2NzgujSD kq7/PsGRnUQUMBY8EnEG3HnfmoyZ9H97gncoNT/za99wpq+LYgiMgyTo9 WKuVZOSOqyFymT7CzkobmAq5Z8lsjG/pUdR5c7blIO6Kky9i43HfdXxAv Hc8IKsc+W/IK+26cfLUTqb/4BhPdeBHkpXD3Lg2QE0Osp+KAoQv7yggpz A==; X-IronPort-AV: E=McAfee;i="6600,9927,10817"; a="355690402" X-IronPort-AV: E=Sophos;i="6.02,210,1688454000"; d="scan'208";a="355690402" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Aug 2023 06:39:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10817"; a="804106888" X-IronPort-AV: E=Sophos;i="6.02,210,1688454000"; d="scan'208";a="804106888" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by fmsmga008.fm.intel.com with ESMTP; 29 Aug 2023 06:39:45 -0700 Received: from fmsmsx602.amr.corp.intel.com (10.18.126.82) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Tue, 29 Aug 2023 06:39:45 -0700 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27 via Frontend Transport; Tue, 29 Aug 2023 06:39:45 -0700 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (104.47.58.168) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.27; Tue, 29 Aug 2023 06:39:44 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=E0om0CZdBzDUj2bi0rp6EjkGhZxC5Tx4HbIKdxDh1uxoaoL3zkC07gEDJdVES9py2J49ZJTwM6nT1AIr87yDWvBLfQlkqpfW3f/02KCCQYceMi+VYzV5TMfNKOHcPw4QFttGyluK64bQooqOlGjkz5fHEFad74veMBfjDSza6M3axIshw949yDunwwkeX6TZk8dl56fvrjIurhjczR+lJ4pVs4/aylsv7OyQkobYkAnrXskTMRGtXzh2wObLpvtrMgBPKqFLHvIsMV4EgtB1wPifmdWSKSz2COjTS4nWdnlAxFReGznQRZ/an7iUgze6h7KDoWwNokE2pdlifIeFRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wGRxHi8tJ7QkXJr37SI1iGZvru1qupq7YXRqmiwPH9k=; b=nmjbpAIda6Fb+9Yw7ZlIFX8gmDD0j4KYrjhr+h7CvoABfDkmlEMDcqTNX05UuJA/6yjC5rn1gAc9+Cy5k/ZkxKX1QfatUzmND1iXDQKEh4de/VqfRoNllTdCdBrEscdKDQnurtGqWx5RdqBr8nOzSLpEo0PhYGFdNcwoy6kJJwGqCDdLRatGymU9bYL4XFtr3Iy1BaidiHoC4T+0SemsqOy+TpQLD8yp2TwmzdTkrkASZr+16Epc+r5PizpLGQhfmmesNGWPpk+42GjNlB7qcRMBzDhPi7u+pwKpzMyPhbWCeScfxif+Di4de9GBVH5gS92ETbW3M6b8ohBQn8mRrQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by CH0PR11MB5475.namprd11.prod.outlook.com (2603:10b6:610:d6::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.35; Tue, 29 Aug 2023 13:39:40 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::43d1:af60:464:347]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::43d1:af60:464:347%5]) with mapi id 15.20.6699.034; Tue, 29 Aug 2023 13:39:40 +0000 Date: Tue, 29 Aug 2023 14:39:32 +0100 From: Bruce Richardson To: Aaron Conole CC: , , Subject: Re: [RFC PATCH] app/test: add support for skipping tests Message-ID: References: <20230817105851.501947-1-bruce.richardson@intel.com> Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: DB7PR05CA0001.eurprd05.prod.outlook.com (2603:10a6:10:36::14) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|CH0PR11MB5475:EE_ X-MS-Office365-Filtering-Correlation-Id: f40cfc9d-09d7-4cf5-5129-08dba89564da X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: uu+nrpce2Oo6o6sj0iGwrWbF9Sz3IvX/hjlJfpbU6ML1ttUL6HZvRoZb48uaIRAYSUDV4MJrxc71smm6bAvHu+5Zzk8I0XQ+sKFNeAk3FA6t2qbpab+ZyyMyArW6FpGn8RnIOg5K/5pMQZD9Yl9HIRBXQS5rLkcvezKYAxuN6KIVIVezLapqpCaalXkgKaaay4ZbT5pfKEvjY98n2JpLijOXnBgcokPzrSAPxeMeT1KtDLgtRTFNecjosSpFjQ3pdz8tN2BEuGA/c68IFxJ8VHFXK5/fWdjoECDRof90w3XxsjeOfwSTyWy2ydJIuaYKtriXPEJ700UB5a46uig61deBJFN463PtE+L+GkS505MKVHw65nHP3C9JN6mftd+D/fdL1LBoPCEKLwpz8YbRLvPFXBhJVnWk99E9lu3hp1HXj2gC5fNcbRRwbajU5Drl3KtNzpBGzRZ54uwZDBBdEknKMLp8OKald8+GkvUP52PMpIMDgUl0XnM+cqUkT9CJPrFn0lfBa7CV1/Sosx5u7/uaXduxKXtqtcOTvytvy5N3WNLd8j2ckK8Kta7u5xQN X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(366004)(136003)(396003)(376002)(39860400002)(346002)(451199024)(1800799009)(186009)(6666004)(6512007)(6506007)(6486002)(86362001)(82960400001)(38100700002)(478600001)(2906002)(26005)(8936002)(41300700001)(66556008)(66476007)(66946007)(44832011)(8676002)(4326008)(5660300002)(6916009)(316002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?pnBnQa1l4+9QPjIiiBa+odBhikYgoomYMMJ9+H4Is4+qTldPXu8Wtargc5H6?= =?us-ascii?Q?f+wtmvQJFz2Ws19xfxyJ5aRJtJggHmLaLd+GrObjPKk5MoGFcAp02B+pzuoZ?= =?us-ascii?Q?gtn92CG1ob1TIwywUFq9V9eJcGzkdgKp1I+9+sOqJfjIrVAVQ5kmEaARXUSh?= =?us-ascii?Q?Q4meaLG+3I+vRiFkDQ545BlXb+ViiZBiq3hSypiWRDYN+ZlMsgl4oJ02EykA?= =?us-ascii?Q?0HboG0iHbadVkwNU4V2S2uUSz8lEPv2Y9Ei4yyNelKA02Tk+gu+55UKyuJLn?= =?us-ascii?Q?7YN75C1Xi1+Wcv0uFyXZVU/U5rDpA1K3lT13wC9ZHw146Cx2v16ySw+XDOrV?= =?us-ascii?Q?RqAtDZRGpmA3f4Vr8637nynllfomnCnfeKu21Bx5VR++BfudRUPNmCW2HKJt?= =?us-ascii?Q?LzFnp/4iykCg8UY7O5TGkblgt30z3anJuf/vxmYPWo3TgPj+tRRKY0Oi3F27?= =?us-ascii?Q?O8gt/wIZUOf+ISOBs9vy3+S7TQPOXkDYWK1CGHXeLKM6LeeTqTy524hMzvsz?= =?us-ascii?Q?0Ww6dskAVHW1ryGx+/M2S6L0lQjDaQi7+1QouzuZH/gx3Oqps7HPVjI2L1nC?= =?us-ascii?Q?3HgbkDMMAQ/zM1Ex5A66mjgPY0Z+Mk5kDyRul+njlTyDL8wUeaWnh8WO54vY?= =?us-ascii?Q?EYf5MHc/R0d2kA0p+tMdzLfDLOOYVKUCJL/c/BkE81TgJ83qS7+HEoMAwVrR?= =?us-ascii?Q?/unxveo2sMvK1TuV2iVnLeWMo4pwc4QqMv9k/seHml7tYD55eO/uXsV1nF8y?= =?us-ascii?Q?lEb/m+pUx9eqiM0aIj9u56PU9TGh9vsVCRwbqLKl/XLBd2qKP9rT2L5XGbVr?= =?us-ascii?Q?ir6MMBs5pR24TZpZvw2SE8/pnKaXSXdo1DJ6vzC7N/ZR0jFhh0MgALbPf8yk?= =?us-ascii?Q?U5pc0pFvPGeIfxBt0FcaNqjf/pXKs8SKmfBmfhOJNKb4qCPxAHGMf1KDPsLo?= =?us-ascii?Q?vPJrbDfs6HRSW8pM5lcDTCVHiZZOkiF71ueHIVep+GTYongcXChBs5xBVSnq?= =?us-ascii?Q?7c21CZyJnvgLXxWpgjGn/+yybt3jbkVK9dLRII+xSb1EDZVKG+zacoZalily?= =?us-ascii?Q?XJSVcj4Qsjc9JZURoMpdQMS/ctAey/fV9kEKR5CvceWPJ31EbpjWsIFAfuXu?= =?us-ascii?Q?Bs4BxdQpNFlz4cjKHTbT31h5IN0NNJbC75Nx2Tx/vvodUIlGPNWTcTJcrGRq?= =?us-ascii?Q?2z1jIJItBKiV4gwfC7/DEqllBh2l/MlULQtAuIGAhvO11C1Q78QVa+vLEO9G?= =?us-ascii?Q?cESX0svQlXzLvhk+A+luKJHu8/27uLyBMyP3cv3sw6NOD4X0ePXNIqoOp4f/?= =?us-ascii?Q?WegKx2WYlLAxjFUWvDvKN8q1XNfG96qAdgGslNSv5VjeZr9270Aq6xUkT9os?= =?us-ascii?Q?ay9yXusNMHIOUoaIAwYo74klaXRgZF9b+9dYc0PPm1IHMnYsi89c54zg621y?= =?us-ascii?Q?PErnCv4QgbhjknLpeVTWfhh79WerppaognodwldDvC2f7qGptzk5H95mDeRd?= =?us-ascii?Q?mvj7P76vL5dHmamQKv1ofArwbpHDKMMoUAgBIjAnZjJC+q0g5jhTf3CdSrXe?= =?us-ascii?Q?OIdiV9Me2uf9LOAo7hcmMpnYR8viqHmLy1MVThCQLYnwhK7gRkRimEAXtUZK?= =?us-ascii?Q?eQ=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: f40cfc9d-09d7-4cf5-5129-08dba89564da X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Aug 2023 13:39:40.0447 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: aQREJ0rIXps0eIFJwNEWw9xMuvzGY47s9+udcw1W7IqNGyQXgteyedxNBoxWUFArM/4AZwaH+SpZ/bXLg10epzXug45BGhEjk3x38jwNL5M= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR11MB5475 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Tue, Aug 29, 2023 at 09:27:23AM -0400, Aaron Conole wrote: > Bruce Richardson writes: > > > When called from automated tools, like meson test, it is often useful to > > skip tests in a test suite, without having to alter the test build. To > > do so, we add support for DPDK_TEST_SKIP environment variable, where one > > can provide a comma-separated list of tests. When the test binary is > > called to run one of the tests on the list via either cmdline parameter > > or environment variable (as done with meson test), the test will not > > actually be run, but will be reported skipped. > > > > Example run: > > $ DPDK_TEST_SKIP=dump_devargs,dump_ring meson test --suite=debug-tests > > ... > > 1/9 DPDK:debug-tests / dump_devargs SKIP 1.11s > > 2/9 DPDK:debug-tests / dump_log_types OK 1.06s > > 3/9 DPDK:debug-tests / dump_malloc_heaps OK 1.11s > > 4/9 DPDK:debug-tests / dump_malloc_stats OK 1.07s > > 5/9 DPDK:debug-tests / dump_mempool OK 1.11s > > 6/9 DPDK:debug-tests / dump_memzone OK 1.06s > > 7/9 DPDK:debug-tests / dump_physmem OK 1.13s > > 8/9 DPDK:debug-tests / dump_ring SKIP 1.04s > > 9/9 DPDK:debug-tests / dump_struct_sizes OK 1.10s > > > > Ok: 7 > > Expected Fail: 0 > > Fail: 0 > > Unexpected Pass: 0 > > Skipped: 2 > > Timeout: 0 > > > > Signed-off-by: Bruce Richardson > > --- > > The idea looks fine to me, although I would really like it if we could > do a command like argument instead of env variable (but that's just a > suggestion to color the shed). > Yes, I prefer cmdline args too, but I believed that for running test suites having the tests to disable set in the environment was a better solution, since it was easy to get at whatever way the tests were run via a test-runner. However, there is support for passing "--test-args" parameter to meson test, which should allow passing this stuff via cmdline. If you prefer the cmdline solution and if it doesn't have any gotchas like that, I can probably look to rework this to use cmdline. I assume that any other test runners in use would similarly be able to append addition cmdline args to the test runs? > Just minor nit stuff below. > > > app/test/test.c | 27 +++++++++++++++++++++++++++ > > 1 file changed, 27 insertions(+) > > > > diff --git a/app/test/test.c b/app/test/test.c > > index fb073ff795..3569c36049 100644 > > --- a/app/test/test.c > > +++ b/app/test/test.c > > @@ -193,6 +193,20 @@ main(int argc, char **argv) > > > > if (test_count > 0) { > > char buf[1024]; > > + char *dpdk_test_skip = getenv("DPDK_TEST_SKIP"); > > + char *skip_tests[128] = {0}; > > + size_t n_skip_tests = 0; > > + > > + if (dpdk_test_skip != NULL && strlen(dpdk_test_skip) > 0){ > > + char *dpdk_test_skip_cp = strdup(dpdk_test_skip); > > + if (dpdk_test_skip_cp == NULL) { > > + ret = -1; > > + goto out; > > + } > > + dpdk_test_skip = dpdk_test_skip_cp; > > + n_skip_tests = rte_strsplit(dpdk_test_skip, strlen(dpdk_test_skip), > > + skip_tests, RTE_DIM(skip_tests), ','); > > We probably should check for n_skip_tests == -1 - although it shouldn't > ever happen for this code. If it ever did happen, we'd walk right off > the for() loop below. Actually, it looks like that error condition for > rte_strsplit isn't documented. > Can't happen for this code. The strsplit only returns -1 if the first or third params are NULL, and the first param (dpdk_test_skip) is checked for NULL before the call, and the third is a local array pointer in this case. > > + } > > > > cl = cmdline_new(main_ctx, "RTE>>", 0, 1); > > if (cl == NULL) { > > @@ -201,6 +215,15 @@ main(int argc, char **argv) > > } > > > > for (i = 0; i < test_count; i++) { > > + /* check if test is to be skipped */ > > + for (size_t j = 0; j < n_skip_tests; j++) { > > We might want to check j < MIN(n_skip_tests, RTE_DIM(skip_tests)) to > prevent a rogue env variable that makes rte_strsplit fail in the future. > rte_strsplit cannot return a value > RTE_DIM(skip_tests). The main loop only increments the return value (tok) at most once per loop, and at the start of the loop checks for tok >= maxtokens. > > + if (strcmp(tests[i], skip_tests[j]) == 0) { > > + fprintf(stderr, "Skipping %s [DPDK_TEST_SKIP]\n", tests[i]); > > + ret = TEST_SKIPPED; > > + goto end_of_cmd; > > + } > > + } > > + Thanks for the review. /Bruce