From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DBAA143C4B; Tue, 5 Mar 2024 16:11:13 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C99CE410E7; Tue, 5 Mar 2024 16:11:13 +0100 (CET) Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) by mails.dpdk.org (Postfix) with ESMTP id 2B0ED4026B; Tue, 5 Mar 2024 16:11:11 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709651472; x=1741187472; h=date:from:to:cc:subject:message-id:references: in-reply-to:mime-version; bh=eLXIgPc6IxbhAN1z+g9QTznsSq4a+ae307yX4ylAgdc=; b=Q+BF3AhL0wdum4aJLCuKa7ExbpFjvQhixeYH2o70S4hbFCCPUnTZF4Fr W0Te7LCluW6ahy3aI1L0cnZii2gc6EiHwUGIr6NrjhuaQPU/ulNpkUEV8 KOfPadDlcdJziAaiASmL+erpXZAa5yUtRsBL82zh8uTwcOxwBQ9hiZ2Ik zT7Y8attAID2NuOCdtI78mKCmVLQH/Qem5SUwY+s5O1iK+Rno2U6e6Qxx 0bSNQLyILyN7d+TbedF4ZBofNZOnI+icG5iOPD20dzPWimdC49D7QMnKp YixQW5dXquNtfYLMEl5bcLM38Qkcx2lpWTgD7Dx4xRq8J30N6Tbc+VS73 w==; X-IronPort-AV: E=McAfee;i="6600,9927,11003"; a="4066193" X-IronPort-AV: E=Sophos;i="6.06,205,1705392000"; d="scan'208";a="4066193" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Mar 2024 07:11:11 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,205,1705392000"; d="scan'208";a="13895751" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by fmviesa005.fm.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 05 Mar 2024 07:11:11 -0800 Received: from fmsmsx611.amr.corp.intel.com (10.18.126.91) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Tue, 5 Mar 2024 07:11:10 -0800 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx611.amr.corp.intel.com (10.18.126.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Tue, 5 Mar 2024 07:11:09 -0800 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35 via Frontend Transport; Tue, 5 Mar 2024 07:11:09 -0800 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.101) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Tue, 5 Mar 2024 07:11:09 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=clurKZYZw2lnNsAXqkCCwkCT63fePZLnk6zI5LPRaiqzOtq59+7vfseEtsTm5tUYYujxhk3LrcN9BxuN3USyDgcWpWfxrYVz9Eg0Y3/8i38+QnfT3UQH7dDkc0p3+d/sM32qpNgOB3bBjdP7b8LYqF0Eghh7Rh4OimeK3XUY3ymq/lJNaJdoReprqAwclrT6WPOAy2UM+GaA/j6MCbZNgBbS4fS4w+3ILscukhp23CXXg10BqWTV3u5D/F8UThQb54dVV/w6DlX1Yj7RYYp4ogdW9MMzmOUIIa/GiMfSqJKz+lsXRu/iFk1xXus7pjffpmuEtZygReyjHoMPZg0FmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=YExYVOnIBfWBWA1iybwT7ZbXIYEm5XhwaDYILMUJd0k=; b=igxJkd7ON7yVJlNk547tY2XcNtX+E0Q2pXbFzO84bMlF7Ymv/7a3Q3yD1gyMBhUhcSMxwSJLBX+b6vScZ3VeulqAlA5mmbCzVjQFwNi5lCwBlwZ3b6D8qiiWfbsD+FfzlhOeQfjuMMAIpsKq9qRdb/6Ch1gJN+oAbiy4asTw4ado0CR5n6uffD/2NLTld6HsL9HJb3QABk+M/Eb0u2SwUhuBijUNsPGrWwwCt0f/vzBGNjXxzzEw3pVm7TV+9Ei3pEpiHYXqjbEirt+jHZQD75PviTcj+fWZ0quI6t9SpoJwOu8l3DA1TJNCLrtNguafTeYTLs04LZQKkxp1NsN6NQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by PH7PR11MB8122.namprd11.prod.outlook.com (2603:10b6:510:235::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7362.24; Tue, 5 Mar 2024 15:11:05 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::d10:3009:a8d3:1d2e]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::d10:3009:a8d3:1d2e%7]) with mapi id 15.20.7362.019; Tue, 5 Mar 2024 15:11:05 +0000 Date: Tue, 5 Mar 2024 15:11:01 +0000 From: Bruce Richardson To: Akhil Goyal CC: "dev@dpdk.org" , Subject: Re: [EXT] [PATCH] app/test: don't count skipped tests as executed Message-ID: References: <20231113150533.249808-1-bruce.richardson@intel.com> Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: DB8PR06CA0033.eurprd06.prod.outlook.com (2603:10a6:10:100::46) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|PH7PR11MB8122:EE_ X-MS-Office365-Filtering-Correlation-Id: 0a0ef781-722e-479a-8f9e-08dc3d267a9b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: C3VSpfhbfEyIYsAYpRcbXT70FPapdN1utsJcXwxKbUY2vMLW0DcyDm2mlNGNTk/AdXzqYc0ElUHdmlp7OwRUUoPWOxSuj2uoeEz72G4Fv1jgdabvhmdaS1ciXVPR3E9y9gPUGsrjrBtqS24FbFfUk8IvTCPG44TweoDfy8G1zbY0cjasgnDsNoBhSQlHKjL+46/LoNQj+/1LRXqukdE5qJwrw2qUz3gm6XRlIlTUyXnfcMMb6/sDOdrjSEVsbUh+zLiytdNWVq1ZdxigzefDayUHFGadcjSP21NcgdAtCBaZewo/4sYOx1iT0xPwYUJMcjJsNI4D2Y9hyPUyqKwkBZjJF7PfdLBi0ed0BeVfQQsvYJ8z5OMv7QW1gn7qcrguy6T8eDvsx+uJKdlRY5AO/V+sAS+Mmch5z5zSpPYtl5UUmGqtaiarHfqdNJZNSbe7GgVkCZpwWtw+t4fHbfNvKyukZztxDF+9vWiNbZHjb1jiHdr8HVRuBs89/Dn0Hat0bJ+ERm14bLm/hclw8uOYWSuzK48tfIU4CumAjoo2Zwk0e5FeydhBNBltKJZGhfr4hfOX79VfYWFHhaJJT7JZvWm1aaBPMY7XTLm4AzwQN6Ok7jpTh+2eRg/0ohv66pceuDnhZwcaMtwipM1LKS9uf/JpmdFb2SWjllvioBpAFTk= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(376005); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?q6cTGtxh2QmWRXqIhfWHD8/uHKXYi5MgkvsgTMXIWkIvjPM/a/jRzqX2Dt6W?= =?us-ascii?Q?voo+wi/OpRuAMYE7vVgPKgd+H4I/USdsWh5TjBkWbGlBW6WxSHiN+N4+g7mV?= =?us-ascii?Q?33NSVSIEdLCvwB+vAsOcaknGoWCVtDonRHrFgJu5+N8JpkLZFdIfeOgJnW5Q?= =?us-ascii?Q?RE3hZW0EDi6iGgIGS1W/xikkkX1bMEbKeLMNtxcOwsEXdcZVQ/DZMEPzP/Ui?= =?us-ascii?Q?R/E3JTITCEyQ4CUMQV6K4NrSL8qfwtUd65NXxiqwwQRW3t+OTpcRb3IwYsba?= =?us-ascii?Q?eb9VgvMLqv3QdoLx3Zc2jACQ/WawtxWWiffx2oio2z78Y+DkNwJDBmg0ZZQV?= =?us-ascii?Q?IGz6OYQ4FjKVi94StHKV9cmmw5wUoJNflfdfOaSZlwgsy7UNkKDFAJKjpnOr?= =?us-ascii?Q?dQl2YFN6M7FjpyfPgNY/cXJMUU/zkifeCghTXQswEnHUiRxWQfngbSdvaYcX?= =?us-ascii?Q?ArRFY2Nae1M4O7HDWw5V/EKZV4X4U61saMaW2myHppqPMd/kcRq54KX/xMes?= =?us-ascii?Q?VlkBjHvauW6z6Nw7mcXXakD/uGX/rTovIHE7Ld+X0o+6PRWtOvL5WnGZ6V5T?= =?us-ascii?Q?TVEFK0CdUxBtjr/b8LlCB9XcxR58QMBbOuk8hFt6Psiqd9Dfo4aezfJfjhKD?= =?us-ascii?Q?DnW/owIdWi4m0qC3/Gacmd4bWmaxOLf4UC9v76RYRIrYIl/SzMyL+gKOZA+m?= =?us-ascii?Q?QuI/Sm5r+JfVCOT7JCG4g9q+FjkRcGyoVEzMRZyjrayZVoERFMK4ADSnDPzM?= =?us-ascii?Q?DtoPr+l2yRYRHDjya1X/A9d+/Gil2s/nZchHecJcydIP7nrqa6ve/edov786?= =?us-ascii?Q?79OPvi6ycdmVdRZW7vYcx70UMulXlfK4XNTtuLQXReLBg9x90BT82HaOEedD?= =?us-ascii?Q?mJEHSDxaMHiWPYtuIMGCgByXQrSp7Yv2lGBbcOCCt+5mE29nr4WjYQCifoOp?= =?us-ascii?Q?W6isrMAgIGSYKF7v3vNfJM3+Orc99DSF9I970KWm3xFZ7jLZKfBDG8hbwAWG?= =?us-ascii?Q?I6lIo9FgCuu2VOVjIoErW1AOqjHAJE78GzUW7smfQHlUmcJvEyy1Q/shYnPB?= =?us-ascii?Q?F9IYIyP0uXz4gBme89mgBc+Njisop9wgarPb7gK+e4ACDqn3uvtEEVC7PyVF?= =?us-ascii?Q?bRUagbWeGi+bjJq26Nc8/I9d3EQGIsNUv9tXESB2Pekz17L6rWSkhlsRssyY?= =?us-ascii?Q?m61IY3pUErpirnmfbs8dwVKu7me+cEPiJ+Ut+VuqVBOw7CFI+JqemPH0v93R?= =?us-ascii?Q?dNNqozAea6Kz8J67CNAENh3I8FgCK/TbAU8U1pNm8wJXE9ndLYhS5V+TeIST?= =?us-ascii?Q?wE+wdPUDK4J2YfJkZp7UDMCcBwLbcAS2YU6xA8cw704vrUFX3/O/loUTTr+8?= =?us-ascii?Q?EqpRNt0mVazUzvi+BqVOQsi0V5auIZiUifA/lZZLHoSfzODXCImTiAbYIFm7?= =?us-ascii?Q?LHy3tmqKc88AbjD5B7EKgv9ygFt/B92T9CBXo5oXNcrzS8CLgaeBMOktbFrm?= =?us-ascii?Q?vPEPD1iAPOIp+nl36PCulHpKtcbd7BNLCozv5+srsY0VGhx/Ny/Tzuw3Ze/V?= =?us-ascii?Q?+Vt+jQDpq30la8rvlINAGN3Zy/QUDYg1VMtB2eklI2y+BnO9bkGBI0kL9aKO?= =?us-ascii?Q?Lw=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 0a0ef781-722e-479a-8f9e-08dc3d267a9b X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Mar 2024 15:11:05.6800 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: X6u7gFEDIG28OvayvKmMccn9zQl2GDOEt4nY3WEHc7hNzJKIQ9JwiIq4vWjCuU7858iccYYYm7dPag6qfHm8anC8hrdtYiK8FWoKFGws0LE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB8122 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Tue, Mar 05, 2024 at 02:36:27PM +0000, Akhil Goyal wrote: > > Subject: [EXT] [PATCH] app/test: don't count skipped tests as executed > > The logic around skipped tests is a little confusing in the unit test > > runner. > > * Any explicitly disabled tests are counted as skipped but not > > executed. > > * Any tests that return TEST_SKIPPED are counted as both skipped and > > executed, using the same statistics counters. > > > > This makes the stats very strange and hard to correlate, since the > > totals don't add up. One would expect that SKIPPED + EXECUTED + > > UNSUPPORTED == TOTAL, and that PASSED + FAILED == EXECUTED. > > > > To achieve this, mark any tests returning TEST_SKIPPED, or ENOTSUP as > > not having executed. > > > > Signed-off-by: Bruce Richardson > > Acked-by: Akhil Goyal Cc: stable@dpdk.org > > Yes this makes sense. > One would say executed should count the unsupported cases as well. > But I think this makes sense to not include them in executed cases. It's a good question and there are arguments either way. I'd say that no test should return ENOTSUP now, and that such tests should return TEST_SKIPPED. For now, I think it's best to treat them the same. > This would give better correlation. > Can we backport this as well? > If LTS maintainers want it, sure. Adding stable on CC.