From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2C2444308C; Thu, 17 Aug 2023 10:46:18 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 27B8940EE2; Thu, 17 Aug 2023 10:46:18 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by mails.dpdk.org (Postfix) with ESMTP id EE66A40685; Thu, 17 Aug 2023 10:46:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1692261977; x=1723797977; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=TOo6+fwAGQqBhbAm5J+tsHXXRkk4sKX6TORkSskDzHc=; b=USzpnAZD2We/WfkB6vQjv7wfm2U7PKlXI12Avsg8sPyh69fBzrhl1X6g 78H0YTcmZORyZtJeD7C9ACTR7KcBfekYiNcIPqbYM8kPu6+p8PySm/X0y MzFIeqU+n9mSCFKqxYtdgvzW53JJs+HTRpfA+N4ua2pigrVdPQyKag+1r DyeZ3kXGr/rLQAUV0Fk9ibGaPgSKoM4+q4ER4ts1BOHvt5FmgAE7DygNn yU8acJT+fL9KoGo9wlJ80qb3NOO18Ef8f5sljupxiaYGtsI87jWJ9LgSm 0J5CK2ct1HDe8M/VzTQd6RSoUkS3JU64wtkCaSsBVaBErlWaifNMcVzby A==; X-IronPort-AV: E=McAfee;i="6600,9927,10803"; a="436657375" X-IronPort-AV: E=Sophos;i="6.01,179,1684825200"; d="scan'208";a="436657375" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Aug 2023 01:46:15 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10803"; a="737623291" X-IronPort-AV: E=Sophos;i="6.01,179,1684825200"; d="scan'208";a="737623291" Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16]) by fmsmga007.fm.intel.com with ESMTP; 17 Aug 2023 01:46:15 -0700 Received: from orsmsx611.amr.corp.intel.com (10.22.229.24) by ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Thu, 17 Aug 2023 01:46:15 -0700 Received: from orsmsx603.amr.corp.intel.com (10.22.229.16) by ORSMSX611.amr.corp.intel.com (10.22.229.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Thu, 17 Aug 2023 01:46:14 -0700 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27 via Frontend Transport; Thu, 17 Aug 2023 01:46:14 -0700 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.176) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.27; Thu, 17 Aug 2023 01:46:14 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hdEyAqq6M5OXS0OmhMMmXzvEVmTU/khYIgxaoi4Tx5Oxufqvmf2HlTnR4im+S1FoTnrBLFIo/7Xr+D4F3Gx1ZyIJI8Yruw+SiKlgrYfDcH6vpziPbgzCrUADuf+f3ib0N2lVgiTh+fw9ntlr7nMnvtFBhmVYdPfc6pZ1GxrzOHHeBr651RnvsK/2/DFKg/L14d85tGRM7iVhWFdWAFJ4UIBR2JvrdZrhu8Ojclmcej0tmgGApfTmCf2oL9XAwJP1egQvrenPW1rKzoBbzcGxSnG9xH8reEYAezcDPtIobdO2Ui2OdNSG46vV9L/pUq/SAekSDgEW/7TWTdyLUPRKSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IlDoqArqERtOoPBy6wPIdfFm9qSAR1RocoMu8Nn7ctU=; b=CT/3j0b36QEYM3KuLAPD/VqC+dTWDp9vibSebssFyO1CqqOR+onGJGglSxZSS2JhvqP7kPqhehQuW+4DyQin7X5WA1cB7IOCeNqHpfCLMoAWUGGXhmZR/0Bm6u3JE9AFE3zkhR6glsskczsyGRMYxEA71EftDl7KX4I7jdK0Mh3Y1Flua3YV6pBCtQp9iglXmRE5uT5Hckj5ugBIADZ+91Luz5uNvQINLCycGopaCT1LDSDrIij2JHaPdZ9/1L8OmyamzhrnmM8JziEB3AQ3HdtZhUrtE12TJKKaTbEXTIoxVz0WvtfJsWUhMmZkLgUxlr4CP8vMGWL+eEvimiuiUQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by MN6PR11MB8101.namprd11.prod.outlook.com (2603:10b6:208:46e::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6678.24; Thu, 17 Aug 2023 08:46:12 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::9802:65f0:c441:780f]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::9802:65f0:c441:780f%7]) with mapi id 15.20.6678.025; Thu, 17 Aug 2023 08:46:12 +0000 Date: Thu, 17 Aug 2023 09:46:06 +0100 From: Bruce Richardson To: David Marchand CC: Patrick Robb , , , Morten =?iso-8859-1?Q?Br=F8rup?= , Honnappa Nagarahalli , "Ruifeng Wang (Arm Technology China)" , Thomas Monjalon , Aaron Conole Subject: Re: [PATCH v5 04/10] app/test: build using per-file dependency matrix Message-ID: References: <20230721115125.55137-1-bruce.richardson@intel.com> <20230815151053.996469-1-bruce.richardson@intel.com> <20230815151053.996469-5-bruce.richardson@intel.com> Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-ClientProxiedBy: DB3PR06CA0010.eurprd06.prod.outlook.com (2603:10a6:8:1::23) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|MN6PR11MB8101:EE_ X-MS-Office365-Filtering-Correlation-Id: 71e32b11-aeb5-48d1-6ea6-08db9efe6915 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: y2rt/gAnegCiV3j3rSt2tje+zxZsoGK2X0MWy+Tji9U2L55FhJXYDpGi113mYIKU2TIU3LDo6jbFLahs+SOfmNdVIUCy1G/pecd8VZCcd/04sOCl9fQECgYqK06FXCedz8+FNIT+1ij6lRhgIttmNijDYViJD/8hDx2B/JuyJHA/OSBSXJ+xT5Pn9f/pvnwTfz3UbU/sn5PBSXIsfYrGvfFanhrFu9d+5+uF8QABrYRz4t+zr/6mZfQkUAz9x4257D1IrW6u638/BDRFNqcIwwUKCbQmxU/pSRZdDM6rVqsCwK+b2xStj6WvmteAqqYo6u/xfVJ6Xxu1COcuU5DRLiy6LwTVhtPwnxTA/O0DaowBjg1EdU2/1U1qca96SYl3+G4kwoRgxVUX7prFTfx+lj2bTqvi95MhOWEwi9RENJppvmaxDX18w/RV7QxjXOZuLEkjUVcPygJEiaIFDVVkmlY/sLX+HTcwAy3YbOL2ZW4zjSQGX9TIcnKlZFxCQlLB+Jr7WhWHpeQCrSdpmt/6fTbF+0Fm68NcXPEL3Yqvk3bdmv4T9Z7t+Otz98MmRwXp X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(396003)(366004)(136003)(346002)(376002)(39860400002)(1800799009)(451199024)(186009)(2906002)(83380400001)(26005)(86362001)(478600001)(6506007)(6666004)(6486002)(6512007)(53546011)(44832011)(5660300002)(41300700001)(54906003)(316002)(66476007)(66946007)(66556008)(6916009)(4326008)(8676002)(8936002)(82960400001)(38100700002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?c0xPYnR0YzZENTZkbWtGQ3cxbTRiVUZYTmZISnA4aEtKazRNemdoN0U3YUtK?= =?utf-8?B?QUt4VWs3QWNPcmljc1ZaR3lTQ1l0VVVhelQwck1IU0tFQmRERWdQSHlGUkJC?= =?utf-8?B?YW1wTENGTkRlYUNQcnRqY05teWxCZ3ZNZS9nY3NWS29tYWFVTFR6bmlCT3dK?= =?utf-8?B?QUtJN1RzVXNVK1drOTZQT2gxb3M0R3dVQUlGWnIzdDlEbVZaNXNhb0Z0U3VT?= =?utf-8?B?S3dBcmUvRHFsbzFFTFNSbWdqV2cwOWovYjZQazlBYzcyTmErUmhuOERsTGVZ?= =?utf-8?B?WnZOeUlsN2dDUW51SkNHcHdITTM3a3gxRWs1N2NwVnU0YStSNUNqY2ZCZDZn?= =?utf-8?B?VkxrMVE0aGhSZ3hJRkZQL1gzdDhKVDU1VGIrODJiRENOWnJFNTRWc2ltdEEr?= =?utf-8?B?VE1nVi92ZytTRThRK2szdGRySlNNUlQ4aDgyQy85UDhJcUZ0NUd2blJ2eFkw?= =?utf-8?B?L2lnUjA3M2cvdGZocUZKcnZwbnJrRHFqa1B2VEg1Q295YlFqMFVoaWQ1WnFF?= =?utf-8?B?ejR4WThFc2h6WmszczJRM1hZWEFBM1owZ0Y5WGxhSHNUUkJud09maEU2VWRk?= =?utf-8?B?akVLZm9KMDdlYkpUOE1admxFUUdocWN2aTVMM3ErTUQ4b2Q1RU1LYU83OUNx?= =?utf-8?B?cjhpQ1ErVWlCVm9rUEU1VDhFclVzVW54M0VwenBLSE1PUDUrTU9xWFA5NGUx?= =?utf-8?B?UWprSmtLTG5LSDFYeGZvL0UrNk9mM3Q1emt1UTdRNlNjN2dZTXhXUUY0Qmk3?= =?utf-8?B?c1lMK241SllpbisrTlVTV3ZiRDNTTDRCMzhHbVhLTWhHbi81YWtsbnVFMjV2?= =?utf-8?B?cUpBbzBqSkxGd3oxVllGYzBZNEU5cEpYUWE5ZmwyZUdmUFFnWGJNcEVDRmpj?= =?utf-8?B?NnBlQnZqVG5sWEJYMWE2YjY0MmduclJiS3lmRmJnQzVVWVN5aUhiN1BNMGZi?= =?utf-8?B?THluYXFuZXhRWFYwTmprS05abTRibm11SWlIVXMrMGdtY3hGdkFMcE5mQXFC?= =?utf-8?B?S29PbGc1T1ZhNFV4N1BMQ3NjMzFBV3hIME1vUDhqZm9OZDdpbHlxNzRkaTEw?= =?utf-8?B?TzdueE8zRUZMUldQNDJ3WTNxc1YxbndROGZuV3pQaitkcjVRL3lVQUNjSVcy?= =?utf-8?B?cE5ja2pOTllNaStyenJidWJvb1VoWTBKZndmU3VjSDRKNnVweERzemJWc2p2?= =?utf-8?B?alRBNWl5ZXVHa0ZZWU0yakRLYVFDM2NCNmRZdTlUMnZpenYvc0x0M093akxa?= =?utf-8?B?dHltZ25pSU9YZWVnclFQU3E5d1BzenoxSXd2b0hyRVRjWmNNc0RQKzA4eURL?= =?utf-8?B?ZHRLZ0tGR04rWi9PRGRSS0ZVSmptZGg1bVFSRzVHcXVhVm8wcDdqRmJmTzlC?= =?utf-8?B?VDJPbEZBdFNKa3FlV1ZUSEVlRHVNV1B5SE5qSXpNWklML0JoYVVRWkRoODh4?= =?utf-8?B?ZW1NakljL1ZVWWhiMW9RUGdKbEZCUFJjdklsRzJPb3p2NUpCYXV4ZkFrSEND?= =?utf-8?B?TU0zUWkzSkUzcXVoSE5ZWW8vdTlPNHF3VHdleUVYMkdQeC9vb2xyVlFXRGlk?= =?utf-8?B?T2N3RG5DSFlkVkhKTlpSVTQyMndwUVIzKzNHTzNoM2NVY20raWd5bWluc0J2?= =?utf-8?B?eFNlQjNSeHVkMVlzMnJSMS8xK0t0TjQ5Q09uZ3FkaEF0SGZHR2tHN01IY285?= =?utf-8?B?NitJR3FuOEFjOHFWcVlXRXY3dy9kbjFzQVdmRU42cUJad0o5U2RmMy9ERUp4?= =?utf-8?B?SnNvNEVxcjVYOXgrZjVZYVduM0c3RGFOaE9PY1VhRHcvUWtpOHFQWkRScEZI?= =?utf-8?B?eEJkdGRjbnI1akpxbURuaXFuS3RsZ2pteFZVU0JWa1FYditxTWZ5Z3dFL093?= =?utf-8?B?Q2QwWHdldkVtNWI3TzVVUnZxZWdvNkN0Z3hac3V1aHIxNnJBQXY4VjFoTmkr?= =?utf-8?B?N1dOdjR0ajdrUUNWem84MUM1MFY4MlpwR0hkRjd6VXI0Q2xOdi8rWS91SlZQ?= =?utf-8?B?TWhhYnZDclB5SitibG5DL3M0U091UEZSWFI3YTNCV2E2V3dyNEkxaVFZSHBN?= =?utf-8?B?cU5jMmRlbzcyekVmR2tNN2hKNWV2b2pMVkY4NFdnL1NVMFBzSGMzYmRGM2k3?= =?utf-8?B?VFhQbG1BQnJhd2lpUUQ1b0lBR2hOT2ozd1lUOU03V1d1NXRUeEJxL00vR2VE?= =?utf-8?B?NWc9PQ==?= X-MS-Exchange-CrossTenant-Network-Message-Id: 71e32b11-aeb5-48d1-6ea6-08db9efe6915 X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Aug 2023 08:46:12.6741 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: xeHHKDxyc/OndHTss7D2LijAUIvdD5TSCsSMoqBkW4quVsLDc/qc54vdABlE0vbH6ydV0OGD/Irje8b00DOj9MpbAWVVhIYLW8vf01jpeO4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN6PR11MB8101 X-OriginatorOrg: intel.com X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org On Thu, Aug 17, 2023 at 09:27:50AM +0200, David Marchand wrote: > On Wed, Aug 16, 2023 at 8:30 PM Patrick Robb wrote: > >> Patching sources from the test tool is a poor solution. > >> In general, developers won't be aware of source patching and will > >> waste time trying to understand why they can't reproduce what the CI > >> reports (it happened to me with DTS on the interrupt stuff with vhost, > >> at least). > >> > >> For this specific case of skipping a test, if nobody can fix the > >> issue, I prefer if the CI can skip some "known broken in my lab" tests > >> via some meson configuration. > >> And, such configuration should be easy to catch in the test report. > >> > > I strongly agree on all points, which is why I said it was probably a good thing anyhow for us to lose this ability. In the case of the disabled fast-test for arm, that was a new discovery coming from adding new environments, not a regression introduced by a patch, and I don't think it made sense then to block the introduction of the entire unit test coverage for arm while they looked into this issue. If it's possible to introduce meson configure functionality to disable specific tests, that does give us more flexibility. And it's obviously a better process than us doing it at the CI end. > > > > We don't currently patch source in any other way in our CI testing. > > It is possible to list tests from meson. > Why not simply list the existing tests and filter the non working one? > > $ meson test -C --suite fast-tests --list > $ meson test -C --test-args ... > > That requires no change in DPDK. > The small issue with that is that the broken test is hidden from reports. While it unblocks things here without having to change DPDK, I wonder if changing DPDK to support this scenario might be worthwhile. For example, one idea I'm now thinking of is to have the test binary itself look in the environment for a DPDK_TEST_SKIPPED variable, and - when processing test names passed on command-line or environment - report any on that list as SKIPPED. This at least flags in the reports that a test is not running, and the test log gives a reason. Worth doing? Or is adding a new environment with broken tests rare enough we don't need this? Of course, this also seems like a feature request for meson. When running all tests, meson allows you to skip a test suite, but doesn't allow skipping a test within a suite. /Bruce