From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F1EA74240D; Wed, 18 Jan 2023 12:43:20 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D2F54400D6; Wed, 18 Jan 2023 12:43:20 +0100 (CET) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2063.outbound.protection.outlook.com [40.107.244.63]) by mails.dpdk.org (Postfix) with ESMTP id CDAFE4003F for ; Wed, 18 Jan 2023 12:43:19 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CH6GRC0Osdx+SLjuIGgAB5UhNoe8hffdRLOt/GX6IKyCBTVArthsiq6lViTGBwlmautXzdjSz/gRirwuVWytJv5ede9eVyUAjX9gDnNxvh5LGwjCbv+FB4ShdzwZePxNjltZXAl8xL2dX1p1B8AKRkrsk3VL/VvV+D6nb/jsviCFB+/G7vHAKb+Zt5xkdJiuVd2AfOvTMTQxS7CJPu2EQbie6byCsy5E5AmJB9Jk+j5CIgPSQqgp1smlYrDEwHrDbt/a4pyrgyi9BIyjMbT8y0v0NIFxGWn6nGrFTn8qc02jrYqMw7YGSfrM71aTwrkO8QyI+khWnio3zeS0TBzJ0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iSkb7woWf12uPwQ9gKOS1VldL/EbjM0p9kq02PUQ8Xs=; b=BtGlwI0XU2xXhXxMn9ytw3MfG2MHvBCasWrsfwKBc/yHxGSFCpOiKk3zVTx/DFdLZxex9a6y/gPseSnuDe/qxSN9nFGOxObcTADgKaFO2qw2i7bC9nBNMbiWz2NVTmFSuofaLlsSFF5sb8x8jp688egBUbgwp9R+ISHSa0BDbW1T1P1bNCKmEVcvgMp9ppN3KXEGZxa+A/FTtsDxs8p8It5pTA754MJQhRgeBiElG29h1aa/X3w98jBELvTI2iRcfUrcEbaQC3Vaxe0FH7QYa1fKFXYS52KvmTZ0BzO3ROSF0G2u2tfJjkEiN9zV/YsgkotSrcbduFtM93eccwhuyg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iSkb7woWf12uPwQ9gKOS1VldL/EbjM0p9kq02PUQ8Xs=; b=To/ce3UwuOMV7GP77Xb0vnLbPKGG+PJPKtlOymR0khW/K4dCnaJexekJLwV69MoROVzURyConkJpptWeIYZ6owqYyBREizAxLiqTdFXDvJdxEPBHtWsKtkEU/4a+GAos8w8MzKB04J77Xa75RnaVH31ZKiuezPddhv6pkq+6RbY= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) by DM4PR12MB6565.namprd12.prod.outlook.com (2603:10b6:8:8c::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.13; Wed, 18 Jan 2023 11:43:17 +0000 Received: from CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::4807:1f44:5e04:e05a]) by CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::4807:1f44:5e04:e05a%8]) with mapi id 15.20.6002.024; Wed, 18 Jan 2023 11:43:17 +0000 Message-ID: <0244668c-0ec3-00e6-30e2-ccd6aae0584d@amd.com> Date: Wed, 18 Jan 2023 11:43:11 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.6.1 Subject: Re: [PATCH v4] devtools: parallelize ABI check Content-Language: en-US To: David Marchand Cc: Thomas Monjalon , dev@dpdk.org, bruce.richardson@intel.com References: <20230107133953.306784-1-thomas@monjalon.net> <20230111195345.1275693-1-thomas@monjalon.net> From: Ferruh Yigit In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-ClientProxiedBy: LO4P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:2c3::17) To CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR12MB4294:EE_|DM4PR12MB6565:EE_ X-MS-Office365-Filtering-Correlation-Id: c3103a36-8f64-40fd-21b1-08daf94930d1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 9Hz9IbTq2kLS6KUMMM607kUS7LM2fxwKnVvbTJoU2tBFbqx81Ko3TAMn/wGx4JkX8DhCHqrlwLfQMiiIQp3Dv6mKMgIgRgSRnMoU7gcr64u/7vD9dhdwPW91EZtZxkxrucxia+PqFXLiGwl6krN4zmbTLMUHi3imX7YHTwnQc1/gMScIgW8QeSJGvdKK/WibBOdae20n+45qijFGyXbOvZVPilQeMyeb/ZRt1HaLcl+ap8/AxgHg+POM04ZeqbskXTu+DBAdAdBCrVc5CDlTltdTgzwn0rwfFCoHRWFjPjqn2S+LS1rsWUO0UnUbgcMgOxKsoU11iFU6Jw2IDT2j7hhfpnbYuhqm1UC454pWN0xxvZrogEYKSDD29qvkHofptzVGKn//HKMacpimd7Xd1dGpWBe5c2PhS5nZCE5wfjR9dA0JNJ/5ictt2M3dIAlhx1Sk2+94vclYFs4CsbJoe5Xwtboi1+AtHNk7OKeyA5c/+Cgd1E6vk/oOUeZ5qNHPRPlLHGOVCWBTDRP09V24d3MKpNwx0EXrqyDWXXsbVaPvdfETMv5qW/SDwHXHqL6TESLjdAqJ1SGjZT8XNuxj9LbG/FERc+P5mPacREjfBglL0S8CHoQQ0sHguwFwH/kdL60EcZfFJ6dVOBwnPxdHU0NXumiYG3/s/eNVr6xri/jwcpcxrsTSyyQdZ9I+puIUtDU12Xa/t48hpeCdu/x4LCzodnOIzlQqfGC2p0CqXoM= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR12MB4294.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(4636009)(39860400002)(366004)(346002)(376002)(396003)(136003)(451199015)(478600001)(6666004)(31696002)(53546011)(6486002)(36756003)(186003)(26005)(6506007)(6512007)(31686004)(86362001)(66476007)(4326008)(316002)(8676002)(6916009)(2616005)(66556008)(66946007)(8936002)(41300700001)(38100700002)(83380400001)(5660300002)(44832011)(2906002)(45980500001)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MEZBZDBoaGRudUUrSEVPTDRNbGcyRDZwcnExZlpyVm1DMDZwaTJzWmtyMlVW?= =?utf-8?B?WEpkQjdmNmNDZmhMb3NjTEtqYkk3WitvVFFLWnoyQWtjMnVKbVlqc0U1RXZN?= =?utf-8?B?YXJQVi9Wb2JlbzhXMFRRVGp3OEdaWkFIQWdrK0ZuRnl6S3ZWK2gyOHBrK1dR?= =?utf-8?B?YWlGd09CYXNkNE9OVEVJRWd1alFoMmZXcVpGZmxiNTFpdmlvVmU5aWptSTFu?= =?utf-8?B?Uk1zcHhZQm90N2RVRnJEb01BNFpzQ2pRbHhNNlRTUDJ2djhib0drZlVYVUIy?= =?utf-8?B?YjBTb3pTV2UrbmM1cVNhaTdNT05hTWt1QW8zUWZTWndwQ1N3dXNLcTVLdHUx?= =?utf-8?B?YlRCazhvbjF0QkUxZGJiUlVNNzlBdTBTbnhBV0QxWmd6UUM1clIrMVRkY2o4?= =?utf-8?B?RVdZeVlwbHoyaFlHcTMyb0k5dFpyVUEvZkMyNk04TkVwTFNRNjlEclJxVVdM?= =?utf-8?B?RnVQa0xQUGlnVjhwSCtuVVIwZCt3TWYrLzMzMFByY3E2elN3R0V6VjhLNkYy?= =?utf-8?B?WTlhTW1yRUZtOS9RUFVhTisyN3dDK2M0RWRmWEtLdndvU1FxYllDZVMwSHJ0?= =?utf-8?B?UmZLNzZiTzVzMTZ4WnplT2xSZXRVT0h4QnpMb1pMZmlWTVo3eFdBbkRlSzhs?= =?utf-8?B?YlNsYkprV0RMNVRNRGhaQkFFNTZZQ0JadXhTbWdvU25IK1NuR3FsSGUvZ3Mz?= =?utf-8?B?dXNoMm5nWlRoNk9CN01Xc3hHRFF0dCtGWW9oTjZFMUdEVHRnRlV2Y3BxUXMw?= =?utf-8?B?OFg5bnBzYlQzdU5mV056UWI2K3lDTmUxQitMNitqelY5VDVwb1J6U2F1N0lX?= =?utf-8?B?dldaYWZTa29PRnlzdmJ6aU5LVVcrWnZycjlPL2RCYUNEU3dRbDlnZFVTQWlV?= =?utf-8?B?NEZ4SlJlakk0ZktkVlhORnF0MTBiOFhTTXQrNFg1T2p5dW8zUGR1QmRJTGNz?= =?utf-8?B?eWUzYjVubEtNbEpMd2NEa1FjZzRSWjBkRVFyUHJ5QTdXWDVwOUZSNTB1UHlP?= =?utf-8?B?Y3pQdVBwb21iWkJycm5kcUF6ZENza2tJcU5SVENRMXQvaHphc0NYanBpZzc4?= =?utf-8?B?V3g3bHhySTM1cGpBSjRFUEc2VlpTQm9tY205QndXUjZkNXVZQ294YjlQOE1O?= =?utf-8?B?SmRRUE1TVkV1cjZ3UGRtRFJTWDdBeWczaUdQQjI4ejFoTERwcXhyN1pYUGda?= =?utf-8?B?NTFuRFRtNk1RN3NvckFsY2JIc1dxQnNKTjVDZ3Y1L3ZFT1VRYWxoaDUwSVlK?= =?utf-8?B?S0dYSzRpaDg4R295YkRTUWZ3dUVNTGxsUTk0bTJsZyszaDNzMU1wbGxmenVH?= =?utf-8?B?Rm0zR21BS2dQMlVKZGtBMllMZksxcCtRN29kd00ramxLR3pGMUxGaFNnM28v?= =?utf-8?B?ME83QmpyRVl5R2wrZWxpUFBTdWdpbDhTZHBPS2dEMFU0a1c1Ykd6bnpZTnp3?= =?utf-8?B?M2c4QWVPTHJGbnM1S0JtU1ZlWnU2UWpnMy9Ma0M0SnlqczB3SkxDb3c3TitJ?= =?utf-8?B?SUVpQXNvVVYrbzNnS2hMemZTVENGdWNiK0xiZE44WllFZDdJMmRjeGpEKzJv?= =?utf-8?B?VDg1bitqem9BcFovejd0b1R1UDFxZVNVTDBlZFBqckRpTEs4dU9kU2hWZWxJ?= =?utf-8?B?eTJYYkNaMWlxbjAwVE1pMllCbUpEYVZZWDhhTktXQU9ETVlZWjkxZ2hiRzRj?= =?utf-8?B?cEZLV3g2QVRmTG1Od2hGWkdRbEQwZW1TU3ZwMndzQjZtS2hjTWMrZHhGL1Rh?= =?utf-8?B?V3JRdG9Xb0xNWXZLYldDaG0waG9VQURzNzQ2Tjh5SU9IM3JkeVZKaXV0YU9H?= =?utf-8?B?UkQ1SnBBTUhYa0RvMjBFRHltUjMxeENJVW1aK0ZPdnRLWTl1TWhKejdUcVZm?= =?utf-8?B?bkovWE5SVGo1S2UvVTRUb2dIVDllcHlVc1JyUHdtU28wUWlvQU53Z3hoUW4v?= =?utf-8?B?dW54aGhBREdXNjlhMzM4dnhBSFRQeDA5Z2xHNXpkVStmK29ZVE1jZ0xBTXJq?= =?utf-8?B?QXZWa016ZzNra1VSVjcvTXJuQjRpdm5Ka3p3VjJJMHVUTmNyUEdFcEs3ZERj?= =?utf-8?B?cTZwSFdaVjFnakk4MGVEWFRXQjUyQ3JpNHN0WWdTRm1KOXExNnVnb08zNzlZ?= =?utf-8?Q?5NRJksCRqqKnv6qoBR9DEA1h4?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: c3103a36-8f64-40fd-21b1-08daf94930d1 X-MS-Exchange-CrossTenant-AuthSource: CH2PR12MB4294.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jan 2023 11:43:17.5943 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /L+LthQmemRXxRufVr4or1QqNpwzm8ykyLj6EKgrmUefTtBllYdmeG8906ZSatmz X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6565 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 1/18/2023 10:45 AM, David Marchand wrote: > On Thu, Jan 12, 2023 at 3:15 PM Ferruh Yigit wrote: >>>> abidiff $ABIDIFF_OPTIONS $dump $dump2 || { >>>> abiret=$? >>>> - echo "Error: ABI issue reported for 'abidiff $ABIDIFF_OPTIONS $dump $dump2'" >&2 >>>> - error=1 >>>> - echo >>>> + echo "Error: ABI issue reported for abidiff $ABIDIFF_OPTIONS $dump $dump2" >&2 >>>> if [ $(($abiret & 3)) -ne 0 ]; then >>>> echo "ABIDIFF_ERROR|ABIDIFF_USAGE_ERROR, this could be a script or environment issue." >&2 >>>> fi >>>> @@ -57,8 +55,13 @@ for dump in $(find $refdir -name "*.dump"); do >>>> if [ $(($abiret & 8)) -ne 0 ]; then >>>> echo "ABIDIFF_ABI_INCOMPATIBLE_CHANGE, this change breaks the ABI." >&2 >>>> fi >>>> - echo >>>> + return 1 >>>> } >>>> -done >>>> +}' >>>> + >>>> +error= >>>> +find $refdir -name "*.dump" | >>>> +xargs -n1 -P0 sh -c 'eval "$diff_func"; run_diff $0' || >>>> +error=1 >>>> >>>> [ -z "$error" ] || [ -n "$warnonly" ] >>> >>> For the record, on my system, calling this script is ~5 times faster: >>> - before >>> real 0m5,447s >>> user 0m4,497s >>> sys 0m0,937s >>> >>> - after >>> real 0m1,202s >>> user 0m10,784s >>> sys 0m2,027s >>> >>> >>>> diff --git a/devtools/gen-abi.sh b/devtools/gen-abi.sh >>>> index f15a3b9aaf..61f7510ea1 100755 >>>> --- a/devtools/gen-abi.sh >>>> +++ b/devtools/gen-abi.sh >>>> @@ -22,5 +22,6 @@ for f in $(find $installdir -name "*.so.*"); do >>>> fi >>>> >>>> libname=$(basename $f) >>>> - abidw --out-file $dumpdir/${libname%.so*}.dump $f >>>> -done >>>> + echo $dumpdir/${libname%.so*}.dump $f >>>> +done | >>>> +xargs -n2 -P0 abidw --out-file >>>> -- >>>> 2.39.0 >>>> >>> >>> - before >>> real 0m8,237s >>> user 0m7,704s >>> sys 0m0,504s >>> >>> - after >>> real 0m2,517s >>> user 0m14,145s >>> sys 0m0,766s >>> >>> >>> Ferruh, I am seeing quite different numbers for running those scripts >>> (clearly not of the minute order). >>> I switched to testing/building in tmpfs some time ago. >>> It requires a good amount of memory (I empirically allocated 40G), but >>> maybe worth a try for you? >>> >> >> I run 'test-meson-builds.sh' script directly and yes I am getting >> different numbers although there is still improvement, not in scale with >> what you are getting, with v4 I have following: >> >> - before >> real 10m3.248s >> user 39m8.664s >> sys 14m52.870s >> >> - after >> real 7m49.086s >> user 39m59.507s >> sys 15m0.598s > > Well, yes, I did not realise which apples you were looking at :-). > The change looks good in any case. > Ack