From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1EC0D423B7; Thu, 12 Jan 2023 15:14:59 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B3D2940E25; Thu, 12 Jan 2023 15:14:58 +0100 (CET) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2074.outbound.protection.outlook.com [40.107.93.74]) by mails.dpdk.org (Postfix) with ESMTP id B8F5540DDC for ; Thu, 12 Jan 2023 15:14:57 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RgeBqSllOcq276GR52KI73HHMoI1vVIFQGDpyJqNiHxY5xNHgdrP3LsTZy1vWlojp6Nuy2gH+s41d7Zg8CbnMiz+yVTxIOFYLkpib0RWE6iyx/fQAhqdu1K6CQVvLabdg/Q2jOuYYVsZj/V4Galqg0/QQnHQ5W31Uk1brOqS+XMNcj2SYdkqjeACVLDNoI2S+6tNUl3TgqwK/55xapK4ctE4KfShFVzCG+ay9a6JYP78w9F0n150GXADvRJ2jk0+IHNyMffkKg6/v45YVDTsY2eiYpbcEW7nK9YITu0U9znN/fLi/RIuRll37wIuhh2pLBtkpTpy7wPITC1TIfRenw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=c4+MVA4mzvYDZPmh4aA4I8zhslMUgCJS2ODC8V53tWo=; b=U8D2UQ+xf2Rr8XP3cDMS4eQIvXoAZgOx44+5ha+yIsK+Crb0do/TwTTunra/MbQ/x2JPFwgMRigsXCTXKxMsIJMFNIyt8PJIyAhAnpUhFW9FGsjc/xA/NJUPtSbn44RKvdk/6vdO6RHS6tFy7FXLs6FtJNYy3ZC3iMhQCuylCgLtFJgL9nLz8X5aJfu/Aexxdj8xnVDN2upxv4Jnja6TTSOJCNxn/Z5PL2TmObCSn37pmA0wG1yysKWdPNA8KN9rlRwSOcJiLBdTMiffqI+/DPHusNkFTunIEApc4igbMnj/o0IF9DWFk+bCEsZ1nFv5v7CkShWfe3jfh/kTOtHDjA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=c4+MVA4mzvYDZPmh4aA4I8zhslMUgCJS2ODC8V53tWo=; b=H+wSdz8QUpR1qjQ0BuJ/5j5+6ombaYjzMsyw+wsDIUJDa/OdV6LxuQtTk7zCsYNtjKDQFfqKrNir2wzoOKCSUK0HQh7otdwGL8RDZvCNZaFppF06TI4MhoxQ752i4BwnP79pxf7iRYdJfZffwHwW4Fs0RZTDa0Lrk6tZ+Zyzll4= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) by PH7PR12MB5593.namprd12.prod.outlook.com (2603:10b6:510:133::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.18; Thu, 12 Jan 2023 14:14:55 +0000 Received: from CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::4807:1f44:5e04:e05a]) by CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::4807:1f44:5e04:e05a%9]) with mapi id 15.20.5986.019; Thu, 12 Jan 2023 14:14:55 +0000 Message-ID: Date: Thu, 12 Jan 2023 14:14:49 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.6.1 Subject: Re: [PATCH v4] devtools: parallelize ABI check Content-Language: en-US To: David Marchand , Thomas Monjalon Cc: dev@dpdk.org, bruce.richardson@intel.com References: <20230107133953.306784-1-thomas@monjalon.net> <20230111195345.1275693-1-thomas@monjalon.net> From: Ferruh Yigit In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-ClientProxiedBy: LO2P123CA0081.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:138::14) To CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR12MB4294:EE_|PH7PR12MB5593:EE_ X-MS-Office365-Filtering-Correlation-Id: 4a54dc58-33ef-43e3-9b9b-08daf4a760ed X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Gv0EfpLDVm0ntTIpwcnVIPnj4sivqe6CORwiKElDfJd5bPK9c3HSVIKiXh4WWj4ikYy6aSjFOXo30fELFjAOp72rcPcrvKFcHNoeLYsC3PQSpZpyi2MRoX3wGA3dSi8MbYnRX/l3Btl+AiO0zHkmpapaIgCIB0UdAN81kYs60OWsiWG1EHmf2fX+wjKueECdzGIe0vRufElSn/ChNx1wRBydfbHlLiqVtIfkjYCqGUDTmEglUq2CjB7UPDrEkrCN5wdKTy2kvH/XgA7YXdjo+/p3s9uQauU7ac9DNMVIzGbVpZhS+B0t1Pf9mvREsaIOf/g/i32XEH/cWbxmZ0ziYCiAI7KW9qG9Kkc6a3iXMBF9cbRSFSQHQYvbTtXm4/CSe0qws646u+UpDqqVi7uupCEqbO6UfdBfcV1tZbtxSm4/84FeFBKUhH/Fm+moOt+XRjx20UMr3GXhqnBeOSx1Fdblq4AInFKoEIE/nE51UiWt3Yd4t63ngeT1t40O8hdSBnmiCNJsHaukQjFBbWty/fohkOL433Lz0lOIBh8uibmTO0c2EnTomBbcJ74eWD0Of1GbVZ0Eg+U9k0WzYMs0sbJ3h3OtBMtMsxkGDHZ3gkLLX3plE5A/8o1x3dgHMN5FzwwxREXUy3RB7V70Kqh1arw3ZqV+SZwPYddRcAOExL3+J8r8irFvJ1A95nwzJ8PMzN92k+BcgYxSaxp/Zg6Z+D4eqZv1QOlMNYRIM+uPRlU= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR12MB4294.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(4636009)(376002)(346002)(39860400002)(136003)(366004)(396003)(451199015)(6506007)(38100700002)(31686004)(6666004)(53546011)(2906002)(6486002)(86362001)(478600001)(2616005)(44832011)(8676002)(186003)(6512007)(316002)(5660300002)(26005)(8936002)(36756003)(83380400001)(41300700001)(31696002)(66556008)(4326008)(110136005)(66946007)(66476007)(45980500001)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bHp0eE9JOHp0RTRFYTZ6UVNYWnd6bktjRkN1d1RCR29ubERibTExdjQrTFZT?= =?utf-8?B?RXFRRU9pRUdSbjQ2ODE1ZlpSa2hZSkUyT0UrL2hiNXNkdmdwRWlNcXNRMGI1?= =?utf-8?B?UVJzZUFTKzR5SkNXZ3oyRUFxY2NEL0IvYjkzL0hIYUtDNmhCSmtTbTUyeXRi?= =?utf-8?B?eDB3OCtHdEY2RDVGalFRM3lCUG5vU29FUjRZZWdZWTdTb1lhYVNxcHVRZG9z?= =?utf-8?B?MCtBNUowdkFjZ0ZieTBpN1M5WmRBVHdRb1BtMEdtSnBvTnhySjZUWVJ3alUv?= =?utf-8?B?YkIvZ0k2RXFidXpLSytVRkw5U1ZaV3JGNHNhRHdpcUJydkw5c1hDYUJ0blBB?= =?utf-8?B?ZFNocWtaZ0ZjL25EamROdVEzTTB4V1JYSUs4MnR4Vk5vRTBLdG01aXQrQjI2?= =?utf-8?B?bFVsYnA1Mjl3aU5xam1iRkIyK1BERTdiQmQ2ZTVJcXdmMmJhN3FES0JHZm84?= =?utf-8?B?bm8vNlgzWjFsSzhKSGZNMDltU3lzSFI5VDVremExM21GMWpNbG5wUDhJbjZn?= =?utf-8?B?VHIzZFhMbGE3S3VXVWFtM2pZdTNFWkVLK2l0M3lieWVZQkFaZzg1VmpZUjdQ?= =?utf-8?B?RFJ4T3pmQlhHWHVlSEs5Y3c1WXFsUWF4eFI0T0EzVjVoUUt2dEdlSGcrOHJy?= =?utf-8?B?WnZvTWlPN2xNT0JtMjJXRE1PbkJJTlQwbEY4aUo1WVJQVjdHV3Jrdms4TjBP?= =?utf-8?B?YVBNbmp1YVNpVWkyc21Mcm5RL21BSGZjNjRPeTVFZE1XQjNCcFRWejQ3c216?= =?utf-8?B?TVlsTXN5L3FuUHc3UVNPZlVtWDFuNE1iZmNSRTRwUzFQcGxyT293MjU1em1t?= =?utf-8?B?bjlKcEpSdXdxWnZLeDh5aGdVMGUrU2drSDcvLzRJa3U5MlAxblVEeVBueDBx?= =?utf-8?B?NVpuZmxrQTREK3UyNFZraUlDeENlZThpSmkzdWdYSkI4czdYSCtNOU9Rb0pK?= =?utf-8?B?MWM2WE12eXNuOFlvemtQNWtWaDF2N3dnZk9YKyt5V2NnTmJVNGplSUhkbWM1?= =?utf-8?B?am0ydXBpYllPZnhxRVBvd3lmTjg3NHRmZm9kRDZuYnNXNWcyTU5Gcjlndm9R?= =?utf-8?B?OGIzSGtSN3ZWanJRekhmcjdQNmpLUWxUT1huNU16VzBUQ2F3YnV4STBjU2xG?= =?utf-8?B?M1orUXBia1UvdkZGcEtRQmhFWmZtNitOK1krc1FSSHV2aG5QMkZyK3kyRkdK?= =?utf-8?B?aU5jcDgxNnBocHYycXVVcjBsclhLa2pROU41MDF4T2RLeCtEdDJmTVhXbjQr?= =?utf-8?B?d0dLZGdmZ3Vwa3ZaUlNGYXNLbXQxZUVhZUg5cTlIQTZpeGE3VXRjbTE4SUVp?= =?utf-8?B?SWhWaGJwT3R5MmxsMlpqdnNaczRjeThWTHJMMm03bG8wRXlGbVlsUGZQNzNt?= =?utf-8?B?Wkw1ZjVhNkFsQ29FSGd3NGZiK29PMGkvQ3JNaGxPbysrUGx6ZmpHRlkwRUhL?= =?utf-8?B?RUtrbnlrZENqU096M2hjUS9QYXNVV09ScitkdGRxZkhXeHNBM1ZKcDFPak55?= =?utf-8?B?MktZdlk0ZFNyS0F2bTE4eDF3RXBEQ2w4RVhQTVZJeTNVeU5OeE5ncmhhNGl6?= =?utf-8?B?Vy8yb0JGc3JZNTNhM3pTd2VUZVJlZytHQ2dsNDlqdHlQa1VIcE9qc0U2U2F5?= =?utf-8?B?aUdhV29WbGJFUHhuTkRuYlorZTVGNEgvMVNpTi81Q2l0TllUdXpSOUViMjRT?= =?utf-8?B?aVY1TmVhdFVLZnVRSDdRYWZqNjNqY3BBdExZcVVEWHBRbUd6MWtMMjRtc09F?= =?utf-8?B?bzgrR285S1BOWkZNWmlja2l6a1M4Tk9YSFJKRmtNdDVZekhiQndoZXdsTGtk?= =?utf-8?B?U0I5NnVpZERhRTdlMHBmUzl0Y01telcrNGFwMU16Y3hqU2hYN0gvbnR1TlF3?= =?utf-8?B?TTRVd3JOckFyY1dNSXV2UjgyUS9qN2lOdmVnMFFxYUZUMDlOd0JoSW8xLzlp?= =?utf-8?B?b2xkcGxxeE1Va1dBMlhxVzA4a0xJdTgyaDUydVZqTlBSYms4OGFuNlJCU2FG?= =?utf-8?B?MU95Uk5CNEx5YUkxRU91QUhVRC9mZytrckp5Z1JUaFNqY0JqMjY4Zmw3MWNt?= =?utf-8?B?TUNXVnNORlgyQjBPL25UUUJEWDhSUmEyVWJ3Y25SY2crNVhyNCtmczRNc201?= =?utf-8?Q?6JvPEEzU+naE1TR1vQ7OG0yyk?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4a54dc58-33ef-43e3-9b9b-08daf4a760ed X-MS-Exchange-CrossTenant-AuthSource: CH2PR12MB4294.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jan 2023 14:14:55.2822 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +UQmZ/uN4n/SohEZRe8uQfhCsKEN8ziJn/kSdNRJdhy/xlNUKepI8O5e+ZQco64N X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5593 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 1/12/2023 10:53 AM, David Marchand wrote: > On Wed, Jan 11, 2023 at 8:53 PM Thomas Monjalon wrote: >> >> Generation and comparison of ABI dumps are done on multiple cores >> thanks to xargs -P0. >> It can accelerate this long step by 5 in my tests. >> >> xargs reports a global error if one of the process has an error. >> >> Running a shell function with xargs requires to export it. >> POSIX shell does not support function export except using an "eval trick". >> Required variables are also exported. >> >> Signed-off-by: Thomas Monjalon >> Tested-by: Ferruh Yigit >> diff --git a/devtools/check-abi.sh b/devtools/check-abi.sh >> index c583eae2fd..31eceb42e6 100755 >> --- a/devtools/check-abi.sh >> +++ b/devtools/check-abi.sh >> @@ -34,20 +34,18 @@ else >> ABIDIFF_OPTIONS="$ABIDIFF_OPTIONS --headers-dir2 $incdir2" >> fi >> >> -error= >> -for dump in $(find $refdir -name "*.dump"); do >> +export newdir ABIDIFF_OPTIONS >> +export diff_func='run_diff() { >> + dump=$1 >> name=$(basename $dump) >> dump2=$(find $newdir -name $name) >> if [ -z "$dump2" ] || [ ! -e "$dump2" ]; then >> echo "Error: cannot find $name in $newdir" >&2 >> - error=1 >> - continue >> - fi >> + return 1 >> + fi; > > No need for ; here. > This can be fixed when applying (I tested both your patch and with > this small fix). > > >> abidiff $ABIDIFF_OPTIONS $dump $dump2 || { >> abiret=$? >> - echo "Error: ABI issue reported for 'abidiff $ABIDIFF_OPTIONS $dump $dump2'" >&2 >> - error=1 >> - echo >> + echo "Error: ABI issue reported for abidiff $ABIDIFF_OPTIONS $dump $dump2" >&2 >> if [ $(($abiret & 3)) -ne 0 ]; then >> echo "ABIDIFF_ERROR|ABIDIFF_USAGE_ERROR, this could be a script or environment issue." >&2 >> fi >> @@ -57,8 +55,13 @@ for dump in $(find $refdir -name "*.dump"); do >> if [ $(($abiret & 8)) -ne 0 ]; then >> echo "ABIDIFF_ABI_INCOMPATIBLE_CHANGE, this change breaks the ABI." >&2 >> fi >> - echo >> + return 1 >> } >> -done >> +}' >> + >> +error= >> +find $refdir -name "*.dump" | >> +xargs -n1 -P0 sh -c 'eval "$diff_func"; run_diff $0' || >> +error=1 >> >> [ -z "$error" ] || [ -n "$warnonly" ] > > For the record, on my system, calling this script is ~5 times faster: > - before > real 0m5,447s > user 0m4,497s > sys 0m0,937s > > - after > real 0m1,202s > user 0m10,784s > sys 0m2,027s > > >> diff --git a/devtools/gen-abi.sh b/devtools/gen-abi.sh >> index f15a3b9aaf..61f7510ea1 100755 >> --- a/devtools/gen-abi.sh >> +++ b/devtools/gen-abi.sh >> @@ -22,5 +22,6 @@ for f in $(find $installdir -name "*.so.*"); do >> fi >> >> libname=$(basename $f) >> - abidw --out-file $dumpdir/${libname%.so*}.dump $f >> -done >> + echo $dumpdir/${libname%.so*}.dump $f >> +done | >> +xargs -n2 -P0 abidw --out-file >> -- >> 2.39.0 >> > > - before > real 0m8,237s > user 0m7,704s > sys 0m0,504s > > - after > real 0m2,517s > user 0m14,145s > sys 0m0,766s > > > Ferruh, I am seeing quite different numbers for running those scripts > (clearly not of the minute order). > I switched to testing/building in tmpfs some time ago. > It requires a good amount of memory (I empirically allocated 40G), but > maybe worth a try for you? > I run 'test-meson-builds.sh' script directly and yes I am getting different numbers although there is still improvement, not in scale with what you are getting, with v4 I have following: - before real 10m3.248s user 39m8.664s sys 14m52.870s - after real 7m49.086s user 39m59.507s sys 15m0.598s What I am running exactly is: time DPDK_ABI_REF_VERSION=v22.11.1 DPDK_ABI_REF_DIR=/tmp/dpdk-abiref DPDK_ABI_REF_SRC=.../dpdk-stable/ ./devtools/test-meson-builds.sh > > In any case, this patch lgtm. > Acked-by: David Marchand > >