From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 75ABA423B5; Thu, 12 Jan 2023 11:54:02 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 62D9240A7D; Thu, 12 Jan 2023 11:54:02 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 6FEAB40156 for ; Thu, 12 Jan 2023 11:54:00 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1673520840; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=2J2y/JAZhKIawps6O4jQ3czU9qvalDkgJ4ATGOzgZvE=; b=OZqt5Vkp7MQvyIllKcZylu5fs7BUVtVJh3CPzyFH2onIVKGspWic1BNMUhHynw0Mem4ZFD /w9XlAfuBAmecNcerACFQHR/ftfKbTfS4T4LFOkMdE9Y15SaNC233VbuS++tbr0DK2iWwm zdWSWg2564X+VN/19Xy9kNzZtltqtlA= Received: from mail-pj1-f69.google.com (mail-pj1-f69.google.com [209.85.216.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-187-B4NVuu6VMtuyNP82jyPThg-1; Thu, 12 Jan 2023 05:53:58 -0500 X-MC-Unique: B4NVuu6VMtuyNP82jyPThg-1 Received: by mail-pj1-f69.google.com with SMTP id h12-20020a17090a604c00b00225b2dbe4cfso8073638pjm.1 for ; Thu, 12 Jan 2023 02:53:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=2J2y/JAZhKIawps6O4jQ3czU9qvalDkgJ4ATGOzgZvE=; b=GcM+qBBTUk2fs3X1gPz2mNOD4IZ8Dxr6oqYIqubmyeKeitf+xOGxA+KAi7MgFUtz5I qu8P1GMA7ibKRjnVbDSc5kLYNl5mdfufuIJGfl8A5slyByXA1fssatB7BBmOmXo59E1s wTwf6u5E2NpEtNxw+t39PsPa8uUTI0RwatN9x7w7n5J/qKoWkwMnHTQpI3TIRRJizZIi pBSPsgs0ZnPiyTr2ciQMTmi5wutX5NicK1bMyNylt06s5xs7UB8c46BXuN3c95ZiPkFT fn1pUToiRBNx0XvrE23N04+4YTCuZd9J2lOSOekowfPCsgIuRkz9t7kzo0PU+DhSxQxK opvA== X-Gm-Message-State: AFqh2krb3+1bmPV8xLajOTsyD/h1ytIxQAV8ZK6f17qUWn00PqrPLN17 PH98KVXYuJJRUlPirSD5MZ+TgYmG1vwpWepQfYkJJG7LDuvMXRKzsOyHI49Y7U9YLbZKDoyMf7O TtEmUHlLAf9zpoK4HbVg= X-Received: by 2002:a17:90a:4f4f:b0:229:cf3:5348 with SMTP id w15-20020a17090a4f4f00b002290cf35348mr57077pjl.30.1673520837768; Thu, 12 Jan 2023 02:53:57 -0800 (PST) X-Google-Smtp-Source: AMrXdXv/cvIe4RunOy7O3g5/mgWkRHoKKm2JSwloyo7EMG3IeOULIAdaJB+Wi1oeAACVKM83YFQyuE5V+EXAtHkUnEo= X-Received: by 2002:a17:90a:4f4f:b0:229:cf3:5348 with SMTP id w15-20020a17090a4f4f00b002290cf35348mr57072pjl.30.1673520837438; Thu, 12 Jan 2023 02:53:57 -0800 (PST) MIME-Version: 1.0 References: <20230107133953.306784-1-thomas@monjalon.net> <20230111195345.1275693-1-thomas@monjalon.net> In-Reply-To: <20230111195345.1275693-1-thomas@monjalon.net> From: David Marchand Date: Thu, 12 Jan 2023 11:53:46 +0100 Message-ID: Subject: Re: [PATCH v4] devtools: parallelize ABI check To: Thomas Monjalon , Ferruh Yigit Cc: dev@dpdk.org, bruce.richardson@intel.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Wed, Jan 11, 2023 at 8:53 PM Thomas Monjalon wrote: > > Generation and comparison of ABI dumps are done on multiple cores > thanks to xargs -P0. > It can accelerate this long step by 5 in my tests. > > xargs reports a global error if one of the process has an error. > > Running a shell function with xargs requires to export it. > POSIX shell does not support function export except using an "eval trick". > Required variables are also exported. > > Signed-off-by: Thomas Monjalon > Tested-by: Ferruh Yigit > diff --git a/devtools/check-abi.sh b/devtools/check-abi.sh > index c583eae2fd..31eceb42e6 100755 > --- a/devtools/check-abi.sh > +++ b/devtools/check-abi.sh > @@ -34,20 +34,18 @@ else > ABIDIFF_OPTIONS="$ABIDIFF_OPTIONS --headers-dir2 $incdir2" > fi > > -error= > -for dump in $(find $refdir -name "*.dump"); do > +export newdir ABIDIFF_OPTIONS > +export diff_func='run_diff() { > + dump=$1 > name=$(basename $dump) > dump2=$(find $newdir -name $name) > if [ -z "$dump2" ] || [ ! -e "$dump2" ]; then > echo "Error: cannot find $name in $newdir" >&2 > - error=1 > - continue > - fi > + return 1 > + fi; No need for ; here. This can be fixed when applying (I tested both your patch and with this small fix). > abidiff $ABIDIFF_OPTIONS $dump $dump2 || { > abiret=$? > - echo "Error: ABI issue reported for 'abidiff $ABIDIFF_OPTIONS $dump $dump2'" >&2 > - error=1 > - echo > + echo "Error: ABI issue reported for abidiff $ABIDIFF_OPTIONS $dump $dump2" >&2 > if [ $(($abiret & 3)) -ne 0 ]; then > echo "ABIDIFF_ERROR|ABIDIFF_USAGE_ERROR, this could be a script or environment issue." >&2 > fi > @@ -57,8 +55,13 @@ for dump in $(find $refdir -name "*.dump"); do > if [ $(($abiret & 8)) -ne 0 ]; then > echo "ABIDIFF_ABI_INCOMPATIBLE_CHANGE, this change breaks the ABI." >&2 > fi > - echo > + return 1 > } > -done > +}' > + > +error= > +find $refdir -name "*.dump" | > +xargs -n1 -P0 sh -c 'eval "$diff_func"; run_diff $0' || > +error=1 > > [ -z "$error" ] || [ -n "$warnonly" ] For the record, on my system, calling this script is ~5 times faster: - before real 0m5,447s user 0m4,497s sys 0m0,937s - after real 0m1,202s user 0m10,784s sys 0m2,027s > diff --git a/devtools/gen-abi.sh b/devtools/gen-abi.sh > index f15a3b9aaf..61f7510ea1 100755 > --- a/devtools/gen-abi.sh > +++ b/devtools/gen-abi.sh > @@ -22,5 +22,6 @@ for f in $(find $installdir -name "*.so.*"); do > fi > > libname=$(basename $f) > - abidw --out-file $dumpdir/${libname%.so*}.dump $f > -done > + echo $dumpdir/${libname%.so*}.dump $f > +done | > +xargs -n2 -P0 abidw --out-file > -- > 2.39.0 > - before real 0m8,237s user 0m7,704s sys 0m0,504s - after real 0m2,517s user 0m14,145s sys 0m0,766s Ferruh, I am seeing quite different numbers for running those scripts (clearly not of the minute order). I switched to testing/building in tmpfs some time ago. It requires a good amount of memory (I empirically allocated 40G), but maybe worth a try for you? In any case, this patch lgtm. Acked-by: David Marchand -- David Marchand