From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp.tuxdriver.com (charlotte.tuxdriver.com [70.61.120.58]) by dpdk.org (Postfix) with ESMTP id E59AFAABA for ; Tue, 5 Jun 2018 14:23:02 +0200 (CEST) Received: from cpe-2606-a000-111b-40b7-640c-26a-4e16-9225.dyn6.twc.com ([2606:a000:111b:40b7:640c:26a:4e16:9225] helo=hmswarspite.think-freely.org) by smtp.tuxdriver.com with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1fQAz8-0006kY-U1; Tue, 05 Jun 2018 08:22:57 -0400 Received: from hmswarspite.think-freely.org (localhost [127.0.0.1]) by hmswarspite.think-freely.org (8.15.2/8.15.2) with ESMTP id w55CMDG1021134; Tue, 5 Jun 2018 08:22:13 -0400 Received: (from nhorman@localhost) by hmswarspite.think-freely.org (8.15.2/8.15.2/Submit) id w55CMBQt021133; Tue, 5 Jun 2018 08:22:11 -0400 From: Neil Horman To: dev@dpdk.org Cc: Neil Horman , thomas@monjalon.net, john.mcnamara@intel.com, bruce.richardson@intel.com, Ferruh Yigit , Stephen Hemminger Date: Tue, 5 Jun 2018 08:21:58 -0400 Message-Id: <20180605122158.21085-1-nhorman@tuxdriver.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180115190545.25687-1-nhorman@tuxdriver.com> References: <20180115190545.25687-1-nhorman@tuxdriver.com> X-Spam-Score: -2.9 (--) X-Spam-Status: No Subject: [dpdk-dev] [PATCH v7] checkpatches.sh: Add checks for ABI symbol addition X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jun 2018 12:23:03 -0000 Recently, some additional patches were added to allow for programmatic marking of C symbols as experimental. The addition of these markers is dependent on the manual addition of exported symbols to the EXPERIMENTAL section of the corresponding libraries version map file. The consensus on review is that, in addition to mandating the addition of symbols to the EXPERIMENTAL version in the map, we need a mechanism to enforce our documented process of mandating that addition when they are introduced. To that end, I am proposing this change. It is an addition to the checkpatches script, which scan incoming patches for additions and removals of symbols to the map file, and warns the user appropriately Signed-off-by: Neil Horman CC: thomas@monjalon.net CC: john.mcnamara@intel.com CC: bruce.richardson@intel.com CC: Ferruh Yigit CC: Stephen Hemminger --- Change notes v2) * Cleaned up and documented awk script (shemminger) * fixed sort/uniq usage (shemminger) * moved checking to new script (tmonjalon) * added maintainer entry (tmonjalon) * added license (tmonjalon) v3) * Changed symbol check script name (tmonjalon) * Trapped exit to clean temp file (tmonjalon) * Honored verbose command (tmonjalon) * Cleaned left over debug bits (tmonjalon) * Updated location in MAINTAINERS file (tmonjalon) v4) * Updated maintainers file (tmonjalon) v5) * undo V4 (tmojalon) v6) * Cleaning up more nits (tmonjalon) * Combining some lines (tmonjalon) * Fixing error print condition (tmonjalon) * Redirect stdin to a file to allow rewinding for Multiple passes on tools (nhorman) v7) * More nits (tmonjalon) * consoloidating some common report lines (tmonjalon) * move SPDX identifier to line 2 (nhorman) * fix some checkpatch errors --- MAINTAINERS | 1 + devtools/check-symbol-change.sh | 145 ++++++++++++++++++++++++++++++++ devtools/checkpatches.sh | 50 +++++++++-- 3 files changed, 189 insertions(+), 7 deletions(-) create mode 100755 devtools/check-symbol-change.sh diff --git a/MAINTAINERS b/MAINTAINERS index 7398749d7..da4735c11 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -122,6 +122,7 @@ M: Neil Horman F: lib/librte_compat/ F: doc/guides/rel_notes/deprecation.rst F: devtools/validate-abi.sh +F: devtools/check-symbol-change.sh F: buildtools/check-experimental-syms.sh Driver information diff --git a/devtools/check-symbol-change.sh b/devtools/check-symbol-change.sh new file mode 100755 index 000000000..76d5562b6 --- /dev/null +++ b/devtools/check-symbol-change.sh @@ -0,0 +1,145 @@ +#!/bin/sh +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2018 Neil Horman + +build_map_changes() +{ + local fname=$1 + local mapdb=$2 + + cat $fname | filterdiff -i *.map | awk ' + # Initialize our variables + BEGIN {map="";sym="";ar="";sec=""; in_sec=0} + + # Anything that starts with + or -, followed by an a + # and ends in the string .map is the name of our map file + # This may appear multiple times in a patch if multiple + # map files are altered, and all section/symbol names + # appearing between a triggering of this rule and the + # next trigger of this rule are associated with this file + /[-+] a\/.*\.map/ {map=$2} + + # Triggering this rule, which starts a line with a + and ends it + # with a { identifies a versioned section. The section name is + # the rest of the line with the + and { symbols remvoed. + # Triggering this rule sets in_sec to 1, which actives the + # symbol rule below + /+.*{/ {gsub("+","");sec=$1; in_sec=1} + + # This rule idenfies the end of a section, and disables the + # symbol rule + /.*}/ {in_sec=0} + + # This rule matches on a + followed by any characters except a : + # (which denotes a global vs local segment), and ends with a ;. + # The semicolon is removed and the symbol is printed with its + # association file name and version section, along with an + # indicator that the symbol is a new addition. Note this rule + # only works if we have found a version section in the rule + # above (hence the in_sec check). Otherwise we flag it as an + # unknown section + /^+[^}].*[^:*];/ {gsub(";","");sym=$2; + if (in_sec == 1) { + print map " " sym " " sec " add" + } else { + print map " " sym " unknown add" + } + } + + # This is the same rule as above, but the rule matches on a + # leading - rather than a +, denoting that the symbol is being + # removed. + /^-[^}].*[^:*];/ {gsub(";","");sym=$2; + if (in_sec == 1) { + print map " " sym " " sec " del" + } else { + print map " " sym " unknown del" + } + }' > ./$mapdb + + sort -u $mapdb > ./$mapdb.2 + mv -f $mapdb.2 $mapdb + +} + +check_for_rule_violations() +{ + local mapdb=$1 + local mname + local symname + local secname + local ar + local ret=0 + + while read mname symname secname ar + do + if [ "$ar" == "add" ] + then + + if [ "$secname" == "unknown" ] + then + # Just inform the user of this occurrence, but + # don't flag it as an error + echo -n "INFO: symbol $syname is added but " + echo -n "patch has insuficient context " + echo -n "to determine the section name " + echo -n "please ensure the version is " + echo "EXPERIMENTAL" + continue + fi + + if [ "$secname" != "EXPERIMENTAL" ] + then + # Symbols that are getting added in a section + # other ithan the experimental section + # to be moving from an already supported + # section or its a violation + grep -q \ + "$mname $symname [^EXPERIMENTAL] del" $mapdb + if [ $? -ne 0 ] + then + echo -n "ERROR: symbol $symname " + echo -n "is added in a section " + echo -n "other than the EXPERIMENTAL " + echo "section of the version map" + ret=1 + fi + fi + else + + if [ "$secname" != "EXPERIMENTAL" ] + then + # Just inform users that non-experimenal + # symbols need to go through a deprecation + # process + echo -n "INFO: symbol $symname is being " + echo -n "removed, ensure that it has " + echo "gone through the deprecation process" + fi + fi + done < $mapdb + + return $ret +} + +trap clean_and_exit_on_sig EXIT + +mapfile=`mktemp mapdb.XXXXXX` +patch=$1 +exit_code=1 + +clean_and_exit_on_sig() +{ + rm -f $mapfile + exit $exit_code +} + +build_map_changes $patch $mapfile +check_for_rule_violations $mapfile +exit_code=$? + +rm -f $mapfile + +exit $exit_code + + diff --git a/devtools/checkpatches.sh b/devtools/checkpatches.sh index 245d5ab10..2e9f30cc8 100755 --- a/devtools/checkpatches.sh +++ b/devtools/checkpatches.sh @@ -7,6 +7,9 @@ # - DPDK_CHECKPATCH_LINE_LENGTH . $(dirname $(readlink -e $0))/load-devel-config + +VALIDATE_NEW_API=$(dirname $(readlink -e $0))/check-symbol-change.sh + length=${DPDK_CHECKPATCH_LINE_LENGTH:-80} # override default Linux options @@ -21,6 +24,15 @@ SPLIT_STRING,LONG_LINE_STRING,\ LINE_SPACING,PARENTHESIS_ALIGNMENT,NETWORKING_BLOCK_COMMENT_STYLE,\ NEW_TYPEDEFS,COMPARISON_TO_NULL" +clean_tmp_files() { + echo $TMPINPUT | grep -q checkpaches + if [ $? -eq 0 ]; then + rm -f $TMPINPUT + fi +} + +trap "clean_tmp_files" SIGINT + print_usage () { cat <<- END_OF_HELP usage: $(basename $0) [-q] [-v] [-nX|patch1 [patch2] ...]] @@ -58,19 +70,43 @@ total=0 status=0 check () { # + local ret=0 + total=$(($total + 1)) ! $verbose || printf '\n### %s\n\n' "$3" if [ -n "$1" ] ; then - report=$($DPDK_CHECKPATCH_PATH $options "$1" 2>/dev/null) + TMPINPUT=$1 elif [ -n "$2" ] ; then - report=$(git format-patch --find-renames --no-stat --stdout -1 $commit | - $DPDK_CHECKPATCH_PATH $options - 2>/dev/null) + TMPINPUT=$(mktemp checkpatches.XXXXXX) + git format-patch --find-renames \ + --no-stat --stdout -1 $commit > ./$TMPINPUT else - report=$($DPDK_CHECKPATCH_PATH $options - 2>/dev/null) + TMPINPUT=$(mktemp checkpatches.XXXXXX) + cat > ./$TMPINPUT + fi + + report=$($DPDK_CHECKPATCH_PATH $options $TMPINPUT 2>/dev/null) + + if [ $? -ne 0 ] + then + $verbose || printf '\n### %s\n\n' "$3" + printf '%s\n' "$report" | sed -n '1,/^total:.*lines checked$/p' + ret=1 + fi + + ! $verbose || printf '\nChecking API additions/removals:\n' + + report=$($VALIDATE_NEW_API "$TMPINPUT") + + if [ $? -ne 0 ]; then + printf '%s\n' "$report" + ret=1 + fi + + clean_tmp_files + if [ $ret -eq 0 ]; then + return 0 fi - [ $? -ne 0 ] || return 0 - $verbose || printf '\n### %s\n\n' "$3" - printf '%s\n' "$report" | sed -n '1,/^total:.*lines checked$/p' status=$(($status + 1)) } -- 2.17.1