From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id 43B31A0C47;
Thu, 7 Oct 2021 12:32:24 +0200 (CEST)
Received: from [217.70.189.124] (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id C698B41166;
Thu, 7 Oct 2021 12:32:23 +0200 (CEST)
Received: from mga11.intel.com (mga11.intel.com [192.55.52.93])
by mails.dpdk.org (Postfix) with ESMTP id DAFF041137
for <dev@dpdk.org>; Thu, 7 Oct 2021 12:32:21 +0200 (CEST)
X-IronPort-AV: E=McAfee;i="6200,9189,10129"; a="223615898"
X-IronPort-AV: E=Sophos;i="5.85,354,1624345200"; d="scan'208";a="223615898"
Received: from orsmga003.jf.intel.com ([10.7.209.27])
by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;
07 Oct 2021 03:32:17 -0700
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.85,354,1624345200"; d="scan'208";a="440220016"
Received: from silpixa00401215.ir.intel.com ([10.55.128.96])
by orsmga003.jf.intel.com with ESMTP; 07 Oct 2021 03:32:16 -0700
From: Sean Morrissey <sean.morrissey@intel.com>
To:
Cc: dev@dpdk.org, Sean Morrissey <sean.morrissey@intel.com>,
Conor Fogarty <conor.fogarty@intel.com>,
Bruce Richardson <bruce.richardson@intel.com>
Date: Thu, 7 Oct 2021 10:25:53 +0000
Message-Id: <20211007102557.188739-2-sean.morrissey@intel.com>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20211007102557.188739-1-sean.morrissey@intel.com>
References: <20211006111329.152938-1-sean.morrissey@intel.com>
<20211007102557.188739-1-sean.morrissey@intel.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Subject: [dpdk-dev] [PATCH v3 1/5] devtools: script to remove unused headers
includes
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>
This script can be used for removing headers flagged for removal by the
include-what-you-use (IWYU) tool. The script has the ability to remove
headers from specified sub-directories or dpdk as a whole and tests the
build after each removal by calling meson compile.
example usages:
Remove headers flagged by iwyu_tool output file
$ ./devtools/process_iwyu.py iwyu.out -b build
Remove headers flagged by iwyu_tool output file from sub-directory
$ ./devtools/process_iwyu.py iwyu.out -b build -d lib/kvargs
Remove headers directly piped from the iwyu_tool
$ iwyu_tool -p build | ./devtools/process_iwyu.py - -b build
Signed-off-by: Sean Morrissey <sean.morrissey@intel.com>
Signed-off-by: Conor Fogarty <conor.fogarty@intel.com>
Reviewed-by: Bruce Richardson <bruce.richardson@intel.com>
---
devtools/process_iwyu.py | 109 +++++++++++++++++++++++++++++++++++++++
1 file changed, 109 insertions(+)
create mode 100755 devtools/process_iwyu.py
diff --git a/devtools/process_iwyu.py b/devtools/process_iwyu.py
new file mode 100755
index 0000000000..50f3d4c5c7
--- /dev/null
+++ b/devtools/process_iwyu.py
@@ -0,0 +1,109 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2021 Intel Corporation
+#
+
+import argparse
+import fileinput
+import sys
+from os.path import abspath, relpath, join
+from pathlib import Path
+from mesonbuild import mesonmain
+
+
+def args_parse():
+ "parse arguments and return the argument object back to main"
+ parser = argparse.ArgumentParser(description="This script can be used to remove includes which are not in use\n")
+ parser.add_argument('-b', '--build_dir', type=str, default='build',
+ help="The path to the build directory in which the IWYU tool was used in.")
+ parser.add_argument('-d', '--sub_dir', type=str, default='',
+ help="The sub-directory to remove headers from.")
+ parser.add_argument('file', type=Path,
+ help="The path to the IWYU log file or output from stdin.")
+
+ return parser.parse_args()
+
+
+def run_meson(args):
+ "Runs a meson command logging output to process.log"
+ with open('process_iwyu.log', 'a') as sys.stdout:
+ ret = mesonmain.run(args, abspath('meson'))
+ sys.stdout = sys.__stdout__
+ return ret
+
+
+def remove_includes(filepath, include, build_dir):
+ "Attempts to remove include, if it fails then revert to original state"
+ with open(filepath) as f:
+ lines = f.readlines() # Read lines when file is opened
+
+ with open(filepath, 'w') as f:
+ for ln in lines: # Removes the include passed in
+ if not ln.startswith(include):
+ f.write(ln)
+
+ # run test build -> call meson on the build folder, meson compile -C build
+ ret = run_meson(['compile', '-C', build_dir])
+ if (ret == 0): # Include is not needed -> build is successful
+ print('SUCCESS')
+ else:
+ # failed, catch the error
+ # return file to original state
+ with open(filepath, 'w') as f:
+ f.writelines(lines)
+ print('FAILED')
+
+
+def get_build_config(builddir, condition):
+ "returns contents of rte_build_config.h"
+ with open(join(builddir, 'rte_build_config.h')) as f:
+ return [ln for ln in f.readlines() if condition(ln)]
+
+
+def uses_libbsd(builddir):
+ "return whether the build uses libbsd or not"
+ return bool(get_build_config(builddir, lambda ln: 'RTE_USE_LIBBSD' in ln))
+
+
+def process(args):
+ "process the iwyu output on a set of files"
+ filepath = None
+ build_dir = abspath(args.build_dir)
+ directory = args.sub_dir
+
+ print("Warning: The results of this script may include false positives which are required for different systems",
+ file=sys.stderr)
+
+ keep_str_fns = uses_libbsd(build_dir) # check for libbsd
+ if keep_str_fns:
+ print("Warning: libbsd is present, build will fail to detect incorrect removal of rte_string_fns.h",
+ file=sys.stderr)
+ # turn on werror
+ run_meson(['configure', build_dir, '-Dwerror=true'])
+ # Use stdin if no iwyu_tool out file given
+ for line in fileinput.input(args.file):
+ if 'should remove' in line:
+ # If the file path in the iwyu_tool output is an absolute path it
+ # means the file is outside of the dpdk directory, therefore ignore it.
+ # Also check to see if the file is within the specified sub directory.
+ filename = line.split()[0]
+ if (filename != abspath(filename) and
+ directory in filename):
+ filepath = relpath(join(build_dir, filename))
+ elif line.startswith('-') and filepath:
+ include = '#include ' + line.split()[2]
+ print(f"Remove {include} from {filepath} ... ", end='', flush=True)
+ if keep_str_fns and '<rte_string_fns.h>' in include:
+ print('skipped')
+ continue
+ remove_includes(filepath, include, build_dir)
+ else:
+ filepath = None
+
+
+def main():
+ process(args_parse())
+
+
+if __name__ == '__main__':
+ main()
--
2.25.1