From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 86CE65A8C for ; Tue, 31 May 2016 10:51:55 +0200 (CEST) Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga102.fm.intel.com with ESMTP; 31 May 2016 01:51:54 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.26,395,1459839600"; d="scan'208";a="987929377" Received: from sie-lab-212-251.ir.intel.com (HELO silpixa00381635.ir.intel.com) ([10.237.212.251]) by orsmga002.jf.intel.com with ESMTP; 31 May 2016 01:51:51 -0700 From: Jasvinder Singh To: dev@dpdk.org Cc: cristian.dumitrescu@intel.com, Guruprasad Mukundarao Date: Tue, 31 May 2016 09:58:05 +0100 Message-Id: <1464685085-95410-1-git-send-email-jasvinder.singh@intel.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1462561464-76782-1-git-send-email-jasvinder.singh@intel.com> References: <1462561464-76782-1-git-send-email-jasvinder.singh@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH v3] ip_pipeline: add script file for pipeline to core mappings X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2016 08:51:56 -0000 From: Guruprasad Mukundarao This script parses the application configuration file and detects all the pipelines specified therein, and then, it generates all the possible mappings of those pipelines on the specified CPU core-list. As a result, each of the possible pipeline-to-core mappings is saved as separate output configuration file. For example- if input file is edge_router_downstream.cfg with 3 pipeline (excluding pipeline 0) and core-list is "1, 2" following combinations will be generated- Pipeline 1 Pipeline 2 Pipeline 3 Core = 1 Core = 1 Core = 2 Core = 1 Core = 2 Core = 1 Core = 2 Core = 1 Core = 1 Core = 2 Core = 2 Core = 1 Core = 2 Core = 1 Core = 2 Core = 1 Core = 2 Core = 2 Core = C1 Core = C1H Core = C2 Core = C1 Core = C2 Core = C1H Core = C2 Core = C1 Core = C1H This script will help users to analyse the performance of application by evaluating all the generated configuration files with different pipelines-to-core mappings and obtaining the application configuration file with best performance. To run the script, issue the following command; ./pipeline-to-core-mapping.py -i -pc "sc, ..." Some optional arguments are given below: -h, --help show this help message and exit -ht {ON,OFF}, --hyper-thread {ON,OFF} enable/disable hyper threading. default is ON -nO, --no-output-file disable output config file generation. Output file generation is enabled by default Signed-off-by: Guruprasad Mukundarao Acked-by: Cristian Dumitrescu --- v3 - update the commit message - create folder to save generated output configuration files - fix the duplicate files creation with same pipeline-to-core mappings - fix checkpatch errors - the generated files will have name in the following format- "__s_c_h.cfg" v2 - correct the copyright header .../ip_pipeline/config/pipeline-to-core-mapping.py | 995 +++++++++++++++++++++ 1 file changed, 995 insertions(+) create mode 100644 examples/ip_pipeline/config/pipeline-to-core-mapping.py diff --git a/examples/ip_pipeline/config/pipeline-to-core-mapping.py b/examples/ip_pipeline/config/pipeline-to-core-mapping.py new file mode 100644 index 0000000..3cabff0 --- /dev/null +++ b/examples/ip_pipeline/config/pipeline-to-core-mapping.py @@ -0,0 +1,995 @@ +#! /usr/bin/python3 + +# BSD LICENSE +# +# Copyright(c) 2016 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +# +# This script maps the set of pipelines identified (MASTER pipelines are +# ignored) from the input configuration file to the set of cores +# provided as input argument and creates configuration files for each of +# the mapping combinations. +# + +import sys +import array +import itertools +import re +import argparse +import os +from collections import namedtuple + +#default values +enable_stage0_traceout = 1 +enable_stage1_traceout = 1 +enable_stage2_traceout = 1 + +#enable_stage0_fileout = 0 +enable_stage1_fileout = 1 +enable_stage2_fileout = 1 + +#pattern for physical core +pattern_phycore = '^(s|S)\d(c|C)[1-9][0-9]*$' +reg_phycore = re.compile(pattern_phycore) + +#----------------------------------------------------------------------------- +def popcount(mask): + return bin(mask).count("1") + +#----------------------------------------------------------------------------- +def len2mask(length): + if (length == 0): + return 0 + + if (length > 64): + sys.exit('error: len2mask - lenght %i > 64. exiting' %length) + + return (0xFFFFFFFFFFFFFFFF >> (64 - length)) + +#----------------------------------------------------------------------------- +def bitstring_write(n, n_bits): + tmpstr = "" + if (n_bits > 64): + return + + i = n_bits - 1 + while (i >= 0): + cond = (n & (1 << i)) + if (cond): + print('1', end='') + tmpstr += '1' + else: + print('0', end='') + tmpstr += '0' + i -= 1 + #end while + return tmpstr +#end function + +#------------------------------------------------------------------------- +Constants = namedtuple('Constants', ['MAX_CORES', 'MAX_PIPELINES']) +constants = Constants(16, 64) + + +#------------------------------------------------------------------------- +class Cores0: + def __init__(self): + self.n_pipelines = 0 + +class Cores1: + def __init__(self): + self.pipelines = 0 + self.n_pipelines = 0 + +class Cores2: + def __init__(self): + self.pipelines = 0 + self.n_pipelines = 0 + self.counter = 0 + self.counter_max = 0 + self.bitpos = array.array("L", itertools.repeat(0,constants.MAX_PIPELINES)) + + +#-------------------------------------------------------------------- +class Context0: + def __init__(self): + self.cores = [Cores0() for i in range(0, constants.MAX_CORES)] + self.n_cores = 0 + self.n_pipelines = 0 + self.n_pipelines0 = 0 + self.pos = 0 + self.file_comment = "" + self.c1 = None + self.c2 = None + + #------------------------------------------------------------- + def stage0_print(self): + print('printing Context0 obj') + print('c0.cores(n_pipelines) = [ ', end='') + for cores_count in range(0, constants.MAX_CORES): + print(self.cores[cores_count].n_pipelines, end=' ') + print(']') + print('c0.n_cores = %d' %self.n_cores) + print('c0.n_pipelines = %d' %self.n_pipelines) + print('c0.n_pipelines0 = %d' %self.n_pipelines0) + print('c0.pos = %d' %self.pos) + print('c0.file_comment = %s' %self.file_comment) + if (self.c1 is not None): + print('c0.c1 = ', end='') + print(repr(self.c1)) + else: + print('c0.c1 = None') + + if (self.c2 is not None): + print('c0.c2 = ', end='') + print(repr(self.c2)) + else: + print('c0.c2 = None') + + + #------------------------------------------------------------- + def stage0_init(self, num_cores, num_pipelines, c1, c2): + self.n_cores = num_cores + self.n_pipelines = num_pipelines + self.c1 = c1 + self.c2 = c2 + + #------------------------------------------------------------- + def stage0_process(self): + #print('inside stage0_process') + + # stage0 init + self.cores[0].n_pipelines = self.n_pipelines + self.n_pipelines0 = 0 + self.pos = 1 + + while True: + #go forward + while True: + if ((self.pos < self.n_cores) and (self.n_pipelines0 > 0)): + self.cores[self.pos].n_pipelines = min(self.cores[self.pos - 1].n_pipelines, self.n_pipelines0) + self.n_pipelines0 -= self.cores[self.pos].n_pipelines + + self.pos += 1 + else: + break + + #end while + + # check solution + if (self.n_pipelines0 == 0): + self.stage0_log() + self.c1.stage1_init(self, self.c2) # self is object c0 + self.c1.stage1_process() + + # go backward + while True: + if (self.pos == 0): + return + + self.pos -= 1 + if ((self.cores[self.pos].n_pipelines >1) and + (self.pos != (self.n_cores -1))): + break + + self.n_pipelines0 += self.cores[self.pos].n_pipelines + self.cores[self.pos].n_pipelines = 0 + #end while + + # rearm + self.cores[self.pos].n_pipelines -= 1 + self.n_pipelines0 += 1 + self.pos += 1 + #end while + #end function + + + #------------------------------------------------------------- + def stage0_log(self): + tmp_file_comment = "" + if(enable_stage0_traceout != 1): + return + + print('STAGE0: ', end='') + tmp_file_comment += 'STAGE0: ' + for cores_count in range(0, self.n_cores): + print('C%d = %d\t' \ + %(cores_count, self.cores[cores_count].n_pipelines), end='') + tmp_file_comment += "C{} = {}\t".format(cores_count, \ + self.cores[cores_count].n_pipelines) + #end for + print('') + self.c1.stage0_file_comment = tmp_file_comment + self.c2.stage0_file_comment = tmp_file_comment + + # end function + +#end class Context0 + + +#------------------------------------------------------------- +class Context1: + #class attribute + _fileTrace = None + + def __init__(self): + self.cores = [Cores1() for i in range(constants.MAX_CORES)] + self.n_cores = 0 + self.n_pipelines = 0 + self.pos = 0 + self.stage0_file_comment = "" + self.stage1_file_comment = "" + + self.c2 = None + self.arr_pipelines2cores = [] + #end init + + #------------------------------------------------------------- + def stage1_reset(self): + for i in range(constants.MAX_CORES): + self.cores[i].pipelines = 0 + self.cores[i].n_pipelines = 0 + + self.n_cores = 0 + self.n_pipelines = 0 + self.pos = 0 + self.c2 = None + self.arr_pipelines2cores.clear() + #end def stage1_reset + + #------------------------------------------------------------- + def stage1_print(self): + print('printing Context1 obj') + print('c1.cores(pipelines,n_pipelines) = [ ', end='') + for cores_count in range(0, constants.MAX_CORES): + print('(%d,%d)' %(self.cores[cores_count].pipelines, + self.cores[cores_count].n_pipelines), end=' ') + print(']') + print('c1.n_cores = %d' %self.n_cores) + print('c1.n_pipelines = %d' %self.n_pipelines) + print('c1.pos = %d' %self.pos) + print('c1.stage0_file_comment = %s' %self.stage0_file_comment) + print('c1.stage1_file_comment = %s' %self.stage1_file_comment) + if (self.c2 is not None): + print('c1.c2 = ', end='') + print(self.c2) + else: + print('c1.c2 = None') + #end stage1_print + + #------------------------------------------------------------- + def stage1_init(self, c0, c2): + self.stage1_reset() + self.n_cores = 0 + while (c0.cores[self.n_cores].n_pipelines > 0): + self.n_cores += 1 + + self.n_pipelines = c0.n_pipelines + self.c2 = c2 + + self.arr_pipelines2cores = [0] * self.n_pipelines + + i = 0 + while (i < self.n_cores): + self.cores[i].n_pipelines = c0.cores[i].n_pipelines + i += 1 + #end while + #end stage1_init + #------------------------------------------------------------- + def stage1_process(self): + pipelines_max = len2mask(self.n_pipelines) + + while True: + pos = 0 + overlap = 0 + + if (self.cores[self.pos].pipelines == pipelines_max): + if (self.pos == 0): + return + + self.cores[self.pos].pipelines = 0 + self.pos -= 1 + continue + #end if + + self.cores[self.pos].pipelines += 1 + + if (popcount(self.cores[self.pos].pipelines) \ + != self.cores[self.pos].n_pipelines): + continue + + overlap = 0 + pos = 0 + while (pos < self.pos): + if ((self.cores[self.pos].pipelines) & \ + (self.cores[pos].pipelines)): + overlap = 1 + break + pos += 1 + #end while + + if (overlap): + continue + + + if ((self.pos > 0) and \ + ((self.cores[self.pos].n_pipelines) == (self.cores[self.pos - 1].n_pipelines)) and \ + ((self.cores[self.pos].pipelines) < (self.cores[self.pos - 1].pipelines))): + continue + + if (self.pos == self.n_cores - 1): + self.stage1_log() + self.c2.stage2_init(self) + self.c2.stage2_process() + + if (self.pos == 0): + return + + self.cores[self.pos].pipelines = 0 + self.pos -= 1 + continue + #endif + + self.pos += 1 + #end for + #end stage1_process + + #------------------------------------------------------------- + def stage1_log(self): + tmp_file_comment = "" + if(enable_stage1_traceout == 1): + print('STAGE1: ', end = '') + tmp_file_comment += 'STAGE1: ' + i = 0 + while (i < self.n_cores): + print('C%d = [' %i, end='') + tmp_file_comment += "C{} = [".format(i) + + j = self.n_pipelines - 1 + while (j >= 0): + cond = ((self.cores[i].pipelines) & (1 << j)) + if (cond): + print('1', end='') + tmp_file_comment += '1' + else: + print('0', end='') + tmp_file_comment += '0' + j -= 1 + + print(']\t', end='') + tmp_file_comment += ']\t' + i += 1 + #end while + print('\n', end ='') + #tmp_file_comment += '\n' + self.stage1_file_comment = tmp_file_comment + self.c2.stage1_file_comment = tmp_file_comment + #endif + + #check if file traceing is enabled + if(enable_stage1_fileout != 1): + return + + #spit out the combination to file + self.stage1_process_file() + #end function stage1_log + + + #------------------------------------------------------------------------ + def stage1_updateCoresInBuf(self, nPipeline, sCore): + + rePipeline = self._fileTrace.arr_pipelines[nPipeline] + rePipeline = rePipeline.replace("[","\[").replace("]","\]") + reCore = 'core\s*=\s*((\d*)|(((s|S)\d)?(c|C)[1-9][0-9]*)).*\n' + sSubs = 'core = ' + sCore + '\n' + + reg_pipeline = re.compile(rePipeline) + search_match = reg_pipeline.search(self._fileTrace.in_buf) + #debug + #print(search_match) + + if(search_match): + pos = search_match.start() + substr1 = self._fileTrace.in_buf[:pos] + substr2 = self._fileTrace.in_buf[pos:] + substr2 = re.sub(reCore, sSubs, substr2,1) + self._fileTrace.in_buf = substr1 + substr2 + #endif + #end function stage1_updateCoresInBuf + + #------------------------------------------------------------------------ + def stage1_process_file(self): + outFileName = os.path.join(self._fileTrace.out_path, \ + self._fileTrace.prefix_outfile) + outFileName += "_{}".format(self._fileTrace.n_cores) + + i = 0 #represents core number + while (i < self.n_cores): + j = self.n_pipelines - 1 + pipeline_idx = 0 + while(j >= 0): + cond = ((self.cores[i].pipelines) & (1 << j)) + if (cond): + #update the pipelines array to match the core + # only in case of cond match + self.arr_pipelines2cores[pipeline_idx] = fileTrace.in_physical_cores[i] + #endif + j -= 1 + pipeline_idx += 1 + #end while + i += 1 + #end while + + # update the in_buf as per the arr_pipelines2cores + for pipeline_idx in range(len(self.arr_pipelines2cores)): + outFileName += "_{}".format(self.arr_pipelines2cores[pipeline_idx]) + self.stage1_updateCoresInBuf(pipeline_idx,self.arr_pipelines2cores[pipeline_idx]) + #endfor + + + #by now the in_buf is all set to be written to file + outFileName += self._fileTrace.suffix_outfile + outputFile = open(outFileName, "w") + + #write out the comments + outputFile.write("; =============== Pipeline-to-Core Mapping ================\n") + outputFile.write("; Generated from file {}\n".format(self._fileTrace.in_file_namepath)) + outputFile.write("; Input pipelines = {}\n; Input cores = {}\n" \ + .format(fileTrace.arr_pipelines, fileTrace.in_physical_cores)) + + strTruncated = ("", "(Truncated)") [self._fileTrace.ncores_truncated] + outputFile.write("; N_PIPELINES = {} N_CORES = {} {} hyper_thread = {} \n"\ + .format(self._fileTrace.n_pipelines, self._fileTrace.n_cores, strTruncated, self._fileTrace.hyper_thread)) + + outputFile.write("; {}\n".format(self.stage0_file_comment)) #stage0 comment + outputFile.write("; {}\n".format(self.stage1_file_comment)) #stage1 comment + #debugging + #outputFile.write("; <<<>>>") + #outputFile.write("; stage1_arr_pipelines2cores = {}".format(self.arr_pipelines2cores)) + outputFile.write("; ========================================================\n") + outputFile.write(";\n") + + # + outputFile.write(self._fileTrace.in_buf) + outputFile.flush() + outputFile.close() + #end function stage1_process_file +# end class Context1 + + +#----------------------------------------------------------------------------- +class Context2: + #class attribute + _fileTrace = None + + def __init__(self): + self.cores = [Cores2() for i in range(constants.MAX_CORES)] + self.n_cores = 0 + self.n_pipelines = 0 + self.pos = 0 + self.stage0_file_comment = "" + self.stage1_file_comment = "" + self.stage2_file_comment = "" + + #each array entry is a pipeline mapped to core stored as string + # pipeline ranging from 1 to n, however stored in zero based array + self.arr2_pipelines2cores = [] + + #------------------------------------------------------------- + def stage2_print(self): + print('printing Context2 obj') + print('c2.cores(pipelines, n_pipelines, counter, counter_max) =') + for cores_count in range(0, constants.MAX_CORES): + print('core[%d] = (%d,%d,%d,%d)' %(cores_count, + self.cores[cores_count].pipelines, \ + self.cores[cores_count].n_pipelines, \ + self.cores[cores_count].counter, \ + self.cores[cores_count].counter_max)) + + print('c2.n_cores = %d' %self.n_cores, end='') + print('c2.n_pipelines = %d' %self.n_pipelines, end='') + print('c2.pos = %d' %self.pos) + print('c2.stage0_file_comment = %s' %self.self.stage0_file_comment) + print('c2.stage1_file_comment = %s' %self.self.stage1_file_comment) + print('c2.stage2_file_comment = %s' %self.self.stage2_file_comment) + #end for + #end function stage2_print + + + #------------------------------------------------------------- + def stage2_reset(self): + for i in range(0, constants.MAX_CORES): + self.cores[i].pipelines = 0 + self.cores[i].n_pipelines = 0; + self.cores[i].counter = 0; + self.cores[i].counter_max = 0 + + for idx in range(0, constants.MAX_PIPELINES): + self.cores[i].bitpos[idx] = 0 + + self.n_cores = 0 + self.n_pipelines = 0 + self.pos = 0 + + self.arr2_pipelines2cores.clear() + #end stage2_reset + + #------------------------------------------------------------- + def bitpos_load(self, coreidx): + i = j = 0 + while (i < self.n_pipelines): + if ((self.cores[coreidx].pipelines) & \ + (1 << i)): + self.cores[coreidx].bitpos[j] = i + j += 1 + i += 1 + self.cores[coreidx].n_pipelines = j + + + #------------------------------------------------------------- + def bitpos_apply(self, in_buf, pos, n_pos): + out = 0 + for i in range(0, n_pos): + out |= (in_buf & (1 << i)) << (pos[i] - i) + + return out + + + #------------------------------------------------------------- + def stage2_init(self, c1): + self.stage2_reset() + self.n_cores = c1.n_cores + self.n_pipelines = c1.n_pipelines + + self.arr2_pipelines2cores = [''] * self.n_pipelines + + core_idx = 0 + while (core_idx < self.n_cores): + self.cores[core_idx].pipelines = c1.cores[core_idx].pipelines + + self.bitpos_load(core_idx) + core_idx += 1 + #end while + #end function stage2_init + + + #------------------------------------------------------------- + def stage2_log(self): + tmp_file_comment = "" + if(enable_stage2_traceout == 1): + print('STAGE2: ', end='') + tmp_file_comment += 'STAGE2: ' + + for i in range(0, self.n_cores): + mask = len2mask(self.cores[i].n_pipelines) + pipelines_ht0 = self.bitpos_apply((~self.cores[i].counter) & mask, \ + self.cores[i].bitpos, \ + self.cores[i].n_pipelines) + + pipelines_ht1 = self.bitpos_apply(self.cores[i].counter, \ + self.cores[i].bitpos, \ + self.cores[i].n_pipelines) + + print('C%dHT0 = [' %i, end='') + tmp_file_comment += "C{}HT0 = [".format(i) + tmp_file_comment += bitstring_write(pipelines_ht0, self.n_pipelines) + + print(']\tC%dHT1 = [' %i, end='') + tmp_file_comment += "]\tC{}HT1 = [".format(i) + tmp_file_comment += bitstring_write(pipelines_ht1, self.n_pipelines) + print(']\t', end='') + tmp_file_comment += ']\t' + #end for + print('') + self.stage2_file_comment = tmp_file_comment + #endif + + #check if file traceing is enabled + if(enable_stage2_fileout != 1): + return + #spit out the combination to file + self.stage2_process_file() + + #end function stage2_log + + #------------------------------------------------------------- + def stage2_updateCoresInBuf(self, nPipeline, sCore): + rePipeline = self._fileTrace.arr_pipelines[nPipeline] + rePipeline = rePipeline.replace("[","\[").replace("]","\]") + reCore = 'core\s*=\s*((\d*)|(((s|S)\d)?(c|C)[1-9][0-9]*)).*\n' + sSubs = 'core = ' + sCore + '\n' + #sSubs = 'core = ' + self._fileTrace.in_physical_cores[sCore] + '\n' + + reg_pipeline = re.compile(rePipeline) + search_match = reg_pipeline.search(self._fileTrace.in_buf) + + if(search_match): + pos = search_match.start() + substr1 = self._fileTrace.in_buf[:pos] + substr2 = self._fileTrace.in_buf[pos:] + substr2 = re.sub(reCore, sSubs, substr2,1) + self._fileTrace.in_buf = substr1 + substr2 + #endif + #end function stage2_updateCoresInBuf + + + #------------------------------------------------------------- + def pipelines2cores(self, n, n_bits, nCore, bHT): + if (n_bits > 64): + return + + i = n_bits - 1 + pipeline_idx = 0 + while (i >= 0): + cond = (n & (1 << i)) + if (cond): + #update the pipelines array to match the core + # only in case of cond match + # PIPELINE0 and core 0 are reserved + if(bHT): + tmpCore = fileTrace.in_physical_cores[nCore] + 'h' + self.arr2_pipelines2cores[pipeline_idx] = tmpCore + else: + self.arr2_pipelines2cores[pipeline_idx] = \ + fileTrace.in_physical_cores[nCore] + #endif + #endif + i -= 1 + pipeline_idx += 1 + #end while + + #end function pipelines2cores + + #------------------------------------------------------------- + def stage2_process_file(self): + outFileName = os.path.join(self._fileTrace.out_path, \ + self._fileTrace.prefix_outfile) + outFileName += "_{}".format(self._fileTrace.n_cores) + + for i in range(0, self.n_cores): + mask = len2mask(self.cores[i].n_pipelines) + pipelines_ht0 = self.bitpos_apply((~self.cores[i].counter) & mask, \ + self.cores[i].bitpos, \ + self.cores[i].n_pipelines) + + pipelines_ht1 = self.bitpos_apply(self.cores[i].counter, \ + self.cores[i].bitpos, \ + self.cores[i].n_pipelines) + + #update pipelines to core mapping + self.pipelines2cores(pipelines_ht0, self.n_pipelines, i , False) + self.pipelines2cores(pipelines_ht1, self.n_pipelines, i, True) + #end for + + # update the in_buf as per the arr_pipelines2cores + for pipeline_idx in range(len(self.arr2_pipelines2cores)): + outFileName += "_{}".format(self.arr2_pipelines2cores[pipeline_idx]) + self.stage2_updateCoresInBuf(pipeline_idx, self.arr2_pipelines2cores[pipeline_idx]) + + #by now the in_buf is all set to be written to file + outFileName += self._fileTrace.suffix_outfile + outputFile = open(outFileName, "w") + + #write out the comments + outputFile.write("; ========= Pipeline-to-Core Mapping =====================\n") + outputFile.write("; Generated from file {}\n".format(self._fileTrace.in_file_namepath)) + outputFile.write("; Input pipelines = {}\n; Input cores = {}\n" \ + .format(fileTrace.arr_pipelines, fileTrace.in_physical_cores)) + + strTruncated = ("", "(Truncated)") [self._fileTrace.ncores_truncated] + outputFile.write("; N_PIPELINES = {} N_CORES = {} {} hyper_thread = {}\n"\ + .format(self._fileTrace.n_pipelines, self._fileTrace.n_cores, strTruncated, self._fileTrace.hyper_thread)) + + outputFile.write("; {}\n".format(self.stage0_file_comment)) #stage0 comment + outputFile.write("; {}\n".format(self.stage1_file_comment)) #stage1 comment + outputFile.write("; {}\n".format(self.stage2_file_comment)) #stage2 comment + outputFile.write("; ========================================================\n") + outputFile.write(";\n") + + outputFile.write(self._fileTrace.in_buf) + outputFile.flush() + outputFile.close() + + #end function stage2_process_file + + #------------------------------------------------------------- + def stage2_process(self): + i = 0 + while(i < self.n_cores): + self.cores[i].counter_max = len2mask(self.cores[i].n_pipelines - 1) + i += 1 + #end while + + self.pos = self.n_cores - 1 + while True: + if (self.pos == self.n_cores - 1): + self.stage2_log() + + if (self.cores[self.pos].counter == self.cores[self.pos].counter_max): + if (self.pos == 0): + return + #endif + + self.cores[self.pos].counter = 0 + self.pos -= 1 + continue + #endif + + self.cores[self.pos].counter += 1 + + if(self.pos < self.n_cores - 1): + self.pos += 1 + #endif + #end while +#end class Context2 + +#----------------------------------------------------------------------------- +class FileTrace: + #initialize default parameters + def __init__(self,filenamepath): + self.in_file_namepath = os.path.abspath(filenamepath) + self.in_filename = os.path.basename(self.in_file_namepath) + self.in_path = os.path.dirname(self.in_file_namepath) + + filenamesplit = self.in_filename.split('.') + self.prefix_outfile = filenamesplit[0] + self.suffix_outfile = ".cfg" + + #output folder: in the same folder as input file + #create new folder in the name of input file + self.out_path = os.path.join(os.path.abspath(os.path.dirname(__file__)), \ + self.prefix_outfile) + os.makedirs(self.out_path, exist_ok=True) + + self.in_buf = None + self.arr_pipelines = [] # holds the positions of search + + self.max_cores = 15 + self.max_pipelines = 15 + + self.in_physical_cores = None + self.hyper_thread = None + + # save the num of pipelines determined from input file + self.n_pipelines = 0 + # save the num of cores input (or the truncated value) + self.n_cores = 0 + self.ncores_truncated = False + + #end init + + def print_TraceFile(self): + print("self.in_file_namepath = ", self.in_file_namepath) + print("self.in_filename = ", self.in_filename) + print("self.in_path = ", self.in_path) + print("self.out_path = ", self.out_path) + print("self.prefix_outfile = ", self.prefix_outfile) + print("self.suffix_outfile = ", self.suffix_outfile) + print("self.in_buf = ", self.in_buf) + print("self.arr_pipelines =", self.arr_pipelines) + print("self.in_physical_cores", self.in_physical_cores) + print("self.hyper_thread", self.hyper_thread) + #end function print_TraceFile + +#end class FileTrace + + +#----------------------------------------------------------------------------- +# main process method +# +def process(n_cores, n_pipelines, fileTrace): + if (n_cores == 0): + sys.exit('N_CORES is 0, exiting') + #endif + + if (n_pipelines == 0): + sys.exit('N_PIPELINES is 0, exiting') + #endif + + if (n_cores > n_pipelines): + print('\nToo many cores, truncating N_CORES to N_PIPELINES') + n_cores = n_pipelines + fileTrace.ncores_truncated = True + #endif + fileTrace.n_pipelines = n_pipelines + fileTrace.n_cores = n_cores + + strTruncated = ("", "(Truncated)") [fileTrace.ncores_truncated] + print("N_PIPELINES = {}, N_CORES = {} {}" \ + .format(n_pipelines,n_cores, strTruncated)) + print("---------------------------------------------------------------") + + c0 = Context0() + c1 = Context1() + c2 = Context2() + + #initialize the class variables + c1._fileTrace = fileTrace + c2._fileTrace = fileTrace + + c0.stage0_init(n_cores, n_pipelines, c1, c2) + c0.stage0_process() + +#end function process + + + +#-------------------------------------------------------------------------- +def validate_core(core): + match = reg_phycore.match(core) + if(match): + return True + else: + return False + #endif +#end function validate_core + + +#-------------------------------------------------------------------------- +def validate_phycores(phy_cores): + #eat up whitespaces + phy_cores = phy_cores.strip().split(',') + + #check if the core list is unique + if(len(phy_cores) != len(set(phy_cores))): + print('list of physical cores has duplicates') + return None + #endif + for core in phy_cores: + if(validate_core(core) != True): + print('invalid physical core specified.') + return None + #endif + #endfor + return phy_cores +#end function validate_phycores + +#-------------------------------------------------------------------------- +def scanconfigfile(fileTrace): + #debug + #fileTrace.print_TraceFile() + + # open file + filetoscan = open(fileTrace.in_file_namepath, 'r') + fileTrace.in_buf = filetoscan.read() + + #reset iterator on open file + filetoscan.seek(0) + + # scan input file for pipelines + # master pipelines to be ignored + pattern_pipeline = r'\[PIPELINE\d*\]' + pattern_mastertype = r'type\s*=\s*MASTER' + + pending_pipeline = False + for line in filetoscan: + match_pipeline = re.search(pattern_pipeline, line) + match_type = re.search('type\s*=', line) + match_mastertype = re.search(pattern_mastertype, line) + + if(match_pipeline): + sPipeline = line[match_pipeline.start():match_pipeline.end()] + #sPipeline = sPipeline.strip('[]') + pending_pipeline = True + elif(match_type): + # found a type definition... + if(match_mastertype == None): + # and this is not a master pipeline... + if(pending_pipeline == True): + # add it to the list of pipelines to be mapped + fileTrace.arr_pipelines.append(sPipeline) + pending_pipeline = False + else: + # and this is a master pipeline... + # ignore the current and move on to next + sPipeline = "" + pending_pipeline = False + #endif + #endif + #endfor + filetoscan.close() + + # validate if pipelines are unique + if(len(fileTrace.arr_pipelines) != len(set(fileTrace.arr_pipelines))): + sys.exit('Error: duplicate pipelines in input file') + #endif + + num_pipelines = len(fileTrace.arr_pipelines) + num_cores = len(fileTrace.in_physical_cores) + + #debug + #print(fileTrace.matches_pipeline) + print("-------------------Pipeline-to-core mapping--------------------") + print("Input pipelines = {}\nInput cores = {}" \ + .format(fileTrace.arr_pipelines, fileTrace.in_physical_cores)) + + #input configuration file validations goes here + if (num_cores > fileTrace.max_cores): + sys.exit('Error: number of cores specified > max_cores (%d)' %fileTrace.max_cores) + + if (num_pipelines > fileTrace.max_pipelines): + sys.exit('Error: number of pipelines in input cfg file > max_pipelines (%d)' %fileTrace.max_pipelines) + + #call process to generate pipeline-to-core mapping, trace and log + process(num_cores, num_pipelines, fileTrace) + +#end function scanconfigfile + + +#----------------------------------------------------------------------------- +# python trick - so that our Python files can act as either reusable modules, +# or as standalone programs +if __name__ == "__main__": + + parser = argparse.ArgumentParser(description='mappipelines') + + reqNamedGrp = parser.add_argument_group('required named args') + reqNamedGrp.add_argument('-i', '--input-file', type=argparse.FileType('r'), help='Input config file', required=True) + + #--physical-cores “, , …”, with = sc + reqNamedGrp.add_argument('-pc', '--physical-cores', type=validate_phycores, help='''Enter available CPU cores in format:\",,...\" + where each core format: \"sc\" + where SOCKETID={0..9}, COREID={1-99}''', required=True) + + + #add optional arguments + parser.add_argument('-ht', '--hyper-thread', help='enable/disable hyper threading. default is ON', default='ON', choices=['ON','OFF']) + + parser.add_argument('-nO', '--no-output-file', help='disable output config file generation. Output file generation is enabled by default', action="store_true") + + args = parser.parse_args() + + if(args.physical_cores == None): + parser.error("invalid physical_cores specified") + #endif + + # create object of FileTrace and initialise + fileTrace = FileTrace(args.input_file.name) + fileTrace.in_physical_cores = args.physical_cores + fileTrace.hyper_thread = args.hyper_thread + + if(fileTrace.hyper_thread == 'OFF'): + print("!!!!disabling stage2 HT!!!!") + enable_stage2_traceout = 0 + enable_stage2_fileout = 0 + elif(fileTrace.hyper_thread == 'ON'): + print("!!!!HT enabled. disabling stage1 file generation.!!!!") + enable_stage1_fileout = 0 + #endif + + if(args.no_output_file == True): + print("!!!!disabling stage1 and stage2 fileout!!!!") + enable_stage1_fileout = 0 + enable_stage2_fileout = 0 + #endif + + scanconfigfile(fileTrace) + +#end main -- 2.5.5