From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id 443F114EC for ; Tue, 6 Nov 2018 09:49:21 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Nov 2018 00:49:20 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,471,1534834800"; d="scan'208";a="87023527" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by orsmga007.jf.intel.com with ESMTP; 06 Nov 2018 00:49:19 -0800 Received: from fmsmsx126.amr.corp.intel.com (10.18.125.43) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.408.0; Tue, 6 Nov 2018 00:49:19 -0800 Received: from lcsmsx153.ger.corp.intel.com (10.186.165.228) by FMSMSX126.amr.corp.intel.com (10.18.125.43) with Microsoft SMTP Server (TLS) id 14.3.408.0; Tue, 6 Nov 2018 00:49:19 -0800 Received: from hasmsx112.ger.corp.intel.com ([169.254.11.97]) by LCSMSX153.ger.corp.intel.com ([169.254.8.236]) with mapi id 14.03.0415.000; Tue, 6 Nov 2018 10:49:18 +0200 From: "Jozwiak, TomaszX" To: "Verma, Shally" , "dev@dpdk.org" , "Trahe, Fiona" , "akhil.goyal@nxp.com" Thread-Topic: [PATCH v2 2/3] app/compress-perf: add performance measurement Thread-Index: AQHUcpCjZhIemjb16Uuf+VxWSS/SKaVAxD6AgAGswOA= Date: Tue, 6 Nov 2018 08:49:17 +0000 Message-ID: References: <1538400427-20164-1-git-send-email-tomaszx.jozwiak@intel.com> <1541151842-8746-1-git-send-email-tomaszx.jozwiak@intel.com> <1541151842-8746-3-git-send-email-tomaszx.jozwiak@intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-version: 11.0.400.15 dlp-reaction: no-action x-originating-ip: [10.104.12.169] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v2 2/3] app/compress-perf: add performance measurement X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2018 08:49:22 -0000 > -----Original Message----- > From: Verma, Shally [mailto:Shally.Verma@cavium.com] > Sent: Monday, November 5, 2018 9:57 AM > To: Jozwiak, TomaszX ; dev@dpdk.org; Trahe, > Fiona ; akhil.goyal@nxp.com > Subject: RE: [PATCH v2 2/3] app/compress-perf: add performance > measurement >=20 >=20 >=20 > >-----Original Message----- > >From: Tomasz Jozwiak > >Sent: 02 November 2018 15:14 > >To: dev@dpdk.org; fiona.trahe@intel.com; tomaszx.jozwiak@intel.com; > >Verma, Shally ; akhil.goyal@nxp.com > >Subject: [PATCH v2 2/3] app/compress-perf: add performance > measurement > > > >External Email > > > >Added performance measurement part into compression perf. test. > > > >Signed-off-by: De Lara Guarch, Pablo > >Signed-off-by: Tomasz Jozwiak > >--- > > app/test-compress-perf/comp_perf_options_parse.c | 8 +- > > app/test-compress-perf/main.c | 886 > ++++++++++++++++++++++- > > 2 files changed, 883 insertions(+), 11 deletions(-) > > > >diff --git a/app/test-compress-perf/comp_perf_options_parse.c > >b/app/test-compress-perf/comp_perf_options_parse.c > >index bef4d2f..e5da3ad 100644 > >--- a/app/test-compress-perf/comp_perf_options_parse.c > >+++ b/app/test-compress-perf/comp_perf_options_parse.c > >@@ -431,10 +431,6 @@ parse_huffman_enc(struct comp_test_data > *test_data, const char *arg) > > { > > "fixed", > > RTE_COMP_HUFFMAN_FIXED > >- }, > >- { > >- "dynamic", > >- RTE_COMP_HUFFMAN_DYNAMIC > > } > > }; > > > >@@ -569,9 +565,9 @@ comp_perf_options_default(struct comp_test_data > *test_data) > > test_data->seg_sz =3D 2048; > > test_data->burst_sz =3D 32; > > test_data->pool_sz =3D 8192; > >- test_data->max_sgl_segs =3D UINT16_MAX; > >+ test_data->max_sgl_segs =3D 16; > > test_data->num_iter =3D 10000; > >- test_data->huffman_enc =3D RTE_COMP_HUFFMAN_DYNAMIC; > >+ test_data->huffman_enc =3D RTE_COMP_HUFFMAN_FIXED; > > test_data->test_op =3D COMPRESS_DECOMPRESS; > > test_data->window_sz =3D -1; > > test_data->level.min =3D 1; > >diff --git a/app/test-compress-perf/main.c > >b/app/test-compress-perf/main.c index f52b98d..e3f4bf6 100644 > >--- a/app/test-compress-perf/main.c > >+++ b/app/test-compress-perf/main.c > >@@ -5,14 +5,728 @@ > > #include > > #include > > #include > >+#include > > #include > > > > #include "comp_perf_options.h" > > > >+#define NUM_MAX_XFORMS 16 > >+#define NUM_MAX_INFLIGHT_OPS 512 > >+#define EXPANSE_RATIO 1.05 > >+#define MIN_ISAL_SIZE 8 > Can we avoid ISAL specific naming ? TJ: yes true :) will be fixed in V3 > >+ > >+#define DIV_CEIL(a, b) ((a) / (b) + ((a) % (b) !=3D 0)) > >+ > >+/* Cleanup state machine */ > >+static enum cleanup_st { > >+ ST_CLEAR =3D 0, > >+ ST_TEST_DATA, > >+ ST_COMPDEV, > >+ ST_INPUT_DATA, > >+ ST_MEMORY_ALLOC, > >+ ST_PREPARE_BUF, > >+ ST_DURING_TEST > >+} cleanup =3D ST_CLEAR; > >+ > >+static int > >+param_range_check(uint16_t size, const struct rte_param_log2_range > >+*range) { > >+ unsigned int next_size; > >+ > >+ /* Check lower/upper bounds */ > >+ if (size < range->min) > >+ return -1; > >+ > >+ if (size > range->max) > >+ return -1; > >+ > >+ /* If range is actually only one value, size is correct */ > >+ if (range->increment =3D=3D 0) > >+ return 0; > >+ > >+ /* Check if value is one of the supported sizes */ > >+ for (next_size =3D range->min; next_size <=3D range->max; > >+ next_size +=3D range->increment) > >+ if (size =3D=3D next_size) > >+ return 0; > >+ > >+ return -1; > >+} > >+ > >+static int > >+comp_perf_check_capabilities(struct comp_test_data *test_data) { > >+ const struct rte_compressdev_capabilities *cap; > >+ > >+ cap =3D rte_compressdev_capability_get(test_data->cdev_id, > >+ RTE_COMP_ALGO_DEFLATE); > >+ > >+ if (cap =3D=3D NULL) { > >+ RTE_LOG(ERR, USER1, > >+ "Compress device does not support DEFLATE\n"); > >+ return -1; > >+ } > >+ > >+ uint64_t comp_flags =3D cap->comp_feature_flags; > >+ > >+ /* Huffman enconding */ > >+ if (test_data->huffman_enc =3D=3D RTE_COMP_HUFFMAN_FIXED && > >+ (comp_flags & RTE_COMP_FF_HUFFMAN_FIXED) =3D=3D = 0) { > >+ RTE_LOG(ERR, USER1, > >+ "Compress device does not supported Fixed Huffma= n\n"); > >+ return -1; > >+ } > >+ > >+ if (test_data->huffman_enc =3D=3D RTE_COMP_HUFFMAN_DYNAMIC && > >+ (comp_flags & RTE_COMP_FF_HUFFMAN_DYNAMIC) =3D= =3D 0) { > >+ RTE_LOG(ERR, USER1, > >+ "Compress device does not supported Dynamic Huff= man\n"); > >+ return -1; > >+ } > >+ > >+ /* Window size */ > >+ if (test_data->window_sz !=3D -1) { > >+ if (param_range_check(test_data->window_sz, &cap- > >window_size) > >+ < 0) { > >+ RTE_LOG(ERR, USER1, > >+ "Compress device does not support " > >+ "this window size\n"); > >+ return -1; > >+ } > >+ } else > >+ /* Set window size to PMD maximum if none was specified = */ > >+ test_data->window_sz =3D cap->window_size.max; > >+ > >+ /* Check if chained mbufs is supported */ > >+ if (test_data->max_sgl_segs > 1 && > >+ (comp_flags & RTE_COMP_FF_OOP_SGL_IN_SGL_OUT) = =3D=3D 0) { > >+ RTE_LOG(INFO, USER1, "Compress device does not support " > >+ "chained mbufs. Max SGL segments set to = 1\n"); > >+ test_data->max_sgl_segs =3D 1; > >+ } > >+ > >+ /* Level 0 support */ > >+ if (test_data->level.min =3D=3D 0 && > >+ (comp_flags & RTE_COMP_FF_NONCOMPRESSED_BLOCKS) = =3D=3D > 0) { > >+ RTE_LOG(ERR, USER1, "Compress device does not support " > >+ "level 0 (no compression)\n"); > >+ return -1; > >+ } > >+ > >+ return 0; > >+} > >+ > >+static int > >+comp_perf_allocate_memory(struct comp_test_data *test_data) { > >+ /* Number of segments for input and output > >+ * (compression and decompression) > >+ */ > >+ uint32_t total_segs =3D DIV_CEIL(test_data->input_data_sz, > >+ test_data->seg_sz); > >+ test_data->comp_buf_pool =3D > rte_pktmbuf_pool_create("comp_buf_pool", > >+ total_segs, > >+ 0, 0, test_data->seg_sz + RTE_PKTMBUF_HE= ADROOM, > >+ rte_socket_id()); > >+ if (test_data->comp_buf_pool =3D=3D NULL) { > >+ RTE_LOG(ERR, USER1, "Mbuf mempool could not be created\n= "); > >+ return -1; > >+ } > >+ > >+ cleanup =3D ST_MEMORY_ALLOC; > >+ test_data->decomp_buf_pool =3D > rte_pktmbuf_pool_create("decomp_buf_pool", > >+ total_segs, > >+ 0, 0, test_data->seg_sz + RTE_PKTMBUF_HE= ADROOM, > >+ rte_socket_id()); > >+ if (test_data->decomp_buf_pool =3D=3D NULL) { > >+ RTE_LOG(ERR, USER1, "Mbuf mempool could not be created\n= "); > >+ return -1; > >+ } > Unless am missing to see it, you need to free pre-allocated memories here > before return call for all failed cases. TJ: There's only one 'freeing stack' at the end of main application functio= n to avoid double freeing resources (which was previously n V1). We have state machine for that stuff (static enum cleanup_st) to know what = should be free and what has been allocated already. In case you mean the state machine is set just after first alloc in line 13= 6: cleanup =3D ST_MEMORY_ALLOC; so we know what should be free at the end of application running in line 89= 1: end: switch (cleanup) { case ST_DURING_TEST: case ST_PREPARE_BUF: free_bufs(test_data); /* fallthrough */ case ST_MEMORY_ALLOC: rte_free(test_data->decomp_bufs); rte_free(test_data->comp_bufs); rte_free(test_data->decompressed_data); rte_free(test_data->compressed_data); rte_mempool_free(test_data->op_pool); rte_mempool_free(test_data->decomp_buf_pool); rte_mempool_free(test_data->comp_buf_pool); /* fallthrough */ case ST_INPUT_DATA: rte_free(test_data->input_data); /* fallthrough */ case ST_COMPDEV: if (test_data->cdev_id !=3D -1) rte_compressdev_stop(test_data->cdev_id); /* fallthrough */ case ST_TEST_DATA: rte_free(test_data); /* fallthrough */ case ST_CLEAR: default: i =3D rte_eal_cleanup(); if (i) { RTE_LOG(ERR, USER1, "Error from rte_eal_cleanup(), %d\n", i); ret =3D i; } break; } return ret; >=20 > >+ > >+ test_data->total_bufs =3D DIV_CEIL(total_segs, > >+ test_data->max_sgl_segs); > >+ > >+ test_data->op_pool =3D rte_comp_op_pool_create("op_pool", > >+ test_data->total_bufs, > >+ 0, 0, rte_socket_id()); > >+ if (test_data->op_pool =3D=3D NULL) { > >+ RTE_LOG(ERR, USER1, "Comp op mempool could not be > created\n"); > >+ return -1; > >+ } > >+ > >+ /* > >+ * Compressed data might be a bit larger than input data, > >+ * if data cannot be compressed > >+ */ > >+ test_data->compressed_data =3D rte_zmalloc_socket(NULL, > >+ test_data->input_data_sz * EXPANSE_RATIO > >+ + > >+ MIN_ISAL_SIZE, 0, > MIN_ISAL_SIZE looks specific to ISAL driver. if so, then is this perf app= specific > to that PMD? or Can we make it somewhat generic? TJ: True will be fixed >=20 > >+ rte_socket_id()); > >+ if (test_data->compressed_data =3D=3D NULL) { > >+ RTE_LOG(ERR, USER1, "Memory to hold the data from the in= put " > >+ "file could not be allocated\n"); > >+ return -1; > >+ } > >+ > >+ test_data->decompressed_data =3D rte_zmalloc_socket(NULL, > >+ test_data->input_data_sz, 0, > >+ rte_socket_id()); > >+ if (test_data->decompressed_data =3D=3D NULL) { > >+ RTE_LOG(ERR, USER1, "Memory to hold the data from the in= put " > >+ "file could not be allocated\n"); > >+ return -1; > >+ } > >+ > >+ test_data->comp_bufs =3D rte_zmalloc_socket(NULL, > >+ test_data->total_bufs * sizeof(struct rte_mbuf *= ), > >+ 0, rte_socket_id()); > >+ if (test_data->comp_bufs =3D=3D NULL) { > >+ RTE_LOG(ERR, USER1, "Memory to hold the compression mbuf= s" > >+ " could not be allocated\n"); > >+ return -1; > >+ } > >+ > >+ test_data->decomp_bufs =3D rte_zmalloc_socket(NULL, > >+ test_data->total_bufs * sizeof(struct rte_mbuf *= ), > >+ 0, rte_socket_id()); > >+ if (test_data->decomp_bufs =3D=3D NULL) { > >+ RTE_LOG(ERR, USER1, "Memory to hold the decompression > mbufs" > >+ " could not be allocated\n"); > >+ return -1; > >+ } > >+ return 0; > >+} > >+ > >+static int > >+comp_perf_dump_input_data(struct comp_test_data *test_data) { > >+ FILE *f =3D fopen(test_data->input_file, "r"); > >+ int ret =3D -1; > >+ > >+ if (f =3D=3D NULL) { > >+ RTE_LOG(ERR, USER1, "Input file could not be opened\n"); > >+ return -1; > >+ } > >+ > >+ if (fseek(f, 0, SEEK_END) !=3D 0) { > >+ RTE_LOG(ERR, USER1, "Size of input could not be calculat= ed\n"); > >+ goto end; > >+ } > >+ size_t actual_file_sz =3D ftell(f); > >+ /* If extended input data size has not been set, > >+ * input data size =3D file size > >+ */ > >+ > >+ if (test_data->input_data_sz =3D=3D 0) > >+ test_data->input_data_sz =3D actual_file_sz; > >+ > >+ if (fseek(f, 0, SEEK_SET) !=3D 0) { > >+ RTE_LOG(ERR, USER1, "Size of input could not be calculat= ed\n"); > >+ goto end; > >+ } > >+ > >+ test_data->input_data =3D rte_zmalloc_socket(NULL, > >+ test_data->input_data_sz, 0, > >+ rte_socket_id()); > >+ > >+ if (test_data->input_data =3D=3D NULL) { > >+ RTE_LOG(ERR, USER1, "Memory to hold the data from the in= put " > >+ "file could not be allocated\n"); > >+ goto end; > >+ } > >+ > >+ size_t remaining_data =3D test_data->input_data_sz; > >+ uint8_t *data =3D test_data->input_data; > >+ > >+ while (remaining_data > 0) { > >+ size_t data_to_read =3D RTE_MIN(remaining_data, > >+ actual_file_sz); > >+ > >+ if (fread(data, data_to_read, 1, f) !=3D 1) { > >+ RTE_LOG(ERR, USER1, "Input file could not be rea= d\n"); > >+ goto end; > >+ } > >+ if (fseek(f, 0, SEEK_SET) !=3D 0) { > >+ RTE_LOG(ERR, USER1, > >+ "Size of input could not be calculated\n= "); > >+ goto end; > >+ } > >+ remaining_data -=3D data_to_read; > >+ data +=3D data_to_read; > >+ } > >+ > >+ if (test_data->input_data_sz > actual_file_sz) > >+ RTE_LOG(INFO, USER1, > >+ "%zu bytes read from file %s, extending the file %.2f = times\n", > >+ test_data->input_data_sz, test_data->input_file, > >+ (double)test_data->input_data_sz/actual_file_sz)= ; > >+ else > >+ RTE_LOG(INFO, USER1, > >+ "%zu bytes read from file %s\n", > >+ test_data->input_data_sz, > >+ test_data->input_file); > >+ > >+ ret =3D 0; > >+ > >+end: > >+ fclose(f); > >+ return ret; > >+} > >+ > >+static int > >+comp_perf_initialize_compressdev(struct comp_test_data *test_data) { > >+ uint8_t enabled_cdev_count; > >+ uint8_t enabled_cdevs[RTE_COMPRESS_MAX_DEVS]; > >+ > >+ enabled_cdev_count =3D rte_compressdev_devices_get(test_data- > >driver_name, > >+ enabled_cdevs, RTE_COMPRESS_MAX_DEVS); > >+ if (enabled_cdev_count =3D=3D 0) { > >+ RTE_LOG(ERR, USER1, "No compress devices type %s availab= le\n", > >+ test_data->driver_name); > >+ return -EINVAL; > >+ } > >+ > >+ if (enabled_cdev_count > 1) > >+ RTE_LOG(INFO, USER1, > >+ "Only the first compress device will be > >+ used\n"); > >+ > >+ test_data->cdev_id =3D enabled_cdevs[0]; > >+ > >+ if (comp_perf_check_capabilities(test_data) < 0) > >+ return -1; > >+ > >+ /* Configure compressdev (one device, one queue pair) */ > >+ struct rte_compressdev_config config =3D { > >+ .socket_id =3D rte_socket_id(), > >+ .nb_queue_pairs =3D 1, > >+ .max_nb_priv_xforms =3D NUM_MAX_XFORMS, > >+ .max_nb_streams =3D 0 > >+ }; > >+ > >+ if (rte_compressdev_configure(test_data->cdev_id, &config) < 0) = { > >+ RTE_LOG(ERR, USER1, "Device configuration failed\n"); > >+ return -1; > >+ } > >+ > >+ if (rte_compressdev_queue_pair_setup(test_data->cdev_id, 0, > >+ NUM_MAX_INFLIGHT_OPS, rte_socket_id()) < 0) { > >+ RTE_LOG(ERR, USER1, "Queue pair setup failed\n"); > >+ return -1; > >+ } > >+ > >+ if (rte_compressdev_start(test_data->cdev_id) < 0) { > >+ RTE_LOG(ERR, USER1, "Device could not be started\n"); > >+ return -1; > >+ } > >+ > >+ return 0; > >+} > >+ > >+static int > >+prepare_bufs(struct comp_test_data *test_data) { > >+ uint32_t remaining_data =3D test_data->input_data_sz; > >+ uint8_t *input_data_ptr =3D test_data->input_data; > >+ size_t data_sz; > >+ uint8_t *data_addr; > >+ uint32_t i, j; > >+ > >+ for (i =3D 0; i < test_data->total_bufs; i++) { > >+ /* Allocate data in input mbuf and copy data from input = file */ > >+ test_data->decomp_bufs[i] =3D > >+ rte_pktmbuf_alloc(test_data->decomp_buf_pool); > >+ if (test_data->decomp_bufs[i] =3D=3D NULL) { > >+ RTE_LOG(ERR, USER1, "Could not allocate mbuf\n")= ; > >+ return -1; > >+ } > >+ > >+ cleanup =3D ST_PREPARE_BUF; > >+ data_sz =3D RTE_MIN(remaining_data, test_data->seg_sz); > >+ data_addr =3D (uint8_t *) rte_pktmbuf_append( > >+ test_data->decomp_bufs[i], data_= sz); > >+ if (data_addr =3D=3D NULL) { > >+ RTE_LOG(ERR, USER1, "Could not append data\n"); > So is here ..free of allocated buffer before return from failed cases >=20 > Thanks > Shally Thx, Tomek