From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 5ECA3239 for ; Tue, 6 Nov 2018 10:06:01 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Nov 2018 01:06:00 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,471,1534834800"; d="scan'208";a="97985202" Received: from fmsmsx104.amr.corp.intel.com ([10.18.124.202]) by orsmga003.jf.intel.com with ESMTP; 06 Nov 2018 01:06:00 -0800 Received: from fmsmsx126.amr.corp.intel.com (10.18.125.43) by fmsmsx104.amr.corp.intel.com (10.18.124.202) with Microsoft SMTP Server (TLS) id 14.3.408.0; Tue, 6 Nov 2018 01:05:59 -0800 Received: from hasmsx108.ger.corp.intel.com (10.184.198.18) by FMSMSX126.amr.corp.intel.com (10.18.125.43) with Microsoft SMTP Server (TLS) id 14.3.408.0; Tue, 6 Nov 2018 01:05:59 -0800 Received: from hasmsx112.ger.corp.intel.com ([169.254.11.97]) by hasmsx108.ger.corp.intel.com ([169.254.9.165]) with mapi id 14.03.0415.000; Tue, 6 Nov 2018 11:05:56 +0200 From: "Jozwiak, TomaszX" To: "Verma, Shally" , "dev@dpdk.org" , "Trahe, Fiona" , "akhil.goyal@nxp.com" Thread-Topic: [dpdk-dev] [PATCH 2/3] app/compress-perf: add performance measurement Thread-Index: AQHUYhSjn5ZjkvYiqUCGRBSCtwq5y6U2CV0AgArVnACAAGXs8IABJz4AgAAtn8A= Date: Tue, 6 Nov 2018 09:05:56 +0000 Message-ID: References: <1538400427-20164-1-git-send-email-tomaszx.jozwiak@intel.com> <1538400427-20164-3-git-send-email-tomaszx.jozwiak@intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-version: 11.0.400.15 dlp-reaction: no-action x-originating-ip: [10.104.12.169] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH 2/3] app/compress-perf: add performance measurement X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2018 09:06:02 -0000 > -----Original Message----- > From: Verma, Shally [mailto:Shally.Verma@cavium.com] > Sent: Tuesday, November 6, 2018 9:16 AM > To: Jozwiak, TomaszX ; dev@dpdk.org; Trahe, > Fiona ; akhil.goyal@nxp.com > Subject: RE: [dpdk-dev] [PATCH 2/3] app/compress-perf: add performance > measurement >=20 >=20 >=20 > >-----Original Message----- > >From: Jozwiak, TomaszX > >Sent: 06 November 2018 13:34 > >To: Verma, Shally ; dev@dpdk.org; Trahe, > Fiona > >; akhil.goyal@nxp.com > >Subject: RE: [dpdk-dev] [PATCH 2/3] app/compress-perf: add performance > >measurement > > > >External Email > > > >Hi Shally, > > > >Please see my comment inline. > > > >> -----Original Message----- > >> From: Verma, Shally [mailto:Shally.Verma@cavium.com] > >> Sent: Monday, November 5, 2018 9:34 AM > >> To: Jozwiak, TomaszX ; dev@dpdk.org; > >> Trahe, Fiona ; akhil.goyal@nxp.com; De Lara > >> Guarch, Pablo > >> Cc: De@dpdk.org; Lara@dpdk.org; Guarch@dpdk.org > >> Subject: RE: [dpdk-dev] [PATCH 2/3] app/compress-perf: add > >> performance measurement > >> > >> > >> > >> >-----Original Message----- > >> >From: Jozwiak, TomaszX > >> >Sent: 02 November 2018 15:29 > >> >To: dev@dpdk.org; Trahe, Fiona ; > >> >akhil.goyal@nxp.com; Verma, Shally ; De > >> >Lara Guarch, Pablo > >> >Cc: De@dpdk.org; Lara@dpdk.org; Guarch@dpdk.org > >> >Subject: RE: [dpdk-dev] [PATCH 2/3] app/compress-perf: add > >> >performance measurement > >> > > >> >External Email > >> > > >> >Hi Shally, > >> > > >> >Sorry for delay - I was on sick leave. > >> >We had some issues with dynamic compression test so I block this > >> >test in V2. May be there's too late to add this into this release > >> >but we've decided > >> to send this V2 to DPDK. > >> > > >> >My comment inline (not all have answer so far, still working on > >> >that) > >> > > >> >> -----Original Message----- > >> >> From: Verma, Shally [mailto:Shally.Verma@cavium.com] > >> >> Sent: Friday, October 12, 2018 12:16 PM > >> >> To: Jozwiak, TomaszX ; dev@dpdk.org; > >> >> Trahe, Fiona ; akhil.goyal@nxp.com; De Lara > >> >> Guarch, Pablo > >> >> Cc: De@dpdk.org; Lara@dpdk.org; Guarch@dpdk.org > >> >> Subject: RE: [dpdk-dev] [PATCH 2/3] app/compress-perf: add > >> >> performance measurement > >> >> > >> >> HI TomaszX > >> >> > >> >> Sorry for delay in response. Comments inline. > >> >> > >> >> >-----Original Message----- > >> >> >From: dev On Behalf Of Tomasz Jozwiak > >> >> >Sent: 01 October 2018 18:57 > >> >> >To: dev@dpdk.org; fiona.trahe@intel.com; > >> >> >tomaszx.jozwiak@intel.com; akhil.goyal@nxp.com; > >> >> >pablo.de.lara.guarch@intel.com > >> >> >Cc: De@dpdk.org; Lara@dpdk.org; Guarch@dpdk.org > >> >> >Subject: [dpdk-dev] [PATCH 2/3] app/compress-perf: add > >> >> >performance measurement > >> >> > > >> >> >External Email > >> >> > > >> >> >Added performance measurement part into compression perf. test. > >> >> > > >> >> >Signed-off-by: De Lara Guarch, Pablo > >> >> > > >> >> >Signed-off-by: Tomasz Jozwiak > >> >> >--- > >> >> > app/test-compress-perf/main.c | 844 > >> >> >++++++++++++++++++++++++++++++++++++++++++ > >> >> > 1 file changed, 844 insertions(+) > >> >> > > >> >> >diff --git a/app/test-compress-perf/main.c > >> >> >b/app/test-compress-perf/main.c index f52b98d..093dfaf 100644 > >> >> >--- a/app/test-compress-perf/main.c > >> >> >+++ b/app/test-compress-perf/main.c > >> >> >@@ -5,13 +5,721 @@ > >> >> > #include > >> >> > #include > >> >> > #include > >> >> >+#include > >> >> > #include > >> >> > > >> >> > #include "comp_perf_options.h" > >> >> > > >> >> >+#define NUM_MAX_XFORMS 16 > >> >> >+#define NUM_MAX_INFLIGHT_OPS 512 #define EXPANSE_RATIO > 1.05 > >> >> >+#define MIN_ISAL_SIZE 8 > >> >> >+ > >> >> >+#define DIV_CEIL(a, b) ((a) / (b) + ((a) % (b) !=3D 0)) > >> >> >+ > >> >> >+static int > >> >> >+param_range_check(uint16_t size, const struct > >> >> >+rte_param_log2_range > >> >> >+*range) { > >> >> >+ unsigned int next_size; > >> >> >+ > >> >> >+ /* Check lower/upper bounds */ > >> >> >+ if (size < range->min) > >> >> >+ return -1; > >> >> >+ > >> >> >+ if (size > range->max) > >> >> >+ return -1; > >> >> >+ > >> >> >+ /* If range is actually only one value, size is correct */ > >> >> >+ if (range->increment =3D=3D 0) > >> >> >+ return 0; > >> >> >+ > >> >> >+ /* Check if value is one of the supported sizes */ > >> >> >+ for (next_size =3D range->min; next_size <=3D range->max; > >> >> >+ next_size +=3D range->increment) > >> >> >+ if (size =3D=3D next_size) > >> >> >+ return 0; > >> >> >+ > >> >> >+ return -1; > >> >> >+} > >> >> >+ > >> >> >+static int > >> >> >+comp_perf_check_capabilities(struct comp_test_data *test_data) { > >> >> >+ const struct rte_compressdev_capabilities *cap; > >> >> >+ > >> >> >+ cap =3D rte_compressdev_capability_get(test_data->cdev_id, > >> >> >+ > >> >> >+ RTE_COMP_ALGO_DEFLATE); > >> >> >+ > >> >> >+ if (cap =3D=3D NULL) { > >> >> >+ RTE_LOG(ERR, USER1, > >> >> >+ "Compress device does not support DEFLATE\= n"); > >> >> >+ return -1; > >> >> >+ } > >> >> >+ > >> >> >+ uint64_t comp_flags =3D cap->comp_feature_flags; > >> >> >+ > >> >> >+ /* Huffman enconding */ > >> >> >+ if (test_data->huffman_enc =3D=3D RTE_COMP_HUFFMAN_FIXED > && > >> >> >+ (comp_flags & RTE_COMP_FF_HUFFMAN_FIXED) = =3D=3D 0) { > >> >> >+ RTE_LOG(ERR, USER1, > >> >> >+ "Compress device does not supported Fixed > Huffman\n"); > >> >> >+ return -1; > >> >> >+ } > >> >> >+ > >> >> >+ if (test_data->huffman_enc =3D=3D > RTE_COMP_HUFFMAN_DYNAMIC > >> && > >> >> >+ (comp_flags & > >> >> >+ RTE_COMP_FF_HUFFMAN_DYNAMIC) =3D=3D 0) > >> { > >> >> >+ RTE_LOG(ERR, USER1, > >> >> >+ "Compress device does not supported > >> >> >+ Dynamic > >> Huffman\n"); > >> >> >+ return -1; > >> >> >+ } > >> >> >+ > >> >> >+ /* Window size */ > >> >> >+ if (test_data->window_sz !=3D -1) { > >> >> >+ if (param_range_check(test_data->window_sz, > >> >> >+ &cap->window_size) > >> >> What if cap->window_size is 0 i.e. implementation default? > >> > > >> >TJ: You probably mean cap->window_size.increment =3D 0 (because > >> >cap->window_size is a structure). In that case we check if > >> >test_data->window_sz >=3Dmin and test_data->window_sz <=3D max only, > >> because increment =3D 0 means (base on compression API) we have only > >> one value of windows_size (no range is supported). > >> But PMD can set min and max too 0 for such case. > > > >TJ: I can't see any issue in that case too. Maybe I don't understand wha= t you > mean but the logic is as follow: > >1) if you pass '--window-sz ...' param. into command line your > >intention is to force that value of window size during test. We check is= this > value is allow (by param_range_check() function). > >2) if you plan to use default value - just don't pass '--window-sz' > >param. in command line at all. In that case we get windows size from > >window_size.max field, so if window_size.min=3D window_size.max=3D0 > test_data->window_sz will be zero, as well. > >If you mean that behavior is not good - I will be grateful for other > suggestions. >=20 > This is fine. but I am thinking of 3rd case here: > c) user pass window sz but PMD window_sz.min =3D max =3D 0, then user > requested windowsz is not applicable right?! In that case - true. There'll be fail : "Compress device does not support this window size\n"); So what is your proposal for that case? >=20 > > > >> > >> > > >> > > >> > > >> .... > >> > >> >> >+ > >> >> >+ if (fread(data, data_to_read, 1, f) !=3D 1) { > >> >> >+ RTE_LOG(ERR, USER1, "Input file could not = be read\n"); > >> >> >+ goto err; > >> >> >+ } > >> >> >+ if (fseek(f, 0, SEEK_SET) !=3D 0) { > >> >> >+ RTE_LOG(ERR, USER1, > >> >> >+ "Size of input could not be calcul= ated\n"); > >> >> >+ goto err; > >> >> >+ } > >> >> >+ remaining_data -=3D data_to_read; > >> >> >+ data +=3D data_to_read; > >> >> It looks like it will run 2nd time only if input file size < input > >> >> data size in which case it will just keep filling input buffer > >> >> with repeated > >> data. > >> >> Is that the intention here? > >> > > >> >TJ: Yes exactly. If test_data->input_data_sz is bigger than > >> >actual_file_sz then we fill the buffer with repeated data from file > >> >to fill > >> whole buffer. > >> I mentioned in one of the earlier reply, wont that then influence the > >> compression behaviour and o/p? my suggestion was to work on actual > >> user provided input to take perf to get actual perf for given content. > > > >TJ: You right, but this solution is flexible. You can pass ' > >--extended-input-sz" or not, so you can use original input data or exten= d it > if you want. > Ok. but still not sure if it's really needed. Might be practically most o= f the time > it wont be exercised. No hard opinion on this though. >=20 > Thanks > Shally > > > >> > >> > > >> >> > >> ... > >> > >> >> >+ if (data_addr =3D=3D NULL) { > >> >> >+ RTE_LOG(ERR, USER1, "Could not > >> >> >+ append data\n"); > >> >> Since a new buffer per segment is allocated, so is it possible for > >> >> append to fail? think, this check is redundant here. > >> > > >> >TJ: Yes, you're right, it should never fail. But I think it's good > >> >coding practice > >> to add the check just in case. > >> > > >> Unless it is called in data path which might cost perf a bit. > > > >TJ: prepare_bufs() is out of perf measurement, so shouldn't impact to > >measurements. The performance measurement is inside > >main_loop() only. > > > > > >Br, Tomek > > > >> > >> Thanks > >> Shally > >> > >> >> >+ return -1; > >> >> >+ } > >> >> >+ > >> >> >+ rte_memcpy(data_addr, input_data_ptr, data= _sz); > >> >> >+ input_data_ptr +=3D data_sz; > >> >> >+ remaining_data -=3D data_sz; > >> >> >+ > >> >> >+ if (rte_pktmbuf_chain(test_data->decomp_bu= fs[i], > >> >> >+ next_seg) < 0) { > >> >> >+ RTE_LOG(ERR, USER1, "Could not cha= in mbufs\n"); > >> >> >+ return -1; > >> >> >+ } > >> >> >+ segs_per_mbuf++; > >> >> >+ } > >> >> >+ > >> >> >+ /* Allocate data in output mbuf */ > >> >> >+ test_data->comp_bufs[i] =3D > >> >> >+ rte_pktmbuf_alloc(test_data->comp_buf_pool= ); > >> >> >+ if (test_data->comp_bufs[i] =3D=3D NULL) { > >> >> >+ RTE_LOG(ERR, USER1, "Could not allocate mb= uf\n"); > >> >> >+ return -1; > >> >> >+ } > >> >> >+ data_addr =3D (uint8_t *) rte_pktmbuf_append( > >> >> >+ test_data->comp_bufs[i], > >> >> >+ test_data->seg_sz); > >> >> >+ if (data_addr =3D=3D NULL) { > >> >> >+ RTE_LOG(ERR, USER1, "Could not append data= \n"); > >> >> >+ return -1; > >> >> >+ } > >> >> >+ > >> >> >+ /* Chain mbufs if needed for output mbufs */ > >> >> >+ for (j =3D 1; j < segs_per_mbuf; j++) { > >> >> >+ struct rte_mbuf *next_seg =3D > >> >> >+ > >> >> >+ rte_pktmbuf_alloc(test_data->comp_buf_pool); > >> >> >+ > >> >> >+ if (next_seg =3D=3D NULL) { > >> >> >+ RTE_LOG(ERR, USER1, > >> >> >+ "Could not allocate mbuf\n= "); > >> >> >+ return -1; > >> >> >+ } > >> >> >+ > >> >> >+ data_addr =3D (uint8_t *)rte_pktmbuf_appen= d(next_seg, > >> >> >+ test_data->seg_sz); > >> >> >+ > >> >> >+ if (data_addr =3D=3D NULL) { > >> >> >+ RTE_LOG(ERR, USER1, "Could not app= end data\n"); > >> >> >+ return -1; > >> >> >+ } > >> >> >+ > >> >> >+ if (rte_pktmbuf_chain(test_data->comp_bufs= [i], > >> >> >+ next_seg) < 0) { > >> >> >+ RTE_LOG(ERR, USER1, "Could not cha= in mbufs\n"); > >> >> >+ return -1; > >> >> >+ } > >> >> >+ } > >> >> >+ } > >> >> >+ > >> >> >+ return 0; > >> >> >+} > >> >> >+ > >> >> >+static void > >> >> >+free_bufs(struct comp_test_data *test_data) { > >> >> >+ uint32_t i; > >> >> >+ > >> >> >+ for (i =3D 0; i < test_data->total_bufs; i++) { > >> >> >+ rte_pktmbuf_free(test_data->comp_bufs[i]); > >> >> >+ rte_pktmbuf_free(test_data->decomp_bufs[i]); > >> >> >+ } > >> >> >+ rte_free(test_data->comp_bufs); > >> >> >+ rte_free(test_data->decomp_bufs); } > >> >> >+ > >> >> >+static int > >> >> >+main_loop(struct comp_test_data *test_data, uint8_t level, > >> >> >+ enum rte_comp_xform_type type, > >> >> >+ uint8_t *output_data_ptr, > >> >> >+ size_t *output_data_sz, > >> >> >+ unsigned int benchmarking) { > >> >> >+ uint8_t dev_id =3D test_data->cdev_id; > >> >> >+ uint32_t i, iter, num_iter; > >> >> >+ struct rte_comp_op **ops, **deq_ops; > >> >> >+ void *priv_xform =3D NULL; > >> >> >+ struct rte_comp_xform xform; > >> >> >+ size_t output_size =3D 0; > >> >> >+ struct rte_mbuf **input_bufs, **output_bufs; > >> >> >+ int res =3D 0; > >> >> >+ int allocated =3D 0; > >> >> >+ > >> >> >+ if (test_data =3D=3D NULL || !test_data->burst_sz) { > >> >> >+ RTE_LOG(ERR, USER1, > >> >> >+ "Unknow burst size\n"); > >> >> >+ return -1; > >> >> >+ } > >> >> >+ > >> >> >+ ops =3D rte_zmalloc_socket(NULL, > >> >> >+ 2 * test_data->total_bufs * sizeof(struct rte_comp= _op *), > >> >> >+ 0, rte_socket_id()); > >> >> >+ > >> >> >+ if (ops =3D=3D NULL) { > >> >> >+ RTE_LOG(ERR, USER1, > >> >> >+ "Can't allocate memory for ops strucures\n= "); > >> >> >+ return -1; > >> >> >+ } > >> >> >+ > >> >> >+ deq_ops =3D &ops[test_data->total_bufs]; > >> >> >+ > >> >> >+ if (type =3D=3D RTE_COMP_COMPRESS) { > >> >> >+ xform =3D (struct rte_comp_xform) { > >> >> >+ .type =3D RTE_COMP_COMPRESS, > >> >> >+ .compress =3D { > >> >> >+ .algo =3D RTE_COMP_ALGO_DEFLATE, > >> >> >+ .deflate.huffman =3D test_data->hu= ffman_enc, > >> >> >+ .level =3D level, > >> >> >+ .window_size =3D test_data->window= _sz, > >> >> >+ .chksum =3D RTE_COMP_CHECKSUM_NONE= , > >> >> >+ .hash_algo =3D RTE_COMP_HASH_ALGO_= NONE > >> >> >+ } > >> >> >+ }; > >> >> >+ input_bufs =3D test_data->decomp_bufs; > >> >> >+ output_bufs =3D test_data->comp_bufs; > >> >> >+ } else { > >> >> >+ xform =3D (struct rte_comp_xform) { > >> >> >+ .type =3D RTE_COMP_DECOMPRESS, > >> >> >+ .decompress =3D { > >> >> >+ .algo =3D RTE_COMP_ALGO_DEFLATE, > >> >> >+ .chksum =3D RTE_COMP_CHECKSUM_NONE= , > >> >> >+ .window_size =3D test_data->window= _sz, > >> >> >+ .hash_algo =3D RTE_COMP_HASH_ALGO_= NONE > >> >> >+ } > >> >> >+ }; > >> >> >+ input_bufs =3D test_data->comp_bufs; > >> >> >+ output_bufs =3D test_data->decomp_bufs; > >> >> >+ } > >> >> >+ > >> >> >+ /* Create private xform */ > >> >> >+ if (rte_compressdev_private_xform_create(dev_id, &xform, > >> >> >+ &priv_xform) < 0) { > >> >> >+ RTE_LOG(ERR, USER1, "Private xform could not be > created\n"); > >> >> >+ res =3D -1; > >> >> >+ goto end; > >> >> >+ } > >> >> >+ > >> >> >+ uint64_t tsc_start, tsc_end, tsc_duration; > >> >> >+ > >> >> >+ tsc_start =3D tsc_end =3D tsc_duration =3D 0; > >> >> >+ if (benchmarking) { > >> >> >+ tsc_start =3D rte_rdtsc(); > >> >> >+ num_iter =3D test_data->num_iter; > >> >> >+ } else > >> >> >+ num_iter =3D 1; > >> >> Looks like in same code we're doing benchmarking and functional > >> validation. > >> >> It can be reorganised to keep validation test separately like done > >> >> in crypto_perf. > >> > > >> >TJ: Ok, makes sense. However in the interests of getting this into > >> >the > >> >18.11 release I'd like to defer this refactoring and the remainder > >> >of your > >> comments below to the next release. > >> > > >> > > >> >Next comments - WIP > >> > > >> > > >> >Br, Tomek