* [PATCH 0/4] compressdev: add LZ4 support
@ 2023-01-09 7:45 Michael Baum
2023-01-09 7:45 ` [PATCH 1/4] compressdev: add LZ4 algorithm support Michael Baum
` (4 more replies)
0 siblings, 5 replies; 31+ messages in thread
From: Michael Baum @ 2023-01-09 7:45 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe, Thomas Monjalon
Add support for LZ4 compression algorithm for both API and app.
In addition, enable app to test de/comp only. This option already
exists in both app doc and arguments but isn't implemented inside the
application.
LZ4: https://github.com/lz4/lz4
RFC: https://patchwork.dpdk.org/project/dpdk/patch/20220410182622.8828-1-rzidane@nvidia.com/
Depends-on: patch-26344 ("compressdev: fix end of comp PMD list macro conflict")
Michael Baum (4):
compressdev: add LZ4 algorithm support
app/test-compress-perf: allow test single compress operation
app/test-compress-perf: add algo option
app/test-compress-perf: add LZ4 support
app/test-compress-perf/comp_perf_options.h | 9 +-
.../comp_perf_options_parse.c | 83 +++++++++++-
.../comp_perf_test_common.c | 126 +++++++++++++-----
.../comp_perf_test_cyclecount.c | 82 +++++++-----
.../comp_perf_test_throughput.c | 75 +++++++----
.../comp_perf_test_verify.c | 75 +++++++----
app/test-compress-perf/main.c | 63 +++++++--
doc/guides/compressdevs/features/default.ini | 4 +
doc/guides/rel_notes/release_23_03.rst | 16 +++
doc/guides/tools/comp_perf.rst | 5 +
lib/compressdev/rte_comp.c | 6 +
lib/compressdev/rte_comp.h | 51 ++++++-
12 files changed, 451 insertions(+), 144 deletions(-)
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH 1/4] compressdev: add LZ4 algorithm support
2023-01-09 7:45 [PATCH 0/4] compressdev: add LZ4 support Michael Baum
@ 2023-01-09 7:45 ` Michael Baum
2023-01-30 18:35 ` [EXT] " Akhil Goyal
2023-01-09 7:45 ` [PATCH 2/4] app/test-compress-perf: allow test single compress operation Michael Baum
` (3 subsequent siblings)
4 siblings, 1 reply; 31+ messages in thread
From: Michael Baum @ 2023-01-09 7:45 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe, Thomas Monjalon
Add support for LZ4 algorithm:
- Add Lz4 param structure to XFORM structures.
- Add capabilities flags for LZ4 params.
- Add xxHash-32 checksum and capabilities flag.
Signed-off-by: Michael Baum <michaelba@nvidia.com>
---
doc/guides/compressdevs/features/default.ini | 4 ++
doc/guides/rel_notes/release_23_03.rst | 7 +++
lib/compressdev/rte_comp.c | 6 +++
lib/compressdev/rte_comp.h | 51 +++++++++++++++++++-
4 files changed, 66 insertions(+), 2 deletions(-)
diff --git a/doc/guides/compressdevs/features/default.ini b/doc/guides/compressdevs/features/default.ini
index e1419ee8db..208b8591cf 100644
--- a/doc/guides/compressdevs/features/default.ini
+++ b/doc/guides/compressdevs/features/default.ini
@@ -20,8 +20,12 @@ OOP SGL In LB Out =
OOP LB In SGL Out =
Deflate =
LZS =
+LZ4 =
Adler32 =
Crc32 =
Adler32&Crc32 =
+xxHash32 =
Fixed =
Dynamic =
+LZ4 Block Checksum =
+LZ4 Block Independence =
diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index b8c5b68d6c..3bcf638544 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -55,6 +55,13 @@ New Features
Also, make sure to start the actual text at the margin.
=======================================================
+* **Added LZ4 algorithm in Compressdev Library.**
+
+ Added new compression algorithm, including:
+
+ * Added support for ``RTE_COMP_ALGO_LZ4``.
+ * Added support for ``RTE_COMP_CHECKSUM_XXHASH32``.
+
Removed Items
-------------
diff --git a/lib/compressdev/rte_comp.c b/lib/compressdev/rte_comp.c
index 320c6dab92..2ad05dbf01 100644
--- a/lib/compressdev/rte_comp.c
+++ b/lib/compressdev/rte_comp.c
@@ -39,6 +39,12 @@ rte_comp_get_feature_name(uint64_t flag)
return "HUFFMAN_FIXED";
case RTE_COMP_FF_HUFFMAN_DYNAMIC:
return "HUFFMAN_DYNAMIC";
+ case RTE_COMP_FF_XXHASH32_CHECKSUM:
+ return "XXHASH32_CHECKSUM";
+ case RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE:
+ return "LZ4_BLOCK_INDEPENDENCE";
+ case RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM:
+ return "LZ4_BLOCK_WITH_CHECKSUM";
default:
return NULL;
}
diff --git a/lib/compressdev/rte_comp.h b/lib/compressdev/rte_comp.h
index a8f398b57b..ec206d17cf 100644
--- a/lib/compressdev/rte_comp.h
+++ b/lib/compressdev/rte_comp.h
@@ -67,6 +67,12 @@ extern "C" {
/**< Fixed huffman encoding is supported */
#define RTE_COMP_FF_HUFFMAN_DYNAMIC (1ULL << 14)
/**< Dynamic huffman encoding is supported */
+#define RTE_COMP_FF_XXHASH32_CHECKSUM (1ULL << 15)
+/**< xxHash-32 Checksum is supported */
+#define RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE (1ULL << 16)
+/**< LZ4 block independent is supported */
+#define RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM (1ULL << 17)
+/**< LZ4 block with checksum is supported */
/** Status of comp operation */
enum rte_comp_op_status {
@@ -109,6 +115,10 @@ enum rte_comp_algorithm {
/**< LZS compression algorithm
* https://tools.ietf.org/html/rfc2395
*/
+ RTE_COMP_ALGO_LZ4,
+ /**< LZ4 compression algorithm
+ * https://github.com/lz4/lz4
+ */
RTE_COMP_ALGO_LIST_END
};
@@ -149,9 +159,12 @@ enum rte_comp_checksum_type {
/**< Generates both Adler-32 and CRC32 checksums, concatenated.
* CRC32 is in the lower 32bits, Adler-32 in the upper 32 bits.
*/
+ RTE_COMP_CHECKSUM_XXHASH32,
+ /**< Generates a xxHash-32 checksum, as used by lz4.
+ * https://github.com/Cyan4973/xxHash/blob/dev/doc/xxhash_spec.md
+ */
};
-
/** Compression Huffman Type - used by DEFLATE algorithm */
enum rte_comp_huffman {
RTE_COMP_HUFFMAN_DEFAULT,
@@ -208,13 +221,41 @@ enum rte_comp_op_type {
*/
};
-
/** Parameters specific to the deflate algorithm */
struct rte_comp_deflate_params {
enum rte_comp_huffman huffman;
/**< Compression huffman encoding type */
};
+/**
+ * Block checksum flag.
+ * If this flag is set, each data block will be followed by a 4-bytes checksum,
+ * calculated by using the xxHash-32 algorithm on the raw (compressed) data
+ * block. The intention is to detect data corruption (storage or transmission
+ * errors) immediately, before decoding. Block checksum usage is optional.
+ */
+#define RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM (1 << 4)
+
+/**
+ * Block Independence flag.
+ * If this flag is set to 1, blocks are independent.
+ * If this flag is set to 0, each block depends on previous ones (up to LZ4
+ * window size, which is 64 KB). In such case, it is necessary to decode all
+ * blocks in sequence.
+ * Block dependency improves compression ratio, especially for small blocks. On
+ * the other hand, it makes random access or multi-threaded decoding impossible.
+ */
+#define RTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE (1 << 5)
+
+/** Parameters specific to the LZ4 algorithm */
+struct rte_comp_lz4_params {
+ uint8_t flags;
+ /**< Compression LZ4 parameter flags.
+ * Based on LZ4 standard flags:
+ * https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor
+ */
+};
+
/** Setup Data for compression */
struct rte_comp_compress_xform {
enum rte_comp_algorithm algo;
@@ -222,6 +263,8 @@ struct rte_comp_compress_xform {
union {
struct rte_comp_deflate_params deflate;
/**< Parameters specific to the deflate algorithm */
+ struct rte_comp_lz4_params lz4;
+ /**< Parameters specific to the LZ4 algorithm */
}; /**< Algorithm specific parameters */
int level;
/**< Compression level */
@@ -251,6 +294,10 @@ struct rte_comp_decompress_xform {
* compressed data. If window size can't be supported by the PMD then
* setup of stream or private_xform should fail.
*/
+ union {
+ struct rte_comp_lz4_params lz4;
+ /**< Parameters specific to the LZ4 algorithm */
+ }; /**< Algorithm specific parameters */
enum rte_comp_hash_algorithm hash_algo;
/**< Hash algorithm to be used with decompress operation. Hash is always
* done on plaintext.
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH 2/4] app/test-compress-perf: allow test single compress operation
2023-01-09 7:45 [PATCH 0/4] compressdev: add LZ4 support Michael Baum
2023-01-09 7:45 ` [PATCH 1/4] compressdev: add LZ4 algorithm support Michael Baum
@ 2023-01-09 7:45 ` Michael Baum
2023-01-09 7:45 ` [PATCH 3/4] app/test-compress-perf: add algo option Michael Baum
` (2 subsequent siblings)
4 siblings, 0 replies; 31+ messages in thread
From: Michael Baum @ 2023-01-09 7:45 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe, Thomas Monjalon
Part of the application options is to test only compress and only
decompress but actually the application ignores this user option and
tries to test always both compress and decompress.
Allow testing only compress and only decompress.
Signed-off-by: Matan Azrad <matan@nvidia.com>
Signed-off-by: Michael Baum <michaelba@nvidia.com>
---
app/test-compress-perf/comp_perf_options.h | 6 +-
.../comp_perf_options_parse.c | 4 +-
.../comp_perf_test_common.c | 126 +++++++++++++-----
.../comp_perf_test_cyclecount.c | 71 ++++++----
.../comp_perf_test_throughput.c | 64 +++++----
.../comp_perf_test_verify.c | 64 +++++----
app/test-compress-perf/main.c | 8 ++
doc/guides/rel_notes/release_23_03.rst | 4 +
8 files changed, 230 insertions(+), 117 deletions(-)
diff --git a/app/test-compress-perf/comp_perf_options.h b/app/test-compress-perf/comp_perf_options.h
index 57dd146330..d00b299247 100644
--- a/app/test-compress-perf/comp_perf_options.h
+++ b/app/test-compress-perf/comp_perf_options.h
@@ -32,9 +32,9 @@ enum cperf_test_type {
};
enum comp_operation {
- COMPRESS_ONLY,
- DECOMPRESS_ONLY,
- COMPRESS_DECOMPRESS
+ COMPRESS = (1 << 0),
+ DECOMPRESS = (1 << 1),
+ COMPRESS_DECOMPRESS = (COMPRESS | DECOMPRESS),
};
struct range_list {
diff --git a/app/test-compress-perf/comp_perf_options_parse.c b/app/test-compress-perf/comp_perf_options_parse.c
index 019eddb7bd..7a992bf43e 100644
--- a/app/test-compress-perf/comp_perf_options_parse.c
+++ b/app/test-compress-perf/comp_perf_options_parse.c
@@ -446,11 +446,11 @@ parse_op_type(struct comp_test_data *test_data, const char *arg)
struct name_id_map optype_namemap[] = {
{
"comp",
- COMPRESS_ONLY
+ COMPRESS
},
{
"decomp",
- DECOMPRESS_ONLY
+ DECOMPRESS
},
{
"comp_and_decomp",
diff --git a/app/test-compress-perf/comp_perf_test_common.c b/app/test-compress-perf/comp_perf_test_common.c
index b402a0d839..cd60958944 100644
--- a/app/test-compress-perf/comp_perf_test_common.c
+++ b/app/test-compress-perf/comp_perf_test_common.c
@@ -227,23 +227,43 @@ comp_perf_allocate_memory(struct comp_test_data *test_data,
{
uint16_t comp_mbuf_size;
uint16_t decomp_mbuf_size;
+ size_t comp_data_size;
+ size_t decomp_data_size;
+ size_t output_data_sz;
test_data->out_seg_sz = find_buf_size(test_data->seg_sz);
- /* Number of segments for input and output
- * (compression and decompression)
- */
- test_data->total_segs = DIV_CEIL(test_data->input_data_sz,
- test_data->seg_sz);
+ if (test_data->test_op & COMPRESS) {
+ /*
+ * Number of segments for input and output
+ * (compression and decompression)
+ */
+ test_data->total_segs = DIV_CEIL(test_data->input_data_sz,
+ test_data->seg_sz);
+ } else {
+ /*
+ * When application does decompression only, input data is
+ * compressed and smaller than the output. The expected size of
+ * uncompressed data given by the user in segment size argument.
+ */
+ test_data->total_segs = test_data->max_sgl_segs;
+ }
+
+ output_data_sz = (size_t) test_data->out_seg_sz * test_data->total_segs;
+ output_data_sz =
+ RTE_MAX(output_data_sz, (size_t) MIN_COMPRESSED_BUF_SIZE);
if (test_data->use_external_mbufs != 0) {
if (comp_perf_allocate_external_mbufs(test_data, mem) < 0)
return -1;
comp_mbuf_size = 0;
decomp_mbuf_size = 0;
- } else {
+ } else if (test_data->test_op & COMPRESS) {
comp_mbuf_size = test_data->out_seg_sz + RTE_PKTMBUF_HEADROOM;
decomp_mbuf_size = test_data->seg_sz + RTE_PKTMBUF_HEADROOM;
+ } else {
+ comp_mbuf_size = test_data->seg_sz + RTE_PKTMBUF_HEADROOM;
+ decomp_mbuf_size = test_data->out_seg_sz + RTE_PKTMBUF_HEADROOM;
}
char pool_name[32] = "";
@@ -287,26 +307,28 @@ comp_perf_allocate_memory(struct comp_test_data *test_data,
return -1;
}
- /*
- * Compressed data might be a bit larger than input data,
- * if data cannot be compressed
- */
- mem->compressed_data = rte_zmalloc_socket(NULL,
- RTE_MAX(
- (size_t) test_data->out_seg_sz *
- test_data->total_segs,
- (size_t) MIN_COMPRESSED_BUF_SIZE),
- 0,
- rte_socket_id());
+ if (test_data->test_op & COMPRESS) {
+ /*
+ * Compressed data might be a bit larger than input data,
+ * if data cannot be compressed
+ */
+ comp_data_size = output_data_sz;
+ decomp_data_size = test_data->input_data_sz;
+ } else {
+ comp_data_size = test_data->input_data_sz;
+ decomp_data_size = output_data_sz;
+ }
+
+ mem->compressed_data = rte_zmalloc_socket(NULL, comp_data_size, 0,
+ rte_socket_id());
if (mem->compressed_data == NULL) {
RTE_LOG(ERR, USER1, "Memory to hold the data from the input "
"file could not be allocated\n");
return -1;
}
- mem->decompressed_data = rte_zmalloc_socket(NULL,
- test_data->input_data_sz, 0,
- rte_socket_id());
+ mem->decompressed_data = rte_zmalloc_socket(NULL, decomp_data_size, 0,
+ rte_socket_id());
if (mem->decompressed_data == NULL) {
RTE_LOG(ERR, USER1, "Memory to hold the data from the input "
"file could not be allocated\n");
@@ -351,6 +373,7 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
uint16_t segs_per_mbuf = 0;
uint32_t cmz = 0;
uint32_t dmz = 0;
+ bool decompress_only = !!(test_data->test_op == DECOMPRESS);
for (i = 0; i < mem->total_bufs; i++) {
/* Allocate data in input mbuf and copy data from input file */
@@ -361,8 +384,6 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
return -1;
}
- data_sz = RTE_MIN(remaining_data, test_data->seg_sz);
-
if (test_data->use_external_mbufs != 0) {
rte_pktmbuf_attach_extbuf(mem->decomp_bufs[i],
mem->decomp_memzones[dmz]->addr,
@@ -372,16 +393,23 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
dmz++;
}
+ if (!decompress_only)
+ data_sz = RTE_MIN(remaining_data, test_data->seg_sz);
+ else
+ data_sz = test_data->out_seg_sz;
+
data_addr = (uint8_t *) rte_pktmbuf_append(
mem->decomp_bufs[i], data_sz);
if (data_addr == NULL) {
RTE_LOG(ERR, USER1, "Could not append data\n");
return -1;
}
- rte_memcpy(data_addr, input_data_ptr, data_sz);
- input_data_ptr += data_sz;
- remaining_data -= data_sz;
+ if (!decompress_only) {
+ rte_memcpy(data_addr, input_data_ptr, data_sz);
+ input_data_ptr += data_sz;
+ remaining_data -= data_sz;
+ }
/* Already one segment in the mbuf */
segs_per_mbuf = 1;
@@ -398,8 +426,6 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
return -1;
}
- data_sz = RTE_MIN(remaining_data, test_data->seg_sz);
-
if (test_data->use_external_mbufs != 0) {
rte_pktmbuf_attach_extbuf(
next_seg,
@@ -410,6 +436,12 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
dmz++;
}
+ if (!decompress_only)
+ data_sz = RTE_MIN(remaining_data,
+ test_data->seg_sz);
+ else
+ data_sz = test_data->out_seg_sz;
+
data_addr = (uint8_t *)rte_pktmbuf_append(next_seg,
data_sz);
@@ -418,9 +450,11 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
return -1;
}
- rte_memcpy(data_addr, input_data_ptr, data_sz);
- input_data_ptr += data_sz;
- remaining_data -= data_sz;
+ if (!decompress_only) {
+ rte_memcpy(data_addr, input_data_ptr, data_sz);
+ input_data_ptr += data_sz;
+ remaining_data -= data_sz;
+ }
if (rte_pktmbuf_chain(mem->decomp_bufs[i],
next_seg) < 0) {
@@ -447,16 +481,26 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
cmz++;
}
- data_addr = (uint8_t *) rte_pktmbuf_append(
- mem->comp_bufs[i],
- test_data->out_seg_sz);
+ if (decompress_only)
+ data_sz = RTE_MIN(remaining_data, test_data->seg_sz);
+ else
+ data_sz = test_data->out_seg_sz;
+
+ data_addr = (uint8_t *) rte_pktmbuf_append(mem->comp_bufs[i],
+ data_sz);
if (data_addr == NULL) {
RTE_LOG(ERR, USER1, "Could not append data\n");
return -1;
}
+ if (decompress_only) {
+ rte_memcpy(data_addr, input_data_ptr, data_sz);
+ input_data_ptr += data_sz;
+ remaining_data -= data_sz;
+ }
+
/* Chain mbufs if needed for output mbufs */
- for (j = 1; j < segs_per_mbuf; j++) {
+ for (j = 1; j < segs_per_mbuf && remaining_data > 0; j++) {
struct rte_mbuf *next_seg =
rte_pktmbuf_alloc(mem->comp_buf_pool);
@@ -476,13 +520,25 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
cmz++;
}
+ if (decompress_only)
+ data_sz = RTE_MIN(remaining_data,
+ test_data->seg_sz);
+ else
+ data_sz = test_data->out_seg_sz;
+
data_addr = (uint8_t *)rte_pktmbuf_append(next_seg,
- test_data->out_seg_sz);
+ data_sz);
if (data_addr == NULL) {
RTE_LOG(ERR, USER1, "Could not append data\n");
return -1;
}
+ if (decompress_only) {
+ rte_memcpy(data_addr, input_data_ptr, data_sz);
+ input_data_ptr += data_sz;
+ remaining_data -= data_sz;
+ }
+
if (rte_pktmbuf_chain(mem->comp_bufs[i],
next_seg) < 0) {
RTE_LOG(ERR, USER1, "Could not chain mbufs\n");
diff --git a/app/test-compress-perf/comp_perf_test_cyclecount.c b/app/test-compress-perf/comp_perf_test_cyclecount.c
index 7473cb6277..ce6c4d7605 100644
--- a/app/test-compress-perf/comp_perf_test_cyclecount.c
+++ b/app/test-compress-perf/comp_perf_test_cyclecount.c
@@ -514,38 +514,55 @@ cperf_cyclecount_test_runner(void *test_ctx)
if (cperf_verify_test_runner(&ctx->ver))
return EXIT_FAILURE;
- /*
- * Run the tests twice, discarding the first performance
- * results, before the cache is warmed up
- */
-
- /* C O M P R E S S */
- for (i = 0; i < 2; i++) {
- if (main_loop(ctx, RTE_COMP_COMPRESS) < 0)
- return EXIT_FAILURE;
- }
+ if (test_data->test_op & COMPRESS) {
+ /*
+ * Run the test twice, discarding the first performance
+ * results, before the cache is warmed up
+ */
+ for (i = 0; i < 2; i++) {
+ if (main_loop(ctx, RTE_COMP_COMPRESS) < 0)
+ return EXIT_FAILURE;
+ }
- ops_enq_retries_comp = ctx->ops_enq_retries;
- ops_deq_retries_comp = ctx->ops_deq_retries;
+ ops_enq_retries_comp = ctx->ops_enq_retries;
+ ops_deq_retries_comp = ctx->ops_deq_retries;
- duration_enq_per_op_comp = ctx->duration_enq /
- (ctx->ver.mem.total_bufs * test_data->num_iter);
- duration_deq_per_op_comp = ctx->duration_deq /
- (ctx->ver.mem.total_bufs * test_data->num_iter);
+ duration_enq_per_op_comp = ctx->duration_enq /
+ (ctx->ver.mem.total_bufs * test_data->num_iter);
+ duration_deq_per_op_comp = ctx->duration_deq /
+ (ctx->ver.mem.total_bufs * test_data->num_iter);
+ } else {
+ ops_enq_retries_comp = 0;
+ ops_deq_retries_comp = 0;
- /* D E C O M P R E S S */
- for (i = 0; i < 2; i++) {
- if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0)
- return EXIT_FAILURE;
+ duration_enq_per_op_comp = 0;
+ duration_deq_per_op_comp = 0;
}
- ops_enq_retries_decomp = ctx->ops_enq_retries;
- ops_deq_retries_decomp = ctx->ops_deq_retries;
+ if (test_data->test_op & DECOMPRESS) {
+ /*
+ * Run the test twice, discarding the first performance
+ * results, before the cache is warmed up
+ */
+ for (i = 0; i < 2; i++) {
+ if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0)
+ return EXIT_FAILURE;
+ }
- duration_enq_per_op_decomp = ctx->duration_enq /
- (ctx->ver.mem.total_bufs * test_data->num_iter);
- duration_deq_per_op_decomp = ctx->duration_deq /
- (ctx->ver.mem.total_bufs * test_data->num_iter);
+ ops_enq_retries_decomp = ctx->ops_enq_retries;
+ ops_deq_retries_decomp = ctx->ops_deq_retries;
+
+ duration_enq_per_op_decomp = ctx->duration_enq /
+ (ctx->ver.mem.total_bufs * test_data->num_iter);
+ duration_deq_per_op_decomp = ctx->duration_deq /
+ (ctx->ver.mem.total_bufs * test_data->num_iter);
+ } else {
+ ops_enq_retries_decomp = 0;
+ ops_deq_retries_decomp = 0;
+
+ duration_enq_per_op_decomp = 0;
+ duration_deq_per_op_decomp = 0;
+ }
duration_setup_per_op = ctx->duration_op /
(ctx->ver.mem.total_bufs * test_data->num_iter);
@@ -563,7 +580,7 @@ cperf_cyclecount_test_runner(void *test_ctx)
" [D-e] - decompression enqueue\n"
" [D-d] - decompression dequeue\n"
" - Cycles section: number of cycles per 'op' for the following operations:\n"
- " setup/op - memory allocation, op configuration and memory dealocation\n"
+ " setup/op - memory allocation, op configuration and memory deallocation\n"
" [C-e] - compression enqueue\n"
" [C-d] - compression dequeue\n"
" [D-e] - decompression enqueue\n"
diff --git a/app/test-compress-perf/comp_perf_test_throughput.c b/app/test-compress-perf/comp_perf_test_throughput.c
index 79cd2b2bf2..c9f8237626 100644
--- a/app/test-compress-perf/comp_perf_test_throughput.c
+++ b/app/test-compress-perf/comp_perf_test_throughput.c
@@ -359,41 +359,53 @@ cperf_throughput_test_runner(void *test_ctx)
* First the verification part is needed
*/
if (cperf_verify_test_runner(&ctx->ver)) {
- ret = EXIT_FAILURE;
+ ret = EXIT_FAILURE;
goto end;
}
- /*
- * Run the tests twice, discarding the first performance
- * results, before the cache is warmed up
- */
- for (i = 0; i < 2; i++) {
- if (main_loop(ctx, RTE_COMP_COMPRESS) < 0) {
- ret = EXIT_FAILURE;
- goto end;
+ if (test_data->test_op & COMPRESS) {
+ /*
+ * Run the test twice, discarding the first performance
+ * results, before the cache is warmed up
+ */
+ for (i = 0; i < 2; i++) {
+ if (main_loop(ctx, RTE_COMP_COMPRESS) < 0) {
+ ret = EXIT_FAILURE;
+ goto end;
+ }
}
- }
- for (i = 0; i < 2; i++) {
- if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0) {
- ret = EXIT_FAILURE;
- goto end;
- }
+ ctx->comp_tsc_byte =
+ (double)(ctx->comp_tsc_duration[test_data->level]) /
+ test_data->input_data_sz;
+ ctx->comp_gbps = rte_get_tsc_hz() / ctx->comp_tsc_byte * 8 /
+ 1000000000;
+ } else {
+ ctx->comp_tsc_byte = 0;
+ ctx->comp_gbps = 0;
}
- ctx->comp_tsc_byte =
- (double)(ctx->comp_tsc_duration[test_data->level]) /
- test_data->input_data_sz;
+ if (test_data->test_op & DECOMPRESS) {
+ /*
+ * Run the test twice, discarding the first performance
+ * results, before the cache is warmed up
+ */
+ for (i = 0; i < 2; i++) {
+ if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0) {
+ ret = EXIT_FAILURE;
+ goto end;
+ }
+ }
- ctx->decomp_tsc_byte =
+ ctx->decomp_tsc_byte =
(double)(ctx->decomp_tsc_duration[test_data->level]) /
- test_data->input_data_sz;
-
- ctx->comp_gbps = rte_get_tsc_hz() / ctx->comp_tsc_byte * 8 /
- 1000000000;
-
- ctx->decomp_gbps = rte_get_tsc_hz() / ctx->decomp_tsc_byte * 8 /
- 1000000000;
+ test_data->input_data_sz;
+ ctx->decomp_gbps = rte_get_tsc_hz() / ctx->decomp_tsc_byte * 8 /
+ 1000000000;
+ } else {
+ ctx->decomp_tsc_byte = 0;
+ ctx->decomp_gbps = 0;
+ }
exp = 0;
if (__atomic_compare_exchange_n(&display_once, &exp, 1, 0,
diff --git a/app/test-compress-perf/comp_perf_test_verify.c b/app/test-compress-perf/comp_perf_test_verify.c
index 8964442891..7d6b6abecd 100644
--- a/app/test-compress-perf/comp_perf_test_verify.c
+++ b/app/test-compress-perf/comp_perf_test_verify.c
@@ -114,7 +114,8 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
output_data_sz = &ctx->decomp_data_sz;
input_bufs = mem->comp_bufs;
output_bufs = mem->decomp_bufs;
- out_seg_sz = test_data->seg_sz;
+ out_seg_sz = (test_data->test_op & COMPRESS) ?
+ test_data->seg_sz : test_data->out_seg_sz;
}
/* Create private xform */
@@ -226,7 +227,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
op->status ==
RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE) {
RTE_LOG(ERR, USER1,
-"Out of space error occurred due to uncompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n");
+"Out of space error occurred due to incompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n");
res = -1;
goto end;
} else if (op->status !=
@@ -311,7 +312,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
op->status ==
RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE) {
RTE_LOG(ERR, USER1,
-"Out of space error occurred due to uncompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n");
+"Out of space error occurred due to incompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n");
res = -1;
goto end;
} else if (op->status !=
@@ -392,44 +393,59 @@ cperf_verify_test_runner(void *test_ctx)
int ret = EXIT_SUCCESS;
static uint16_t display_once;
uint32_t lcore = rte_lcore_id();
+ uint16_t exp = 0;
ctx->mem.lcore_id = lcore;
test_data->ratio = 0;
- if (main_loop(ctx, RTE_COMP_COMPRESS) < 0) {
- ret = EXIT_FAILURE;
- goto end;
+ if (test_data->test_op & COMPRESS) {
+ if (main_loop(ctx, RTE_COMP_COMPRESS) < 0) {
+ ret = EXIT_FAILURE;
+ goto end;
+ }
}
- if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0) {
- ret = EXIT_FAILURE;
- goto end;
- }
+ if (test_data->test_op & DECOMPRESS) {
+ if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0) {
+ ret = EXIT_FAILURE;
+ goto end;
+ }
- if (ctx->decomp_data_sz != test_data->input_data_sz) {
- RTE_LOG(ERR, USER1,
- "Decompressed data length not equal to input data length\n");
- RTE_LOG(ERR, USER1,
- "Decompressed size = %zu, expected = %zu\n",
- ctx->decomp_data_sz, test_data->input_data_sz);
- ret = EXIT_FAILURE;
- goto end;
- } else {
- if (memcmp(ctx->mem.decompressed_data,
- test_data->input_data,
- test_data->input_data_sz) != 0) {
+ if (!(test_data->test_op & COMPRESS)) {
+ /*
+ * For DECOMPRESS_ONLY mode there is no more
+ * verifications, reset the 'ratio' and 'comp_data_sz'
+ * fields for other tests report.
+ */
+ ctx->comp_data_sz = 0;
+ ctx->ratio = 0;
+ goto end;
+ }
+
+ if (ctx->decomp_data_sz != test_data->input_data_sz) {
+ RTE_LOG(ERR, USER1,
+ "Decompressed data length not equal to input data length\n");
RTE_LOG(ERR, USER1,
- "Decompressed data is not the same as file data\n");
+ "Decompressed size = %zu, expected = %zu\n",
+ ctx->decomp_data_sz, test_data->input_data_sz);
ret = EXIT_FAILURE;
goto end;
+ } else {
+ if (memcmp(ctx->mem.decompressed_data,
+ test_data->input_data,
+ test_data->input_data_sz) != 0) {
+ RTE_LOG(ERR, USER1,
+ "Decompressed data is not the same as file data\n");
+ ret = EXIT_FAILURE;
+ goto end;
+ }
}
}
ctx->ratio = (double) ctx->comp_data_sz /
test_data->input_data_sz * 100;
- uint16_t exp = 0;
if (!ctx->silent) {
if (__atomic_compare_exchange_n(&display_once, &exp, 1, 0,
__ATOMIC_RELAXED, __ATOMIC_RELAXED)) {
diff --git a/app/test-compress-perf/main.c b/app/test-compress-perf/main.c
index 41b8edc2bd..bbb4c7917b 100644
--- a/app/test-compress-perf/main.c
+++ b/app/test-compress-perf/main.c
@@ -254,6 +254,14 @@ comp_perf_dump_input_data(struct comp_test_data *test_data)
goto end;
}
+ if (!(test_data->test_op & COMPRESS) &&
+ test_data->input_data_sz >
+ (size_t) test_data->seg_sz * (size_t) test_data->max_sgl_segs) {
+ RTE_LOG(ERR, USER1,
+ "Size of input must be less than total segments\n");
+ goto end;
+ }
+
test_data->input_data = rte_zmalloc_socket(NULL,
test_data->input_data_sz, 0, rte_socket_id());
diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index 3bcf638544..1f879da6b8 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -62,6 +62,10 @@ New Features
* Added support for ``RTE_COMP_ALGO_LZ4``.
* Added support for ``RTE_COMP_CHECKSUM_XXHASH32``.
+* **Allowed test single compress operation in test-compress-perf.**
+
+ Enable the application options for testing only compress and only decompress.
+
Removed Items
-------------
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH 3/4] app/test-compress-perf: add algo option
2023-01-09 7:45 [PATCH 0/4] compressdev: add LZ4 support Michael Baum
2023-01-09 7:45 ` [PATCH 1/4] compressdev: add LZ4 algorithm support Michael Baum
2023-01-09 7:45 ` [PATCH 2/4] app/test-compress-perf: allow test single compress operation Michael Baum
@ 2023-01-09 7:45 ` Michael Baum
2023-01-09 7:45 ` [PATCH 4/4] app/test-compress-perf: add LZ4 support Michael Baum
2023-02-02 9:13 ` [PATCH v2 0/4] compressdev: " Michael Baum
4 siblings, 0 replies; 31+ messages in thread
From: Michael Baum @ 2023-01-09 7:45 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe, Thomas Monjalon
Add a command line option `algo` to select the compress algorithm
supported by the compress API: null (DMA), deflate and lzs.
Default for deflate.
Signed-off-by: Matan Azrad <matan@nvidia.com>
Signed-off-by: Michael Baum <michaelba@nvidia.com>
---
app/test-compress-perf/comp_perf_options.h | 2 +
.../comp_perf_options_parse.c | 40 ++++++++++++++++++-
.../comp_perf_test_cyclecount.c | 4 +-
.../comp_perf_test_throughput.c | 4 +-
.../comp_perf_test_verify.c | 4 +-
app/test-compress-perf/main.c | 38 +++++++++++-------
doc/guides/rel_notes/release_23_03.rst | 4 ++
doc/guides/tools/comp_perf.rst | 2 +
8 files changed, 76 insertions(+), 22 deletions(-)
diff --git a/app/test-compress-perf/comp_perf_options.h b/app/test-compress-perf/comp_perf_options.h
index d00b299247..5e5227a700 100644
--- a/app/test-compress-perf/comp_perf_options.h
+++ b/app/test-compress-perf/comp_perf_options.h
@@ -63,6 +63,8 @@ struct comp_test_data {
enum rte_comp_huffman huffman_enc;
enum comp_operation test_op;
+ enum rte_comp_algorithm test_algo;
+
int window_sz;
struct range_list level_lst;
uint8_t level;
diff --git a/app/test-compress-perf/comp_perf_options_parse.c b/app/test-compress-perf/comp_perf_options_parse.c
index 7a992bf43e..97ddff87ef 100644
--- a/app/test-compress-perf/comp_perf_options_parse.c
+++ b/app/test-compress-perf/comp_perf_options_parse.c
@@ -25,6 +25,7 @@
#define CPERF_MAX_SGL_SEGS ("max-num-sgl-segs")
#define CPERF_NUM_ITER ("num-iter")
#define CPERF_OPTYPE ("operation")
+#define CPERF_ALGO ("algo")
#define CPERF_HUFFMAN_ENC ("huffman-enc")
#define CPERF_LEVEL ("compress-level")
#define CPERF_WINDOW_SIZE ("window-sz")
@@ -56,6 +57,8 @@ usage(char *progname)
" compressed/decompressed (default: 10000)\n"
" --operation [comp/decomp/comp_and_decomp]: perform test on\n"
" compression, decompression or both operations\n"
+ " --algo [null/deflate/lzs]: perform test on algorithm\n"
+ " null(DMA), deflate or lzs (default: deflate)\n"
" --huffman-enc [fixed/dynamic/default]: Huffman encoding\n"
" (default: dynamic)\n"
" --compress-level N: compression level, which could be a single value, list or range\n"
@@ -470,6 +473,36 @@ parse_op_type(struct comp_test_data *test_data, const char *arg)
return 0;
}
+static int
+parse_algo(struct comp_test_data *test_data, const char *arg)
+{
+ struct name_id_map algo_namemap[] = {
+ {
+ "null",
+ RTE_COMP_ALGO_NULL
+ },
+ {
+ "deflate",
+ RTE_COMP_ALGO_DEFLATE
+ },
+ {
+ "lzs",
+ RTE_COMP_ALGO_LZS
+ }
+ };
+
+ int id = get_str_key_id_mapping(algo_namemap,
+ RTE_DIM(algo_namemap), arg);
+ if (id < 0) {
+ RTE_LOG(ERR, USER1, "Invalid algorithm specified\n");
+ return -1;
+ }
+
+ test_data->test_algo = (enum rte_comp_algorithm)id;
+
+ return 0;
+}
+
static int
parse_huffman_enc(struct comp_test_data *test_data, const char *arg)
{
@@ -491,7 +524,7 @@ parse_huffman_enc(struct comp_test_data *test_data, const char *arg)
int id = get_str_key_id_mapping(huffman_namemap,
RTE_DIM(huffman_namemap), arg);
if (id < 0) {
- RTE_LOG(ERR, USER1, "Invalid Huffmane encoding specified\n");
+ RTE_LOG(ERR, USER1, "Invalid Huffman encoding specified\n");
return -1;
}
@@ -507,7 +540,7 @@ parse_level(struct comp_test_data *test_data, const char *arg)
/*
* Try parsing the argument as a range, if it fails,
- * arse it as a list
+ * parse it as a list
*/
if (parse_range(arg, &test_data->level_lst.min,
&test_data->level_lst.max,
@@ -572,6 +605,7 @@ static struct option lgopts[] = {
{ CPERF_MAX_SGL_SEGS, required_argument, 0, 0},
{ CPERF_NUM_ITER, required_argument, 0, 0 },
{ CPERF_OPTYPE, required_argument, 0, 0 },
+ { CPERF_ALGO, required_argument, 0, 0 },
{ CPERF_HUFFMAN_ENC, required_argument, 0, 0 },
{ CPERF_LEVEL, required_argument, 0, 0 },
{ CPERF_WINDOW_SIZE, required_argument, 0, 0 },
@@ -594,6 +628,7 @@ comp_perf_opts_parse_long(int opt_idx, struct comp_test_data *test_data)
{ CPERF_MAX_SGL_SEGS, parse_max_num_sgl_segs },
{ CPERF_NUM_ITER, parse_num_iter },
{ CPERF_OPTYPE, parse_op_type },
+ { CPERF_ALGO, parse_algo },
{ CPERF_HUFFMAN_ENC, parse_huffman_enc },
{ CPERF_LEVEL, parse_level },
{ CPERF_WINDOW_SIZE, parse_window_sz },
@@ -649,6 +684,7 @@ comp_perf_options_default(struct comp_test_data *test_data)
test_data->num_iter = 10000;
test_data->huffman_enc = RTE_COMP_HUFFMAN_DYNAMIC;
test_data->test_op = COMPRESS_DECOMPRESS;
+ test_data->test_algo = RTE_COMP_ALGO_DEFLATE;
test_data->window_sz = -1;
test_data->level_lst.min = RTE_COMP_LEVEL_MIN;
test_data->level_lst.max = RTE_COMP_LEVEL_MAX;
diff --git a/app/test-compress-perf/comp_perf_test_cyclecount.c b/app/test-compress-perf/comp_perf_test_cyclecount.c
index ce6c4d7605..81c3d30038 100644
--- a/app/test-compress-perf/comp_perf_test_cyclecount.c
+++ b/app/test-compress-perf/comp_perf_test_cyclecount.c
@@ -193,7 +193,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_COMPRESS,
.compress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.deflate.huffman = test_data->huffman_enc,
.level = test_data->level,
.window_size = test_data->window_sz,
@@ -208,7 +208,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_DECOMPRESS,
.decompress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.chksum = RTE_COMP_CHECKSUM_NONE,
.window_size = test_data->window_sz,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
diff --git a/app/test-compress-perf/comp_perf_test_throughput.c b/app/test-compress-perf/comp_perf_test_throughput.c
index c9f8237626..2545ee9925 100644
--- a/app/test-compress-perf/comp_perf_test_throughput.c
+++ b/app/test-compress-perf/comp_perf_test_throughput.c
@@ -84,7 +84,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_COMPRESS,
.compress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.deflate.huffman = test_data->huffman_enc,
.level = test_data->level,
.window_size = test_data->window_sz,
@@ -99,7 +99,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_DECOMPRESS,
.decompress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.chksum = RTE_COMP_CHECKSUM_NONE,
.window_size = test_data->window_sz,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
diff --git a/app/test-compress-perf/comp_perf_test_verify.c b/app/test-compress-perf/comp_perf_test_verify.c
index 7d6b6abecd..88f4f41851 100644
--- a/app/test-compress-perf/comp_perf_test_verify.c
+++ b/app/test-compress-perf/comp_perf_test_verify.c
@@ -87,7 +87,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_COMPRESS,
.compress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.deflate.huffman = test_data->huffman_enc,
.level = test_data->level,
.window_size = test_data->window_sz,
@@ -104,7 +104,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_DECOMPRESS,
.decompress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.chksum = RTE_COMP_CHECKSUM_NONE,
.window_size = test_data->window_sz,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
diff --git a/app/test-compress-perf/main.c b/app/test-compress-perf/main.c
index bbb4c7917b..d049527ba1 100644
--- a/app/test-compress-perf/main.c
+++ b/app/test-compress-perf/main.c
@@ -57,29 +57,39 @@ comp_perf_check_capabilities(struct comp_test_data *test_data, uint8_t cdev_id)
{
const struct rte_compressdev_capabilities *cap;
- cap = rte_compressdev_capability_get(cdev_id,
- RTE_COMP_ALGO_DEFLATE);
+ cap = rte_compressdev_capability_get(cdev_id, test_data->test_algo);
if (cap == NULL) {
RTE_LOG(ERR, USER1,
- "Compress device does not support DEFLATE\n");
+ "Compress device does not support %u algorithm\n",
+ test_data->test_algo);
return -1;
}
uint64_t comp_flags = cap->comp_feature_flags;
- /* Huffman encoding */
- if (test_data->huffman_enc == RTE_COMP_HUFFMAN_FIXED &&
- (comp_flags & RTE_COMP_FF_HUFFMAN_FIXED) == 0) {
- RTE_LOG(ERR, USER1,
- "Compress device does not supported Fixed Huffman\n");
- return -1;
- }
+ /* Algorithm type */
+ switch (test_data->test_algo) {
+ case RTE_COMP_ALGO_DEFLATE:
+ /* Huffman encoding */
+ if (test_data->huffman_enc == RTE_COMP_HUFFMAN_FIXED &&
+ (comp_flags & RTE_COMP_FF_HUFFMAN_FIXED) == 0) {
+ RTE_LOG(ERR, USER1,
+ "Compress device does not supported Fixed Huffman\n");
+ return -1;
+ }
- if (test_data->huffman_enc == RTE_COMP_HUFFMAN_DYNAMIC &&
- (comp_flags & RTE_COMP_FF_HUFFMAN_DYNAMIC) == 0) {
- RTE_LOG(ERR, USER1,
- "Compress device does not supported Dynamic Huffman\n");
+ if (test_data->huffman_enc == RTE_COMP_HUFFMAN_DYNAMIC &&
+ (comp_flags & RTE_COMP_FF_HUFFMAN_DYNAMIC) == 0) {
+ RTE_LOG(ERR, USER1,
+ "Compress device does not supported Dynamic Huffman\n");
+ return -1;
+ }
+ break;
+ case RTE_COMP_ALGO_LZS:
+ case RTE_COMP_ALGO_NULL:
+ break;
+ default:
return -1;
}
diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index 1f879da6b8..107d4d7603 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -66,6 +66,10 @@ New Features
Enable the application options for testing only compress and only decompress.
+* **Added algo option in test-compress-perf.**
+
+ Added support for testing other algorithms except for DEFLAT.
+
Removed Items
-------------
diff --git a/doc/guides/tools/comp_perf.rst b/doc/guides/tools/comp_perf.rst
index 9d2f4dbe4a..cf9035cc23 100644
--- a/doc/guides/tools/comp_perf.rst
+++ b/doc/guides/tools/comp_perf.rst
@@ -84,6 +84,8 @@ Application Options
``--operation [comp/decomp/comp_and_decomp]``: perform test on compression, decompression or both operations
+ ``--algo [null/deflate/lzs]`` : perform test on algorithm null(DMA), Deflate or lzs (default: Deflate)
+
``--huffman-enc [fixed/dynamic/default]``: Huffman encoding (default: dynamic)
``--compress-level N``: compression level, which could be a single value, list or range (default: range between 1 and 9)
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH 4/4] app/test-compress-perf: add LZ4 support
2023-01-09 7:45 [PATCH 0/4] compressdev: add LZ4 support Michael Baum
` (2 preceding siblings ...)
2023-01-09 7:45 ` [PATCH 3/4] app/test-compress-perf: add algo option Michael Baum
@ 2023-01-09 7:45 ` Michael Baum
2023-02-02 9:13 ` [PATCH v2 0/4] compressdev: " Michael Baum
4 siblings, 0 replies; 31+ messages in thread
From: Michael Baum @ 2023-01-09 7:45 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe, Thomas Monjalon
Add support for LZ4 algorithm and add a new parameter for it.
Signed-off-by: Michael Baum <michaelba@nvidia.com>
---
app/test-compress-perf/comp_perf_options.h | 1 +
.../comp_perf_options_parse.c | 43 ++++++++++++++++++-
.../comp_perf_test_cyclecount.c | 7 ++-
.../comp_perf_test_throughput.c | 7 ++-
.../comp_perf_test_verify.c | 7 ++-
app/test-compress-perf/main.c | 17 ++++++++
doc/guides/rel_notes/release_23_03.rst | 1 +
doc/guides/tools/comp_perf.rst | 5 ++-
8 files changed, 82 insertions(+), 6 deletions(-)
diff --git a/app/test-compress-perf/comp_perf_options.h b/app/test-compress-perf/comp_perf_options.h
index 5e5227a700..828a7309d8 100644
--- a/app/test-compress-perf/comp_perf_options.h
+++ b/app/test-compress-perf/comp_perf_options.h
@@ -61,6 +61,7 @@ struct comp_test_data {
uint16_t max_sgl_segs;
uint32_t total_segs;
+ uint8_t lz4_flags;
enum rte_comp_huffman huffman_enc;
enum comp_operation test_op;
enum rte_comp_algorithm test_algo;
diff --git a/app/test-compress-perf/comp_perf_options_parse.c b/app/test-compress-perf/comp_perf_options_parse.c
index 97ddff87ef..6d8c370fc2 100644
--- a/app/test-compress-perf/comp_perf_options_parse.c
+++ b/app/test-compress-perf/comp_perf_options_parse.c
@@ -27,6 +27,7 @@
#define CPERF_OPTYPE ("operation")
#define CPERF_ALGO ("algo")
#define CPERF_HUFFMAN_ENC ("huffman-enc")
+#define CPERF_LZ4_FLAGS ("lz4-flags")
#define CPERF_LEVEL ("compress-level")
#define CPERF_WINDOW_SIZE ("window-sz")
#define CPERF_EXTERNAL_MBUFS ("external-mbufs")
@@ -57,10 +58,11 @@ usage(char *progname)
" compressed/decompressed (default: 10000)\n"
" --operation [comp/decomp/comp_and_decomp]: perform test on\n"
" compression, decompression or both operations\n"
- " --algo [null/deflate/lzs]: perform test on algorithm\n"
- " null(DMA), deflate or lzs (default: deflate)\n"
+ " --algo [null/deflate/lzs/lz4]: perform test on algorithm\n"
+ " null(DMA), deflate, lzs or lz4 (default: deflate)\n"
" --huffman-enc [fixed/dynamic/default]: Huffman encoding\n"
" (default: dynamic)\n"
+ " --lz4-flags N: flags to configure LZ4 algorithm (default: 0)\n"
" --compress-level N: compression level, which could be a single value, list or range\n"
" (default: range between 1 and 9)\n"
" --window-sz N: base two log value of compression window size\n"
@@ -153,6 +155,23 @@ parse_uint16_t(uint16_t *value, const char *arg)
return 0;
}
+static int
+parse_uint8_t(uint8_t *value, const char *arg)
+{
+ uint32_t val = 0;
+ int ret = parse_uint32_t(&val, arg);
+
+ if (ret < 0)
+ return ret;
+
+ if (val > UINT8_MAX)
+ return -ERANGE;
+
+ *value = (uint8_t) val;
+
+ return 0;
+}
+
static int
parse_range(const char *arg, uint8_t *min, uint8_t *max, uint8_t *inc)
{
@@ -488,6 +507,10 @@ parse_algo(struct comp_test_data *test_data, const char *arg)
{
"lzs",
RTE_COMP_ALGO_LZS
+ },
+ {
+ "lz4",
+ RTE_COMP_ALGO_LZ4
}
};
@@ -533,6 +556,19 @@ parse_huffman_enc(struct comp_test_data *test_data, const char *arg)
return 0;
}
+static int
+parse_lz4_flags(struct comp_test_data *test_data, const char *arg)
+{
+ int ret = parse_uint8_t(&test_data->lz4_flags, arg);
+
+ if (ret) {
+ RTE_LOG(ERR, USER1, "Failed to parse LZ4 flags\n");
+ return -1;
+ }
+
+ return 0;
+}
+
static int
parse_level(struct comp_test_data *test_data, const char *arg)
{
@@ -607,6 +643,7 @@ static struct option lgopts[] = {
{ CPERF_OPTYPE, required_argument, 0, 0 },
{ CPERF_ALGO, required_argument, 0, 0 },
{ CPERF_HUFFMAN_ENC, required_argument, 0, 0 },
+ { CPERF_LZ4_FLAGS, required_argument, 0, 0 },
{ CPERF_LEVEL, required_argument, 0, 0 },
{ CPERF_WINDOW_SIZE, required_argument, 0, 0 },
{ CPERF_EXTERNAL_MBUFS, 0, 0, 0 },
@@ -630,6 +667,7 @@ comp_perf_opts_parse_long(int opt_idx, struct comp_test_data *test_data)
{ CPERF_OPTYPE, parse_op_type },
{ CPERF_ALGO, parse_algo },
{ CPERF_HUFFMAN_ENC, parse_huffman_enc },
+ { CPERF_LZ4_FLAGS, parse_lz4_flags },
{ CPERF_LEVEL, parse_level },
{ CPERF_WINDOW_SIZE, parse_window_sz },
{ CPERF_EXTERNAL_MBUFS, parse_external_mbufs },
@@ -682,6 +720,7 @@ comp_perf_options_default(struct comp_test_data *test_data)
test_data->pool_sz = 8192;
test_data->max_sgl_segs = 16;
test_data->num_iter = 10000;
+ test_data->lz4_flags = 0;
test_data->huffman_enc = RTE_COMP_HUFFMAN_DYNAMIC;
test_data->test_op = COMPRESS_DECOMPRESS;
test_data->test_algo = RTE_COMP_ALGO_DEFLATE;
diff --git a/app/test-compress-perf/comp_perf_test_cyclecount.c b/app/test-compress-perf/comp_perf_test_cyclecount.c
index 81c3d30038..4d336ec8d6 100644
--- a/app/test-compress-perf/comp_perf_test_cyclecount.c
+++ b/app/test-compress-perf/comp_perf_test_cyclecount.c
@@ -194,13 +194,16 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type)
.type = RTE_COMP_COMPRESS,
.compress = {
.algo = test_data->test_algo,
- .deflate.huffman = test_data->huffman_enc,
.level = test_data->level,
.window_size = test_data->window_sz,
.chksum = RTE_COMP_CHECKSUM_NONE,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_DEFLATE)
+ xform.compress.deflate.huffman = test_data->huffman_enc;
+ else if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.compress.lz4.flags = test_data->lz4_flags;
input_bufs = mem->decomp_bufs;
output_bufs = mem->comp_bufs;
out_seg_sz = test_data->out_seg_sz;
@@ -214,6 +217,8 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type)
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.decompress.lz4.flags = test_data->lz4_flags;
input_bufs = mem->comp_bufs;
output_bufs = mem->decomp_bufs;
out_seg_sz = test_data->seg_sz;
diff --git a/app/test-compress-perf/comp_perf_test_throughput.c b/app/test-compress-perf/comp_perf_test_throughput.c
index 2545ee9925..1f7072d223 100644
--- a/app/test-compress-perf/comp_perf_test_throughput.c
+++ b/app/test-compress-perf/comp_perf_test_throughput.c
@@ -85,13 +85,16 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type)
.type = RTE_COMP_COMPRESS,
.compress = {
.algo = test_data->test_algo,
- .deflate.huffman = test_data->huffman_enc,
.level = test_data->level,
.window_size = test_data->window_sz,
.chksum = RTE_COMP_CHECKSUM_NONE,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_DEFLATE)
+ xform.compress.deflate.huffman = test_data->huffman_enc;
+ else if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.compress.lz4.flags = test_data->lz4_flags;
input_bufs = mem->decomp_bufs;
output_bufs = mem->comp_bufs;
out_seg_sz = test_data->out_seg_sz;
@@ -105,6 +108,8 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type)
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.decompress.lz4.flags = test_data->lz4_flags;
input_bufs = mem->comp_bufs;
output_bufs = mem->decomp_bufs;
out_seg_sz = test_data->seg_sz;
diff --git a/app/test-compress-perf/comp_perf_test_verify.c b/app/test-compress-perf/comp_perf_test_verify.c
index 88f4f41851..7bd18073cf 100644
--- a/app/test-compress-perf/comp_perf_test_verify.c
+++ b/app/test-compress-perf/comp_perf_test_verify.c
@@ -88,13 +88,16 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
.type = RTE_COMP_COMPRESS,
.compress = {
.algo = test_data->test_algo,
- .deflate.huffman = test_data->huffman_enc,
.level = test_data->level,
.window_size = test_data->window_sz,
.chksum = RTE_COMP_CHECKSUM_NONE,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_DEFLATE)
+ xform.compress.deflate.huffman = test_data->huffman_enc;
+ else if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.compress.lz4.flags = test_data->lz4_flags;
output_data_ptr = ctx->mem.compressed_data;
output_data_sz = &ctx->comp_data_sz;
input_bufs = mem->decomp_bufs;
@@ -110,6 +113,8 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.decompress.lz4.flags = test_data->lz4_flags;
output_data_ptr = ctx->mem.decompressed_data;
output_data_sz = &ctx->decomp_data_sz;
input_bufs = mem->comp_bufs;
diff --git a/app/test-compress-perf/main.c b/app/test-compress-perf/main.c
index d049527ba1..fa366123ed 100644
--- a/app/test-compress-perf/main.c
+++ b/app/test-compress-perf/main.c
@@ -86,6 +86,23 @@ comp_perf_check_capabilities(struct comp_test_data *test_data, uint8_t cdev_id)
return -1;
}
break;
+ case RTE_COMP_ALGO_LZ4:
+ /* LZ4 flags */
+ if ((test_data->lz4_flags & RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM) &&
+ (comp_flags & RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM) == 0) {
+ RTE_LOG(ERR, USER1,
+ "Compress device does not support LZ4 block with checksum\n");
+ return -1;
+ }
+
+ if ((test_data->lz4_flags &
+ RTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE) &&
+ (comp_flags & RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE) == 0) {
+ RTE_LOG(ERR, USER1,
+ "Compress device does not support LZ4 independent blocks\n");
+ return -1;
+ }
+ break;
case RTE_COMP_ALGO_LZS:
case RTE_COMP_ALGO_NULL:
break;
diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index 107d4d7603..7ae012ae33 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -61,6 +61,7 @@ New Features
* Added support for ``RTE_COMP_ALGO_LZ4``.
* Added support for ``RTE_COMP_CHECKSUM_XXHASH32``.
+ * Added support for ``lz4`` in test-compress-perf algo options.
* **Allowed test single compress operation in test-compress-perf.**
diff --git a/doc/guides/tools/comp_perf.rst b/doc/guides/tools/comp_perf.rst
index cf9035cc23..5eb05a095c 100644
--- a/doc/guides/tools/comp_perf.rst
+++ b/doc/guides/tools/comp_perf.rst
@@ -84,10 +84,13 @@ Application Options
``--operation [comp/decomp/comp_and_decomp]``: perform test on compression, decompression or both operations
- ``--algo [null/deflate/lzs]`` : perform test on algorithm null(DMA), Deflate or lzs (default: Deflate)
+ ``--algo [null/deflate/lzs/lz4]`` : perform test on algorithm null(DMA), Deflate, lzs or lz4 (default: Deflate)
``--huffman-enc [fixed/dynamic/default]``: Huffman encoding (default: dynamic)
+ ``--lz4-flags N``: flags to for LZ4 parameters,
+ see `LZ4 Frame Descriptor <https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor>`_ (default: no flags)
+
``--compress-level N``: compression level, which could be a single value, list or range (default: range between 1 and 9)
``--window-sz N``: base two log value of compression window size (default: max supported by PMD)
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* RE: [EXT] [PATCH 1/4] compressdev: add LZ4 algorithm support
2023-01-09 7:45 ` [PATCH 1/4] compressdev: add LZ4 algorithm support Michael Baum
@ 2023-01-30 18:35 ` Akhil Goyal
2023-01-30 20:36 ` Michael Baum
0 siblings, 1 reply; 31+ messages in thread
From: Akhil Goyal @ 2023-01-30 18:35 UTC (permalink / raw)
To: Michael Baum, dev, Mahipal Challa, Fan Zhang, Ashish Gupta
Cc: Matan Azrad, Fiona Trahe, Thomas Monjalon
> +/**
> + * Block checksum flag.
> + * If this flag is set, each data block will be followed by a 4-bytes checksum,
> + * calculated by using the xxHash-32 algorithm on the raw (compressed) data
> + * block. The intention is to detect data corruption (storage or transmission
> + * errors) immediately, before decoding. Block checksum usage is optional.
> + */
> +#define RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM (1 << 4)
> +
> +/**
> + * Block Independence flag.
> + * If this flag is set to 1, blocks are independent.
> + * If this flag is set to 0, each block depends on previous ones (up to LZ4
> + * window size, which is 64 KB). In such case, it is necessary to decode all
> + * blocks in sequence.
> + * Block dependency improves compression ratio, especially for small blocks.
> On
> + * the other hand, it makes random access or multi-threaded decoding
> impossible.
> + */
> +#define RTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE (1 << 5)
Why did you start with 4th and 5th bit of the flags? Why not first two bits?
++ more people for review.
^ permalink raw reply [flat|nested] 31+ messages in thread
* RE: [EXT] [PATCH 1/4] compressdev: add LZ4 algorithm support
2023-01-30 18:35 ` [EXT] " Akhil Goyal
@ 2023-01-30 20:36 ` Michael Baum
2023-01-31 6:29 ` Akhil Goyal
0 siblings, 1 reply; 31+ messages in thread
From: Michael Baum @ 2023-01-30 20:36 UTC (permalink / raw)
To: Akhil Goyal, dev, Mahipal Challa, Fan Zhang, Ashish Gupta
Cc: Matan Azrad, Fiona Trahe, NBU-Contact-Thomas Monjalon (EXTERNAL)
On Mon, Jan 30, 2023 at 20:35 PM Akhil Goyal <gakhil@marvell.com> wrote:
>
> > +/**
> > + * Block checksum flag.
> > + * If this flag is set, each data block will be followed by a 4-bytes checksum,
> > + * calculated by using the xxHash-32 algorithm on the raw (compressed) data
> > + * block. The intention is to detect data corruption (storage or transmission
> > + * errors) immediately, before decoding. Block checksum usage is optional.
> > + */
> > +#define RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM (1 << 4)
> > +
> > +/**
> > + * Block Independence flag.
> > + * If this flag is set to 1, blocks are independent.
> > + * If this flag is set to 0, each block depends on previous ones (up to LZ4
> > + * window size, which is 64 KB). In such case, it is necessary to decode all
> > + * blocks in sequence.
> > + * Block dependency improves compression ratio, especially for small blocks.
> > On
> > + * the other hand, it makes random access or multi-threaded decoding
> > impossible.
> > + */
> > +#define RTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE (1 << 5)
>
> Why did you start with 4th and 5th bit of the flags? Why not first two bits?
I didn't choose the values by myself, I took them from LZ4 standard:
https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor
>
> ++ more people for review.
^ permalink raw reply [flat|nested] 31+ messages in thread
* RE: [EXT] [PATCH 1/4] compressdev: add LZ4 algorithm support
2023-01-30 20:36 ` Michael Baum
@ 2023-01-31 6:29 ` Akhil Goyal
0 siblings, 0 replies; 31+ messages in thread
From: Akhil Goyal @ 2023-01-31 6:29 UTC (permalink / raw)
To: Michael Baum, dev, Mahipal Challa, Fan Zhang, Ashish Gupta
Cc: Matan Azrad, Fiona Trahe, NBU-Contact-Thomas Monjalon (EXTERNAL), Kai Ji
> -----Original Message-----
> From: Michael Baum <michaelba@nvidia.com>
> Sent: Tuesday, January 31, 2023 2:07 AM
> To: Akhil Goyal <gakhil@marvell.com>; dev@dpdk.org; Mahipal Challa
> <mchalla@marvell.com>; Fan Zhang <fanzhang.oss@gmail.com>; Ashish Gupta
> <ashishg@marvell.com>
> Cc: Matan Azrad <matan@nvidia.com>; Fiona Trahe <fiona.trahe@intel.com>;
> NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>
> Subject: RE: [EXT] [PATCH 1/4] compressdev: add LZ4 algorithm support
>
> On Mon, Jan 30, 2023 at 20:35 PM Akhil Goyal <gakhil@marvell.com> wrote:
> >
> > > +/**
> > > + * Block checksum flag.
> > > + * If this flag is set, each data block will be followed by a 4-bytes checksum,
> > > + * calculated by using the xxHash-32 algorithm on the raw (compressed)
> data
> > > + * block. The intention is to detect data corruption (storage or transmission
> > > + * errors) immediately, before decoding. Block checksum usage is optional.
> > > + */
> > > +#define RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM (1 << 4)
> > > +
> > > +/**
> > > + * Block Independence flag.
> > > + * If this flag is set to 1, blocks are independent.
> > > + * If this flag is set to 0, each block depends on previous ones (up to LZ4
> > > + * window size, which is 64 KB). In such case, it is necessary to decode all
> > > + * blocks in sequence.
> > > + * Block dependency improves compression ratio, especially for small
> blocks.
> > > On
> > > + * the other hand, it makes random access or multi-threaded decoding
> > > impossible.
> > > + */
> > > +#define RTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE (1 << 5)
> >
> > Why did you start with 4th and 5th bit of the flags? Why not first two bits?
>
> I didn't choose the values by myself, I took them from LZ4 standard:
> https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor
Ok, Are we defining and making the frame descriptor inside library?
Also, why all the bits not added?
I believe these are just flags which PMD would use to make the frame descriptors.
If so, we should start with 0th bit and let the PMD handle bit shifting.
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v2 0/4] compressdev: add LZ4 support
2023-01-09 7:45 [PATCH 0/4] compressdev: add LZ4 support Michael Baum
` (3 preceding siblings ...)
2023-01-09 7:45 ` [PATCH 4/4] app/test-compress-perf: add LZ4 support Michael Baum
@ 2023-02-02 9:13 ` Michael Baum
2023-02-02 9:13 ` [PATCH v2 1/4] compressdev: add LZ4 algorithm support Michael Baum
` (4 more replies)
4 siblings, 5 replies; 31+ messages in thread
From: Michael Baum @ 2023-02-02 9:13 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe, Thomas Monjalon
Add support for LZ4 compression algorithm for both API and app.
In addition, enable app to test de/comp only. This option already
exists in both app doc and arguments but isn't implemented inside the
application.
LZ4: https://github.com/lz4/lz4
RFC:
https://patchwork.dpdk.org/project/dpdk/patch/20220410182622.8828-1-rzidane@nvidia.com/
v2:
- Add macros for all LZ4 flags in the standard.
- Rebase.
Depends-on: series-26738 ("compressdev: fix end of list enums conflict")
Michael Baum (4):
compressdev: add LZ4 algorithm support
app/test-compress-perf: allow test single compress operation
app/test-compress-perf: add algo option
app/test-compress-perf: add LZ4 support
app/test-compress-perf/comp_perf_options.h | 9 +-
.../comp_perf_options_parse.c | 83 +++++++++++-
.../comp_perf_test_common.c | 126 +++++++++++++-----
.../comp_perf_test_cyclecount.c | 82 +++++++-----
.../comp_perf_test_throughput.c | 75 +++++++----
.../comp_perf_test_verify.c | 75 +++++++----
app/test-compress-perf/main.c | 63 +++++++--
doc/guides/compressdevs/features/default.ini | 7 +
doc/guides/rel_notes/release_23_03.rst | 16 +++
doc/guides/tools/comp_perf.rst | 5 +
lib/compressdev/rte_comp.c | 12 ++
lib/compressdev/rte_comp.h | 79 ++++++++++-
12 files changed, 488 insertions(+), 144 deletions(-)
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v2 1/4] compressdev: add LZ4 algorithm support
2023-02-02 9:13 ` [PATCH v2 0/4] compressdev: " Michael Baum
@ 2023-02-02 9:13 ` Michael Baum
2023-02-05 17:20 ` [EXT] " Akhil Goyal
2023-02-02 9:13 ` [PATCH v2 2/4] app/test-compress-perf: allow test single compress operation Michael Baum
` (3 subsequent siblings)
4 siblings, 1 reply; 31+ messages in thread
From: Michael Baum @ 2023-02-02 9:13 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe, Thomas Monjalon
Add support for LZ4 algorithm:
- Add Lz4 param structure to XFORM structures.
- Add capabilities flags for LZ4 params.
- Add xxHash-32 checksum and capabilities flag.
Signed-off-by: Michael Baum <michaelba@nvidia.com>
---
doc/guides/compressdevs/features/default.ini | 7 ++
doc/guides/rel_notes/release_23_03.rst | 7 ++
lib/compressdev/rte_comp.c | 12 +++
lib/compressdev/rte_comp.h | 79 +++++++++++++++++++-
4 files changed, 103 insertions(+), 2 deletions(-)
diff --git a/doc/guides/compressdevs/features/default.ini b/doc/guides/compressdevs/features/default.ini
index e1419ee8db..2f178c5efd 100644
--- a/doc/guides/compressdevs/features/default.ini
+++ b/doc/guides/compressdevs/features/default.ini
@@ -20,8 +20,15 @@ OOP SGL In LB Out =
OOP LB In SGL Out =
Deflate =
LZS =
+LZ4 =
Adler32 =
Crc32 =
Adler32&Crc32 =
+xxHash32 =
Fixed =
Dynamic =
+LZ4 Dictionary ID =
+LZ4 Content Checksum =
+LZ4 Content Size =
+LZ4 Block Checksum =
+LZ4 Block Independence =
diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index 73f5d94e14..2722296180 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -78,6 +78,13 @@ New Features
``rte_event_dev_config::nb_single_link_event_port_queues`` parameter
required for eth_rx, eth_tx, crypto and timer eventdev adapters.
+* **Added LZ4 algorithm in Compressdev Library.**
+
+ Added new compression algorithm, including:
+
+ * Added support for ``RTE_COMP_ALGO_LZ4``.
+ * Added support for ``RTE_COMP_CHECKSUM_XXHASH32``.
+
Removed Items
-------------
diff --git a/lib/compressdev/rte_comp.c b/lib/compressdev/rte_comp.c
index 320c6dab92..f060c68557 100644
--- a/lib/compressdev/rte_comp.c
+++ b/lib/compressdev/rte_comp.c
@@ -39,6 +39,18 @@ rte_comp_get_feature_name(uint64_t flag)
return "HUFFMAN_FIXED";
case RTE_COMP_FF_HUFFMAN_DYNAMIC:
return "HUFFMAN_DYNAMIC";
+ case RTE_COMP_FF_XXHASH32_CHECKSUM:
+ return "XXHASH32_CHECKSUM";
+ case RTE_COMP_FF_LZ4_DICT_ID:
+ return "LZ4_DICT_ID";
+ case RTE_COMP_FF_LZ4_CONTENT_WITH_CHECKSUM:
+ return "LZ4_CONTENT_WITH_CHECKSUM";
+ case RTE_COMP_FF_LZ4_CONTENT_SIZE:
+ return "LZ4_CONTENT_SIZE";
+ case RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE:
+ return "LZ4_BLOCK_INDEPENDENCE";
+ case RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM:
+ return "LZ4_BLOCK_WITH_CHECKSUM";
default:
return NULL;
}
diff --git a/lib/compressdev/rte_comp.h b/lib/compressdev/rte_comp.h
index 5bd711fda1..2096fb2407 100644
--- a/lib/compressdev/rte_comp.h
+++ b/lib/compressdev/rte_comp.h
@@ -67,6 +67,18 @@ extern "C" {
/**< Fixed huffman encoding is supported */
#define RTE_COMP_FF_HUFFMAN_DYNAMIC (1ULL << 14)
/**< Dynamic huffman encoding is supported */
+#define RTE_COMP_FF_XXHASH32_CHECKSUM (1ULL << 15)
+/**< xxHash-32 Checksum is supported */
+#define RTE_COMP_FF_LZ4_DICT_ID (1ULL << 16)
+/**< LZ4 dictionary ID is supported */
+#define RTE_COMP_FF_LZ4_CONTENT_WITH_CHECKSUM (1ULL << 17)
+/**< LZ4 content with checksum is supported */
+#define RTE_COMP_FF_LZ4_CONTENT_SIZE (1ULL << 18)
+/**< LZ4 content size is supported */
+#define RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE (1ULL << 19)
+/**< LZ4 block independent is supported */
+#define RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM (1ULL << 20)
+/**< LZ4 block with checksum is supported */
/** Status of comp operation */
enum rte_comp_op_status {
@@ -109,6 +121,10 @@ enum rte_comp_algorithm {
/**< LZS compression algorithm
* https://tools.ietf.org/html/rfc2395
*/
+ RTE_COMP_ALGO_LZ4,
+ /**< LZ4 compression algorithm
+ * https://github.com/lz4/lz4
+ */
};
/** Compression Hash Algorithms */
@@ -147,9 +163,12 @@ enum rte_comp_checksum_type {
/**< Generates both Adler-32 and CRC32 checksums, concatenated.
* CRC32 is in the lower 32bits, Adler-32 in the upper 32 bits.
*/
+ RTE_COMP_CHECKSUM_XXHASH32,
+ /**< Generates a xxHash-32 checksum, as used by lz4.
+ * https://github.com/Cyan4973/xxHash/blob/dev/doc/xxhash_spec.md
+ */
};
-
/** Compression Huffman Type - used by DEFLATE algorithm */
enum rte_comp_huffman {
RTE_COMP_HUFFMAN_DEFAULT,
@@ -206,13 +225,63 @@ enum rte_comp_op_type {
*/
};
-
/** Parameters specific to the deflate algorithm */
struct rte_comp_deflate_params {
enum rte_comp_huffman huffman;
/**< Compression huffman encoding type */
};
+/**
+ * Dictionary ID flag
+ * If this flag is set, a 4-bytes Dict-ID field will be present, after the
+ * descriptor flags and the Content Size.
+ */
+#define RTE_COMP_LZ4_FLAG_DICT_ID (1 << 0)
+
+/**
+ * Content Checksum flag
+ * If this flag is set, a 32-bits content checksum will be appended after the
+ * EndMark.
+ */
+#define RTE_COMP_LZ4_FLAG_CONTENT_CHECKSUM (1 << 2)
+
+/**
+ * Content Size flag
+ * If this flag is set, the uncompressed size of data included within the frame
+ * will be present as an 8 bytes unsigned little-endian value, after the flags.
+ * Content Size usage is optional.
+ */
+#define RTE_COMP_LZ4_FLAG_CONTENT_SIZE (1 << 3)
+
+/**
+ * Block Checksum flag.
+ * If this flag is set, each data block will be followed by a 4-bytes checksum,
+ * calculated by using the xxHash-32 algorithm on the raw (compressed) data
+ * block. The intention is to detect data corruption (storage or transmission
+ * errors) immediately, before decoding. Block checksum usage is optional.
+ */
+#define RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM (1 << 4)
+
+/**
+ * Block Independence flag.
+ * If this flag is set to 1, blocks are independent.
+ * If this flag is set to 0, each block depends on previous ones (up to LZ4
+ * window size, which is 64 KB). In such case, it is necessary to decode all
+ * blocks in sequence.
+ * Block dependency improves compression ratio, especially for small blocks. On
+ * the other hand, it makes random access or multi-threaded decoding impossible.
+ */
+#define RTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE (1 << 5)
+
+/** Parameters specific to the LZ4 algorithm */
+struct rte_comp_lz4_params {
+ uint8_t flags;
+ /**< Compression LZ4 parameter flags.
+ * Based on LZ4 standard flags:
+ * https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor
+ */
+};
+
/** Setup Data for compression */
struct rte_comp_compress_xform {
enum rte_comp_algorithm algo;
@@ -220,6 +289,8 @@ struct rte_comp_compress_xform {
union {
struct rte_comp_deflate_params deflate;
/**< Parameters specific to the deflate algorithm */
+ struct rte_comp_lz4_params lz4;
+ /**< Parameters specific to the LZ4 algorithm */
}; /**< Algorithm specific parameters */
int level;
/**< Compression level */
@@ -249,6 +320,10 @@ struct rte_comp_decompress_xform {
* compressed data. If window size can't be supported by the PMD then
* setup of stream or private_xform should fail.
*/
+ union {
+ struct rte_comp_lz4_params lz4;
+ /**< Parameters specific to the LZ4 algorithm */
+ }; /**< Algorithm specific parameters */
enum rte_comp_hash_algorithm hash_algo;
/**< Hash algorithm to be used with decompress operation. Hash is always
* done on plaintext.
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v2 2/4] app/test-compress-perf: allow test single compress operation
2023-02-02 9:13 ` [PATCH v2 0/4] compressdev: " Michael Baum
2023-02-02 9:13 ` [PATCH v2 1/4] compressdev: add LZ4 algorithm support Michael Baum
@ 2023-02-02 9:13 ` Michael Baum
2023-02-02 9:13 ` [PATCH v2 3/4] app/test-compress-perf: add algo option Michael Baum
` (2 subsequent siblings)
4 siblings, 0 replies; 31+ messages in thread
From: Michael Baum @ 2023-02-02 9:13 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe, Thomas Monjalon
Part of the application options is to test only compress and only
decompress but actually the application ignores this user option and
tries to test always both compress and decompress.
Allow testing only compress and only decompress.
Signed-off-by: Matan Azrad <matan@nvidia.com>
Signed-off-by: Michael Baum <michaelba@nvidia.com>
---
app/test-compress-perf/comp_perf_options.h | 6 +-
.../comp_perf_options_parse.c | 4 +-
.../comp_perf_test_common.c | 126 +++++++++++++-----
.../comp_perf_test_cyclecount.c | 71 ++++++----
.../comp_perf_test_throughput.c | 64 +++++----
.../comp_perf_test_verify.c | 64 +++++----
app/test-compress-perf/main.c | 8 ++
doc/guides/rel_notes/release_23_03.rst | 4 +
8 files changed, 230 insertions(+), 117 deletions(-)
diff --git a/app/test-compress-perf/comp_perf_options.h b/app/test-compress-perf/comp_perf_options.h
index 57dd146330..d00b299247 100644
--- a/app/test-compress-perf/comp_perf_options.h
+++ b/app/test-compress-perf/comp_perf_options.h
@@ -32,9 +32,9 @@ enum cperf_test_type {
};
enum comp_operation {
- COMPRESS_ONLY,
- DECOMPRESS_ONLY,
- COMPRESS_DECOMPRESS
+ COMPRESS = (1 << 0),
+ DECOMPRESS = (1 << 1),
+ COMPRESS_DECOMPRESS = (COMPRESS | DECOMPRESS),
};
struct range_list {
diff --git a/app/test-compress-perf/comp_perf_options_parse.c b/app/test-compress-perf/comp_perf_options_parse.c
index 019eddb7bd..7a992bf43e 100644
--- a/app/test-compress-perf/comp_perf_options_parse.c
+++ b/app/test-compress-perf/comp_perf_options_parse.c
@@ -446,11 +446,11 @@ parse_op_type(struct comp_test_data *test_data, const char *arg)
struct name_id_map optype_namemap[] = {
{
"comp",
- COMPRESS_ONLY
+ COMPRESS
},
{
"decomp",
- DECOMPRESS_ONLY
+ DECOMPRESS
},
{
"comp_and_decomp",
diff --git a/app/test-compress-perf/comp_perf_test_common.c b/app/test-compress-perf/comp_perf_test_common.c
index b402a0d839..cd60958944 100644
--- a/app/test-compress-perf/comp_perf_test_common.c
+++ b/app/test-compress-perf/comp_perf_test_common.c
@@ -227,23 +227,43 @@ comp_perf_allocate_memory(struct comp_test_data *test_data,
{
uint16_t comp_mbuf_size;
uint16_t decomp_mbuf_size;
+ size_t comp_data_size;
+ size_t decomp_data_size;
+ size_t output_data_sz;
test_data->out_seg_sz = find_buf_size(test_data->seg_sz);
- /* Number of segments for input and output
- * (compression and decompression)
- */
- test_data->total_segs = DIV_CEIL(test_data->input_data_sz,
- test_data->seg_sz);
+ if (test_data->test_op & COMPRESS) {
+ /*
+ * Number of segments for input and output
+ * (compression and decompression)
+ */
+ test_data->total_segs = DIV_CEIL(test_data->input_data_sz,
+ test_data->seg_sz);
+ } else {
+ /*
+ * When application does decompression only, input data is
+ * compressed and smaller than the output. The expected size of
+ * uncompressed data given by the user in segment size argument.
+ */
+ test_data->total_segs = test_data->max_sgl_segs;
+ }
+
+ output_data_sz = (size_t) test_data->out_seg_sz * test_data->total_segs;
+ output_data_sz =
+ RTE_MAX(output_data_sz, (size_t) MIN_COMPRESSED_BUF_SIZE);
if (test_data->use_external_mbufs != 0) {
if (comp_perf_allocate_external_mbufs(test_data, mem) < 0)
return -1;
comp_mbuf_size = 0;
decomp_mbuf_size = 0;
- } else {
+ } else if (test_data->test_op & COMPRESS) {
comp_mbuf_size = test_data->out_seg_sz + RTE_PKTMBUF_HEADROOM;
decomp_mbuf_size = test_data->seg_sz + RTE_PKTMBUF_HEADROOM;
+ } else {
+ comp_mbuf_size = test_data->seg_sz + RTE_PKTMBUF_HEADROOM;
+ decomp_mbuf_size = test_data->out_seg_sz + RTE_PKTMBUF_HEADROOM;
}
char pool_name[32] = "";
@@ -287,26 +307,28 @@ comp_perf_allocate_memory(struct comp_test_data *test_data,
return -1;
}
- /*
- * Compressed data might be a bit larger than input data,
- * if data cannot be compressed
- */
- mem->compressed_data = rte_zmalloc_socket(NULL,
- RTE_MAX(
- (size_t) test_data->out_seg_sz *
- test_data->total_segs,
- (size_t) MIN_COMPRESSED_BUF_SIZE),
- 0,
- rte_socket_id());
+ if (test_data->test_op & COMPRESS) {
+ /*
+ * Compressed data might be a bit larger than input data,
+ * if data cannot be compressed
+ */
+ comp_data_size = output_data_sz;
+ decomp_data_size = test_data->input_data_sz;
+ } else {
+ comp_data_size = test_data->input_data_sz;
+ decomp_data_size = output_data_sz;
+ }
+
+ mem->compressed_data = rte_zmalloc_socket(NULL, comp_data_size, 0,
+ rte_socket_id());
if (mem->compressed_data == NULL) {
RTE_LOG(ERR, USER1, "Memory to hold the data from the input "
"file could not be allocated\n");
return -1;
}
- mem->decompressed_data = rte_zmalloc_socket(NULL,
- test_data->input_data_sz, 0,
- rte_socket_id());
+ mem->decompressed_data = rte_zmalloc_socket(NULL, decomp_data_size, 0,
+ rte_socket_id());
if (mem->decompressed_data == NULL) {
RTE_LOG(ERR, USER1, "Memory to hold the data from the input "
"file could not be allocated\n");
@@ -351,6 +373,7 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
uint16_t segs_per_mbuf = 0;
uint32_t cmz = 0;
uint32_t dmz = 0;
+ bool decompress_only = !!(test_data->test_op == DECOMPRESS);
for (i = 0; i < mem->total_bufs; i++) {
/* Allocate data in input mbuf and copy data from input file */
@@ -361,8 +384,6 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
return -1;
}
- data_sz = RTE_MIN(remaining_data, test_data->seg_sz);
-
if (test_data->use_external_mbufs != 0) {
rte_pktmbuf_attach_extbuf(mem->decomp_bufs[i],
mem->decomp_memzones[dmz]->addr,
@@ -372,16 +393,23 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
dmz++;
}
+ if (!decompress_only)
+ data_sz = RTE_MIN(remaining_data, test_data->seg_sz);
+ else
+ data_sz = test_data->out_seg_sz;
+
data_addr = (uint8_t *) rte_pktmbuf_append(
mem->decomp_bufs[i], data_sz);
if (data_addr == NULL) {
RTE_LOG(ERR, USER1, "Could not append data\n");
return -1;
}
- rte_memcpy(data_addr, input_data_ptr, data_sz);
- input_data_ptr += data_sz;
- remaining_data -= data_sz;
+ if (!decompress_only) {
+ rte_memcpy(data_addr, input_data_ptr, data_sz);
+ input_data_ptr += data_sz;
+ remaining_data -= data_sz;
+ }
/* Already one segment in the mbuf */
segs_per_mbuf = 1;
@@ -398,8 +426,6 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
return -1;
}
- data_sz = RTE_MIN(remaining_data, test_data->seg_sz);
-
if (test_data->use_external_mbufs != 0) {
rte_pktmbuf_attach_extbuf(
next_seg,
@@ -410,6 +436,12 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
dmz++;
}
+ if (!decompress_only)
+ data_sz = RTE_MIN(remaining_data,
+ test_data->seg_sz);
+ else
+ data_sz = test_data->out_seg_sz;
+
data_addr = (uint8_t *)rte_pktmbuf_append(next_seg,
data_sz);
@@ -418,9 +450,11 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
return -1;
}
- rte_memcpy(data_addr, input_data_ptr, data_sz);
- input_data_ptr += data_sz;
- remaining_data -= data_sz;
+ if (!decompress_only) {
+ rte_memcpy(data_addr, input_data_ptr, data_sz);
+ input_data_ptr += data_sz;
+ remaining_data -= data_sz;
+ }
if (rte_pktmbuf_chain(mem->decomp_bufs[i],
next_seg) < 0) {
@@ -447,16 +481,26 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
cmz++;
}
- data_addr = (uint8_t *) rte_pktmbuf_append(
- mem->comp_bufs[i],
- test_data->out_seg_sz);
+ if (decompress_only)
+ data_sz = RTE_MIN(remaining_data, test_data->seg_sz);
+ else
+ data_sz = test_data->out_seg_sz;
+
+ data_addr = (uint8_t *) rte_pktmbuf_append(mem->comp_bufs[i],
+ data_sz);
if (data_addr == NULL) {
RTE_LOG(ERR, USER1, "Could not append data\n");
return -1;
}
+ if (decompress_only) {
+ rte_memcpy(data_addr, input_data_ptr, data_sz);
+ input_data_ptr += data_sz;
+ remaining_data -= data_sz;
+ }
+
/* Chain mbufs if needed for output mbufs */
- for (j = 1; j < segs_per_mbuf; j++) {
+ for (j = 1; j < segs_per_mbuf && remaining_data > 0; j++) {
struct rte_mbuf *next_seg =
rte_pktmbuf_alloc(mem->comp_buf_pool);
@@ -476,13 +520,25 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
cmz++;
}
+ if (decompress_only)
+ data_sz = RTE_MIN(remaining_data,
+ test_data->seg_sz);
+ else
+ data_sz = test_data->out_seg_sz;
+
data_addr = (uint8_t *)rte_pktmbuf_append(next_seg,
- test_data->out_seg_sz);
+ data_sz);
if (data_addr == NULL) {
RTE_LOG(ERR, USER1, "Could not append data\n");
return -1;
}
+ if (decompress_only) {
+ rte_memcpy(data_addr, input_data_ptr, data_sz);
+ input_data_ptr += data_sz;
+ remaining_data -= data_sz;
+ }
+
if (rte_pktmbuf_chain(mem->comp_bufs[i],
next_seg) < 0) {
RTE_LOG(ERR, USER1, "Could not chain mbufs\n");
diff --git a/app/test-compress-perf/comp_perf_test_cyclecount.c b/app/test-compress-perf/comp_perf_test_cyclecount.c
index 7473cb6277..ce6c4d7605 100644
--- a/app/test-compress-perf/comp_perf_test_cyclecount.c
+++ b/app/test-compress-perf/comp_perf_test_cyclecount.c
@@ -514,38 +514,55 @@ cperf_cyclecount_test_runner(void *test_ctx)
if (cperf_verify_test_runner(&ctx->ver))
return EXIT_FAILURE;
- /*
- * Run the tests twice, discarding the first performance
- * results, before the cache is warmed up
- */
-
- /* C O M P R E S S */
- for (i = 0; i < 2; i++) {
- if (main_loop(ctx, RTE_COMP_COMPRESS) < 0)
- return EXIT_FAILURE;
- }
+ if (test_data->test_op & COMPRESS) {
+ /*
+ * Run the test twice, discarding the first performance
+ * results, before the cache is warmed up
+ */
+ for (i = 0; i < 2; i++) {
+ if (main_loop(ctx, RTE_COMP_COMPRESS) < 0)
+ return EXIT_FAILURE;
+ }
- ops_enq_retries_comp = ctx->ops_enq_retries;
- ops_deq_retries_comp = ctx->ops_deq_retries;
+ ops_enq_retries_comp = ctx->ops_enq_retries;
+ ops_deq_retries_comp = ctx->ops_deq_retries;
- duration_enq_per_op_comp = ctx->duration_enq /
- (ctx->ver.mem.total_bufs * test_data->num_iter);
- duration_deq_per_op_comp = ctx->duration_deq /
- (ctx->ver.mem.total_bufs * test_data->num_iter);
+ duration_enq_per_op_comp = ctx->duration_enq /
+ (ctx->ver.mem.total_bufs * test_data->num_iter);
+ duration_deq_per_op_comp = ctx->duration_deq /
+ (ctx->ver.mem.total_bufs * test_data->num_iter);
+ } else {
+ ops_enq_retries_comp = 0;
+ ops_deq_retries_comp = 0;
- /* D E C O M P R E S S */
- for (i = 0; i < 2; i++) {
- if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0)
- return EXIT_FAILURE;
+ duration_enq_per_op_comp = 0;
+ duration_deq_per_op_comp = 0;
}
- ops_enq_retries_decomp = ctx->ops_enq_retries;
- ops_deq_retries_decomp = ctx->ops_deq_retries;
+ if (test_data->test_op & DECOMPRESS) {
+ /*
+ * Run the test twice, discarding the first performance
+ * results, before the cache is warmed up
+ */
+ for (i = 0; i < 2; i++) {
+ if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0)
+ return EXIT_FAILURE;
+ }
- duration_enq_per_op_decomp = ctx->duration_enq /
- (ctx->ver.mem.total_bufs * test_data->num_iter);
- duration_deq_per_op_decomp = ctx->duration_deq /
- (ctx->ver.mem.total_bufs * test_data->num_iter);
+ ops_enq_retries_decomp = ctx->ops_enq_retries;
+ ops_deq_retries_decomp = ctx->ops_deq_retries;
+
+ duration_enq_per_op_decomp = ctx->duration_enq /
+ (ctx->ver.mem.total_bufs * test_data->num_iter);
+ duration_deq_per_op_decomp = ctx->duration_deq /
+ (ctx->ver.mem.total_bufs * test_data->num_iter);
+ } else {
+ ops_enq_retries_decomp = 0;
+ ops_deq_retries_decomp = 0;
+
+ duration_enq_per_op_decomp = 0;
+ duration_deq_per_op_decomp = 0;
+ }
duration_setup_per_op = ctx->duration_op /
(ctx->ver.mem.total_bufs * test_data->num_iter);
@@ -563,7 +580,7 @@ cperf_cyclecount_test_runner(void *test_ctx)
" [D-e] - decompression enqueue\n"
" [D-d] - decompression dequeue\n"
" - Cycles section: number of cycles per 'op' for the following operations:\n"
- " setup/op - memory allocation, op configuration and memory dealocation\n"
+ " setup/op - memory allocation, op configuration and memory deallocation\n"
" [C-e] - compression enqueue\n"
" [C-d] - compression dequeue\n"
" [D-e] - decompression enqueue\n"
diff --git a/app/test-compress-perf/comp_perf_test_throughput.c b/app/test-compress-perf/comp_perf_test_throughput.c
index 79cd2b2bf2..c9f8237626 100644
--- a/app/test-compress-perf/comp_perf_test_throughput.c
+++ b/app/test-compress-perf/comp_perf_test_throughput.c
@@ -359,41 +359,53 @@ cperf_throughput_test_runner(void *test_ctx)
* First the verification part is needed
*/
if (cperf_verify_test_runner(&ctx->ver)) {
- ret = EXIT_FAILURE;
+ ret = EXIT_FAILURE;
goto end;
}
- /*
- * Run the tests twice, discarding the first performance
- * results, before the cache is warmed up
- */
- for (i = 0; i < 2; i++) {
- if (main_loop(ctx, RTE_COMP_COMPRESS) < 0) {
- ret = EXIT_FAILURE;
- goto end;
+ if (test_data->test_op & COMPRESS) {
+ /*
+ * Run the test twice, discarding the first performance
+ * results, before the cache is warmed up
+ */
+ for (i = 0; i < 2; i++) {
+ if (main_loop(ctx, RTE_COMP_COMPRESS) < 0) {
+ ret = EXIT_FAILURE;
+ goto end;
+ }
}
- }
- for (i = 0; i < 2; i++) {
- if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0) {
- ret = EXIT_FAILURE;
- goto end;
- }
+ ctx->comp_tsc_byte =
+ (double)(ctx->comp_tsc_duration[test_data->level]) /
+ test_data->input_data_sz;
+ ctx->comp_gbps = rte_get_tsc_hz() / ctx->comp_tsc_byte * 8 /
+ 1000000000;
+ } else {
+ ctx->comp_tsc_byte = 0;
+ ctx->comp_gbps = 0;
}
- ctx->comp_tsc_byte =
- (double)(ctx->comp_tsc_duration[test_data->level]) /
- test_data->input_data_sz;
+ if (test_data->test_op & DECOMPRESS) {
+ /*
+ * Run the test twice, discarding the first performance
+ * results, before the cache is warmed up
+ */
+ for (i = 0; i < 2; i++) {
+ if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0) {
+ ret = EXIT_FAILURE;
+ goto end;
+ }
+ }
- ctx->decomp_tsc_byte =
+ ctx->decomp_tsc_byte =
(double)(ctx->decomp_tsc_duration[test_data->level]) /
- test_data->input_data_sz;
-
- ctx->comp_gbps = rte_get_tsc_hz() / ctx->comp_tsc_byte * 8 /
- 1000000000;
-
- ctx->decomp_gbps = rte_get_tsc_hz() / ctx->decomp_tsc_byte * 8 /
- 1000000000;
+ test_data->input_data_sz;
+ ctx->decomp_gbps = rte_get_tsc_hz() / ctx->decomp_tsc_byte * 8 /
+ 1000000000;
+ } else {
+ ctx->decomp_tsc_byte = 0;
+ ctx->decomp_gbps = 0;
+ }
exp = 0;
if (__atomic_compare_exchange_n(&display_once, &exp, 1, 0,
diff --git a/app/test-compress-perf/comp_perf_test_verify.c b/app/test-compress-perf/comp_perf_test_verify.c
index 8964442891..7d6b6abecd 100644
--- a/app/test-compress-perf/comp_perf_test_verify.c
+++ b/app/test-compress-perf/comp_perf_test_verify.c
@@ -114,7 +114,8 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
output_data_sz = &ctx->decomp_data_sz;
input_bufs = mem->comp_bufs;
output_bufs = mem->decomp_bufs;
- out_seg_sz = test_data->seg_sz;
+ out_seg_sz = (test_data->test_op & COMPRESS) ?
+ test_data->seg_sz : test_data->out_seg_sz;
}
/* Create private xform */
@@ -226,7 +227,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
op->status ==
RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE) {
RTE_LOG(ERR, USER1,
-"Out of space error occurred due to uncompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n");
+"Out of space error occurred due to incompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n");
res = -1;
goto end;
} else if (op->status !=
@@ -311,7 +312,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
op->status ==
RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE) {
RTE_LOG(ERR, USER1,
-"Out of space error occurred due to uncompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n");
+"Out of space error occurred due to incompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n");
res = -1;
goto end;
} else if (op->status !=
@@ -392,44 +393,59 @@ cperf_verify_test_runner(void *test_ctx)
int ret = EXIT_SUCCESS;
static uint16_t display_once;
uint32_t lcore = rte_lcore_id();
+ uint16_t exp = 0;
ctx->mem.lcore_id = lcore;
test_data->ratio = 0;
- if (main_loop(ctx, RTE_COMP_COMPRESS) < 0) {
- ret = EXIT_FAILURE;
- goto end;
+ if (test_data->test_op & COMPRESS) {
+ if (main_loop(ctx, RTE_COMP_COMPRESS) < 0) {
+ ret = EXIT_FAILURE;
+ goto end;
+ }
}
- if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0) {
- ret = EXIT_FAILURE;
- goto end;
- }
+ if (test_data->test_op & DECOMPRESS) {
+ if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0) {
+ ret = EXIT_FAILURE;
+ goto end;
+ }
- if (ctx->decomp_data_sz != test_data->input_data_sz) {
- RTE_LOG(ERR, USER1,
- "Decompressed data length not equal to input data length\n");
- RTE_LOG(ERR, USER1,
- "Decompressed size = %zu, expected = %zu\n",
- ctx->decomp_data_sz, test_data->input_data_sz);
- ret = EXIT_FAILURE;
- goto end;
- } else {
- if (memcmp(ctx->mem.decompressed_data,
- test_data->input_data,
- test_data->input_data_sz) != 0) {
+ if (!(test_data->test_op & COMPRESS)) {
+ /*
+ * For DECOMPRESS_ONLY mode there is no more
+ * verifications, reset the 'ratio' and 'comp_data_sz'
+ * fields for other tests report.
+ */
+ ctx->comp_data_sz = 0;
+ ctx->ratio = 0;
+ goto end;
+ }
+
+ if (ctx->decomp_data_sz != test_data->input_data_sz) {
+ RTE_LOG(ERR, USER1,
+ "Decompressed data length not equal to input data length\n");
RTE_LOG(ERR, USER1,
- "Decompressed data is not the same as file data\n");
+ "Decompressed size = %zu, expected = %zu\n",
+ ctx->decomp_data_sz, test_data->input_data_sz);
ret = EXIT_FAILURE;
goto end;
+ } else {
+ if (memcmp(ctx->mem.decompressed_data,
+ test_data->input_data,
+ test_data->input_data_sz) != 0) {
+ RTE_LOG(ERR, USER1,
+ "Decompressed data is not the same as file data\n");
+ ret = EXIT_FAILURE;
+ goto end;
+ }
}
}
ctx->ratio = (double) ctx->comp_data_sz /
test_data->input_data_sz * 100;
- uint16_t exp = 0;
if (!ctx->silent) {
if (__atomic_compare_exchange_n(&display_once, &exp, 1, 0,
__ATOMIC_RELAXED, __ATOMIC_RELAXED)) {
diff --git a/app/test-compress-perf/main.c b/app/test-compress-perf/main.c
index 41b8edc2bd..bbb4c7917b 100644
--- a/app/test-compress-perf/main.c
+++ b/app/test-compress-perf/main.c
@@ -254,6 +254,14 @@ comp_perf_dump_input_data(struct comp_test_data *test_data)
goto end;
}
+ if (!(test_data->test_op & COMPRESS) &&
+ test_data->input_data_sz >
+ (size_t) test_data->seg_sz * (size_t) test_data->max_sgl_segs) {
+ RTE_LOG(ERR, USER1,
+ "Size of input must be less than total segments\n");
+ goto end;
+ }
+
test_data->input_data = rte_zmalloc_socket(NULL,
test_data->input_data_sz, 0, rte_socket_id());
diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index 2722296180..15fa8e812b 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -85,6 +85,10 @@ New Features
* Added support for ``RTE_COMP_ALGO_LZ4``.
* Added support for ``RTE_COMP_CHECKSUM_XXHASH32``.
+* **Allowed test single compress operation in test-compress-perf.**
+
+ Enable the application options for testing only compress and only decompress.
+
Removed Items
-------------
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v2 3/4] app/test-compress-perf: add algo option
2023-02-02 9:13 ` [PATCH v2 0/4] compressdev: " Michael Baum
2023-02-02 9:13 ` [PATCH v2 1/4] compressdev: add LZ4 algorithm support Michael Baum
2023-02-02 9:13 ` [PATCH v2 2/4] app/test-compress-perf: allow test single compress operation Michael Baum
@ 2023-02-02 9:13 ` Michael Baum
2023-02-02 9:13 ` [PATCH v2 4/4] app/test-compress-perf: add LZ4 support Michael Baum
2023-02-13 6:11 ` [PATCH v3 0/4] compressdev: " Michael Baum
4 siblings, 0 replies; 31+ messages in thread
From: Michael Baum @ 2023-02-02 9:13 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe, Thomas Monjalon
Add a command line option `algo` to select the compress algorithm
supported by the compress API: null (DMA), deflate and lzs.
Default for deflate.
Signed-off-by: Matan Azrad <matan@nvidia.com>
Signed-off-by: Michael Baum <michaelba@nvidia.com>
---
app/test-compress-perf/comp_perf_options.h | 2 +
.../comp_perf_options_parse.c | 40 ++++++++++++++++++-
.../comp_perf_test_cyclecount.c | 4 +-
.../comp_perf_test_throughput.c | 4 +-
.../comp_perf_test_verify.c | 4 +-
app/test-compress-perf/main.c | 38 +++++++++++-------
doc/guides/rel_notes/release_23_03.rst | 4 ++
doc/guides/tools/comp_perf.rst | 2 +
8 files changed, 76 insertions(+), 22 deletions(-)
diff --git a/app/test-compress-perf/comp_perf_options.h b/app/test-compress-perf/comp_perf_options.h
index d00b299247..5e5227a700 100644
--- a/app/test-compress-perf/comp_perf_options.h
+++ b/app/test-compress-perf/comp_perf_options.h
@@ -63,6 +63,8 @@ struct comp_test_data {
enum rte_comp_huffman huffman_enc;
enum comp_operation test_op;
+ enum rte_comp_algorithm test_algo;
+
int window_sz;
struct range_list level_lst;
uint8_t level;
diff --git a/app/test-compress-perf/comp_perf_options_parse.c b/app/test-compress-perf/comp_perf_options_parse.c
index 7a992bf43e..97ddff87ef 100644
--- a/app/test-compress-perf/comp_perf_options_parse.c
+++ b/app/test-compress-perf/comp_perf_options_parse.c
@@ -25,6 +25,7 @@
#define CPERF_MAX_SGL_SEGS ("max-num-sgl-segs")
#define CPERF_NUM_ITER ("num-iter")
#define CPERF_OPTYPE ("operation")
+#define CPERF_ALGO ("algo")
#define CPERF_HUFFMAN_ENC ("huffman-enc")
#define CPERF_LEVEL ("compress-level")
#define CPERF_WINDOW_SIZE ("window-sz")
@@ -56,6 +57,8 @@ usage(char *progname)
" compressed/decompressed (default: 10000)\n"
" --operation [comp/decomp/comp_and_decomp]: perform test on\n"
" compression, decompression or both operations\n"
+ " --algo [null/deflate/lzs]: perform test on algorithm\n"
+ " null(DMA), deflate or lzs (default: deflate)\n"
" --huffman-enc [fixed/dynamic/default]: Huffman encoding\n"
" (default: dynamic)\n"
" --compress-level N: compression level, which could be a single value, list or range\n"
@@ -470,6 +473,36 @@ parse_op_type(struct comp_test_data *test_data, const char *arg)
return 0;
}
+static int
+parse_algo(struct comp_test_data *test_data, const char *arg)
+{
+ struct name_id_map algo_namemap[] = {
+ {
+ "null",
+ RTE_COMP_ALGO_NULL
+ },
+ {
+ "deflate",
+ RTE_COMP_ALGO_DEFLATE
+ },
+ {
+ "lzs",
+ RTE_COMP_ALGO_LZS
+ }
+ };
+
+ int id = get_str_key_id_mapping(algo_namemap,
+ RTE_DIM(algo_namemap), arg);
+ if (id < 0) {
+ RTE_LOG(ERR, USER1, "Invalid algorithm specified\n");
+ return -1;
+ }
+
+ test_data->test_algo = (enum rte_comp_algorithm)id;
+
+ return 0;
+}
+
static int
parse_huffman_enc(struct comp_test_data *test_data, const char *arg)
{
@@ -491,7 +524,7 @@ parse_huffman_enc(struct comp_test_data *test_data, const char *arg)
int id = get_str_key_id_mapping(huffman_namemap,
RTE_DIM(huffman_namemap), arg);
if (id < 0) {
- RTE_LOG(ERR, USER1, "Invalid Huffmane encoding specified\n");
+ RTE_LOG(ERR, USER1, "Invalid Huffman encoding specified\n");
return -1;
}
@@ -507,7 +540,7 @@ parse_level(struct comp_test_data *test_data, const char *arg)
/*
* Try parsing the argument as a range, if it fails,
- * arse it as a list
+ * parse it as a list
*/
if (parse_range(arg, &test_data->level_lst.min,
&test_data->level_lst.max,
@@ -572,6 +605,7 @@ static struct option lgopts[] = {
{ CPERF_MAX_SGL_SEGS, required_argument, 0, 0},
{ CPERF_NUM_ITER, required_argument, 0, 0 },
{ CPERF_OPTYPE, required_argument, 0, 0 },
+ { CPERF_ALGO, required_argument, 0, 0 },
{ CPERF_HUFFMAN_ENC, required_argument, 0, 0 },
{ CPERF_LEVEL, required_argument, 0, 0 },
{ CPERF_WINDOW_SIZE, required_argument, 0, 0 },
@@ -594,6 +628,7 @@ comp_perf_opts_parse_long(int opt_idx, struct comp_test_data *test_data)
{ CPERF_MAX_SGL_SEGS, parse_max_num_sgl_segs },
{ CPERF_NUM_ITER, parse_num_iter },
{ CPERF_OPTYPE, parse_op_type },
+ { CPERF_ALGO, parse_algo },
{ CPERF_HUFFMAN_ENC, parse_huffman_enc },
{ CPERF_LEVEL, parse_level },
{ CPERF_WINDOW_SIZE, parse_window_sz },
@@ -649,6 +684,7 @@ comp_perf_options_default(struct comp_test_data *test_data)
test_data->num_iter = 10000;
test_data->huffman_enc = RTE_COMP_HUFFMAN_DYNAMIC;
test_data->test_op = COMPRESS_DECOMPRESS;
+ test_data->test_algo = RTE_COMP_ALGO_DEFLATE;
test_data->window_sz = -1;
test_data->level_lst.min = RTE_COMP_LEVEL_MIN;
test_data->level_lst.max = RTE_COMP_LEVEL_MAX;
diff --git a/app/test-compress-perf/comp_perf_test_cyclecount.c b/app/test-compress-perf/comp_perf_test_cyclecount.c
index ce6c4d7605..81c3d30038 100644
--- a/app/test-compress-perf/comp_perf_test_cyclecount.c
+++ b/app/test-compress-perf/comp_perf_test_cyclecount.c
@@ -193,7 +193,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_COMPRESS,
.compress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.deflate.huffman = test_data->huffman_enc,
.level = test_data->level,
.window_size = test_data->window_sz,
@@ -208,7 +208,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_DECOMPRESS,
.decompress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.chksum = RTE_COMP_CHECKSUM_NONE,
.window_size = test_data->window_sz,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
diff --git a/app/test-compress-perf/comp_perf_test_throughput.c b/app/test-compress-perf/comp_perf_test_throughput.c
index c9f8237626..2545ee9925 100644
--- a/app/test-compress-perf/comp_perf_test_throughput.c
+++ b/app/test-compress-perf/comp_perf_test_throughput.c
@@ -84,7 +84,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_COMPRESS,
.compress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.deflate.huffman = test_data->huffman_enc,
.level = test_data->level,
.window_size = test_data->window_sz,
@@ -99,7 +99,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_DECOMPRESS,
.decompress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.chksum = RTE_COMP_CHECKSUM_NONE,
.window_size = test_data->window_sz,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
diff --git a/app/test-compress-perf/comp_perf_test_verify.c b/app/test-compress-perf/comp_perf_test_verify.c
index 7d6b6abecd..88f4f41851 100644
--- a/app/test-compress-perf/comp_perf_test_verify.c
+++ b/app/test-compress-perf/comp_perf_test_verify.c
@@ -87,7 +87,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_COMPRESS,
.compress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.deflate.huffman = test_data->huffman_enc,
.level = test_data->level,
.window_size = test_data->window_sz,
@@ -104,7 +104,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_DECOMPRESS,
.decompress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.chksum = RTE_COMP_CHECKSUM_NONE,
.window_size = test_data->window_sz,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
diff --git a/app/test-compress-perf/main.c b/app/test-compress-perf/main.c
index bbb4c7917b..d049527ba1 100644
--- a/app/test-compress-perf/main.c
+++ b/app/test-compress-perf/main.c
@@ -57,29 +57,39 @@ comp_perf_check_capabilities(struct comp_test_data *test_data, uint8_t cdev_id)
{
const struct rte_compressdev_capabilities *cap;
- cap = rte_compressdev_capability_get(cdev_id,
- RTE_COMP_ALGO_DEFLATE);
+ cap = rte_compressdev_capability_get(cdev_id, test_data->test_algo);
if (cap == NULL) {
RTE_LOG(ERR, USER1,
- "Compress device does not support DEFLATE\n");
+ "Compress device does not support %u algorithm\n",
+ test_data->test_algo);
return -1;
}
uint64_t comp_flags = cap->comp_feature_flags;
- /* Huffman encoding */
- if (test_data->huffman_enc == RTE_COMP_HUFFMAN_FIXED &&
- (comp_flags & RTE_COMP_FF_HUFFMAN_FIXED) == 0) {
- RTE_LOG(ERR, USER1,
- "Compress device does not supported Fixed Huffman\n");
- return -1;
- }
+ /* Algorithm type */
+ switch (test_data->test_algo) {
+ case RTE_COMP_ALGO_DEFLATE:
+ /* Huffman encoding */
+ if (test_data->huffman_enc == RTE_COMP_HUFFMAN_FIXED &&
+ (comp_flags & RTE_COMP_FF_HUFFMAN_FIXED) == 0) {
+ RTE_LOG(ERR, USER1,
+ "Compress device does not supported Fixed Huffman\n");
+ return -1;
+ }
- if (test_data->huffman_enc == RTE_COMP_HUFFMAN_DYNAMIC &&
- (comp_flags & RTE_COMP_FF_HUFFMAN_DYNAMIC) == 0) {
- RTE_LOG(ERR, USER1,
- "Compress device does not supported Dynamic Huffman\n");
+ if (test_data->huffman_enc == RTE_COMP_HUFFMAN_DYNAMIC &&
+ (comp_flags & RTE_COMP_FF_HUFFMAN_DYNAMIC) == 0) {
+ RTE_LOG(ERR, USER1,
+ "Compress device does not supported Dynamic Huffman\n");
+ return -1;
+ }
+ break;
+ case RTE_COMP_ALGO_LZS:
+ case RTE_COMP_ALGO_NULL:
+ break;
+ default:
return -1;
}
diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index 15fa8e812b..da345ded48 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -89,6 +89,10 @@ New Features
Enable the application options for testing only compress and only decompress.
+* **Added algo option in test-compress-perf.**
+
+ Added support for testing other algorithms except for DEFLAT.
+
Removed Items
-------------
diff --git a/doc/guides/tools/comp_perf.rst b/doc/guides/tools/comp_perf.rst
index 9d2f4dbe4a..cf9035cc23 100644
--- a/doc/guides/tools/comp_perf.rst
+++ b/doc/guides/tools/comp_perf.rst
@@ -84,6 +84,8 @@ Application Options
``--operation [comp/decomp/comp_and_decomp]``: perform test on compression, decompression or both operations
+ ``--algo [null/deflate/lzs]`` : perform test on algorithm null(DMA), Deflate or lzs (default: Deflate)
+
``--huffman-enc [fixed/dynamic/default]``: Huffman encoding (default: dynamic)
``--compress-level N``: compression level, which could be a single value, list or range (default: range between 1 and 9)
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v2 4/4] app/test-compress-perf: add LZ4 support
2023-02-02 9:13 ` [PATCH v2 0/4] compressdev: " Michael Baum
` (2 preceding siblings ...)
2023-02-02 9:13 ` [PATCH v2 3/4] app/test-compress-perf: add algo option Michael Baum
@ 2023-02-02 9:13 ` Michael Baum
2023-02-13 6:11 ` [PATCH v3 0/4] compressdev: " Michael Baum
4 siblings, 0 replies; 31+ messages in thread
From: Michael Baum @ 2023-02-02 9:13 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe, Thomas Monjalon
Add support for LZ4 algorithm and add a new parameter for it.
Signed-off-by: Michael Baum <michaelba@nvidia.com>
---
app/test-compress-perf/comp_perf_options.h | 1 +
.../comp_perf_options_parse.c | 43 ++++++++++++++++++-
.../comp_perf_test_cyclecount.c | 7 ++-
.../comp_perf_test_throughput.c | 7 ++-
.../comp_perf_test_verify.c | 7 ++-
app/test-compress-perf/main.c | 17 ++++++++
doc/guides/rel_notes/release_23_03.rst | 1 +
doc/guides/tools/comp_perf.rst | 5 ++-
8 files changed, 82 insertions(+), 6 deletions(-)
diff --git a/app/test-compress-perf/comp_perf_options.h b/app/test-compress-perf/comp_perf_options.h
index 5e5227a700..828a7309d8 100644
--- a/app/test-compress-perf/comp_perf_options.h
+++ b/app/test-compress-perf/comp_perf_options.h
@@ -61,6 +61,7 @@ struct comp_test_data {
uint16_t max_sgl_segs;
uint32_t total_segs;
+ uint8_t lz4_flags;
enum rte_comp_huffman huffman_enc;
enum comp_operation test_op;
enum rte_comp_algorithm test_algo;
diff --git a/app/test-compress-perf/comp_perf_options_parse.c b/app/test-compress-perf/comp_perf_options_parse.c
index 97ddff87ef..6d8c370fc2 100644
--- a/app/test-compress-perf/comp_perf_options_parse.c
+++ b/app/test-compress-perf/comp_perf_options_parse.c
@@ -27,6 +27,7 @@
#define CPERF_OPTYPE ("operation")
#define CPERF_ALGO ("algo")
#define CPERF_HUFFMAN_ENC ("huffman-enc")
+#define CPERF_LZ4_FLAGS ("lz4-flags")
#define CPERF_LEVEL ("compress-level")
#define CPERF_WINDOW_SIZE ("window-sz")
#define CPERF_EXTERNAL_MBUFS ("external-mbufs")
@@ -57,10 +58,11 @@ usage(char *progname)
" compressed/decompressed (default: 10000)\n"
" --operation [comp/decomp/comp_and_decomp]: perform test on\n"
" compression, decompression or both operations\n"
- " --algo [null/deflate/lzs]: perform test on algorithm\n"
- " null(DMA), deflate or lzs (default: deflate)\n"
+ " --algo [null/deflate/lzs/lz4]: perform test on algorithm\n"
+ " null(DMA), deflate, lzs or lz4 (default: deflate)\n"
" --huffman-enc [fixed/dynamic/default]: Huffman encoding\n"
" (default: dynamic)\n"
+ " --lz4-flags N: flags to configure LZ4 algorithm (default: 0)\n"
" --compress-level N: compression level, which could be a single value, list or range\n"
" (default: range between 1 and 9)\n"
" --window-sz N: base two log value of compression window size\n"
@@ -153,6 +155,23 @@ parse_uint16_t(uint16_t *value, const char *arg)
return 0;
}
+static int
+parse_uint8_t(uint8_t *value, const char *arg)
+{
+ uint32_t val = 0;
+ int ret = parse_uint32_t(&val, arg);
+
+ if (ret < 0)
+ return ret;
+
+ if (val > UINT8_MAX)
+ return -ERANGE;
+
+ *value = (uint8_t) val;
+
+ return 0;
+}
+
static int
parse_range(const char *arg, uint8_t *min, uint8_t *max, uint8_t *inc)
{
@@ -488,6 +507,10 @@ parse_algo(struct comp_test_data *test_data, const char *arg)
{
"lzs",
RTE_COMP_ALGO_LZS
+ },
+ {
+ "lz4",
+ RTE_COMP_ALGO_LZ4
}
};
@@ -533,6 +556,19 @@ parse_huffman_enc(struct comp_test_data *test_data, const char *arg)
return 0;
}
+static int
+parse_lz4_flags(struct comp_test_data *test_data, const char *arg)
+{
+ int ret = parse_uint8_t(&test_data->lz4_flags, arg);
+
+ if (ret) {
+ RTE_LOG(ERR, USER1, "Failed to parse LZ4 flags\n");
+ return -1;
+ }
+
+ return 0;
+}
+
static int
parse_level(struct comp_test_data *test_data, const char *arg)
{
@@ -607,6 +643,7 @@ static struct option lgopts[] = {
{ CPERF_OPTYPE, required_argument, 0, 0 },
{ CPERF_ALGO, required_argument, 0, 0 },
{ CPERF_HUFFMAN_ENC, required_argument, 0, 0 },
+ { CPERF_LZ4_FLAGS, required_argument, 0, 0 },
{ CPERF_LEVEL, required_argument, 0, 0 },
{ CPERF_WINDOW_SIZE, required_argument, 0, 0 },
{ CPERF_EXTERNAL_MBUFS, 0, 0, 0 },
@@ -630,6 +667,7 @@ comp_perf_opts_parse_long(int opt_idx, struct comp_test_data *test_data)
{ CPERF_OPTYPE, parse_op_type },
{ CPERF_ALGO, parse_algo },
{ CPERF_HUFFMAN_ENC, parse_huffman_enc },
+ { CPERF_LZ4_FLAGS, parse_lz4_flags },
{ CPERF_LEVEL, parse_level },
{ CPERF_WINDOW_SIZE, parse_window_sz },
{ CPERF_EXTERNAL_MBUFS, parse_external_mbufs },
@@ -682,6 +720,7 @@ comp_perf_options_default(struct comp_test_data *test_data)
test_data->pool_sz = 8192;
test_data->max_sgl_segs = 16;
test_data->num_iter = 10000;
+ test_data->lz4_flags = 0;
test_data->huffman_enc = RTE_COMP_HUFFMAN_DYNAMIC;
test_data->test_op = COMPRESS_DECOMPRESS;
test_data->test_algo = RTE_COMP_ALGO_DEFLATE;
diff --git a/app/test-compress-perf/comp_perf_test_cyclecount.c b/app/test-compress-perf/comp_perf_test_cyclecount.c
index 81c3d30038..4d336ec8d6 100644
--- a/app/test-compress-perf/comp_perf_test_cyclecount.c
+++ b/app/test-compress-perf/comp_perf_test_cyclecount.c
@@ -194,13 +194,16 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type)
.type = RTE_COMP_COMPRESS,
.compress = {
.algo = test_data->test_algo,
- .deflate.huffman = test_data->huffman_enc,
.level = test_data->level,
.window_size = test_data->window_sz,
.chksum = RTE_COMP_CHECKSUM_NONE,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_DEFLATE)
+ xform.compress.deflate.huffman = test_data->huffman_enc;
+ else if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.compress.lz4.flags = test_data->lz4_flags;
input_bufs = mem->decomp_bufs;
output_bufs = mem->comp_bufs;
out_seg_sz = test_data->out_seg_sz;
@@ -214,6 +217,8 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type)
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.decompress.lz4.flags = test_data->lz4_flags;
input_bufs = mem->comp_bufs;
output_bufs = mem->decomp_bufs;
out_seg_sz = test_data->seg_sz;
diff --git a/app/test-compress-perf/comp_perf_test_throughput.c b/app/test-compress-perf/comp_perf_test_throughput.c
index 2545ee9925..1f7072d223 100644
--- a/app/test-compress-perf/comp_perf_test_throughput.c
+++ b/app/test-compress-perf/comp_perf_test_throughput.c
@@ -85,13 +85,16 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type)
.type = RTE_COMP_COMPRESS,
.compress = {
.algo = test_data->test_algo,
- .deflate.huffman = test_data->huffman_enc,
.level = test_data->level,
.window_size = test_data->window_sz,
.chksum = RTE_COMP_CHECKSUM_NONE,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_DEFLATE)
+ xform.compress.deflate.huffman = test_data->huffman_enc;
+ else if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.compress.lz4.flags = test_data->lz4_flags;
input_bufs = mem->decomp_bufs;
output_bufs = mem->comp_bufs;
out_seg_sz = test_data->out_seg_sz;
@@ -105,6 +108,8 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type)
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.decompress.lz4.flags = test_data->lz4_flags;
input_bufs = mem->comp_bufs;
output_bufs = mem->decomp_bufs;
out_seg_sz = test_data->seg_sz;
diff --git a/app/test-compress-perf/comp_perf_test_verify.c b/app/test-compress-perf/comp_perf_test_verify.c
index 88f4f41851..7bd18073cf 100644
--- a/app/test-compress-perf/comp_perf_test_verify.c
+++ b/app/test-compress-perf/comp_perf_test_verify.c
@@ -88,13 +88,16 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
.type = RTE_COMP_COMPRESS,
.compress = {
.algo = test_data->test_algo,
- .deflate.huffman = test_data->huffman_enc,
.level = test_data->level,
.window_size = test_data->window_sz,
.chksum = RTE_COMP_CHECKSUM_NONE,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_DEFLATE)
+ xform.compress.deflate.huffman = test_data->huffman_enc;
+ else if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.compress.lz4.flags = test_data->lz4_flags;
output_data_ptr = ctx->mem.compressed_data;
output_data_sz = &ctx->comp_data_sz;
input_bufs = mem->decomp_bufs;
@@ -110,6 +113,8 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.decompress.lz4.flags = test_data->lz4_flags;
output_data_ptr = ctx->mem.decompressed_data;
output_data_sz = &ctx->decomp_data_sz;
input_bufs = mem->comp_bufs;
diff --git a/app/test-compress-perf/main.c b/app/test-compress-perf/main.c
index d049527ba1..fa366123ed 100644
--- a/app/test-compress-perf/main.c
+++ b/app/test-compress-perf/main.c
@@ -86,6 +86,23 @@ comp_perf_check_capabilities(struct comp_test_data *test_data, uint8_t cdev_id)
return -1;
}
break;
+ case RTE_COMP_ALGO_LZ4:
+ /* LZ4 flags */
+ if ((test_data->lz4_flags & RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM) &&
+ (comp_flags & RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM) == 0) {
+ RTE_LOG(ERR, USER1,
+ "Compress device does not support LZ4 block with checksum\n");
+ return -1;
+ }
+
+ if ((test_data->lz4_flags &
+ RTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE) &&
+ (comp_flags & RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE) == 0) {
+ RTE_LOG(ERR, USER1,
+ "Compress device does not support LZ4 independent blocks\n");
+ return -1;
+ }
+ break;
case RTE_COMP_ALGO_LZS:
case RTE_COMP_ALGO_NULL:
break;
diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index da345ded48..aedc5767ff 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -84,6 +84,7 @@ New Features
* Added support for ``RTE_COMP_ALGO_LZ4``.
* Added support for ``RTE_COMP_CHECKSUM_XXHASH32``.
+ * Added support for ``lz4`` in test-compress-perf algo options.
* **Allowed test single compress operation in test-compress-perf.**
diff --git a/doc/guides/tools/comp_perf.rst b/doc/guides/tools/comp_perf.rst
index cf9035cc23..5eb05a095c 100644
--- a/doc/guides/tools/comp_perf.rst
+++ b/doc/guides/tools/comp_perf.rst
@@ -84,10 +84,13 @@ Application Options
``--operation [comp/decomp/comp_and_decomp]``: perform test on compression, decompression or both operations
- ``--algo [null/deflate/lzs]`` : perform test on algorithm null(DMA), Deflate or lzs (default: Deflate)
+ ``--algo [null/deflate/lzs/lz4]`` : perform test on algorithm null(DMA), Deflate, lzs or lz4 (default: Deflate)
``--huffman-enc [fixed/dynamic/default]``: Huffman encoding (default: dynamic)
+ ``--lz4-flags N``: flags to for LZ4 parameters,
+ see `LZ4 Frame Descriptor <https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor>`_ (default: no flags)
+
``--compress-level N``: compression level, which could be a single value, list or range (default: range between 1 and 9)
``--window-sz N``: base two log value of compression window size (default: max supported by PMD)
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* RE: [EXT] [PATCH v2 1/4] compressdev: add LZ4 algorithm support
2023-02-02 9:13 ` [PATCH v2 1/4] compressdev: add LZ4 algorithm support Michael Baum
@ 2023-02-05 17:20 ` Akhil Goyal
0 siblings, 0 replies; 31+ messages in thread
From: Akhil Goyal @ 2023-02-05 17:20 UTC (permalink / raw)
To: dev, Kai Ji, Mahipal Challa, Ashish Gupta, Fiona Trahe
Cc: Matan Azrad, Thomas Monjalon, Michael Baum
> Add support for LZ4 algorithm:
> - Add Lz4 param structure to XFORM structures.
> - Add capabilities flags for LZ4 params.
> - Add xxHash-32 checksum and capabilities flag.
>
> Signed-off-by: Michael Baum <michaelba@nvidia.com>
> ---
Please review the series, or else it will be merged in RC1.
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v3 0/4] compressdev: add LZ4 support
2023-02-02 9:13 ` [PATCH v2 0/4] compressdev: " Michael Baum
` (3 preceding siblings ...)
2023-02-02 9:13 ` [PATCH v2 4/4] app/test-compress-perf: add LZ4 support Michael Baum
@ 2023-02-13 6:11 ` Michael Baum
2023-02-13 6:11 ` [PATCH v3 1/4] compressdev: add LZ4 algorithm support Michael Baum
` (5 more replies)
4 siblings, 6 replies; 31+ messages in thread
From: Michael Baum @ 2023-02-13 6:11 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe, Thomas Monjalon
Add support for LZ4 compression algorithm for both API and app.
In addition, enable app to test de/comp only. This option already
exists in both app doc and arguments but isn't implemented inside the
application.
LZ4: https://github.com/lz4/lz4
RFC:
https://patchwork.dpdk.org/project/dpdk/patch/20220410182622.8828-1-rzidane@nvidia.com/
v2:
- Add macros for all LZ4 flags in the standard.
- Rebase.
v3:
- Rebase.
Michael Baum (4):
compressdev: add LZ4 algorithm support
app/test-compress-perf: allow test single compress operation
app/test-compress-perf: add algo option
app/test-compress-perf: add LZ4 support
app/test-compress-perf/comp_perf_options.h | 9 +-
.../comp_perf_options_parse.c | 83 +++++++++++-
.../comp_perf_test_common.c | 126 +++++++++++++-----
.../comp_perf_test_cyclecount.c | 82 +++++++-----
.../comp_perf_test_throughput.c | 75 +++++++----
.../comp_perf_test_verify.c | 75 +++++++----
app/test-compress-perf/main.c | 63 +++++++--
doc/guides/compressdevs/features/default.ini | 7 +
doc/guides/rel_notes/release_23_03.rst | 16 +++
doc/guides/tools/comp_perf.rst | 5 +
lib/compressdev/rte_comp.c | 12 ++
lib/compressdev/rte_comp.h | 79 ++++++++++-
12 files changed, 488 insertions(+), 144 deletions(-)
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v3 1/4] compressdev: add LZ4 algorithm support
2023-02-13 6:11 ` [PATCH v3 0/4] compressdev: " Michael Baum
@ 2023-02-13 6:11 ` Michael Baum
2023-02-13 6:11 ` [PATCH v3 2/4] app/test-compress-perf: allow test single compress operation Michael Baum
` (4 subsequent siblings)
5 siblings, 0 replies; 31+ messages in thread
From: Michael Baum @ 2023-02-13 6:11 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe, Thomas Monjalon
Add support for LZ4 algorithm:
- Add Lz4 param structure to XFORM structures.
- Add capabilities flags for LZ4 params.
- Add xxHash-32 checksum and capabilities flag.
Signed-off-by: Michael Baum <michaelba@nvidia.com>
---
doc/guides/compressdevs/features/default.ini | 7 ++
doc/guides/rel_notes/release_23_03.rst | 7 ++
lib/compressdev/rte_comp.c | 12 +++
lib/compressdev/rte_comp.h | 79 +++++++++++++++++++-
4 files changed, 103 insertions(+), 2 deletions(-)
diff --git a/doc/guides/compressdevs/features/default.ini b/doc/guides/compressdevs/features/default.ini
index e1419ee8db..2f178c5efd 100644
--- a/doc/guides/compressdevs/features/default.ini
+++ b/doc/guides/compressdevs/features/default.ini
@@ -20,8 +20,15 @@ OOP SGL In LB Out =
OOP LB In SGL Out =
Deflate =
LZS =
+LZ4 =
Adler32 =
Crc32 =
Adler32&Crc32 =
+xxHash32 =
Fixed =
Dynamic =
+LZ4 Dictionary ID =
+LZ4 Content Checksum =
+LZ4 Content Size =
+LZ4 Block Checksum =
+LZ4 Block Independence =
diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index 7527c6d57f..790c2a5eef 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -105,6 +105,13 @@ New Features
* Added support to capture packets at each graph node with packet metadata and
node name.
+* **Added LZ4 algorithm in Compressdev Library.**
+
+ Added new compression algorithm, including:
+
+ * Added support for ``RTE_COMP_ALGO_LZ4``.
+ * Added support for ``RTE_COMP_CHECKSUM_XXHASH32``.
+
Removed Items
-------------
diff --git a/lib/compressdev/rte_comp.c b/lib/compressdev/rte_comp.c
index 320c6dab92..f060c68557 100644
--- a/lib/compressdev/rte_comp.c
+++ b/lib/compressdev/rte_comp.c
@@ -39,6 +39,18 @@ rte_comp_get_feature_name(uint64_t flag)
return "HUFFMAN_FIXED";
case RTE_COMP_FF_HUFFMAN_DYNAMIC:
return "HUFFMAN_DYNAMIC";
+ case RTE_COMP_FF_XXHASH32_CHECKSUM:
+ return "XXHASH32_CHECKSUM";
+ case RTE_COMP_FF_LZ4_DICT_ID:
+ return "LZ4_DICT_ID";
+ case RTE_COMP_FF_LZ4_CONTENT_WITH_CHECKSUM:
+ return "LZ4_CONTENT_WITH_CHECKSUM";
+ case RTE_COMP_FF_LZ4_CONTENT_SIZE:
+ return "LZ4_CONTENT_SIZE";
+ case RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE:
+ return "LZ4_BLOCK_INDEPENDENCE";
+ case RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM:
+ return "LZ4_BLOCK_WITH_CHECKSUM";
default:
return NULL;
}
diff --git a/lib/compressdev/rte_comp.h b/lib/compressdev/rte_comp.h
index 5bd711fda1..2096fb2407 100644
--- a/lib/compressdev/rte_comp.h
+++ b/lib/compressdev/rte_comp.h
@@ -67,6 +67,18 @@ extern "C" {
/**< Fixed huffman encoding is supported */
#define RTE_COMP_FF_HUFFMAN_DYNAMIC (1ULL << 14)
/**< Dynamic huffman encoding is supported */
+#define RTE_COMP_FF_XXHASH32_CHECKSUM (1ULL << 15)
+/**< xxHash-32 Checksum is supported */
+#define RTE_COMP_FF_LZ4_DICT_ID (1ULL << 16)
+/**< LZ4 dictionary ID is supported */
+#define RTE_COMP_FF_LZ4_CONTENT_WITH_CHECKSUM (1ULL << 17)
+/**< LZ4 content with checksum is supported */
+#define RTE_COMP_FF_LZ4_CONTENT_SIZE (1ULL << 18)
+/**< LZ4 content size is supported */
+#define RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE (1ULL << 19)
+/**< LZ4 block independent is supported */
+#define RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM (1ULL << 20)
+/**< LZ4 block with checksum is supported */
/** Status of comp operation */
enum rte_comp_op_status {
@@ -109,6 +121,10 @@ enum rte_comp_algorithm {
/**< LZS compression algorithm
* https://tools.ietf.org/html/rfc2395
*/
+ RTE_COMP_ALGO_LZ4,
+ /**< LZ4 compression algorithm
+ * https://github.com/lz4/lz4
+ */
};
/** Compression Hash Algorithms */
@@ -147,9 +163,12 @@ enum rte_comp_checksum_type {
/**< Generates both Adler-32 and CRC32 checksums, concatenated.
* CRC32 is in the lower 32bits, Adler-32 in the upper 32 bits.
*/
+ RTE_COMP_CHECKSUM_XXHASH32,
+ /**< Generates a xxHash-32 checksum, as used by lz4.
+ * https://github.com/Cyan4973/xxHash/blob/dev/doc/xxhash_spec.md
+ */
};
-
/** Compression Huffman Type - used by DEFLATE algorithm */
enum rte_comp_huffman {
RTE_COMP_HUFFMAN_DEFAULT,
@@ -206,13 +225,63 @@ enum rte_comp_op_type {
*/
};
-
/** Parameters specific to the deflate algorithm */
struct rte_comp_deflate_params {
enum rte_comp_huffman huffman;
/**< Compression huffman encoding type */
};
+/**
+ * Dictionary ID flag
+ * If this flag is set, a 4-bytes Dict-ID field will be present, after the
+ * descriptor flags and the Content Size.
+ */
+#define RTE_COMP_LZ4_FLAG_DICT_ID (1 << 0)
+
+/**
+ * Content Checksum flag
+ * If this flag is set, a 32-bits content checksum will be appended after the
+ * EndMark.
+ */
+#define RTE_COMP_LZ4_FLAG_CONTENT_CHECKSUM (1 << 2)
+
+/**
+ * Content Size flag
+ * If this flag is set, the uncompressed size of data included within the frame
+ * will be present as an 8 bytes unsigned little-endian value, after the flags.
+ * Content Size usage is optional.
+ */
+#define RTE_COMP_LZ4_FLAG_CONTENT_SIZE (1 << 3)
+
+/**
+ * Block Checksum flag.
+ * If this flag is set, each data block will be followed by a 4-bytes checksum,
+ * calculated by using the xxHash-32 algorithm on the raw (compressed) data
+ * block. The intention is to detect data corruption (storage or transmission
+ * errors) immediately, before decoding. Block checksum usage is optional.
+ */
+#define RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM (1 << 4)
+
+/**
+ * Block Independence flag.
+ * If this flag is set to 1, blocks are independent.
+ * If this flag is set to 0, each block depends on previous ones (up to LZ4
+ * window size, which is 64 KB). In such case, it is necessary to decode all
+ * blocks in sequence.
+ * Block dependency improves compression ratio, especially for small blocks. On
+ * the other hand, it makes random access or multi-threaded decoding impossible.
+ */
+#define RTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE (1 << 5)
+
+/** Parameters specific to the LZ4 algorithm */
+struct rte_comp_lz4_params {
+ uint8_t flags;
+ /**< Compression LZ4 parameter flags.
+ * Based on LZ4 standard flags:
+ * https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor
+ */
+};
+
/** Setup Data for compression */
struct rte_comp_compress_xform {
enum rte_comp_algorithm algo;
@@ -220,6 +289,8 @@ struct rte_comp_compress_xform {
union {
struct rte_comp_deflate_params deflate;
/**< Parameters specific to the deflate algorithm */
+ struct rte_comp_lz4_params lz4;
+ /**< Parameters specific to the LZ4 algorithm */
}; /**< Algorithm specific parameters */
int level;
/**< Compression level */
@@ -249,6 +320,10 @@ struct rte_comp_decompress_xform {
* compressed data. If window size can't be supported by the PMD then
* setup of stream or private_xform should fail.
*/
+ union {
+ struct rte_comp_lz4_params lz4;
+ /**< Parameters specific to the LZ4 algorithm */
+ }; /**< Algorithm specific parameters */
enum rte_comp_hash_algorithm hash_algo;
/**< Hash algorithm to be used with decompress operation. Hash is always
* done on plaintext.
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v3 2/4] app/test-compress-perf: allow test single compress operation
2023-02-13 6:11 ` [PATCH v3 0/4] compressdev: " Michael Baum
2023-02-13 6:11 ` [PATCH v3 1/4] compressdev: add LZ4 algorithm support Michael Baum
@ 2023-02-13 6:11 ` Michael Baum
2023-02-13 7:23 ` [EXT] " Akhil Goyal
2023-02-13 6:11 ` [PATCH v3 3/4] app/test-compress-perf: add algo option Michael Baum
` (3 subsequent siblings)
5 siblings, 1 reply; 31+ messages in thread
From: Michael Baum @ 2023-02-13 6:11 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe, Thomas Monjalon
Part of the application options is to test only compress and only
decompress but actually the application ignores this user option and
tries to test always both compress and decompress.
Allow testing only compress and only decompress.
Signed-off-by: Matan Azrad <matan@nvidia.com>
Signed-off-by: Michael Baum <michaelba@nvidia.com>
---
app/test-compress-perf/comp_perf_options.h | 6 +-
.../comp_perf_options_parse.c | 4 +-
.../comp_perf_test_common.c | 126 +++++++++++++-----
.../comp_perf_test_cyclecount.c | 71 ++++++----
.../comp_perf_test_throughput.c | 64 +++++----
.../comp_perf_test_verify.c | 64 +++++----
app/test-compress-perf/main.c | 8 ++
doc/guides/rel_notes/release_23_03.rst | 4 +
8 files changed, 230 insertions(+), 117 deletions(-)
diff --git a/app/test-compress-perf/comp_perf_options.h b/app/test-compress-perf/comp_perf_options.h
index 57dd146330..d00b299247 100644
--- a/app/test-compress-perf/comp_perf_options.h
+++ b/app/test-compress-perf/comp_perf_options.h
@@ -32,9 +32,9 @@ enum cperf_test_type {
};
enum comp_operation {
- COMPRESS_ONLY,
- DECOMPRESS_ONLY,
- COMPRESS_DECOMPRESS
+ COMPRESS = (1 << 0),
+ DECOMPRESS = (1 << 1),
+ COMPRESS_DECOMPRESS = (COMPRESS | DECOMPRESS),
};
struct range_list {
diff --git a/app/test-compress-perf/comp_perf_options_parse.c b/app/test-compress-perf/comp_perf_options_parse.c
index 019eddb7bd..7a992bf43e 100644
--- a/app/test-compress-perf/comp_perf_options_parse.c
+++ b/app/test-compress-perf/comp_perf_options_parse.c
@@ -446,11 +446,11 @@ parse_op_type(struct comp_test_data *test_data, const char *arg)
struct name_id_map optype_namemap[] = {
{
"comp",
- COMPRESS_ONLY
+ COMPRESS
},
{
"decomp",
- DECOMPRESS_ONLY
+ DECOMPRESS
},
{
"comp_and_decomp",
diff --git a/app/test-compress-perf/comp_perf_test_common.c b/app/test-compress-perf/comp_perf_test_common.c
index b402a0d839..cd60958944 100644
--- a/app/test-compress-perf/comp_perf_test_common.c
+++ b/app/test-compress-perf/comp_perf_test_common.c
@@ -227,23 +227,43 @@ comp_perf_allocate_memory(struct comp_test_data *test_data,
{
uint16_t comp_mbuf_size;
uint16_t decomp_mbuf_size;
+ size_t comp_data_size;
+ size_t decomp_data_size;
+ size_t output_data_sz;
test_data->out_seg_sz = find_buf_size(test_data->seg_sz);
- /* Number of segments for input and output
- * (compression and decompression)
- */
- test_data->total_segs = DIV_CEIL(test_data->input_data_sz,
- test_data->seg_sz);
+ if (test_data->test_op & COMPRESS) {
+ /*
+ * Number of segments for input and output
+ * (compression and decompression)
+ */
+ test_data->total_segs = DIV_CEIL(test_data->input_data_sz,
+ test_data->seg_sz);
+ } else {
+ /*
+ * When application does decompression only, input data is
+ * compressed and smaller than the output. The expected size of
+ * uncompressed data given by the user in segment size argument.
+ */
+ test_data->total_segs = test_data->max_sgl_segs;
+ }
+
+ output_data_sz = (size_t) test_data->out_seg_sz * test_data->total_segs;
+ output_data_sz =
+ RTE_MAX(output_data_sz, (size_t) MIN_COMPRESSED_BUF_SIZE);
if (test_data->use_external_mbufs != 0) {
if (comp_perf_allocate_external_mbufs(test_data, mem) < 0)
return -1;
comp_mbuf_size = 0;
decomp_mbuf_size = 0;
- } else {
+ } else if (test_data->test_op & COMPRESS) {
comp_mbuf_size = test_data->out_seg_sz + RTE_PKTMBUF_HEADROOM;
decomp_mbuf_size = test_data->seg_sz + RTE_PKTMBUF_HEADROOM;
+ } else {
+ comp_mbuf_size = test_data->seg_sz + RTE_PKTMBUF_HEADROOM;
+ decomp_mbuf_size = test_data->out_seg_sz + RTE_PKTMBUF_HEADROOM;
}
char pool_name[32] = "";
@@ -287,26 +307,28 @@ comp_perf_allocate_memory(struct comp_test_data *test_data,
return -1;
}
- /*
- * Compressed data might be a bit larger than input data,
- * if data cannot be compressed
- */
- mem->compressed_data = rte_zmalloc_socket(NULL,
- RTE_MAX(
- (size_t) test_data->out_seg_sz *
- test_data->total_segs,
- (size_t) MIN_COMPRESSED_BUF_SIZE),
- 0,
- rte_socket_id());
+ if (test_data->test_op & COMPRESS) {
+ /*
+ * Compressed data might be a bit larger than input data,
+ * if data cannot be compressed
+ */
+ comp_data_size = output_data_sz;
+ decomp_data_size = test_data->input_data_sz;
+ } else {
+ comp_data_size = test_data->input_data_sz;
+ decomp_data_size = output_data_sz;
+ }
+
+ mem->compressed_data = rte_zmalloc_socket(NULL, comp_data_size, 0,
+ rte_socket_id());
if (mem->compressed_data == NULL) {
RTE_LOG(ERR, USER1, "Memory to hold the data from the input "
"file could not be allocated\n");
return -1;
}
- mem->decompressed_data = rte_zmalloc_socket(NULL,
- test_data->input_data_sz, 0,
- rte_socket_id());
+ mem->decompressed_data = rte_zmalloc_socket(NULL, decomp_data_size, 0,
+ rte_socket_id());
if (mem->decompressed_data == NULL) {
RTE_LOG(ERR, USER1, "Memory to hold the data from the input "
"file could not be allocated\n");
@@ -351,6 +373,7 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
uint16_t segs_per_mbuf = 0;
uint32_t cmz = 0;
uint32_t dmz = 0;
+ bool decompress_only = !!(test_data->test_op == DECOMPRESS);
for (i = 0; i < mem->total_bufs; i++) {
/* Allocate data in input mbuf and copy data from input file */
@@ -361,8 +384,6 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
return -1;
}
- data_sz = RTE_MIN(remaining_data, test_data->seg_sz);
-
if (test_data->use_external_mbufs != 0) {
rte_pktmbuf_attach_extbuf(mem->decomp_bufs[i],
mem->decomp_memzones[dmz]->addr,
@@ -372,16 +393,23 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
dmz++;
}
+ if (!decompress_only)
+ data_sz = RTE_MIN(remaining_data, test_data->seg_sz);
+ else
+ data_sz = test_data->out_seg_sz;
+
data_addr = (uint8_t *) rte_pktmbuf_append(
mem->decomp_bufs[i], data_sz);
if (data_addr == NULL) {
RTE_LOG(ERR, USER1, "Could not append data\n");
return -1;
}
- rte_memcpy(data_addr, input_data_ptr, data_sz);
- input_data_ptr += data_sz;
- remaining_data -= data_sz;
+ if (!decompress_only) {
+ rte_memcpy(data_addr, input_data_ptr, data_sz);
+ input_data_ptr += data_sz;
+ remaining_data -= data_sz;
+ }
/* Already one segment in the mbuf */
segs_per_mbuf = 1;
@@ -398,8 +426,6 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
return -1;
}
- data_sz = RTE_MIN(remaining_data, test_data->seg_sz);
-
if (test_data->use_external_mbufs != 0) {
rte_pktmbuf_attach_extbuf(
next_seg,
@@ -410,6 +436,12 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
dmz++;
}
+ if (!decompress_only)
+ data_sz = RTE_MIN(remaining_data,
+ test_data->seg_sz);
+ else
+ data_sz = test_data->out_seg_sz;
+
data_addr = (uint8_t *)rte_pktmbuf_append(next_seg,
data_sz);
@@ -418,9 +450,11 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
return -1;
}
- rte_memcpy(data_addr, input_data_ptr, data_sz);
- input_data_ptr += data_sz;
- remaining_data -= data_sz;
+ if (!decompress_only) {
+ rte_memcpy(data_addr, input_data_ptr, data_sz);
+ input_data_ptr += data_sz;
+ remaining_data -= data_sz;
+ }
if (rte_pktmbuf_chain(mem->decomp_bufs[i],
next_seg) < 0) {
@@ -447,16 +481,26 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
cmz++;
}
- data_addr = (uint8_t *) rte_pktmbuf_append(
- mem->comp_bufs[i],
- test_data->out_seg_sz);
+ if (decompress_only)
+ data_sz = RTE_MIN(remaining_data, test_data->seg_sz);
+ else
+ data_sz = test_data->out_seg_sz;
+
+ data_addr = (uint8_t *) rte_pktmbuf_append(mem->comp_bufs[i],
+ data_sz);
if (data_addr == NULL) {
RTE_LOG(ERR, USER1, "Could not append data\n");
return -1;
}
+ if (decompress_only) {
+ rte_memcpy(data_addr, input_data_ptr, data_sz);
+ input_data_ptr += data_sz;
+ remaining_data -= data_sz;
+ }
+
/* Chain mbufs if needed for output mbufs */
- for (j = 1; j < segs_per_mbuf; j++) {
+ for (j = 1; j < segs_per_mbuf && remaining_data > 0; j++) {
struct rte_mbuf *next_seg =
rte_pktmbuf_alloc(mem->comp_buf_pool);
@@ -476,13 +520,25 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
cmz++;
}
+ if (decompress_only)
+ data_sz = RTE_MIN(remaining_data,
+ test_data->seg_sz);
+ else
+ data_sz = test_data->out_seg_sz;
+
data_addr = (uint8_t *)rte_pktmbuf_append(next_seg,
- test_data->out_seg_sz);
+ data_sz);
if (data_addr == NULL) {
RTE_LOG(ERR, USER1, "Could not append data\n");
return -1;
}
+ if (decompress_only) {
+ rte_memcpy(data_addr, input_data_ptr, data_sz);
+ input_data_ptr += data_sz;
+ remaining_data -= data_sz;
+ }
+
if (rte_pktmbuf_chain(mem->comp_bufs[i],
next_seg) < 0) {
RTE_LOG(ERR, USER1, "Could not chain mbufs\n");
diff --git a/app/test-compress-perf/comp_perf_test_cyclecount.c b/app/test-compress-perf/comp_perf_test_cyclecount.c
index 7473cb6277..ce6c4d7605 100644
--- a/app/test-compress-perf/comp_perf_test_cyclecount.c
+++ b/app/test-compress-perf/comp_perf_test_cyclecount.c
@@ -514,38 +514,55 @@ cperf_cyclecount_test_runner(void *test_ctx)
if (cperf_verify_test_runner(&ctx->ver))
return EXIT_FAILURE;
- /*
- * Run the tests twice, discarding the first performance
- * results, before the cache is warmed up
- */
-
- /* C O M P R E S S */
- for (i = 0; i < 2; i++) {
- if (main_loop(ctx, RTE_COMP_COMPRESS) < 0)
- return EXIT_FAILURE;
- }
+ if (test_data->test_op & COMPRESS) {
+ /*
+ * Run the test twice, discarding the first performance
+ * results, before the cache is warmed up
+ */
+ for (i = 0; i < 2; i++) {
+ if (main_loop(ctx, RTE_COMP_COMPRESS) < 0)
+ return EXIT_FAILURE;
+ }
- ops_enq_retries_comp = ctx->ops_enq_retries;
- ops_deq_retries_comp = ctx->ops_deq_retries;
+ ops_enq_retries_comp = ctx->ops_enq_retries;
+ ops_deq_retries_comp = ctx->ops_deq_retries;
- duration_enq_per_op_comp = ctx->duration_enq /
- (ctx->ver.mem.total_bufs * test_data->num_iter);
- duration_deq_per_op_comp = ctx->duration_deq /
- (ctx->ver.mem.total_bufs * test_data->num_iter);
+ duration_enq_per_op_comp = ctx->duration_enq /
+ (ctx->ver.mem.total_bufs * test_data->num_iter);
+ duration_deq_per_op_comp = ctx->duration_deq /
+ (ctx->ver.mem.total_bufs * test_data->num_iter);
+ } else {
+ ops_enq_retries_comp = 0;
+ ops_deq_retries_comp = 0;
- /* D E C O M P R E S S */
- for (i = 0; i < 2; i++) {
- if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0)
- return EXIT_FAILURE;
+ duration_enq_per_op_comp = 0;
+ duration_deq_per_op_comp = 0;
}
- ops_enq_retries_decomp = ctx->ops_enq_retries;
- ops_deq_retries_decomp = ctx->ops_deq_retries;
+ if (test_data->test_op & DECOMPRESS) {
+ /*
+ * Run the test twice, discarding the first performance
+ * results, before the cache is warmed up
+ */
+ for (i = 0; i < 2; i++) {
+ if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0)
+ return EXIT_FAILURE;
+ }
- duration_enq_per_op_decomp = ctx->duration_enq /
- (ctx->ver.mem.total_bufs * test_data->num_iter);
- duration_deq_per_op_decomp = ctx->duration_deq /
- (ctx->ver.mem.total_bufs * test_data->num_iter);
+ ops_enq_retries_decomp = ctx->ops_enq_retries;
+ ops_deq_retries_decomp = ctx->ops_deq_retries;
+
+ duration_enq_per_op_decomp = ctx->duration_enq /
+ (ctx->ver.mem.total_bufs * test_data->num_iter);
+ duration_deq_per_op_decomp = ctx->duration_deq /
+ (ctx->ver.mem.total_bufs * test_data->num_iter);
+ } else {
+ ops_enq_retries_decomp = 0;
+ ops_deq_retries_decomp = 0;
+
+ duration_enq_per_op_decomp = 0;
+ duration_deq_per_op_decomp = 0;
+ }
duration_setup_per_op = ctx->duration_op /
(ctx->ver.mem.total_bufs * test_data->num_iter);
@@ -563,7 +580,7 @@ cperf_cyclecount_test_runner(void *test_ctx)
" [D-e] - decompression enqueue\n"
" [D-d] - decompression dequeue\n"
" - Cycles section: number of cycles per 'op' for the following operations:\n"
- " setup/op - memory allocation, op configuration and memory dealocation\n"
+ " setup/op - memory allocation, op configuration and memory deallocation\n"
" [C-e] - compression enqueue\n"
" [C-d] - compression dequeue\n"
" [D-e] - decompression enqueue\n"
diff --git a/app/test-compress-perf/comp_perf_test_throughput.c b/app/test-compress-perf/comp_perf_test_throughput.c
index 79cd2b2bf2..c9f8237626 100644
--- a/app/test-compress-perf/comp_perf_test_throughput.c
+++ b/app/test-compress-perf/comp_perf_test_throughput.c
@@ -359,41 +359,53 @@ cperf_throughput_test_runner(void *test_ctx)
* First the verification part is needed
*/
if (cperf_verify_test_runner(&ctx->ver)) {
- ret = EXIT_FAILURE;
+ ret = EXIT_FAILURE;
goto end;
}
- /*
- * Run the tests twice, discarding the first performance
- * results, before the cache is warmed up
- */
- for (i = 0; i < 2; i++) {
- if (main_loop(ctx, RTE_COMP_COMPRESS) < 0) {
- ret = EXIT_FAILURE;
- goto end;
+ if (test_data->test_op & COMPRESS) {
+ /*
+ * Run the test twice, discarding the first performance
+ * results, before the cache is warmed up
+ */
+ for (i = 0; i < 2; i++) {
+ if (main_loop(ctx, RTE_COMP_COMPRESS) < 0) {
+ ret = EXIT_FAILURE;
+ goto end;
+ }
}
- }
- for (i = 0; i < 2; i++) {
- if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0) {
- ret = EXIT_FAILURE;
- goto end;
- }
+ ctx->comp_tsc_byte =
+ (double)(ctx->comp_tsc_duration[test_data->level]) /
+ test_data->input_data_sz;
+ ctx->comp_gbps = rte_get_tsc_hz() / ctx->comp_tsc_byte * 8 /
+ 1000000000;
+ } else {
+ ctx->comp_tsc_byte = 0;
+ ctx->comp_gbps = 0;
}
- ctx->comp_tsc_byte =
- (double)(ctx->comp_tsc_duration[test_data->level]) /
- test_data->input_data_sz;
+ if (test_data->test_op & DECOMPRESS) {
+ /*
+ * Run the test twice, discarding the first performance
+ * results, before the cache is warmed up
+ */
+ for (i = 0; i < 2; i++) {
+ if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0) {
+ ret = EXIT_FAILURE;
+ goto end;
+ }
+ }
- ctx->decomp_tsc_byte =
+ ctx->decomp_tsc_byte =
(double)(ctx->decomp_tsc_duration[test_data->level]) /
- test_data->input_data_sz;
-
- ctx->comp_gbps = rte_get_tsc_hz() / ctx->comp_tsc_byte * 8 /
- 1000000000;
-
- ctx->decomp_gbps = rte_get_tsc_hz() / ctx->decomp_tsc_byte * 8 /
- 1000000000;
+ test_data->input_data_sz;
+ ctx->decomp_gbps = rte_get_tsc_hz() / ctx->decomp_tsc_byte * 8 /
+ 1000000000;
+ } else {
+ ctx->decomp_tsc_byte = 0;
+ ctx->decomp_gbps = 0;
+ }
exp = 0;
if (__atomic_compare_exchange_n(&display_once, &exp, 1, 0,
diff --git a/app/test-compress-perf/comp_perf_test_verify.c b/app/test-compress-perf/comp_perf_test_verify.c
index 8964442891..7d6b6abecd 100644
--- a/app/test-compress-perf/comp_perf_test_verify.c
+++ b/app/test-compress-perf/comp_perf_test_verify.c
@@ -114,7 +114,8 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
output_data_sz = &ctx->decomp_data_sz;
input_bufs = mem->comp_bufs;
output_bufs = mem->decomp_bufs;
- out_seg_sz = test_data->seg_sz;
+ out_seg_sz = (test_data->test_op & COMPRESS) ?
+ test_data->seg_sz : test_data->out_seg_sz;
}
/* Create private xform */
@@ -226,7 +227,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
op->status ==
RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE) {
RTE_LOG(ERR, USER1,
-"Out of space error occurred due to uncompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n");
+"Out of space error occurred due to incompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n");
res = -1;
goto end;
} else if (op->status !=
@@ -311,7 +312,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
op->status ==
RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE) {
RTE_LOG(ERR, USER1,
-"Out of space error occurred due to uncompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n");
+"Out of space error occurred due to incompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n");
res = -1;
goto end;
} else if (op->status !=
@@ -392,44 +393,59 @@ cperf_verify_test_runner(void *test_ctx)
int ret = EXIT_SUCCESS;
static uint16_t display_once;
uint32_t lcore = rte_lcore_id();
+ uint16_t exp = 0;
ctx->mem.lcore_id = lcore;
test_data->ratio = 0;
- if (main_loop(ctx, RTE_COMP_COMPRESS) < 0) {
- ret = EXIT_FAILURE;
- goto end;
+ if (test_data->test_op & COMPRESS) {
+ if (main_loop(ctx, RTE_COMP_COMPRESS) < 0) {
+ ret = EXIT_FAILURE;
+ goto end;
+ }
}
- if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0) {
- ret = EXIT_FAILURE;
- goto end;
- }
+ if (test_data->test_op & DECOMPRESS) {
+ if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0) {
+ ret = EXIT_FAILURE;
+ goto end;
+ }
- if (ctx->decomp_data_sz != test_data->input_data_sz) {
- RTE_LOG(ERR, USER1,
- "Decompressed data length not equal to input data length\n");
- RTE_LOG(ERR, USER1,
- "Decompressed size = %zu, expected = %zu\n",
- ctx->decomp_data_sz, test_data->input_data_sz);
- ret = EXIT_FAILURE;
- goto end;
- } else {
- if (memcmp(ctx->mem.decompressed_data,
- test_data->input_data,
- test_data->input_data_sz) != 0) {
+ if (!(test_data->test_op & COMPRESS)) {
+ /*
+ * For DECOMPRESS_ONLY mode there is no more
+ * verifications, reset the 'ratio' and 'comp_data_sz'
+ * fields for other tests report.
+ */
+ ctx->comp_data_sz = 0;
+ ctx->ratio = 0;
+ goto end;
+ }
+
+ if (ctx->decomp_data_sz != test_data->input_data_sz) {
+ RTE_LOG(ERR, USER1,
+ "Decompressed data length not equal to input data length\n");
RTE_LOG(ERR, USER1,
- "Decompressed data is not the same as file data\n");
+ "Decompressed size = %zu, expected = %zu\n",
+ ctx->decomp_data_sz, test_data->input_data_sz);
ret = EXIT_FAILURE;
goto end;
+ } else {
+ if (memcmp(ctx->mem.decompressed_data,
+ test_data->input_data,
+ test_data->input_data_sz) != 0) {
+ RTE_LOG(ERR, USER1,
+ "Decompressed data is not the same as file data\n");
+ ret = EXIT_FAILURE;
+ goto end;
+ }
}
}
ctx->ratio = (double) ctx->comp_data_sz /
test_data->input_data_sz * 100;
- uint16_t exp = 0;
if (!ctx->silent) {
if (__atomic_compare_exchange_n(&display_once, &exp, 1, 0,
__ATOMIC_RELAXED, __ATOMIC_RELAXED)) {
diff --git a/app/test-compress-perf/main.c b/app/test-compress-perf/main.c
index 41b8edc2bd..bbb4c7917b 100644
--- a/app/test-compress-perf/main.c
+++ b/app/test-compress-perf/main.c
@@ -254,6 +254,14 @@ comp_perf_dump_input_data(struct comp_test_data *test_data)
goto end;
}
+ if (!(test_data->test_op & COMPRESS) &&
+ test_data->input_data_sz >
+ (size_t) test_data->seg_sz * (size_t) test_data->max_sgl_segs) {
+ RTE_LOG(ERR, USER1,
+ "Size of input must be less than total segments\n");
+ goto end;
+ }
+
test_data->input_data = rte_zmalloc_socket(NULL,
test_data->input_data_sz, 0, rte_socket_id());
diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index 790c2a5eef..b2a8c921b9 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -112,6 +112,10 @@ New Features
* Added support for ``RTE_COMP_ALGO_LZ4``.
* Added support for ``RTE_COMP_CHECKSUM_XXHASH32``.
+* **Allowed test single compress operation in test-compress-perf.**
+
+ Enable the application options for testing only compress and only decompress.
+
Removed Items
-------------
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v3 3/4] app/test-compress-perf: add algo option
2023-02-13 6:11 ` [PATCH v3 0/4] compressdev: " Michael Baum
2023-02-13 6:11 ` [PATCH v3 1/4] compressdev: add LZ4 algorithm support Michael Baum
2023-02-13 6:11 ` [PATCH v3 2/4] app/test-compress-perf: allow test single compress operation Michael Baum
@ 2023-02-13 6:11 ` Michael Baum
2023-02-13 7:29 ` [EXT] " Akhil Goyal
2023-02-13 6:11 ` [PATCH v3 4/4] app/test-compress-perf: add LZ4 support Michael Baum
` (2 subsequent siblings)
5 siblings, 1 reply; 31+ messages in thread
From: Michael Baum @ 2023-02-13 6:11 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe, Thomas Monjalon
Add a command line option `algo` to select the compress algorithm
supported by the compress API: null (DMA), deflate and lzs.
Default for deflate.
Signed-off-by: Matan Azrad <matan@nvidia.com>
Signed-off-by: Michael Baum <michaelba@nvidia.com>
---
app/test-compress-perf/comp_perf_options.h | 2 +
.../comp_perf_options_parse.c | 40 ++++++++++++++++++-
.../comp_perf_test_cyclecount.c | 4 +-
.../comp_perf_test_throughput.c | 4 +-
.../comp_perf_test_verify.c | 4 +-
app/test-compress-perf/main.c | 38 +++++++++++-------
doc/guides/rel_notes/release_23_03.rst | 4 ++
doc/guides/tools/comp_perf.rst | 2 +
8 files changed, 76 insertions(+), 22 deletions(-)
diff --git a/app/test-compress-perf/comp_perf_options.h b/app/test-compress-perf/comp_perf_options.h
index d00b299247..5e5227a700 100644
--- a/app/test-compress-perf/comp_perf_options.h
+++ b/app/test-compress-perf/comp_perf_options.h
@@ -63,6 +63,8 @@ struct comp_test_data {
enum rte_comp_huffman huffman_enc;
enum comp_operation test_op;
+ enum rte_comp_algorithm test_algo;
+
int window_sz;
struct range_list level_lst;
uint8_t level;
diff --git a/app/test-compress-perf/comp_perf_options_parse.c b/app/test-compress-perf/comp_perf_options_parse.c
index 7a992bf43e..97ddff87ef 100644
--- a/app/test-compress-perf/comp_perf_options_parse.c
+++ b/app/test-compress-perf/comp_perf_options_parse.c
@@ -25,6 +25,7 @@
#define CPERF_MAX_SGL_SEGS ("max-num-sgl-segs")
#define CPERF_NUM_ITER ("num-iter")
#define CPERF_OPTYPE ("operation")
+#define CPERF_ALGO ("algo")
#define CPERF_HUFFMAN_ENC ("huffman-enc")
#define CPERF_LEVEL ("compress-level")
#define CPERF_WINDOW_SIZE ("window-sz")
@@ -56,6 +57,8 @@ usage(char *progname)
" compressed/decompressed (default: 10000)\n"
" --operation [comp/decomp/comp_and_decomp]: perform test on\n"
" compression, decompression or both operations\n"
+ " --algo [null/deflate/lzs]: perform test on algorithm\n"
+ " null(DMA), deflate or lzs (default: deflate)\n"
" --huffman-enc [fixed/dynamic/default]: Huffman encoding\n"
" (default: dynamic)\n"
" --compress-level N: compression level, which could be a single value, list or range\n"
@@ -470,6 +473,36 @@ parse_op_type(struct comp_test_data *test_data, const char *arg)
return 0;
}
+static int
+parse_algo(struct comp_test_data *test_data, const char *arg)
+{
+ struct name_id_map algo_namemap[] = {
+ {
+ "null",
+ RTE_COMP_ALGO_NULL
+ },
+ {
+ "deflate",
+ RTE_COMP_ALGO_DEFLATE
+ },
+ {
+ "lzs",
+ RTE_COMP_ALGO_LZS
+ }
+ };
+
+ int id = get_str_key_id_mapping(algo_namemap,
+ RTE_DIM(algo_namemap), arg);
+ if (id < 0) {
+ RTE_LOG(ERR, USER1, "Invalid algorithm specified\n");
+ return -1;
+ }
+
+ test_data->test_algo = (enum rte_comp_algorithm)id;
+
+ return 0;
+}
+
static int
parse_huffman_enc(struct comp_test_data *test_data, const char *arg)
{
@@ -491,7 +524,7 @@ parse_huffman_enc(struct comp_test_data *test_data, const char *arg)
int id = get_str_key_id_mapping(huffman_namemap,
RTE_DIM(huffman_namemap), arg);
if (id < 0) {
- RTE_LOG(ERR, USER1, "Invalid Huffmane encoding specified\n");
+ RTE_LOG(ERR, USER1, "Invalid Huffman encoding specified\n");
return -1;
}
@@ -507,7 +540,7 @@ parse_level(struct comp_test_data *test_data, const char *arg)
/*
* Try parsing the argument as a range, if it fails,
- * arse it as a list
+ * parse it as a list
*/
if (parse_range(arg, &test_data->level_lst.min,
&test_data->level_lst.max,
@@ -572,6 +605,7 @@ static struct option lgopts[] = {
{ CPERF_MAX_SGL_SEGS, required_argument, 0, 0},
{ CPERF_NUM_ITER, required_argument, 0, 0 },
{ CPERF_OPTYPE, required_argument, 0, 0 },
+ { CPERF_ALGO, required_argument, 0, 0 },
{ CPERF_HUFFMAN_ENC, required_argument, 0, 0 },
{ CPERF_LEVEL, required_argument, 0, 0 },
{ CPERF_WINDOW_SIZE, required_argument, 0, 0 },
@@ -594,6 +628,7 @@ comp_perf_opts_parse_long(int opt_idx, struct comp_test_data *test_data)
{ CPERF_MAX_SGL_SEGS, parse_max_num_sgl_segs },
{ CPERF_NUM_ITER, parse_num_iter },
{ CPERF_OPTYPE, parse_op_type },
+ { CPERF_ALGO, parse_algo },
{ CPERF_HUFFMAN_ENC, parse_huffman_enc },
{ CPERF_LEVEL, parse_level },
{ CPERF_WINDOW_SIZE, parse_window_sz },
@@ -649,6 +684,7 @@ comp_perf_options_default(struct comp_test_data *test_data)
test_data->num_iter = 10000;
test_data->huffman_enc = RTE_COMP_HUFFMAN_DYNAMIC;
test_data->test_op = COMPRESS_DECOMPRESS;
+ test_data->test_algo = RTE_COMP_ALGO_DEFLATE;
test_data->window_sz = -1;
test_data->level_lst.min = RTE_COMP_LEVEL_MIN;
test_data->level_lst.max = RTE_COMP_LEVEL_MAX;
diff --git a/app/test-compress-perf/comp_perf_test_cyclecount.c b/app/test-compress-perf/comp_perf_test_cyclecount.c
index ce6c4d7605..81c3d30038 100644
--- a/app/test-compress-perf/comp_perf_test_cyclecount.c
+++ b/app/test-compress-perf/comp_perf_test_cyclecount.c
@@ -193,7 +193,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_COMPRESS,
.compress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.deflate.huffman = test_data->huffman_enc,
.level = test_data->level,
.window_size = test_data->window_sz,
@@ -208,7 +208,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_DECOMPRESS,
.decompress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.chksum = RTE_COMP_CHECKSUM_NONE,
.window_size = test_data->window_sz,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
diff --git a/app/test-compress-perf/comp_perf_test_throughput.c b/app/test-compress-perf/comp_perf_test_throughput.c
index c9f8237626..2545ee9925 100644
--- a/app/test-compress-perf/comp_perf_test_throughput.c
+++ b/app/test-compress-perf/comp_perf_test_throughput.c
@@ -84,7 +84,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_COMPRESS,
.compress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.deflate.huffman = test_data->huffman_enc,
.level = test_data->level,
.window_size = test_data->window_sz,
@@ -99,7 +99,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_DECOMPRESS,
.decompress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.chksum = RTE_COMP_CHECKSUM_NONE,
.window_size = test_data->window_sz,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
diff --git a/app/test-compress-perf/comp_perf_test_verify.c b/app/test-compress-perf/comp_perf_test_verify.c
index 7d6b6abecd..88f4f41851 100644
--- a/app/test-compress-perf/comp_perf_test_verify.c
+++ b/app/test-compress-perf/comp_perf_test_verify.c
@@ -87,7 +87,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_COMPRESS,
.compress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.deflate.huffman = test_data->huffman_enc,
.level = test_data->level,
.window_size = test_data->window_sz,
@@ -104,7 +104,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_DECOMPRESS,
.decompress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.chksum = RTE_COMP_CHECKSUM_NONE,
.window_size = test_data->window_sz,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
diff --git a/app/test-compress-perf/main.c b/app/test-compress-perf/main.c
index bbb4c7917b..d049527ba1 100644
--- a/app/test-compress-perf/main.c
+++ b/app/test-compress-perf/main.c
@@ -57,29 +57,39 @@ comp_perf_check_capabilities(struct comp_test_data *test_data, uint8_t cdev_id)
{
const struct rte_compressdev_capabilities *cap;
- cap = rte_compressdev_capability_get(cdev_id,
- RTE_COMP_ALGO_DEFLATE);
+ cap = rte_compressdev_capability_get(cdev_id, test_data->test_algo);
if (cap == NULL) {
RTE_LOG(ERR, USER1,
- "Compress device does not support DEFLATE\n");
+ "Compress device does not support %u algorithm\n",
+ test_data->test_algo);
return -1;
}
uint64_t comp_flags = cap->comp_feature_flags;
- /* Huffman encoding */
- if (test_data->huffman_enc == RTE_COMP_HUFFMAN_FIXED &&
- (comp_flags & RTE_COMP_FF_HUFFMAN_FIXED) == 0) {
- RTE_LOG(ERR, USER1,
- "Compress device does not supported Fixed Huffman\n");
- return -1;
- }
+ /* Algorithm type */
+ switch (test_data->test_algo) {
+ case RTE_COMP_ALGO_DEFLATE:
+ /* Huffman encoding */
+ if (test_data->huffman_enc == RTE_COMP_HUFFMAN_FIXED &&
+ (comp_flags & RTE_COMP_FF_HUFFMAN_FIXED) == 0) {
+ RTE_LOG(ERR, USER1,
+ "Compress device does not supported Fixed Huffman\n");
+ return -1;
+ }
- if (test_data->huffman_enc == RTE_COMP_HUFFMAN_DYNAMIC &&
- (comp_flags & RTE_COMP_FF_HUFFMAN_DYNAMIC) == 0) {
- RTE_LOG(ERR, USER1,
- "Compress device does not supported Dynamic Huffman\n");
+ if (test_data->huffman_enc == RTE_COMP_HUFFMAN_DYNAMIC &&
+ (comp_flags & RTE_COMP_FF_HUFFMAN_DYNAMIC) == 0) {
+ RTE_LOG(ERR, USER1,
+ "Compress device does not supported Dynamic Huffman\n");
+ return -1;
+ }
+ break;
+ case RTE_COMP_ALGO_LZS:
+ case RTE_COMP_ALGO_NULL:
+ break;
+ default:
return -1;
}
diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index b2a8c921b9..6aa3f31246 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -116,6 +116,10 @@ New Features
Enable the application options for testing only compress and only decompress.
+* **Added algo option in test-compress-perf.**
+
+ Added support for testing other algorithms except for DEFLAT.
+
Removed Items
-------------
diff --git a/doc/guides/tools/comp_perf.rst b/doc/guides/tools/comp_perf.rst
index 9d2f4dbe4a..cf9035cc23 100644
--- a/doc/guides/tools/comp_perf.rst
+++ b/doc/guides/tools/comp_perf.rst
@@ -84,6 +84,8 @@ Application Options
``--operation [comp/decomp/comp_and_decomp]``: perform test on compression, decompression or both operations
+ ``--algo [null/deflate/lzs]`` : perform test on algorithm null(DMA), Deflate or lzs (default: Deflate)
+
``--huffman-enc [fixed/dynamic/default]``: Huffman encoding (default: dynamic)
``--compress-level N``: compression level, which could be a single value, list or range (default: range between 1 and 9)
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v3 4/4] app/test-compress-perf: add LZ4 support
2023-02-13 6:11 ` [PATCH v3 0/4] compressdev: " Michael Baum
` (2 preceding siblings ...)
2023-02-13 6:11 ` [PATCH v3 3/4] app/test-compress-perf: add algo option Michael Baum
@ 2023-02-13 6:11 ` Michael Baum
2023-02-13 15:47 ` [PATCH v3 0/4] compressdev: " Zhang, Fan
2023-02-14 17:40 ` [PATCH v4 " Michael Baum
5 siblings, 0 replies; 31+ messages in thread
From: Michael Baum @ 2023-02-13 6:11 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe, Thomas Monjalon
Add support for LZ4 algorithm and add a new parameter for it.
Signed-off-by: Michael Baum <michaelba@nvidia.com>
---
app/test-compress-perf/comp_perf_options.h | 1 +
.../comp_perf_options_parse.c | 43 ++++++++++++++++++-
.../comp_perf_test_cyclecount.c | 7 ++-
.../comp_perf_test_throughput.c | 7 ++-
.../comp_perf_test_verify.c | 7 ++-
app/test-compress-perf/main.c | 17 ++++++++
doc/guides/rel_notes/release_23_03.rst | 1 +
doc/guides/tools/comp_perf.rst | 5 ++-
8 files changed, 82 insertions(+), 6 deletions(-)
diff --git a/app/test-compress-perf/comp_perf_options.h b/app/test-compress-perf/comp_perf_options.h
index 5e5227a700..828a7309d8 100644
--- a/app/test-compress-perf/comp_perf_options.h
+++ b/app/test-compress-perf/comp_perf_options.h
@@ -61,6 +61,7 @@ struct comp_test_data {
uint16_t max_sgl_segs;
uint32_t total_segs;
+ uint8_t lz4_flags;
enum rte_comp_huffman huffman_enc;
enum comp_operation test_op;
enum rte_comp_algorithm test_algo;
diff --git a/app/test-compress-perf/comp_perf_options_parse.c b/app/test-compress-perf/comp_perf_options_parse.c
index 97ddff87ef..6d8c370fc2 100644
--- a/app/test-compress-perf/comp_perf_options_parse.c
+++ b/app/test-compress-perf/comp_perf_options_parse.c
@@ -27,6 +27,7 @@
#define CPERF_OPTYPE ("operation")
#define CPERF_ALGO ("algo")
#define CPERF_HUFFMAN_ENC ("huffman-enc")
+#define CPERF_LZ4_FLAGS ("lz4-flags")
#define CPERF_LEVEL ("compress-level")
#define CPERF_WINDOW_SIZE ("window-sz")
#define CPERF_EXTERNAL_MBUFS ("external-mbufs")
@@ -57,10 +58,11 @@ usage(char *progname)
" compressed/decompressed (default: 10000)\n"
" --operation [comp/decomp/comp_and_decomp]: perform test on\n"
" compression, decompression or both operations\n"
- " --algo [null/deflate/lzs]: perform test on algorithm\n"
- " null(DMA), deflate or lzs (default: deflate)\n"
+ " --algo [null/deflate/lzs/lz4]: perform test on algorithm\n"
+ " null(DMA), deflate, lzs or lz4 (default: deflate)\n"
" --huffman-enc [fixed/dynamic/default]: Huffman encoding\n"
" (default: dynamic)\n"
+ " --lz4-flags N: flags to configure LZ4 algorithm (default: 0)\n"
" --compress-level N: compression level, which could be a single value, list or range\n"
" (default: range between 1 and 9)\n"
" --window-sz N: base two log value of compression window size\n"
@@ -153,6 +155,23 @@ parse_uint16_t(uint16_t *value, const char *arg)
return 0;
}
+static int
+parse_uint8_t(uint8_t *value, const char *arg)
+{
+ uint32_t val = 0;
+ int ret = parse_uint32_t(&val, arg);
+
+ if (ret < 0)
+ return ret;
+
+ if (val > UINT8_MAX)
+ return -ERANGE;
+
+ *value = (uint8_t) val;
+
+ return 0;
+}
+
static int
parse_range(const char *arg, uint8_t *min, uint8_t *max, uint8_t *inc)
{
@@ -488,6 +507,10 @@ parse_algo(struct comp_test_data *test_data, const char *arg)
{
"lzs",
RTE_COMP_ALGO_LZS
+ },
+ {
+ "lz4",
+ RTE_COMP_ALGO_LZ4
}
};
@@ -533,6 +556,19 @@ parse_huffman_enc(struct comp_test_data *test_data, const char *arg)
return 0;
}
+static int
+parse_lz4_flags(struct comp_test_data *test_data, const char *arg)
+{
+ int ret = parse_uint8_t(&test_data->lz4_flags, arg);
+
+ if (ret) {
+ RTE_LOG(ERR, USER1, "Failed to parse LZ4 flags\n");
+ return -1;
+ }
+
+ return 0;
+}
+
static int
parse_level(struct comp_test_data *test_data, const char *arg)
{
@@ -607,6 +643,7 @@ static struct option lgopts[] = {
{ CPERF_OPTYPE, required_argument, 0, 0 },
{ CPERF_ALGO, required_argument, 0, 0 },
{ CPERF_HUFFMAN_ENC, required_argument, 0, 0 },
+ { CPERF_LZ4_FLAGS, required_argument, 0, 0 },
{ CPERF_LEVEL, required_argument, 0, 0 },
{ CPERF_WINDOW_SIZE, required_argument, 0, 0 },
{ CPERF_EXTERNAL_MBUFS, 0, 0, 0 },
@@ -630,6 +667,7 @@ comp_perf_opts_parse_long(int opt_idx, struct comp_test_data *test_data)
{ CPERF_OPTYPE, parse_op_type },
{ CPERF_ALGO, parse_algo },
{ CPERF_HUFFMAN_ENC, parse_huffman_enc },
+ { CPERF_LZ4_FLAGS, parse_lz4_flags },
{ CPERF_LEVEL, parse_level },
{ CPERF_WINDOW_SIZE, parse_window_sz },
{ CPERF_EXTERNAL_MBUFS, parse_external_mbufs },
@@ -682,6 +720,7 @@ comp_perf_options_default(struct comp_test_data *test_data)
test_data->pool_sz = 8192;
test_data->max_sgl_segs = 16;
test_data->num_iter = 10000;
+ test_data->lz4_flags = 0;
test_data->huffman_enc = RTE_COMP_HUFFMAN_DYNAMIC;
test_data->test_op = COMPRESS_DECOMPRESS;
test_data->test_algo = RTE_COMP_ALGO_DEFLATE;
diff --git a/app/test-compress-perf/comp_perf_test_cyclecount.c b/app/test-compress-perf/comp_perf_test_cyclecount.c
index 81c3d30038..4d336ec8d6 100644
--- a/app/test-compress-perf/comp_perf_test_cyclecount.c
+++ b/app/test-compress-perf/comp_perf_test_cyclecount.c
@@ -194,13 +194,16 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type)
.type = RTE_COMP_COMPRESS,
.compress = {
.algo = test_data->test_algo,
- .deflate.huffman = test_data->huffman_enc,
.level = test_data->level,
.window_size = test_data->window_sz,
.chksum = RTE_COMP_CHECKSUM_NONE,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_DEFLATE)
+ xform.compress.deflate.huffman = test_data->huffman_enc;
+ else if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.compress.lz4.flags = test_data->lz4_flags;
input_bufs = mem->decomp_bufs;
output_bufs = mem->comp_bufs;
out_seg_sz = test_data->out_seg_sz;
@@ -214,6 +217,8 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type)
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.decompress.lz4.flags = test_data->lz4_flags;
input_bufs = mem->comp_bufs;
output_bufs = mem->decomp_bufs;
out_seg_sz = test_data->seg_sz;
diff --git a/app/test-compress-perf/comp_perf_test_throughput.c b/app/test-compress-perf/comp_perf_test_throughput.c
index 2545ee9925..1f7072d223 100644
--- a/app/test-compress-perf/comp_perf_test_throughput.c
+++ b/app/test-compress-perf/comp_perf_test_throughput.c
@@ -85,13 +85,16 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type)
.type = RTE_COMP_COMPRESS,
.compress = {
.algo = test_data->test_algo,
- .deflate.huffman = test_data->huffman_enc,
.level = test_data->level,
.window_size = test_data->window_sz,
.chksum = RTE_COMP_CHECKSUM_NONE,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_DEFLATE)
+ xform.compress.deflate.huffman = test_data->huffman_enc;
+ else if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.compress.lz4.flags = test_data->lz4_flags;
input_bufs = mem->decomp_bufs;
output_bufs = mem->comp_bufs;
out_seg_sz = test_data->out_seg_sz;
@@ -105,6 +108,8 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type)
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.decompress.lz4.flags = test_data->lz4_flags;
input_bufs = mem->comp_bufs;
output_bufs = mem->decomp_bufs;
out_seg_sz = test_data->seg_sz;
diff --git a/app/test-compress-perf/comp_perf_test_verify.c b/app/test-compress-perf/comp_perf_test_verify.c
index 88f4f41851..7bd18073cf 100644
--- a/app/test-compress-perf/comp_perf_test_verify.c
+++ b/app/test-compress-perf/comp_perf_test_verify.c
@@ -88,13 +88,16 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
.type = RTE_COMP_COMPRESS,
.compress = {
.algo = test_data->test_algo,
- .deflate.huffman = test_data->huffman_enc,
.level = test_data->level,
.window_size = test_data->window_sz,
.chksum = RTE_COMP_CHECKSUM_NONE,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_DEFLATE)
+ xform.compress.deflate.huffman = test_data->huffman_enc;
+ else if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.compress.lz4.flags = test_data->lz4_flags;
output_data_ptr = ctx->mem.compressed_data;
output_data_sz = &ctx->comp_data_sz;
input_bufs = mem->decomp_bufs;
@@ -110,6 +113,8 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.decompress.lz4.flags = test_data->lz4_flags;
output_data_ptr = ctx->mem.decompressed_data;
output_data_sz = &ctx->decomp_data_sz;
input_bufs = mem->comp_bufs;
diff --git a/app/test-compress-perf/main.c b/app/test-compress-perf/main.c
index d049527ba1..fa366123ed 100644
--- a/app/test-compress-perf/main.c
+++ b/app/test-compress-perf/main.c
@@ -86,6 +86,23 @@ comp_perf_check_capabilities(struct comp_test_data *test_data, uint8_t cdev_id)
return -1;
}
break;
+ case RTE_COMP_ALGO_LZ4:
+ /* LZ4 flags */
+ if ((test_data->lz4_flags & RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM) &&
+ (comp_flags & RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM) == 0) {
+ RTE_LOG(ERR, USER1,
+ "Compress device does not support LZ4 block with checksum\n");
+ return -1;
+ }
+
+ if ((test_data->lz4_flags &
+ RTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE) &&
+ (comp_flags & RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE) == 0) {
+ RTE_LOG(ERR, USER1,
+ "Compress device does not support LZ4 independent blocks\n");
+ return -1;
+ }
+ break;
case RTE_COMP_ALGO_LZS:
case RTE_COMP_ALGO_NULL:
break;
diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index 6aa3f31246..c05fa2930b 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -111,6 +111,7 @@ New Features
* Added support for ``RTE_COMP_ALGO_LZ4``.
* Added support for ``RTE_COMP_CHECKSUM_XXHASH32``.
+ * Added support for ``lz4`` in test-compress-perf algo options.
* **Allowed test single compress operation in test-compress-perf.**
diff --git a/doc/guides/tools/comp_perf.rst b/doc/guides/tools/comp_perf.rst
index cf9035cc23..5eb05a095c 100644
--- a/doc/guides/tools/comp_perf.rst
+++ b/doc/guides/tools/comp_perf.rst
@@ -84,10 +84,13 @@ Application Options
``--operation [comp/decomp/comp_and_decomp]``: perform test on compression, decompression or both operations
- ``--algo [null/deflate/lzs]`` : perform test on algorithm null(DMA), Deflate or lzs (default: Deflate)
+ ``--algo [null/deflate/lzs/lz4]`` : perform test on algorithm null(DMA), Deflate, lzs or lz4 (default: Deflate)
``--huffman-enc [fixed/dynamic/default]``: Huffman encoding (default: dynamic)
+ ``--lz4-flags N``: flags to for LZ4 parameters,
+ see `LZ4 Frame Descriptor <https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor>`_ (default: no flags)
+
``--compress-level N``: compression level, which could be a single value, list or range (default: range between 1 and 9)
``--window-sz N``: base two log value of compression window size (default: max supported by PMD)
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* RE: [EXT] [PATCH v3 2/4] app/test-compress-perf: allow test single compress operation
2023-02-13 6:11 ` [PATCH v3 2/4] app/test-compress-perf: allow test single compress operation Michael Baum
@ 2023-02-13 7:23 ` Akhil Goyal
2023-02-14 16:37 ` Michael Baum
0 siblings, 1 reply; 31+ messages in thread
From: Akhil Goyal @ 2023-02-13 7:23 UTC (permalink / raw)
To: Michael Baum, dev; +Cc: Matan Azrad, Ashish Gupta, Fiona Trahe, Thomas Monjalon
> Part of the application options is to test only compress and only
> decompress but actually the application ignores this user option and
> tries to test always both compress and decompress.
From the description it seems to be a bug in previous implementation.
Can we back port it? And update the patch title accordingly?
Also, release note update is not required for this fix.
>
> Allow testing only compress and only decompress.
>
> Signed-off-by: Matan Azrad <matan@nvidia.com>
> Signed-off-by: Michael Baum <michaelba@nvidia.com>
^ permalink raw reply [flat|nested] 31+ messages in thread
* RE: [EXT] [PATCH v3 3/4] app/test-compress-perf: add algo option
2023-02-13 6:11 ` [PATCH v3 3/4] app/test-compress-perf: add algo option Michael Baum
@ 2023-02-13 7:29 ` Akhil Goyal
2023-02-14 16:32 ` Michael Baum
0 siblings, 1 reply; 31+ messages in thread
From: Akhil Goyal @ 2023-02-13 7:29 UTC (permalink / raw)
To: Michael Baum, dev; +Cc: Matan Azrad, Ashish Gupta, Fiona Trahe, Thomas Monjalon
> Add a command line option `algo` to select the compress algorithm
> supported by the compress API: null (DMA), deflate and lzs.
>
> Default for deflate.
>
> Signed-off-by: Matan Azrad <matan@nvidia.com>
> Signed-off-by: Michael Baum <michaelba@nvidia.com>
> ---
> app/test-compress-perf/comp_perf_options.h | 2 +
> .../comp_perf_options_parse.c | 40 ++++++++++++++++++-
> .../comp_perf_test_cyclecount.c | 4 +-
> .../comp_perf_test_throughput.c | 4 +-
> .../comp_perf_test_verify.c | 4 +-
> app/test-compress-perf/main.c | 38 +++++++++++-------
> doc/guides/rel_notes/release_23_03.rst | 4 ++
> doc/guides/tools/comp_perf.rst | 2 +
> 8 files changed, 76 insertions(+), 22 deletions(-)
>
> diff --git a/app/test-compress-perf/comp_perf_options.h b/app/test-compress-
> perf/comp_perf_options.h
> index d00b299247..5e5227a700 100644
> --- a/app/test-compress-perf/comp_perf_options.h
> +++ b/app/test-compress-perf/comp_perf_options.h
> @@ -63,6 +63,8 @@ struct comp_test_data {
>
> enum rte_comp_huffman huffman_enc;
> enum comp_operation test_op;
> + enum rte_comp_algorithm test_algo;
> +
> int window_sz;
> struct range_list level_lst;
> uint8_t level;
> diff --git a/app/test-compress-perf/comp_perf_options_parse.c b/app/test-
> compress-perf/comp_perf_options_parse.c
> index 7a992bf43e..97ddff87ef 100644
> --- a/app/test-compress-perf/comp_perf_options_parse.c
> +++ b/app/test-compress-perf/comp_perf_options_parse.c
> @@ -25,6 +25,7 @@
> #define CPERF_MAX_SGL_SEGS ("max-num-sgl-segs")
> #define CPERF_NUM_ITER ("num-iter")
> #define CPERF_OPTYPE ("operation")
> +#define CPERF_ALGO ("algo")
> #define CPERF_HUFFMAN_ENC ("huffman-enc")
> #define CPERF_LEVEL ("compress-level")
> #define CPERF_WINDOW_SIZE ("window-sz")
> @@ -56,6 +57,8 @@ usage(char *progname)
> " compressed/decompressed (default: 10000)\n"
> " --operation [comp/decomp/comp_and_decomp]: perform
> test on\n"
> " compression, decompression or both
> operations\n"
> + " --algo [null/deflate/lzs]: perform test on algorithm\n"
> + " null(DMA), deflate or lzs (default: deflate)\n"
Since LZ4 is also supported now, is it not good to add that also in this patch?
> " --huffman-enc [fixed/dynamic/default]: Huffman encoding\n"
> " (default: dynamic)\n"
> " --compress-level N: compression level, which could be a single
> value, list or range\n"
> @@ -470,6 +473,36 @@ parse_op_type(struct comp_test_data *test_data,
> const char *arg)
> return 0;
> }
>
> +static int
> +parse_algo(struct comp_test_data *test_data, const char *arg)
> +{
> + struct name_id_map algo_namemap[] = {
> + {
> + "null",
> + RTE_COMP_ALGO_NULL
> + },
> + {
> + "deflate",
> + RTE_COMP_ALGO_DEFLATE
> + },
> + {
> + "lzs",
> + RTE_COMP_ALGO_LZS
> + }
> + };
> +
> + int id = get_str_key_id_mapping(algo_namemap,
> + RTE_DIM(algo_namemap), arg);
> + if (id < 0) {
> + RTE_LOG(ERR, USER1, "Invalid algorithm specified\n");
> + return -1;
> + }
> +
> + test_data->test_algo = (enum rte_comp_algorithm)id;
> +
> + return 0;
> +}
> +
> static int
> parse_huffman_enc(struct comp_test_data *test_data, const char *arg)
> {
> @@ -491,7 +524,7 @@ parse_huffman_enc(struct comp_test_data *test_data,
> const char *arg)
> int id = get_str_key_id_mapping(huffman_namemap,
> RTE_DIM(huffman_namemap), arg);
> if (id < 0) {
> - RTE_LOG(ERR, USER1, "Invalid Huffmane encoding
> specified\n");
> + RTE_LOG(ERR, USER1, "Invalid Huffman encoding specified\n");
This change is not related to this patch.
Should be a separate patch and should be backported.
> return -1;
> }
>
> @@ -507,7 +540,7 @@ parse_level(struct comp_test_data *test_data, const
> char *arg)
>
> /*
> * Try parsing the argument as a range, if it fails,
> - * arse it as a list
> + * parse it as a list
Same here.
> */
> if (parse_range(arg, &test_data->level_lst.min,
> &test_data->level_lst.max,
> @@ -572,6 +605,7 @@ static struct option lgopts[] = {
> { CPERF_MAX_SGL_SEGS, required_argument, 0, 0},
> { CPERF_NUM_ITER, required_argument, 0, 0 },
> { CPERF_OPTYPE, required_argument, 0, 0 },
> + { CPERF_ALGO, required_argument, 0, 0 },
> { CPERF_HUFFMAN_ENC, required_argument, 0, 0 },
> { CPERF_LEVEL, required_argument, 0, 0 },
> { CPERF_WINDOW_SIZE, required_argument, 0, 0 },
> @@ -594,6 +628,7 @@ comp_perf_opts_parse_long(int opt_idx, struct
> comp_test_data *test_data)
> { CPERF_MAX_SGL_SEGS, parse_max_num_sgl_segs },
> { CPERF_NUM_ITER, parse_num_iter },
> { CPERF_OPTYPE, parse_op_type },
> + { CPERF_ALGO, parse_algo },
> { CPERF_HUFFMAN_ENC, parse_huffman_enc },
> { CPERF_LEVEL, parse_level },
> { CPERF_WINDOW_SIZE, parse_window_sz },
> @@ -649,6 +684,7 @@ comp_perf_options_default(struct comp_test_data
> *test_data)
> test_data->num_iter = 10000;
> test_data->huffman_enc = RTE_COMP_HUFFMAN_DYNAMIC;
> test_data->test_op = COMPRESS_DECOMPRESS;
> + test_data->test_algo = RTE_COMP_ALGO_DEFLATE;
> test_data->window_sz = -1;
> test_data->level_lst.min = RTE_COMP_LEVEL_MIN;
> test_data->level_lst.max = RTE_COMP_LEVEL_MAX;
> diff --git a/app/test-compress-perf/comp_perf_test_cyclecount.c b/app/test-
> compress-perf/comp_perf_test_cyclecount.c
> index ce6c4d7605..81c3d30038 100644
> --- a/app/test-compress-perf/comp_perf_test_cyclecount.c
> +++ b/app/test-compress-perf/comp_perf_test_cyclecount.c
> @@ -193,7 +193,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum
> rte_comp_xform_type type)
> xform = (struct rte_comp_xform) {
> .type = RTE_COMP_COMPRESS,
> .compress = {
> - .algo = RTE_COMP_ALGO_DEFLATE,
> + .algo = test_data->test_algo,
> .deflate.huffman = test_data->huffman_enc,
> .level = test_data->level,
> .window_size = test_data->window_sz,
> @@ -208,7 +208,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum
> rte_comp_xform_type type)
> xform = (struct rte_comp_xform) {
> .type = RTE_COMP_DECOMPRESS,
> .decompress = {
> - .algo = RTE_COMP_ALGO_DEFLATE,
> + .algo = test_data->test_algo,
> .chksum = RTE_COMP_CHECKSUM_NONE,
> .window_size = test_data->window_sz,
> .hash_algo = RTE_COMP_HASH_ALGO_NONE
> diff --git a/app/test-compress-perf/comp_perf_test_throughput.c b/app/test-
> compress-perf/comp_perf_test_throughput.c
> index c9f8237626..2545ee9925 100644
> --- a/app/test-compress-perf/comp_perf_test_throughput.c
> +++ b/app/test-compress-perf/comp_perf_test_throughput.c
> @@ -84,7 +84,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum
> rte_comp_xform_type type)
> xform = (struct rte_comp_xform) {
> .type = RTE_COMP_COMPRESS,
> .compress = {
> - .algo = RTE_COMP_ALGO_DEFLATE,
> + .algo = test_data->test_algo,
> .deflate.huffman = test_data->huffman_enc,
> .level = test_data->level,
> .window_size = test_data->window_sz,
> @@ -99,7 +99,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum
> rte_comp_xform_type type)
> xform = (struct rte_comp_xform) {
> .type = RTE_COMP_DECOMPRESS,
> .decompress = {
> - .algo = RTE_COMP_ALGO_DEFLATE,
> + .algo = test_data->test_algo,
> .chksum = RTE_COMP_CHECKSUM_NONE,
> .window_size = test_data->window_sz,
> .hash_algo = RTE_COMP_HASH_ALGO_NONE
> diff --git a/app/test-compress-perf/comp_perf_test_verify.c b/app/test-
> compress-perf/comp_perf_test_verify.c
> index 7d6b6abecd..88f4f41851 100644
> --- a/app/test-compress-perf/comp_perf_test_verify.c
> +++ b/app/test-compress-perf/comp_perf_test_verify.c
> @@ -87,7 +87,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum
> rte_comp_xform_type type)
> xform = (struct rte_comp_xform) {
> .type = RTE_COMP_COMPRESS,
> .compress = {
> - .algo = RTE_COMP_ALGO_DEFLATE,
> + .algo = test_data->test_algo,
> .deflate.huffman = test_data->huffman_enc,
> .level = test_data->level,
> .window_size = test_data->window_sz,
> @@ -104,7 +104,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum
> rte_comp_xform_type type)
> xform = (struct rte_comp_xform) {
> .type = RTE_COMP_DECOMPRESS,
> .decompress = {
> - .algo = RTE_COMP_ALGO_DEFLATE,
> + .algo = test_data->test_algo,
> .chksum = RTE_COMP_CHECKSUM_NONE,
> .window_size = test_data->window_sz,
> .hash_algo = RTE_COMP_HASH_ALGO_NONE
> diff --git a/app/test-compress-perf/main.c b/app/test-compress-perf/main.c
> index bbb4c7917b..d049527ba1 100644
> --- a/app/test-compress-perf/main.c
> +++ b/app/test-compress-perf/main.c
> @@ -57,29 +57,39 @@ comp_perf_check_capabilities(struct comp_test_data
> *test_data, uint8_t cdev_id)
> {
> const struct rte_compressdev_capabilities *cap;
>
> - cap = rte_compressdev_capability_get(cdev_id,
> - RTE_COMP_ALGO_DEFLATE);
> + cap = rte_compressdev_capability_get(cdev_id, test_data->test_algo);
>
> if (cap == NULL) {
> RTE_LOG(ERR, USER1,
> - "Compress device does not support DEFLATE\n");
> + "Compress device does not support %u algorithm\n",
> + test_data->test_algo);
> return -1;
> }
>
> uint64_t comp_flags = cap->comp_feature_flags;
>
> - /* Huffman encoding */
> - if (test_data->huffman_enc == RTE_COMP_HUFFMAN_FIXED &&
> - (comp_flags & RTE_COMP_FF_HUFFMAN_FIXED) == 0)
> {
> - RTE_LOG(ERR, USER1,
> - "Compress device does not supported Fixed
> Huffman\n");
> - return -1;
> - }
> + /* Algorithm type */
> + switch (test_data->test_algo) {
> + case RTE_COMP_ALGO_DEFLATE:
> + /* Huffman encoding */
> + if (test_data->huffman_enc == RTE_COMP_HUFFMAN_FIXED
> &&
> + (comp_flags & RTE_COMP_FF_HUFFMAN_FIXED) == 0) {
> + RTE_LOG(ERR, USER1,
> + "Compress device does not supported Fixed
> Huffman\n");
> + return -1;
> + }
>
> - if (test_data->huffman_enc == RTE_COMP_HUFFMAN_DYNAMIC &&
> - (comp_flags & RTE_COMP_FF_HUFFMAN_DYNAMIC)
> == 0) {
> - RTE_LOG(ERR, USER1,
> - "Compress device does not supported Dynamic
> Huffman\n");
> + if (test_data->huffman_enc ==
> RTE_COMP_HUFFMAN_DYNAMIC &&
> + (comp_flags & RTE_COMP_FF_HUFFMAN_DYNAMIC) == 0) {
> + RTE_LOG(ERR, USER1,
> + "Compress device does not supported Dynamic
> Huffman\n");
> + return -1;
> + }
> + break;
> + case RTE_COMP_ALGO_LZS:
> + case RTE_COMP_ALGO_NULL:
> + break;
> + default:
> return -1;
> }
>
> diff --git a/doc/guides/rel_notes/release_23_03.rst
> b/doc/guides/rel_notes/release_23_03.rst
> index b2a8c921b9..6aa3f31246 100644
> --- a/doc/guides/rel_notes/release_23_03.rst
> +++ b/doc/guides/rel_notes/release_23_03.rst
> @@ -116,6 +116,10 @@ New Features
>
> Enable the application options for testing only compress and only decompress.
>
> +* **Added algo option in test-compress-perf.**
> +
> + Added support for testing other algorithms except for DEFLAT.
> +
This patch is only adding a new option to the test app.
I believe release notes are not needed.
>
> Removed Items
> -------------
> diff --git a/doc/guides/tools/comp_perf.rst b/doc/guides/tools/comp_perf.rst
> index 9d2f4dbe4a..cf9035cc23 100644
> --- a/doc/guides/tools/comp_perf.rst
> +++ b/doc/guides/tools/comp_perf.rst
> @@ -84,6 +84,8 @@ Application Options
>
> ``--operation [comp/decomp/comp_and_decomp]``: perform test on
> compression, decompression or both operations
>
> + ``--algo [null/deflate/lzs]`` : perform test on algorithm null(DMA), Deflate or lzs
> (default: Deflate)
> +
> ``--huffman-enc [fixed/dynamic/default]``: Huffman encoding (default:
> dynamic)
>
> ``--compress-level N``: compression level, which could be a single value, list or
> range (default: range between 1 and 9)
> --
> 2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v3 0/4] compressdev: add LZ4 support
2023-02-13 6:11 ` [PATCH v3 0/4] compressdev: " Michael Baum
` (3 preceding siblings ...)
2023-02-13 6:11 ` [PATCH v3 4/4] app/test-compress-perf: add LZ4 support Michael Baum
@ 2023-02-13 15:47 ` Zhang, Fan
2023-02-13 15:56 ` Michael Baum
2023-02-14 17:40 ` [PATCH v4 " Michael Baum
5 siblings, 1 reply; 31+ messages in thread
From: Zhang, Fan @ 2023-02-13 15:47 UTC (permalink / raw)
To: dev
Hi,
The patchset only have library change and added performance tests.
We shall have at least one PMD to support the added algorithms and unit
test updated for the PMD/algorithms.
Regards,
Fan
On 2/13/2023 6:11 AM, Michael Baum wrote:
> Add support for LZ4 compression algorithm for both API and app.
>
> In addition, enable app to test de/comp only. This option already
> exists in both app doc and arguments but isn't implemented inside the
> application.
>
> LZ4: https://github.com/lz4/lz4
> RFC:
> https://patchwork.dpdk.org/project/dpdk/patch/20220410182622.8828-1-rzidane@nvidia.com/
>
> v2:
> - Add macros for all LZ4 flags in the standard.
> - Rebase.
>
> v3:
> - Rebase.
>
> Michael Baum (4):
> compressdev: add LZ4 algorithm support
> app/test-compress-perf: allow test single compress operation
> app/test-compress-perf: add algo option
> app/test-compress-perf: add LZ4 support
>
> app/test-compress-perf/comp_perf_options.h | 9 +-
> .../comp_perf_options_parse.c | 83 +++++++++++-
> .../comp_perf_test_common.c | 126 +++++++++++++-----
> .../comp_perf_test_cyclecount.c | 82 +++++++-----
> .../comp_perf_test_throughput.c | 75 +++++++----
> .../comp_perf_test_verify.c | 75 +++++++----
> app/test-compress-perf/main.c | 63 +++++++--
> doc/guides/compressdevs/features/default.ini | 7 +
> doc/guides/rel_notes/release_23_03.rst | 16 +++
> doc/guides/tools/comp_perf.rst | 5 +
> lib/compressdev/rte_comp.c | 12 ++
> lib/compressdev/rte_comp.h | 79 ++++++++++-
> 12 files changed, 488 insertions(+), 144 deletions(-)
>
^ permalink raw reply [flat|nested] 31+ messages in thread
* RE: [PATCH v3 0/4] compressdev: add LZ4 support
2023-02-13 15:47 ` [PATCH v3 0/4] compressdev: " Zhang, Fan
@ 2023-02-13 15:56 ` Michael Baum
0 siblings, 0 replies; 31+ messages in thread
From: Michael Baum @ 2023-02-13 15:56 UTC (permalink / raw)
To: Zhang, Fan, dev
On 2/13/2023 17:47 PM, Zhang, Fan wrote:
> Hi,
>
> The patchset only have library change and added performance tests.
>
> We shall have at least one PMD to support the added algorithms and unit test
> updated for the PMD/algorithms.
There is another patchset which adds it in MLX5 compress PMD:
series-26766 ("compress/mlx5: add LZ4 support") https://patchwork.dpdk.org/project/dpdk/cover/20230202162537.1067595-1-michaelba@nvidia.com/
>
> Regards,
>
> Fan
>
> On 2/13/2023 6:11 AM, Michael Baum wrote:
> > Add support for LZ4 compression algorithm for both API and app.
> >
> > In addition, enable app to test de/comp only. This option already
> > exists in both app doc and arguments but isn't implemented inside the
> > application.
> >
> > LZ4: https://github.com/lz4/lz4
> > RFC:
> > https://patchwork.dpdk.org/project/dpdk/patch/20220410182622.8828-1-rz
> > idane@nvidia.com/
> >
> > v2:
> > - Add macros for all LZ4 flags in the standard.
> > - Rebase.
> >
> > v3:
> > - Rebase.
> >
> > Michael Baum (4):
> > compressdev: add LZ4 algorithm support
> > app/test-compress-perf: allow test single compress operation
> > app/test-compress-perf: add algo option
> > app/test-compress-perf: add LZ4 support
> >
> > app/test-compress-perf/comp_perf_options.h | 9 +-
> > .../comp_perf_options_parse.c | 83 +++++++++++-
> > .../comp_perf_test_common.c | 126 +++++++++++++-----
> > .../comp_perf_test_cyclecount.c | 82 +++++++-----
> > .../comp_perf_test_throughput.c | 75 +++++++----
> > .../comp_perf_test_verify.c | 75 +++++++----
> > app/test-compress-perf/main.c | 63 +++++++--
> > doc/guides/compressdevs/features/default.ini | 7 +
> > doc/guides/rel_notes/release_23_03.rst | 16 +++
> > doc/guides/tools/comp_perf.rst | 5 +
> > lib/compressdev/rte_comp.c | 12 ++
> > lib/compressdev/rte_comp.h | 79 ++++++++++-
> > 12 files changed, 488 insertions(+), 144 deletions(-)
> >
^ permalink raw reply [flat|nested] 31+ messages in thread
* RE: [EXT] [PATCH v3 3/4] app/test-compress-perf: add algo option
2023-02-13 7:29 ` [EXT] " Akhil Goyal
@ 2023-02-14 16:32 ` Michael Baum
0 siblings, 0 replies; 31+ messages in thread
From: Michael Baum @ 2023-02-14 16:32 UTC (permalink / raw)
To: Akhil Goyal, dev
Cc: Matan Azrad, Ashish Gupta, Fiona Trahe,
NBU-Contact-Thomas Monjalon (EXTERNAL)
On Monday, 13 February 2023 9:29, Akhil Goyal wrote:
>
> > Add a command line option `algo` to select the compress algorithm
> > supported by the compress API: null (DMA), deflate and lzs.
> >
> > Default for deflate.
> >
> > Signed-off-by: Matan Azrad <matan@nvidia.com>
> > Signed-off-by: Michael Baum <michaelba@nvidia.com>
> > ---
> > app/test-compress-perf/comp_perf_options.h | 2 +
> > .../comp_perf_options_parse.c | 40 ++++++++++++++++++-
> > .../comp_perf_test_cyclecount.c | 4 +-
> > .../comp_perf_test_throughput.c | 4 +-
> > .../comp_perf_test_verify.c | 4 +-
> > app/test-compress-perf/main.c | 38 +++++++++++-------
> > doc/guides/rel_notes/release_23_03.rst | 4 ++
> > doc/guides/tools/comp_perf.rst | 2 +
> > 8 files changed, 76 insertions(+), 22 deletions(-)
> >
> > diff --git a/app/test-compress-perf/comp_perf_options.h
> > b/app/test-compress- perf/comp_perf_options.h index
> > d00b299247..5e5227a700 100644
> > --- a/app/test-compress-perf/comp_perf_options.h
> > +++ b/app/test-compress-perf/comp_perf_options.h
> > @@ -63,6 +63,8 @@ struct comp_test_data {
> >
> > enum rte_comp_huffman huffman_enc;
> > enum comp_operation test_op;
> > + enum rte_comp_algorithm test_algo;
> > +
> > int window_sz;
> > struct range_list level_lst;
> > uint8_t level;
> > diff --git a/app/test-compress-perf/comp_perf_options_parse.c
> > b/app/test- compress-perf/comp_perf_options_parse.c
> > index 7a992bf43e..97ddff87ef 100644
> > --- a/app/test-compress-perf/comp_perf_options_parse.c
> > +++ b/app/test-compress-perf/comp_perf_options_parse.c
> > @@ -25,6 +25,7 @@
> > #define CPERF_MAX_SGL_SEGS ("max-num-sgl-segs")
> > #define CPERF_NUM_ITER ("num-iter")
> > #define CPERF_OPTYPE ("operation")
> > +#define CPERF_ALGO ("algo")
> > #define CPERF_HUFFMAN_ENC ("huffman-enc")
> > #define CPERF_LEVEL ("compress-level")
> > #define CPERF_WINDOW_SIZE ("window-sz")
> > @@ -56,6 +57,8 @@ usage(char *progname)
> > " compressed/decompressed (default: 10000)\n"
> > " --operation [comp/decomp/comp_and_decomp]: perform
> > test on\n"
> > " compression, decompression or both
> > operations\n"
> > + " --algo [null/deflate/lzs]: perform test on algorithm\n"
> > + " null(DMA), deflate or lzs (default: deflate)\n"
>
> Since LZ4 is also supported now, is it not good to add that also in this patch?
I agree, I'll merge them in the next version.
>
> > " --huffman-enc [fixed/dynamic/default]: Huffman encoding\n"
> > " (default: dynamic)\n"
> > " --compress-level N: compression level, which could be
> > a single value, list or range\n"
> > @@ -470,6 +473,36 @@ parse_op_type(struct comp_test_data *test_data,
> > const char *arg)
> > return 0;
> > }
> >
> > +static int
> > +parse_algo(struct comp_test_data *test_data, const char *arg) {
> > + struct name_id_map algo_namemap[] = {
> > + {
> > + "null",
> > + RTE_COMP_ALGO_NULL
> > + },
> > + {
> > + "deflate",
> > + RTE_COMP_ALGO_DEFLATE
> > + },
> > + {
> > + "lzs",
> > + RTE_COMP_ALGO_LZS
> > + }
> > + };
> > +
> > + int id = get_str_key_id_mapping(algo_namemap,
> > + RTE_DIM(algo_namemap), arg);
> > + if (id < 0) {
> > + RTE_LOG(ERR, USER1, "Invalid algorithm specified\n");
> > + return -1;
> > + }
> > +
> > + test_data->test_algo = (enum rte_comp_algorithm)id;
> > +
> > + return 0;
> > +}
> > +
> > static int
> > parse_huffman_enc(struct comp_test_data *test_data, const char *arg)
> > { @@ -491,7 +524,7 @@ parse_huffman_enc(struct comp_test_data
> > *test_data, const char *arg)
> > int id = get_str_key_id_mapping(huffman_namemap,
> > RTE_DIM(huffman_namemap), arg);
> > if (id < 0) {
> > - RTE_LOG(ERR, USER1, "Invalid Huffmane encoding
> > specified\n");
> > + RTE_LOG(ERR, USER1, "Invalid Huffman encoding
> > + specified\n");
>
> This change is not related to this patch.
> Should be a separate patch and should be backported.
I'll separate it and the next to different commit.
>
> > return -1;
> > }
> >
> > @@ -507,7 +540,7 @@ parse_level(struct comp_test_data *test_data,
> > const char *arg)
> >
> > /*
> > * Try parsing the argument as a range, if it fails,
> > - * arse it as a list
> > + * parse it as a list
>
> Same here.
>
> > */
> > if (parse_range(arg, &test_data->level_lst.min,
> > &test_data->level_lst.max, @@ -572,6 +605,7 @@
> > static struct option lgopts[] = {
> > { CPERF_MAX_SGL_SEGS, required_argument, 0, 0},
> > { CPERF_NUM_ITER, required_argument, 0, 0 },
> > { CPERF_OPTYPE, required_argument, 0, 0 },
> > + { CPERF_ALGO, required_argument, 0, 0 },
> > { CPERF_HUFFMAN_ENC, required_argument, 0, 0 },
> > { CPERF_LEVEL, required_argument, 0, 0 },
> > { CPERF_WINDOW_SIZE, required_argument, 0, 0 }, @@ -594,6 +628,7
> > @@ comp_perf_opts_parse_long(int opt_idx, struct comp_test_data
> > *test_data)
> > { CPERF_MAX_SGL_SEGS, parse_max_num_sgl_segs },
> > { CPERF_NUM_ITER, parse_num_iter },
> > { CPERF_OPTYPE, parse_op_type },
> > + { CPERF_ALGO, parse_algo },
> > { CPERF_HUFFMAN_ENC, parse_huffman_enc },
> > { CPERF_LEVEL, parse_level },
> > { CPERF_WINDOW_SIZE, parse_window_sz },
> > @@ -649,6 +684,7 @@ comp_perf_options_default(struct comp_test_data
> > *test_data)
> > test_data->num_iter = 10000;
> > test_data->huffman_enc = RTE_COMP_HUFFMAN_DYNAMIC;
> > test_data->test_op = COMPRESS_DECOMPRESS;
> > + test_data->test_algo = RTE_COMP_ALGO_DEFLATE;
> > test_data->window_sz = -1;
> > test_data->level_lst.min = RTE_COMP_LEVEL_MIN;
> > test_data->level_lst.max = RTE_COMP_LEVEL_MAX; diff --git
> > a/app/test-compress-perf/comp_perf_test_cyclecount.c b/app/test-
> > compress-perf/comp_perf_test_cyclecount.c
> > index ce6c4d7605..81c3d30038 100644
> > --- a/app/test-compress-perf/comp_perf_test_cyclecount.c
> > +++ b/app/test-compress-perf/comp_perf_test_cyclecount.c
> > @@ -193,7 +193,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum
> > rte_comp_xform_type type)
> > xform = (struct rte_comp_xform) {
> > .type = RTE_COMP_COMPRESS,
> > .compress = {
> > - .algo = RTE_COMP_ALGO_DEFLATE,
> > + .algo = test_data->test_algo,
> > .deflate.huffman = test_data->huffman_enc,
> > .level = test_data->level,
> > .window_size = test_data->window_sz, @@
> > -208,7 +208,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum
> > rte_comp_xform_type type)
> > xform = (struct rte_comp_xform) {
> > .type = RTE_COMP_DECOMPRESS,
> > .decompress = {
> > - .algo = RTE_COMP_ALGO_DEFLATE,
> > + .algo = test_data->test_algo,
> > .chksum = RTE_COMP_CHECKSUM_NONE,
> > .window_size = test_data->window_sz,
> > .hash_algo = RTE_COMP_HASH_ALGO_NONE
> > diff --git a/app/test-compress-perf/comp_perf_test_throughput.c
> > b/app/test- compress-perf/comp_perf_test_throughput.c
> > index c9f8237626..2545ee9925 100644
> > --- a/app/test-compress-perf/comp_perf_test_throughput.c
> > +++ b/app/test-compress-perf/comp_perf_test_throughput.c
> > @@ -84,7 +84,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum
> > rte_comp_xform_type type)
> > xform = (struct rte_comp_xform) {
> > .type = RTE_COMP_COMPRESS,
> > .compress = {
> > - .algo = RTE_COMP_ALGO_DEFLATE,
> > + .algo = test_data->test_algo,
> > .deflate.huffman = test_data->huffman_enc,
> > .level = test_data->level,
> > .window_size = test_data->window_sz, @@
> > -99,7 +99,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum
> > rte_comp_xform_type type)
> > xform = (struct rte_comp_xform) {
> > .type = RTE_COMP_DECOMPRESS,
> > .decompress = {
> > - .algo = RTE_COMP_ALGO_DEFLATE,
> > + .algo = test_data->test_algo,
> > .chksum = RTE_COMP_CHECKSUM_NONE,
> > .window_size = test_data->window_sz,
> > .hash_algo = RTE_COMP_HASH_ALGO_NONE
> > diff --git a/app/test-compress-perf/comp_perf_test_verify.c
> > b/app/test- compress-perf/comp_perf_test_verify.c
> > index 7d6b6abecd..88f4f41851 100644
> > --- a/app/test-compress-perf/comp_perf_test_verify.c
> > +++ b/app/test-compress-perf/comp_perf_test_verify.c
> > @@ -87,7 +87,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum
> > rte_comp_xform_type type)
> > xform = (struct rte_comp_xform) {
> > .type = RTE_COMP_COMPRESS,
> > .compress = {
> > - .algo = RTE_COMP_ALGO_DEFLATE,
> > + .algo = test_data->test_algo,
> > .deflate.huffman = test_data->huffman_enc,
> > .level = test_data->level,
> > .window_size = test_data->window_sz, @@
> > -104,7 +104,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum
> > rte_comp_xform_type type)
> > xform = (struct rte_comp_xform) {
> > .type = RTE_COMP_DECOMPRESS,
> > .decompress = {
> > - .algo = RTE_COMP_ALGO_DEFLATE,
> > + .algo = test_data->test_algo,
> > .chksum = RTE_COMP_CHECKSUM_NONE,
> > .window_size = test_data->window_sz,
> > .hash_algo = RTE_COMP_HASH_ALGO_NONE
> > diff --git a/app/test-compress-perf/main.c
> > b/app/test-compress-perf/main.c index bbb4c7917b..d049527ba1 100644
> > --- a/app/test-compress-perf/main.c
> > +++ b/app/test-compress-perf/main.c
> > @@ -57,29 +57,39 @@ comp_perf_check_capabilities(struct comp_test_data
> > *test_data, uint8_t cdev_id) {
> > const struct rte_compressdev_capabilities *cap;
> >
> > - cap = rte_compressdev_capability_get(cdev_id,
> > - RTE_COMP_ALGO_DEFLATE);
> > + cap = rte_compressdev_capability_get(cdev_id,
> > + test_data->test_algo);
> >
> > if (cap == NULL) {
> > RTE_LOG(ERR, USER1,
> > - "Compress device does not support DEFLATE\n");
> > + "Compress device does not support %u algorithm\n",
> > + test_data->test_algo);
> > return -1;
> > }
> >
> > uint64_t comp_flags = cap->comp_feature_flags;
> >
> > - /* Huffman encoding */
> > - if (test_data->huffman_enc == RTE_COMP_HUFFMAN_FIXED &&
> > - (comp_flags & RTE_COMP_FF_HUFFMAN_FIXED) == 0)
> > {
> > - RTE_LOG(ERR, USER1,
> > - "Compress device does not supported Fixed
> > Huffman\n");
> > - return -1;
> > - }
> > + /* Algorithm type */
> > + switch (test_data->test_algo) {
> > + case RTE_COMP_ALGO_DEFLATE:
> > + /* Huffman encoding */
> > + if (test_data->huffman_enc == RTE_COMP_HUFFMAN_FIXED
> > &&
> > + (comp_flags & RTE_COMP_FF_HUFFMAN_FIXED) == 0) {
> > + RTE_LOG(ERR, USER1,
> > + "Compress device does not supported
> > + Fixed
> > Huffman\n");
> > + return -1;
> > + }
> >
> > - if (test_data->huffman_enc == RTE_COMP_HUFFMAN_DYNAMIC &&
> > - (comp_flags & RTE_COMP_FF_HUFFMAN_DYNAMIC)
> > == 0) {
> > - RTE_LOG(ERR, USER1,
> > - "Compress device does not supported Dynamic
> > Huffman\n");
> > + if (test_data->huffman_enc ==
> > RTE_COMP_HUFFMAN_DYNAMIC &&
> > + (comp_flags & RTE_COMP_FF_HUFFMAN_DYNAMIC) == 0) {
> > + RTE_LOG(ERR, USER1,
> > + "Compress device does not supported
> > + Dynamic
> > Huffman\n");
> > + return -1;
> > + }
> > + break;
> > + case RTE_COMP_ALGO_LZS:
> > + case RTE_COMP_ALGO_NULL:
> > + break;
> > + default:
> > return -1;
> > }
> >
> > diff --git a/doc/guides/rel_notes/release_23_03.rst
> > b/doc/guides/rel_notes/release_23_03.rst
> > index b2a8c921b9..6aa3f31246 100644
> > --- a/doc/guides/rel_notes/release_23_03.rst
> > +++ b/doc/guides/rel_notes/release_23_03.rst
> > @@ -116,6 +116,10 @@ New Features
> >
> > Enable the application options for testing only compress and only
> decompress.
> >
> > +* **Added algo option in test-compress-perf.**
> > +
> > + Added support for testing other algorithms except for DEFLAT.
> > +
>
> This patch is only adding a new option to the test app.
> I believe release notes are not needed.
Thank you, I'm removing it.
>
>
> >
> > Removed Items
> > -------------
> > diff --git a/doc/guides/tools/comp_perf.rst
> > b/doc/guides/tools/comp_perf.rst index 9d2f4dbe4a..cf9035cc23 100644
> > --- a/doc/guides/tools/comp_perf.rst
> > +++ b/doc/guides/tools/comp_perf.rst
> > @@ -84,6 +84,8 @@ Application Options
> >
> > ``--operation [comp/decomp/comp_and_decomp]``: perform test on
> > compression, decompression or both operations
> >
> > + ``--algo [null/deflate/lzs]`` : perform test on algorithm null(DMA),
> > + Deflate or lzs
> > (default: Deflate)
> > +
> > ``--huffman-enc [fixed/dynamic/default]``: Huffman encoding (default:
> > dynamic)
> >
> > ``--compress-level N``: compression level, which could be a single
> > value, list or range (default: range between 1 and 9)
> > --
> > 2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* RE: [EXT] [PATCH v3 2/4] app/test-compress-perf: allow test single compress operation
2023-02-13 7:23 ` [EXT] " Akhil Goyal
@ 2023-02-14 16:37 ` Michael Baum
0 siblings, 0 replies; 31+ messages in thread
From: Michael Baum @ 2023-02-14 16:37 UTC (permalink / raw)
To: Akhil Goyal, dev
Cc: Matan Azrad, Ashish Gupta, Fiona Trahe,
NBU-Contact-Thomas Monjalon (EXTERNAL)
On Monday, 13 February 2023 9:23, Akhil Goyal wrote:
>
> From the description it seems to be a bug in previous implementation.
> Can we back port it? And update the patch title accordingly?
I'll update and send a new version.
> Also, release note update is not required for this fix.
I'll remove it, thank you.
>
> >
> > Allow testing only compress and only decompress.
> >
> > Signed-off-by: Matan Azrad <matan@nvidia.com>
> > Signed-off-by: Michael Baum <michaelba@nvidia.com>
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v4 0/4] compressdev: add LZ4 support
2023-02-13 6:11 ` [PATCH v3 0/4] compressdev: " Michael Baum
` (4 preceding siblings ...)
2023-02-13 15:47 ` [PATCH v3 0/4] compressdev: " Zhang, Fan
@ 2023-02-14 17:40 ` Michael Baum
2023-02-14 17:40 ` [PATCH v4 1/4] compressdev: add LZ4 algorithm support Michael Baum
` (4 more replies)
5 siblings, 5 replies; 31+ messages in thread
From: Michael Baum @ 2023-02-14 17:40 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe, Thomas Monjalon
Add support for LZ4 compression algorithm for both API and app.
In addition, enable app to test de/comp only. This option already
exists in both app doc and arguments but isn't implemented inside the
application.
LZ4: https://github.com/lz4/lz4
RFC:
https://patchwork.dpdk.org/project/dpdk/patch/20220410182622.8828-1-rzidane@nvidia.com/
v2:
- Add macros for all LZ4 flags in the standard.
- Rebase.
v3:
- Rebase.
v4:
- Change "allow test single" commit to fix and add "Cc: stable@dpdk.org".
- Take all typo fixes into separate commit.
- Squash "add LZ4 algo" commit into "add algo option" commit.
- Remove unnecessary release notes updates.
Michael Baum (4):
compressdev: add LZ4 algorithm support
app/test-compress-perf: fix some typos
app/test-compress-perf: fix missing test single compress op
app/test-compress-perf: add algo option
app/test-compress-perf/comp_perf_options.h | 9 +-
.../comp_perf_options_parse.c | 83 +++++++++++-
.../comp_perf_test_common.c | 126 +++++++++++++-----
.../comp_perf_test_cyclecount.c | 82 +++++++-----
.../comp_perf_test_throughput.c | 75 +++++++----
.../comp_perf_test_verify.c | 75 +++++++----
app/test-compress-perf/main.c | 63 +++++++--
doc/guides/compressdevs/features/default.ini | 7 +
doc/guides/rel_notes/release_23_03.rst | 7 +
doc/guides/tools/comp_perf.rst | 5 +
lib/compressdev/rte_comp.c | 12 ++
lib/compressdev/rte_comp.h | 79 ++++++++++-
12 files changed, 479 insertions(+), 144 deletions(-)
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v4 1/4] compressdev: add LZ4 algorithm support
2023-02-14 17:40 ` [PATCH v4 " Michael Baum
@ 2023-02-14 17:40 ` Michael Baum
2023-02-14 17:40 ` [PATCH v4 2/4] app/test-compress-perf: fix some typos Michael Baum
` (3 subsequent siblings)
4 siblings, 0 replies; 31+ messages in thread
From: Michael Baum @ 2023-02-14 17:40 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe, Thomas Monjalon
Add support for LZ4 algorithm:
- Add Lz4 param structure to XFORM structures.
- Add capabilities flags for LZ4 params.
- Add xxHash-32 checksum and capabilities flag.
Signed-off-by: Michael Baum <michaelba@nvidia.com>
---
doc/guides/compressdevs/features/default.ini | 7 ++
doc/guides/rel_notes/release_23_03.rst | 7 ++
lib/compressdev/rte_comp.c | 12 +++
lib/compressdev/rte_comp.h | 79 +++++++++++++++++++-
4 files changed, 103 insertions(+), 2 deletions(-)
diff --git a/doc/guides/compressdevs/features/default.ini b/doc/guides/compressdevs/features/default.ini
index e1419ee8db..2f178c5efd 100644
--- a/doc/guides/compressdevs/features/default.ini
+++ b/doc/guides/compressdevs/features/default.ini
@@ -20,8 +20,15 @@ OOP SGL In LB Out =
OOP LB In SGL Out =
Deflate =
LZS =
+LZ4 =
Adler32 =
Crc32 =
Adler32&Crc32 =
+xxHash32 =
Fixed =
Dynamic =
+LZ4 Dictionary ID =
+LZ4 Content Checksum =
+LZ4 Content Size =
+LZ4 Block Checksum =
+LZ4 Block Independence =
diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index 7527c6d57f..790c2a5eef 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -105,6 +105,13 @@ New Features
* Added support to capture packets at each graph node with packet metadata and
node name.
+* **Added LZ4 algorithm in Compressdev Library.**
+
+ Added new compression algorithm, including:
+
+ * Added support for ``RTE_COMP_ALGO_LZ4``.
+ * Added support for ``RTE_COMP_CHECKSUM_XXHASH32``.
+
Removed Items
-------------
diff --git a/lib/compressdev/rte_comp.c b/lib/compressdev/rte_comp.c
index 320c6dab92..f060c68557 100644
--- a/lib/compressdev/rte_comp.c
+++ b/lib/compressdev/rte_comp.c
@@ -39,6 +39,18 @@ rte_comp_get_feature_name(uint64_t flag)
return "HUFFMAN_FIXED";
case RTE_COMP_FF_HUFFMAN_DYNAMIC:
return "HUFFMAN_DYNAMIC";
+ case RTE_COMP_FF_XXHASH32_CHECKSUM:
+ return "XXHASH32_CHECKSUM";
+ case RTE_COMP_FF_LZ4_DICT_ID:
+ return "LZ4_DICT_ID";
+ case RTE_COMP_FF_LZ4_CONTENT_WITH_CHECKSUM:
+ return "LZ4_CONTENT_WITH_CHECKSUM";
+ case RTE_COMP_FF_LZ4_CONTENT_SIZE:
+ return "LZ4_CONTENT_SIZE";
+ case RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE:
+ return "LZ4_BLOCK_INDEPENDENCE";
+ case RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM:
+ return "LZ4_BLOCK_WITH_CHECKSUM";
default:
return NULL;
}
diff --git a/lib/compressdev/rte_comp.h b/lib/compressdev/rte_comp.h
index 5bd711fda1..2096fb2407 100644
--- a/lib/compressdev/rte_comp.h
+++ b/lib/compressdev/rte_comp.h
@@ -67,6 +67,18 @@ extern "C" {
/**< Fixed huffman encoding is supported */
#define RTE_COMP_FF_HUFFMAN_DYNAMIC (1ULL << 14)
/**< Dynamic huffman encoding is supported */
+#define RTE_COMP_FF_XXHASH32_CHECKSUM (1ULL << 15)
+/**< xxHash-32 Checksum is supported */
+#define RTE_COMP_FF_LZ4_DICT_ID (1ULL << 16)
+/**< LZ4 dictionary ID is supported */
+#define RTE_COMP_FF_LZ4_CONTENT_WITH_CHECKSUM (1ULL << 17)
+/**< LZ4 content with checksum is supported */
+#define RTE_COMP_FF_LZ4_CONTENT_SIZE (1ULL << 18)
+/**< LZ4 content size is supported */
+#define RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE (1ULL << 19)
+/**< LZ4 block independent is supported */
+#define RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM (1ULL << 20)
+/**< LZ4 block with checksum is supported */
/** Status of comp operation */
enum rte_comp_op_status {
@@ -109,6 +121,10 @@ enum rte_comp_algorithm {
/**< LZS compression algorithm
* https://tools.ietf.org/html/rfc2395
*/
+ RTE_COMP_ALGO_LZ4,
+ /**< LZ4 compression algorithm
+ * https://github.com/lz4/lz4
+ */
};
/** Compression Hash Algorithms */
@@ -147,9 +163,12 @@ enum rte_comp_checksum_type {
/**< Generates both Adler-32 and CRC32 checksums, concatenated.
* CRC32 is in the lower 32bits, Adler-32 in the upper 32 bits.
*/
+ RTE_COMP_CHECKSUM_XXHASH32,
+ /**< Generates a xxHash-32 checksum, as used by lz4.
+ * https://github.com/Cyan4973/xxHash/blob/dev/doc/xxhash_spec.md
+ */
};
-
/** Compression Huffman Type - used by DEFLATE algorithm */
enum rte_comp_huffman {
RTE_COMP_HUFFMAN_DEFAULT,
@@ -206,13 +225,63 @@ enum rte_comp_op_type {
*/
};
-
/** Parameters specific to the deflate algorithm */
struct rte_comp_deflate_params {
enum rte_comp_huffman huffman;
/**< Compression huffman encoding type */
};
+/**
+ * Dictionary ID flag
+ * If this flag is set, a 4-bytes Dict-ID field will be present, after the
+ * descriptor flags and the Content Size.
+ */
+#define RTE_COMP_LZ4_FLAG_DICT_ID (1 << 0)
+
+/**
+ * Content Checksum flag
+ * If this flag is set, a 32-bits content checksum will be appended after the
+ * EndMark.
+ */
+#define RTE_COMP_LZ4_FLAG_CONTENT_CHECKSUM (1 << 2)
+
+/**
+ * Content Size flag
+ * If this flag is set, the uncompressed size of data included within the frame
+ * will be present as an 8 bytes unsigned little-endian value, after the flags.
+ * Content Size usage is optional.
+ */
+#define RTE_COMP_LZ4_FLAG_CONTENT_SIZE (1 << 3)
+
+/**
+ * Block Checksum flag.
+ * If this flag is set, each data block will be followed by a 4-bytes checksum,
+ * calculated by using the xxHash-32 algorithm on the raw (compressed) data
+ * block. The intention is to detect data corruption (storage or transmission
+ * errors) immediately, before decoding. Block checksum usage is optional.
+ */
+#define RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM (1 << 4)
+
+/**
+ * Block Independence flag.
+ * If this flag is set to 1, blocks are independent.
+ * If this flag is set to 0, each block depends on previous ones (up to LZ4
+ * window size, which is 64 KB). In such case, it is necessary to decode all
+ * blocks in sequence.
+ * Block dependency improves compression ratio, especially for small blocks. On
+ * the other hand, it makes random access or multi-threaded decoding impossible.
+ */
+#define RTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE (1 << 5)
+
+/** Parameters specific to the LZ4 algorithm */
+struct rte_comp_lz4_params {
+ uint8_t flags;
+ /**< Compression LZ4 parameter flags.
+ * Based on LZ4 standard flags:
+ * https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor
+ */
+};
+
/** Setup Data for compression */
struct rte_comp_compress_xform {
enum rte_comp_algorithm algo;
@@ -220,6 +289,8 @@ struct rte_comp_compress_xform {
union {
struct rte_comp_deflate_params deflate;
/**< Parameters specific to the deflate algorithm */
+ struct rte_comp_lz4_params lz4;
+ /**< Parameters specific to the LZ4 algorithm */
}; /**< Algorithm specific parameters */
int level;
/**< Compression level */
@@ -249,6 +320,10 @@ struct rte_comp_decompress_xform {
* compressed data. If window size can't be supported by the PMD then
* setup of stream or private_xform should fail.
*/
+ union {
+ struct rte_comp_lz4_params lz4;
+ /**< Parameters specific to the LZ4 algorithm */
+ }; /**< Algorithm specific parameters */
enum rte_comp_hash_algorithm hash_algo;
/**< Hash algorithm to be used with decompress operation. Hash is always
* done on plaintext.
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v4 2/4] app/test-compress-perf: fix some typos
2023-02-14 17:40 ` [PATCH v4 " Michael Baum
2023-02-14 17:40 ` [PATCH v4 1/4] compressdev: add LZ4 algorithm support Michael Baum
@ 2023-02-14 17:40 ` Michael Baum
2023-02-14 17:40 ` [PATCH v4 3/4] app/test-compress-perf: fix missing test single compress op Michael Baum
` (2 subsequent siblings)
4 siblings, 0 replies; 31+ messages in thread
From: Michael Baum @ 2023-02-14 17:40 UTC (permalink / raw)
To: dev
Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe,
Thomas Monjalon, tomaszx.jozwiak, arturx.trybula,
adamx.dybkowski, stable
Fix some typos in comments and prints in this app.
Fixes: e0b6287c035d ("app/compress-perf: add parser")
Fixes: 2695db95a147 ("test/compress: add cycle-count mode to perf tool")
Fixes: 971d89f5116b ("app/compress-perf: prevent output buffer overflow")
Cc: tomaszx.jozwiak@intel.com
Cc: arturx.trybula@intel.com
Cc: adamx.dybkowski@intel.com
Cc: stable@dpdk.org
Signed-off-by: Michael Baum <michaelba@nvidia.com>
---
app/test-compress-perf/comp_perf_options_parse.c | 4 ++--
app/test-compress-perf/comp_perf_test_cyclecount.c | 2 +-
app/test-compress-perf/comp_perf_test_verify.c | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/app/test-compress-perf/comp_perf_options_parse.c b/app/test-compress-perf/comp_perf_options_parse.c
index 019eddb7bd..9b9d4e6554 100644
--- a/app/test-compress-perf/comp_perf_options_parse.c
+++ b/app/test-compress-perf/comp_perf_options_parse.c
@@ -491,7 +491,7 @@ parse_huffman_enc(struct comp_test_data *test_data, const char *arg)
int id = get_str_key_id_mapping(huffman_namemap,
RTE_DIM(huffman_namemap), arg);
if (id < 0) {
- RTE_LOG(ERR, USER1, "Invalid Huffmane encoding specified\n");
+ RTE_LOG(ERR, USER1, "Invalid Huffman encoding specified\n");
return -1;
}
@@ -507,7 +507,7 @@ parse_level(struct comp_test_data *test_data, const char *arg)
/*
* Try parsing the argument as a range, if it fails,
- * arse it as a list
+ * parse it as a list
*/
if (parse_range(arg, &test_data->level_lst.min,
&test_data->level_lst.max,
diff --git a/app/test-compress-perf/comp_perf_test_cyclecount.c b/app/test-compress-perf/comp_perf_test_cyclecount.c
index 7473cb6277..ce457fefbb 100644
--- a/app/test-compress-perf/comp_perf_test_cyclecount.c
+++ b/app/test-compress-perf/comp_perf_test_cyclecount.c
@@ -563,7 +563,7 @@ cperf_cyclecount_test_runner(void *test_ctx)
" [D-e] - decompression enqueue\n"
" [D-d] - decompression dequeue\n"
" - Cycles section: number of cycles per 'op' for the following operations:\n"
- " setup/op - memory allocation, op configuration and memory dealocation\n"
+ " setup/op - memory allocation, op configuration and memory deallocation\n"
" [C-e] - compression enqueue\n"
" [C-d] - compression dequeue\n"
" [D-e] - decompression enqueue\n"
diff --git a/app/test-compress-perf/comp_perf_test_verify.c b/app/test-compress-perf/comp_perf_test_verify.c
index 8964442891..6b61a9194f 100644
--- a/app/test-compress-perf/comp_perf_test_verify.c
+++ b/app/test-compress-perf/comp_perf_test_verify.c
@@ -226,7 +226,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
op->status ==
RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE) {
RTE_LOG(ERR, USER1,
-"Out of space error occurred due to uncompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n");
+"Out of space error occurred due to incompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n");
res = -1;
goto end;
} else if (op->status !=
@@ -311,7 +311,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
op->status ==
RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE) {
RTE_LOG(ERR, USER1,
-"Out of space error occurred due to uncompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n");
+"Out of space error occurred due to incompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n");
res = -1;
goto end;
} else if (op->status !=
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v4 3/4] app/test-compress-perf: fix missing test single compress op
2023-02-14 17:40 ` [PATCH v4 " Michael Baum
2023-02-14 17:40 ` [PATCH v4 1/4] compressdev: add LZ4 algorithm support Michael Baum
2023-02-14 17:40 ` [PATCH v4 2/4] app/test-compress-perf: fix some typos Michael Baum
@ 2023-02-14 17:40 ` Michael Baum
2023-02-14 17:40 ` [PATCH v4 4/4] app/test-compress-perf: add algo option Michael Baum
2023-02-16 8:01 ` [EXT] [PATCH v4 0/4] compressdev: add LZ4 support Akhil Goyal
4 siblings, 0 replies; 31+ messages in thread
From: Michael Baum @ 2023-02-14 17:40 UTC (permalink / raw)
To: dev
Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe,
Thomas Monjalon, tomaszx.jozwiak, stable
Part of the application options is to test only compress and only
decompress but actually the application ignores this user option and
tries to test always both compress and decompress.
Allow testing only compress and only decompress.
Fixes: e0b6287c035d ("app/compress-perf: add parser")
Cc: tomaszx.jozwiak@intel.com
Cc: stable@dpdk.org
Signed-off-by: Matan Azrad <matan@nvidia.com>
Signed-off-by: Michael Baum <michaelba@nvidia.com>
---
app/test-compress-perf/comp_perf_options.h | 6 +-
.../comp_perf_options_parse.c | 4 +-
.../comp_perf_test_common.c | 126 +++++++++++++-----
.../comp_perf_test_cyclecount.c | 69 ++++++----
.../comp_perf_test_throughput.c | 64 +++++----
.../comp_perf_test_verify.c | 60 ++++++---
app/test-compress-perf/main.c | 8 ++
7 files changed, 223 insertions(+), 114 deletions(-)
diff --git a/app/test-compress-perf/comp_perf_options.h b/app/test-compress-perf/comp_perf_options.h
index 57dd146330..d00b299247 100644
--- a/app/test-compress-perf/comp_perf_options.h
+++ b/app/test-compress-perf/comp_perf_options.h
@@ -32,9 +32,9 @@ enum cperf_test_type {
};
enum comp_operation {
- COMPRESS_ONLY,
- DECOMPRESS_ONLY,
- COMPRESS_DECOMPRESS
+ COMPRESS = (1 << 0),
+ DECOMPRESS = (1 << 1),
+ COMPRESS_DECOMPRESS = (COMPRESS | DECOMPRESS),
};
struct range_list {
diff --git a/app/test-compress-perf/comp_perf_options_parse.c b/app/test-compress-perf/comp_perf_options_parse.c
index 9b9d4e6554..303e714cda 100644
--- a/app/test-compress-perf/comp_perf_options_parse.c
+++ b/app/test-compress-perf/comp_perf_options_parse.c
@@ -446,11 +446,11 @@ parse_op_type(struct comp_test_data *test_data, const char *arg)
struct name_id_map optype_namemap[] = {
{
"comp",
- COMPRESS_ONLY
+ COMPRESS
},
{
"decomp",
- DECOMPRESS_ONLY
+ DECOMPRESS
},
{
"comp_and_decomp",
diff --git a/app/test-compress-perf/comp_perf_test_common.c b/app/test-compress-perf/comp_perf_test_common.c
index b402a0d839..cd60958944 100644
--- a/app/test-compress-perf/comp_perf_test_common.c
+++ b/app/test-compress-perf/comp_perf_test_common.c
@@ -227,23 +227,43 @@ comp_perf_allocate_memory(struct comp_test_data *test_data,
{
uint16_t comp_mbuf_size;
uint16_t decomp_mbuf_size;
+ size_t comp_data_size;
+ size_t decomp_data_size;
+ size_t output_data_sz;
test_data->out_seg_sz = find_buf_size(test_data->seg_sz);
- /* Number of segments for input and output
- * (compression and decompression)
- */
- test_data->total_segs = DIV_CEIL(test_data->input_data_sz,
- test_data->seg_sz);
+ if (test_data->test_op & COMPRESS) {
+ /*
+ * Number of segments for input and output
+ * (compression and decompression)
+ */
+ test_data->total_segs = DIV_CEIL(test_data->input_data_sz,
+ test_data->seg_sz);
+ } else {
+ /*
+ * When application does decompression only, input data is
+ * compressed and smaller than the output. The expected size of
+ * uncompressed data given by the user in segment size argument.
+ */
+ test_data->total_segs = test_data->max_sgl_segs;
+ }
+
+ output_data_sz = (size_t) test_data->out_seg_sz * test_data->total_segs;
+ output_data_sz =
+ RTE_MAX(output_data_sz, (size_t) MIN_COMPRESSED_BUF_SIZE);
if (test_data->use_external_mbufs != 0) {
if (comp_perf_allocate_external_mbufs(test_data, mem) < 0)
return -1;
comp_mbuf_size = 0;
decomp_mbuf_size = 0;
- } else {
+ } else if (test_data->test_op & COMPRESS) {
comp_mbuf_size = test_data->out_seg_sz + RTE_PKTMBUF_HEADROOM;
decomp_mbuf_size = test_data->seg_sz + RTE_PKTMBUF_HEADROOM;
+ } else {
+ comp_mbuf_size = test_data->seg_sz + RTE_PKTMBUF_HEADROOM;
+ decomp_mbuf_size = test_data->out_seg_sz + RTE_PKTMBUF_HEADROOM;
}
char pool_name[32] = "";
@@ -287,26 +307,28 @@ comp_perf_allocate_memory(struct comp_test_data *test_data,
return -1;
}
- /*
- * Compressed data might be a bit larger than input data,
- * if data cannot be compressed
- */
- mem->compressed_data = rte_zmalloc_socket(NULL,
- RTE_MAX(
- (size_t) test_data->out_seg_sz *
- test_data->total_segs,
- (size_t) MIN_COMPRESSED_BUF_SIZE),
- 0,
- rte_socket_id());
+ if (test_data->test_op & COMPRESS) {
+ /*
+ * Compressed data might be a bit larger than input data,
+ * if data cannot be compressed
+ */
+ comp_data_size = output_data_sz;
+ decomp_data_size = test_data->input_data_sz;
+ } else {
+ comp_data_size = test_data->input_data_sz;
+ decomp_data_size = output_data_sz;
+ }
+
+ mem->compressed_data = rte_zmalloc_socket(NULL, comp_data_size, 0,
+ rte_socket_id());
if (mem->compressed_data == NULL) {
RTE_LOG(ERR, USER1, "Memory to hold the data from the input "
"file could not be allocated\n");
return -1;
}
- mem->decompressed_data = rte_zmalloc_socket(NULL,
- test_data->input_data_sz, 0,
- rte_socket_id());
+ mem->decompressed_data = rte_zmalloc_socket(NULL, decomp_data_size, 0,
+ rte_socket_id());
if (mem->decompressed_data == NULL) {
RTE_LOG(ERR, USER1, "Memory to hold the data from the input "
"file could not be allocated\n");
@@ -351,6 +373,7 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
uint16_t segs_per_mbuf = 0;
uint32_t cmz = 0;
uint32_t dmz = 0;
+ bool decompress_only = !!(test_data->test_op == DECOMPRESS);
for (i = 0; i < mem->total_bufs; i++) {
/* Allocate data in input mbuf and copy data from input file */
@@ -361,8 +384,6 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
return -1;
}
- data_sz = RTE_MIN(remaining_data, test_data->seg_sz);
-
if (test_data->use_external_mbufs != 0) {
rte_pktmbuf_attach_extbuf(mem->decomp_bufs[i],
mem->decomp_memzones[dmz]->addr,
@@ -372,16 +393,23 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
dmz++;
}
+ if (!decompress_only)
+ data_sz = RTE_MIN(remaining_data, test_data->seg_sz);
+ else
+ data_sz = test_data->out_seg_sz;
+
data_addr = (uint8_t *) rte_pktmbuf_append(
mem->decomp_bufs[i], data_sz);
if (data_addr == NULL) {
RTE_LOG(ERR, USER1, "Could not append data\n");
return -1;
}
- rte_memcpy(data_addr, input_data_ptr, data_sz);
- input_data_ptr += data_sz;
- remaining_data -= data_sz;
+ if (!decompress_only) {
+ rte_memcpy(data_addr, input_data_ptr, data_sz);
+ input_data_ptr += data_sz;
+ remaining_data -= data_sz;
+ }
/* Already one segment in the mbuf */
segs_per_mbuf = 1;
@@ -398,8 +426,6 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
return -1;
}
- data_sz = RTE_MIN(remaining_data, test_data->seg_sz);
-
if (test_data->use_external_mbufs != 0) {
rte_pktmbuf_attach_extbuf(
next_seg,
@@ -410,6 +436,12 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
dmz++;
}
+ if (!decompress_only)
+ data_sz = RTE_MIN(remaining_data,
+ test_data->seg_sz);
+ else
+ data_sz = test_data->out_seg_sz;
+
data_addr = (uint8_t *)rte_pktmbuf_append(next_seg,
data_sz);
@@ -418,9 +450,11 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
return -1;
}
- rte_memcpy(data_addr, input_data_ptr, data_sz);
- input_data_ptr += data_sz;
- remaining_data -= data_sz;
+ if (!decompress_only) {
+ rte_memcpy(data_addr, input_data_ptr, data_sz);
+ input_data_ptr += data_sz;
+ remaining_data -= data_sz;
+ }
if (rte_pktmbuf_chain(mem->decomp_bufs[i],
next_seg) < 0) {
@@ -447,16 +481,26 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
cmz++;
}
- data_addr = (uint8_t *) rte_pktmbuf_append(
- mem->comp_bufs[i],
- test_data->out_seg_sz);
+ if (decompress_only)
+ data_sz = RTE_MIN(remaining_data, test_data->seg_sz);
+ else
+ data_sz = test_data->out_seg_sz;
+
+ data_addr = (uint8_t *) rte_pktmbuf_append(mem->comp_bufs[i],
+ data_sz);
if (data_addr == NULL) {
RTE_LOG(ERR, USER1, "Could not append data\n");
return -1;
}
+ if (decompress_only) {
+ rte_memcpy(data_addr, input_data_ptr, data_sz);
+ input_data_ptr += data_sz;
+ remaining_data -= data_sz;
+ }
+
/* Chain mbufs if needed for output mbufs */
- for (j = 1; j < segs_per_mbuf; j++) {
+ for (j = 1; j < segs_per_mbuf && remaining_data > 0; j++) {
struct rte_mbuf *next_seg =
rte_pktmbuf_alloc(mem->comp_buf_pool);
@@ -476,13 +520,25 @@ prepare_bufs(struct comp_test_data *test_data, struct cperf_mem_resources *mem)
cmz++;
}
+ if (decompress_only)
+ data_sz = RTE_MIN(remaining_data,
+ test_data->seg_sz);
+ else
+ data_sz = test_data->out_seg_sz;
+
data_addr = (uint8_t *)rte_pktmbuf_append(next_seg,
- test_data->out_seg_sz);
+ data_sz);
if (data_addr == NULL) {
RTE_LOG(ERR, USER1, "Could not append data\n");
return -1;
}
+ if (decompress_only) {
+ rte_memcpy(data_addr, input_data_ptr, data_sz);
+ input_data_ptr += data_sz;
+ remaining_data -= data_sz;
+ }
+
if (rte_pktmbuf_chain(mem->comp_bufs[i],
next_seg) < 0) {
RTE_LOG(ERR, USER1, "Could not chain mbufs\n");
diff --git a/app/test-compress-perf/comp_perf_test_cyclecount.c b/app/test-compress-perf/comp_perf_test_cyclecount.c
index ce457fefbb..ce6c4d7605 100644
--- a/app/test-compress-perf/comp_perf_test_cyclecount.c
+++ b/app/test-compress-perf/comp_perf_test_cyclecount.c
@@ -514,38 +514,55 @@ cperf_cyclecount_test_runner(void *test_ctx)
if (cperf_verify_test_runner(&ctx->ver))
return EXIT_FAILURE;
- /*
- * Run the tests twice, discarding the first performance
- * results, before the cache is warmed up
- */
-
- /* C O M P R E S S */
- for (i = 0; i < 2; i++) {
- if (main_loop(ctx, RTE_COMP_COMPRESS) < 0)
- return EXIT_FAILURE;
- }
+ if (test_data->test_op & COMPRESS) {
+ /*
+ * Run the test twice, discarding the first performance
+ * results, before the cache is warmed up
+ */
+ for (i = 0; i < 2; i++) {
+ if (main_loop(ctx, RTE_COMP_COMPRESS) < 0)
+ return EXIT_FAILURE;
+ }
- ops_enq_retries_comp = ctx->ops_enq_retries;
- ops_deq_retries_comp = ctx->ops_deq_retries;
+ ops_enq_retries_comp = ctx->ops_enq_retries;
+ ops_deq_retries_comp = ctx->ops_deq_retries;
- duration_enq_per_op_comp = ctx->duration_enq /
- (ctx->ver.mem.total_bufs * test_data->num_iter);
- duration_deq_per_op_comp = ctx->duration_deq /
- (ctx->ver.mem.total_bufs * test_data->num_iter);
+ duration_enq_per_op_comp = ctx->duration_enq /
+ (ctx->ver.mem.total_bufs * test_data->num_iter);
+ duration_deq_per_op_comp = ctx->duration_deq /
+ (ctx->ver.mem.total_bufs * test_data->num_iter);
+ } else {
+ ops_enq_retries_comp = 0;
+ ops_deq_retries_comp = 0;
- /* D E C O M P R E S S */
- for (i = 0; i < 2; i++) {
- if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0)
- return EXIT_FAILURE;
+ duration_enq_per_op_comp = 0;
+ duration_deq_per_op_comp = 0;
}
- ops_enq_retries_decomp = ctx->ops_enq_retries;
- ops_deq_retries_decomp = ctx->ops_deq_retries;
+ if (test_data->test_op & DECOMPRESS) {
+ /*
+ * Run the test twice, discarding the first performance
+ * results, before the cache is warmed up
+ */
+ for (i = 0; i < 2; i++) {
+ if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0)
+ return EXIT_FAILURE;
+ }
- duration_enq_per_op_decomp = ctx->duration_enq /
- (ctx->ver.mem.total_bufs * test_data->num_iter);
- duration_deq_per_op_decomp = ctx->duration_deq /
- (ctx->ver.mem.total_bufs * test_data->num_iter);
+ ops_enq_retries_decomp = ctx->ops_enq_retries;
+ ops_deq_retries_decomp = ctx->ops_deq_retries;
+
+ duration_enq_per_op_decomp = ctx->duration_enq /
+ (ctx->ver.mem.total_bufs * test_data->num_iter);
+ duration_deq_per_op_decomp = ctx->duration_deq /
+ (ctx->ver.mem.total_bufs * test_data->num_iter);
+ } else {
+ ops_enq_retries_decomp = 0;
+ ops_deq_retries_decomp = 0;
+
+ duration_enq_per_op_decomp = 0;
+ duration_deq_per_op_decomp = 0;
+ }
duration_setup_per_op = ctx->duration_op /
(ctx->ver.mem.total_bufs * test_data->num_iter);
diff --git a/app/test-compress-perf/comp_perf_test_throughput.c b/app/test-compress-perf/comp_perf_test_throughput.c
index 79cd2b2bf2..c9f8237626 100644
--- a/app/test-compress-perf/comp_perf_test_throughput.c
+++ b/app/test-compress-perf/comp_perf_test_throughput.c
@@ -359,41 +359,53 @@ cperf_throughput_test_runner(void *test_ctx)
* First the verification part is needed
*/
if (cperf_verify_test_runner(&ctx->ver)) {
- ret = EXIT_FAILURE;
+ ret = EXIT_FAILURE;
goto end;
}
- /*
- * Run the tests twice, discarding the first performance
- * results, before the cache is warmed up
- */
- for (i = 0; i < 2; i++) {
- if (main_loop(ctx, RTE_COMP_COMPRESS) < 0) {
- ret = EXIT_FAILURE;
- goto end;
+ if (test_data->test_op & COMPRESS) {
+ /*
+ * Run the test twice, discarding the first performance
+ * results, before the cache is warmed up
+ */
+ for (i = 0; i < 2; i++) {
+ if (main_loop(ctx, RTE_COMP_COMPRESS) < 0) {
+ ret = EXIT_FAILURE;
+ goto end;
+ }
}
- }
- for (i = 0; i < 2; i++) {
- if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0) {
- ret = EXIT_FAILURE;
- goto end;
- }
+ ctx->comp_tsc_byte =
+ (double)(ctx->comp_tsc_duration[test_data->level]) /
+ test_data->input_data_sz;
+ ctx->comp_gbps = rte_get_tsc_hz() / ctx->comp_tsc_byte * 8 /
+ 1000000000;
+ } else {
+ ctx->comp_tsc_byte = 0;
+ ctx->comp_gbps = 0;
}
- ctx->comp_tsc_byte =
- (double)(ctx->comp_tsc_duration[test_data->level]) /
- test_data->input_data_sz;
+ if (test_data->test_op & DECOMPRESS) {
+ /*
+ * Run the test twice, discarding the first performance
+ * results, before the cache is warmed up
+ */
+ for (i = 0; i < 2; i++) {
+ if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0) {
+ ret = EXIT_FAILURE;
+ goto end;
+ }
+ }
- ctx->decomp_tsc_byte =
+ ctx->decomp_tsc_byte =
(double)(ctx->decomp_tsc_duration[test_data->level]) /
- test_data->input_data_sz;
-
- ctx->comp_gbps = rte_get_tsc_hz() / ctx->comp_tsc_byte * 8 /
- 1000000000;
-
- ctx->decomp_gbps = rte_get_tsc_hz() / ctx->decomp_tsc_byte * 8 /
- 1000000000;
+ test_data->input_data_sz;
+ ctx->decomp_gbps = rte_get_tsc_hz() / ctx->decomp_tsc_byte * 8 /
+ 1000000000;
+ } else {
+ ctx->decomp_tsc_byte = 0;
+ ctx->decomp_gbps = 0;
+ }
exp = 0;
if (__atomic_compare_exchange_n(&display_once, &exp, 1, 0,
diff --git a/app/test-compress-perf/comp_perf_test_verify.c b/app/test-compress-perf/comp_perf_test_verify.c
index 6b61a9194f..7d6b6abecd 100644
--- a/app/test-compress-perf/comp_perf_test_verify.c
+++ b/app/test-compress-perf/comp_perf_test_verify.c
@@ -114,7 +114,8 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
output_data_sz = &ctx->decomp_data_sz;
input_bufs = mem->comp_bufs;
output_bufs = mem->decomp_bufs;
- out_seg_sz = test_data->seg_sz;
+ out_seg_sz = (test_data->test_op & COMPRESS) ?
+ test_data->seg_sz : test_data->out_seg_sz;
}
/* Create private xform */
@@ -392,44 +393,59 @@ cperf_verify_test_runner(void *test_ctx)
int ret = EXIT_SUCCESS;
static uint16_t display_once;
uint32_t lcore = rte_lcore_id();
+ uint16_t exp = 0;
ctx->mem.lcore_id = lcore;
test_data->ratio = 0;
- if (main_loop(ctx, RTE_COMP_COMPRESS) < 0) {
- ret = EXIT_FAILURE;
- goto end;
+ if (test_data->test_op & COMPRESS) {
+ if (main_loop(ctx, RTE_COMP_COMPRESS) < 0) {
+ ret = EXIT_FAILURE;
+ goto end;
+ }
}
- if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0) {
- ret = EXIT_FAILURE;
- goto end;
- }
+ if (test_data->test_op & DECOMPRESS) {
+ if (main_loop(ctx, RTE_COMP_DECOMPRESS) < 0) {
+ ret = EXIT_FAILURE;
+ goto end;
+ }
- if (ctx->decomp_data_sz != test_data->input_data_sz) {
- RTE_LOG(ERR, USER1,
- "Decompressed data length not equal to input data length\n");
- RTE_LOG(ERR, USER1,
- "Decompressed size = %zu, expected = %zu\n",
- ctx->decomp_data_sz, test_data->input_data_sz);
- ret = EXIT_FAILURE;
- goto end;
- } else {
- if (memcmp(ctx->mem.decompressed_data,
- test_data->input_data,
- test_data->input_data_sz) != 0) {
+ if (!(test_data->test_op & COMPRESS)) {
+ /*
+ * For DECOMPRESS_ONLY mode there is no more
+ * verifications, reset the 'ratio' and 'comp_data_sz'
+ * fields for other tests report.
+ */
+ ctx->comp_data_sz = 0;
+ ctx->ratio = 0;
+ goto end;
+ }
+
+ if (ctx->decomp_data_sz != test_data->input_data_sz) {
+ RTE_LOG(ERR, USER1,
+ "Decompressed data length not equal to input data length\n");
RTE_LOG(ERR, USER1,
- "Decompressed data is not the same as file data\n");
+ "Decompressed size = %zu, expected = %zu\n",
+ ctx->decomp_data_sz, test_data->input_data_sz);
ret = EXIT_FAILURE;
goto end;
+ } else {
+ if (memcmp(ctx->mem.decompressed_data,
+ test_data->input_data,
+ test_data->input_data_sz) != 0) {
+ RTE_LOG(ERR, USER1,
+ "Decompressed data is not the same as file data\n");
+ ret = EXIT_FAILURE;
+ goto end;
+ }
}
}
ctx->ratio = (double) ctx->comp_data_sz /
test_data->input_data_sz * 100;
- uint16_t exp = 0;
if (!ctx->silent) {
if (__atomic_compare_exchange_n(&display_once, &exp, 1, 0,
__ATOMIC_RELAXED, __ATOMIC_RELAXED)) {
diff --git a/app/test-compress-perf/main.c b/app/test-compress-perf/main.c
index 41b8edc2bd..bbb4c7917b 100644
--- a/app/test-compress-perf/main.c
+++ b/app/test-compress-perf/main.c
@@ -254,6 +254,14 @@ comp_perf_dump_input_data(struct comp_test_data *test_data)
goto end;
}
+ if (!(test_data->test_op & COMPRESS) &&
+ test_data->input_data_sz >
+ (size_t) test_data->seg_sz * (size_t) test_data->max_sgl_segs) {
+ RTE_LOG(ERR, USER1,
+ "Size of input must be less than total segments\n");
+ goto end;
+ }
+
test_data->input_data = rte_zmalloc_socket(NULL,
test_data->input_data_sz, 0, rte_socket_id());
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v4 4/4] app/test-compress-perf: add algo option
2023-02-14 17:40 ` [PATCH v4 " Michael Baum
` (2 preceding siblings ...)
2023-02-14 17:40 ` [PATCH v4 3/4] app/test-compress-perf: fix missing test single compress op Michael Baum
@ 2023-02-14 17:40 ` Michael Baum
2023-02-16 8:01 ` [EXT] [PATCH v4 0/4] compressdev: add LZ4 support Akhil Goyal
4 siblings, 0 replies; 31+ messages in thread
From: Michael Baum @ 2023-02-14 17:40 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Akhil Goyal, Ashish Gupta, Fiona Trahe, Thomas Monjalon
Add a command line option `algo` to select the compress algorithm
supported by the compress API: null (DMA), deflate, lz4 and lzs.
Default for deflate.
Signed-off-by: Matan Azrad <matan@nvidia.com>
Signed-off-by: Michael Baum <michaelba@nvidia.com>
---
app/test-compress-perf/comp_perf_options.h | 3 +
.../comp_perf_options_parse.c | 75 +++++++++++++++++++
.../comp_perf_test_cyclecount.c | 11 ++-
.../comp_perf_test_throughput.c | 11 ++-
.../comp_perf_test_verify.c | 11 ++-
app/test-compress-perf/main.c | 55 ++++++++++----
doc/guides/tools/comp_perf.rst | 5 ++
7 files changed, 148 insertions(+), 23 deletions(-)
diff --git a/app/test-compress-perf/comp_perf_options.h b/app/test-compress-perf/comp_perf_options.h
index d00b299247..828a7309d8 100644
--- a/app/test-compress-perf/comp_perf_options.h
+++ b/app/test-compress-perf/comp_perf_options.h
@@ -61,8 +61,11 @@ struct comp_test_data {
uint16_t max_sgl_segs;
uint32_t total_segs;
+ uint8_t lz4_flags;
enum rte_comp_huffman huffman_enc;
enum comp_operation test_op;
+ enum rte_comp_algorithm test_algo;
+
int window_sz;
struct range_list level_lst;
uint8_t level;
diff --git a/app/test-compress-perf/comp_perf_options_parse.c b/app/test-compress-perf/comp_perf_options_parse.c
index 303e714cda..6d8c370fc2 100644
--- a/app/test-compress-perf/comp_perf_options_parse.c
+++ b/app/test-compress-perf/comp_perf_options_parse.c
@@ -25,7 +25,9 @@
#define CPERF_MAX_SGL_SEGS ("max-num-sgl-segs")
#define CPERF_NUM_ITER ("num-iter")
#define CPERF_OPTYPE ("operation")
+#define CPERF_ALGO ("algo")
#define CPERF_HUFFMAN_ENC ("huffman-enc")
+#define CPERF_LZ4_FLAGS ("lz4-flags")
#define CPERF_LEVEL ("compress-level")
#define CPERF_WINDOW_SIZE ("window-sz")
#define CPERF_EXTERNAL_MBUFS ("external-mbufs")
@@ -56,8 +58,11 @@ usage(char *progname)
" compressed/decompressed (default: 10000)\n"
" --operation [comp/decomp/comp_and_decomp]: perform test on\n"
" compression, decompression or both operations\n"
+ " --algo [null/deflate/lzs/lz4]: perform test on algorithm\n"
+ " null(DMA), deflate, lzs or lz4 (default: deflate)\n"
" --huffman-enc [fixed/dynamic/default]: Huffman encoding\n"
" (default: dynamic)\n"
+ " --lz4-flags N: flags to configure LZ4 algorithm (default: 0)\n"
" --compress-level N: compression level, which could be a single value, list or range\n"
" (default: range between 1 and 9)\n"
" --window-sz N: base two log value of compression window size\n"
@@ -150,6 +155,23 @@ parse_uint16_t(uint16_t *value, const char *arg)
return 0;
}
+static int
+parse_uint8_t(uint8_t *value, const char *arg)
+{
+ uint32_t val = 0;
+ int ret = parse_uint32_t(&val, arg);
+
+ if (ret < 0)
+ return ret;
+
+ if (val > UINT8_MAX)
+ return -ERANGE;
+
+ *value = (uint8_t) val;
+
+ return 0;
+}
+
static int
parse_range(const char *arg, uint8_t *min, uint8_t *max, uint8_t *inc)
{
@@ -470,6 +492,40 @@ parse_op_type(struct comp_test_data *test_data, const char *arg)
return 0;
}
+static int
+parse_algo(struct comp_test_data *test_data, const char *arg)
+{
+ struct name_id_map algo_namemap[] = {
+ {
+ "null",
+ RTE_COMP_ALGO_NULL
+ },
+ {
+ "deflate",
+ RTE_COMP_ALGO_DEFLATE
+ },
+ {
+ "lzs",
+ RTE_COMP_ALGO_LZS
+ },
+ {
+ "lz4",
+ RTE_COMP_ALGO_LZ4
+ }
+ };
+
+ int id = get_str_key_id_mapping(algo_namemap,
+ RTE_DIM(algo_namemap), arg);
+ if (id < 0) {
+ RTE_LOG(ERR, USER1, "Invalid algorithm specified\n");
+ return -1;
+ }
+
+ test_data->test_algo = (enum rte_comp_algorithm)id;
+
+ return 0;
+}
+
static int
parse_huffman_enc(struct comp_test_data *test_data, const char *arg)
{
@@ -500,6 +556,19 @@ parse_huffman_enc(struct comp_test_data *test_data, const char *arg)
return 0;
}
+static int
+parse_lz4_flags(struct comp_test_data *test_data, const char *arg)
+{
+ int ret = parse_uint8_t(&test_data->lz4_flags, arg);
+
+ if (ret) {
+ RTE_LOG(ERR, USER1, "Failed to parse LZ4 flags\n");
+ return -1;
+ }
+
+ return 0;
+}
+
static int
parse_level(struct comp_test_data *test_data, const char *arg)
{
@@ -572,7 +641,9 @@ static struct option lgopts[] = {
{ CPERF_MAX_SGL_SEGS, required_argument, 0, 0},
{ CPERF_NUM_ITER, required_argument, 0, 0 },
{ CPERF_OPTYPE, required_argument, 0, 0 },
+ { CPERF_ALGO, required_argument, 0, 0 },
{ CPERF_HUFFMAN_ENC, required_argument, 0, 0 },
+ { CPERF_LZ4_FLAGS, required_argument, 0, 0 },
{ CPERF_LEVEL, required_argument, 0, 0 },
{ CPERF_WINDOW_SIZE, required_argument, 0, 0 },
{ CPERF_EXTERNAL_MBUFS, 0, 0, 0 },
@@ -594,7 +665,9 @@ comp_perf_opts_parse_long(int opt_idx, struct comp_test_data *test_data)
{ CPERF_MAX_SGL_SEGS, parse_max_num_sgl_segs },
{ CPERF_NUM_ITER, parse_num_iter },
{ CPERF_OPTYPE, parse_op_type },
+ { CPERF_ALGO, parse_algo },
{ CPERF_HUFFMAN_ENC, parse_huffman_enc },
+ { CPERF_LZ4_FLAGS, parse_lz4_flags },
{ CPERF_LEVEL, parse_level },
{ CPERF_WINDOW_SIZE, parse_window_sz },
{ CPERF_EXTERNAL_MBUFS, parse_external_mbufs },
@@ -647,8 +720,10 @@ comp_perf_options_default(struct comp_test_data *test_data)
test_data->pool_sz = 8192;
test_data->max_sgl_segs = 16;
test_data->num_iter = 10000;
+ test_data->lz4_flags = 0;
test_data->huffman_enc = RTE_COMP_HUFFMAN_DYNAMIC;
test_data->test_op = COMPRESS_DECOMPRESS;
+ test_data->test_algo = RTE_COMP_ALGO_DEFLATE;
test_data->window_sz = -1;
test_data->level_lst.min = RTE_COMP_LEVEL_MIN;
test_data->level_lst.max = RTE_COMP_LEVEL_MAX;
diff --git a/app/test-compress-perf/comp_perf_test_cyclecount.c b/app/test-compress-perf/comp_perf_test_cyclecount.c
index ce6c4d7605..4d336ec8d6 100644
--- a/app/test-compress-perf/comp_perf_test_cyclecount.c
+++ b/app/test-compress-perf/comp_perf_test_cyclecount.c
@@ -193,14 +193,17 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_COMPRESS,
.compress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
- .deflate.huffman = test_data->huffman_enc,
+ .algo = test_data->test_algo,
.level = test_data->level,
.window_size = test_data->window_sz,
.chksum = RTE_COMP_CHECKSUM_NONE,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_DEFLATE)
+ xform.compress.deflate.huffman = test_data->huffman_enc;
+ else if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.compress.lz4.flags = test_data->lz4_flags;
input_bufs = mem->decomp_bufs;
output_bufs = mem->comp_bufs;
out_seg_sz = test_data->out_seg_sz;
@@ -208,12 +211,14 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_DECOMPRESS,
.decompress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.chksum = RTE_COMP_CHECKSUM_NONE,
.window_size = test_data->window_sz,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.decompress.lz4.flags = test_data->lz4_flags;
input_bufs = mem->comp_bufs;
output_bufs = mem->decomp_bufs;
out_seg_sz = test_data->seg_sz;
diff --git a/app/test-compress-perf/comp_perf_test_throughput.c b/app/test-compress-perf/comp_perf_test_throughput.c
index c9f8237626..1f7072d223 100644
--- a/app/test-compress-perf/comp_perf_test_throughput.c
+++ b/app/test-compress-perf/comp_perf_test_throughput.c
@@ -84,14 +84,17 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_COMPRESS,
.compress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
- .deflate.huffman = test_data->huffman_enc,
+ .algo = test_data->test_algo,
.level = test_data->level,
.window_size = test_data->window_sz,
.chksum = RTE_COMP_CHECKSUM_NONE,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_DEFLATE)
+ xform.compress.deflate.huffman = test_data->huffman_enc;
+ else if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.compress.lz4.flags = test_data->lz4_flags;
input_bufs = mem->decomp_bufs;
output_bufs = mem->comp_bufs;
out_seg_sz = test_data->out_seg_sz;
@@ -99,12 +102,14 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_DECOMPRESS,
.decompress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.chksum = RTE_COMP_CHECKSUM_NONE,
.window_size = test_data->window_sz,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.decompress.lz4.flags = test_data->lz4_flags;
input_bufs = mem->comp_bufs;
output_bufs = mem->decomp_bufs;
out_seg_sz = test_data->seg_sz;
diff --git a/app/test-compress-perf/comp_perf_test_verify.c b/app/test-compress-perf/comp_perf_test_verify.c
index 7d6b6abecd..7bd18073cf 100644
--- a/app/test-compress-perf/comp_perf_test_verify.c
+++ b/app/test-compress-perf/comp_perf_test_verify.c
@@ -87,14 +87,17 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_COMPRESS,
.compress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
- .deflate.huffman = test_data->huffman_enc,
+ .algo = test_data->test_algo,
.level = test_data->level,
.window_size = test_data->window_sz,
.chksum = RTE_COMP_CHECKSUM_NONE,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_DEFLATE)
+ xform.compress.deflate.huffman = test_data->huffman_enc;
+ else if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.compress.lz4.flags = test_data->lz4_flags;
output_data_ptr = ctx->mem.compressed_data;
output_data_sz = &ctx->comp_data_sz;
input_bufs = mem->decomp_bufs;
@@ -104,12 +107,14 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type)
xform = (struct rte_comp_xform) {
.type = RTE_COMP_DECOMPRESS,
.decompress = {
- .algo = RTE_COMP_ALGO_DEFLATE,
+ .algo = test_data->test_algo,
.chksum = RTE_COMP_CHECKSUM_NONE,
.window_size = test_data->window_sz,
.hash_algo = RTE_COMP_HASH_ALGO_NONE
}
};
+ if (test_data->test_algo == RTE_COMP_ALGO_LZ4)
+ xform.decompress.lz4.flags = test_data->lz4_flags;
output_data_ptr = ctx->mem.decompressed_data;
output_data_sz = &ctx->decomp_data_sz;
input_bufs = mem->comp_bufs;
diff --git a/app/test-compress-perf/main.c b/app/test-compress-perf/main.c
index bbb4c7917b..fa366123ed 100644
--- a/app/test-compress-perf/main.c
+++ b/app/test-compress-perf/main.c
@@ -57,29 +57,56 @@ comp_perf_check_capabilities(struct comp_test_data *test_data, uint8_t cdev_id)
{
const struct rte_compressdev_capabilities *cap;
- cap = rte_compressdev_capability_get(cdev_id,
- RTE_COMP_ALGO_DEFLATE);
+ cap = rte_compressdev_capability_get(cdev_id, test_data->test_algo);
if (cap == NULL) {
RTE_LOG(ERR, USER1,
- "Compress device does not support DEFLATE\n");
+ "Compress device does not support %u algorithm\n",
+ test_data->test_algo);
return -1;
}
uint64_t comp_flags = cap->comp_feature_flags;
- /* Huffman encoding */
- if (test_data->huffman_enc == RTE_COMP_HUFFMAN_FIXED &&
- (comp_flags & RTE_COMP_FF_HUFFMAN_FIXED) == 0) {
- RTE_LOG(ERR, USER1,
- "Compress device does not supported Fixed Huffman\n");
- return -1;
- }
+ /* Algorithm type */
+ switch (test_data->test_algo) {
+ case RTE_COMP_ALGO_DEFLATE:
+ /* Huffman encoding */
+ if (test_data->huffman_enc == RTE_COMP_HUFFMAN_FIXED &&
+ (comp_flags & RTE_COMP_FF_HUFFMAN_FIXED) == 0) {
+ RTE_LOG(ERR, USER1,
+ "Compress device does not supported Fixed Huffman\n");
+ return -1;
+ }
- if (test_data->huffman_enc == RTE_COMP_HUFFMAN_DYNAMIC &&
- (comp_flags & RTE_COMP_FF_HUFFMAN_DYNAMIC) == 0) {
- RTE_LOG(ERR, USER1,
- "Compress device does not supported Dynamic Huffman\n");
+ if (test_data->huffman_enc == RTE_COMP_HUFFMAN_DYNAMIC &&
+ (comp_flags & RTE_COMP_FF_HUFFMAN_DYNAMIC) == 0) {
+ RTE_LOG(ERR, USER1,
+ "Compress device does not supported Dynamic Huffman\n");
+ return -1;
+ }
+ break;
+ case RTE_COMP_ALGO_LZ4:
+ /* LZ4 flags */
+ if ((test_data->lz4_flags & RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM) &&
+ (comp_flags & RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM) == 0) {
+ RTE_LOG(ERR, USER1,
+ "Compress device does not support LZ4 block with checksum\n");
+ return -1;
+ }
+
+ if ((test_data->lz4_flags &
+ RTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE) &&
+ (comp_flags & RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE) == 0) {
+ RTE_LOG(ERR, USER1,
+ "Compress device does not support LZ4 independent blocks\n");
+ return -1;
+ }
+ break;
+ case RTE_COMP_ALGO_LZS:
+ case RTE_COMP_ALGO_NULL:
+ break;
+ default:
return -1;
}
diff --git a/doc/guides/tools/comp_perf.rst b/doc/guides/tools/comp_perf.rst
index 9d2f4dbe4a..5eb05a095c 100644
--- a/doc/guides/tools/comp_perf.rst
+++ b/doc/guides/tools/comp_perf.rst
@@ -84,8 +84,13 @@ Application Options
``--operation [comp/decomp/comp_and_decomp]``: perform test on compression, decompression or both operations
+ ``--algo [null/deflate/lzs/lz4]`` : perform test on algorithm null(DMA), Deflate, lzs or lz4 (default: Deflate)
+
``--huffman-enc [fixed/dynamic/default]``: Huffman encoding (default: dynamic)
+ ``--lz4-flags N``: flags to for LZ4 parameters,
+ see `LZ4 Frame Descriptor <https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor>`_ (default: no flags)
+
``--compress-level N``: compression level, which could be a single value, list or range (default: range between 1 and 9)
``--window-sz N``: base two log value of compression window size (default: max supported by PMD)
--
2.25.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* RE: [EXT] [PATCH v4 0/4] compressdev: add LZ4 support
2023-02-14 17:40 ` [PATCH v4 " Michael Baum
` (3 preceding siblings ...)
2023-02-14 17:40 ` [PATCH v4 4/4] app/test-compress-perf: add algo option Michael Baum
@ 2023-02-16 8:01 ` Akhil Goyal
4 siblings, 0 replies; 31+ messages in thread
From: Akhil Goyal @ 2023-02-16 8:01 UTC (permalink / raw)
To: Michael Baum, dev; +Cc: Matan Azrad, Ashish Gupta, Fiona Trahe, Thomas Monjalon
> Add support for LZ4 compression algorithm for both API and app.
>
> In addition, enable app to test de/comp only. This option already
> exists in both app doc and arguments but isn't implemented inside the
> application.
>
> LZ4: https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__github.com_lz4_lz4&d=DwIDAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=DnL7Si2
> wl_PRwpZ9TWey3eu68gBzn7DkPwuqhd6WNyo&m=zuvFwq-
> wCfo4E5qcsXuVW4Tg-xq4cnKWlMdx8BrlcwM9-
> mjeKpQgAejfzaUzSwyP&s=89JL5ZLAeh2gshZ3Gqohsr3JZomgK9pjaCnLRHpnG0w
> &e=
> RFC:
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__patchwork.dpdk.org_project_dpdk_patch_20220410182622.8828-2D1-
> 2Drzidane-
> 40nvidia.com_&d=DwIDAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=DnL7Si2wl_PRwpZ
> 9TWey3eu68gBzn7DkPwuqhd6WNyo&m=zuvFwq-wCfo4E5qcsXuVW4Tg-
> xq4cnKWlMdx8BrlcwM9-
> mjeKpQgAejfzaUzSwyP&s=HhD8pa6ivCYKy89RMuLN2OhehLXxbluIyVWxF0QdjXE
> &e=
>
> v2:
> - Add macros for all LZ4 flags in the standard.
> - Rebase.
>
> v3:
> - Rebase.
>
> v4:
> - Change "allow test single" commit to fix and add "Cc: stable@dpdk.org".
> - Take all typo fixes into separate commit.
> - Squash "add LZ4 algo" commit into "add algo option" commit.
> - Remove unnecessary release notes updates.
Applied to dpdk-next-crypto
Thanks
^ permalink raw reply [flat|nested] 31+ messages in thread
end of thread, other threads:[~2023-02-16 8:01 UTC | newest]
Thread overview: 31+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-01-09 7:45 [PATCH 0/4] compressdev: add LZ4 support Michael Baum
2023-01-09 7:45 ` [PATCH 1/4] compressdev: add LZ4 algorithm support Michael Baum
2023-01-30 18:35 ` [EXT] " Akhil Goyal
2023-01-30 20:36 ` Michael Baum
2023-01-31 6:29 ` Akhil Goyal
2023-01-09 7:45 ` [PATCH 2/4] app/test-compress-perf: allow test single compress operation Michael Baum
2023-01-09 7:45 ` [PATCH 3/4] app/test-compress-perf: add algo option Michael Baum
2023-01-09 7:45 ` [PATCH 4/4] app/test-compress-perf: add LZ4 support Michael Baum
2023-02-02 9:13 ` [PATCH v2 0/4] compressdev: " Michael Baum
2023-02-02 9:13 ` [PATCH v2 1/4] compressdev: add LZ4 algorithm support Michael Baum
2023-02-05 17:20 ` [EXT] " Akhil Goyal
2023-02-02 9:13 ` [PATCH v2 2/4] app/test-compress-perf: allow test single compress operation Michael Baum
2023-02-02 9:13 ` [PATCH v2 3/4] app/test-compress-perf: add algo option Michael Baum
2023-02-02 9:13 ` [PATCH v2 4/4] app/test-compress-perf: add LZ4 support Michael Baum
2023-02-13 6:11 ` [PATCH v3 0/4] compressdev: " Michael Baum
2023-02-13 6:11 ` [PATCH v3 1/4] compressdev: add LZ4 algorithm support Michael Baum
2023-02-13 6:11 ` [PATCH v3 2/4] app/test-compress-perf: allow test single compress operation Michael Baum
2023-02-13 7:23 ` [EXT] " Akhil Goyal
2023-02-14 16:37 ` Michael Baum
2023-02-13 6:11 ` [PATCH v3 3/4] app/test-compress-perf: add algo option Michael Baum
2023-02-13 7:29 ` [EXT] " Akhil Goyal
2023-02-14 16:32 ` Michael Baum
2023-02-13 6:11 ` [PATCH v3 4/4] app/test-compress-perf: add LZ4 support Michael Baum
2023-02-13 15:47 ` [PATCH v3 0/4] compressdev: " Zhang, Fan
2023-02-13 15:56 ` Michael Baum
2023-02-14 17:40 ` [PATCH v4 " Michael Baum
2023-02-14 17:40 ` [PATCH v4 1/4] compressdev: add LZ4 algorithm support Michael Baum
2023-02-14 17:40 ` [PATCH v4 2/4] app/test-compress-perf: fix some typos Michael Baum
2023-02-14 17:40 ` [PATCH v4 3/4] app/test-compress-perf: fix missing test single compress op Michael Baum
2023-02-14 17:40 ` [PATCH v4 4/4] app/test-compress-perf: add algo option Michael Baum
2023-02-16 8:01 ` [EXT] [PATCH v4 0/4] compressdev: add LZ4 support Akhil Goyal
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).