[dpdk-dev] [PATCH] compressdev: implement API

[dpdk-dev] [PATCH] compressdev: implement API

[Fiona Trahe]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 84286 bytes --]

With the vast amounts of data being transported around networks
and stored in storage systems, reducing data size is becoming ever
more important.

There are both software libraries and hardware devices available
that provide compression, but no common API.
This API is proposed in this commit, which supports the following features:

- Deflate Algorithm (https://tools.ietf.org/html/rfc1951)
- LZS algorithm (https://tools.ietf.org/html/rfc2395)
- Static and Dynamic Huffman encoding.
- Compression levels
- Checksum generation
- Asynchronous burst API
- Session-based (a session contains immutable data only and is useable across devices)
- stream-based to maintain state and history data for stateful flows.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
---

Changes since RFCv3:

- Added missing dependencies for shared lib
- Used SPDX license header
- Used dynamic logging
- Changed window size type to uint32_t
- Removed some unnecessary API
- Replaced phys_addr_t with rte_iova_t
- Resolved checkpatch issues
- Resolved Doxygen issues
- Added default mx nb qps for virtual pmds
- Removed unnecessary extern keywords
- Resolved enqueue/dequeued prototype issue
- Resolved capabilities issues
- Completed API documentation
- Added missing API in version.map file
- Added experimental tag
- Added compressdev Doxygen doc in index
- Deleted TODOs
- Deleted unused event callback mechanism
- Clarified flush flag and window size behaviour
- Added capability RTE_COMP_FF_NONCOMPRESSED_BLOCKS
- Simplified xform struct - no need for separate
  common and stateful structs
- Renamed device features to COMPDEV to distinguish from
  service features

Items to be addressed in v2
 - Add hash feature
 - Implement stream functions
 - Add int rte_comp_stream_create_in_op_priv()
 - Add meson build
 - Set/clear cache-aligned keyword
 - Description of stateless/stateful behaviour in
   rte_comp_enqueue_burst function header
 - Add capability helper functions 

 config/common_base                                 |   6 +
 doc/api/doxy-api-index.md                          |   1 +
 doc/api/doxy-api.conf                              |   1 +
 lib/Makefile                                       |   3 +
 lib/librte_compressdev/Makefile                    |  29 +
 lib/librte_compressdev/rte_comp.h                  | 503 ++++++++++++
 lib/librte_compressdev/rte_compressdev.c           | 902 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 757 +++++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.c       | 163 ++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 439 ++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  47 ++
 lib/librte_eal/common/include/rte_log.h            |   1 +
 mk/rte.app.mk                                      |   1 +
 13 files changed, 2853 insertions(+)
 create mode 100644 lib/librte_compressdev/Makefile
 create mode 100644 lib/librte_compressdev/rte_comp.h
 create mode 100644 lib/librte_compressdev/rte_compressdev.c
 create mode 100644 lib/librte_compressdev/rte_compressdev.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.c
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_version.map

diff --git a/config/common_base b/config/common_base
index ad03cf4..e0e5768 100644
--- a/config/common_base
+++ b/config/common_base
@@ -535,6 +535,12 @@ CONFIG_RTE_LIBRTE_PMD_MRVL_CRYPTO=n
 CONFIG_RTE_LIBRTE_PMD_MRVL_CRYPTO_DEBUG=n
 
 #
+# Compile generic compression device library
+#
+CONFIG_RTE_LIBRTE_COMPRESSDEV=y
+CONFIG_RTE_COMPRESS_MAX_DEVS=64
+
+#
 # Compile generic security library
 #
 CONFIG_RTE_LIBRTE_SECURITY=y
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index d77f205..07b8e75 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -43,6 +43,7 @@ The public API headers are grouped by topics:
   [rte_tm]             (@ref rte_tm.h),
   [rte_mtr]            (@ref rte_mtr.h),
   [bbdev]              (@ref rte_bbdev.h),
+  [compressdev]        (@ref rte_compressdev.h),
   [cryptodev]          (@ref rte_cryptodev.h),
   [security]           (@ref rte_security.h),
   [eventdev]           (@ref rte_eventdev.h),
diff --git a/doc/api/doxy-api.conf b/doc/api/doxy-api.conf
index cda52fd..06432c3 100644
--- a/doc/api/doxy-api.conf
+++ b/doc/api/doxy-api.conf
@@ -45,6 +45,7 @@ INPUT                   = doc/api/doxy-api-index.md \
                           lib/librte_cfgfile \
                           lib/librte_cmdline \
                           lib/librte_compat \
+                          lib/librte_compressdev \
                           lib/librte_cryptodev \
                           lib/librte_distributor \
                           lib/librte_efd \
diff --git a/lib/Makefile b/lib/Makefile
index ec965a6..19396da 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -26,6 +26,9 @@ DEPDIRS-librte_bbdev := librte_eal librte_mempool librte_mbuf
 DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += librte_cryptodev
 DEPDIRS-librte_cryptodev := librte_eal librte_mempool librte_ring librte_mbuf
 DEPDIRS-librte_cryptodev += librte_kvargs
+DIRS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += librte_compressdev
+DEPDIRS-librte_compressdev := librte_eal librte_mempool librte_ring librte_mbuf
+DEPDIRS-librte_compressdev += librte_kvargs
 DIRS-$(CONFIG_RTE_LIBRTE_SECURITY) += librte_security
 DEPDIRS-librte_security := librte_eal librte_mempool librte_ring librte_mbuf
 DEPDIRS-librte_security += librte_ether
diff --git a/lib/librte_compressdev/Makefile b/lib/librte_compressdev/Makefile
new file mode 100644
index 0000000..6f1546a
--- /dev/null
+++ b/lib/librte_compressdev/Makefile
@@ -0,0 +1,29 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2017-2018 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_compressdev.a
+
+# library version
+LIBABIVER := 1
+
+# build flags
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+LDLIBS += -lrte_eal -lrte_mempool -lrte_kvargs
+
+# library source files
+SRCS-y += rte_compressdev.c rte_compressdev_pmd.c
+
+# export include files
+SYMLINK-y-include += rte_comp.h
+SYMLINK-y-include += rte_compressdev.h
+SYMLINK-y-include += rte_compressdev_pmd.h
+
+# versioning export map
+EXPORT_MAP := rte_compressdev_version.map
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_compressdev/rte_comp.h b/lib/librte_compressdev/rte_comp.h
new file mode 100644
index 0000000..0f5ee32
--- /dev/null
+++ b/lib/librte_compressdev/rte_comp.h
@@ -0,0 +1,503 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMP_H_
+#define _RTE_COMP_H_
+
+/**
+ * @file rte_comp.h
+ *
+ * RTE definitions for Data Compression Service
+ *
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <string.h>
+#include <rte_mempool.h>
+
+
+/** Status of comp operation */
+enum rte_comp_op_status {
+	RTE_COMP_OP_STATUS_SUCCESS = 0,
+	/**< Operation completed successfully */
+	RTE_COMP_OP_STATUS_NOT_PROCESSED,
+	/**< Operation has not yet been processed by the device */
+	RTE_COMP_OP_STATUS_INVALID_SESSION,
+	/**< Operation failed due to invalid session arguments */
+	RTE_COMP_OP_STATUS_INVALID_ARGS,
+	/**< Operation failed due to invalid arguments in request */
+	RTE_COMP_OP_STATUS_ERROR,
+	/**< Error handling operation */
+	RTE_COMP_OP_STATUS_INVALID_STATE,
+	/**< Operation is invoked in invalid state */
+	RTE_COMP_OP_STATUS_OUT_OF_SPACE,
+	/**< Output buffer ran out of space before operation completed */
+};
+
+
+/** Compression Algorithms */
+enum rte_comp_algorithm {
+	RTE_COMP_NULL = 0,
+	/**< No compression.
+	 * Pass-through, data is copied unchanged from source buffer to
+	 * destination buffer.
+	 */
+	RTE_COMP_DEFLATE,
+	/**< DEFLATE compression algorithm
+	 * https://tools.ietf.org/html/rfc1951
+	 */
+	RTE_COMP_LZS,
+	/**< LZS compression algorithm
+	 * https://tools.ietf.org/html/rfc2395
+	 */
+	RTE_COMP_ALGO_LIST_END
+};
+
+/**< Compression Level.
+ * The number is interpreted by each PMD differently. However, lower numbers
+ * give fastest compression, at the expense of compression ratio while
+ * higher numbers may give better compression ratios but are likely slower.
+ */
+#define	RTE_COMP_LEVEL_PMD_DEFAULT	(-1)
+/** Use PMD Default */
+#define	RTE_COMP_LEVEL_NONE		(0)
+/** Output uncompressed blocks if supported by the specified algorithm */
+#define RTE_COMP_LEVEL_MIN		(1)
+/** Use minimum compression level supported by the PMD */
+#define RTE_COMP_LEVEL_MAX		(9)
+/** Use maximum compression level supported by the PMD */
+
+/** Compression checksum types */
+enum rte_comp_checksum_type {
+	RTE_COMP_NONE,
+	/**< No checksum generated */
+	RTE_COMP_CRC32,
+	/**< Generates a CRC32 checksum, as used by gzip */
+	RTE_COMP_ADLER32,
+	/**< Generates an Adler-32 checksum, as used by zlib */
+	RTE_COMP_CRC32_ADLER32,
+	/**< Generates both Adler-32 and CRC32 checksums, concatenated.
+	 * CRC32 is in the lower 32bits, Adler-32 in the upper 32 bits.
+	 */
+};
+
+
+/** Compression Huffman Type - used by DEFLATE algorithm */
+enum rte_comp_huffman {
+	RTE_COMP_DEFAULT,
+	/**< PMD may choose which Huffman codes to use */
+	RTE_COMP_FIXED,
+	/**< Use Fixed Huffman codes */
+	RTE_COMP_DYNAMIC,
+	/**< Use Dynamic Huffman codes */
+};
+
+enum rte_comp_flush_flag {
+	RTE_COMP_FLUSH_NONE,
+	/**< Data is not flushed. Output may remain in the compressor and be
+	 * processed during a following op. It may not be possible to decompress
+	 * output until a later op with some other flush flag has been sent.
+	 */
+	RTE_COMP_FLUSH_SYNC,
+	/**< All data should be flushed to output buffer. Output data can be
+	 * decompressed. However state and history is not cleared, so future
+	 * operations may use history from this operation.
+	 */
+	RTE_COMP_FLUSH_FULL,
+	/**< All data should be flushed to output buffer. Output data can be
+	 * decompressed. State and history data is cleared, so future
+	 * ops will be independent of ops processed before this.
+	 */
+	RTE_COMP_FLUSH_FINAL
+	/**< Same as RTE_COMP_FLUSH_FULL but if op.algo is RTE_COMP_DEFLATE
+	 * then bfinal bit is set in the last block.
+	 */
+};
+
+/** Compression transform types */
+enum rte_comp_xform_type {
+	RTE_COMP_COMPRESS,
+	/**< Compression service - compress */
+	RTE_COMP_DECOMPRESS,
+	/**< Compression service - decompress */
+};
+
+enum rte_comp_op_type {
+	RTE_COMP_OP_STATELESS,
+	/**< All data to be processed is submitted in the op, no state or history
+	 * from previous ops is used and none will be stored for future ops.
+	 * Flush flag must be set to either FLUSH_FULL or FLUSH_FINAL.
+	 */
+	RTE_COMP_OP_STATEFUL
+	/**< There may be more data to be processed after this op, it's part of a
+	 * stream of data. State and history from previous ops can be used
+	 * and resulting state and history can be stored for future ops,
+	 * depending on flush flag.
+	 */
+};
+
+
+/** Parameters specific to the deflate algorithm */
+struct rte_comp_deflate_params {
+	enum rte_comp_huffman huffman;
+	/**< Compression huffman encoding type */
+};
+
+/** Session Setup Data for compression */
+struct rte_comp_compress_xform {
+	enum rte_comp_algorithm algo;
+	/**< Algorithm to use for compress operation */
+	union {
+		struct rte_comp_deflate_params deflate;
+		/**< Parameters specific to the deflate algorithm */
+	}; /**< Algorithm specific parameters */
+	int level;
+	/**< Compression level */
+	uint32_t window_size;
+	/**< Depth of sliding window to be used. If window size can't be
+	 * supported by the PMD then it may fall back to a smaller size. This
+	 * is likely to result in a worse compression ratio.
+	 */
+	enum rte_comp_checksum_type chksum;
+	/**< Type of checksum to generate on the uncompressed data */
+};
+
+/**
+ * Session Setup Data for decompression.
+ */
+struct rte_comp_decompress_xform {
+	enum rte_comp_algorithm algo;
+	/**< Algorithm to use for decompression */
+	enum rte_comp_checksum_type chksum;
+	/**< Type of checksum to generate on the decompressed data */
+	uint32_t window_size;
+	/**< Max depth of sliding window which was used to generate compressed
+	 * data. If window size can't be supported by the PMD then session
+	 * setup should fail.
+	 */
+};
+
+/**
+ * Compression transform structure.
+ *
+ * This is used to specify the compression transforms required.
+ * Each transform structure can hold a single transform, the type field is
+ * used to specify which transform is contained within the union.
+ * There are no chain cases currently supported, just single xforms of
+ *  - compress-only
+ *  - decompress-only
+ *
+ */
+struct rte_comp_xform {
+	struct rte_comp_xform *next;
+	/**< next xform in chain */
+	enum rte_comp_xform_type type;
+	/**< xform type */
+	union {
+		struct rte_comp_compress_xform compress;
+		/**< xform for compress operation */
+		struct rte_comp_decompress_xform decompress;
+		/**< decompress xform */
+	};
+};
+
+
+struct rte_comp_session;
+/**
+ * Compression Operation.
+ *
+ * This structure contains data relating to performing a compression
+ * operation on the referenced mbuf data buffers.
+ *
+ * Comp operations are enqueued and dequeued in comp PMDs using the
+ * rte_compressdev_enqueue_burst() / rte_compressdev_dequeue_burst() APIs
+ */
+struct rte_comp_op {
+
+	enum rte_comp_op_type op_type;
+	void *stream_private;
+	/* Handle for an initialised stream, which holds state and history data.
+	 * rte_comp_stream_create() must be called for all data streams whether
+	 * op_type is STATEFUL or STATELESS and the resulting stream attached
+	 * to the one or more operations associated with the data stream.
+	 */
+	struct rte_comp_session *session;
+	/**< Handle for the initialised session context */
+	struct rte_mempool *mempool;
+	/**< Pool from which operation is allocated */
+	rte_iova_t phys_addr;
+	/**< Physical address of this operation */
+	struct rte_mbuf *m_src;
+	/**< source mbuf
+	 * The total size of the input buffer(s) can be retrieved using
+	 * rte_pktmbuf_data_len(m_src)
+	 */
+	struct rte_mbuf *m_dst;
+	/**< destination mbuf
+	 * The total size of the output buffer(s) can be retrieved using
+	 * rte_pktmbuf_data_len(m_dst)
+	 */
+
+	struct {
+		uint32_t offset;
+		/**< Starting point for compression or decompression,
+		 * specified as number of bytes from start of packet in
+		 * source buffer.
+		 * Starting point for checksum generation in compress direction.
+		 */
+		uint32_t length;
+		/**< The length, in bytes, of the data in source buffer
+		 * to be compressed or decompressed.
+		 * Also the length of the data over which the checksum
+		 * should be generated in compress direction
+		 */
+	} src;
+	struct {
+		uint32_t offset;
+		/**< Starting point for writing output data, specified as
+		 * number of bytes from start of packet in dest
+		 * buffer. Starting point for checksum generation in
+		 * decompress direction.
+		 */
+	} dst;
+	enum rte_comp_flush_flag flush_flag;
+	/**< Defines flush characteristics for the output data.
+	 * Only applicable in compress direction
+	 */
+	uint64_t input_chksum;
+	/**< An input checksum can be provided to generate a
+	 * cumulative checksum across sequential blocks in a STATELESS stream.
+	 * Checksum type is as specified in xform chksum_type
+	 */
+	uint64_t output_chksum;
+	/**< If a checksum is generated it will be written in here.
+	 * Checksum type is as specified in xform chksum_type.
+	 */
+	uint32_t consumed;
+	/**< The number of bytes from the source buffer
+	 * which were compressed/decompressed.
+	 */
+	uint32_t produced;
+	/**< The number of bytes written to the destination buffer
+	 * which were compressed/decompressed.
+	 */
+	uint64_t debug_status;
+	/**<
+	 * Status of the operation is returned in the status param.
+	 * This field allows the PMD to pass back extra
+	 * pmd-specific debug information. Value is not defined on the API.
+	 */
+	uint8_t status;
+	/**<
+	 * Operation status - use values from enum rte_comp_status.
+	 * This is reset to
+	 * RTE_COMP_OP_STATUS_NOT_PROCESSED on allocation from mempool and
+	 * will be set to RTE_COMP_OP_STATUS_SUCCESS after operation
+	 * is successfully processed by a PMD
+	 */
+};
+
+
+/**
+ * Reset the fields of an operation to their default values.
+ *
+ * @note The private data associated with the operation is not zeroed.
+ *
+ * @param op
+ *   The operation to be reset
+ */
+static inline void
+__rte_comp_op_reset(struct rte_comp_op *op)
+{
+	struct rte_mempool *tmp_mp = op->mempool;
+	phys_addr_t tmp_phys_addr = op->phys_addr;
+
+	memset(op, 0, sizeof(struct rte_comp_op));
+	op->status = RTE_COMP_OP_STATUS_NOT_PROCESSED;
+	op->phys_addr = tmp_phys_addr;
+	op->mempool = tmp_mp;
+}
+
+/**
+ * Private data structure belonging to an operation pool.
+ */
+struct rte_comp_op_pool_private {
+	uint16_t user_size;
+	/**< Size of private user data with each operation. */
+};
+
+
+/**
+ * Returns the size of private user data allocated with each object in
+ * the mempool
+ *
+ * @param mempool
+ *   Mempool for operations
+ * @return
+ *   user data size
+ */
+static inline uint16_t
+__rte_comp_op_get_user_data_size(struct rte_mempool *mempool)
+{
+	struct rte_comp_op_pool_private *priv =
+	    (struct rte_comp_op_pool_private *)rte_mempool_get_priv(mempool);
+
+	return priv->user_size;
+}
+
+
+/**
+ * Creates an operation pool
+ *
+ * @param name
+ *   Compress pool name
+ * @param nb_elts
+ *   Number of elements in pool
+ * @param cache_size
+ *   Number of elements to cache on lcore, see
+ *   *rte_mempool_create* for further details about cache size
+ * @param user_size
+ *   Size of private data to allocate for user with each operation
+ * @param socket_id
+ *   Socket to identifier allocate memory on
+ * @return
+ *  - On success pointer to mempool
+ *  - On failure NULL
+ */
+struct rte_mempool *
+rte_comp_op_pool_create(const char *name,
+		unsigned int nb_elts, unsigned int cache_size,
+		uint16_t user_size, int socket_id);
+
+/**
+ * Bulk allocate raw element from mempool and return as comp operations
+ *
+ * @param mempool
+ *   Compress operation mempool
+ * @param ops
+ *   Array to place allocated operations
+ * @param nb_ops
+ *   Number of operations to allocate
+ * @return
+ * - On success returns  number of ops allocated
+ */
+static inline int
+__rte_comp_op_raw_bulk_alloc(struct rte_mempool *mempool,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	if (rte_mempool_get_bulk(mempool, (void **)ops, nb_ops) == 0)
+		return nb_ops;
+
+	return 0;
+}
+
+/**
+ * Allocate an operation from a mempool with default parameters set
+ *
+ * @param mempool
+ *   Compress operation mempool
+ *
+ * @return
+ * - On success returns a valid rte_comp_op structure
+ * - On failure returns NULL
+ */
+static inline struct rte_comp_op *
+rte_comp_op_alloc(struct rte_mempool *mempool)
+{
+	struct rte_comp_op *op = NULL;
+	int retval;
+
+	retval = __rte_comp_op_raw_bulk_alloc(mempool, &op, 1);
+	if (unlikely(retval != 1))
+		return NULL;
+
+	__rte_comp_op_reset(op);
+
+	return op;
+}
+
+
+/**
+ * Bulk allocate operations from a mempool with default parameters set
+ *
+ * @param mempool
+ *   Compress operation mempool
+ * @param ops
+ *   Array to place allocated operations
+ * @param nb_ops
+ *   Number of operations to allocate
+ * @return
+ * - nb_ops if the number of operations requested were allocated.
+ * - 0 if the requested number of ops are not available.
+ *   None are allocated in this case.
+ */
+static inline unsigned
+rte_comp_op_bulk_alloc(struct rte_mempool *mempool,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	int i;
+
+	if (unlikely(__rte_comp_op_raw_bulk_alloc(mempool, ops, nb_ops)
+			!= nb_ops))
+		return 0;
+
+	for (i = 0; i < nb_ops; i++)
+		__rte_comp_op_reset(ops[i]);
+
+	return nb_ops;
+}
+
+
+
+/**
+ * Returns a pointer to the private user data of an operation if
+ * that operation has enough capacity for requested size.
+ *
+ * @param op
+ *   Compress operation
+ * @param size
+ *   Size of space requested in private data
+ * @return
+ * - if sufficient space available returns pointer to start of user data
+ * - if insufficient space returns NULL
+ */
+static inline void *
+__rte_comp_op_get_user_data(struct rte_comp_op *op, uint32_t size)
+{
+	uint32_t user_size;
+
+	if (likely(op->mempool != NULL)) {
+		user_size = __rte_comp_op_get_user_data_size(op->mempool);
+
+		if (likely(user_size >= size))
+			return (void *)(op + 1);
+
+	}
+
+	return NULL;
+}
+
+/**
+ * free operation structure
+ * If operation has been allocate from a rte_mempool, then the operation will
+ * be returned to the mempool.
+ *
+ * @param op
+ *   Compress operation
+ */
+static inline void
+rte_comp_op_free(struct rte_comp_op *op)
+{
+	if (op != NULL && op->mempool != NULL)
+		rte_mempool_put(op->mempool, op);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_COMP_H_ */
diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
new file mode 100644
index 0000000..8fc6674
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -0,0 +1,902 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#include <sys/types.h>
+#include <sys/queue.h>
+#include <ctype.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stdarg.h>
+#include <errno.h>
+#include <stdint.h>
+#include <inttypes.h>
+#include <netinet/in.h>
+
+#include <rte_byteorder.h>
+#include <rte_log.h>
+#include <rte_debug.h>
+#include <rte_dev.h>
+#include <rte_interrupts.h>
+#include <rte_memory.h>
+#include <rte_memcpy.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_tailq.h>
+#include <rte_eal.h>
+#include <rte_per_lcore.h>
+#include <rte_lcore.h>
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_mempool.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_errno.h>
+#include <rte_spinlock.h>
+#include <rte_string_fns.h>
+
+#include "rte_comp.h"
+#include "rte_compressdev.h"
+#include "rte_compressdev_pmd.h"
+
+static uint8_t nb_drivers;
+
+struct rte_compressdev rte_comp_devices[RTE_COMPRESS_MAX_DEVS];
+
+struct rte_compressdev *rte_compressdevs = &rte_comp_devices[0];
+
+static struct rte_compressdev_global compressdev_globals = {
+		.devs			= &rte_comp_devices[0],
+		.data			= { NULL },
+		.nb_devs		= 0,
+		.max_devs		= RTE_COMPRESS_MAX_DEVS
+};
+
+struct rte_compressdev_global *rte_compressdev_globals = &compressdev_globals;
+
+/**
+ * The compression algorithm strings identifiers.
+ * It could be used in application command line.
+ */
+const char *
+rte_comp_algorithm_strings[] = {
+	[RTE_COMP_DEFLATE]		= "deflate",
+	[RTE_COMP_LZS]			= "lzs"
+};
+
+
+#define param_range_check(x, y) \
+	(((x < y.min) || (x > y.max)) || \
+	(y.increment != 0 && (x % y.increment) != 0))
+
+
+const char * __rte_experimental
+rte_compressdev_get_feature_name(uint64_t flag)
+{
+	switch (flag) {
+	case RTE_COMPDEV_FF_HW_ACCELERATED:
+		return "HW_ACCELERATED";
+	case RTE_COMPDEV_FF_CPU_SSE:
+		return "CPU_SSE";
+	case RTE_COMPDEV_FF_CPU_AVX:
+		return "CPU_AVX";
+	case RTE_COMPDEV_FF_CPU_AVX2:
+		return "CPU_AVX2";
+	case RTE_COMPDEV_FF_CPU_AVX512:
+		return "CPU_AVX512";
+	case RTE_COMPDEV_FF_CPU_NEON:
+		return "CPU_NEON";
+	default:
+		return NULL;
+	}
+}
+
+const char * __rte_experimental
+rte_comp_get_feature_name(uint64_t flag)
+{
+	switch (flag) {
+	case RTE_COMP_FF_STATEFUL_COMPRESSION:
+		return "STATEFUL_COMPRESSION";
+	case RTE_COMP_FF_STATEFUL_DECOMPRESSION:
+		return "STATEFUL_DECOMPRESSION";
+	case RTE_COMP_FF_MBUF_SCATTER_GATHER:
+		return "MBUF_SCATTER_GATHER";
+	case RTE_COMP_FF_MULTI_PKT_CHECKSUM:
+		return "MULTI_PKT_CHECKSUM";
+	case RTE_COMP_FF_ADLER32_CHECKSUM:
+		return "ADLER32_CHECKSUM";
+	case RTE_COMP_FF_CRC32_CHECKSUM:
+		return "CRC32_CHECKSUM";
+	case RTE_COMP_FF_CRC32_ADLER32_CHECKSUM:
+		return "CRC32_ADLER32_CHECKSUM";
+	case RTE_COMP_FF_NONCOMPRESSED_BLOCKS:
+		return "NONCOMPRESSED_BLOCKS";
+	default:
+		return NULL;
+	}
+}
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_get_dev(uint8_t dev_id)
+{
+	return &rte_compressdev_globals->devs[dev_id];
+}
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_get_named_dev(const char *name)
+{
+	struct rte_compressdev *dev;
+	unsigned int i;
+
+	if (name == NULL)
+		return NULL;
+
+	for (i = 0; i < rte_compressdev_globals->max_devs; i++) {
+		dev = &rte_compressdev_globals->devs[i];
+
+		if ((dev->attached == RTE_COMPRESSDEV_ATTACHED) &&
+				(strcmp(dev->data->name, name) == 0))
+			return dev;
+	}
+
+	return NULL;
+}
+
+unsigned int __rte_experimental
+rte_compressdev_pmd_is_valid_dev(uint8_t dev_id)
+{
+	struct rte_compressdev *dev = NULL;
+
+	if (dev_id >= rte_compressdev_globals->nb_devs)
+		return 0;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+	if (dev->attached != RTE_COMPRESSDEV_ATTACHED)
+		return 0;
+	else
+		return 1;
+}
+
+
+int __rte_experimental
+rte_compressdev_get_dev_id(const char *name)
+{
+	unsigned int i;
+
+	if (name == NULL)
+		return -1;
+
+	for (i = 0; i < rte_compressdev_globals->nb_devs; i++)
+		if ((strcmp(rte_compressdev_globals->devs[i].data->name, name)
+				== 0) &&
+				(rte_compressdev_globals->devs[i].attached ==
+						RTE_COMPRESSDEV_ATTACHED))
+			return i;
+
+	return -1;
+}
+
+uint8_t __rte_experimental
+rte_compressdev_count(void)
+{
+	return rte_compressdev_globals->nb_devs;
+}
+
+uint8_t __rte_experimental
+rte_compressdev_device_count_by_driver(uint8_t driver_id)
+{
+	uint8_t i, dev_count = 0;
+
+	for (i = 0; i < rte_compressdev_globals->max_devs; i++)
+		if (rte_compressdev_globals->devs[i].driver_id == driver_id &&
+			rte_compressdev_globals->devs[i].attached ==
+					RTE_COMPRESSDEV_ATTACHED)
+			dev_count++;
+
+	return dev_count;
+}
+
+uint8_t __rte_experimental
+rte_compressdev_devices_get(const char *driver_name, uint8_t *devices,
+	uint8_t nb_devices)
+{
+	uint8_t i, count = 0;
+	struct rte_compressdev *devs = rte_compressdev_globals->devs;
+	uint8_t max_devs = rte_compressdev_globals->max_devs;
+
+	for (i = 0; i < max_devs && count < nb_devices;	i++) {
+
+		if (devs[i].attached == RTE_COMPRESSDEV_ATTACHED) {
+			int cmp;
+
+			cmp = strncmp(devs[i].device->driver->name,
+					driver_name,
+					strlen(driver_name));
+
+			if (cmp == 0)
+				devices[count++] = devs[i].data->dev_id;
+		}
+	}
+
+	return count;
+}
+
+
+int __rte_experimental
+rte_compressdev_socket_id(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_pmd_is_valid_dev(dev_id))
+		return -1;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	return dev->data->socket_id;
+}
+
+static inline int
+rte_compressdev_data_alloc(uint8_t dev_id, struct rte_compressdev_data **data,
+		int socket_id)
+{
+	char mz_name[RTE_COMPRESSDEV_NAME_MAX_LEN];
+	const struct rte_memzone *mz;
+	int n;
+
+	/* generate memzone name */
+	n = snprintf(mz_name, sizeof(mz_name),
+			"rte_compressdev_data_%u", dev_id);
+	if (n >= (int)sizeof(mz_name))
+		return -EINVAL;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		mz = rte_memzone_reserve(mz_name,
+				sizeof(struct rte_compressdev_data),
+				socket_id, 0);
+	} else
+		mz = rte_memzone_lookup(mz_name);
+
+	if (mz == NULL)
+		return -ENOMEM;
+
+	*data = mz->addr;
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		memset(*data, 0, sizeof(struct rte_compressdev_data));
+
+	return 0;
+}
+
+static uint8_t
+rte_compressdev_find_free_device_index(void)
+{
+	uint8_t dev_id;
+
+	for (dev_id = 0; dev_id < RTE_COMPRESS_MAX_DEVS; dev_id++) {
+		if (rte_comp_devices[dev_id].attached ==
+				RTE_COMPRESSDEV_DETACHED)
+			return dev_id;
+	}
+	return RTE_COMPRESS_MAX_DEVS;
+}
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_allocate(const char *name, int socket_id)
+{
+	struct rte_compressdev *compressdev;
+	uint8_t dev_id;
+
+	if (rte_compressdev_pmd_get_named_dev(name) != NULL) {
+		COMPRESSDEV_LOG(ERR,
+			"comp device with name %s already allocated!", name);
+		return NULL;
+	}
+
+	dev_id = rte_compressdev_find_free_device_index();
+	if (dev_id == RTE_COMPRESS_MAX_DEVS) {
+		COMPRESSDEV_LOG(ERR, "Reached maximum number of comp devices");
+		return NULL;
+	}
+
+	compressdev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (compressdev->data == NULL) {
+		struct rte_compressdev_data *compressdev_data =
+				compressdev_globals.data[dev_id];
+
+		int retval = rte_compressdev_data_alloc(dev_id,
+				&compressdev_data, socket_id);
+
+		if (retval < 0 || compressdev_data == NULL)
+			return NULL;
+
+		compressdev->data = compressdev_data;
+
+		snprintf(compressdev->data->name, RTE_COMPRESSDEV_NAME_MAX_LEN,
+				"%s", name);
+
+		compressdev->data->dev_id = dev_id;
+		compressdev->data->socket_id = socket_id;
+		compressdev->data->dev_started = 0;
+
+		compressdev->attached = RTE_COMPRESSDEV_ATTACHED;
+
+		compressdev_globals.nb_devs++;
+	}
+
+	return compressdev;
+}
+
+int __rte_experimental
+rte_compressdev_pmd_release_device(struct rte_compressdev *compressdev)
+{
+	int ret;
+
+	if (compressdev == NULL)
+		return -EINVAL;
+
+	/* Close device only if device operations have been set */
+	if (compressdev->dev_ops) {
+		ret = rte_compressdev_close(compressdev->data->dev_id);
+		if (ret < 0)
+			return ret;
+	}
+
+	compressdev->attached = RTE_COMPRESSDEV_DETACHED;
+	compressdev_globals.nb_devs--;
+	return 0;
+}
+
+uint16_t __rte_experimental
+rte_compressdev_queue_pair_count(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	dev = &rte_comp_devices[dev_id];
+	return dev->data->nb_queue_pairs;
+}
+
+static int
+rte_compressdev_queue_pairs_config(struct rte_compressdev *dev,
+		uint16_t nb_qpairs, int socket_id)
+{
+	struct rte_compressdev_info dev_info;
+	void **qp;
+	unsigned int i;
+
+	if ((dev == NULL) || (nb_qpairs < 1)) {
+		COMPRESSDEV_LOG(ERR, "invalid param: dev %p, nb_queues %u",
+							dev, nb_qpairs);
+		return -EINVAL;
+	}
+
+	COMPRESSDEV_LOG(DEBUG, "Setup %d queues pairs on device %u",
+			nb_qpairs, dev->data->dev_id);
+
+	memset(&dev_info, 0, sizeof(struct rte_compressdev_info));
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_infos_get, -ENOTSUP);
+	(*dev->dev_ops->dev_infos_get)(dev, &dev_info);
+
+	if (nb_qpairs > (dev_info.max_nb_queue_pairs)) {
+		COMPRESSDEV_LOG(ERR, "Invalid num queue_pairs (%u) for dev %u",
+				nb_qpairs, dev->data->dev_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->queue_pairs == NULL) { /* first time configuration */
+		dev->data->queue_pairs = rte_zmalloc_socket(
+				"compressdev->queue_pairs",
+				sizeof(dev->data->queue_pairs[0]) * nb_qpairs,
+				RTE_CACHE_LINE_SIZE, socket_id);
+
+		if (dev->data->queue_pairs == NULL) {
+			dev->data->nb_queue_pairs = 0;
+			COMPRESSDEV_LOG(ERR,
+			"failed to get memory for qp meta data, nb_queues %u",
+							nb_qpairs);
+			return -(ENOMEM);
+		}
+	} else { /* re-configure */
+		int ret;
+		uint16_t old_nb_queues = dev->data->nb_queue_pairs;
+
+		qp = dev->data->queue_pairs;
+
+		RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->queue_pair_release,
+				-ENOTSUP);
+
+		for (i = nb_qpairs; i < old_nb_queues; i++) {
+			ret = (*dev->dev_ops->queue_pair_release)(dev, i);
+			if (ret < 0)
+				return ret;
+		}
+
+		qp = rte_realloc(qp, sizeof(qp[0]) * nb_qpairs,
+				RTE_CACHE_LINE_SIZE);
+		if (qp == NULL) {
+			COMPRESSDEV_LOG(ERR,
+			"failed to realloc qp meta data, nb_queues %u",
+						nb_qpairs);
+			return -(ENOMEM);
+		}
+
+		if (nb_qpairs > old_nb_queues) {
+			uint16_t new_qs = nb_qpairs - old_nb_queues;
+
+			memset(qp + old_nb_queues, 0,
+				sizeof(qp[0]) * new_qs);
+		}
+
+		dev->data->queue_pairs = qp;
+
+	}
+	dev->data->nb_queue_pairs = nb_qpairs;
+	return 0;
+}
+
+int __rte_experimental
+rte_compressdev_configure(uint8_t dev_id, struct rte_compressdev_config *config)
+{
+	struct rte_compressdev *dev;
+	int diag;
+
+	if (!rte_compressdev_pmd_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	if (dev->data->dev_started) {
+		COMPRESSDEV_LOG(ERR,
+		    "device %d must be stopped to allow configuration", dev_id);
+		return -EBUSY;
+	}
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_configure, -ENOTSUP);
+
+	/* Setup new number of queue pairs and reconfigure device. */
+	diag = rte_compressdev_queue_pairs_config(dev, config->nb_queue_pairs,
+			config->socket_id);
+	if (diag != 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev%d rte_comp_dev_queue_pairs_config = %d",
+				dev_id, diag);
+		return diag;
+	}
+
+	return (*dev->dev_ops->dev_configure)(dev, config);
+}
+
+
+int __rte_experimental
+rte_compressdev_start(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+	int diag;
+
+	COMPRESSDEV_LOG(DEBUG, "Start dev_id=%" PRIu8, dev_id);
+
+	if (!rte_compressdev_pmd_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_start, -ENOTSUP);
+
+	if (dev->data->dev_started != 0) {
+		COMPRESSDEV_LOG(ERR,
+		    "Device with dev_id=%" PRIu8 " already started", dev_id);
+		return 0;
+	}
+
+	diag = (*dev->dev_ops->dev_start)(dev);
+	if (diag == 0)
+		dev->data->dev_started = 1;
+	else
+		return diag;
+
+	return 0;
+}
+
+void __rte_experimental
+rte_compressdev_stop(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_pmd_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_stop);
+
+	if (dev->data->dev_started == 0) {
+		COMPRESSDEV_LOG(ERR,
+		    "Device with dev_id=%" PRIu8 " already stopped", dev_id);
+		return;
+	}
+
+	(*dev->dev_ops->dev_stop)(dev);
+	dev->data->dev_started = 0;
+}
+
+int __rte_experimental
+rte_compressdev_close(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+	int retval;
+
+	if (!rte_compressdev_pmd_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -1;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	/* Device must be stopped before it can be closed */
+	if (dev->data->dev_started == 1) {
+		COMPRESSDEV_LOG(ERR, "Device %u must be stopped before closing",
+				dev_id);
+		return -EBUSY;
+	}
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_close, -ENOTSUP);
+	retval = (*dev->dev_ops->dev_close)(dev);
+
+	if (retval < 0)
+		return retval;
+
+	return 0;
+}
+
+int __rte_experimental
+rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
+		uint32_t max_inflight_ops, int socket_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_pmd_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+	if (queue_pair_id >= dev->data->nb_queue_pairs) {
+		COMPRESSDEV_LOG(ERR, "Invalid queue_pair_id=%d", queue_pair_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->dev_started) {
+		COMPRESSDEV_LOG(ERR,
+		    "device %d must be stopped to allow configuration", dev_id);
+		return -EBUSY;
+	}
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->queue_pair_setup, -ENOTSUP);
+
+	return (*dev->dev_ops->queue_pair_setup)(dev, queue_pair_id,
+			max_inflight_ops, socket_id);
+}
+
+
+int __rte_experimental
+rte_compressdev_stats_get(uint8_t dev_id, struct rte_compressdev_stats *stats)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_pmd_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%d", dev_id);
+		return -ENODEV;
+	}
+
+	if (stats == NULL) {
+		COMPRESSDEV_LOG(ERR, "Invalid stats ptr");
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+	memset(stats, 0, sizeof(*stats));
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->stats_get, -ENOTSUP);
+	(*dev->dev_ops->stats_get)(dev, stats);
+	return 0;
+}
+
+void __rte_experimental
+rte_compressdev_stats_reset(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_pmd_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	RTE_FUNC_PTR_OR_RET(*dev->dev_ops->stats_reset);
+	(*dev->dev_ops->stats_reset)(dev);
+}
+
+
+void __rte_experimental
+rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
+{
+	struct rte_compressdev *dev;
+
+	if (dev_id >= compressdev_globals.nb_devs) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%d", dev_id);
+		return;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	memset(dev_info, 0, sizeof(struct rte_compressdev_info));
+
+	RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
+	(*dev->dev_ops->dev_infos_get)(dev, dev_info);
+
+	dev_info->driver_name = dev->device->driver->name;
+}
+
+
+int __rte_experimental
+rte_compressdev_session_init(uint8_t dev_id,
+		struct rte_comp_session *sess,
+		struct rte_comp_xform *xforms,
+		struct rte_mempool *mp)
+{
+	struct rte_compressdev *dev;
+	uint8_t index;
+	int ret;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (sess == NULL || xforms == NULL || dev == NULL)
+		return -EINVAL;
+
+	index = dev->driver_id;
+
+	if (sess->sess_private_data[index] == NULL) {
+		ret = dev->dev_ops->session_configure(dev, xforms, sess, mp);
+		if (ret < 0) {
+			COMPRESSDEV_LOG(ERR,
+				"dev_id %d failed to configure session details",
+				dev_id);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+struct rte_comp_session * __rte_experimental
+rte_compressdev_session_create(struct rte_mempool *mp)
+{
+	struct rte_comp_session *sess;
+
+	/* Allocate a session structure from the session pool */
+	if (rte_mempool_get(mp, (void *)&sess)) {
+		COMPRESSDEV_LOG(ERR, "couldn't get object from session mempool");
+		return NULL;
+	}
+
+	/* Clear device session pointer */
+	memset(sess, 0, (sizeof(void *) * nb_drivers));
+
+	return sess;
+}
+
+int __rte_experimental
+rte_compressdev_session_clear(uint8_t dev_id,
+		struct rte_comp_session *sess)
+{
+	struct rte_compressdev *dev;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (dev == NULL || sess == NULL)
+		return -EINVAL;
+
+	dev->dev_ops->session_clear(dev, sess);
+
+	return 0;
+}
+
+int __rte_experimental
+rte_compressdev_session_terminate(struct rte_comp_session *sess)
+{
+	uint8_t i;
+	void *sess_priv;
+	struct rte_mempool *sess_mp;
+
+	if (sess == NULL)
+		return -EINVAL;
+
+	/* Check that all device private data has been freed */
+	for (i = 0; i < nb_drivers; i++) {
+		sess_priv = get_session_private_data(sess, i);
+		if (sess_priv != NULL)
+			return -EBUSY;
+	}
+
+	/* Return session to mempool */
+	sess_mp = rte_mempool_from_obj(sess);
+	rte_mempool_put(sess_mp, sess);
+
+	return 0;
+}
+
+unsigned int __rte_experimental
+rte_compressdev_get_header_session_size(void)
+{
+	/*
+	 * Header contains pointers to the private data
+	 * of all registered drivers
+	 */
+	return (sizeof(void *) * nb_drivers);
+}
+
+unsigned int __rte_experimental
+rte_compressdev_get_private_session_size(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_pmd_is_valid_dev(dev_id))
+		return 0;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (*dev->dev_ops->session_get_size == NULL)
+		return 0;
+
+	return (*dev->dev_ops->session_get_size)(dev);
+
+}
+
+/** Initialise rte_comp_op mempool element */
+static void
+rte_comp_op_init(struct rte_mempool *mempool,
+		__rte_unused void *opaque_arg,
+		void *_op_data,
+		__rte_unused unsigned int i)
+{
+	struct rte_comp_op *op = _op_data;
+
+	memset(_op_data, 0, mempool->elt_size);
+
+	op->status = RTE_COMP_OP_STATUS_NOT_PROCESSED;
+	op->phys_addr = rte_mem_virt2iova(_op_data);
+	op->mempool = mempool;
+}
+
+
+struct rte_mempool * __rte_experimental
+rte_comp_op_pool_create(const char *name,
+		unsigned int nb_elts, unsigned int cache_size,
+		uint16_t user_size, int socket_id)
+{
+	struct rte_comp_op_pool_private *priv;
+
+	unsigned int elt_size = sizeof(struct rte_comp_op) + user_size;
+
+	/* lookup mempool in case already allocated */
+	struct rte_mempool *mp = rte_mempool_lookup(name);
+
+	if (mp != NULL) {
+		priv = (struct rte_comp_op_pool_private *)
+				rte_mempool_get_priv(mp);
+
+		if (mp->elt_size != elt_size ||
+				mp->cache_size < cache_size ||
+				mp->size < nb_elts ||
+				priv->user_size <  user_size) {
+			mp = NULL;
+			COMPRESSDEV_LOG(ERR,
+		"Mempool %s already exists but with incompatible parameters",
+					name);
+			return NULL;
+		}
+		return mp;
+	}
+
+	mp = rte_mempool_create(
+			name,
+			nb_elts,
+			elt_size,
+			cache_size,
+			sizeof(struct rte_comp_op_pool_private),
+			NULL,
+			NULL,
+			rte_comp_op_init,
+			NULL,
+			socket_id,
+			0);
+
+	if (mp == NULL) {
+		COMPRESSDEV_LOG(ERR, "Failed to create mempool %s", name);
+		return NULL;
+	}
+
+	priv = (struct rte_comp_op_pool_private *)
+			rte_mempool_get_priv(mp);
+
+	priv->user_size = user_size;
+
+	return mp;
+}
+
+TAILQ_HEAD(compressdev_driver_list, compressdev_driver);
+
+static struct compressdev_driver_list compressdev_driver_list =
+	TAILQ_HEAD_INITIALIZER(compressdev_driver_list);
+
+int __rte_experimental
+rte_compressdev_driver_id_get(const char *name)
+{
+	struct compressdev_driver *driver;
+	const char *driver_name;
+
+	if (name == NULL) {
+		RTE_LOG(DEBUG, COMPRESSDEV, "name pointer NULL");
+		return -1;
+	}
+
+	TAILQ_FOREACH(driver, &compressdev_driver_list, next) {
+		driver_name = driver->driver->name;
+		if (strncmp(driver_name, name, strlen(driver_name)) == 0)
+			return driver->id;
+	}
+	return -1;
+}
+
+const char * __rte_experimental
+rte_compressdev_name_get(uint8_t dev_id)
+{
+	struct rte_compressdev *dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (dev == NULL)
+		return NULL;
+
+	return dev->data->name;
+}
+
+const char * __rte_experimental
+rte_compressdev_driver_name_get(uint8_t driver_id)
+{
+	struct compressdev_driver *driver;
+
+	TAILQ_FOREACH(driver, &compressdev_driver_list, next)
+		if (driver->id == driver_id)
+			return driver->driver->name;
+	return NULL;
+}
+
+uint8_t
+rte_compressdev_allocate_driver(struct compressdev_driver *comp_drv,
+		const struct rte_driver *drv)
+{
+	comp_drv->driver = drv;
+	comp_drv->id = nb_drivers;
+
+	TAILQ_INSERT_TAIL(&compressdev_driver_list, comp_drv, next);
+
+	return nb_drivers++;
+}
+
+RTE_INIT(rte_compressdev_log);
+
+static void
+rte_compressdev_log(void)
+{
+	compressdev_logtype = rte_log_register("librte.compressdev");
+	if (compressdev_logtype >= 0)
+		rte_log_set_level(compressdev_logtype, RTE_LOG_NOTICE);
+}
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
new file mode 100644
index 0000000..72390b4
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -0,0 +1,757 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMPRESSDEV_H_
+#define _RTE_COMPRESSDEV_H_
+
+/**
+ * @file rte_compressdev.h
+ *
+ * RTE Compression Device APIs
+ *
+ * Defines comp device APIs for the provisioning of compression operations.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include "rte_kvargs.h"
+#include "rte_comp.h"
+#include "rte_dev.h"
+#include <rte_common.h>
+
+extern const char **rte_cyptodev_names;
+
+/* Logging Macros */
+extern int compressdev_logtype;
+#define COMPRESSDEV_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, compressdev_logtype, "%s(): "fmt "\n", \
+			__func__, ##args)
+
+#define RTE_COMPRESSDEV_DETACHED  (0)
+#define RTE_COMPRESSDEV_ATTACHED  (1)
+
+#define RTE_COMPRESSDEV_NAME_MAX_LEN	(64)
+
+/**< Max length of name of comp PMD */
+/**
+ * A macro that points to an offset from the start
+ * of the comp operation structure (rte_comp_op)
+ *
+ * The returned pointer is cast to type t.
+ *
+ * @param c
+ *   The comp operation
+ * @param o
+ *   The offset from the start of the comp operation
+ * @param t
+ *   The type to cast the result into
+ */
+#define rte_comp_op_ctod_offset(c, t, o)	\
+	((t)((char *)(c) + (o)))
+
+/**
+ * A macro that returns the physical address that points
+ * to an offset from the start of the comp operation
+ * (rte_comp_op).
+ *
+ * @param c
+ *   The comp operation
+ * @param o
+ *   The offset from the start of the comp operation
+ *   to calculate address from
+ */
+#define rte_comp_op_ctophys_offset(c, o)	\
+	(rte_iova_t)((c)->phys_addr + (o))
+
+/**
+ * Parameter log base 2 range description.
+ * Final value will be 2^value.
+ */
+struct rte_param_log2_range {
+	uint8_t min;	/**< Minimum log2 value */
+	uint8_t max;	/**< Maximum log2 value */
+	uint8_t increment;
+	/**< If a range of sizes are supported,
+	 * this parameter is used to indicate
+	 * increments in log base 2 byte value
+	 * that are supported between the minimum and maximum
+	 */
+};
+
+/** Structure used to capture a capability of a comp device */
+struct rte_compressdev_capabilities {
+	enum rte_comp_algorithm algo;
+	/* Compression algorithm */
+	uint64_t comp_feature_flags;
+	/**< Bitmask of flags for compression service features */
+	struct rte_param_log2_range window_size;
+	/**< Window size range in bytes */
+};
+
+
+/** Macro used at end of comp PMD list */
+#define RTE_COMP_END_OF_CAPABILITIES_LIST() \
+	{ RTE_COMP_ALGO_LIST_END }
+
+/**
+ * compression device supported feature flags
+ *
+ * @note New features flags should be added to the end of the list
+ *
+ * Keep these flags synchronised with rte_compressdev_get_feature_name()
+ */
+
+#define	RTE_COMPDEV_FF_HW_ACCELERATED		(1ULL << 0)
+/**< Operations are off-loaded to an external hardware accelerator */
+#define	RTE_COMPDEV_FF_CPU_SSE			(1ULL << 1)
+/**< Utilises CPU SIMD SSE instructions */
+#define	RTE_COMPDEV_FF_CPU_AVX			(1ULL << 2)
+/**< Utilises CPU SIMD AVX instructions */
+#define	RTE_COMPDEV_FF_CPU_AVX2			(1ULL << 3)
+/**< Utilises CPU SIMD AVX2 instructions */
+#define	RTE_COMPDEV_FF_CPU_AVX512		(1ULL << 4)
+/**< Utilises CPU SIMD AVX512 instructions */
+#define	RTE_COMPDEV_FF_CPU_NEON			(1ULL << 5)
+/**< Utilises CPU NEON instructions */
+
+/**
+ * compression service feature flags
+ *
+ * @note New features flags should be added to the end of the list
+ *
+ * Keep these flags synchronised with rte_comp_get_feature_name()
+ */
+#define RTE_COMP_FF_STATEFUL_COMPRESSION	(1ULL << 0)
+/**< Stateful compression is supported */
+#define RTE_COMP_FF_STATEFUL_DECOMPRESSION	(1ULL << 1)
+/**< Stateful decompression is supported */
+#define	RTE_COMP_FF_MBUF_SCATTER_GATHER		(1ULL << 2)
+/**< Scatter-gather mbufs are supported */
+#define RTE_COMP_FF_ADLER32_CHECKSUM		(1ULL << 3)
+/**< Adler-32 Checksum is supported */
+#define RTE_COMP_FF_CRC32_CHECKSUM		(1ULL << 4)
+/**< CRC32 Checksum is supported */
+#define RTE_COMP_FF_CRC32_ADLER32_CHECKSUM	(1ULL << 5)
+/**< Adler-32/CRC32 Checksum is supported */
+#define RTE_COMP_FF_MULTI_PKT_CHECKSUM		(1ULL << 6)
+/**< Generation of checksum across multiple stateless packets is supported */
+#define RTE_COMP_FF_NONCOMPRESSED_BLOCKS	(1ULL << 7)
+/**< Creation of non-compressed blocks using RTE_COMP_LEVEL_NONE is supported */
+
+/**
+ * Get the name of a compress device feature flag.
+ *
+ * @param flag
+ *   The mask describing the flag
+ *
+ * @return
+ *   The name of this flag, or NULL if it's not a valid feature flag.
+ */
+const char * __rte_experimental
+rte_compressdev_get_feature_name(uint64_t flag);
+
+/**
+ * Get the name of a compress service feature flag
+ *
+ * @param flag
+ *   The mask describing the flag
+ *
+ * @return
+ *   The name of this flag, or NULL if it's not a valid feature flag.
+ */
+const char * __rte_experimental
+rte_comp_get_feature_name(uint64_t flag);
+
+/**  comp device information */
+struct rte_compressdev_info {
+	const char *driver_name;		/**< Driver name. */
+	uint8_t driver_id;			/**< Driver identifier */
+	uint64_t feature_flags;			/**< Feature flags */
+	const struct rte_compressdev_capabilities *capabilities;
+	/**< Array of devices supported capabilities */
+	unsigned int max_nb_queue_pairs;
+	/**< Maximum number of queues pairs supported by device. */
+};
+
+
+/** comp device statistics */
+struct rte_compressdev_stats {
+	uint64_t enqueued_count;
+	/**< Count of all operations enqueued */
+	uint64_t dequeued_count;
+	/**< Count of all operations dequeued */
+
+	uint64_t enqueue_err_count;
+	/**< Total error count on operations enqueued */
+	uint64_t dequeue_err_count;
+	/**< Total error count on operations dequeued */
+};
+
+
+/**
+ * Get the device identifier for the named compress device.
+ *
+ * @param name
+ *   Device name to select the device structure
+ * @return
+ *   - Returns compress device identifier on success.
+ *   - Return -1 on failure to find named compress device.
+ */
+int __rte_experimental
+rte_compressdev_get_dev_id(const char *name);
+
+/**
+ * Get the compress device name given a device identifier.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - Returns compress device name.
+ *   - Returns NULL if compress device is not present.
+ */
+const char * __rte_experimental
+rte_compressdev_name_get(uint8_t dev_id);
+
+/**
+ * Get the total number of compress devices that have been successfully
+ * initialised.
+ *
+ * @return
+ *   - The total number of usable compress devices.
+ */
+uint8_t __rte_experimental
+rte_compressdev_count(void);
+
+/**
+ * Get number of comp device defined type.
+ *
+ * @param driver_id
+ *   Driver identifier
+ * @return
+ *   Returns number of comp device.
+ */
+uint8_t __rte_experimental
+rte_compressdev_device_count_by_driver(uint8_t driver_id);
+
+/**
+ * Get number and identifiers of attached comp devices that
+ * use the same compress driver.
+ *
+ * @param driver_name
+ *   Driver name
+ * @param devices
+ *   Output devices identifiers
+ * @param nb_devices
+ *   Maximal number of devices
+ *
+ * @return
+ *   Returns number of attached compress devices.
+ */
+uint8_t __rte_experimental
+rte_compressdev_devices_get(const char *driver_name, uint8_t *devices,
+		uint8_t nb_devices);
+/*
+ * Return the NUMA socket to which a device is connected.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   The NUMA socket id to which the device is connected or
+ *   a default of zero if the socket could not be determined.
+ *   -1 if returned is the dev_id value is out of range.
+ */
+int __rte_experimental
+rte_compressdev_socket_id(uint8_t dev_id);
+
+/** Compress device configuration structure */
+struct rte_compressdev_config {
+	int socket_id;
+	/**< Socket on which to allocate resources */
+	uint16_t nb_queue_pairs;
+	/**< Total number of queue pairs to configure on a device */
+};
+
+/**
+ * Configure a device.
+ *
+ * This function must be invoked first before any other function in the
+ * API. This function can also be re-invoked when a device is in the
+ * stopped state.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param config
+ *   The compress device configuration
+ * @return
+ *   - 0: Success, device configured.
+ *   - <0: Error code returned by the driver configuration function.
+ */
+int __rte_experimental
+rte_compressdev_configure(uint8_t dev_id,
+			struct rte_compressdev_config *config);
+
+/**
+ * Start an device.
+ *
+ * The device start step is the last one and consists of setting the configured
+ * offload features and in starting the transmit and the receive units of the
+ * device.
+ * On success, all basic functions exported by the API (link status,
+ * receive/transmit, and so on) can be invoked.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - 0: Success, device started.
+ *   - <0: Error code of the driver device start function.
+ */
+int __rte_experimental
+rte_compressdev_start(uint8_t dev_id);
+
+/**
+ * Stop an device. The device can be restarted with a call to
+ * rte_compressdev_start()
+ *
+ * @param dev_id
+ *   Compress device identifier
+ */
+void __rte_experimental
+rte_compressdev_stop(uint8_t dev_id);
+
+/**
+ * Close an device. The device cannot be restarted!
+ *
+ * @param dev_id
+ *   Compress device identifier
+ *
+ * @return
+ *  - 0 on successfully closing device
+ *  - <0 on failure to close device
+ */
+int __rte_experimental
+rte_compressdev_close(uint8_t dev_id);
+
+/**
+ * Allocate and set up a receive queue pair for a device.
+ *
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param queue_pair_id
+ *   The index of the queue pairs to set up. The
+ *   value must be in the range [0, nb_queue_pair - 1]
+ *   previously supplied to rte_compressdev_configure()
+ * @param max_inflight_ops
+ *   Max number of ops which the qp will have to
+ *   accommodate simultaneously
+ * @param socket_id
+ *   The *socket_id* argument is the socket identifier
+ *   in case of NUMA. The value can be *SOCKET_ID_ANY*
+ *   if there is no NUMA constraint for the DMA memory
+ *   allocated for the receive queue pair
+ * @return
+ *   - 0: Success, queue pair correctly set up.
+ *   - <0: Queue pair configuration failed
+ */
+int __rte_experimental
+rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
+		uint32_t max_inflight_ops, int socket_id);
+
+/**
+ * Get the number of queue pairs on a specific comp device
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - The number of configured queue pairs.
+ */
+uint16_t __rte_experimental
+rte_compressdev_queue_pair_count(uint8_t dev_id);
+
+
+/**
+ * Retrieve the general I/O statistics of a device.
+ *
+ * @param dev_id
+ *   The identifier of the device
+ * @param stats
+ *   A pointer to a structure of type
+ *   *rte_compressdev_stats* to be filled with the
+ *   values of device counters
+ * @return
+ *   - Zero if successful.
+ *   - Non-zero otherwise.
+ */
+int __rte_experimental
+rte_compressdev_stats_get(uint8_t dev_id, struct rte_compressdev_stats *stats);
+
+/**
+ * Reset the general I/O statistics of a device.
+ *
+ * @param dev_id
+ *   The identifier of the device.
+ */
+void __rte_experimental
+rte_compressdev_stats_reset(uint8_t dev_id);
+
+/**
+ * Retrieve the contextual information of a device.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param dev_info
+ *   A pointer to a structure of type *rte_compressdev_info*
+ *   to be filled with the contextual information of the device
+ *
+ * @note The capabilities field of dev_info is set to point to the first
+ * element of an array of struct rte_compressdev_capabilities.
+ * The element after the last valid element has it's op field set to
+ * RTE_COMP_ALGO_LIST_END.
+ */
+void __rte_experimental
+rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info);
+
+
+typedef uint16_t (*compress_dequeue_pkt_burst_t)(void *qp,
+		struct rte_comp_op **ops, uint16_t nb_ops);
+/**< Dequeue processed packets from queue pair of a device. */
+
+typedef uint16_t (*compress_enqueue_pkt_burst_t)(void *qp,
+		struct rte_comp_op **ops, uint16_t nb_ops);
+/**< Enqueue packets for processing on queue pair of a device. */
+
+
+/** The data structure associated with each comp device. */
+struct rte_compressdev {
+	compress_dequeue_pkt_burst_t dequeue_burst;
+	/**< Pointer to PMD receive function */
+	compress_enqueue_pkt_burst_t enqueue_burst;
+	/**< Pointer to PMD transmit function */
+
+	struct rte_compressdev_data *data;
+	/**< Pointer to device data */
+	struct rte_compressdev_ops *dev_ops;
+	/**< Functions exported by PMD */
+	uint64_t feature_flags;
+	/**< Supported features */
+	struct rte_device *device;
+	/**< Backing device */
+
+	uint8_t driver_id;
+	/**< comp driver identifier*/
+
+	__extension__
+	uint8_t attached : 1;
+	/**< Flag indicating the device is attached */
+} __rte_cache_aligned;
+
+
+/**
+ *
+ * The data part, with no function pointers, associated with each device.
+ *
+ * This structure is safe to place in shared memory to be common among
+ * different processes in a multi-process configuration.
+ */
+struct rte_compressdev_data {
+	uint8_t dev_id;
+	/**< Compress device identifier */
+	uint8_t socket_id;
+	/**< Socket identifier where memory is allocated */
+	char name[RTE_COMPRESSDEV_NAME_MAX_LEN];
+	/**< Unique identifier name */
+
+	__extension__
+	uint8_t dev_started : 1;
+	/**< Device state: STARTED(1)/STOPPED(0) */
+
+	void **queue_pairs;
+	/**< Array of pointers to queue pairs. */
+	uint16_t nb_queue_pairs;
+	/**< Number of device queue pairs */
+
+	void *dev_private;
+	/**< PMD-specific private data */
+} __rte_cache_aligned;
+
+struct rte_compressdev *rte_compressdevs;
+/**
+ *
+ * Dequeue a burst of processed compression operations from a queue on the comp
+ * device. The dequeued operation are stored in *rte_comp_op* structures
+ * whose pointers are supplied in the *ops* array.
+ *
+ * The rte_compressdev_dequeue_burst() function returns the number of ops
+ * actually dequeued, which is the number of *rte_comp_op* data structures
+ * effectively supplied into the *ops* array.
+ *
+ * A return value equal to *nb_ops* indicates that the queue contained
+ * at least *nb_ops* operations, and this is likely to signify that other
+ * processed operations remain in the devices output queue. Applications
+ * implementing a "retrieve as many processed operations as possible" policy
+ * can check this specific case and keep invoking the
+ * rte_compressdev_dequeue_burst() function until a value less than
+ * *nb_ops* is returned.
+ *
+ * The rte_compressdev_dequeue_burst() function does not provide any error
+ * notification to avoid the corresponding overhead.
+ *
+ * Note: operation ordering is not maintained within the queue pair.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param qp_id
+ *   The index of the queue pair from which to retrieve
+ *   processed operations. The value must be in the range
+ *   [0, nb_queue_pair - 1] previously supplied to
+ *   rte_compressdev_configure()
+ * @param ops
+ *   The address of an array of pointers to
+ *   *rte_comp_op* structures that must be
+ *   large enough to store *nb_ops* pointers in it
+ * @param nb_ops
+ *   The maximum number of operations to dequeue
+ * @return
+ *   - The number of operations actually dequeued, which is the number
+ *   of pointers to *rte_comp_op* structures effectively supplied to the
+ *   *ops* array.
+ */
+static inline uint16_t
+rte_compressdev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	struct rte_compressdev *dev = &rte_compressdevs[dev_id];
+
+	nb_ops = (*dev->dequeue_burst)
+			(dev->data->queue_pairs[qp_id], ops, nb_ops);
+
+	return nb_ops;
+}
+
+/**
+ * Enqueue a burst of operations for processing on a compression device.
+ *
+ * The rte_compressdev_enqueue_burst() function is invoked to place
+ * comp operations on the queue *qp_id* of the device designated by
+ * its *dev_id*.
+ *
+ * The *nb_ops* parameter is the number of operations to process which are
+ * supplied in the *ops* array of *rte_comp_op* structures.
+ *
+ * The rte_compressdev_enqueue_burst() function returns the number of
+ * operations it actually enqueued for processing. A return value equal to
+ * *nb_ops* means that all packets have been enqueued.
+ *
+ * @note All compression operations are Out-of-place (OOP) operations,
+ * as the size of the output data is different to the size of the input data.
+ *
+ * @note The flush flag only applies to operations which return SUCCESS.
+ * In OUT_OF_SPACE case whether STATEFUL or STATELESS, data in dest buffer
+ * is as if flush flag was FLUSH_NONE.
+ * @note flush flag only applies in compression direction. It has no meaning
+ * for decompression.
+ * @note: operation ordering is not maintained within the queue pair.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param qp_id
+ *   The index of the queue pair on which operations
+ *   are to be enqueued for processing. The value
+ *   must be in the range [0, nb_queue_pairs - 1]
+ *   previously supplied to *rte_compressdev_configure*
+ * @param ops
+ *   The address of an array of *nb_ops* pointers
+ *   to *rte_comp_op* structures which contain
+ *   the operations to be processed
+ * @param nb_ops
+ *   The number of operations to process
+ * @return
+ *   The number of operations actually enqueued on the device. The return
+ *   value can be less than the value of the *nb_ops* parameter when the
+ *   comp devices queue is full or if invalid parameters are specified in
+ *   a *rte_comp_op*.
+ */
+static inline uint16_t
+rte_compressdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	struct rte_compressdev *dev = &rte_compressdevs[dev_id];
+
+	return (*dev->enqueue_burst)(
+			dev->data->queue_pairs[qp_id], ops, nb_ops);
+}
+
+
+/** Compressdev session */
+struct rte_comp_session {
+	__extension__ void *sess_private_data[0];
+	/**< Private session material */
+};
+
+
+/**
+ * Create compression session header (generic with no private data)
+ *
+ * @param mempool
+ *   Symmetric session mempool to allocate session objects from
+ * @return
+ *  - On success return pointer to compression session
+ *  - On failure returns NULL
+ */
+struct rte_comp_session * __rte_experimental
+rte_compressdev_session_create(struct rte_mempool *mempool);
+
+/**
+ * Frees comp session header, after checking that all
+ * the device private data has been freed, returning it
+ * to its original mempool.
+ *
+ * @param sess
+ *   Session header to be freed
+ *
+ * @return
+ *  - 0 if successful.
+ *  - -EINVAL if session is NULL.
+ *  - -EBUSY if not all device private data has been freed.
+ */
+int __rte_experimental
+rte_compressdev_session_terminate(struct rte_comp_session *sess);
+
+/**
+ * Fill out private session data for the device, based on its device id.
+ * The same private session data is shared by all devices exposed by a driver
+ * so this only needs to be called for one device per driver-type.
+ * All private data stored must be shareable across devices, so read-only.
+ * A session initialised for more than one device (of different driver types)
+ * must used the same xform for each init.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param sess
+ *   Session where the private data will be attached to
+ * @param xforms
+ *   Compress transform operations to apply on flow
+ *   processed with this session
+ * @param mempool
+ *   Mempool from where the private data should be allocated
+ *
+ * @return
+ *  - On success, zero.
+ *  - -EINVAL if input parameters are invalid.
+ *  - -ENOTSUP if comp device does not support the comp transform.
+ *  - -ENOMEM if the private session could not be allocated.
+ */
+int __rte_experimental
+rte_compressdev_session_init(uint8_t dev_id,
+			struct rte_comp_session *sess,
+			struct rte_comp_xform *xforms,
+			struct rte_mempool *mempool);
+
+/**
+ * Frees private data for the device id, based on its device type,
+ * returning it to its mempool.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param sess
+ *   Session containing the reference to the private data
+ *
+ * @return
+ *  - 0 if successful.
+ *  - -EINVAL if device is invalid or session is NULL.
+ */
+int __rte_experimental
+rte_compressdev_session_clear(uint8_t dev_id,
+			struct rte_comp_session *sess);
+
+/**
+ * Get the size of the header session, for all registered drivers.
+ *
+ * @return
+ *   Size of the header session.
+ */
+unsigned int __rte_experimental
+rte_compressdev_get_header_session_size(void);
+
+/**
+ * Get the size of the private session data for a device.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ *
+ * @return
+ *   - Size of the private data, if successful
+ *   - 0 if device is invalid or does not have private session
+ */
+unsigned int __rte_experimental
+rte_compressdev_get_private_session_size(uint8_t dev_id);
+
+/**
+ * This should alloc a stream from the device's mempool and initialise it.
+ * This handle will be passed to the PMD with every op in the stream.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param session
+ *   Session previously allocated by
+ *   *rte_compressdev_session_create*
+ * @param stream
+ *   Pointer to PMD's private stream data
+ * @param op_type
+ *   Op type for which the stream will be used
+ *
+ * @return
+ *
+ * TODO: Should qp_id also be added, with constraint that all ops in the same
+ * stream should be sent to the same qp?
+ *
+ */
+int __rte_experimental
+rte_comp_stream_create(uint8_t dev_id, struct rte_comp_session *sess,
+			void **stream, enum rte_comp_op_type op_type);
+
+/**
+ * This should clear the stream and return it to the device's mempool.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ *
+ * @param stream
+ *   PMD's private stream data
+ *
+ *
+ * @return
+ */
+int __rte_experimental
+rte_comp_stream_free(uint8_t dev_id, void *stream);
+
+/**
+ * Provide driver identifier.
+ *
+ * @param name
+ *   Compress driver name
+ * @return
+ *  The driver type identifier or -1 if no driver found
+ */
+int __rte_experimental
+rte_compressdev_driver_id_get(const char *name);
+
+/**
+ * Provide driver name.
+ *
+ * @param driver_id
+ *   The driver identifier
+ * @return
+ *  The driver name or null if no driver found
+ */
+const char * __rte_experimental
+rte_compressdev_driver_name_get(uint8_t driver_id);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_COMPRESSDEV_H_ */
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.c b/lib/librte_compressdev/rte_compressdev_pmd.c
new file mode 100644
index 0000000..7f032b9
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_pmd.c
@@ -0,0 +1,163 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#include <rte_malloc.h>
+
+#include "rte_compressdev_pmd.h"
+
+int compressdev_logtype;
+
+/**
+ * Parse name from argument
+ */
+static int
+rte_compressdev_pmd_parse_name_arg(const char *key __rte_unused,
+		const char *value, void *extra_args)
+{
+	struct rte_compressdev_pmd_init_params *params = extra_args;
+	int n;
+
+	n = snprintf(params->name, RTE_COMPRESSDEV_NAME_MAX_LEN, "%s", value);
+	if (n >= RTE_COMPRESSDEV_NAME_MAX_LEN)
+		return -EINVAL;
+
+	return 0;
+}
+
+/**
+ * Parse unsigned integer from argument
+ */
+static int
+rte_compressdev_pmd_parse_uint_arg(const char *key __rte_unused,
+		const char *value, void *extra_args)
+{
+	int i;
+	char *end;
+
+	errno = 0;
+	i = strtol(value, &end, 10);
+	if (*end != 0 || errno != 0 || i < 0)
+		return -EINVAL;
+
+	*((uint32_t *)extra_args) = i;
+	return 0;
+}
+
+int
+rte_compressdev_pmd_parse_input_args(
+		struct rte_compressdev_pmd_init_params *params,
+		const char *args)
+{
+	struct rte_kvargs *kvlist = NULL;
+	int ret = 0;
+
+	if (params == NULL)
+		return -EINVAL;
+
+	if (args) {
+		kvlist = rte_kvargs_parse(args,	compressdev_pmd_valid_params);
+		if (kvlist == NULL)
+			return -EINVAL;
+
+		ret = rte_kvargs_process(kvlist,
+				RTE_COMPRESSDEV_PMD_MAX_NB_QP_ARG,
+				&rte_compressdev_pmd_parse_uint_arg,
+				&params->max_nb_queue_pairs);
+		if (ret < 0)
+			goto free_kvlist;
+
+		ret = rte_kvargs_process(kvlist,
+				RTE_COMPRESSDEV_PMD_SOCKET_ID_ARG,
+				&rte_compressdev_pmd_parse_uint_arg,
+				&params->socket_id);
+		if (ret < 0)
+			goto free_kvlist;
+
+		ret = rte_kvargs_process(kvlist,
+				RTE_COMPRESSDEV_PMD_NAME_ARG,
+				&rte_compressdev_pmd_parse_name_arg,
+				params);
+		if (ret < 0)
+			goto free_kvlist;
+	}
+
+free_kvlist:
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_create(const char *name,
+		struct rte_device *device,
+		struct rte_compressdev_pmd_init_params *params)
+{
+	struct rte_compressdev *compressdev;
+
+	if (params->name[0] != '\0') {
+		COMPRESSDEV_LOG(INFO, "[%s] User specified device name = %s\n",
+				device->driver->name, params->name);
+		name = params->name;
+	}
+
+	COMPRESSDEV_LOG(INFO, "[%s] - Creating compressdev %s\n",
+			device->driver->name, name);
+
+	COMPRESSDEV_LOG(INFO,
+	"[%s] - Init parameters - name: %s, socket id: %d, max queue pairs: %u",
+			device->driver->name, name,
+			params->socket_id, params->max_nb_queue_pairs);
+
+	/* allocate device structure */
+	compressdev = rte_compressdev_pmd_allocate(name, params->socket_id);
+	if (compressdev == NULL) {
+		COMPRESSDEV_LOG(ERR, "[%s] Failed to allocate comp device for %s",
+				device->driver->name, name);
+		return NULL;
+	}
+
+	/* allocate private device structure */
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		compressdev->data->dev_private =
+				rte_zmalloc_socket("compressdev device private",
+						params->private_data_size,
+						RTE_CACHE_LINE_SIZE,
+						params->socket_id);
+
+		if (compressdev->data->dev_private == NULL) {
+			COMPRESSDEV_LOG(ERR,
+		"[%s] Cannot allocate memory for compressdev %s private data",
+					device->driver->name, name);
+
+			rte_compressdev_pmd_release_device(compressdev);
+			return NULL;
+		}
+	}
+
+	compressdev->device = device;
+
+	return compressdev;
+}
+
+int __rte_experimental
+rte_compressdev_pmd_destroy(struct rte_compressdev *compressdev)
+{
+	int retval;
+
+	COMPRESSDEV_LOG(INFO, "[%s] Closing comp device %s",
+			compressdev->device->driver->name,
+			compressdev->device->name);
+
+	/* free comp device */
+	retval = rte_compressdev_pmd_release_device(compressdev);
+	if (retval)
+		return retval;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		rte_free(compressdev->data->dev_private);
+
+	compressdev->device = NULL;
+	compressdev->data = NULL;
+
+	return 0;
+}
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
new file mode 100644
index 0000000..0e30cb4
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -0,0 +1,439 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMPRESSDEV_PMD_H_
+#define _RTE_COMPRESSDEV_PMD_H_
+
+/** @file
+ * RTE comp PMD APIs
+ *
+ * @note
+ * These APIs are for comp PMDs only and user applications should not call
+ * them directly.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <string.h>
+
+#include <rte_dev.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_mempool.h>
+#include <rte_log.h>
+#include <rte_common.h>
+
+#include "rte_comp.h"
+#include "rte_compressdev.h"
+
+#define RTE_COMPRESSDEV_PMD_DEFAULT_MAX_NB_QPS		8
+
+#define RTE_COMPRESSDEV_PMD_NAME_ARG			("name")
+#define RTE_COMPRESSDEV_PMD_MAX_NB_QP_ARG		("max_nb_queue_pairs")
+#define RTE_COMPRESSDEV_PMD_SOCKET_ID_ARG		("socket_id")
+
+static const char * const compressdev_pmd_valid_params[] = {
+	RTE_COMPRESSDEV_PMD_NAME_ARG,
+	RTE_COMPRESSDEV_PMD_MAX_NB_QP_ARG,
+	RTE_COMPRESSDEV_PMD_SOCKET_ID_ARG
+};
+
+/**
+ * @internal
+ * Initialisation parameters for comp devices
+ */
+struct rte_compressdev_pmd_init_params {
+	char name[RTE_COMPRESSDEV_NAME_MAX_LEN];
+	size_t private_data_size;
+	int socket_id;
+	unsigned int max_nb_queue_pairs;
+};
+
+/** Global structure used for maintaining state of allocated comp devices */
+struct rte_compressdev_global {
+	struct rte_compressdev *devs;	/**< Device information array */
+	struct rte_compressdev_data *data[RTE_COMPRESS_MAX_DEVS];
+	/**< Device private data */
+	uint8_t nb_devs;		/**< Number of devices found */
+	uint8_t max_devs;		/**< Max number of devices */
+};
+
+/* compressdev driver, containing the driver identifier */
+struct compressdev_driver {
+	TAILQ_ENTRY(compressdev_driver) next; /**< Next in list. */
+	const struct rte_driver *driver;
+	uint8_t id;
+};
+
+/** Pointer to global array of comp devices */
+extern struct rte_compressdev *rte_compressdevs;
+/** Pointer to global comp devices data structure */
+extern struct rte_compressdev_global *rte_compressdev_globals;
+
+/**
+ * Get the rte_compressdev structure device pointer for the device. Assumes a
+ * valid device index.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - The rte_compressdev structure pointer for the given device identifier.
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_get_dev(uint8_t dev_id);
+
+/**
+ * Get the rte_compressdev structure device pointer for the named device.
+ *
+ * @param name
+ *   Compress device name
+ * @return
+ *   - The rte_compressdev structure pointer for the given device identifier.
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_get_named_dev(const char *name);
+
+/**
+ * Validate if the comp device index is valid attached comp device.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - If the device index is valid (1) or not (0).
+ */
+unsigned int __rte_experimental
+rte_compressdev_pmd_is_valid_dev(uint8_t dev_id);
+
+/**
+ * Definitions of all functions exported by a driver through the
+ * the generic structure of type *comp_dev_ops* supplied in the
+ * *rte_compressdev* structure associated with a device.
+ */
+
+/**
+ *	Function used to configure device.
+ *
+ * @param dev
+ *   Compress device
+ * @param config
+ *   Compress device configurations
+ * @return
+ *   Returns 0 on success
+ */
+typedef int (*compressdev_configure_t)(struct rte_compressdev *dev,
+		struct rte_compressdev_config *config);
+
+/**
+ * Function used to start a configured device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *   Returns 0 on success
+ */
+typedef int (*compressdev_start_t)(struct rte_compressdev *dev);
+
+/**
+ * Function used to stop a configured device.
+ *
+ * @param dev
+ *   Compress device
+ */
+typedef void (*compressdev_stop_t)(struct rte_compressdev *dev);
+
+/**
+ * Function used to close a configured device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ * - 0 on success.
+ * - EAGAIN if can't close as device is busy
+ */
+typedef int (*compressdev_close_t)(struct rte_compressdev *dev);
+
+
+/**
+ * Function used to get statistics of a device.
+ *
+ * @param dev
+ *   Compress device
+ * @param stats
+ *   Compress device stats to populate
+ */
+typedef void (*compressdev_stats_get_t)(struct rte_compressdev *dev,
+				struct rte_compressdev_stats *stats);
+
+
+/**
+ * Function used to reset statistics of a device.
+ *
+ * @param dev
+ *   Compress device
+ */
+typedef void (*compressdev_stats_reset_t)(struct rte_compressdev *dev);
+
+
+/**
+ * Function used to get specific information of a device.
+ *
+ * @param dev
+ *   Compress device
+ */
+typedef void (*compressdev_info_get_t)(struct rte_compressdev *dev,
+				struct rte_compressdev_info *dev_info);
+
+/**
+ * Setup a queue pair for a device.
+ *
+ * @param dev
+ *   Compress device
+ * @param qp_id
+ *   Queue pair identifier
+ * @param max_inflight_ops
+ *   Max inflight ops which qp must accommodate
+ * @param socket_id
+ *   Socket identifier
+ * @return
+ *   Returns 0 on success.
+ */
+typedef int (*compressdev_queue_pair_setup_t)(struct rte_compressdev *dev,
+		uint16_t qp_id,	uint32_t max_inflight_ops, int socket_id);
+
+/**
+ * Release memory resources allocated by given queue pair.
+ *
+ * @param dev
+ *   Compress device
+ * @param qp_id
+ *   Queue pair identifier
+ * @return
+ * - 0 on success.
+ * - EAGAIN if can't close as device is busy
+ */
+typedef int (*compressdev_queue_pair_release_t)(struct rte_compressdev *dev,
+		uint16_t qp_id);
+
+/**
+ * Get number of available queue pairs of a device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *   Returns number of queue pairs on success.
+ */
+typedef uint32_t (*compressdev_queue_pair_count_t)(struct rte_compressdev *dev);
+
+/**
+ * Get the size of a compressdev session
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *  - On success returns the size of the session structure for device
+ *  - On failure returns 0
+ */
+typedef unsigned int (*compressdev_get_session_private_size_t)(
+		struct rte_compressdev *dev);
+
+/**
+ * Configure a comp session on a device.
+ *
+ * @param dev
+ *   Compress device
+ * @param xform
+ *   Single or chain of comp xforms
+ * @param session
+ *   Compressdev session
+ * @param mp
+ *   Mempool where the private session is allocated
+ * @return
+ *  - Returns 0 if private session structure have been created successfully.
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private session could not be allocated.
+ */
+typedef int (*compressdev_configure_session_t)(struct rte_compressdev *dev,
+		struct rte_comp_xform *xform,
+		struct rte_comp_session *session,
+		struct rte_mempool *mp);
+
+/**
+ * Free driver private session data.
+ *
+ * @param dev
+ *   Compress device
+ * @param sess
+ *   Compressdev session
+ */
+typedef void (*compressdev_free_session_t)(struct rte_compressdev *dev,
+		struct rte_comp_session *sess);
+
+/** comp device operations function pointer table */
+struct rte_compressdev_ops {
+	compressdev_configure_t dev_configure;	/**< Configure device. */
+	compressdev_start_t dev_start;		/**< Start device. */
+	compressdev_stop_t dev_stop;		/**< Stop device. */
+	compressdev_close_t dev_close;		/**< Close device. */
+
+	compressdev_info_get_t dev_infos_get;	/**< Get device info. */
+
+	compressdev_stats_get_t stats_get;
+	/**< Get device statistics. */
+	compressdev_stats_reset_t stats_reset;
+	/**< Reset device statistics. */
+
+	compressdev_queue_pair_setup_t queue_pair_setup;
+	/**< Set up a device queue pair. */
+	compressdev_queue_pair_release_t queue_pair_release;
+	/**< Release a queue pair. */
+	compressdev_queue_pair_count_t queue_pair_count;
+	/**< Get count of the queue pairs. */
+
+	compressdev_get_session_private_size_t session_get_size;
+	/**< Return private session. */
+	compressdev_configure_session_t session_configure;
+	/**< Configure a comp session. */
+	compressdev_free_session_t session_clear;
+	/**< Clear a comp sessions private data. */
+};
+
+
+/**
+ * @internal
+ *
+ * Function for internal use by dummy drivers primarily, e.g. ring-based
+ * driver.
+ * Allocates a new compressdev slot for an comp device and returns the pointer
+ * to that slot for the driver to use.
+ *
+ * @param name
+ *   Unique identifier name for each device
+ * @param socket_id
+ *   Socket to allocate resources on
+ * @return
+ *   - Slot in the rte_dev_devices array for a new device;
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_allocate(const char *name, int socket_id);
+
+/**
+ * @internal
+ *
+ * Function for internal use by dummy drivers primarily, e.g. ring-based
+ * driver.
+ * Release the specified compressdev device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *   - 0 on success, negative on error
+ */
+int __rte_experimental
+rte_compressdev_pmd_release_device(struct rte_compressdev *dev);
+
+
+/**
+ * @internal
+ *
+ * PMD assist function to parse initialisation arguments for comp driver
+ * when creating a new comp PMD device instance.
+ *
+ * PMD driver should set default values for that PMD before calling function,
+ * these default values will be over-written with successfully parsed values
+ * from args string.
+ *
+ * @param params
+ *   Parsed PMD initialisation parameters
+ * @param args
+ *   Input argument string to parse
+ * @return
+ *  - 0 on success
+ *  - errno on failure
+ */
+int __rte_experimental
+rte_compressdev_pmd_parse_input_args(
+		struct rte_compressdev_pmd_init_params *params,
+		const char *args);
+
+/**
+ * @internal
+ *
+ * PMD assist function to provide boiler plate code for comp driver to create
+ * and allocate resources for a new comp PMD device instance.
+ *
+ * @param name
+ *   Compress device name
+ * @param device
+ *   Base device instance
+ * @param params
+ *   PMD initialisation parameters
+ * @return
+ *  - comp device instance on success
+ *  - NULL on creation failure
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_create(const char *name,
+		struct rte_device *device,
+		struct rte_compressdev_pmd_init_params *params);
+
+/**
+ * @internal
+ *
+ * PMD assist function to provide boiler plate code for comp driver to
+ * destroy and free resources associated with a comp PMD device instance.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *  - 0 on success
+ *  - errno on failure
+ */
+int __rte_experimental
+rte_compressdev_pmd_destroy(struct rte_compressdev *dev);
+
+
+/**
+ * @internal
+ * Allocate compressdev driver.
+ *
+ * @param comp_drv
+ *   Compressdev driver
+ * @param drv
+ *   Rte_driver
+ * @return
+ *  The driver type identifier
+ */
+uint8_t __rte_experimental
+rte_compressdev_allocate_driver(struct compressdev_driver *comp_drv,
+		const struct rte_driver *drv);
+
+
+#define RTE_PMD_REGISTER_COMPRESSDEV_DRIVER(comp_drv, drv, driver_id)\
+RTE_INIT(init_ ##driver_id);\
+static void init_ ##driver_id(void)\
+{\
+	driver_id = rte_compressdev_allocate_driver(&comp_drv, &(drv).driver);\
+}
+
+static inline void *
+get_session_private_data(const struct rte_comp_session *sess,
+		uint8_t driver_id) {
+	return sess->sess_private_data[driver_id];
+}
+
+static inline void
+set_session_private_data(struct rte_comp_session *sess,
+		uint8_t driver_id, void *private_data)
+{
+	sess->sess_private_data[driver_id] = private_data;
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_COMPRESSDEV_PMD_H_ */
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
new file mode 100644
index 0000000..4554cbb
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -0,0 +1,47 @@
+EXPERIMENTAL {
+        global:
+
+        rte_compressdevs;
+	rte_compressdev_allocate_driver;
+	rte_compressdev_callback_register;
+	rte_compressdev_callback_unregister;
+	rte_compressdev_close;
+	rte_compressdev_configure;
+	rte_compressdev_count;
+	rte_compressdev_device_count_by_driver;
+	rte_compressdev_devices_get;
+	rte_compressdev_driver_id_get;
+	rte_compressdev_driver_name_get;
+	rte_compressdev_get_dev_id;
+	rte_compressdev_get_feature_name;
+	rte_compressdev_get_header_session_size;
+	rte_compressdev_get_private_session_size;
+	rte_compressdev_info_get;
+	rte_compressdev_name_get;
+	rte_compressdev_pmd_allocate;
+	rte_compressdev_pmd_callback_process;
+	rte_compressdev_pmd_create;
+	rte_compressdev_pmd_destroy;
+	rte_compressdev_pmd_get_dev;
+	rte_compressdev_pmd_get_named_dev;
+	rte_compressdev_pmd_is_valid_dev;
+	rte_compressdev_pmd_parse_input_args;
+	rte_compressdev_pmd_release_device;
+	rte_compressdev_queue_pair_count;
+	rte_compressdev_queue_pair_setup;
+	rte_compressdev_session_create;
+	rte_compressdev_session_init;
+	rte_compressdev_session_clear;
+	rte_compressdev_session_terminate;
+	rte_compressdev_socket_id;
+	rte_compressdev_start;
+	rte_compressdev_stats_get;
+	rte_compressdev_stats_reset;
+	rte_compressdev_stop;
+	rte_comp_get_feature_name;
+	rte_comp_op_pool_create;
+	rte_comp_stream_create;
+	rte_comp_stream_free;
+
+        local: *;
+};
diff --git a/lib/librte_eal/common/include/rte_log.h b/lib/librte_eal/common/include/rte_log.h
index 9029c78..30a88ff 100644
--- a/lib/librte_eal/common/include/rte_log.h
+++ b/lib/librte_eal/common/include/rte_log.h
@@ -60,6 +60,7 @@ extern struct rte_logs rte_logs;
 #define RTE_LOGTYPE_EFD       18 /**< Log related to EFD. */
 #define RTE_LOGTYPE_EVENTDEV  19 /**< Log related to eventdev. */
 #define RTE_LOGTYPE_GSO       20 /**< Log related to GSO. */
+#define RTE_LOGTYPE_COMPRESSDEV 21 /**< Log related to compressdev. */
 
 /* these log types can be used in an application */
 #define RTE_LOGTYPE_USER1     24 /**< User-defined log type 1. */
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 3eb41d1..cff710a 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -94,6 +94,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF)           += -lrte_mbuf
 _LDLIBS-$(CONFIG_RTE_LIBRTE_NET)            += -lrte_net
 _LDLIBS-$(CONFIG_RTE_LIBRTE_ETHER)          += -lrte_ethdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_BBDEV)          += -lrte_bbdev
+_LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV)    += -lrte_compressdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_CRYPTODEV)      += -lrte_cryptodev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY)       += -lrte_security
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV)       += -lrte_eventdev
-- 
2.5.0

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Thomas Monjalon]

02/02/2018 19:25, Fiona Trahe:
> - Used dynamic logging
[...]
> --- a/lib/librte_eal/common/include/rte_log.h
> +++ b/lib/librte_eal/common/include/rte_log.h
> @@ -60,6 +60,7 @@ extern struct rte_logs rte_logs;
>  #define RTE_LOGTYPE_EFD       18 /**< Log related to EFD. */
>  #define RTE_LOGTYPE_EVENTDEV  19 /**< Log related to eventdev. */
>  #define RTE_LOGTYPE_GSO       20 /**< Log related to GSO. */
> +#define RTE_LOGTYPE_COMPRESSDEV 21 /**< Log related to compressdev. */

You don't need this define with dynamic logging.

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Trahe, Fiona]

We've been struggling with the idea of session in compressdev.

Is it really a session? 
 - It's not in the same sense as cryptodev where it's used to hold a key, and maps to a Security Association.
 - It's a set of immutable data that is needed with the op and stream to perform the operation.
 - It inherited from cryptodev the ability to be set up for multiple driver types and used across any 
    devices of those types. For stateful ops this facility can't be used. 
    For stateless we don't think it's important, and think it's unlikely to be used.
 - Drivers use it to prepare private data, set up resources, do pre-work, so there's
    less work to be done on the data path. Initially we didn't have a stream, we do now,
    this may be a better alternative place for that work.
So we've been toying with the idea of getting rid of the session. 

We also struggle with the idea of setting up a stream for stateless ops.
  - Well, really I just think the name is misleading, i.e. there's no problem with setting 
    up some private PMD data to use with stateless operations, just calling it a
    stream doesn't seem right.

So putting above thoughts together I want to propose:
-	Removal of the session and all associated APIs.
-	Passing in one of three data types in the rte_comp_op
    
    union {
        struct rte_comp_xform *xform;
        /**< Immutable compress/decompress params */
        void *pmd_stateless_data;
        /**< Stateless private PMD data derived from an rte_comp_xform
         * rte_comp_stateless_data_init() must be called on a device 
         * before sending any STATELESS operations. If the PMD returns a non-NULL
         * value the handle must be attached to subsequent STATELESS operations.
         * If a PMD returns NULL, then the xform should be passed directly to each op 
         */
        void *stream;
        /* Private PMD data derived initially from an rte_comp_xform, which holds state
         * and history data and evolves as operations are processed.
         * rte_comp_stream_create() must be called on a device for all STATEFUL 
         * data streams and the resulting stream attached
         * to the one or more operations associated with the data stream.
         * All operations in a stream must be sent to the same device.
         */
    }

Notes: 
1. Internally if a PMD wants to use the exact same data structure for both it can do, 
     just on the API I think it's better if they're named differently with 
     different comments.
2. I'm not clear of the constraints if any, which attach to the pmd_stateless_data
     For our PMD it would only hold immutable data as the session did, and so
     could be attached to many ops in parallel. 
     Is this true for all PMDs or are there constraints which should be called out?
     Is it limited to a specific device, qp, or to be used on one op at a time?
3. Am open to other naming suggestions, just trying to capture the essence
    of these data structs better than our current API does.

We would put some more helper fns and structure around the above code if people
are in agreement, just want to see if the concept flies before going further?

Fiona
 

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Ahmed Mansour]

Hi Fiona,

Thanks for starting this discussion. In the current API the user must
make 12 API calls just to get information to compress. Maybe there is a
way to simplify. At least for some use cases (stateless). I think a call
sometime next week would be good to help clarify coalesce some of the
complexity.

I added specific comments inline.

Thanks,

Ahmed

On 2/21/2018 2:12 PM, Trahe, Fiona wrote:
> We've been struggling with the idea of session in compressdev.
>
> Is it really a session? 
>  - It's not in the same sense as cryptodev where it's used to hold a key, and maps to a Security Association.
>  - It's a set of immutable data that is needed with the op and stream to perform the operation.
>  - It inherited from cryptodev the ability to be set up for multiple driver types and used across any 
>     devices of those types. For stateful ops this facility can't be used. 
>     For stateless we don't think it's important, and think it's unlikely to be used.
>  - Drivers use it to prepare private data, set up resources, do pre-work, so there's
>     less work to be done on the data path. Initially we didn't have a stream, we do now,
>     this may be a better alternative place for that work.
> So we've been toying with the idea of getting rid of the session. 
[Ahmed] In our proprietary API the stream and session are one. A session
holds many properties like the op-type, instead of having this
information in the op itself.  This way we lower the per op setup cost.
This also allows rapid reuse of stateful infrastructure, once a stream
is closed on a stateful session, the next op (stream) on this session
reuses the stateful storage. Obviously if a stream is in "pause mode" on
a session, all following ops that may be unrelated to this
stream/session must also wait until this current stream is closed or
aborted before the infrastructure can be reused.
>
> We also struggle with the idea of setting up a stream for stateless ops.
>   - Well, really I just think the name is misleading, i.e. there's no problem with setting 
>     up some private PMD data to use with stateless operations, just calling it a
>     stream doesn't seem right.
[Ahmed] I agree. The op has all the necessary information to process it
in the current API? Both the stream and the op are one time use. We
can't attach multiple similar ops to a single stream/session and rely on
their properties to simplify op setup, so why the hassle.
>
> So putting above thoughts together I want to propose:
> -	Removal of the session and all associated APIs.
> -	Passing in one of three data types in the rte_comp_op
>     
>     union {
>         struct rte_comp_xform *xform;
>         /**< Immutable compress/decompress params */
>         void *pmd_stateless_data;
>         /**< Stateless private PMD data derived from an rte_comp_xform
>          * rte_comp_stateless_data_init() must be called on a device 
>          * before sending any STATELESS operations. If the PMD returns a non-NULL
>          * value the handle must be attached to subsequent STATELESS operations.
>          * If a PMD returns NULL, then the xform should be passed directly to each op 
>          */
>         void *stream;
>         /* Private PMD data derived initially from an rte_comp_xform, which holds state
>          * and history data and evolves as operations are processed.
>          * rte_comp_stream_create() must be called on a device for all STATEFUL 
>          * data streams and the resulting stream attached
>          * to the one or more operations associated with the data stream.
>          * All operations in a stream must be sent to the same device.
>          */
>     }
[Ahmed] I like this setup, but I am not sure in what cases the xform
immutable would be used. I understand the other two.
> Notes: 
> 1. Internally if a PMD wants to use the exact same data structure for both it can do, 
>      just on the API I think it's better if they're named differently with 
>      different comments.
> 2. I'm not clear of the constraints if any, which attach to the pmd_stateless_data
>      For our PMD it would only hold immutable data as the session did, and so
>      could be attached to many ops in parallel.
>      Is this true for all PMDs or are there constraints which should be called out?
>      Is it limited to a specific device, qp, or to be used on one op at a time?
> 3. Am open to other naming suggestions, just trying to capture the essence
>     of these data structs better than our current API does.
>
> We would put some more helper fns and structure around the above code if people
> are in agreement, just want to see if the concept flies before going further?
>
> Fiona
>  
>
>

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Trahe, Fiona]

> -----Original Message-----
> From: Ahmed Mansour [mailto:ahmed.mansour@nxp.com]
> Sent: Saturday, February 24, 2018 1:17 AM
> To: Trahe, Fiona <fiona.trahe@intel.com>; dev@dpdk.org; Shally.Verma@cavium.com
> Cc: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Athreya, Narayana Prasad
> <NarayanaPrasad.Athreya@cavium.com>; Gupta, Ashish <Ashish.Gupta@cavium.com>; Sahu, Sunila
> <Sunila.Sahu@cavium.com>; Challa, Mahipal <Mahipal.Challa@cavium.com>; Jain, Deepak K
> <deepak.k.jain@intel.com>; Hemant Agrawal <hemant.agrawal@nxp.com>; Roy Pledge
> <roy.pledge@nxp.com>; Youri Querry <youri.querry_1@nxp.com>
> Subject: Re: [dpdk-dev] [PATCH] compressdev: implement API
> 
> Hi Fiona,
> 
> Thanks for starting this discussion. In the current API the user must
> make 12 API calls just to get information to compress. Maybe there is a
> way to simplify. At least for some use cases (stateless). I think a call
> sometime next week would be good to help clarify coalesce some of the
> complexity.
[Fiona] Would 10:30 GMT on Wednesday 28th Feb suit?
> 
> I added specific comments inline.
> 
> Thanks,
> 
> Ahmed
> 
> On 2/21/2018 2:12 PM, Trahe, Fiona wrote:
> > We've been struggling with the idea of session in compressdev.
> >
> > Is it really a session?
> >  - It's not in the same sense as cryptodev where it's used to hold a key, and maps to a Security
> Association.
> >  - It's a set of immutable data that is needed with the op and stream to perform the operation.
> >  - It inherited from cryptodev the ability to be set up for multiple driver types and used across any
> >     devices of those types. For stateful ops this facility can't be used.
> >     For stateless we don't think it's important, and think it's unlikely to be used.
> >  - Drivers use it to prepare private data, set up resources, do pre-work, so there's
> >     less work to be done on the data path. Initially we didn't have a stream, we do now,
> >     this may be a better alternative place for that work.
> > So we've been toying with the idea of getting rid of the session.
> [Ahmed] In our proprietary API the stream and session are one. A session
> holds many properties like the op-type, instead of having this
> information in the op itself.  This way we lower the per op setup cost.
> This also allows rapid reuse of stateful infrastructure, once a stream
> is closed on a stateful session, the next op (stream) on this session
> reuses the stateful storage. Obviously if a stream is in "pause mode" on
> a session, all following ops that may be unrelated to this
> stream/session must also wait until this current stream is closed or
> aborted before the infrastructure can be reused.
> >
> > We also struggle with the idea of setting up a stream for stateless ops.
> >   - Well, really I just think the name is misleading, i.e. there's no problem with setting
> >     up some private PMD data to use with stateless operations, just calling it a
> >     stream doesn't seem right.
> [Ahmed] I agree. The op has all the necessary information to process it
> in the current API? Both the stream and the op are one time use. We
> can't attach multiple similar ops to a single stream/session and rely on
> their properties to simplify op setup, so why the hassle.
> >
> > So putting above thoughts together I want to propose:
> > -	Removal of the session and all associated APIs.
> > -	Passing in one of three data types in the rte_comp_op
> >
> >     union {
> >         struct rte_comp_xform *xform;
> >         /**< Immutable compress/decompress params */
> >         void *pmd_stateless_data;
> >         /**< Stateless private PMD data derived from an rte_comp_xform
> >          * rte_comp_stateless_data_init() must be called on a device
> >          * before sending any STATELESS operations. If the PMD returns a non-NULL
> >          * value the handle must be attached to subsequent STATELESS operations.
> >          * If a PMD returns NULL, then the xform should be passed directly to each op
> >          */
> >         void *stream;
> >         /* Private PMD data derived initially from an rte_comp_xform, which holds state
> >          * and history data and evolves as operations are processed.
> >          * rte_comp_stream_create() must be called on a device for all STATEFUL
> >          * data streams and the resulting stream attached
> >          * to the one or more operations associated with the data stream.
> >          * All operations in a stream must be sent to the same device.
> >          */
> >     }
> [Ahmed] I like this setup, but I am not sure in what cases the xform
> immutable would be used. I understand the other two.
[Fiona] The xform is there because I don't know yet what limitations may apply to the
pmd_stateless_data. If it has no limitation and once set up once on a device 
can be attached simultaneously to any op sent to any qp on that device
then we don't need the xform. But I understood from Shally's earlier request for
setting up a stream on a stateless request that some resources are
allocated, so we may need to document some limitations.
In this case the xform may be a better path for PMDs which don't have the same
limitations.  


> > Notes:
> > 1. Internally if a PMD wants to use the exact same data structure for both it can do,
> >      just on the API I think it's better if they're named differently with
> >      different comments.
> > 2. I'm not clear of the constraints if any, which attach to the pmd_stateless_data
> >      For our PMD it would only hold immutable data as the session did, and so
> >      could be attached to many ops in parallel.
> >      Is this true for all PMDs or are there constraints which should be called out?
> >      Is it limited to a specific device, qp, or to be used on one op at a time?
> > 3. Am open to other naming suggestions, just trying to capture the essence
> >     of these data structs better than our current API does.
> >
> > We would put some more helper fns and structure around the above code if people
> > are in agreement, just want to see if the concept flies before going further?
> >
> > Fiona
> >
> >
> >

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Ahmed Mansour]

>> Hi Fiona,
>>
>> Thanks for starting this discussion. In the current API the user must
>> make 12 API calls just to get information to compress. Maybe there is a
>> way to simplify. At least for some use cases (stateless). I think a call
>> sometime next week would be good to help clarify coalesce some of the
>> complexity.
> [Fiona] Would 10:30 GMT on Wednesday 28th Feb suit?
[Ahmed] I am in Ottawa Canada so Wednesday 12:00 or 1:00 GMT would be
better. Does that work?
>> I added specific comments inline.
>>
>> Thanks,
>>
>> Ahmed
>>
>> On 2/21/2018 2:12 PM, Trahe, Fiona wrote:
>>> We've been struggling with the idea of session in compressdev.
>>>
>>> Is it really a session?
>>>  - It's not in the same sense as cryptodev where it's used to hold a key, and maps to a Security
>> Association.
>>>  - It's a set of immutable data that is needed with the op and stream to perform the operation.
>>>  - It inherited from cryptodev the ability to be set up for multiple driver types and used across any
>>>     devices of those types. For stateful ops this facility can't be used.
>>>     For stateless we don't think it's important, and think it's unlikely to be used.
>>>  - Drivers use it to prepare private data, set up resources, do pre-work, so there's
>>>     less work to be done on the data path. Initially we didn't have a stream, we do now,
>>>     this may be a better alternative place for that work.
>>> So we've been toying with the idea of getting rid of the session.
>> [Ahmed] In our proprietary API the stream and session are one. A session
>> holds many properties like the op-type, instead of having this
>> information in the op itself.  This way we lower the per op setup cost.
>> This also allows rapid reuse of stateful infrastructure, once a stream
>> is closed on a stateful session, the next op (stream) on this session
>> reuses the stateful storage. Obviously if a stream is in "pause mode" on
>> a session, all following ops that may be unrelated to this
>> stream/session must also wait until this current stream is closed or
>> aborted before the infrastructure can be reused.
>>> We also struggle with the idea of setting up a stream for stateless ops.
>>>   - Well, really I just think the name is misleading, i.e. there's no problem with setting
>>>     up some private PMD data to use with stateless operations, just calling it a
>>>     stream doesn't seem right.
>> [Ahmed] I agree. The op has all the necessary information to process it
>> in the current API? Both the stream and the op are one time use. We
>> can't attach multiple similar ops to a single stream/session and rely on
>> their properties to simplify op setup, so why the hassle.
>>> So putting above thoughts together I want to propose:
>>> -	Removal of the session and all associated APIs.
>>> -	Passing in one of three data types in the rte_comp_op
>>>
>>>     union {
>>>         struct rte_comp_xform *xform;
>>>         /**< Immutable compress/decompress params */
>>>         void *pmd_stateless_data;
>>>         /**< Stateless private PMD data derived from an rte_comp_xform
>>>          * rte_comp_stateless_data_init() must be called on a device
>>>          * before sending any STATELESS operations. If the PMD returns a non-NULL
>>>          * value the handle must be attached to subsequent STATELESS operations.
>>>          * If a PMD returns NULL, then the xform should be passed directly to each op
>>>          */
>>>         void *stream;
>>>         /* Private PMD data derived initially from an rte_comp_xform, which holds state
>>>          * and history data and evolves as operations are processed.
>>>          * rte_comp_stream_create() must be called on a device for all STATEFUL
>>>          * data streams and the resulting stream attached
>>>          * to the one or more operations associated with the data stream.
>>>          * All operations in a stream must be sent to the same device.
>>>          */
>>>     }
>> [Ahmed] I like this setup, but I am not sure in what cases the xform
>> immutable would be used. I understand the other two.
> [Fiona] The xform is there because I don't know yet what limitations may apply to the
> pmd_stateless_data. If it has no limitation and once set up once on a device 
> can be attached simultaneously to any op sent to any qp on that device
> then we don't need the xform. But I understood from Shally's earlier request for
> setting up a stream on a stateless request that some resources are
> allocated, so we may need to document some limitations.
> In this case the xform may be a better path for PMDs which don't have the same
> limitations.  
>
>
>>> Notes:
>>> 1. Internally if a PMD wants to use the exact same data structure for both it can do,
>>>      just on the API I think it's better if they're named differently with
>>>      different comments.
>>> 2. I'm not clear of the constraints if any, which attach to the pmd_stateless_data
>>>      For our PMD it would only hold immutable data as the session did, and so
>>>      could be attached to many ops in parallel.
>>>      Is this true for all PMDs or are there constraints which should be called out?
>>>      Is it limited to a specific device, qp, or to be used on one op at a time?
>>> 3. Am open to other naming suggestions, just trying to capture the essence
>>>     of these data structs better than our current API does.
>>>
>>> We would put some more helper fns and structure around the above code if people
>>> are in agreement, just want to see if the concept flies before going further?
>>>
>>> Fiona
>>>
>>>
>>>
>

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Verma, Shally]

>-----Original Message-----
>From: Ahmed Mansour [mailto:ahmed.mansour@nxp.com]
>Sent: 27 February 2018 00:46
>To: Trahe, Fiona <fiona.trahe@intel.com>; dev@dpdk.org; Verma, Shally <Shally.Verma@cavium.com>
>Cc: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Athreya, Narayana Prasad <NarayanaPrasad.Athreya@cavium.com>;
>Gupta, Ashish <Ashish.Gupta@cavium.com>; Sahu, Sunila <Sunila.Sahu@cavium.com>; Challa, Mahipal
><Mahipal.Challa@cavium.com>; Jain, Deepak K <deepak.k.jain@intel.com>; Hemant Agrawal <hemant.agrawal@nxp.com>; Roy
>Pledge <roy.pledge@nxp.com>; Youri Querry <youri.querry_1@nxp.com>
>Subject: Re: [dpdk-dev] [PATCH] compressdev: implement API
>
>>> Hi Fiona,
>>>
>>> Thanks for starting this discussion. In the current API the user must
>>> make 12 API calls just to get information to compress. Maybe there is a
>>> way to simplify. At least for some use cases (stateless). I think a call
>>> sometime next week would be good to help clarify coalesce some of the
>>> complexity.
>> [Fiona] Would 10:30 GMT on Wednesday 28th Feb suit?
>[Ahmed] I am in Ottawa Canada so Wednesday 12:00 or 1:00 GMT would be
>better. Does that work?

[Shally] Am in India time zone. Either works for me.

//snip

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Verma, Shally]

Hi Fiona, Ahmed

>-----Original Message-----
>From: Ahmed Mansour [mailto:ahmed.mansour@nxp.com]
>Sent: 24 February 2018 06:47
>To: Trahe, Fiona <fiona.trahe@intel.com>; dev@dpdk.org; Verma, Shally <Shally.Verma@cavium.com>
>Cc: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Athreya, Narayana Prasad <NarayanaPrasad.Athreya@cavium.com>;
>Gupta, Ashish <Ashish.Gupta@cavium.com>; Sahu, Sunila <Sunila.Sahu@cavium.com>; Challa, Mahipal
><Mahipal.Challa@cavium.com>; Jain, Deepak K <deepak.k.jain@intel.com>; Hemant Agrawal <hemant.agrawal@nxp.com>; Roy
>Pledge <roy.pledge@nxp.com>; Youri Querry <youri.querry_1@nxp.com>
>Subject: Re: [dpdk-dev] [PATCH] compressdev: implement API
>
>Hi Fiona,
>
>Thanks for starting this discussion. In the current API the user must
>make 12 API calls just to get information to compress. Maybe there is a
>way to simplify. At least for some use cases (stateless). I think a call
>sometime next week would be good to help clarify coalesce some of the
>complexity.
>
>I added specific comments inline.
>
>Thanks,
>
>Ahmed
>
>On 2/21/2018 2:12 PM, Trahe, Fiona wrote:
>> We've been struggling with the idea of session in compressdev.
>>
>> Is it really a session?
>>  - It's not in the same sense as cryptodev where it's used to hold a key, and maps to a Security Association.
>>  - It's a set of immutable data that is needed with the op and stream to perform the operation.
>>  - It inherited from cryptodev the ability to be set up for multiple driver types and used across any
>>     devices of those types. For stateful ops this facility can't be used.
>>     For stateless we don't think it's important, and think it's unlikely to be used.
>>  - Drivers use it to prepare private data, set up resources, do pre-work, so there's
>>     less work to be done on the data path. Initially we didn't have a stream, we do now,
>>     this may be a better alternative place for that work.
>> So we've been toying with the idea of getting rid of the session.
>[Ahmed] In our proprietary API the stream and session are one. A session
>holds many properties like the op-type, instead of having this
>information in the op itself.  This way we lower the per op setup cost.
>This also allows rapid reuse of stateful infrastructure, once a stream
>is closed on a stateful session, the next op (stream) on this session
>reuses the stateful storage. Obviously if a stream is in "pause mode" on
>a session, all following ops that may be unrelated to this
>stream/session must also wait until this current stream is closed or
>aborted before the infrastructure can be reused.
>>
>> We also struggle with the idea of setting up a stream for stateless ops.
>>   - Well, really I just think the name is misleading, i.e. there's no problem with setting
>>     up some private PMD data to use with stateless operations, just calling it a
>>     stream doesn't seem right.
>[Ahmed] I agree. The op has all the necessary information to process it
>in the current API? Both the stream and the op are one time use. We
>can't attach multiple similar ops to a single stream/session and rely on
>their properties to simplify op setup, so why the hassle.

[Shally]  As per my knowledge, session came with idea in DPDK, if system has multiple devices setup to do similar jobs then application can fan out ops to any of them for load-balancing. Though it is not possible for stateful ops but it still can work for stateless. If there's an application which only have stateless ops to process then I see this is still useful feature to support.

In current proposal, stream logically represent data and hold its specific information and session is generic information that can be applied on multiple data. If we want to combine stream and session. Then one way to look at this is:

"let application only allocate and initialize session with rte_comp_xform (and possibly op type) information so that PMD can do one-time setup and allocate enough resources. Once attached to op, cannot be reused until that op is fully processed. So, if app has 16 data elements to process in a burst, it will setup 16 sessions."

This is same as what Ahmed suggested. For a particular load-balancing case suggested above, If application want, can initialize different sessions on multiple devices with same xform so that each is prepared to process ops.  Application can then fanout stateless ops to multiple devices for load-balancing but then it would need to keep map of device & a session map.

If this sound feasible, then I too believe we can rather get rid of either and keep one (possibly session but am open with stream as well). 
However, regardless of case whether we live with name stream or session, I don't see much deviation from current API spec except description and few modifications/additions as identified.
So, then I see it as:

- A stream(or session whichever name is chosen) can be used with only one-op at-a-time
- It can be re-used when previously attached op is processed
-  if it is stream then currently it is allocated from PMD managed pool whereas Sessions are allocated from application created mempool.
   In either of case, I would expect to review pool management API

With this in mind, below are few of my comments

>>
>> So putting above thoughts together I want to propose:
>> -	Removal of the session and all associated APIs.
>> -	Passing in one of three data types in the rte_comp_op
>>
>>     union {
>>         struct rte_comp_xform *xform;
>>         /**< Immutable compress/decompress params */
>>         void *pmd_stateless_data;
>>         /**< Stateless private PMD data derived from an rte_comp_xform
>>          * rte_comp_stateless_data_init() must be called on a device
>>          * before sending any STATELESS operations. If the PMD returns a non-NULL
>>          * value the handle must be attached to subsequent STATELESS operations.
>>          * If a PMD returns NULL, then the xform should be passed directly to each op
>>          */
[Shally] It sounds like stateless_data_init() nothing more than a replacement of session_init(). 
	So, this is needed neither if we retain session concept nor if we retain stream concept ( rte_comp_stream_create() with op_type: stateless can serve same purpose). 
	It should be sufficient to provide either stream (or session) pointer.

>>         void *stream;
>>         /* Private PMD data derived initially from an rte_comp_xform, which holds state
>>          * and history data and evolves as operations are processed.
>>          * rte_comp_stream_create() must be called on a device for all STATEFUL
>>          * data streams and the resulting stream attached
>>          * to the one or more operations associated with the data stream.
>>          * All operations in a stream must be sent to the same device.
>>          */
>>     }
>[Ahmed] I like this setup, but I am not sure in what cases the xform
>immutable would be used. I understand the other two.

[Shally] my understanding is xform will be mapped by PMD to its internally managed stream(or session data structure). And then we  can remove STATEFUL reference here and just say stream(or session) it belongs to. However, This condition still apply:
       *All operations that belong to same stream must be sent to the same device.*

>> Notes:
>> 1. Internally if a PMD wants to use the exact same data structure for both it can do,
>>      just on the API I think it's better if they're named differently with
>>      different comments.
>> 2. I'm not clear of the constraints if any, which attach to the pmd_stateless_data
>>      For our PMD it would only hold immutable data as the session did, and so
>>      could be attached to many ops in parallel.
>>      Is this true for all PMDs or are there constraints which should be called out?
>>      Is it limited to a specific device, qp, or to be used on one op at a time?
>> 3. Am open to other naming suggestions, just trying to capture the essence
>>     of these data structs better than our current API does.
>>
>> We would put some more helper fns and structure around the above code if people
>> are in agreement, just want to see if the concept flies before going further?
>>
>> Fiona
>>
>>
>>

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Ahmed Mansour]

> Hi Fiona, Ahmed
>> Hi Fiona,
>>
>> Thanks for starting this discussion. In the current API the user must
>> make 12 API calls just to get information to compress. Maybe there is a
>> way to simplify. At least for some use cases (stateless). I think a call
>> sometime next week would be good to help clarify coalesce some of the
>> complexity.
>>
>> I added specific comments inline.
>>
>> Thanks,
>>
>> Ahmed
>>
>> On 2/21/2018 2:12 PM, Trahe, Fiona wrote:
>>> We've been struggling with the idea of session in compressdev.
>>>
>>> Is it really a session?
>>>  - It's not in the same sense as cryptodev where it's used to hold a key, and maps to a Security Association.
>>>  - It's a set of immutable data that is needed with the op and stream to perform the operation.
>>>  - It inherited from cryptodev the ability to be set up for multiple driver types and used across any
>>>     devices of those types. For stateful ops this facility can't be used.
>>>     For stateless we don't think it's important, and think it's unlikely to be used.
>>>  - Drivers use it to prepare private data, set up resources, do pre-work, so there's
>>>     less work to be done on the data path. Initially we didn't have a stream, we do now,
>>>     this may be a better alternative place for that work.
>>> So we've been toying with the idea of getting rid of the session.
>> [Ahmed] In our proprietary API the stream and session are one. A session
>> holds many properties like the op-type, instead of having this
>> information in the op itself.  This way we lower the per op setup cost.
>> This also allows rapid reuse of stateful infrastructure, once a stream
>> is closed on a stateful session, the next op (stream) on this session
>> reuses the stateful storage. Obviously if a stream is in "pause mode" on
>> a session, all following ops that may be unrelated to this
>> stream/session must also wait until this current stream is closed or
>> aborted before the infrastructure can be reused.
>>> We also struggle with the idea of setting up a stream for stateless ops.
>>>   - Well, really I just think the name is misleading, i.e. there's no problem with setting
>>>     up some private PMD data to use with stateless operations, just calling it a
>>>     stream doesn't seem right.
>> [Ahmed] I agree. The op has all the necessary information to process it
>> in the current API? Both the stream and the op are one time use. We
>> can't attach multiple similar ops to a single stream/session and rely on
>> their properties to simplify op setup, so why the hassle.
> [Shally]  As per my knowledge, session came with idea in DPDK, if system has multiple devices setup to do similar jobs then application can fan out ops to any of them for load-balancing. Though it is not possible for stateful ops but it still can work for stateless. If there's an application which only have stateless ops to process then I see this is still useful feature to support.
[Ahmed] Is there an advantage to exposing load balancing to the user? I
do not see load balancing as a feature within itself. Can the PMD take
care of this? I guess a system that has
> In current proposal, stream logically represent data and hold its specific information and session is generic information that can be applied on multiple data. If we want to combine stream and session. Then one way to look at this is:
>
> "let application only allocate and initialize session with rte_comp_xform (and possibly op type) information so that PMD can do one-time setup and allocate enough resources. Once attached to op, cannot be reused until that op is fully processed. So, if app has 16 data elements to process in a burst, it will setup 16 sessions."
[Ahmed] Why not allow multiple inflight stateless ops with the same
session? Stateless by definition guarantees that the resources used to
work on one up will be free after the op is processed. That means that
even if an op fails to process correctly on a session, it will have no
effect on the next op since there is not interdependence. This assumes
that the resources are shareable between hardware instances for
stateless. That is not a bad assumption since hardware should not need
more than the data of the op itself to work on a statelss op.
> This is same as what Ahmed suggested. For a particular load-balancing case suggested above, If application want, can initialize different sessions on multiple devices with same xform so that each is prepared to process ops.  Application can then fanout stateless ops to multiple devices for load-balancing but then it would need to keep map of device & a session map.
>
> If this sound feasible, then I too believe we can rather get rid of either and keep one (possibly session but am open with stream as well). 
> However, regardless of case whether we live with name stream or session, I don't see much deviation from current API spec except description and few modifications/additions as identified.
> So, then I see it as:
>
> - A stream(or session whichever name is chosen) can be used with only one-op at-a-time
> - It can be re-used when previously attached op is processed
> -  if it is stream then currently it is allocated from PMD managed pool whereas Sessions are allocated from application created mempool.
>    In either of case, I would expect to review pool management API
>
> With this in mind, below are few of my comments
>
>>> So putting above thoughts together I want to propose:
>>> -	Removal of the session and all associated APIs.
>>> -	Passing in one of three data types in the rte_comp_op
>>>
>>>     union {
>>>         struct rte_comp_xform *xform;
>>>         /**< Immutable compress/decompress params */
>>>         void *pmd_stateless_data;
>>>         /**< Stateless private PMD data derived from an rte_comp_xform
>>>          * rte_comp_stateless_data_init() must be called on a device
>>>          * before sending any STATELESS operations. If the PMD returns a non-NULL
>>>          * value the handle must be attached to subsequent STATELESS operations.
>>>          * If a PMD returns NULL, then the xform should be passed directly to each op
>>>          */
> [Shally] It sounds like stateless_data_init() nothing more than a replacement of session_init(). 
> 	So, this is needed neither if we retain session concept nor if we retain stream concept ( rte_comp_stream_create() with op_type: stateless can serve same purpose). 
> 	It should be sufficient to provide either stream (or session) pointer.
>
>>>         void *stream;
>>>         /* Private PMD data derived initially from an rte_comp_xform, which holds state
>>>          * and history data and evolves as operations are processed.
>>>          * rte_comp_stream_create() must be called on a device for all STATEFUL
>>>          * data streams and the resulting stream attached
>>>          * to the one or more operations associated with the data stream.
>>>          * All operations in a stream must be sent to the same device.
>>>          */
>>>     }
>> [Ahmed] I like this setup, but I am not sure in what cases the xform
>> immutable would be used. I understand the other two.
> [Shally] my understanding is xform will be mapped by PMD to its internally managed stream(or session data structure). And then we  can remove STATEFUL reference here and just say stream(or session) it belongs to. However, This condition still apply:
>        *All operations that belong to same stream must be sent to the same device.*
>
>>> Notes:
>>> 1. Internally if a PMD wants to use the exact same data structure for both it can do,
>>>      just on the API I think it's better if they're named differently with
>>>      different comments.
>>> 2. I'm not clear of the constraints if any, which attach to the pmd_stateless_data
>>>      For our PMD it would only hold immutable data as the session did, and so
>>>      could be attached to many ops in parallel.
>>>      Is this true for all PMDs or are there constraints which should be called out?
>>>      Is it limited to a specific device, qp, or to be used on one op at a time?
>>> 3. Am open to other naming suggestions, just trying to capture the essence
>>>     of these data structs better than our current API does.
>>>
>>> We would put some more helper fns and structure around the above code if people
>>> are in agreement, just want to see if the concept flies before going further?
>>>
>>> Fiona
>>>
>>>
>>>
>

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Verma, Shally]

>-----Original Message-----
>From: Ahmed Mansour [mailto:ahmed.mansour@nxp.com]
>Sent: 27 February 2018 03:05
>To: Verma, Shally <Shally.Verma@cavium.com>; Trahe, Fiona <fiona.trahe@intel.com>; dev@dpdk.org
>Cc: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Athreya, Narayana Prasad <NarayanaPrasad.Athreya@cavium.com>;
>Gupta, Ashish <Ashish.Gupta@cavium.com>; Sahu, Sunila <Sunila.Sahu@cavium.com>; Challa, Mahipal
><Mahipal.Challa@cavium.com>; Jain, Deepak K <deepak.k.jain@intel.com>; Hemant Agrawal <hemant.agrawal@nxp.com>; Roy
>Pledge <roy.pledge@nxp.com>; Youri Querry <youri.querry_1@nxp.com>
>Subject: Re: [dpdk-dev] [PATCH] compressdev: implement API
>
>> Hi Fiona, Ahmed
>>> Hi Fiona,
>>>
>>> Thanks for starting this discussion. In the current API the user must
>>> make 12 API calls just to get information to compress. Maybe there is a
>>> way to simplify. At least for some use cases (stateless). I think a call
>>> sometime next week would be good to help clarify coalesce some of the
>>> complexity.
>>>
>>> I added specific comments inline.
>>>
>>> Thanks,
>>>
>>> Ahmed
>>>
>>> On 2/21/2018 2:12 PM, Trahe, Fiona wrote:
>>>> We've been struggling with the idea of session in compressdev.
>>>>
>>>> Is it really a session?
>>>>  - It's not in the same sense as cryptodev where it's used to hold a key, and maps to a Security Association.
>>>>  - It's a set of immutable data that is needed with the op and stream to perform the operation.
>>>>  - It inherited from cryptodev the ability to be set up for multiple driver types and used across any
>>>>     devices of those types. For stateful ops this facility can't be used.
>>>>     For stateless we don't think it's important, and think it's unlikely to be used.
>>>>  - Drivers use it to prepare private data, set up resources, do pre-work, so there's
>>>>     less work to be done on the data path. Initially we didn't have a stream, we do now,
>>>>     this may be a better alternative place for that work.
>>>> So we've been toying with the idea of getting rid of the session.
>>> [Ahmed] In our proprietary API the stream and session are one. A session
>>> holds many properties like the op-type, instead of having this
>>> information in the op itself.  This way we lower the per op setup cost.
>>> This also allows rapid reuse of stateful infrastructure, once a stream
>>> is closed on a stateful session, the next op (stream) on this session
>>> reuses the stateful storage. Obviously if a stream is in "pause mode" on
>>> a session, all following ops that may be unrelated to this
>>> stream/session must also wait until this current stream is closed or
>>> aborted before the infrastructure can be reused.
>>>> We also struggle with the idea of setting up a stream for stateless ops.
>>>>   - Well, really I just think the name is misleading, i.e. there's no problem with setting
>>>>     up some private PMD data to use with stateless operations, just calling it a
>>>>     stream doesn't seem right.
>>> [Ahmed] I agree. The op has all the necessary information to process it
>>> in the current API? Both the stream and the op are one time use. We
>>> can't attach multiple similar ops to a single stream/session and rely on
>>> their properties to simplify op setup, so why the hassle.
>> [Shally]  As per my knowledge, session came with idea in DPDK, if system has multiple devices setup to do similar jobs then
>application can fan out ops to any of them for load-balancing. Though it is not possible for stateful ops but it still can work for stateless.
>If there's an application which only have stateless ops to process then I see this is still useful feature to support.
>[Ahmed] Is there an advantage to exposing load balancing to the user? I
>do not see load balancing as a feature within itself. Can the PMD take
>care of this? I guess a system that has

[Shally] I assume idea was to leverage multiple PMDs that are available in system (say QAT+SW ZLIB) and I believe matter of load-balancing came out of one of the earlier discussion with Fiona on RFC v1. http://dev.dpdk.narkive.com/CHS5l01B/dpdk-dev-rfc-v1-doc-compression-api-for-dpdk#post3
So, I wait for her comments on this. But in any case, with changed notion too it looks achievable to me, if so is desired.

>> In current proposal, stream logically represent data and hold its specific information and session is generic information that can be
>applied on multiple data. If we want to combine stream and session. Then one way to look at this is:
>>
>> "let application only allocate and initialize session with rte_comp_xform (and possibly op type) information so that PMD can do one-
>time setup and allocate enough resources. Once attached to op, cannot be reused until that op is fully processed. So, if app has 16
>data elements to process in a burst, it will setup 16 sessions."
>[Ahmed] Why not allow multiple inflight stateless ops with the same
>session? Stateless by definition guarantees that the resources used to
>work on one up will be free after the op is processed. That means that
>even if an op fails to process correctly on a session, it will have no
>effect on the next op since there is not interdependence. This assumes
>that the resources are shareable between hardware instances for
>stateless. That is not a bad assumption since hardware should not need
>more than the data of the op itself to work on a statelss op.

[Shally]  multiple ops in-flight can connect to same session but I assume you agree then they cannot execute in parallel i.e. only one op at-a-time can use session here? And as far as I understand your PMD works this way. Your HW execute one op at-a-time from queue?!

>> This is same as what Ahmed suggested. For a particular load-balancing case suggested above, If application want, can initialize
>different sessions on multiple devices with same xform so that each is prepared to process ops.  Application can then fanout stateless
>ops to multiple devices for load-balancing but then it would need to keep map of device & a session map.
>>
>> If this sound feasible, then I too believe we can rather get rid of either and keep one (possibly session but am open with stream as
>well).
>> However, regardless of case whether we live with name stream or session, I don't see much deviation from current API spec except
>description and few modifications/additions as identified.
>> So, then I see it as:
>>
>> - A stream(or session whichever name is chosen) can be used with only one-op at-a-time
>> - It can be re-used when previously attached op is processed
>> -  if it is stream then currently it is allocated from PMD managed pool whereas Sessions are allocated from application created
>mempool.
>>    In either of case, I would expect to review pool management API
>>
>> With this in mind, below are few of my comments
>>
>>>> So putting above thoughts together I want to propose:
>>>> -	Removal of the session and all associated APIs.
>>>> -	Passing in one of three data types in the rte_comp_op
>>>>
>>>>     union {
>>>>         struct rte_comp_xform *xform;
>>>>         /**< Immutable compress/decompress params */
>>>>         void *pmd_stateless_data;
>>>>         /**< Stateless private PMD data derived from an rte_comp_xform
>>>>          * rte_comp_stateless_data_init() must be called on a device
>>>>          * before sending any STATELESS operations. If the PMD returns a non-NULL
>>>>          * value the handle must be attached to subsequent STATELESS operations.
>>>>          * If a PMD returns NULL, then the xform should be passed directly to each op
>>>>          */
>> [Shally] It sounds like stateless_data_init() nothing more than a replacement of session_init().
>> 	So, this is needed neither if we retain session concept nor if we retain stream concept ( rte_comp_stream_create() with
>op_type: stateless can serve same purpose).
>> 	It should be sufficient to provide either stream (or session) pointer.
>>
>>>>         void *stream;
>>>>         /* Private PMD data derived initially from an rte_comp_xform, which holds state
>>>>          * and history data and evolves as operations are processed.
>>>>          * rte_comp_stream_create() must be called on a device for all STATEFUL
>>>>          * data streams and the resulting stream attached
>>>>          * to the one or more operations associated with the data stream.
>>>>          * All operations in a stream must be sent to the same device.
>>>>          */
>>>>     }
>>> [Ahmed] I like this setup, but I am not sure in what cases the xform
>>> immutable would be used. I understand the other two.
>> [Shally] my understanding is xform will be mapped by PMD to its internally managed stream(or session data structure). And then we
>can remove STATEFUL reference here and just say stream(or session) it belongs to. However, This condition still apply:
>>        *All operations that belong to same stream must be sent to the same device.*
>>
>>>> Notes:
>>>> 1. Internally if a PMD wants to use the exact same data structure for both it can do,
>>>>      just on the API I think it's better if they're named differently with
>>>>      different comments.
>>>> 2. I'm not clear of the constraints if any, which attach to the pmd_stateless_data
>>>>      For our PMD it would only hold immutable data as the session did, and so
>>>>      could be attached to many ops in parallel.
>>>>      Is this true for all PMDs or are there constraints which should be called out?
>>>>      Is it limited to a specific device, qp, or to be used on one op at a time?
>>>> 3. Am open to other naming suggestions, just trying to capture the essence
>>>>     of these data structs better than our current API does.
>>>>
>>>> We would put some more helper fns and structure around the above code if people
>>>> are in agreement, just want to see if the concept flies before going further?
>>>>
>>>> Fiona
>>>>
>>>>
>>>>
>>

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Trahe, Fiona]

Hi Ahmed, Shally,

So just to capture what we concluded in the call today:

 - There's no requirement for a device-agnostic session to facilitate load-balancing.
 - For stateful data a stream is compulsory. Xform is passed to stream on creation. 
    So no need for a session in stateful op.

Re session data for stateless ops:
 - All PMDs could cope with just passing in a xform with a stateless op. But it might 
   not be performant. 
 - Some PMDs need to allocate some resources which can only be used by one op
   at a time. For stateful ops these resources can be attached to a stream. For stateless
   they could allocate the resources on each op enqueue, but it would be better if
   the resources were setup once based on the xform and could be re-used on ops,
   though only by one op at a time. 
 - Some PMDs don't need to allocate such resources, but could benefit by
   setting up some pmd data based on the xform. This data would not be 
   constrained, could be used in parallel by any op or qp of the device. 
 - The name pmd_stateless_data was not popular, maybe something like 
    xform_private_data can be used. On creation of this data, the PMD can return 
    an indication of whether it should be used by one op at a time or shared.  
  
So I'll 
 - remove the session completely from the API.
 - add an initialiser API for the data to be attached to stateless ops
 - add a union to the op:

 union {
        void *pmd_private_xform;
        /**< Stateless private PMD data derived from an rte_comp_xform
         * rte_comp_xform_init() must be called on a device 
         * before sending any STATELESS operations. The PMD returns a handle
         * which must be attached to subsequent STATELESS operations.
         * The PMD also returns a flag, if this is COMP_PRIVATE_XFORM_SHAREABLE
         * then the xform can be attached to multiple ops at the same time, 
         * if it's COMP_PRIVATE_XFORM_SINGLE_OP then it can only be
         * be used on one op at a time, other private xforms must be initialised
         * to send other ops in parallel. 
         */
        void *stream;
        /* Private PMD data derived initially from an rte_comp_xform, which holds state
         * and history data and evolves as operations are processed.
         * rte_comp_stream_create() must be called on a device for all STATEFUL 
         * data streams and the resulting stream attached
         * to the one or more operations associated with the data stream.
         * All operations in a stream must be sent to the same device.
         */
    }

Previous startup flow before sending a stateful op: 
rte_comp_get_private_size(devid)
rte_comp_mempool_create() - returns sess_pool
rte_comp_session_create(sess_pool)
rte_comp_session_init(devid, sess, sess_pool, xform)
rte_comp_stream_create(devid, sess, **stream, op_type)

simplified to:
rte_comp_xform_init(devid, xform, **priv_xform, *flag) - returns handle and flag 
(pool is within the PMD)

Note, I don't think we bottomed out on removing the xform from the union, but I don't
think we need it with above solution. 

Other discussion:
 - we should document on API that qp is not thread-safe, so enqueue
   and dequeue should be performed by same thread.

device and qp flow:
 - dev_info_get() - application reads device capabilities, including the max qps the device can support.
 - dev_config() - application specifies how many qps it intends to use - typically one per thread, must be < device max
 - qp_setup() - called per qp. Creates the qp based on the size indicated by max_inflights
 - dev_start() - once started device can't be reconfigured, must call dev_stop to reconfigure.


Regards,
Fiona

> -----Original Message-----
> From: Verma, Shally [mailto:Shally.Verma@cavium.com]
> Sent: Tuesday, February 27, 2018 5:54 AM
> To: Ahmed Mansour <ahmed.mansour@nxp.com>; Trahe, Fiona <fiona.trahe@intel.com>;
> dev@dpdk.org
> Cc: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Athreya, Narayana Prasad
> <NarayanaPrasad.Athreya@cavium.com>; Gupta, Ashish <Ashish.Gupta@cavium.com>; Sahu, Sunila
> <Sunila.Sahu@cavium.com>; Challa, Mahipal <Mahipal.Challa@cavium.com>; Jain, Deepak K
> <deepak.k.jain@intel.com>; Hemant Agrawal <hemant.agrawal@nxp.com>; Roy Pledge
> <roy.pledge@nxp.com>; Youri Querry <youri.querry_1@nxp.com>
> Subject: RE: [dpdk-dev] [PATCH] compressdev: implement API
> 
> 
> 
> >-----Original Message-----
> >From: Ahmed Mansour [mailto:ahmed.mansour@nxp.com]
> >Sent: 27 February 2018 03:05
> >To: Verma, Shally <Shally.Verma@cavium.com>; Trahe, Fiona <fiona.trahe@intel.com>; dev@dpdk.org
> >Cc: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Athreya, Narayana Prasad
> <NarayanaPrasad.Athreya@cavium.com>;
> >Gupta, Ashish <Ashish.Gupta@cavium.com>; Sahu, Sunila <Sunila.Sahu@cavium.com>; Challa,
> Mahipal
> ><Mahipal.Challa@cavium.com>; Jain, Deepak K <deepak.k.jain@intel.com>; Hemant Agrawal
> <hemant.agrawal@nxp.com>; Roy
> >Pledge <roy.pledge@nxp.com>; Youri Querry <youri.querry_1@nxp.com>
> >Subject: Re: [dpdk-dev] [PATCH] compressdev: implement API
> >
> >> Hi Fiona, Ahmed
> >>> Hi Fiona,
> >>>
> >>> Thanks for starting this discussion. In the current API the user must
> >>> make 12 API calls just to get information to compress. Maybe there is a
> >>> way to simplify. At least for some use cases (stateless). I think a call
> >>> sometime next week would be good to help clarify coalesce some of the
> >>> complexity.
> >>>
> >>> I added specific comments inline.
> >>>
> >>> Thanks,
> >>>
> >>> Ahmed
> >>>
> >>> On 2/21/2018 2:12 PM, Trahe, Fiona wrote:
> >>>> We've been struggling with the idea of session in compressdev.
> >>>>
> >>>> Is it really a session?
> >>>>  - It's not in the same sense as cryptodev where it's used to hold a key, and maps to a Security
> Association.
> >>>>  - It's a set of immutable data that is needed with the op and stream to perform the operation.
> >>>>  - It inherited from cryptodev the ability to be set up for multiple driver types and used across any
> >>>>     devices of those types. For stateful ops this facility can't be used.
> >>>>     For stateless we don't think it's important, and think it's unlikely to be used.
> >>>>  - Drivers use it to prepare private data, set up resources, do pre-work, so there's
> >>>>     less work to be done on the data path. Initially we didn't have a stream, we do now,
> >>>>     this may be a better alternative place for that work.
> >>>> So we've been toying with the idea of getting rid of the session.
> >>> [Ahmed] In our proprietary API the stream and session are one. A session
> >>> holds many properties like the op-type, instead of having this
> >>> information in the op itself.  This way we lower the per op setup cost.
> >>> This also allows rapid reuse of stateful infrastructure, once a stream
> >>> is closed on a stateful session, the next op (stream) on this session
> >>> reuses the stateful storage. Obviously if a stream is in "pause mode" on
> >>> a session, all following ops that may be unrelated to this
> >>> stream/session must also wait until this current stream is closed or
> >>> aborted before the infrastructure can be reused.
> >>>> We also struggle with the idea of setting up a stream for stateless ops.
> >>>>   - Well, really I just think the name is misleading, i.e. there's no problem with setting
> >>>>     up some private PMD data to use with stateless operations, just calling it a
> >>>>     stream doesn't seem right.
> >>> [Ahmed] I agree. The op has all the necessary information to process it
> >>> in the current API? Both the stream and the op are one time use. We
> >>> can't attach multiple similar ops to a single stream/session and rely on
> >>> their properties to simplify op setup, so why the hassle.
> >> [Shally]  As per my knowledge, session came with idea in DPDK, if system has multiple devices setup
> to do similar jobs then
> >application can fan out ops to any of them for load-balancing. Though it is not possible for stateful ops
> but it still can work for stateless.
> >If there's an application which only have stateless ops to process then I see this is still useful feature to
> support.
> >[Ahmed] Is there an advantage to exposing load balancing to the user? I
> >do not see load balancing as a feature within itself. Can the PMD take
> >care of this? I guess a system that has
> 
> [Shally] I assume idea was to leverage multiple PMDs that are available in system (say QAT+SW ZLIB)
> and I believe matter of load-balancing came out of one of the earlier discussion with Fiona on RFC v1.
> http://dev.dpdk.narkive.com/CHS5l01B/dpdk-dev-rfc-v1-doc-compression-api-for-dpdk#post3
> So, I wait for her comments on this. But in any case, with changed notion too it looks achievable to me,
> if so is desired.
> 
> >> In current proposal, stream logically represent data and hold its specific information and session is
> generic information that can be
> >applied on multiple data. If we want to combine stream and session. Then one way to look at this is:
> >>
> >> "let application only allocate and initialize session with rte_comp_xform (and possibly op type)
> information so that PMD can do one-
> >time setup and allocate enough resources. Once attached to op, cannot be reused until that op is fully
> processed. So, if app has 16
> >data elements to process in a burst, it will setup 16 sessions."
> >[Ahmed] Why not allow multiple inflight stateless ops with the same
> >session? Stateless by definition guarantees that the resources used to
> >work on one up will be free after the op is processed. That means that
> >even if an op fails to process correctly on a session, it will have no
> >effect on the next op since there is not interdependence. This assumes
> >that the resources are shareable between hardware instances for
> >stateless. That is not a bad assumption since hardware should not need
> >more than the data of the op itself to work on a statelss op.
> 
> [Shally]  multiple ops in-flight can connect to same session but I assume you agree then they cannot
> execute in parallel i.e. only one op at-a-time can use session here? And as far as I understand your PMD
> works this way. Your HW execute one op at-a-time from queue?!
> 
> >> This is same as what Ahmed suggested. For a particular load-balancing case suggested above, If
> application want, can initialize
> >different sessions on multiple devices with same xform so that each is prepared to process ops.
> Application can then fanout stateless
> >ops to multiple devices for load-balancing but then it would need to keep map of device & a session
> map.
> >>
> >> If this sound feasible, then I too believe we can rather get rid of either and keep one (possibly session
> but am open with stream as
> >well).
> >> However, regardless of case whether we live with name stream or session, I don't see much deviation
> from current API spec except
> >description and few modifications/additions as identified.
> >> So, then I see it as:
> >>
> >> - A stream(or session whichever name is chosen) can be used with only one-op at-a-time
> >> - It can be re-used when previously attached op is processed
> >> -  if it is stream then currently it is allocated from PMD managed pool whereas Sessions are allocated
> from application created
> >mempool.
> >>    In either of case, I would expect to review pool management API
> >>
> >> With this in mind, below are few of my comments
> >>
> >>>> So putting above thoughts together I want to propose:
> >>>> -	Removal of the session and all associated APIs.
> >>>> -	Passing in one of three data types in the rte_comp_op
> >>>>
> >>>>     union {
> >>>>         struct rte_comp_xform *xform;
> >>>>         /**< Immutable compress/decompress params */
> >>>>         void *pmd_stateless_data;
> >>>>         /**< Stateless private PMD data derived from an rte_comp_xform
> >>>>          * rte_comp_stateless_data_init() must be called on a device
> >>>>          * before sending any STATELESS operations. If the PMD returns a non-NULL
> >>>>          * value the handle must be attached to subsequent STATELESS operations.
> >>>>          * If a PMD returns NULL, then the xform should be passed directly to each op
> >>>>          */
> >> [Shally] It sounds like stateless_data_init() nothing more than a replacement of session_init().
> >> 	So, this is needed neither if we retain session concept nor if we retain stream concept (
> rte_comp_stream_create() with
> >op_type: stateless can serve same purpose).
> >> 	It should be sufficient to provide either stream (or session) pointer.
> >>
> >>>>         void *stream;
> >>>>         /* Private PMD data derived initially from an rte_comp_xform, which holds state
> >>>>          * and history data and evolves as operations are processed.
> >>>>          * rte_comp_stream_create() must be called on a device for all STATEFUL
> >>>>          * data streams and the resulting stream attached
> >>>>          * to the one or more operations associated with the data stream.
> >>>>          * All operations in a stream must be sent to the same device.
> >>>>          */
> >>>>     }
> >>> [Ahmed] I like this setup, but I am not sure in what cases the xform
> >>> immutable would be used. I understand the other two.
> >> [Shally] my understanding is xform will be mapped by PMD to its internally managed stream(or
> session data structure). And then we
> >can remove STATEFUL reference here and just say stream(or session) it belongs to. However, This
> condition still apply:
> >>        *All operations that belong to same stream must be sent to the same device.*
> >>
> >>>> Notes:
> >>>> 1. Internally if a PMD wants to use the exact same data structure for both it can do,
> >>>>      just on the API I think it's better if they're named differently with
> >>>>      different comments.
> >>>> 2. I'm not clear of the constraints if any, which attach to the pmd_stateless_data
> >>>>      For our PMD it would only hold immutable data as the session did, and so
> >>>>      could be attached to many ops in parallel.
> >>>>      Is this true for all PMDs or are there constraints which should be called out?
> >>>>      Is it limited to a specific device, qp, or to be used on one op at a time?
> >>>> 3. Am open to other naming suggestions, just trying to capture the essence
> >>>>     of these data structs better than our current API does.
> >>>>
> >>>> We would put some more helper fns and structure around the above code if people
> >>>> are in agreement, just want to see if the concept flies before going further?
> >>>>
> >>>> Fiona
> >>>>
> >>>>
> >>>>
> >>

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Verma, Shally]

Hi Fiona

>-----Original Message-----
>From: Trahe, Fiona [mailto:fiona.trahe@intel.com]
>Sent: 01 March 2018 00:09
>To: Verma, Shally <Shally.Verma@cavium.com>; Ahmed Mansour <ahmed.mansour@nxp.com>; dev@dpdk.org
>Cc: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Athreya, Narayana Prasad <NarayanaPrasad.Athreya@cavium.com>;
>Gupta, Ashish <Ashish.Gupta@cavium.com>; Sahu, Sunila <Sunila.Sahu@cavium.com>; Challa, Mahipal
><Mahipal.Challa@cavium.com>; Jain, Deepak K <deepak.k.jain@intel.com>; Hemant Agrawal <hemant.agrawal@nxp.com>; Roy
>Pledge <roy.pledge@nxp.com>; Youri Querry <youri.querry_1@nxp.com>; Trahe, Fiona <fiona.trahe@intel.com>
>Subject: RE: [dpdk-dev] [PATCH] compressdev: implement API
>
>Hi Ahmed, Shally,
>
>So just to capture what we concluded in the call today:
>
> - There's no requirement for a device-agnostic session to facilitate load-balancing.
> - For stateful data a stream is compulsory. Xform is passed to stream on creation.
>    So no need for a session in stateful op.
>
>Re session data for stateless ops:
> - All PMDs could cope with just passing in a xform with a stateless op. But it might
>   not be performant.
> - Some PMDs need to allocate some resources which can only be used by one op
>   at a time. For stateful ops these resources can be attached to a stream. For stateless
>   they could allocate the resources on each op enqueue, but it would be better if
>   the resources were setup once based on the xform and could be re-used on ops,
>   though only by one op at a time.
> - Some PMDs don't need to allocate such resources, but could benefit by
>   setting up some pmd data based on the xform. This data would not be
>   constrained, could be used in parallel by any op or qp of the device.
> - The name pmd_stateless_data was not popular, maybe something like
>    xform_private_data can be used. On creation of this data, the PMD can return
>    an indication of whether it should be used by one op at a time or shared.
>
>So I'll
> - remove the session completely from the API.
> - add an initialiser API for the data to be attached to stateless ops
> - add a union to the op:
>
> union {
>        void *pmd_private_xform;
>        /**< Stateless private PMD data derived from an rte_comp_xform
>         * rte_comp_xform_init() must be called on a device
>         * before sending any STATELESS operations. The PMD returns a handle
>         * which must be attached to subsequent STATELESS operations.
>         * The PMD also returns a flag, if this is COMP_PRIVATE_XFORM_SHAREABLE
>         * then the xform can be attached to multiple ops at the same time,
>         * if it's COMP_PRIVATE_XFORM_SINGLE_OP then it can only be
>         * be used on one op at a time, other private xforms must be initialised
>         * to send other ops in parallel.
>         */
>        void *stream;
>        /* Private PMD data derived initially from an rte_comp_xform, which holds state
>         * and history data and evolves as operations are processed.
>         * rte_comp_stream_create() must be called on a device for all STATEFUL
>         * data streams and the resulting stream attached
>         * to the one or more operations associated with the data stream.
>         * All operations in a stream must be sent to the same device.
>         */
>    }
>
>Previous startup flow before sending a stateful op:
>rte_comp_get_private_size(devid)
>rte_comp_mempool_create() - returns sess_pool
>rte_comp_session_create(sess_pool)
>rte_comp_session_init(devid, sess, sess_pool, xform)
>rte_comp_stream_create(devid, sess, **stream, op_type)
>
>simplified to:
>rte_comp_xform_init(devid, xform, **priv_xform, *flag) - returns handle and flag
>(pool is within the PMD)
>
>Note, I don't think we bottomed out on removing the xform from the union, but I don't
>think we need it with above solution.

[Shally] This looks better to me. So it mean app would always call xform_init() for stateless and attach an updated priv_xform to ops (depending upon if there's shareable or not). So it does not need to have  NULL pointer on priv_xform. right?


>
>Other discussion:
> - we should document on API that qp is not thread-safe, so enqueue
>   and dequeue should be performed by same thread.
>
>device and qp flow:
> - dev_info_get() - application reads device capabilities, including the max qps the device can support.
> - dev_config() - application specifies how many qps it intends to use - typically one per thread, must be < device max
> - qp_setup() - called per qp. Creates the qp based on the size indicated by max_inflights
> - dev_start() - once started device can't be reconfigured, must call dev_stop to reconfigure.
>
>
>Regards,
>Fiona
>
>> -----Original Message-----
>> From: Verma, Shally [mailto:Shally.Verma@cavium.com]
>> Sent: Tuesday, February 27, 2018 5:54 AM
>> To: Ahmed Mansour <ahmed.mansour@nxp.com>; Trahe, Fiona <fiona.trahe@intel.com>;
>> dev@dpdk.org
>> Cc: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Athreya, Narayana Prasad
>> <NarayanaPrasad.Athreya@cavium.com>; Gupta, Ashish <Ashish.Gupta@cavium.com>; Sahu, Sunila
>> <Sunila.Sahu@cavium.com>; Challa, Mahipal <Mahipal.Challa@cavium.com>; Jain, Deepak K
>> <deepak.k.jain@intel.com>; Hemant Agrawal <hemant.agrawal@nxp.com>; Roy Pledge
>> <roy.pledge@nxp.com>; Youri Querry <youri.querry_1@nxp.com>
>> Subject: RE: [dpdk-dev] [PATCH] compressdev: implement API
>>
>>
>>
>> >-----Original Message-----
>> >From: Ahmed Mansour [mailto:ahmed.mansour@nxp.com]
>> >Sent: 27 February 2018 03:05
>> >To: Verma, Shally <Shally.Verma@cavium.com>; Trahe, Fiona <fiona.trahe@intel.com>; dev@dpdk.org
>> >Cc: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Athreya, Narayana Prasad
>> <NarayanaPrasad.Athreya@cavium.com>;
>> >Gupta, Ashish <Ashish.Gupta@cavium.com>; Sahu, Sunila <Sunila.Sahu@cavium.com>; Challa,
>> Mahipal
>> ><Mahipal.Challa@cavium.com>; Jain, Deepak K <deepak.k.jain@intel.com>; Hemant Agrawal
>> <hemant.agrawal@nxp.com>; Roy
>> >Pledge <roy.pledge@nxp.com>; Youri Querry <youri.querry_1@nxp.com>
>> >Subject: Re: [dpdk-dev] [PATCH] compressdev: implement API
>> >
>> >> Hi Fiona, Ahmed
>> >>> Hi Fiona,
>> >>>
>> >>> Thanks for starting this discussion. In the current API the user must
>> >>> make 12 API calls just to get information to compress. Maybe there is a
>> >>> way to simplify. At least for some use cases (stateless). I think a call
>> >>> sometime next week would be good to help clarify coalesce some of the
>> >>> complexity.
>> >>>
>> >>> I added specific comments inline.
>> >>>
>> >>> Thanks,
>> >>>
>> >>> Ahmed
>> >>>
>> >>> On 2/21/2018 2:12 PM, Trahe, Fiona wrote:
>> >>>> We've been struggling with the idea of session in compressdev.
>> >>>>
>> >>>> Is it really a session?
>> >>>>  - It's not in the same sense as cryptodev where it's used to hold a key, and maps to a Security
>> Association.
>> >>>>  - It's a set of immutable data that is needed with the op and stream to perform the operation.
>> >>>>  - It inherited from cryptodev the ability to be set up for multiple driver types and used across any
>> >>>>     devices of those types. For stateful ops this facility can't be used.
>> >>>>     For stateless we don't think it's important, and think it's unlikely to be used.
>> >>>>  - Drivers use it to prepare private data, set up resources, do pre-work, so there's
>> >>>>     less work to be done on the data path. Initially we didn't have a stream, we do now,
>> >>>>     this may be a better alternative place for that work.
>> >>>> So we've been toying with the idea of getting rid of the session.
>> >>> [Ahmed] In our proprietary API the stream and session are one. A session
>> >>> holds many properties like the op-type, instead of having this
>> >>> information in the op itself.  This way we lower the per op setup cost.
>> >>> This also allows rapid reuse of stateful infrastructure, once a stream
>> >>> is closed on a stateful session, the next op (stream) on this session
>> >>> reuses the stateful storage. Obviously if a stream is in "pause mode" on
>> >>> a session, all following ops that may be unrelated to this
>> >>> stream/session must also wait until this current stream is closed or
>> >>> aborted before the infrastructure can be reused.
>> >>>> We also struggle with the idea of setting up a stream for stateless ops.
>> >>>>   - Well, really I just think the name is misleading, i.e. there's no problem with setting
>> >>>>     up some private PMD data to use with stateless operations, just calling it a
>> >>>>     stream doesn't seem right.
>> >>> [Ahmed] I agree. The op has all the necessary information to process it
>> >>> in the current API? Both the stream and the op are one time use. We
>> >>> can't attach multiple similar ops to a single stream/session and rely on
>> >>> their properties to simplify op setup, so why the hassle.
>> >> [Shally]  As per my knowledge, session came with idea in DPDK, if system has multiple devices setup
>> to do similar jobs then
>> >application can fan out ops to any of them for load-balancing. Though it is not possible for stateful ops
>> but it still can work for stateless.
>> >If there's an application which only have stateless ops to process then I see this is still useful feature to
>> support.
>> >[Ahmed] Is there an advantage to exposing load balancing to the user? I
>> >do not see load balancing as a feature within itself. Can the PMD take
>> >care of this? I guess a system that has
>>
>> [Shally] I assume idea was to leverage multiple PMDs that are available in system (say QAT+SW ZLIB)
>> and I believe matter of load-balancing came out of one of the earlier discussion with Fiona on RFC v1.
>> http://dev.dpdk.narkive.com/CHS5l01B/dpdk-dev-rfc-v1-doc-compression-api-for-dpdk#post3
>> So, I wait for her comments on this. But in any case, with changed notion too it looks achievable to me,
>> if so is desired.
>>
>> >> In current proposal, stream logically represent data and hold its specific information and session is
>> generic information that can be
>> >applied on multiple data. If we want to combine stream and session. Then one way to look at this is:
>> >>
>> >> "let application only allocate and initialize session with rte_comp_xform (and possibly op type)
>> information so that PMD can do one-
>> >time setup and allocate enough resources. Once attached to op, cannot be reused until that op is fully
>> processed. So, if app has 16
>> >data elements to process in a burst, it will setup 16 sessions."
>> >[Ahmed] Why not allow multiple inflight stateless ops with the same
>> >session? Stateless by definition guarantees that the resources used to
>> >work on one up will be free after the op is processed. That means that
>> >even if an op fails to process correctly on a session, it will have no
>> >effect on the next op since there is not interdependence. This assumes
>> >that the resources are shareable between hardware instances for
>> >stateless. That is not a bad assumption since hardware should not need
>> >more than the data of the op itself to work on a statelss op.
>>
>> [Shally]  multiple ops in-flight can connect to same session but I assume you agree then they cannot
>> execute in parallel i.e. only one op at-a-time can use session here? And as far as I understand your PMD
>> works this way. Your HW execute one op at-a-time from queue?!
>>
>> >> This is same as what Ahmed suggested. For a particular load-balancing case suggested above, If
>> application want, can initialize
>> >different sessions on multiple devices with same xform so that each is prepared to process ops.
>> Application can then fanout stateless
>> >ops to multiple devices for load-balancing but then it would need to keep map of device & a session
>> map.
>> >>
>> >> If this sound feasible, then I too believe we can rather get rid of either and keep one (possibly session
>> but am open with stream as
>> >well).
>> >> However, regardless of case whether we live with name stream or session, I don't see much deviation
>> from current API spec except
>> >description and few modifications/additions as identified.
>> >> So, then I see it as:
>> >>
>> >> - A stream(or session whichever name is chosen) can be used with only one-op at-a-time
>> >> - It can be re-used when previously attached op is processed
>> >> -  if it is stream then currently it is allocated from PMD managed pool whereas Sessions are allocated
>> from application created
>> >mempool.
>> >>    In either of case, I would expect to review pool management API
>> >>
>> >> With this in mind, below are few of my comments
>> >>
>> >>>> So putting above thoughts together I want to propose:
>> >>>> -	Removal of the session and all associated APIs.
>> >>>> -	Passing in one of three data types in the rte_comp_op
>> >>>>
>> >>>>     union {
>> >>>>         struct rte_comp_xform *xform;
>> >>>>         /**< Immutable compress/decompress params */
>> >>>>         void *pmd_stateless_data;
>> >>>>         /**< Stateless private PMD data derived from an rte_comp_xform
>> >>>>          * rte_comp_stateless_data_init() must be called on a device
>> >>>>          * before sending any STATELESS operations. If the PMD returns a non-NULL
>> >>>>          * value the handle must be attached to subsequent STATELESS operations.
>> >>>>          * If a PMD returns NULL, then the xform should be passed directly to each op
>> >>>>          */
>> >> [Shally] It sounds like stateless_data_init() nothing more than a replacement of session_init().
>> >> 	So, this is needed neither if we retain session concept nor if we retain stream concept (
>> rte_comp_stream_create() with
>> >op_type: stateless can serve same purpose).
>> >> 	It should be sufficient to provide either stream (or session) pointer.
>> >>
>> >>>>         void *stream;
>> >>>>         /* Private PMD data derived initially from an rte_comp_xform, which holds state
>> >>>>          * and history data and evolves as operations are processed.
>> >>>>          * rte_comp_stream_create() must be called on a device for all STATEFUL
>> >>>>          * data streams and the resulting stream attached
>> >>>>          * to the one or more operations associated with the data stream.
>> >>>>          * All operations in a stream must be sent to the same device.
>> >>>>          */
>> >>>>     }
>> >>> [Ahmed] I like this setup, but I am not sure in what cases the xform
>> >>> immutable would be used. I understand the other two.
>> >> [Shally] my understanding is xform will be mapped by PMD to its internally managed stream(or
>> session data structure). And then we
>> >can remove STATEFUL reference here and just say stream(or session) it belongs to. However, This
>> condition still apply:
>> >>        *All operations that belong to same stream must be sent to the same device.*
>> >>
>> >>>> Notes:
>> >>>> 1. Internally if a PMD wants to use the exact same data structure for both it can do,
>> >>>>      just on the API I think it's better if they're named differently with
>> >>>>      different comments.
>> >>>> 2. I'm not clear of the constraints if any, which attach to the pmd_stateless_data
>> >>>>      For our PMD it would only hold immutable data as the session did, and so
>> >>>>      could be attached to many ops in parallel.
>> >>>>      Is this true for all PMDs or are there constraints which should be called out?
>> >>>>      Is it limited to a specific device, qp, or to be used on one op at a time?
>> >>>> 3. Am open to other naming suggestions, just trying to capture the essence
>> >>>>     of these data structs better than our current API does.
>> >>>>
>> >>>> We would put some more helper fns and structure around the above code if people
>> >>>> are in agreement, just want to see if the concept flies before going further?
>> >>>>
>> >>>> Fiona
>> >>>>
>> >>>>
>> >>>>
>> >>

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Trahe, Fiona]

Hi Shally

//snip//
> [Shally] This looks better to me. So it mean app would always call xform_init() for stateless and attach an
> updated priv_xform to ops (depending upon if there's shareable or not). So it does not need to have
> NULL pointer on priv_xform. right?
> 
[Fiona] yes. The PMD must return a valid priv_xform pointer.

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Ahmed Mansour]

On 3/1/2018 9:41 AM, Trahe, Fiona wrote:
> Hi Shally
>
> //snip//
>> [Shally] This looks better to me. So it mean app would always call xform_init() for stateless and attach an
>> updated priv_xform to ops (depending upon if there's shareable or not). So it does not need to have
>> NULL pointer on priv_xform. right?
>>
> [Fiona] yes. The PMD must return a valid priv_xform pointer.

[Ahmed] What I understood is that the xform_init will be called once
initially. if the @flag returned is NONE_SHAREABLE then the application
must not attach two inflight ops to the same @priv_xform? Otherwise the
application can attach many ops in flight to the @priv_xform?

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Trahe, Fiona]

> -----Original Message-----
> From: Ahmed Mansour [mailto:ahmed.mansour@nxp.com]
> Sent: Friday, March 2, 2018 12:56 AM
> To: Trahe, Fiona <fiona.trahe@intel.com>; Verma, Shally <Shally.Verma@cavium.com>; dev@dpdk.org
> Cc: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Athreya, Narayana Prasad
> <NarayanaPrasad.Athreya@cavium.com>; Gupta, Ashish <Ashish.Gupta@cavium.com>; Sahu, Sunila
> <Sunila.Sahu@cavium.com>; Challa, Mahipal <Mahipal.Challa@cavium.com>; Jain, Deepak K
> <deepak.k.jain@intel.com>; Hemant Agrawal <hemant.agrawal@nxp.com>; Roy Pledge
> <roy.pledge@nxp.com>; Youri Querry <youri.querry_1@nxp.com>
> Subject: Re: [dpdk-dev] [PATCH] compressdev: implement API
> 
> On 3/1/2018 9:41 AM, Trahe, Fiona wrote:
> > Hi Shally
> >
> > //snip//
> >> [Shally] This looks better to me. So it mean app would always call xform_init() for stateless and attach
> an
> >> updated priv_xform to ops (depending upon if there's shareable or not). So it does not need to have
> >> NULL pointer on priv_xform. right?
> >>
> > [Fiona] yes. The PMD must return a valid priv_xform pointer.
> 
> [Ahmed] What I understood is that the xform_init will be called once
> initially. if the @flag returned is NONE_SHAREABLE then the application
> must not attach two inflight ops to the same @priv_xform? Otherwise the
> application can attach many ops in flight to the @priv_xform?
[Fiona Yes. App calls the xform_init() once on a device where it plans to send stateless ops.
If PMD returns shareable, then it doesn't need to call again and can attach this to every stateless op going to that device.
If PMD returns SINGLE_OP then it must call xform_init() before every other 
stateless op it wants to have inflight simultaneously. This does not mean it must be called before every op, 
but probably will set up a batch of priv_xforms  - it can reuse each priv_xform once the op finishes with it.

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Ahmed Mansour]

On 3/2/2018 4:53 AM, Trahe, Fiona wrote:
>
>> On 3/1/2018 9:41 AM, Trahe, Fiona wrote:
>>> Hi Shally
>>>
>>> //snip//
>>>> [Shally] This looks better to me. So it mean app would always call xform_init() for stateless and attach
>> an
>>>> updated priv_xform to ops (depending upon if there's shareable or not). So it does not need to have
>>>> NULL pointer on priv_xform. right?
>>>>
>>> [Fiona] yes. The PMD must return a valid priv_xform pointer.
>> [Ahmed] What I understood is that the xform_init will be called once
>> initially. if the @flag returned is NONE_SHAREABLE then the application
>> must not attach two inflight ops to the same @priv_xform? Otherwise the
>> application can attach many ops in flight to the @priv_xform?
> [Fiona Yes. App calls the xform_init() once on a device where it plans to send stateless ops.
> If PMD returns shareable, then it doesn't need to call again and can attach this to every stateless op going to that device.
> If PMD returns SINGLE_OP then it must call xform_init() before every other 
> stateless op it wants to have inflight simultaneously. This does not mean it must be called before every op, 
> but probably will set up a batch of priv_xforms  - it can reuse each priv_xform once the op finishes with it.
[Ahmed] @Shally Can this complexity of managing the NONE_SHAREABLE mode
be pushed into the PMD? A flexible stockpile can be kept and maintained
by the PMD and it can be increased or decreased based on
low-water/high-water thresholds

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Verma, Shally]

>-----Original Message-----
>From: Ahmed Mansour [mailto:ahmed.mansour@nxp.com]
>Sent: 03 March 2018 01:19
>To: Trahe, Fiona <fiona.trahe@intel.com>; Verma, Shally <Shally.Verma@cavium.com>; dev@dpdk.org
>Cc: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Athreya, Narayana Prasad <NarayanaPrasad.Athreya@cavium.com>;
>Gupta, Ashish <Ashish.Gupta@cavium.com>; Sahu, Sunila <Sunila.Sahu@cavium.com>; Challa, Mahipal
><Mahipal.Challa@cavium.com>; Jain, Deepak K <deepak.k.jain@intel.com>; Hemant Agrawal <hemant.agrawal@nxp.com>; Roy
>Pledge <roy.pledge@nxp.com>; Youri Querry <youri.querry_1@nxp.com>
>Subject: Re: [dpdk-dev] [PATCH] compressdev: implement API
>
>On 3/2/2018 4:53 AM, Trahe, Fiona wrote:
>>
>>> On 3/1/2018 9:41 AM, Trahe, Fiona wrote:
>>>> Hi Shally
>>>>
>>>> //snip//
>>>>> [Shally] This looks better to me. So it mean app would always call xform_init() for stateless and attach
>>> an
>>>>> updated priv_xform to ops (depending upon if there's shareable or not). So it does not need to have
>>>>> NULL pointer on priv_xform. right?
>>>>>
>>>> [Fiona] yes. The PMD must return a valid priv_xform pointer.
>>> [Ahmed] What I understood is that the xform_init will be called once
>>> initially. if the @flag returned is NONE_SHAREABLE then the application
>>> must not attach two inflight ops to the same @priv_xform? Otherwise the
>>> application can attach many ops in flight to the @priv_xform?
>> [Fiona Yes. App calls the xform_init() once on a device where it plans to send stateless ops.
>> If PMD returns shareable, then it doesn't need to call again and can attach this to every stateless op going to that device.
>> If PMD returns SINGLE_OP then it must call xform_init() before every other
>> stateless op it wants to have inflight simultaneously. This does not mean it must be called before every op,
>> but probably will set up a batch of priv_xforms  - it can reuse each priv_xform once the op finishes with it.
>[Ahmed] @Shally Can this complexity of managing the NONE_SHAREABLE mode
>be pushed into the PMD? A flexible stockpile can be kept and maintained
>by the PMD and it can be increased or decreased based on
>low-water/high-water thresholds
[Shally] It is doable to manage within PMD but need to do hands on to evaluate effectiveness. So far, we have never exercised this way and left it to application to attach different session (or stream) to op for maximum performance gain. So, I would say, may it be ok to have flag feature in first place and deprecate later, if it not required?! Or just have API without any flag option and add a feature flag to indicate PMD support for SHAREABLE/NON-SHAREABLE xform_priv handle?!

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Ahmed Mansour]

On 3/5/2018 9:32 AM, Verma, Shally wrote:
>
>> -----Original Message-----
>> From: Ahmed Mansour [mailto:ahmed.mansour@nxp.com]
>> Sent: 03 March 2018 01:19
>> To: Trahe, Fiona <fiona.trahe@intel.com>; Verma, Shally <Shally.Verma@cavium.com>; dev@dpdk.org
>> Cc: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Athreya, Narayana Prasad <NarayanaPrasad.Athreya@cavium.com>;
>> Gupta, Ashish <Ashish.Gupta@cavium.com>; Sahu, Sunila <Sunila.Sahu@cavium.com>; Challa, Mahipal
>> <Mahipal.Challa@cavium.com>; Jain, Deepak K <deepak.k.jain@intel.com>; Hemant Agrawal <hemant.agrawal@nxp.com>; Roy
>> Pledge <roy.pledge@nxp.com>; Youri Querry <youri.querry_1@nxp.com>
>> Subject: Re: [dpdk-dev] [PATCH] compressdev: implement API
>>
>> On 3/2/2018 4:53 AM, Trahe, Fiona wrote:
>>>> On 3/1/2018 9:41 AM, Trahe, Fiona wrote:
>>>>> Hi Shally
>>>>>
>>>>> //snip//
>>>>>> [Shally] This looks better to me. So it mean app would always call xform_init() for stateless and attach
>>>> an
>>>>>> updated priv_xform to ops (depending upon if there's shareable or not). So it does not need to have
>>>>>> NULL pointer on priv_xform. right?
>>>>>>
>>>>> [Fiona] yes. The PMD must return a valid priv_xform pointer.
>>>> [Ahmed] What I understood is that the xform_init will be called once
>>>> initially. if the @flag returned is NONE_SHAREABLE then the application
>>>> must not attach two inflight ops to the same @priv_xform? Otherwise the
>>>> application can attach many ops in flight to the @priv_xform?
>>> [Fiona Yes. App calls the xform_init() once on a device where it plans to send stateless ops.
>>> If PMD returns shareable, then it doesn't need to call again and can attach this to every stateless op going to that device.
>>> If PMD returns SINGLE_OP then it must call xform_init() before every other
>>> stateless op it wants to have inflight simultaneously. This does not mean it must be called before every op,
>>> but probably will set up a batch of priv_xforms  - it can reuse each priv_xform once the op finishes with it.
>> [Ahmed] @Shally Can this complexity of managing the NONE_SHAREABLE mode
>> be pushed into the PMD? A flexible stockpile can be kept and maintained
>> by the PMD and it can be increased or decreased based on
>> low-water/high-water thresholds
> [Shally] It is doable to manage within PMD but need to do hands on to evaluate effectiveness. So far, we have never exercised this way and left it to application to attach different session (or stream) to op for maximum performance gain. So, I would say, may it be ok to have flag feature in first place and deprecate later, if it not required?! Or just have API without any flag option and add a feature flag to indicate PMD support for SHAREABLE/NON-SHAREABLE xform_priv handle?!
[Ahmed] Either way looks ok to me. I see your point about performance.
If this is in the PMD it will have to constantly guess how much memory
the user needs and accommodate dynamically. The user can implement a
similar scheme or if the application is simple they can pre-allocate and
reduce CPU allocation de-allocation overhead.

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Ahmed Mansour]

On 2/28/2018 1:39 PM, Trahe, Fiona wrote:
> Hi Ahmed, Shally,
>
> So just to capture what we concluded in the call today:
>
>  - There's no requirement for a device-agnostic session to facilitate load-balancing.
>  - For stateful data a stream is compulsory. Xform is passed to stream on creation. 
>     So no need for a session in stateful op.
>
> Re session data for stateless ops:
>  - All PMDs could cope with just passing in a xform with a stateless op. But it might 
>    not be performant. 
>  - Some PMDs need to allocate some resources which can only be used by one op
>    at a time. For stateful ops these resources can be attached to a stream. For stateless
>    they could allocate the resources on each op enqueue, but it would be better if
>    the resources were setup once based on the xform and could be re-used on ops,
>    though only by one op at a time. 
>  - Some PMDs don't need to allocate such resources, but could benefit by
>    setting up some pmd data based on the xform. This data would not be 
>    constrained, could be used in parallel by any op or qp of the device. 
>  - The name pmd_stateless_data was not popular, maybe something like 
>     xform_private_data can be used. On creation of this data, the PMD can return 
>     an indication of whether it should be used by one op at a time or shared.  
>   
> So I'll 
>  - remove the session completely from the API.
>  - add an initialiser API for the data to be attached to stateless ops
>  - add a union to the op:
>
>  union {
>         void *pmd_private_xform;
>         /**< Stateless private PMD data derived from an rte_comp_xform
>          * rte_comp_xform_init() must be called on a device 
>          * before sending any STATELESS operations. The PMD returns a handle
>          * which must be attached to subsequent STATELESS operations.
>          * The PMD also returns a flag, if this is COMP_PRIVATE_XFORM_SHAREABLE
>          * then the xform can be attached to multiple ops at the same time, 
>          * if it's COMP_PRIVATE_XFORM_SINGLE_OP then it can only be
>          * be used on one op at a time, other private xforms must be initialised
>          * to send other ops in parallel. 
>          */
>         void *stream;
>         /* Private PMD data derived initially from an rte_comp_xform, which holds state
>          * and history data and evolves as operations are processed.
>          * rte_comp_stream_create() must be called on a device for all STATEFUL 
>          * data streams and the resulting stream attached
>          * to the one or more operations associated with the data stream.
>          * All operations in a stream must be sent to the same device.
>          */
>     }
>
> Previous startup flow before sending a stateful op: 
> rte_comp_get_private_size(devid)
> rte_comp_mempool_create() - returns sess_pool
> rte_comp_session_create(sess_pool)
> rte_comp_session_init(devid, sess, sess_pool, xform)
> rte_comp_stream_create(devid, sess, **stream, op_type)
>
> simplified to:
> rte_comp_xform_init(devid, xform, **priv_xform, *flag) - returns handle and flag 
> (pool is within the PMD)
>
> Note, I don't think we bottomed out on removing the xform from the union, but I don't
> think we need it with above solution. 
>
> Other discussion:
>  - we should document on API that qp is not thread-safe, so enqueue
>    and dequeue should be performed by same thread.
[Ahmed] - I understand a qp should represent a single software user.
This is good because we will not have to add locking as you mentioned,
but are you sure that dequeues cannot be performed by another thread
without adding significant overhead? it would enable producer consumer
applications, and the name queue pair implies some independence.
- Another question. I want to be sure. A qp can be used to send both
compress and decompress ops.
- Nitpick: The name queue pair implies order preservation to the user.
Maybe we should change it to something that does not imply that.
>
> device and qp flow:
>  - dev_info_get() - application reads device capabilities, including the max qps the device can support.
>  - dev_config() - application specifies how many qps it intends to use - typically one per thread, must be < device max
>  - qp_setup() - called per qp. Creates the qp based on the size indicated by max_inflights
>  - dev_start() - once started device can't be reconfigured, must call dev_stop to reconfigure.
>
>
> Regards,
> Fiona
>
>> -----Original Message-----
>> From: Verma, Shally [mailto:Shally.Verma@cavium.com]
>> Sent: Tuesday, February 27, 2018 5:54 AM
>> To: Ahmed Mansour <ahmed.mansour@nxp.com>; Trahe, Fiona <fiona.trahe@intel.com>;
>> dev@dpdk.org
>> Cc: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Athreya, Narayana Prasad
>> <NarayanaPrasad.Athreya@cavium.com>; Gupta, Ashish <Ashish.Gupta@cavium.com>; Sahu, Sunila
>> <Sunila.Sahu@cavium.com>; Challa, Mahipal <Mahipal.Challa@cavium.com>; Jain, Deepak K
>> <deepak.k.jain@intel.com>; Hemant Agrawal <hemant.agrawal@nxp.com>; Roy Pledge
>> <roy.pledge@nxp.com>; Youri Querry <youri.querry_1@nxp.com>
>> Subject: RE: [dpdk-dev] [PATCH] compressdev: implement API
>>
>>
>>
>>> -----Original Message-----
>>> From: Ahmed Mansour [mailto:ahmed.mansour@nxp.com]
>>> Sent: 27 February 2018 03:05
>>> To: Verma, Shally <Shally.Verma@cavium.com>; Trahe, Fiona <fiona.trahe@intel.com>; dev@dpdk.org
>>> Cc: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Athreya, Narayana Prasad
>> <NarayanaPrasad.Athreya@cavium.com>;
>>> Gupta, Ashish <Ashish.Gupta@cavium.com>; Sahu, Sunila <Sunila.Sahu@cavium.com>; Challa,
>> Mahipal
>>> <Mahipal.Challa@cavium.com>; Jain, Deepak K <deepak.k.jain@intel.com>; Hemant Agrawal
>> <hemant.agrawal@nxp.com>; Roy
>>> Pledge <roy.pledge@nxp.com>; Youri Querry <youri.querry_1@nxp.com>
>>> Subject: Re: [dpdk-dev] [PATCH] compressdev: implement API
>>>
>>>> Hi Fiona, Ahmed
>>>>> Hi Fiona,
>>>>>
>>>>> Thanks for starting this discussion. In the current API the user must
>>>>> make 12 API calls just to get information to compress. Maybe there is a
>>>>> way to simplify. At least for some use cases (stateless). I think a call
>>>>> sometime next week would be good to help clarify coalesce some of the
>>>>> complexity.
>>>>>
>>>>> I added specific comments inline.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Ahmed
>>>>>
>>>>> On 2/21/2018 2:12 PM, Trahe, Fiona wrote:
>>>>>> We've been struggling with the idea of session in compressdev.
>>>>>>
>>>>>> Is it really a session?
>>>>>>  - It's not in the same sense as cryptodev where it's used to hold a key, and maps to a Security
>> Association.
>>>>>>  - It's a set of immutable data that is needed with the op and stream to perform the operation.
>>>>>>  - It inherited from cryptodev the ability to be set up for multiple driver types and used across any
>>>>>>     devices of those types. For stateful ops this facility can't be used.
>>>>>>     For stateless we don't think it's important, and think it's unlikely to be used.
>>>>>>  - Drivers use it to prepare private data, set up resources, do pre-work, so there's
>>>>>>     less work to be done on the data path. Initially we didn't have a stream, we do now,
>>>>>>     this may be a better alternative place for that work.
>>>>>> So we've been toying with the idea of getting rid of the session.
>>>>> [Ahmed] In our proprietary API the stream and session are one. A session
>>>>> holds many properties like the op-type, instead of having this
>>>>> information in the op itself.  This way we lower the per op setup cost.
>>>>> This also allows rapid reuse of stateful infrastructure, once a stream
>>>>> is closed on a stateful session, the next op (stream) on this session
>>>>> reuses the stateful storage. Obviously if a stream is in "pause mode" on
>>>>> a session, all following ops that may be unrelated to this
>>>>> stream/session must also wait until this current stream is closed or
>>>>> aborted before the infrastructure can be reused.
>>>>>> We also struggle with the idea of setting up a stream for stateless ops.
>>>>>>   - Well, really I just think the name is misleading, i.e. there's no problem with setting
>>>>>>     up some private PMD data to use with stateless operations, just calling it a
>>>>>>     stream doesn't seem right.
>>>>> [Ahmed] I agree. The op has all the necessary information to process it
>>>>> in the current API? Both the stream and the op are one time use. We
>>>>> can't attach multiple similar ops to a single stream/session and rely on
>>>>> their properties to simplify op setup, so why the hassle.
>>>> [Shally]  As per my knowledge, session came with idea in DPDK, if system has multiple devices setup
>> to do similar jobs then
>>> application can fan out ops to any of them for load-balancing. Though it is not possible for stateful ops
>> but it still can work for stateless.
>>> If there's an application which only have stateless ops to process then I see this is still useful feature to
>> support.
>>> [Ahmed] Is there an advantage to exposing load balancing to the user? I
>>> do not see load balancing as a feature within itself. Can the PMD take
>>> care of this? I guess a system that has
>> [Shally] I assume idea was to leverage multiple PMDs that are available in system (say QAT+SW ZLIB)
>> and I believe matter of load-balancing came out of one of the earlier discussion with Fiona on RFC v1.
>> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdev.dpdk.narkive.com%2FCHS5l01B%2Fdpdk-dev-rfc-v1-doc-compression-api-for-dpdk%23post3&data=02%7C01%7Cahmed.mansour%40nxp.com%7C4299d16d58144e417f5208d57eda9ba9%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636554399768115871&sdata=MHWrD0qD%2FMKL%2FjX4j1kOdDeElh0cG1gQj9d1862K4V0%3D&reserved=0
>> So, I wait for her comments on this. But in any case, with changed notion too it looks achievable to me,
>> if so is desired.
>>
>>>> In current proposal, stream logically represent data and hold its specific information and session is
>> generic information that can be
>>> applied on multiple data. If we want to combine stream and session. Then one way to look at this is:
>>>> "let application only allocate and initialize session with rte_comp_xform (and possibly op type)
>> information so that PMD can do one-
>>> time setup and allocate enough resources. Once attached to op, cannot be reused until that op is fully
>> processed. So, if app has 16
>>> data elements to process in a burst, it will setup 16 sessions."
>>> [Ahmed] Why not allow multiple inflight stateless ops with the same
>>> session? Stateless by definition guarantees that the resources used to
>>> work on one up will be free after the op is processed. That means that
>>> even if an op fails to process correctly on a session, it will have no
>>> effect on the next op since there is not interdependence. This assumes
>>> that the resources are shareable between hardware instances for
>>> stateless. That is not a bad assumption since hardware should not need
>>> more than the data of the op itself to work on a statelss op.
>> [Shally]  multiple ops in-flight can connect to same session but I assume you agree then they cannot
>> execute in parallel i.e. only one op at-a-time can use session here? And as far as I understand your PMD
>> works this way. Your HW execute one op at-a-time from queue?!
>>
>>>> This is same as what Ahmed suggested. For a particular load-balancing case suggested above, If
>> application want, can initialize
>>> different sessions on multiple devices with same xform so that each is prepared to process ops.
>> Application can then fanout stateless
>>> ops to multiple devices for load-balancing but then it would need to keep map of device & a session
>> map.
>>>> If this sound feasible, then I too believe we can rather get rid of either and keep one (possibly session
>> but am open with stream as
>>> well).
>>>> However, regardless of case whether we live with name stream or session, I don't see much deviation
>> from current API spec except
>>> description and few modifications/additions as identified.
>>>> So, then I see it as:
>>>>
>>>> - A stream(or session whichever name is chosen) can be used with only one-op at-a-time
>>>> - It can be re-used when previously attached op is processed
>>>> -  if it is stream then currently it is allocated from PMD managed pool whereas Sessions are allocated
>> from application created
>>> mempool.
>>>>    In either of case, I would expect to review pool management API
>>>>
>>>> With this in mind, below are few of my comments
>>>>
>>>>>> So putting above thoughts together I want to propose:
>>>>>> -	Removal of the session and all associated APIs.
>>>>>> -	Passing in one of three data types in the rte_comp_op
>>>>>>
>>>>>>     union {
>>>>>>         struct rte_comp_xform *xform;
>>>>>>         /**< Immutable compress/decompress params */
>>>>>>         void *pmd_stateless_data;
>>>>>>         /**< Stateless private PMD data derived from an rte_comp_xform
>>>>>>          * rte_comp_stateless_data_init() must be called on a device
>>>>>>          * before sending any STATELESS operations. If the PMD returns a non-NULL
>>>>>>          * value the handle must be attached to subsequent STATELESS operations.
>>>>>>          * If a PMD returns NULL, then the xform should be passed directly to each op
>>>>>>          */
>>>> [Shally] It sounds like stateless_data_init() nothing more than a replacement of session_init().
>>>> 	So, this is needed neither if we retain session concept nor if we retain stream concept (
>> rte_comp_stream_create() with
>>> op_type: stateless can serve same purpose).
>>>> 	It should be sufficient to provide either stream (or session) pointer.
>>>>
>>>>>>         void *stream;
>>>>>>         /* Private PMD data derived initially from an rte_comp_xform, which holds state
>>>>>>          * and history data and evolves as operations are processed.
>>>>>>          * rte_comp_stream_create() must be called on a device for all STATEFUL
>>>>>>          * data streams and the resulting stream attached
>>>>>>          * to the one or more operations associated with the data stream.
>>>>>>          * All operations in a stream must be sent to the same device.
>>>>>>          */
>>>>>>     }
>>>>> [Ahmed] I like this setup, but I am not sure in what cases the xform
>>>>> immutable would be used. I understand the other two.
>>>> [Shally] my understanding is xform will be mapped by PMD to its internally managed stream(or
>> session data structure). And then we
>>> can remove STATEFUL reference here and just say stream(or session) it belongs to. However, This
>> condition still apply:
>>>>        *All operations that belong to same stream must be sent to the same device.*
>>>>
>>>>>> Notes:
>>>>>> 1. Internally if a PMD wants to use the exact same data structure for both it can do,
>>>>>>      just on the API I think it's better if they're named differently with
>>>>>>      different comments.
>>>>>> 2. I'm not clear of the constraints if any, which attach to the pmd_stateless_data
>>>>>>      For our PMD it would only hold immutable data as the session did, and so
>>>>>>      could be attached to many ops in parallel.
>>>>>>      Is this true for all PMDs or are there constraints which should be called out?
>>>>>>      Is it limited to a specific device, qp, or to be used on one op at a time?
>>>>>> 3. Am open to other naming suggestions, just trying to capture the essence
>>>>>>     of these data structs better than our current API does.
>>>>>>
>>>>>> We would put some more helper fns and structure around the above code if people
>>>>>> are in agreement, just want to see if the concept flies before going further?
>>>>>>
>>>>>> Fiona
>>>>>>
>>>>>>
>>>>>>
>

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Thomas Monjalon]

02/02/2018 19:25, Fiona Trahe:
>  config/common_base                                 |   6 +
>  doc/api/doxy-api-index.md                          |   1 +
>  doc/api/doxy-api.conf                              |   1 +
>  lib/Makefile                                       |   3 +
>  lib/librte_compressdev/Makefile                    |  29 +
>  lib/librte_compressdev/rte_comp.h                  | 503 ++++++++++++

Why rte_comp.h instead of the more consistent rte_compress.h?

>  lib/librte_compressdev/rte_compressdev.c           | 902 +++++++++++++++++++++
>  lib/librte_compressdev/rte_compressdev.h           | 757 +++++++++++++++++
>  lib/librte_compressdev/rte_compressdev_pmd.c       | 163 ++++
>  lib/librte_compressdev/rte_compressdev_pmd.h       | 439 ++++++++++
>  lib/librte_compressdev/rte_compressdev_version.map |  47 ++
>  lib/librte_eal/common/include/rte_log.h            |   1 +
>  mk/rte.app.mk                                      |   1 +
>  13 files changed, 2853 insertions(+)

Please update MAINTAINERS file and release notes.

Maybe it is worth splitting this patch in few shorter parts?


> --- a/config/common_base
> +++ b/config/common_base
> @@ -535,6 +535,12 @@ CONFIG_RTE_LIBRTE_PMD_MRVL_CRYPTO=n
>  CONFIG_RTE_LIBRTE_PMD_MRVL_CRYPTO_DEBUG=n
>  
>  #
> +# Compile generic compression device library
> +#
> +CONFIG_RTE_LIBRTE_COMPRESSDEV=y
> +CONFIG_RTE_COMPRESS_MAX_DEVS=64
> +
> +#
>  # Compile generic security library
>  #
>  CONFIG_RTE_LIBRTE_SECURITY=y
> diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
> index d77f205..07b8e75 100644
> --- a/doc/api/doxy-api-index.md
> +++ b/doc/api/doxy-api-index.md
> @@ -43,6 +43,7 @@ The public API headers are grouped by topics:
>    [rte_tm]             (@ref rte_tm.h),
>    [rte_mtr]            (@ref rte_mtr.h),
>    [bbdev]              (@ref rte_bbdev.h),
> +  [compressdev]        (@ref rte_compressdev.h),
>    [cryptodev]          (@ref rte_cryptodev.h),
>    [security]           (@ref rte_security.h),
>    [eventdev]           (@ref rte_eventdev.h),

Please move it between security and eventdev in these lists.

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Trahe, Fiona]

Hi Tomas,
Sorry for the delay in replying to this

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Sunday, February 4, 2018 2:25 PM
> To: Trahe, Fiona <fiona.trahe@intel.com>
> Cc: dev@dpdk.org; ahmed.mansour@nxp.com; Shally.Verma@cavium.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>
> Subject: Re: [dpdk-dev] [PATCH] compressdev: implement API
> 
> 02/02/2018 19:25, Fiona Trahe:
> >  config/common_base                                 |   6 +
> >  doc/api/doxy-api-index.md                          |   1 +
> >  doc/api/doxy-api.conf                              |   1 +
> >  lib/Makefile                                       |   3 +
> >  lib/librte_compressdev/Makefile                    |  29 +
> >  lib/librte_compressdev/rte_comp.h                  | 503 ++++++++++++
> 
> Why rte_comp.h instead of the more consistent rte_compress.h?
[Fiona]  I did originally... but ran into difficulty with horribly names like
RTE_COMPRESS_COMPRESS
RTE_COMPRESS_DECOMPRESS
rte_compress_compress_xform
rte_compress_decompress_xform
So compress is both the module prefix and the name of one of the actions.
I could have used compressdev - but names were very long.
So decided to opt for using 
_compressdev_ in names to do with the device and
_comp_ in names to do with the compression service

Also I could have used compdev instead of compressdev, 
but I felt compress should be in the lib name

> 
> >  lib/librte_compressdev/rte_compressdev.c           | 902 +++++++++++++++++++++
> >  lib/librte_compressdev/rte_compressdev.h           | 757 +++++++++++++++++
> >  lib/librte_compressdev/rte_compressdev_pmd.c       | 163 ++++
> >  lib/librte_compressdev/rte_compressdev_pmd.h       | 439 ++++++++++
> >  lib/librte_compressdev/rte_compressdev_version.map |  47 ++
> >  lib/librte_eal/common/include/rte_log.h            |   1 +
> >  mk/rte.app.mk                                      |   1 +
> >  13 files changed, 2853 insertions(+)
> 
> Please update MAINTAINERS file and release notes.
> 
> Maybe it is worth splitting this patch in few shorter parts?
[Fiona] will do
> 
> 
> > --- a/config/common_base
> > +++ b/config/common_base
> > @@ -535,6 +535,12 @@ CONFIG_RTE_LIBRTE_PMD_MRVL_CRYPTO=n
> >  CONFIG_RTE_LIBRTE_PMD_MRVL_CRYPTO_DEBUG=n
> >
> >  #
> > +# Compile generic compression device library
> > +#
> > +CONFIG_RTE_LIBRTE_COMPRESSDEV=y
> > +CONFIG_RTE_COMPRESS_MAX_DEVS=64
> > +
> > +#
> >  # Compile generic security library
> >  #
> >  CONFIG_RTE_LIBRTE_SECURITY=y
> > diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
> > index d77f205..07b8e75 100644
> > --- a/doc/api/doxy-api-index.md
> > +++ b/doc/api/doxy-api-index.md
> > @@ -43,6 +43,7 @@ The public API headers are grouped by topics:
> >    [rte_tm]             (@ref rte_tm.h),
> >    [rte_mtr]            (@ref rte_mtr.h),
> >    [bbdev]              (@ref rte_bbdev.h),
> > +  [compressdev]        (@ref rte_compressdev.h),
> >    [cryptodev]          (@ref rte_cryptodev.h),
> >    [security]           (@ref rte_security.h),
> >    [eventdev]           (@ref rte_eventdev.h),
> 
> Please move it between security and eventdev in these lists.
ok

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Thomas Monjalon]

23/03/2018 19:08, Trahe, Fiona:
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > 02/02/2018 19:25, Fiona Trahe:
> > >  lib/librte_compressdev/rte_comp.h                  | 503 ++++++++++++
> > 
> > Why rte_comp.h instead of the more consistent rte_compress.h?
> [Fiona]  I did originally... but ran into difficulty with horribly names like
> RTE_COMPRESS_COMPRESS
> RTE_COMPRESS_DECOMPRESS
> rte_compress_compress_xform
> rte_compress_decompress_xform
> So compress is both the module prefix and the name of one of the actions.
> I could have used compressdev - but names were very long.
> So decided to opt for using 
> _compressdev_ in names to do with the device and
> _comp_ in names to do with the compression service
> 
> Also I could have used compdev instead of compressdev, 
> but I felt compress should be in the lib name

I understand your concerns.
I don't like "comp" very much because it sounds like "comparison".
However, I don't have a better idea.
Sometimes naming is more difficult than coding :)

Re: [dpdk-dev] [PATCH] compressdev: implement API

[Trahe, Fiona]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Saturday, March 24, 2018 1:02 AM
> To: Trahe, Fiona <fiona.trahe@intel.com>
> Cc: dev@dpdk.org; ahmed.mansour@nxp.com; Shally.Verma@cavium.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; fiona.trahe@gmail.com
> Subject: Re: [dpdk-dev] [PATCH] compressdev: implement API
> 
> 23/03/2018 19:08, Trahe, Fiona:
> > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > 02/02/2018 19:25, Fiona Trahe:
> > > >  lib/librte_compressdev/rte_comp.h                  | 503 ++++++++++++
> > >
> > > Why rte_comp.h instead of the more consistent rte_compress.h?
> > [Fiona]  I did originally... but ran into difficulty with horribly names like
> > RTE_COMPRESS_COMPRESS
> > RTE_COMPRESS_DECOMPRESS
> > rte_compress_compress_xform
> > rte_compress_decompress_xform
> > So compress is both the module prefix and the name of one of the actions.
> > I could have used compressdev - but names were very long.
> > So decided to opt for using
> > _compressdev_ in names to do with the device and
> > _comp_ in names to do with the compression service
> >
> > Also I could have used compdev instead of compressdev,
> > but I felt compress should be in the lib name
> 
> I understand your concerns.
> I don't like "comp" very much because it sounds like "comparison".
> However, I don't have a better idea.
> Sometimes naming is more difficult than coding :)
[Fiona] True :)

[dpdk-dev] [PATCH v2 0/3] implement compression API

[Fiona Trahe]

With the vast amounts of data being transported around networks
and stored in storage systems, reducing data size is becoming ever
more important.

There are both software libraries and hardware devices available
that provide compression, but no common API. Such an API is proposed
in this commit, which supports the following features:

- Deflate Algorithm (https://tools.ietf.org/html/rfc1951)
- LZS algorithm (https://tools.ietf.org/html/rfc2395)
- Static and Dynamic Huffman encoding.
- Compression levels
- Checksum generation
- Asynchronous burst API
- private_xform - a place for PMDs to hold private data derived from 
  a xform and used by stateless operations.
- stream - a place for PMDs to hold private data derived from
  a xform and also maintain state and history data. For
  stateful flows.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara pablo.de.lara.guarch@intel.com
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>

Opens:
 - creation of private_xform and stream mempools. In v2, config
   added to give PMD the size params needed to create mempools.
   Still open as to whether the pools should be created by
   application, in API layer or by PMDs. Expect to resolve in a v3. 
 - alternative to mbufs for passing data between application
   and accelerator. A lot of varied opinions on this, expect to
   address in a future release.
 - addition of hash feature - to be added by Cavium
 - addition of feature to add capability to PMD to allow more than one
   inflight operation from a stateful stream - to be added by NXP 

Changes in v2:
 - Add stream APIs 
 - Remove session
 - Add SHAREABLE / NON_SHAREABLE private_xform types
 - Add algo enum 'UNSPECIFIED' to fix warning in capabilities
 - Change one remaining log to use dynamic logging.
 - Add rte_cache_aligned keyword to op
 - Rename enums with better names _ALGO, __CHECKSUM, _HUFFMAN_
 - Use const keyword when passing xform
 - Remove qp_count fn from dev_ops as never used
 - Remove max_nb_queue-pairs from compressdev_init_param as never used
 - Clarify device configure and start sequence
 - Replace OUT_OF_SPACE with OUT_OF_SPACE_RECOVERABLE and TERMINATED
   and clarified usage. 
 - Add stream and private_xform sizes to device config for use in
   mempool creation
 - Add capability helper fn
 - Use Base2 log value for window size on xforms
 - Add Meson build
 - Update MAINTAINERS
 - Update Map file
 - Change order in doxy file
 - Update Release note 


Fiona Trahe (3):
  compressdev: add structs and enum for compression service
  compressdev: implement API
  doc: update doxy and release note for compressdev

 MAINTAINERS                                        |   7 +
 config/common_base                                 |   6 +
 config/rte_config.h                                |   3 +
 doc/api/doxy-api-index.md                          |   1 +
 doc/api/doxy-api.conf                              |   1 +
 doc/guides/rel_notes/release_18_05.rst             |   6 +
 lib/Makefile                                       |   3 +
 lib/librte_compressdev/Makefile                    |  29 +
 lib/librte_compressdev/meson.build                 |   9 +
 lib/librte_compressdev/rte_comp.h                  | 528 ++++++++++++
 lib/librte_compressdev/rte_compressdev.c           | 904 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 727 +++++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.c       | 156 ++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 450 ++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  43 +
 lib/meson.build                                    |   2 +-
 mk/rte.app.mk                                      |   1 +
 17 files changed, 2875 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_compressdev/Makefile
 create mode 100644 lib/librte_compressdev/meson.build
 create mode 100644 lib/librte_compressdev/rte_comp.h
 create mode 100644 lib/librte_compressdev/rte_compressdev.c
 create mode 100644 lib/librte_compressdev/rte_compressdev.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.c
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_version.map

-- 
2.7.4

[dpdk-dev] [PATCH v3 00/13] Implement compression API

[Pablo de Lara]

With the vast amounts of data being transported around networks
and stored in storage systems, reducing data size is becoming ever more important.

There are both software libraries and hardware devices available that
provide compression, but no common API.
Such an API is proposed in this patchset, which supports the following features:

- Deflate Algorithm (https://tools.ietf.org/html/rfc1951)
- LZS algorithm (https://tools.ietf.org/html/rfc2395)
- Static and Dynamic Huffman encoding.
- Compression levels
- Checksum generation
- Asynchronous burst API
- private_xform - a place for PMDs to hold private data derived from
  a xform and used by stateless operations.
- stream - a place for PMDs to hold private data derived from
  a xform and also maintain state and history data. For
  stateful flows.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara pablo.de.lara.guarch@intel.com
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>

Opens:
 - creation of private_xform and stream mempools. In v3, config
   added to give PMD the size params needed to create mempools.
   Still open as to whether the pools should be created by
   application, in API layer or by PMDs. Expect to resolve in a v4. 
 - alternative to mbufs for passing data between application
   and accelerator. A lot of varied opinions on this, expect to
   address in a future release.
 - addition of feature to add capability to PMD to allow more than one
   inflight operation from a stateful stream - to be added by NXP 

Changes in v3:
 - Remove rte_comp_op_ctod helper functions
 - Remove param_range_check macro
 - Rename from phys_addr to iova_addr
 - Remove rte_comp_algo_strings
 - Rename rte_compressdev_pmd_is_valid_dev to rte_compressdev_is_valid_dev
 - Remove feature flags from compressdev
 - Support hash operations
 - Add shareable priv xform in feature flags, instead of returnin it
   on xform creation
 - Allow max number of queue pairs to be 0, meaning that there is no
   limit.
 - Add invalid configuration checks
 - Add capability helper functions

Changes in v2:
 - Add stream APIs
 - Remove session
 - Add SHAREABLE / NON_SHAREABLE private_xform types
 - Add algo enum 'UNSPECIFIED' to fix warning in capabilities
 - Change one remaining log to use dynamic logging.
 - Add rte_cache_aligned keyword to op
 - Rename enums with better names _ALGO, __CHECKSUM, _HUFFMAN_
 - Use const keyword when passing xform
 - Remove qp_count fn from dev_ops as never used
 - Remove max_nb_queue-pairs from compressdev_init_param as never used
 - Clarify device configure and start sequence
 - Replace OUT_OF_SPACE with OUT_OF_SPACE_RECOVERABLE and TERMINATED
   and clarified usage. 
 - Add stream and private_xform sizes to device config for use in
   mempool creation
 - Add capability helper fn
 - Use Base2 log value for window size on xforms
 - Add Meson build
 - Update MAINTAINERS
 - Update Map file
 - Change order in doxy file
 - Update Release note 

Fiona Trahe (12):
  compressdev: add basic device management
  compressdev: add queue pair management
  compressdev: add compression specific data
  compressdev: add enqueue/dequeue functions
  compressdev: add operation management
  compressdev: support stateless operations
  compressdev: support stateful operations
  compressdev: add device feature flags
  compressdev: add compression service feature flags
  compressdev: add device stats
  compressdev: add device capabilities
  compressdev: get device id from name

Shally Verma (1):
  compressdev: support hash operations

 MAINTAINERS                                        |   7 +
 config/common_base                                 |   6 +
 config/rte_config.h                                |   3 +
 doc/api/doxy-api-index.md                          |   1 +
 doc/api/doxy-api.conf                              |   1 +
 doc/guides/rel_notes/release_18_05.rst             |   6 +
 lib/Makefile                                       |   3 +
 lib/librte_compressdev/Makefile                    |  29 +
 lib/librte_compressdev/meson.build                 |   9 +
 lib/librte_compressdev/rte_comp.h                  | 539 ++++++++++++
 lib/librte_compressdev/rte_compressdev.c           | 902 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 701 ++++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.c       | 157 ++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 437 ++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  44 +
 lib/meson.build                                    |   2 +-
 mk/rte.app.mk                                      |   1 +
 17 files changed, 2847 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_compressdev/Makefile
 create mode 100644 lib/librte_compressdev/meson.build
 create mode 100644 lib/librte_compressdev/rte_comp.h
 create mode 100644 lib/librte_compressdev/rte_compressdev.c
 create mode 100644 lib/librte_compressdev/rte_compressdev.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.c
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_version.map

-- 
2.14.3

[dpdk-dev] [PATCH v3 01/13] compressdev: add basic device management

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Add basic functions to manage compress devices,
including driver and device allocation, and the basic
interface with compressdev PMDs.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 MAINTAINERS                                        |   7 +
 config/common_base                                 |   6 +
 config/rte_config.h                                |   3 +
 doc/api/doxy-api-index.md                          |   1 +
 doc/api/doxy-api.conf                              |   1 +
 doc/guides/rel_notes/release_18_05.rst             |   6 +
 lib/Makefile                                       |   3 +
 lib/librte_compressdev/Makefile                    |  28 ++
 lib/librte_compressdev/meson.build                 |   8 +
 lib/librte_compressdev/rte_compressdev.c           | 464 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 267 ++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.c       | 157 +++++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 283 +++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  31 ++
 lib/meson.build                                    |   2 +-
 mk/rte.app.mk                                      |   1 +
 16 files changed, 1267 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_compressdev/Makefile
 create mode 100644 lib/librte_compressdev/meson.build
 create mode 100644 lib/librte_compressdev/rte_compressdev.c
 create mode 100644 lib/librte_compressdev/rte_compressdev.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.c
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 1081d71aa..75f13f92e 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -325,6 +325,13 @@ F: drivers/raw/skeleton_rawdev/
 F: test/test/test_rawdev.c
 F: doc/guides/prog_guide/rawdev.rst
 
+Compression API - EXPERIMENTAL
+M: Fiona Trahe <fiona.trahe@intel.com>
+M: Pablo de Lara <pablo.de.lara.guarch@intel.com>
+M: Ashish Gupta <ashish.gupta@caviumnetworks.com>
+T: git://dpdk.org/next/dpdk-next-crypto
+F: lib/librte_compressdev/
+
 
 Bus Drivers
 -----------
diff --git a/config/common_base b/config/common_base
index ad4706267..90f205a21 100644
--- a/config/common_base
+++ b/config/common_base
@@ -538,6 +538,12 @@ CONFIG_RTE_LIBRTE_PMD_CCP_CPU_AUTH=n
 CONFIG_RTE_LIBRTE_PMD_MRVL_CRYPTO=n
 CONFIG_RTE_LIBRTE_PMD_MRVL_CRYPTO_DEBUG=n
 
+#
+# Compile generic compression device library
+#
+CONFIG_RTE_LIBRTE_COMPRESSDEV=y
+CONFIG_RTE_COMPRESS_MAX_DEVS=64
+
 #
 # Compile generic security library
 #
diff --git a/config/rte_config.h b/config/rte_config.h
index db6ceb6cd..f731eaace 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -47,6 +47,9 @@
 #define RTE_ETHDEV_QUEUE_STAT_CNTRS 16
 #define RTE_ETHDEV_RXTX_CALLBACKS 1
 
+/* compressdev defines */
+#define RTE_COMPRESS_MAX_DEVS 64
+
 /* cryptodev defines */
 #define RTE_CRYPTO_MAX_DEVS 64
 #define RTE_CRYPTODEV_NAME_LEN 64
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index d77f205bb..530808e9d 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -45,6 +45,7 @@ The public API headers are grouped by topics:
   [bbdev]              (@ref rte_bbdev.h),
   [cryptodev]          (@ref rte_cryptodev.h),
   [security]           (@ref rte_security.h),
+  [compressdev]        (@ref rte_compressdev.h),
   [eventdev]           (@ref rte_eventdev.h),
   [event_eth_rx_adapter]   (@ref rte_event_eth_rx_adapter.h),
   [rawdev]             (@ref rte_rawdev.h),
diff --git a/doc/api/doxy-api.conf b/doc/api/doxy-api.conf
index cda52fdfb..06432c3aa 100644
--- a/doc/api/doxy-api.conf
+++ b/doc/api/doxy-api.conf
@@ -45,6 +45,7 @@ INPUT                   = doc/api/doxy-api-index.md \
                           lib/librte_cfgfile \
                           lib/librte_cmdline \
                           lib/librte_compat \
+                          lib/librte_compressdev \
                           lib/librte_cryptodev \
                           lib/librte_distributor \
                           lib/librte_efd \
diff --git a/doc/guides/rel_notes/release_18_05.rst b/doc/guides/rel_notes/release_18_05.rst
index 7c428439e..77d3c1bf8 100644
--- a/doc/guides/rel_notes/release_18_05.rst
+++ b/doc/guides/rel_notes/release_18_05.rst
@@ -70,6 +70,11 @@ New Features
 
   * AES-CMAC (128-bit key).
 
+* **Added compressdev API, a generic compression service library.**
+
+  The compressdev library provides an API for offload of compression and
+  decompression operations to hardware or software accelerator devices.
+
 
 API Changes
 -----------
@@ -155,6 +160,7 @@ The libraries prepended with a plus sign were incremented in this version.
      librte_cfgfile.so.2
      librte_cmdline.so.2
    + librte_common_octeontx.so.1
+   + librte_compressdev.so.1
      librte_cryptodev.so.4
      librte_distributor.so.1
    + librte_eal.so.7
diff --git a/lib/Makefile b/lib/Makefile
index ec965a606..19396daff 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -26,6 +26,9 @@ DEPDIRS-librte_bbdev := librte_eal librte_mempool librte_mbuf
 DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += librte_cryptodev
 DEPDIRS-librte_cryptodev := librte_eal librte_mempool librte_ring librte_mbuf
 DEPDIRS-librte_cryptodev += librte_kvargs
+DIRS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += librte_compressdev
+DEPDIRS-librte_compressdev := librte_eal librte_mempool librte_ring librte_mbuf
+DEPDIRS-librte_compressdev += librte_kvargs
 DIRS-$(CONFIG_RTE_LIBRTE_SECURITY) += librte_security
 DEPDIRS-librte_security := librte_eal librte_mempool librte_ring librte_mbuf
 DEPDIRS-librte_security += librte_ether
diff --git a/lib/librte_compressdev/Makefile b/lib/librte_compressdev/Makefile
new file mode 100644
index 000000000..5f67ab817
--- /dev/null
+++ b/lib/librte_compressdev/Makefile
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2017-2018 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_compressdev.a
+
+# library version
+LIBABIVER := 1
+
+# build flags
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+LDLIBS += -lrte_eal -lrte_mempool -lrte_kvargs
+
+# library source files
+SRCS-y += rte_compressdev.c rte_compressdev_pmd.c
+
+# export include files
+SYMLINK-y-include += rte_compressdev.h
+SYMLINK-y-include += rte_compressdev_pmd.h
+
+# versioning export map
+EXPORT_MAP := rte_compressdev_version.map
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_compressdev/meson.build b/lib/librte_compressdev/meson.build
new file mode 100644
index 000000000..fc5eaf009
--- /dev/null
+++ b/lib/librte_compressdev/meson.build
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+allow_experimental_apis = true
+sources = files('rte_compressdev.c', 'rte_compressdev_pmd.c')
+headers = files('rte_compressdev.h',
+	'rte_compressdev_pmd.h')
+deps += ['kvargs', 'mbuf']
diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
new file mode 100644
index 000000000..d635953b2
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -0,0 +1,464 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#include <sys/types.h>
+#include <sys/queue.h>
+#include <ctype.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stdarg.h>
+#include <errno.h>
+#include <stdint.h>
+#include <inttypes.h>
+#include <netinet/in.h>
+
+#include <rte_byteorder.h>
+#include <rte_log.h>
+#include <rte_debug.h>
+#include <rte_dev.h>
+#include <rte_interrupts.h>
+#include <rte_memory.h>
+#include <rte_memcpy.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_tailq.h>
+#include <rte_eal.h>
+#include <rte_per_lcore.h>
+#include <rte_lcore.h>
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_mempool.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_errno.h>
+#include <rte_spinlock.h>
+#include <rte_string_fns.h>
+
+#include "rte_comp.h"
+#include "rte_compressdev.h"
+#include "rte_compressdev_pmd.h"
+
+static uint8_t nb_drivers;
+
+struct rte_compressdev rte_comp_devices[RTE_COMPRESS_MAX_DEVS];
+
+struct rte_compressdev *rte_compressdevs = &rte_comp_devices[0];
+
+static struct rte_compressdev_global compressdev_globals = {
+		.devs			= &rte_comp_devices[0],
+		.data			= { NULL },
+		.nb_devs		= 0,
+		.max_devs		= RTE_COMPRESS_MAX_DEVS
+};
+
+struct rte_compressdev_global *rte_compressdev_globals = &compressdev_globals;
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_get_dev(uint8_t dev_id)
+{
+	return &rte_compressdev_globals->devs[dev_id];
+}
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_get_named_dev(const char *name)
+{
+	struct rte_compressdev *dev;
+	unsigned int i;
+
+	if (name == NULL)
+		return NULL;
+
+	for (i = 0; i < rte_compressdev_globals->max_devs; i++) {
+		dev = &rte_compressdev_globals->devs[i];
+
+		if ((dev->attached == RTE_COMPRESSDEV_ATTACHED) &&
+				(strcmp(dev->data->name, name) == 0))
+			return dev;
+	}
+
+	return NULL;
+}
+
+unsigned int __rte_experimental
+rte_compressdev_is_valid_dev(uint8_t dev_id)
+{
+	struct rte_compressdev *dev = NULL;
+
+	if (dev_id >= rte_compressdev_globals->nb_devs)
+		return 0;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+	if (dev->attached != RTE_COMPRESSDEV_ATTACHED)
+		return 0;
+	else
+		return 1;
+}
+
+
+uint8_t __rte_experimental
+rte_compressdev_count(void)
+{
+	return rte_compressdev_globals->nb_devs;
+}
+
+uint8_t __rte_experimental
+rte_compressdev_device_count_by_driver(uint8_t driver_id)
+{
+	uint8_t i, dev_count = 0;
+
+	for (i = 0; i < rte_compressdev_globals->max_devs; i++)
+		if (rte_compressdev_globals->devs[i].driver_id == driver_id &&
+			rte_compressdev_globals->devs[i].attached ==
+					RTE_COMPRESSDEV_ATTACHED)
+			dev_count++;
+
+	return dev_count;
+}
+
+uint8_t __rte_experimental
+rte_compressdev_devices_get(const char *driver_name, uint8_t *devices,
+	uint8_t nb_devices)
+{
+	uint8_t i, count = 0;
+	struct rte_compressdev *devs = rte_compressdev_globals->devs;
+	uint8_t max_devs = rte_compressdev_globals->max_devs;
+
+	for (i = 0; i < max_devs && count < nb_devices;	i++) {
+
+		if (devs[i].attached == RTE_COMPRESSDEV_ATTACHED) {
+			int cmp;
+
+			cmp = strncmp(devs[i].device->driver->name,
+					driver_name,
+					strlen(driver_name));
+
+			if (cmp == 0)
+				devices[count++] = devs[i].data->dev_id;
+		}
+	}
+
+	return count;
+}
+
+int __rte_experimental
+rte_compressdev_socket_id(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id))
+		return -1;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	return dev->data->socket_id;
+}
+
+static inline int
+rte_compressdev_data_alloc(uint8_t dev_id, struct rte_compressdev_data **data,
+		int socket_id)
+{
+	char mz_name[RTE_COMPRESSDEV_NAME_MAX_LEN];
+	const struct rte_memzone *mz;
+	int n;
+
+	/* generate memzone name */
+	n = snprintf(mz_name, sizeof(mz_name),
+			"rte_compressdev_data_%u", dev_id);
+	if (n >= (int)sizeof(mz_name))
+		return -EINVAL;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		mz = rte_memzone_reserve(mz_name,
+				sizeof(struct rte_compressdev_data),
+				socket_id, 0);
+	} else
+		mz = rte_memzone_lookup(mz_name);
+
+	if (mz == NULL)
+		return -ENOMEM;
+
+	*data = mz->addr;
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		memset(*data, 0, sizeof(struct rte_compressdev_data));
+
+	return 0;
+}
+
+static uint8_t
+rte_compressdev_find_free_device_index(void)
+{
+	uint8_t dev_id;
+
+	for (dev_id = 0; dev_id < RTE_COMPRESS_MAX_DEVS; dev_id++) {
+		if (rte_comp_devices[dev_id].attached ==
+				RTE_COMPRESSDEV_DETACHED)
+			return dev_id;
+	}
+	return RTE_COMPRESS_MAX_DEVS;
+}
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_allocate(const char *name, int socket_id)
+{
+	struct rte_compressdev *compressdev;
+	uint8_t dev_id;
+
+	if (rte_compressdev_pmd_get_named_dev(name) != NULL) {
+		COMPRESSDEV_LOG(ERR,
+			"comp device with name %s already allocated!", name);
+		return NULL;
+	}
+
+	dev_id = rte_compressdev_find_free_device_index();
+	if (dev_id == RTE_COMPRESS_MAX_DEVS) {
+		COMPRESSDEV_LOG(ERR, "Reached maximum number of comp devices");
+		return NULL;
+	}
+	compressdev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (compressdev->data == NULL) {
+		struct rte_compressdev_data *compressdev_data =
+				compressdev_globals.data[dev_id];
+
+		int retval = rte_compressdev_data_alloc(dev_id,
+				&compressdev_data, socket_id);
+
+		if (retval < 0 || compressdev_data == NULL)
+			return NULL;
+
+		compressdev->data = compressdev_data;
+
+		snprintf(compressdev->data->name, RTE_COMPRESSDEV_NAME_MAX_LEN,
+				"%s", name);
+
+		compressdev->data->dev_id = dev_id;
+		compressdev->data->socket_id = socket_id;
+		compressdev->data->dev_started = 0;
+
+		compressdev->attached = RTE_COMPRESSDEV_ATTACHED;
+
+		compressdev_globals.nb_devs++;
+	}
+
+	return compressdev;
+}
+
+int __rte_experimental
+rte_compressdev_pmd_release_device(struct rte_compressdev *compressdev)
+{
+	int ret;
+
+	if (compressdev == NULL)
+		return -EINVAL;
+
+	/* Close device only if device operations have been set */
+	if (compressdev->dev_ops) {
+		ret = rte_compressdev_close(compressdev->data->dev_id);
+		if (ret < 0)
+			return ret;
+	}
+
+	compressdev->attached = RTE_COMPRESSDEV_DETACHED;
+	compressdev_globals.nb_devs--;
+	return 0;
+}
+
+int __rte_experimental
+rte_compressdev_configure(uint8_t dev_id, struct rte_compressdev_config *config)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	if (dev->data->dev_started) {
+		COMPRESSDEV_LOG(ERR,
+		    "device %d must be stopped to allow configuration", dev_id);
+		return -EBUSY;
+	}
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_configure, -ENOTSUP);
+
+	return (*dev->dev_ops->dev_configure)(dev, config);
+}
+
+
+int __rte_experimental
+rte_compressdev_start(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+	int diag;
+
+	COMPRESSDEV_LOG(DEBUG, "Start dev_id=%" PRIu8, dev_id);
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_start, -ENOTSUP);
+
+	if (dev->data->dev_started != 0) {
+		COMPRESSDEV_LOG(ERR,
+		    "Device with dev_id=%" PRIu8 " already started", dev_id);
+		return 0;
+	}
+
+	diag = (*dev->dev_ops->dev_start)(dev);
+	if (diag == 0)
+		dev->data->dev_started = 1;
+	else
+		return diag;
+
+	return 0;
+}
+
+void __rte_experimental
+rte_compressdev_stop(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_stop);
+
+	if (dev->data->dev_started == 0) {
+		COMPRESSDEV_LOG(ERR,
+		    "Device with dev_id=%" PRIu8 " already stopped", dev_id);
+		return;
+	}
+
+	(*dev->dev_ops->dev_stop)(dev);
+	dev->data->dev_started = 0;
+}
+
+int __rte_experimental
+rte_compressdev_close(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+	int retval;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -1;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	/* Device must be stopped before it can be closed */
+	if (dev->data->dev_started == 1) {
+		COMPRESSDEV_LOG(ERR, "Device %u must be stopped before closing",
+				dev_id);
+		return -EBUSY;
+	}
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_close, -ENOTSUP);
+	retval = (*dev->dev_ops->dev_close)(dev);
+
+	if (retval < 0)
+		return retval;
+
+	return 0;
+}
+
+void __rte_experimental
+rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
+{
+	struct rte_compressdev *dev;
+
+	if (dev_id >= compressdev_globals.nb_devs) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%d", dev_id);
+		return;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	memset(dev_info, 0, sizeof(struct rte_compressdev_info));
+
+	RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
+	(*dev->dev_ops->dev_infos_get)(dev, dev_info);
+
+	dev_info->driver_name = dev->device->driver->name;
+}
+
+TAILQ_HEAD(compressdev_driver_list, compressdev_driver);
+
+static struct compressdev_driver_list compressdev_driver_list =
+	TAILQ_HEAD_INITIALIZER(compressdev_driver_list);
+
+int __rte_experimental
+rte_compressdev_driver_id_get(const char *name)
+{
+	struct compressdev_driver *driver;
+	const char *driver_name;
+
+	if (name == NULL) {
+		COMPRESSDEV_LOG(DEBUG, "name pointer NULL");
+		return -1;
+	}
+
+	TAILQ_FOREACH(driver, &compressdev_driver_list, next) {
+		driver_name = driver->driver->name;
+		if (strncmp(driver_name, name, strlen(driver_name)) == 0)
+			return driver->id;
+	}
+	return -1;
+}
+
+const char * __rte_experimental
+rte_compressdev_name_get(uint8_t dev_id)
+{
+	struct rte_compressdev *dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (dev == NULL)
+		return NULL;
+
+	return dev->data->name;
+}
+
+const char * __rte_experimental
+rte_compressdev_driver_name_get(uint8_t driver_id)
+{
+	struct compressdev_driver *driver;
+
+	TAILQ_FOREACH(driver, &compressdev_driver_list, next)
+		if (driver->id == driver_id)
+			return driver->driver->name;
+	return NULL;
+}
+
+uint8_t
+rte_compressdev_allocate_driver(struct compressdev_driver *comp_drv,
+		const struct rte_driver *drv)
+{
+	comp_drv->driver = drv;
+	comp_drv->id = nb_drivers;
+
+	TAILQ_INSERT_TAIL(&compressdev_driver_list, comp_drv, next);
+
+	return nb_drivers++;
+}
+
+RTE_INIT(rte_compressdev_log);
+
+static void
+rte_compressdev_log(void)
+{
+	compressdev_logtype = rte_log_register("lib.compressdev");
+	if (compressdev_logtype >= 0)
+		rte_log_set_level(compressdev_logtype, RTE_LOG_NOTICE);
+}
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
new file mode 100644
index 000000000..c5de61378
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -0,0 +1,267 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMPRESSDEV_H_
+#define _RTE_COMPRESSDEV_H_
+
+/**
+ * @file rte_compressdev.h
+ *
+ * RTE Compression Device APIs
+ *
+ * Defines comp device APIs for the provisioning of compression operations.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include "rte_kvargs.h"
+#include "rte_comp.h"
+#include "rte_dev.h"
+#include <rte_common.h>
+
+/* Logging Macros */
+extern int compressdev_logtype;
+#define COMPRESSDEV_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, compressdev_logtype, "%s(): "fmt "\n", \
+			__func__, ##args)
+
+#define RTE_COMPRESSDEV_DETACHED  (0)
+#define RTE_COMPRESSDEV_ATTACHED  (1)
+
+#define RTE_COMPRESSDEV_NAME_MAX_LEN	(64)
+/**< Max length of name of comp PMD */
+
+/**  comp device information */
+struct rte_compressdev_info {
+	const char *driver_name;		/**< Driver name. */
+	uint8_t driver_id;			/**< Driver identifier */
+};
+
+/**
+ * Get the compress device name given a device identifier.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - Returns compress device name.
+ *   - Returns NULL if compress device is not present.
+ */
+const char * __rte_experimental
+rte_compressdev_name_get(uint8_t dev_id);
+
+/**
+ * Get the total number of compress devices that have been successfully
+ * initialised.
+ *
+ * @return
+ *   - The total number of usable compress devices.
+ */
+uint8_t __rte_experimental
+rte_compressdev_count(void);
+
+/**
+ * Get number of comp device defined type.
+ *
+ * @param driver_id
+ *   Driver identifier
+ * @return
+ *   Returns number of comp device.
+ */
+uint8_t __rte_experimental
+rte_compressdev_device_count_by_driver(uint8_t driver_id);
+
+/**
+ * Get number and identifiers of attached comp devices that
+ * use the same compress driver.
+ *
+ * @param driver_name
+ *   Driver name
+ * @param devices
+ *   Output devices identifiers
+ * @param nb_devices
+ *   Maximal number of devices
+ *
+ * @return
+ *   Returns number of attached compress devices.
+ */
+uint8_t __rte_experimental
+rte_compressdev_devices_get(const char *driver_name, uint8_t *devices,
+		uint8_t nb_devices);
+
+/**
+ * Validate if the comp device index is valid attached comp device.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - If the device index is valid (1) or not (0).
+ */
+unsigned int __rte_experimental
+rte_compressdev_is_valid_dev(uint8_t dev_id);
+
+/*
+ * Return the NUMA socket to which a device is connected.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   The NUMA socket id to which the device is connected or
+ *   a default of zero if the socket could not be determined.
+ *   -1 if returned is the dev_id value is out of range.
+ */
+int __rte_experimental
+rte_compressdev_socket_id(uint8_t dev_id);
+
+/** Compress device configuration structure */
+struct rte_compressdev_config {
+	int socket_id;
+};
+
+/**
+ * Configure a device.
+ *
+ * This function must be invoked first before any other function in the
+ * API. This function can also be re-invoked when a device is in the
+ * stopped state.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param config
+ *   The compress device configuration
+ * @return
+ *   - 0: Success, device configured.
+ *   - <0: Error code returned by the driver configuration function.
+ */
+int __rte_experimental
+rte_compressdev_configure(uint8_t dev_id,
+			struct rte_compressdev_config *config);
+
+/**
+ * Start a device.
+ *
+ * The device start step is called after configuring the device and setting up
+ * its queue pairs.
+ * On success, data-path functions exported by the API (enqueue/dequeue, etc)
+ * can be invoked.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - 0: Success, device started.
+ *   - <0: Error code of the driver device start function.
+ */
+int __rte_experimental
+rte_compressdev_start(uint8_t dev_id);
+
+/**
+ * Stop a device. The device can be restarted with a call to
+ * rte_compressdev_start()
+ *
+ * @param dev_id
+ *   Compress device identifier
+ */
+void __rte_experimental
+rte_compressdev_stop(uint8_t dev_id);
+
+/**
+ * Close an device. The device cannot be restarted!
+ *
+ * @param dev_id
+ *   Compress device identifier
+ *
+ * @return
+ *  - 0 on successfully closing device
+ *  - <0 on failure to close device
+ */
+int __rte_experimental
+rte_compressdev_close(uint8_t dev_id);
+
+/**
+ * Retrieve the contextual information of a device.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param dev_info
+ *   A pointer to a structure of type *rte_compressdev_info*
+ *   to be filled with the contextual information of the device
+ *
+ * @note The capabilities field of dev_info is set to point to the first
+ * element of an array of struct rte_compressdev_capabilities.
+ * The element after the last valid element has it's op field set to
+ * RTE_COMP_ALGO_LIST_END.
+ */
+void __rte_experimental
+rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info);
+
+/** The data structure associated with each comp device. */
+struct rte_compressdev {
+	struct rte_compressdev_data *data;
+	/**< Pointer to device data */
+	struct rte_compressdev_ops *dev_ops;
+	/**< Functions exported by PMD */
+	struct rte_device *device;
+	/**< Backing device */
+
+	uint8_t driver_id;
+	/**< comp driver identifier*/
+
+	__extension__
+	uint8_t attached : 1;
+	/**< Flag indicating the device is attached */
+} __rte_cache_aligned;
+
+/**
+ *
+ * The data part, with no function pointers, associated with each device.
+ *
+ * This structure is safe to place in shared memory to be common among
+ * different processes in a multi-process configuration.
+ */
+struct rte_compressdev_data {
+	uint8_t dev_id;
+	/**< Compress device identifier */
+	uint8_t socket_id;
+	/**< Socket identifier where memory is allocated */
+	char name[RTE_COMPRESSDEV_NAME_MAX_LEN];
+	/**< Unique identifier name */
+
+	__extension__
+	uint8_t dev_started : 1;
+	/**< Device state: STARTED(1)/STOPPED(0) */
+
+	void *dev_private;
+	/**< PMD-specific private data */
+} __rte_cache_aligned;
+
+struct rte_compressdev *rte_compressdevs;
+
+/**
+ * Provide driver identifier.
+ *
+ * @param name
+ *   Compress driver name
+ * @return
+ *  The driver type identifier or -1 if no driver found
+ */
+int __rte_experimental
+rte_compressdev_driver_id_get(const char *name);
+
+/**
+ * Provide driver name.
+ *
+ * @param driver_id
+ *   The driver identifier
+ * @return
+ *  The driver name or null if no driver found
+ */
+const char * __rte_experimental
+rte_compressdev_driver_name_get(uint8_t driver_id);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_COMPRESSDEV_H_ */
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.c b/lib/librte_compressdev/rte_compressdev_pmd.c
new file mode 100644
index 000000000..148596e0b
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_pmd.c
@@ -0,0 +1,157 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#include <rte_malloc.h>
+#include <rte_kvargs.h>
+
+#include "rte_compressdev_pmd.h"
+
+int compressdev_logtype;
+
+/**
+ * Parse name from argument
+ */
+static int
+rte_compressdev_pmd_parse_name_arg(const char *key __rte_unused,
+		const char *value, void *extra_args)
+{
+	struct rte_compressdev_pmd_init_params *params = extra_args;
+	int n;
+
+	n = snprintf(params->name, RTE_COMPRESSDEV_NAME_MAX_LEN, "%s", value);
+	if (n >= RTE_COMPRESSDEV_NAME_MAX_LEN)
+		return -EINVAL;
+
+	return 0;
+}
+
+/**
+ * Parse unsigned integer from argument
+ */
+static int
+rte_compressdev_pmd_parse_uint_arg(const char *key __rte_unused,
+		const char *value, void *extra_args)
+{
+	int i;
+	char *end;
+
+	errno = 0;
+	i = strtol(value, &end, 10);
+	if (*end != 0 || errno != 0 || i < 0)
+		return -EINVAL;
+
+	*((uint32_t *)extra_args) = i;
+	return 0;
+}
+
+int
+rte_compressdev_pmd_parse_input_args(
+		struct rte_compressdev_pmd_init_params *params,
+		const char *args)
+{
+	struct rte_kvargs *kvlist = NULL;
+	int ret = 0;
+
+	if (params == NULL)
+		return -EINVAL;
+
+	if (args) {
+		kvlist = rte_kvargs_parse(args,	compressdev_pmd_valid_params);
+		if (kvlist == NULL)
+			return -EINVAL;
+
+		ret = rte_kvargs_process(kvlist,
+				RTE_COMPRESSDEV_PMD_SOCKET_ID_ARG,
+				&rte_compressdev_pmd_parse_uint_arg,
+				&params->socket_id);
+		if (ret < 0)
+			goto free_kvlist;
+
+		ret = rte_kvargs_process(kvlist,
+				RTE_COMPRESSDEV_PMD_NAME_ARG,
+				&rte_compressdev_pmd_parse_name_arg,
+				params);
+		if (ret < 0)
+			goto free_kvlist;
+	}
+
+free_kvlist:
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_create(const char *name,
+		struct rte_device *device,
+		struct rte_compressdev_pmd_init_params *params)
+{
+	struct rte_compressdev *compressdev;
+
+	if (params->name[0] != '\0') {
+		COMPRESSDEV_LOG(INFO, "[%s] User specified device name = %s\n",
+				device->driver->name, params->name);
+		name = params->name;
+	}
+
+	COMPRESSDEV_LOG(INFO, "[%s] - Creating compressdev %s\n",
+			device->driver->name, name);
+
+	COMPRESSDEV_LOG(INFO,
+	"[%s] - Init parameters - name: %s, socket id: %d",
+			device->driver->name, name,
+			params->socket_id);
+
+	/* allocate device structure */
+	compressdev = rte_compressdev_pmd_allocate(name, params->socket_id);
+	if (compressdev == NULL) {
+		COMPRESSDEV_LOG(ERR, "[%s] Failed to allocate comp device %s",
+				device->driver->name, name);
+		return NULL;
+	}
+
+	/* allocate private device structure */
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		compressdev->data->dev_private =
+				rte_zmalloc_socket("compressdev device private",
+						params->private_data_size,
+						RTE_CACHE_LINE_SIZE,
+						params->socket_id);
+
+		if (compressdev->data->dev_private == NULL) {
+			COMPRESSDEV_LOG(ERR,
+		"[%s] Cannot allocate memory for compressdev %s private data",
+					device->driver->name, name);
+
+			rte_compressdev_pmd_release_device(compressdev);
+			return NULL;
+		}
+	}
+
+	compressdev->device = device;
+
+	return compressdev;
+}
+
+int __rte_experimental
+rte_compressdev_pmd_destroy(struct rte_compressdev *compressdev)
+{
+	int retval;
+
+	COMPRESSDEV_LOG(INFO, "[%s] Closing comp device %s",
+			compressdev->device->driver->name,
+			compressdev->device->name);
+
+	/* free comp device */
+	retval = rte_compressdev_pmd_release_device(compressdev);
+	if (retval)
+		return retval;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		rte_free(compressdev->data->dev_private);
+
+	compressdev->device = NULL;
+	compressdev->data = NULL;
+
+	return 0;
+}
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
new file mode 100644
index 000000000..1b49b48a6
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -0,0 +1,283 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMPRESSDEV_PMD_H_
+#define _RTE_COMPRESSDEV_PMD_H_
+
+/** @file
+ * RTE comp PMD APIs
+ *
+ * @note
+ * These APIs are for comp PMDs only and user applications should not call
+ * them directly.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <string.h>
+
+#include <rte_dev.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_mempool.h>
+#include <rte_log.h>
+#include <rte_common.h>
+
+#include "rte_compressdev.h"
+
+#define RTE_COMPRESSDEV_PMD_NAME_ARG			("name")
+#define RTE_COMPRESSDEV_PMD_SOCKET_ID_ARG		("socket_id")
+
+static const char * const compressdev_pmd_valid_params[] = {
+	RTE_COMPRESSDEV_PMD_NAME_ARG,
+	RTE_COMPRESSDEV_PMD_SOCKET_ID_ARG
+};
+
+/**
+ * @internal
+ * Initialisation parameters for comp devices
+ */
+struct rte_compressdev_pmd_init_params {
+	char name[RTE_COMPRESSDEV_NAME_MAX_LEN];
+	size_t private_data_size;
+	int socket_id;
+};
+
+/** Global structure used for maintaining state of allocated comp devices */
+struct rte_compressdev_global {
+	struct rte_compressdev *devs;	/**< Device information array */
+	struct rte_compressdev_data *data[RTE_COMPRESS_MAX_DEVS];
+	/**< Device private data */
+	uint8_t nb_devs;		/**< Number of devices found */
+	uint8_t max_devs;		/**< Max number of devices */
+};
+
+/* compressdev driver, containing the driver identifier */
+struct compressdev_driver {
+	TAILQ_ENTRY(compressdev_driver) next; /**< Next in list. */
+	const struct rte_driver *driver;
+	uint8_t id;
+};
+
+/** Pointer to global array of comp devices */
+extern struct rte_compressdev *rte_compressdevs;
+/** Pointer to global comp devices data structure */
+extern struct rte_compressdev_global *rte_compressdev_globals;
+
+/**
+ * Get the rte_compressdev structure device pointer for the device. Assumes a
+ * valid device index.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - The rte_compressdev structure pointer for the given device identifier.
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_get_dev(uint8_t dev_id);
+
+/**
+ * Get the rte_compressdev structure device pointer for the named device.
+ *
+ * @param name
+ *   Compress device name
+ * @return
+ *   - The rte_compressdev structure pointer for the given device identifier.
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_get_named_dev(const char *name);
+
+
+/**
+ * Definitions of all functions exported by a driver through the
+ * the generic structure of type *comp_dev_ops* supplied in the
+ * *rte_compressdev* structure associated with a device.
+ */
+
+/**
+ * Function used to configure device.
+ *
+ * @param dev
+ *   Compress device
+ * @param config
+ *   Compress device configurations
+ * @return
+ *   Returns 0 on success
+ */
+typedef int (*compressdev_configure_t)(struct rte_compressdev *dev,
+		struct rte_compressdev_config *config);
+
+/**
+ * Function used to start a configured device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *   Returns 0 on success
+ */
+typedef int (*compressdev_start_t)(struct rte_compressdev *dev);
+
+/**
+ * Function used to stop a configured device.
+ *
+ * @param dev
+ *   Compress device
+ */
+typedef void (*compressdev_stop_t)(struct rte_compressdev *dev);
+
+/**
+ * Function used to close a configured device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ * - 0 on success.
+ * - EAGAIN if can't close as device is busy
+ */
+typedef int (*compressdev_close_t)(struct rte_compressdev *dev);
+
+
+/**
+ * Function used to get specific information of a device.
+ *
+ * @param dev
+ *   Compress device
+ */
+typedef void (*compressdev_info_get_t)(struct rte_compressdev *dev,
+				struct rte_compressdev_info *dev_info);
+
+/** comp device operations function pointer table */
+struct rte_compressdev_ops {
+	compressdev_configure_t dev_configure;	/**< Configure device. */
+	compressdev_start_t dev_start;		/**< Start device. */
+	compressdev_stop_t dev_stop;		/**< Stop device. */
+	compressdev_close_t dev_close;		/**< Close device. */
+
+	compressdev_info_get_t dev_infos_get;	/**< Get device info. */
+};
+
+/**
+ * @internal
+ *
+ * Function for internal use by dummy drivers primarily, e.g. ring-based
+ * driver.
+ * Allocates a new compressdev slot for an comp device and returns the pointer
+ * to that slot for the driver to use.
+ *
+ * @param name
+ *   Unique identifier name for each device
+ * @param socket_id
+ *   Socket to allocate resources on
+ * @return
+ *   - Slot in the rte_dev_devices array for a new device;
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_allocate(const char *name, int socket_id);
+
+/**
+ * @internal
+ *
+ * Function for internal use by dummy drivers primarily, e.g. ring-based
+ * driver.
+ * Release the specified compressdev device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *   - 0 on success, negative on error
+ */
+int __rte_experimental
+rte_compressdev_pmd_release_device(struct rte_compressdev *dev);
+
+
+/**
+ * @internal
+ *
+ * PMD assist function to parse initialisation arguments for comp driver
+ * when creating a new comp PMD device instance.
+ *
+ * PMD driver should set default values for that PMD before calling function,
+ * these default values will be over-written with successfully parsed values
+ * from args string.
+ *
+ * @param params
+ *   Parsed PMD initialisation parameters
+ * @param args
+ *   Input argument string to parse
+ * @return
+ *  - 0 on success
+ *  - errno on failure
+ */
+int __rte_experimental
+rte_compressdev_pmd_parse_input_args(
+		struct rte_compressdev_pmd_init_params *params,
+		const char *args);
+
+/**
+ * @internal
+ *
+ * PMD assist function to provide boiler plate code for comp driver to create
+ * and allocate resources for a new comp PMD device instance.
+ *
+ * @param name
+ *   Compress device name
+ * @param device
+ *   Base device instance
+ * @param params
+ *   PMD initialisation parameters
+ * @return
+ *  - comp device instance on success
+ *  - NULL on creation failure
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_create(const char *name,
+		struct rte_device *device,
+		struct rte_compressdev_pmd_init_params *params);
+
+/**
+ * @internal
+ *
+ * PMD assist function to provide boiler plate code for comp driver to
+ * destroy and free resources associated with a comp PMD device instance.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *  - 0 on success
+ *  - errno on failure
+ */
+int __rte_experimental
+rte_compressdev_pmd_destroy(struct rte_compressdev *dev);
+
+
+/**
+ * @internal
+ * Allocate compressdev driver.
+ *
+ * @param comp_drv
+ *   Compressdev driver
+ * @param drv
+ *   Rte_driver
+ * @return
+ *  The driver type identifier
+ */
+uint8_t __rte_experimental
+rte_compressdev_allocate_driver(struct compressdev_driver *comp_drv,
+		const struct rte_driver *drv);
+
+
+#define RTE_PMD_REGISTER_COMPRESSDEV_DRIVER(comp_drv, drv, driver_id)\
+RTE_INIT(init_ ##driver_id);\
+static void init_ ##driver_id(void)\
+{\
+	driver_id = rte_compressdev_allocate_driver(&comp_drv, &(drv).driver);\
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_COMPRESSDEV_PMD_H_ */
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
new file mode 100644
index 000000000..11bdc1317
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -0,0 +1,31 @@
+EXPERIMENTAL {
+        global:
+
+	rte_compressdevs;
+	rte_compressdev_allocate_driver;
+	rte_compressdev_callback_register;
+	rte_compressdev_callback_unregister;
+	rte_compressdev_close;
+	rte_compressdev_configure;
+	rte_compressdev_count;
+	rte_compressdev_device_count_by_driver;
+	rte_compressdev_devices_get;
+	rte_compressdev_driver_id_get;
+	rte_compressdev_driver_name_get;
+	rte_compressdev_info_get;
+	rte_compressdev_is_valid_dev;
+	rte_compressdev_name_get;
+	rte_compressdev_pmd_allocate;
+	rte_compressdev_pmd_callback_process;
+	rte_compressdev_pmd_create;
+	rte_compressdev_pmd_destroy;
+	rte_compressdev_pmd_get_dev;
+	rte_compressdev_pmd_get_named_dev;
+	rte_compressdev_pmd_parse_input_args;
+	rte_compressdev_pmd_release_device;
+	rte_compressdev_socket_id;
+	rte_compressdev_start;
+	rte_compressdev_stop;
+
+        local: *;
+};
diff --git a/lib/meson.build b/lib/meson.build
index ef6159170..ae718aa3f 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -14,7 +14,7 @@ libraries = [ 'compat', # just a header, used for versioning
 	'hash',    # efd depends on this
 	'kvargs',  # cryptodev depends on this
 	'acl', 'bbdev', 'bitratestats', 'cfgfile',
-	'cmdline', 'cryptodev',
+	'cmdline', 'compressdev', 'cryptodev',
 	'distributor', 'efd', 'eventdev',
 	'gro', 'gso', 'ip_frag', 'jobstats',
 	'kni', 'latencystats', 'lpm', 'member',
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index afd0aff8c..db89cbc16 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -94,6 +94,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF)           += -lrte_mbuf
 _LDLIBS-$(CONFIG_RTE_LIBRTE_NET)            += -lrte_net
 _LDLIBS-$(CONFIG_RTE_LIBRTE_ETHER)          += -lrte_ethdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_BBDEV)          += -lrte_bbdev
+_LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV)    += -lrte_compressdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_CRYPTODEV)      += -lrte_cryptodev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY)       += -lrte_security
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV)       += -lrte_eventdev
-- 
2.14.3

[dpdk-dev] [PATCH v3 02/13] compressdev: add queue pair management

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Add functions to manage device queue pairs.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 135 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           |  50 ++++++++
 lib/librte_compressdev/rte_compressdev_pmd.h       |  47 +++++++
 lib/librte_compressdev/rte_compressdev_version.map |   2 +
 4 files changed, 234 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index d635953b2..c90e4beaf 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -266,10 +266,100 @@ rte_compressdev_pmd_release_device(struct rte_compressdev *compressdev)
 	return 0;
 }
 
+uint16_t __rte_experimental
+rte_compressdev_queue_pair_count(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	dev = &rte_comp_devices[dev_id];
+	return dev->data->nb_queue_pairs;
+}
+
+static int
+rte_compressdev_queue_pairs_config(struct rte_compressdev *dev,
+		uint16_t nb_qpairs, int socket_id)
+{
+	struct rte_compressdev_info dev_info;
+	void **qp;
+	unsigned int i;
+
+	if ((dev == NULL) || (nb_qpairs < 1)) {
+		COMPRESSDEV_LOG(ERR, "invalid param: dev %p, nb_queues %u",
+							dev, nb_qpairs);
+		return -EINVAL;
+	}
+
+	COMPRESSDEV_LOG(DEBUG, "Setup %d queues pairs on device %u",
+			nb_qpairs, dev->data->dev_id);
+
+	memset(&dev_info, 0, sizeof(struct rte_compressdev_info));
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_infos_get, -ENOTSUP);
+	(*dev->dev_ops->dev_infos_get)(dev, &dev_info);
+
+	if ((dev_info.max_nb_queue_pairs != 0) &&
+			(nb_qpairs > dev_info.max_nb_queue_pairs)) {
+		COMPRESSDEV_LOG(ERR, "Invalid num queue_pairs (%u) for dev %u",
+				nb_qpairs, dev->data->dev_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->queue_pairs == NULL) { /* first time configuration */
+		dev->data->queue_pairs = rte_zmalloc_socket(
+				"compressdev->queue_pairs",
+				sizeof(dev->data->queue_pairs[0]) * nb_qpairs,
+				RTE_CACHE_LINE_SIZE, socket_id);
+
+		if (dev->data->queue_pairs == NULL) {
+			dev->data->nb_queue_pairs = 0;
+			COMPRESSDEV_LOG(ERR,
+			"failed to get memory for qp meta data, nb_queues %u",
+							nb_qpairs);
+			return -(ENOMEM);
+		}
+	} else { /* re-configure */
+		int ret;
+		uint16_t old_nb_queues = dev->data->nb_queue_pairs;
+
+		qp = dev->data->queue_pairs;
+
+		RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->queue_pair_release,
+				-ENOTSUP);
+
+		for (i = nb_qpairs; i < old_nb_queues; i++) {
+			ret = (*dev->dev_ops->queue_pair_release)(dev, i);
+			if (ret < 0)
+				return ret;
+		}
+
+		qp = rte_realloc(qp, sizeof(qp[0]) * nb_qpairs,
+				RTE_CACHE_LINE_SIZE);
+		if (qp == NULL) {
+			COMPRESSDEV_LOG(ERR,
+			"failed to realloc qp meta data, nb_queues %u",
+						nb_qpairs);
+			return -(ENOMEM);
+		}
+
+		if (nb_qpairs > old_nb_queues) {
+			uint16_t new_qs = nb_qpairs - old_nb_queues;
+
+			memset(qp + old_nb_queues, 0,
+				sizeof(qp[0]) * new_qs);
+		}
+
+		dev->data->queue_pairs = qp;
+
+	}
+	dev->data->nb_queue_pairs = nb_qpairs;
+	return 0;
+}
+
 int __rte_experimental
 rte_compressdev_configure(uint8_t dev_id, struct rte_compressdev_config *config)
 {
 	struct rte_compressdev *dev;
+	int diag;
 
 	if (!rte_compressdev_is_valid_dev(dev_id)) {
 		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
@@ -286,6 +376,16 @@ rte_compressdev_configure(uint8_t dev_id, struct rte_compressdev_config *config)
 
 	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_configure, -ENOTSUP);
 
+	/* Setup new number of queue pairs and reconfigure device. */
+	diag = rte_compressdev_queue_pairs_config(dev, config->nb_queue_pairs,
+			config->socket_id);
+	if (diag != 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev%d rte_comp_dev_queue_pairs_config = %d",
+				dev_id, diag);
+		return diag;
+	}
+
 	return (*dev->dev_ops->dev_configure)(dev, config);
 }
 
@@ -375,6 +475,41 @@ rte_compressdev_close(uint8_t dev_id)
 	return 0;
 }
 
+int __rte_experimental
+rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
+		uint32_t max_inflight_ops, int socket_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+	if (queue_pair_id >= dev->data->nb_queue_pairs) {
+		COMPRESSDEV_LOG(ERR, "Invalid queue_pair_id=%d", queue_pair_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->dev_started) {
+		COMPRESSDEV_LOG(ERR,
+		    "device %d must be stopped to allow configuration", dev_id);
+		return -EBUSY;
+	}
+
+	if (max_inflight_ops == 0) {
+		COMPRESSDEV_LOG(ERR,
+			"Invalid maximum number of inflight operations");
+		return -EINVAL;
+	}
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->queue_pair_setup, -ENOTSUP);
+
+	return (*dev->dev_ops->queue_pair_setup)(dev, queue_pair_id,
+			max_inflight_ops, socket_id);
+}
+
 void __rte_experimental
 rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index c5de61378..883a5c273 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -38,6 +38,10 @@ extern int compressdev_logtype;
 struct rte_compressdev_info {
 	const char *driver_name;		/**< Driver name. */
 	uint8_t driver_id;			/**< Driver identifier */
+	uint16_t max_nb_queue_pairs;
+	/**< Maximum number of queues pairs supported by device.
+	 * (If 0, there is no limit in maximum number of queue pairs)
+	 */
 };
 
 /**
@@ -118,6 +122,9 @@ rte_compressdev_socket_id(uint8_t dev_id);
 /** Compress device configuration structure */
 struct rte_compressdev_config {
 	int socket_id;
+	/**< Socket on which to allocate resources */
+	uint16_t nb_queue_pairs;
+	/**< Total number of queue pairs to configure on a device */
 };
 
 /**
@@ -179,6 +186,44 @@ rte_compressdev_stop(uint8_t dev_id);
 int __rte_experimental
 rte_compressdev_close(uint8_t dev_id);
 
+/**
+ * Allocate and set up a receive queue pair for a device.
+ * This should only be called when the device is stopped.
+ *
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param queue_pair_id
+ *   The index of the queue pairs to set up. The
+ *   value must be in the range [0, nb_queue_pair - 1]
+ *   previously supplied to rte_compressdev_configure()
+ * @param max_inflight_ops
+ *   Max number of ops which the qp will have to
+ *   accommodate simultaneously
+ * @param socket_id
+ *   The *socket_id* argument is the socket identifier
+ *   in case of NUMA. The value can be *SOCKET_ID_ANY*
+ *   if there is no NUMA constraint for the DMA memory
+ *   allocated for the receive queue pair
+ * @return
+ *   - 0: Success, queue pair correctly set up.
+ *   - <0: Queue pair configuration failed
+ */
+int __rte_experimental
+rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
+		uint32_t max_inflight_ops, int socket_id);
+
+/**
+ * Get the number of queue pairs on a specific comp device
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - The number of configured queue pairs.
+ */
+uint16_t __rte_experimental
+rte_compressdev_queue_pair_count(uint8_t dev_id);
+
 /**
  * Retrieve the contextual information of a device.
  *
@@ -232,6 +277,11 @@ struct rte_compressdev_data {
 	uint8_t dev_started : 1;
 	/**< Device state: STARTED(1)/STOPPED(0) */
 
+	void **queue_pairs;
+	/**< Array of pointers to queue pairs. */
+	uint16_t nb_queue_pairs;
+	/**< Number of device queue pairs */
+
 	void *dev_private;
 	/**< PMD-specific private data */
 } __rte_cache_aligned;
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
index 1b49b48a6..b71391659 100644
--- a/lib/librte_compressdev/rte_compressdev_pmd.h
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -149,6 +149,48 @@ typedef int (*compressdev_close_t)(struct rte_compressdev *dev);
 typedef void (*compressdev_info_get_t)(struct rte_compressdev *dev,
 				struct rte_compressdev_info *dev_info);
 
+/**
+ * Setup a queue pair for a device.
+ *
+ * @param dev
+ *   Compress device
+ * @param qp_id
+ *   Queue pair identifier
+ * @param max_inflight_ops
+ *   Max inflight ops which qp must accommodate
+ * @param socket_id
+ *   Socket identifier
+ * @return
+ *   Returns 0 on success.
+ */
+typedef int (*compressdev_queue_pair_setup_t)(struct rte_compressdev *dev,
+		uint16_t qp_id,	uint32_t max_inflight_ops, int socket_id);
+
+/**
+ * Release memory resources allocated by given queue pair.
+ *
+ * @param dev
+ *   Compress device
+ * @param qp_id
+ *   Queue pair identifier
+ * @return
+ * - 0 on success.
+ * - EAGAIN if can't close as device is busy
+ */
+typedef int (*compressdev_queue_pair_release_t)(struct rte_compressdev *dev,
+		uint16_t qp_id);
+
+/**
+ * Get number of available queue pairs of a device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *   Returns number of queue pairs on success.
+ */
+typedef uint32_t (*compressdev_queue_pair_count_t)(struct rte_compressdev *dev);
+
+
 /** comp device operations function pointer table */
 struct rte_compressdev_ops {
 	compressdev_configure_t dev_configure;	/**< Configure device. */
@@ -157,6 +199,11 @@ struct rte_compressdev_ops {
 	compressdev_close_t dev_close;		/**< Close device. */
 
 	compressdev_info_get_t dev_infos_get;	/**< Get device info. */
+
+	compressdev_queue_pair_setup_t queue_pair_setup;
+	/**< Set up a device queue pair. */
+	compressdev_queue_pair_release_t queue_pair_release;
+	/**< Release a queue pair. */
 };
 
 /**
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 11bdc1317..78ac8d904 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -23,6 +23,8 @@ EXPERIMENTAL {
 	rte_compressdev_pmd_get_named_dev;
 	rte_compressdev_pmd_parse_input_args;
 	rte_compressdev_pmd_release_device;
+	rte_compressdev_queue_pair_count;
+	rte_compressdev_queue_pair_setup;
 	rte_compressdev_socket_id;
 	rte_compressdev_start;
 	rte_compressdev_stop;
-- 
2.14.3

[dpdk-dev] [PATCH v3 03/13] compressdev: add compression specific data

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added structures and enums specific to compression,
including the compression operation structure and the
different supported algorithms, checksums and compression
levels.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/Makefile    |   1 +
 lib/librte_compressdev/meson.build |   3 +-
 lib/librte_compressdev/rte_comp.h  | 312 +++++++++++++++++++++++++++++++++++++
 3 files changed, 315 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_compressdev/rte_comp.h

diff --git a/lib/librte_compressdev/Makefile b/lib/librte_compressdev/Makefile
index 5f67ab817..6f1546afd 100644
--- a/lib/librte_compressdev/Makefile
+++ b/lib/librte_compressdev/Makefile
@@ -19,6 +19,7 @@ LDLIBS += -lrte_eal -lrte_mempool -lrte_kvargs
 SRCS-y += rte_compressdev.c rte_compressdev_pmd.c
 
 # export include files
+SYMLINK-y-include += rte_comp.h
 SYMLINK-y-include += rte_compressdev.h
 SYMLINK-y-include += rte_compressdev_pmd.h
 
diff --git a/lib/librte_compressdev/meson.build b/lib/librte_compressdev/meson.build
index fc5eaf009..a72d4cea5 100644
--- a/lib/librte_compressdev/meson.build
+++ b/lib/librte_compressdev/meson.build
@@ -4,5 +4,6 @@
 allow_experimental_apis = true
 sources = files('rte_compressdev.c', 'rte_compressdev_pmd.c')
 headers = files('rte_compressdev.h',
-	'rte_compressdev_pmd.h')
+	'rte_compressdev_pmd.h',
+	'rte_comp.h')
 deps += ['kvargs', 'mbuf']
diff --git a/lib/librte_compressdev/rte_comp.h b/lib/librte_compressdev/rte_comp.h
new file mode 100644
index 000000000..cf0f3c999
--- /dev/null
+++ b/lib/librte_compressdev/rte_comp.h
@@ -0,0 +1,312 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMP_H_
+#define _RTE_COMP_H_
+
+/**
+ * @file rte_comp.h
+ *
+ * RTE definitions for Data Compression Service
+ *
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <string.h>
+#include <rte_mempool.h>
+
+/** Status of comp operation */
+enum rte_comp_op_status {
+	RTE_COMP_OP_STATUS_SUCCESS = 0,
+	/**< Operation completed successfully */
+	RTE_COMP_OP_STATUS_NOT_PROCESSED,
+	/**< Operation has not yet been processed by the device */
+	RTE_COMP_OP_STATUS_INVALID_ARGS,
+	/**< Operation failed due to invalid arguments in request */
+	RTE_COMP_OP_STATUS_ERROR,
+	/**< Error handling operation */
+	RTE_COMP_OP_STATUS_INVALID_STATE,
+	/**< Operation is invoked in invalid state */
+	RTE_COMP_OP_STATUS_OUT_OF_SPACE,
+	/**< Output buffer ran out of space before operation completed */
+};
+
+/** Compression Algorithms */
+enum rte_comp_algorithm {
+	RTE_COMP_ALGO_UNSPECIFIED = 0,
+	/** No Compression algorithm */
+	RTE_COMP_ALGO_NULL,
+	/**< No compression.
+	 * Pass-through, data is copied unchanged from source buffer to
+	 * destination buffer.
+	 */
+	RTE_COMP_ALGO_DEFLATE,
+	/**< DEFLATE compression algorithm
+	 * https://tools.ietf.org/html/rfc1951
+	 */
+	RTE_COMP_ALGO_LZS,
+	/**< LZS compression algorithm
+	 * https://tools.ietf.org/html/rfc2395
+	 */
+	RTE_COMP_ALGO_LIST_END
+};
+
+/**< Compression Level.
+ * The number is interpreted by each PMD differently. However, lower numbers
+ * give fastest compression, at the expense of compression ratio while
+ * higher numbers may give better compression ratios but are likely slower.
+ */
+#define	RTE_COMP_LEVEL_PMD_DEFAULT	(-1)
+/** Use PMD Default */
+#define	RTE_COMP_LEVEL_NONE		(0)
+/** Output uncompressed blocks if supported by the specified algorithm */
+#define RTE_COMP_LEVEL_MIN		(1)
+/** Use minimum compression level supported by the PMD */
+#define RTE_COMP_LEVEL_MAX		(9)
+/** Use maximum compression level supported by the PMD */
+
+/** Compression checksum types */
+enum rte_comp_checksum_type {
+	RTE_COMP_CHECKSUM_NONE,
+	/**< No checksum generated */
+	RTE_COMP_CHECKSUM_CRC32,
+	/**< Generates a CRC32 checksum, as used by gzip */
+	RTE_COMP_CHECKSUM_ADLER32,
+	/**< Generates an Adler-32 checksum, as used by zlib */
+	RTE_COMP_CHECKSUM_CRC32_ADLER32,
+	/**< Generates both Adler-32 and CRC32 checksums, concatenated.
+	 * CRC32 is in the lower 32bits, Adler-32 in the upper 32 bits.
+	 */
+};
+
+
+/** Compression Huffman Type - used by DEFLATE algorithm */
+enum rte_comp_huffman {
+	RTE_COMP_HUFFMAN_DEFAULT,
+	/**< PMD may choose which Huffman codes to use */
+	RTE_COMP_HUFFMAN_FIXED,
+	/**< Use Fixed Huffman codes */
+	RTE_COMP_HUFFMAN_DYNAMIC,
+	/**< Use Dynamic Huffman codes */
+};
+
+enum rte_comp_flush_flag {
+	RTE_COMP_FLUSH_NONE,
+	/**< Data is not flushed. Output may remain in the compressor and be
+	 * processed during a following op. It may not be possible to decompress
+	 * output until a later op with some other flush flag has been sent.
+	 */
+	RTE_COMP_FLUSH_SYNC,
+	/**< All data should be flushed to output buffer. Output data can be
+	 * decompressed. However state and history is not cleared, so future
+	 * operations may use history from this operation.
+	 */
+	RTE_COMP_FLUSH_FULL,
+	/**< All data should be flushed to output buffer. Output data can be
+	 * decompressed. State and history data is cleared, so future
+	 * ops will be independent of ops processed before this.
+	 */
+	RTE_COMP_FLUSH_FINAL
+	/**< Same as RTE_COMP_FLUSH_FULL but if op.algo is RTE_COMP_ALGO_DEFLATE
+	 * then bfinal bit is set in the last block.
+	 */
+};
+
+/** Compression transform types */
+enum rte_comp_xform_type {
+	RTE_COMP_COMPRESS,
+	/**< Compression service - compress */
+	RTE_COMP_DECOMPRESS,
+	/**< Compression service - decompress */
+};
+
+enum rte_comp_op_type {
+	RTE_COMP_OP_STATELESS,
+	/**< All data to be processed is submitted in the op, no state or
+	 * history from previous ops is used and none will be stored for future
+	 * ops. Flush flag must be set to either FLUSH_FULL or FLUSH_FINAL.
+	 */
+	RTE_COMP_OP_STATEFUL
+	/**< There may be more data to be processed after this op, it's part of
+	 * a stream of data. State and history from previous ops can be used
+	 * and resulting state and history can be stored for future ops,
+	 * depending on flush flag.
+	 */
+};
+
+
+/** Parameters specific to the deflate algorithm */
+struct rte_comp_deflate_params {
+	enum rte_comp_huffman huffman;
+	/**< Compression huffman encoding type */
+};
+
+/** Setup Data for compression */
+struct rte_comp_compress_xform {
+	enum rte_comp_algorithm algo;
+	/**< Algorithm to use for compress operation */
+	union {
+		struct rte_comp_deflate_params deflate;
+		/**< Parameters specific to the deflate algorithm */
+	}; /**< Algorithm specific parameters */
+	int level;
+	/**< Compression level */
+	uint8_t window_size;
+	/**< Base two log value of sliding window to be used. If window size
+	 * can't be supported by the PMD then it may fall back to a smaller
+	 * size. This is likely to result in a worse compression ratio.
+	 */
+	enum rte_comp_checksum_type chksum;
+	/**< Type of checksum to generate on the uncompressed data */
+};
+
+/**
+ * Setup Data for decompression.
+ */
+struct rte_comp_decompress_xform {
+	enum rte_comp_algorithm algo;
+	/**< Algorithm to use for decompression */
+	enum rte_comp_checksum_type chksum;
+	/**< Type of checksum to generate on the decompressed data */
+	uint8_t window_size;
+	/**< Base two log value of sliding window which was used to generate
+	 * compressed data. If window size can't be supported by the PMD then
+	 * setup of stream or private_xform should fail.
+	 */
+};
+
+/**
+ * Compression transform structure.
+ *
+ * This is used to specify the compression transforms required.
+ * Each transform structure can hold a single transform, the type field is
+ * used to specify which transform is contained within the union.
+ * There are no chain cases currently supported, just single xforms of
+ *  - compress-only
+ *  - decompress-only
+ *
+ */
+struct rte_comp_xform {
+	struct rte_comp_xform *next;
+	/**< next xform in chain */
+	enum rte_comp_xform_type type;
+	/**< xform type */
+	union {
+		struct rte_comp_compress_xform compress;
+		/**< xform for compress operation */
+		struct rte_comp_decompress_xform decompress;
+		/**< decompress xform */
+	};
+};
+
+/**
+ * Compression Operation.
+ *
+ * This structure contains data relating to performing a compression
+ * operation on the referenced mbuf data buffers.
+ *
+ * Comp operations are enqueued and dequeued in comp PMDs using the
+ * rte_compressdev_enqueue_burst() / rte_compressdev_dequeue_burst() APIs
+ */
+struct rte_comp_op {
+	enum rte_comp_op_type op_type;
+	union {
+		void *private_xform;
+		/**< Stateless private PMD data derived from an rte_comp_xform.
+		 * A handle returned by rte_compressdev_private_xform_create()
+		 * must be attached to operations of op_type RTE_COMP_STATELESS.
+		 */
+		void *stream;
+		/**< Private PMD data derived initially from an rte_comp_xform,
+		 * which holds state and history data and evolves as operations
+		 * are processed. rte_compressdev_stream_create() must be called
+		 * on a device for all STATEFUL data streams and the resulting
+		 * stream attached to the one or more operations associated
+		 * with the data stream.
+		 * All operations in a stream must be sent to the same device.
+		 */
+	};
+
+	struct rte_mempool *mempool;
+	/**< Pool from which operation is allocated */
+	rte_iova_t iova_addr;
+	/**< IOVA address of this operation */
+	struct rte_mbuf *m_src;
+	/**< source mbuf
+	 * The total size of the input buffer(s) can be retrieved using
+	 * rte_pktmbuf_data_len(m_src)
+	 */
+	struct rte_mbuf *m_dst;
+	/**< destination mbuf
+	 * The total size of the output buffer(s) can be retrieved using
+	 * rte_pktmbuf_data_len(m_dst)
+	 */
+
+	struct {
+		uint32_t offset;
+		/**< Starting point for compression or decompression,
+		 * specified as number of bytes from start of packet in
+		 * source buffer.
+		 * Starting point for checksum generation in compress direction.
+		 */
+		uint32_t length;
+		/**< The length, in bytes, of the data in source buffer
+		 * to be compressed or decompressed.
+		 * Also the length of the data over which the checksum
+		 * should be generated in compress direction
+		 */
+	} src;
+	struct {
+		uint32_t offset;
+		/**< Starting point for writing output data, specified as
+		 * number of bytes from start of packet in dest
+		 * buffer. Starting point for checksum generation in
+		 * decompress direction.
+		 */
+	} dst;
+	enum rte_comp_flush_flag flush_flag;
+	/**< Defines flush characteristics for the output data.
+	 * Only applicable in compress direction
+	 */
+	uint64_t input_chksum;
+	/**< An input checksum can be provided to generate a
+	 * cumulative checksum across sequential blocks in a STATELESS stream.
+	 * Checksum type is as specified in xform chksum_type
+	 */
+	uint64_t output_chksum;
+	/**< If a checksum is generated it will be written in here.
+	 * Checksum type is as specified in xform chksum_type.
+	 */
+	uint32_t consumed;
+	/**< The number of bytes from the source buffer
+	 * which were compressed/decompressed.
+	 */
+	uint32_t produced;
+	/**< The number of bytes written to the destination buffer
+	 * which were compressed/decompressed.
+	 */
+	uint64_t debug_status;
+	/**<
+	 * Status of the operation is returned in the status param.
+	 * This field allows the PMD to pass back extra
+	 * pmd-specific debug information. Value is not defined on the API.
+	 */
+	uint8_t status;
+	/**<
+	 * Operation status - use values from enum rte_comp_status.
+	 * This is reset to
+	 * RTE_COMP_OP_STATUS_NOT_PROCESSED on allocation from mempool and
+	 * will be set to RTE_COMP_OP_STATUS_SUCCESS after operation
+	 * is successfully processed by a PMD
+	 */
+} __rte_cache_aligned;
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_COMP_H_ */
-- 
2.14.3

[dpdk-dev] [PATCH v3 04/13] compressdev: add enqueue/dequeue functions

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.h | 121 +++++++++++++++++++++++++++++++
 1 file changed, 121 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 883a5c273..06f9ee135 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -241,8 +241,23 @@ rte_compressdev_queue_pair_count(uint8_t dev_id);
 void __rte_experimental
 rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info);
 
+
+typedef uint16_t (*compress_dequeue_pkt_burst_t)(void *qp,
+		struct rte_comp_op **ops, uint16_t nb_ops);
+/**< Dequeue processed packets from queue pair of a device. */
+
+typedef uint16_t (*compress_enqueue_pkt_burst_t)(void *qp,
+		struct rte_comp_op **ops, uint16_t nb_ops);
+/**< Enqueue packets for processing on queue pair of a device. */
+
+
 /** The data structure associated with each comp device. */
 struct rte_compressdev {
+	compress_dequeue_pkt_burst_t dequeue_burst;
+	/**< Pointer to PMD receive function */
+	compress_enqueue_pkt_burst_t enqueue_burst;
+	/**< Pointer to PMD transmit function */
+
 	struct rte_compressdev_data *data;
 	/**< Pointer to device data */
 	struct rte_compressdev_ops *dev_ops;
@@ -288,6 +303,112 @@ struct rte_compressdev_data {
 
 struct rte_compressdev *rte_compressdevs;
 
+/**
+ *
+ * Dequeue a burst of processed compression operations from a queue on the comp
+ * device. The dequeued operation are stored in *rte_comp_op* structures
+ * whose pointers are supplied in the *ops* array.
+ *
+ * The rte_compressdev_dequeue_burst() function returns the number of ops
+ * actually dequeued, which is the number of *rte_comp_op* data structures
+ * effectively supplied into the *ops* array.
+ *
+ * A return value equal to *nb_ops* indicates that the queue contained
+ * at least *nb_ops* operations, and this is likely to signify that other
+ * processed operations remain in the devices output queue. Applications
+ * implementing a "retrieve as many processed operations as possible" policy
+ * can check this specific case and keep invoking the
+ * rte_compressdev_dequeue_burst() function until a value less than
+ * *nb_ops* is returned.
+ *
+ * The rte_compressdev_dequeue_burst() function does not provide any error
+ * notification to avoid the corresponding overhead.
+ *
+ * Note: operation ordering is not maintained within the queue pair.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param qp_id
+ *   The index of the queue pair from which to retrieve
+ *   processed operations. The value must be in the range
+ *   [0, nb_queue_pair - 1] previously supplied to
+ *   rte_compressdev_configure()
+ * @param ops
+ *   The address of an array of pointers to
+ *   *rte_comp_op* structures that must be
+ *   large enough to store *nb_ops* pointers in it
+ * @param nb_ops
+ *   The maximum number of operations to dequeue
+ * @return
+ *   - The number of operations actually dequeued, which is the number
+ *   of pointers to *rte_comp_op* structures effectively supplied to the
+ *   *ops* array.
+ */
+static inline uint16_t
+rte_compressdev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	struct rte_compressdev *dev = &rte_compressdevs[dev_id];
+
+	nb_ops = (*dev->dequeue_burst)
+			(dev->data->queue_pairs[qp_id], ops, nb_ops);
+
+	return nb_ops;
+}
+
+/**
+ * Enqueue a burst of operations for processing on a compression device.
+ *
+ * The rte_compressdev_enqueue_burst() function is invoked to place
+ * comp operations on the queue *qp_id* of the device designated by
+ * its *dev_id*.
+ *
+ * The *nb_ops* parameter is the number of operations to process which are
+ * supplied in the *ops* array of *rte_comp_op* structures.
+ *
+ * The rte_compressdev_enqueue_burst() function returns the number of
+ * operations it actually enqueued for processing. A return value equal to
+ * *nb_ops* means that all packets have been enqueued.
+ *
+ * @note All compression operations are Out-of-place (OOP) operations,
+ * as the size of the output data is different to the size of the input data.
+ *
+ * @note The flush flag only applies to operations which return SUCCESS.
+ * In OUT_OF_SPACE case whether STATEFUL or STATELESS, data in dest buffer
+ * is as if flush flag was FLUSH_NONE.
+ * @note flush flag only applies in compression direction. It has no meaning
+ * for decompression.
+ * @note: operation ordering is not maintained within the queue pair.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param qp_id
+ *   The index of the queue pair on which operations
+ *   are to be enqueued for processing. The value
+ *   must be in the range [0, nb_queue_pairs - 1]
+ *   previously supplied to *rte_compressdev_configure*
+ * @param ops
+ *   The address of an array of *nb_ops* pointers
+ *   to *rte_comp_op* structures which contain
+ *   the operations to be processed
+ * @param nb_ops
+ *   The number of operations to process
+ * @return
+ *   The number of operations actually enqueued on the device. The return
+ *   value can be less than the value of the *nb_ops* parameter when the
+ *   comp devices queue is full or if invalid parameters are specified in
+ *   a *rte_comp_op*.
+ */
+static inline uint16_t
+rte_compressdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	struct rte_compressdev *dev = &rte_compressdevs[dev_id];
+
+	return (*dev->enqueue_burst)(
+			dev->data->queue_pairs[qp_id], ops, nb_ops);
+}
+
 /**
  * Provide driver identifier.
  *
-- 
2.14.3

[dpdk-dev] [PATCH v3 05/13] compressdev: add operation management

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added functions to allocate and free compression operations.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_comp.h                  | 195 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.c           |  72 ++++++++
 lib/librte_compressdev/rte_compressdev_version.map |   1 +
 3 files changed, 268 insertions(+)

diff --git a/lib/librte_compressdev/rte_comp.h b/lib/librte_compressdev/rte_comp.h
index cf0f3c999..4cf84c5db 100644
--- a/lib/librte_compressdev/rte_comp.h
+++ b/lib/librte_compressdev/rte_comp.h
@@ -305,6 +305,201 @@ struct rte_comp_op {
 	 */
 } __rte_cache_aligned;
 
+
+/**
+ * Reset the fields of an operation to their default values.
+ *
+ * @note The private data associated with the operation is not zeroed.
+ *
+ * @param op
+ *   The operation to be reset
+ */
+static inline void
+__rte_comp_op_reset(struct rte_comp_op *op)
+{
+	struct rte_mempool *tmp_mp = op->mempool;
+	rte_iova_t tmp_iova_addr = op->iova_addr;
+
+	memset(op, 0, sizeof(struct rte_comp_op));
+	op->status = RTE_COMP_OP_STATUS_NOT_PROCESSED;
+	op->iova_addr = tmp_iova_addr;
+	op->mempool = tmp_mp;
+}
+
+/**
+ * Private data structure belonging to an operation pool.
+ */
+struct rte_comp_op_pool_private {
+	uint16_t user_size;
+	/**< Size of private user data with each operation. */
+};
+
+
+/**
+ * Returns the size of private user data allocated with each object in
+ * the mempool
+ *
+ * @param mempool
+ *   Mempool for operations
+ * @return
+ *   user data size
+ */
+static inline uint16_t
+__rte_comp_op_get_user_data_size(struct rte_mempool *mempool)
+{
+	struct rte_comp_op_pool_private *priv =
+	    (struct rte_comp_op_pool_private *)rte_mempool_get_priv(mempool);
+
+	return priv->user_size;
+}
+
+
+/**
+ * Creates an operation pool
+ *
+ * @param name
+ *   Compress pool name
+ * @param nb_elts
+ *   Number of elements in pool
+ * @param cache_size
+ *   Number of elements to cache on lcore, see
+ *   *rte_mempool_create* for further details about cache size
+ * @param user_size
+ *   Size of private data to allocate for user with each operation
+ * @param socket_id
+ *   Socket to identifier allocate memory on
+ * @return
+ *  - On success pointer to mempool
+ *  - On failure NULL
+ */
+struct rte_mempool *
+rte_comp_op_pool_create(const char *name,
+		unsigned int nb_elts, unsigned int cache_size,
+		uint16_t user_size, int socket_id);
+
+/**
+ * Bulk allocate raw element from mempool and return as comp operations
+ *
+ * @param mempool
+ *   Compress operation mempool
+ * @param ops
+ *   Array to place allocated operations
+ * @param nb_ops
+ *   Number of operations to allocate
+ * @return
+ * - On success returns  number of ops allocated
+ */
+static inline int
+__rte_comp_op_raw_bulk_alloc(struct rte_mempool *mempool,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	if (rte_mempool_get_bulk(mempool, (void **)ops, nb_ops) == 0)
+		return nb_ops;
+
+	return 0;
+}
+
+/**
+ * Allocate an operation from a mempool with default parameters set
+ *
+ * @param mempool
+ *   Compress operation mempool
+ *
+ * @return
+ * - On success returns a valid rte_comp_op structure
+ * - On failure returns NULL
+ */
+static inline struct rte_comp_op *
+rte_comp_op_alloc(struct rte_mempool *mempool)
+{
+	struct rte_comp_op *op = NULL;
+	int retval;
+
+	retval = __rte_comp_op_raw_bulk_alloc(mempool, &op, 1);
+	if (unlikely(retval != 1))
+		return NULL;
+
+	__rte_comp_op_reset(op);
+
+	return op;
+}
+
+
+/**
+ * Bulk allocate operations from a mempool with default parameters set
+ *
+ * @param mempool
+ *   Compress operation mempool
+ * @param ops
+ *   Array to place allocated operations
+ * @param nb_ops
+ *   Number of operations to allocate
+ * @return
+ * - nb_ops if the number of operations requested were allocated.
+ * - 0 if the requested number of ops are not available.
+ *   None are allocated in this case.
+ */
+static inline unsigned
+rte_comp_op_bulk_alloc(struct rte_mempool *mempool,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	int i;
+
+	if (unlikely(__rte_comp_op_raw_bulk_alloc(mempool, ops, nb_ops)
+			!= nb_ops))
+		return 0;
+
+	for (i = 0; i < nb_ops; i++)
+		__rte_comp_op_reset(ops[i]);
+
+	return nb_ops;
+}
+
+
+
+/**
+ * Returns a pointer to the private user data of an operation if
+ * that operation has enough capacity for requested size.
+ *
+ * @param op
+ *   Compress operation
+ * @param size
+ *   Size of space requested in private data
+ * @return
+ * - if sufficient space available returns pointer to start of user data
+ * - if insufficient space returns NULL
+ */
+static inline void *
+__rte_comp_op_get_user_data(struct rte_comp_op *op, uint32_t size)
+{
+	uint32_t user_size;
+
+	if (likely(op->mempool != NULL)) {
+		user_size = __rte_comp_op_get_user_data_size(op->mempool);
+
+		if (likely(user_size >= size))
+			return (void *)(op + 1);
+
+	}
+
+	return NULL;
+}
+
+/**
+ * free operation structure
+ * If operation has been allocate from a rte_mempool, then the operation will
+ * be returned to the mempool.
+ *
+ * @param op
+ *   Compress operation
+ */
+static inline void
+rte_comp_op_free(struct rte_comp_op *op)
+{
+	if (op != NULL && op->mempool != NULL)
+		rte_mempool_put(op->mempool, op);
+}
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index c90e4beaf..0dab92650 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -530,6 +530,78 @@ rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
 	dev_info->driver_name = dev->device->driver->name;
 }
 
+/** Initialise rte_comp_op mempool element */
+static void
+rte_comp_op_init(struct rte_mempool *mempool,
+		__rte_unused void *opaque_arg,
+		void *_op_data,
+		__rte_unused unsigned int i)
+{
+	struct rte_comp_op *op = _op_data;
+
+	memset(_op_data, 0, mempool->elt_size);
+
+	op->status = RTE_COMP_OP_STATUS_NOT_PROCESSED;
+	op->iova_addr = rte_mem_virt2iova(_op_data);
+	op->mempool = mempool;
+}
+
+
+struct rte_mempool * __rte_experimental
+rte_comp_op_pool_create(const char *name,
+		unsigned int nb_elts, unsigned int cache_size,
+		uint16_t user_size, int socket_id)
+{
+	struct rte_comp_op_pool_private *priv;
+
+	unsigned int elt_size = sizeof(struct rte_comp_op) + user_size;
+
+	/* lookup mempool in case already allocated */
+	struct rte_mempool *mp = rte_mempool_lookup(name);
+
+	if (mp != NULL) {
+		priv = (struct rte_comp_op_pool_private *)
+				rte_mempool_get_priv(mp);
+
+		if (mp->elt_size != elt_size ||
+				mp->cache_size < cache_size ||
+				mp->size < nb_elts ||
+				priv->user_size <  user_size) {
+			mp = NULL;
+			COMPRESSDEV_LOG(ERR,
+		"Mempool %s already exists but with incompatible parameters",
+					name);
+			return NULL;
+		}
+		return mp;
+	}
+
+	mp = rte_mempool_create(
+			name,
+			nb_elts,
+			elt_size,
+			cache_size,
+			sizeof(struct rte_comp_op_pool_private),
+			NULL,
+			NULL,
+			rte_comp_op_init,
+			NULL,
+			socket_id,
+			0);
+
+	if (mp == NULL) {
+		COMPRESSDEV_LOG(ERR, "Failed to create mempool %s", name);
+		return NULL;
+	}
+
+	priv = (struct rte_comp_op_pool_private *)
+			rte_mempool_get_priv(mp);
+
+	priv->user_size = user_size;
+
+	return mp;
+}
+
 TAILQ_HEAD(compressdev_driver_list, compressdev_driver);
 
 static struct compressdev_driver_list compressdev_driver_list =
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 78ac8d904..6a3596429 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -28,6 +28,7 @@ EXPERIMENTAL {
 	rte_compressdev_socket_id;
 	rte_compressdev_start;
 	rte_compressdev_stop;
+	rte_comp_op_pool_create;
 
         local: *;
 };
-- 
2.14.3

[dpdk-dev] [PATCH v3 06/13] compressdev: support stateless operations

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added private transform data (priv_xform) in compression
operation, which will contain the private data from each
PMD to support stateless operations.
Also, added functions to create/free this data.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 48 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 48 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 40 ++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  2 +
 4 files changed, 138 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 0dab92650..3843a6bbf 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -530,6 +530,54 @@ rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
 	dev_info->driver_name = dev->device->driver->name;
 }
 
+int __rte_experimental
+rte_compressdev_private_xform_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **priv_xform)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (xform == NULL || priv_xform == NULL || dev == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->private_xform_create, -ENOTSUP);
+	ret = (*dev->dev_ops->private_xform_create)(dev, xform, priv_xform);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to create private_xform: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
+int __rte_experimental
+rte_compressdev_private_xform_free(uint8_t dev_id, void *priv_xform)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (dev == NULL || priv_xform == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->private_xform_free, -ENOTSUP);
+	ret = dev->dev_ops->private_xform_free(dev, priv_xform);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to free private xform: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
 /** Initialise rte_comp_op mempool element */
 static void
 rte_comp_op_init(struct rte_mempool *mempool,
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 06f9ee135..917c0d764 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -125,6 +125,8 @@ struct rte_compressdev_config {
 	/**< Socket on which to allocate resources */
 	uint16_t nb_queue_pairs;
 	/**< Total number of queue pairs to configure on a device */
+	uint16_t max_nb_priv_xforms;
+	/**< Max number of private_xforms which will be created on the device */
 };
 
 /**
@@ -409,6 +411,52 @@ rte_compressdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 			dev->data->queue_pairs[qp_id], ops, nb_ops);
 }
 
+/**
+ * This should alloc a private_xform from the device's mempool and initialise
+ * it. The application should call this API when setting up for stateless
+ * processing on a device. If it returns non-shareable, then the appl cannot
+ * share this handle with multiple in-flight ops and should call this API again
+ * to get a separate handle for every in-flight op.
+ * The handle returned is only valid for use with ops of op_type STATELESS.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param xform
+ *   xform data
+ * @param private_xform
+ *   Pointer to where PMD's private_xform handle should be stored
+ *
+ * @return
+ *  - if successful returns 0
+ *    and valid private_xform handle
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private_xform could not be allocated.
+ */
+int __rte_experimental
+rte_compressdev_private_xform_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **private_xform);
+
+/**
+ * This should clear the private_xform and return it to the device's mempool.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ *
+ * @param private_xform
+ *   PMD's private_xform data
+ *
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -EBUSY if can't free private_xform due to inflight operations
+ */
+int __rte_experimental
+rte_compressdev_private_xform_free(uint8_t dev_id, void *private_xform);
+
 /**
  * Provide driver identifier.
  *
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
index b71391659..a8b868510 100644
--- a/lib/librte_compressdev/rte_compressdev_pmd.h
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -190,6 +190,41 @@ typedef int (*compressdev_queue_pair_release_t)(struct rte_compressdev *dev,
  */
 typedef uint32_t (*compressdev_queue_pair_count_t)(struct rte_compressdev *dev);
 
+/**
+ * Create driver private_xform data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param xform
+ *   xform data
+ * @param private_xform
+ *   ptr where handle of pmd's private_xform data should be stored
+ * @return
+ *  - if successful returns 0
+ *    and valid private_xform handle
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private_xform could not be allocated.
+ */
+typedef int (*compressdev_private_xform_create_t)(struct rte_compressdev *dev,
+		const struct rte_comp_xform *xform, void **private_xform);
+
+/**
+ * Free driver private_xform data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param private_xform
+ *   handle of pmd's private_xform data
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -EBUSY if can't free private_xform due to inflight operations
+ */
+typedef int (*compressdev_private_xform_free_t)(struct rte_compressdev *dev,
+		void *private_xform);
 
 /** comp device operations function pointer table */
 struct rte_compressdev_ops {
@@ -204,6 +239,11 @@ struct rte_compressdev_ops {
 	/**< Set up a device queue pair. */
 	compressdev_queue_pair_release_t queue_pair_release;
 	/**< Release a queue pair. */
+
+	compressdev_private_xform_create_t private_xform_create;
+	/**< Create a comp private_xform and initialise its private data. */
+	compressdev_private_xform_free_t private_xform_free;
+	/**< Free a comp private_xform's data. */
 };
 
 /**
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 6a3596429..6636decc2 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -23,6 +23,8 @@ EXPERIMENTAL {
 	rte_compressdev_pmd_get_named_dev;
 	rte_compressdev_pmd_parse_input_args;
 	rte_compressdev_pmd_release_device;
+	rte_compressdev_private_xform_create;
+	rte_compressdev_private_xform_free;
 	rte_compressdev_queue_pair_count;
 	rte_compressdev_queue_pair_setup;
 	rte_compressdev_socket_id;
-- 
2.14.3

[dpdk-dev] [PATCH v3 07/13] compressdev: support stateful operations

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added stream data (stream) in compression operation,
which will contain the private data from each PMD
to support stateful operations.
Also, added functions to create/free this data.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 49 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 50 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 41 ++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  2 +
 4 files changed, 142 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 3843a6bbf..433973aaa 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -578,6 +578,55 @@ rte_compressdev_private_xform_free(uint8_t dev_id, void *priv_xform)
 	return 0;
 }
 
+int __rte_experimental
+rte_compressdev_stream_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **stream)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (xform == NULL || dev == NULL || stream == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->stream_create, -ENOTSUP);
+	ret = (*dev->dev_ops->stream_create)(dev, xform, stream);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to create stream: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
+
+int __rte_experimental
+rte_compressdev_stream_free(uint8_t dev_id, void *stream)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (dev == NULL || stream == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->stream_free, -ENOTSUP);
+	ret = dev->dev_ops->stream_free(dev, stream);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to free stream: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
 /** Initialise rte_comp_op mempool element */
 static void
 rte_comp_op_init(struct rte_mempool *mempool,
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 917c0d764..0e148dd30 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -127,6 +127,8 @@ struct rte_compressdev_config {
 	/**< Total number of queue pairs to configure on a device */
 	uint16_t max_nb_priv_xforms;
 	/**< Max number of private_xforms which will be created on the device */
+	uint16_t max_nb_streams;
+	/**< Max number of streams which will be created on the device */
 };
 
 /**
@@ -411,6 +413,54 @@ rte_compressdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 			dev->data->queue_pairs[qp_id], ops, nb_ops);
 }
 
+/**
+ * This should alloc a stream from the device's mempool and initialise it.
+ * The application should call this API when setting up for the stateful
+ * processing of a set of data on a device. The API can be called multiple
+ * times to set up a stream for each data set. The handle returned is only for
+ * use with ops of op_type STATEFUL and must be passed to the PMD
+ * with every op in the data stream
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param xform
+ *   xform data
+ * @param stream
+ *   Pointer to where PMD's private stream handle should be stored
+ *
+ * @return
+ *  - 0 if successful and valid stream handle
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private stream could not be allocated.
+ *
+ */
+int __rte_experimental
+rte_compressdev_stream_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **stream);
+
+/**
+ * This should clear the stream and return it to the device's mempool.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ *
+ * @param stream
+ *   PMD's private stream data
+ *
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -EBUSY if can't free stream as there are inflight operations
+ */
+int __rte_experimental
+rte_compressdev_stream_free(uint8_t dev_id, void *stream);
+
 /**
  * This should alloc a private_xform from the device's mempool and initialise
  * it. The application should call this API when setting up for stateless
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
index a8b868510..59bcb688c 100644
--- a/lib/librte_compressdev/rte_compressdev_pmd.h
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -190,6 +190,42 @@ typedef int (*compressdev_queue_pair_release_t)(struct rte_compressdev *dev,
  */
 typedef uint32_t (*compressdev_queue_pair_count_t)(struct rte_compressdev *dev);
 
+/**
+ * Create driver private stream data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param xform
+ *   xform data
+ * @param stream
+ *   ptr where handle of pmd's private stream data should be stored
+ * @return
+ *  - Returns 0 if private stream structure has been created successfully.
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private stream could not be allocated.
+ */
+typedef int (*compressdev_stream_create_t)(struct rte_compressdev *dev,
+		const struct rte_comp_xform *xform, void **stream);
+
+/**
+ * Free driver private stream data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param stream
+ *   handle of pmd's private stream data
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -EBUSY if can't free stream as there are inflight operations
+ */
+typedef int (*compressdev_stream_free_t)(struct rte_compressdev *dev,
+		void *stream);
+
 /**
  * Create driver private_xform data.
  *
@@ -240,6 +276,11 @@ struct rte_compressdev_ops {
 	compressdev_queue_pair_release_t queue_pair_release;
 	/**< Release a queue pair. */
 
+	compressdev_stream_create_t stream_create;
+	/**< Create a comp stream and initialise its private data. */
+	compressdev_stream_free_t stream_free;
+	/**< Free a comp stream's private data. */
+
 	compressdev_private_xform_create_t private_xform_create;
 	/**< Create a comp private_xform and initialise its private data. */
 	compressdev_private_xform_free_t private_xform_free;
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 6636decc2..58cb5205a 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -30,6 +30,8 @@ EXPERIMENTAL {
 	rte_compressdev_socket_id;
 	rte_compressdev_start;
 	rte_compressdev_stop;
+	rte_compressdev_stream_create;
+	rte_compressdev_stream_free;
 	rte_comp_op_pool_create;
 
         local: *;
-- 
2.14.3

[dpdk-dev] [PATCH v3 08/13] compressdev: support hash operations

[Pablo de Lara]

From: Shally Verma <shally.verma@caviumnetworks.com>

- Added hash algo enumeration and params in xform and rte_comp_op
- Updated compress/decompress xform to input hash algorithm
- Updated struct rte_comp_op to input hash buffer

User in capability query will know about support hashes via
device info comp_feature_flag. If supported, application can initialize
desired algorithm enumeration in xform structure and pass valid hash
buffer during enqueue_burst().

Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Sunila Sahu <sunila.sahu@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_comp.h | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/lib/librte_compressdev/rte_comp.h b/lib/librte_compressdev/rte_comp.h
index 4cf84c5db..055f2ee75 100644
--- a/lib/librte_compressdev/rte_comp.h
+++ b/lib/librte_compressdev/rte_comp.h
@@ -55,6 +55,17 @@ enum rte_comp_algorithm {
 	RTE_COMP_ALGO_LIST_END
 };
 
+/** Compression Hash Algorithms */
+enum rte_comp_hash_algorithm {
+	RTE_COMP_HASH_ALGO_UNSPECIFIED = 0,
+	/**< No hash */
+	RTE_COMP_HASH_ALGO_SHA1,
+	/**< SHA1 hash algorithm */
+	RTE_COMP_HASH_ALGO_SHA2_256,
+	/**< SHA256 hash algorithm of SHA2 family */
+	RTE_COMP_HASH_ALGO_LIST_END
+};
+
 /**< Compression Level.
  * The number is interpreted by each PMD differently. However, lower numbers
  * give fastest compression, at the expense of compression ratio while
@@ -162,6 +173,10 @@ struct rte_comp_compress_xform {
 	 */
 	enum rte_comp_checksum_type chksum;
 	/**< Type of checksum to generate on the uncompressed data */
+	enum rte_comp_hash_algorithm hash_algo;
+	/**< Hash algorithm to be used with compress operation. Hash is always
+	 * done on plaintext.
+	 */
 };
 
 /**
@@ -177,6 +192,10 @@ struct rte_comp_decompress_xform {
 	 * compressed data. If window size can't be supported by the PMD then
 	 * setup of stream or private_xform should fail.
 	 */
+	enum rte_comp_hash_algorithm hash_algo;
+	/**< Hash algorithm to be used with decompress operation. Hash is always
+	 * done on plaintext.
+	 */
 };
 
 /**
@@ -268,6 +287,19 @@ struct rte_comp_op {
 		 * decompress direction.
 		 */
 	} dst;
+	struct {
+		uint8_t *digest;
+		/**< Output buffer to store hash output, if enabled in xform.
+		 * Buffer would contain valid value only after an op with
+		 * flush flag = RTE_COMP_FLUSH_FULL/FLUSH_FINAL is processed
+		 * successfully.
+		 *
+		 * Length of buffer should be contiguous and large enough to
+		 * accommodate digest produced by specific hash algo.
+		 */
+		rte_iova_t iova_addr;
+		/**< IO address of the buffer */
+	} hash;
 	enum rte_comp_flush_flag flush_flag;
 	/**< Defines flush characteristics for the output data.
 	 * Only applicable in compress direction
-- 
2.14.3

[dpdk-dev] [PATCH v3 09/13] compressdev: add device feature flags

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 21 ++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 33 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  1 +
 3 files changed, 55 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 433973aaa..b962ccf23 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -56,6 +56,27 @@ static struct rte_compressdev_global compressdev_globals = {
 
 struct rte_compressdev_global *rte_compressdev_globals = &compressdev_globals;
 
+const char * __rte_experimental
+rte_compressdev_get_feature_name(uint64_t flag)
+{
+	switch (flag) {
+	case RTE_COMPDEV_FF_HW_ACCELERATED:
+		return "HW_ACCELERATED";
+	case RTE_COMPDEV_FF_CPU_SSE:
+		return "CPU_SSE";
+	case RTE_COMPDEV_FF_CPU_AVX:
+		return "CPU_AVX";
+	case RTE_COMPDEV_FF_CPU_AVX2:
+		return "CPU_AVX2";
+	case RTE_COMPDEV_FF_CPU_AVX512:
+		return "CPU_AVX512";
+	case RTE_COMPDEV_FF_CPU_NEON:
+		return "CPU_NEON";
+	default:
+		return NULL;
+	}
+}
+
 struct rte_compressdev * __rte_experimental
 rte_compressdev_pmd_get_dev(uint8_t dev_id)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 0e148dd30..72c46dd96 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -34,10 +34,43 @@ extern int compressdev_logtype;
 #define RTE_COMPRESSDEV_NAME_MAX_LEN	(64)
 /**< Max length of name of comp PMD */
 
+/**
+ * compression device supported feature flags
+ *
+ * @note New features flags should be added to the end of the list
+ *
+ * Keep these flags synchronised with rte_compressdev_get_feature_name()
+ */
+#define	RTE_COMPDEV_FF_HW_ACCELERATED		(1ULL << 0)
+/**< Operations are off-loaded to an external hardware accelerator */
+#define	RTE_COMPDEV_FF_CPU_SSE			(1ULL << 1)
+/**< Utilises CPU SIMD SSE instructions */
+#define	RTE_COMPDEV_FF_CPU_AVX			(1ULL << 2)
+/**< Utilises CPU SIMD AVX instructions */
+#define	RTE_COMPDEV_FF_CPU_AVX2			(1ULL << 3)
+/**< Utilises CPU SIMD AVX2 instructions */
+#define	RTE_COMPDEV_FF_CPU_AVX512		(1ULL << 4)
+/**< Utilises CPU SIMD AVX512 instructions */
+#define	RTE_COMPDEV_FF_CPU_NEON			(1ULL << 5)
+/**< Utilises CPU NEON instructions */
+
+/**
+ * Get the name of a compress device feature flag.
+ *
+ * @param flag
+ *   The mask describing the flag
+ *
+ * @return
+ *   The name of this flag, or NULL if it's not a valid feature flag.
+ */
+const char * __rte_experimental
+rte_compressdev_get_feature_name(uint64_t flag);
+
 /**  comp device information */
 struct rte_compressdev_info {
 	const char *driver_name;		/**< Driver name. */
 	uint8_t driver_id;			/**< Driver identifier */
+	uint64_t feature_flags;			/**< Feature flags */
 	uint16_t max_nb_queue_pairs;
 	/**< Maximum number of queues pairs supported by device.
 	 * (If 0, there is no limit in maximum number of queue pairs)
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 58cb5205a..01ddc7a07 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -12,6 +12,7 @@ EXPERIMENTAL {
 	rte_compressdev_devices_get;
 	rte_compressdev_driver_id_get;
 	rte_compressdev_driver_name_get;
+	rte_compressdev_get_feature_name;
 	rte_compressdev_info_get;
 	rte_compressdev_is_valid_dev;
 	rte_compressdev_name_get;
-- 
2.14.3

[dpdk-dev] [PATCH v3 10/13] compressdev: add compression service feature flags

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 31 +++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 45 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  1 +
 3 files changed, 77 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index b962ccf23..6f9c78f74 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -77,6 +77,37 @@ rte_compressdev_get_feature_name(uint64_t flag)
 	}
 }
 
+const char * __rte_experimental
+rte_comp_get_feature_name(uint64_t flag)
+{
+	switch (flag) {
+	case RTE_COMP_FF_STATEFUL_COMPRESSION:
+		return "STATEFUL_COMPRESSION";
+	case RTE_COMP_FF_STATEFUL_DECOMPRESSION:
+		return "STATEFUL_DECOMPRESSION";
+	case RTE_COMP_FF_MBUF_SCATTER_GATHER:
+		return "MBUF_SCATTER_GATHER";
+	case RTE_COMP_FF_MULTI_PKT_CHECKSUM:
+		return "MULTI_PKT_CHECKSUM";
+	case RTE_COMP_FF_ADLER32_CHECKSUM:
+		return "ADLER32_CHECKSUM";
+	case RTE_COMP_FF_CRC32_CHECKSUM:
+		return "CRC32_CHECKSUM";
+	case RTE_COMP_FF_CRC32_ADLER32_CHECKSUM:
+		return "CRC32_ADLER32_CHECKSUM";
+	case RTE_COMP_FF_NONCOMPRESSED_BLOCKS:
+		return "NONCOMPRESSED_BLOCKS";
+	case RTE_COMP_FF_SHA1_HASH:
+		return "SHA1_HASH";
+	case RTE_COMP_FF_SHA2_SHA256_HASH:
+		return "SHA2_SHA256_HASH";
+	case RTE_COMP_FF_SHAREABLE_PRIV_XFORM:
+		return "SHAREABLE_PRIV_XFORM"
+	default:
+		return NULL;
+	}
+}
+
 struct rte_compressdev * __rte_experimental
 rte_compressdev_pmd_get_dev(uint8_t dev_id)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 72c46dd96..02a6c531c 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -54,6 +54,39 @@ extern int compressdev_logtype;
 #define	RTE_COMPDEV_FF_CPU_NEON			(1ULL << 5)
 /**< Utilises CPU NEON instructions */
 
+/**
+ * compression service feature flags
+ *
+ * @note New features flags should be added to the end of the list
+ *
+ * Keep these flags synchronised with rte_comp_get_feature_name()
+ */
+#define RTE_COMP_FF_STATEFUL_COMPRESSION	(1ULL << 0)
+/**< Stateful compression is supported */
+#define RTE_COMP_FF_STATEFUL_DECOMPRESSION	(1ULL << 1)
+/**< Stateful decompression is supported */
+#define	RTE_COMP_FF_MBUF_SCATTER_GATHER		(1ULL << 2)
+/**< Scatter-gather mbufs are supported */
+#define RTE_COMP_FF_ADLER32_CHECKSUM		(1ULL << 3)
+/**< Adler-32 Checksum is supported */
+#define RTE_COMP_FF_CRC32_CHECKSUM		(1ULL << 4)
+/**< CRC32 Checksum is supported */
+#define RTE_COMP_FF_CRC32_ADLER32_CHECKSUM	(1ULL << 5)
+/**< Adler-32/CRC32 Checksum is supported */
+#define RTE_COMP_FF_MULTI_PKT_CHECKSUM		(1ULL << 6)
+/**< Generation of checksum across multiple stateless packets is supported */
+#define RTE_COMP_FF_SHA1_HASH			(1ULL << 7)
+/**< SHA1 Hash is supported */
+#define RTE_COMP_FF_SHA2_SHA256_HASH		(1ULL << 8)
+/**< SHA256 Hash of SHA2 family is supported */
+#define RTE_COMP_FF_NONCOMPRESSED_BLOCKS	(1ULL << 9)
+/**< Creation of non-compressed blocks using RTE_COMP_LEVEL_NONE is supported */
+#define RTE_COMP_FF_SHAREABLE_PRIV_XFORM	(1ULL << 10)
+/**< Private xforms created by the PMD can be shared
+ * across multiple stateless operations. If not set, then app needs
+ * to create as many priv_xforms as many expected in flight.
+ */
+
 /**
  * Get the name of a compress device feature flag.
  *
@@ -66,6 +99,18 @@ extern int compressdev_logtype;
 const char * __rte_experimental
 rte_compressdev_get_feature_name(uint64_t flag);
 
+/**
+ * Get the name of a compress service feature flag
+ *
+ * @param flag
+ *   The mask describing the flag
+ *
+ * @return
+ *   The name of this flag, or NULL if it's not a valid feature flag.
+ */
+const char * __rte_experimental
+rte_comp_get_feature_name(uint64_t flag);
+
 /**  comp device information */
 struct rte_compressdev_info {
 	const char *driver_name;		/**< Driver name. */
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 01ddc7a07..f538b9f57 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -33,6 +33,7 @@ EXPERIMENTAL {
 	rte_compressdev_stop;
 	rte_compressdev_stream_create;
 	rte_compressdev_stream_free;
+	rte_comp_get_feature_name;
 	rte_comp_op_pool_create;
 
         local: *;
-- 
2.14.3

[dpdk-dev] [PATCH v3 11/13] compressdev: add device stats

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 41 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 39 ++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 26 ++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  2 ++
 4 files changed, 108 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 6f9c78f74..b327f9915 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -562,6 +562,47 @@ rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
 			max_inflight_ops, socket_id);
 }
 
+
+int __rte_experimental
+rte_compressdev_stats_get(uint8_t dev_id, struct rte_compressdev_stats *stats)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%d", dev_id);
+		return -ENODEV;
+	}
+
+	if (stats == NULL) {
+		COMPRESSDEV_LOG(ERR, "Invalid stats ptr");
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+	memset(stats, 0, sizeof(*stats));
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->stats_get, -ENOTSUP);
+	(*dev->dev_ops->stats_get)(dev, stats);
+	return 0;
+}
+
+void __rte_experimental
+rte_compressdev_stats_reset(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	RTE_FUNC_PTR_OR_RET(*dev->dev_ops->stats_reset);
+	(*dev->dev_ops->stats_reset)(dev);
+}
+
+
 void __rte_experimental
 rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 02a6c531c..f8aab528c 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -122,6 +122,19 @@ struct rte_compressdev_info {
 	 */
 };
 
+/** comp device statistics */
+struct rte_compressdev_stats {
+	uint64_t enqueued_count;
+	/**< Count of all operations enqueued */
+	uint64_t dequeued_count;
+	/**< Count of all operations dequeued */
+
+	uint64_t enqueue_err_count;
+	/**< Total error count on operations enqueued */
+	uint64_t dequeue_err_count;
+	/**< Total error count on operations dequeued */
+};
+
 /**
  * Get the compress device name given a device identifier.
  *
@@ -306,6 +319,32 @@ rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
 uint16_t __rte_experimental
 rte_compressdev_queue_pair_count(uint8_t dev_id);
 
+
+/**
+ * Retrieve the general I/O statistics of a device.
+ *
+ * @param dev_id
+ *   The identifier of the device
+ * @param stats
+ *   A pointer to a structure of type
+ *   *rte_compressdev_stats* to be filled with the
+ *   values of device counters
+ * @return
+ *   - Zero if successful.
+ *   - Non-zero otherwise.
+ */
+int __rte_experimental
+rte_compressdev_stats_get(uint8_t dev_id, struct rte_compressdev_stats *stats);
+
+/**
+ * Reset the general I/O statistics of a device.
+ *
+ * @param dev_id
+ *   The identifier of the device.
+ */
+void __rte_experimental
+rte_compressdev_stats_reset(uint8_t dev_id);
+
 /**
  * Retrieve the contextual information of a device.
  *
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
index 59bcb688c..e08c69e36 100644
--- a/lib/librte_compressdev/rte_compressdev_pmd.h
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -140,6 +140,27 @@ typedef void (*compressdev_stop_t)(struct rte_compressdev *dev);
 typedef int (*compressdev_close_t)(struct rte_compressdev *dev);
 
 
+/**
+ * Function used to get statistics of a device.
+ *
+ * @param dev
+ *   Compress device
+ * @param stats
+ *   Compress device stats to populate
+ */
+typedef void (*compressdev_stats_get_t)(struct rte_compressdev *dev,
+				struct rte_compressdev_stats *stats);
+
+
+/**
+ * Function used to reset statistics of a device.
+ *
+ * @param dev
+ *   Compress device
+ */
+typedef void (*compressdev_stats_reset_t)(struct rte_compressdev *dev);
+
+
 /**
  * Function used to get specific information of a device.
  *
@@ -271,6 +292,11 @@ struct rte_compressdev_ops {
 
 	compressdev_info_get_t dev_infos_get;	/**< Get device info. */
 
+	compressdev_stats_get_t stats_get;
+	/**< Get device statistics. */
+	compressdev_stats_reset_t stats_reset;
+	/**< Reset device statistics. */
+
 	compressdev_queue_pair_setup_t queue_pair_setup;
 	/**< Set up a device queue pair. */
 	compressdev_queue_pair_release_t queue_pair_release;
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index f538b9f57..7bdc58a38 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -30,6 +30,8 @@ EXPERIMENTAL {
 	rte_compressdev_queue_pair_setup;
 	rte_compressdev_socket_id;
 	rte_compressdev_start;
+	rte_compressdev_stats_get;
+	rte_compressdev_stats_reset;
 	rte_compressdev_stop;
 	rte_compressdev_stream_create;
 	rte_compressdev_stream_free;
-- 
2.14.3

[dpdk-dev] [PATCH v3 12/13] compressdev: add device capabilities

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added structure which each PMD will fill out,
providing the capabilities of each driver
(containing mainly which compression services
it supports).

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 23 ++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 35 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  1 +
 3 files changed, 59 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index b327f9915..70d5de92c 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -56,6 +56,29 @@ static struct rte_compressdev_global compressdev_globals = {
 
 struct rte_compressdev_global *rte_compressdev_globals = &compressdev_globals;
 
+const struct rte_compressdev_capabilities * __rte_experimental
+rte_compressdev_capability_get(uint8_t dev_id,
+			enum rte_comp_algorithm algo)
+{
+	const struct rte_compressdev_capabilities *capability;
+	struct rte_compressdev_info dev_info;
+	int i = 0;
+
+	if (dev_id >= compressdev_globals.nb_devs) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%d", dev_id);
+		return NULL;
+	}
+	rte_compressdev_info_get(dev_id, &dev_info);
+
+	while ((capability = &dev_info.capabilities[i++])->algo !=
+			RTE_COMP_ALGO_UNSPECIFIED){
+		if (capability->algo == algo)
+			return capability;
+	}
+
+	return NULL;
+}
+
 const char * __rte_experimental
 rte_compressdev_get_feature_name(uint64_t flag)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index f8aab528c..e9cb212ec 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -34,6 +34,39 @@ extern int compressdev_logtype;
 #define RTE_COMPRESSDEV_NAME_MAX_LEN	(64)
 /**< Max length of name of comp PMD */
 
+/**
+ * Parameter log base 2 range description.
+ * Final value will be 2^value.
+ */
+struct rte_param_log2_range {
+	uint8_t min;	/**< Minimum log2 value */
+	uint8_t max;	/**< Maximum log2 value */
+	uint8_t increment;
+	/**< If a range of sizes are supported,
+	 * this parameter is used to indicate
+	 * increments in base 2 log byte value
+	 * that are supported between the minimum and maximum
+	 */
+};
+
+/** Structure used to capture a capability of a comp device */
+struct rte_compressdev_capabilities {
+	enum rte_comp_algorithm algo;
+	/* Compression algorithm */
+	uint64_t comp_feature_flags;
+	/**< Bitmask of flags for compression service features */
+	struct rte_param_log2_range window_size;
+	/**< Window size range in base two log byte values */
+};
+
+/** Macro used at end of comp PMD list */
+#define RTE_COMP_END_OF_CAPABILITIES_LIST() \
+	{ RTE_COMP_ALGO_UNSPECIFIED }
+
+const struct rte_compressdev_capabilities * __rte_experimental
+rte_compressdev_capability_get(uint8_t dev_id,
+			enum rte_comp_algorithm algo);
+
 /**
  * compression device supported feature flags
  *
@@ -116,6 +149,8 @@ struct rte_compressdev_info {
 	const char *driver_name;		/**< Driver name. */
 	uint8_t driver_id;			/**< Driver identifier */
 	uint64_t feature_flags;			/**< Feature flags */
+	const struct rte_compressdev_capabilities *capabilities;
+	/**< Array of devices supported capabilities */
 	uint16_t max_nb_queue_pairs;
 	/**< Maximum number of queues pairs supported by device.
 	 * (If 0, there is no limit in maximum number of queue pairs)
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 7bdc58a38..dec73fcff 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -5,6 +5,7 @@ EXPERIMENTAL {
 	rte_compressdev_allocate_driver;
 	rte_compressdev_callback_register;
 	rte_compressdev_callback_unregister;
+	rte_compressdev_capability_get;
 	rte_compressdev_close;
 	rte_compressdev_configure;
 	rte_compressdev_count;
-- 
2.14.3

[dpdk-dev] [PATCH v3 13/13] compressdev: get device id from name

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added API to retrieve the device id provided the device name.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 18 ++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 13 +++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  1 +
 3 files changed, 32 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 70d5de92c..a29255cc0 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -173,6 +173,24 @@ rte_compressdev_is_valid_dev(uint8_t dev_id)
 }
 
 
+int __rte_experimental
+rte_compressdev_get_dev_id(const char *name)
+{
+	unsigned int i;
+
+	if (name == NULL)
+		return -1;
+
+	for (i = 0; i < rte_compressdev_globals->nb_devs; i++)
+		if ((strcmp(rte_compressdev_globals->devs[i].data->name, name)
+				== 0) &&
+				(rte_compressdev_globals->devs[i].attached ==
+						RTE_COMPRESSDEV_ATTACHED))
+			return i;
+
+	return -1;
+}
+
 uint8_t __rte_experimental
 rte_compressdev_count(void)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index e9cb212ec..1d1a072ef 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -170,6 +170,19 @@ struct rte_compressdev_stats {
 	/**< Total error count on operations dequeued */
 };
 
+
+/**
+ * Get the device identifier for the named compress device.
+ *
+ * @param name
+ *   Device name to select the device structure
+ * @return
+ *   - Returns compress device identifier on success.
+ *   - Return -1 on failure to find named compress device.
+ */
+int __rte_experimental
+rte_compressdev_get_dev_id(const char *name);
+
 /**
  * Get the compress device name given a device identifier.
  *
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index dec73fcff..46bdda88b 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -13,6 +13,7 @@ EXPERIMENTAL {
 	rte_compressdev_devices_get;
 	rte_compressdev_driver_id_get;
 	rte_compressdev_driver_name_get;
+	rte_compressdev_get_dev_id;
 	rte_compressdev_get_feature_name;
 	rte_compressdev_info_get;
 	rte_compressdev_is_valid_dev;
-- 
2.14.3

[dpdk-dev] [PATCH v4 00/13] Implement compression API

[Pablo de Lara]

With the vast amounts of data being transported around networks
and stored in storage systems, reducing data size is becoming ever more important.

There are both software libraries and hardware devices available that
provide compression, but no common API.
Such an API is proposed in this patchset, which supports the following features:

- Deflate Algorithm (https://tools.ietf.org/html/rfc1951)
- LZS algorithm (https://tools.ietf.org/html/rfc2395)
- Static and Dynamic Huffman encoding.
- Compression levels
- Checksum generation
- Asynchronous burst API
- private_xform - a place for PMDs to hold private data derived from
  a xform and used by stateless operations.
- stream - a place for PMDs to hold private data derived from
  a xform and also maintain state and history data. For
  stateful flows.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara pablo.de.lara.guarch@intel.com
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>

Opens:
 - creation of private_xform and stream mempools. In v3, config
   added to give PMD the size params needed to create mempools.
   Still open as to whether the pools should be created by
   application, in API layer or by PMDs. Expect to resolve in a v4. 
 - alternative to mbufs for passing data between application
   and accelerator. A lot of varied opinions on this, expect to
   address in a future release.
 - addition of feature to add capability to PMD to allow more than one
   inflight operation from a stateful stream - to be added by NXP 

Changes in v4:
 - Fix build (missing ";")
 - Change order in common_base, so compression is placed after security

Changes in v3:
 - Remove rte_comp_op_ctod helper functions
 - Remove param_range_check macro
 - Rename from phys_addr to iova_addr
 - Remove rte_comp_algo_strings
 - Rename rte_compressdev_pmd_is_valid_dev to rte_compressdev_is_valid_dev
 - Remove feature flags from compressdev
 - Support hash operations
 - Add shareable priv xform in feature flags, instead of returnin it
   on xform creation
 - Allow max number of queue pairs to be 0, meaning that there is no
   limit.
 - Add invalid configuration checks
 - Add capability helper functions

Changes in v2:
 - Add stream APIs
 - Remove session
 - Add SHAREABLE / NON_SHAREABLE private_xform types
 - Add algo enum 'UNSPECIFIED' to fix warning in capabilities
 - Change one remaining log to use dynamic logging.
 - Add rte_cache_aligned keyword to op
 - Rename enums with better names _ALGO, __CHECKSUM, _HUFFMAN_
 - Use const keyword when passing xform
 - Remove qp_count fn from dev_ops as never used
 - Remove max_nb_queue-pairs from compressdev_init_param as never used
 - Clarify device configure and start sequence
 - Replace OUT_OF_SPACE with OUT_OF_SPACE_RECOVERABLE and TERMINATED
   and clarified usage. 
 - Add stream and private_xform sizes to device config for use in
   mempool creation
 - Add capability helper fn
 - Use Base2 log value for window size on xforms
 - Add Meson build
 - Update MAINTAINERS
 - Update Map file
 - Change order in doxy file
 - Update Release note 

Fiona Trahe (12):
  compressdev: add basic device management
  compressdev: add queue pair management
  compressdev: add compression specific data
  compressdev: add enqueue/dequeue functions
  compressdev: add operation management
  compressdev: support stateless operations
  compressdev: support stateful operations
  compressdev: add device feature flags
  compressdev: add compression service feature flags
  compressdev: add device stats
  compressdev: add device capabilities
  compressdev: get device id from name

Shally Verma (1):
  compressdev: support hash operations

 MAINTAINERS                                        |   7 +
 config/common_base                                 |   6 +
 config/rte_config.h                                |   3 +
 doc/api/doxy-api-index.md                          |   1 +
 doc/api/doxy-api.conf                              |   1 +
 doc/guides/rel_notes/release_18_05.rst             |   6 +
 lib/Makefile                                       |   3 +
 lib/librte_compressdev/Makefile                    |  29 +
 lib/librte_compressdev/meson.build                 |   9 +
 lib/librte_compressdev/rte_comp.h                  | 539 ++++++++++++
 lib/librte_compressdev/rte_compressdev.c           | 902 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 701 ++++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.c       | 157 ++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 437 ++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  44 +
 lib/meson.build                                    |   2 +-
 mk/rte.app.mk                                      |   1 +
 17 files changed, 2847 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_compressdev/Makefile
 create mode 100644 lib/librte_compressdev/meson.build
 create mode 100644 lib/librte_compressdev/rte_comp.h
 create mode 100644 lib/librte_compressdev/rte_compressdev.c
 create mode 100644 lib/librte_compressdev/rte_compressdev.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.c
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_version.map

-- 
2.14.3

[dpdk-dev] [PATCH v4 01/13] compressdev: add basic device management

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Add basic functions to manage compress devices,
including driver and device allocation, and the basic
interface with compressdev PMDs.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 MAINTAINERS                                        |   7 +
 config/common_base                                 |   6 +
 config/rte_config.h                                |   3 +
 doc/api/doxy-api-index.md                          |   1 +
 doc/api/doxy-api.conf                              |   1 +
 doc/guides/rel_notes/release_18_05.rst             |   6 +
 lib/Makefile                                       |   3 +
 lib/librte_compressdev/Makefile                    |  28 ++
 lib/librte_compressdev/meson.build                 |   8 +
 lib/librte_compressdev/rte_compressdev.c           | 464 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 267 ++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.c       | 157 +++++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 283 +++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  31 ++
 lib/meson.build                                    |   2 +-
 mk/rte.app.mk                                      |   1 +
 16 files changed, 1267 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_compressdev/Makefile
 create mode 100644 lib/librte_compressdev/meson.build
 create mode 100644 lib/librte_compressdev/rte_compressdev.c
 create mode 100644 lib/librte_compressdev/rte_compressdev.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.c
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 1081d71aa..75f13f92e 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -325,6 +325,13 @@ F: drivers/raw/skeleton_rawdev/
 F: test/test/test_rawdev.c
 F: doc/guides/prog_guide/rawdev.rst
 
+Compression API - EXPERIMENTAL
+M: Fiona Trahe <fiona.trahe@intel.com>
+M: Pablo de Lara <pablo.de.lara.guarch@intel.com>
+M: Ashish Gupta <ashish.gupta@caviumnetworks.com>
+T: git://dpdk.org/next/dpdk-next-crypto
+F: lib/librte_compressdev/
+
 
 Bus Drivers
 -----------
diff --git a/config/common_base b/config/common_base
index ad4706267..f40354487 100644
--- a/config/common_base
+++ b/config/common_base
@@ -543,6 +543,12 @@ CONFIG_RTE_LIBRTE_PMD_MRVL_CRYPTO_DEBUG=n
 #
 CONFIG_RTE_LIBRTE_SECURITY=y
 
+#
+# Compile generic compression device library
+#
+CONFIG_RTE_LIBRTE_COMPRESSDEV=y
+CONFIG_RTE_COMPRESS_MAX_DEVS=64
+
 #
 # Compile generic event device library
 #
diff --git a/config/rte_config.h b/config/rte_config.h
index db6ceb6cd..949071f6e 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -51,6 +51,9 @@
 #define RTE_CRYPTO_MAX_DEVS 64
 #define RTE_CRYPTODEV_NAME_LEN 64
 
+/* compressdev defines */
+#define RTE_COMPRESS_MAX_DEVS 64
+
 /* eventdev defines */
 #define RTE_EVENT_MAX_DEVS 16
 #define RTE_EVENT_MAX_QUEUES_PER_DEV 64
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index d77f205bb..530808e9d 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -45,6 +45,7 @@ The public API headers are grouped by topics:
   [bbdev]              (@ref rte_bbdev.h),
   [cryptodev]          (@ref rte_cryptodev.h),
   [security]           (@ref rte_security.h),
+  [compressdev]        (@ref rte_compressdev.h),
   [eventdev]           (@ref rte_eventdev.h),
   [event_eth_rx_adapter]   (@ref rte_event_eth_rx_adapter.h),
   [rawdev]             (@ref rte_rawdev.h),
diff --git a/doc/api/doxy-api.conf b/doc/api/doxy-api.conf
index cda52fdfb..06432c3aa 100644
--- a/doc/api/doxy-api.conf
+++ b/doc/api/doxy-api.conf
@@ -45,6 +45,7 @@ INPUT                   = doc/api/doxy-api-index.md \
                           lib/librte_cfgfile \
                           lib/librte_cmdline \
                           lib/librte_compat \
+                          lib/librte_compressdev \
                           lib/librte_cryptodev \
                           lib/librte_distributor \
                           lib/librte_efd \
diff --git a/doc/guides/rel_notes/release_18_05.rst b/doc/guides/rel_notes/release_18_05.rst
index 7c428439e..77d3c1bf8 100644
--- a/doc/guides/rel_notes/release_18_05.rst
+++ b/doc/guides/rel_notes/release_18_05.rst
@@ -70,6 +70,11 @@ New Features
 
   * AES-CMAC (128-bit key).
 
+* **Added compressdev API, a generic compression service library.**
+
+  The compressdev library provides an API for offload of compression and
+  decompression operations to hardware or software accelerator devices.
+
 
 API Changes
 -----------
@@ -155,6 +160,7 @@ The libraries prepended with a plus sign were incremented in this version.
      librte_cfgfile.so.2
      librte_cmdline.so.2
    + librte_common_octeontx.so.1
+   + librte_compressdev.so.1
      librte_cryptodev.so.4
      librte_distributor.so.1
    + librte_eal.so.7
diff --git a/lib/Makefile b/lib/Makefile
index ec965a606..19396daff 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -26,6 +26,9 @@ DEPDIRS-librte_bbdev := librte_eal librte_mempool librte_mbuf
 DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += librte_cryptodev
 DEPDIRS-librte_cryptodev := librte_eal librte_mempool librte_ring librte_mbuf
 DEPDIRS-librte_cryptodev += librte_kvargs
+DIRS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += librte_compressdev
+DEPDIRS-librte_compressdev := librte_eal librte_mempool librte_ring librte_mbuf
+DEPDIRS-librte_compressdev += librte_kvargs
 DIRS-$(CONFIG_RTE_LIBRTE_SECURITY) += librte_security
 DEPDIRS-librte_security := librte_eal librte_mempool librte_ring librte_mbuf
 DEPDIRS-librte_security += librte_ether
diff --git a/lib/librte_compressdev/Makefile b/lib/librte_compressdev/Makefile
new file mode 100644
index 000000000..5f67ab817
--- /dev/null
+++ b/lib/librte_compressdev/Makefile
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2017-2018 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_compressdev.a
+
+# library version
+LIBABIVER := 1
+
+# build flags
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+LDLIBS += -lrte_eal -lrte_mempool -lrte_kvargs
+
+# library source files
+SRCS-y += rte_compressdev.c rte_compressdev_pmd.c
+
+# export include files
+SYMLINK-y-include += rte_compressdev.h
+SYMLINK-y-include += rte_compressdev_pmd.h
+
+# versioning export map
+EXPORT_MAP := rte_compressdev_version.map
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_compressdev/meson.build b/lib/librte_compressdev/meson.build
new file mode 100644
index 000000000..fc5eaf009
--- /dev/null
+++ b/lib/librte_compressdev/meson.build
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+allow_experimental_apis = true
+sources = files('rte_compressdev.c', 'rte_compressdev_pmd.c')
+headers = files('rte_compressdev.h',
+	'rte_compressdev_pmd.h')
+deps += ['kvargs', 'mbuf']
diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
new file mode 100644
index 000000000..d635953b2
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -0,0 +1,464 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#include <sys/types.h>
+#include <sys/queue.h>
+#include <ctype.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stdarg.h>
+#include <errno.h>
+#include <stdint.h>
+#include <inttypes.h>
+#include <netinet/in.h>
+
+#include <rte_byteorder.h>
+#include <rte_log.h>
+#include <rte_debug.h>
+#include <rte_dev.h>
+#include <rte_interrupts.h>
+#include <rte_memory.h>
+#include <rte_memcpy.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_tailq.h>
+#include <rte_eal.h>
+#include <rte_per_lcore.h>
+#include <rte_lcore.h>
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_mempool.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_errno.h>
+#include <rte_spinlock.h>
+#include <rte_string_fns.h>
+
+#include "rte_comp.h"
+#include "rte_compressdev.h"
+#include "rte_compressdev_pmd.h"
+
+static uint8_t nb_drivers;
+
+struct rte_compressdev rte_comp_devices[RTE_COMPRESS_MAX_DEVS];
+
+struct rte_compressdev *rte_compressdevs = &rte_comp_devices[0];
+
+static struct rte_compressdev_global compressdev_globals = {
+		.devs			= &rte_comp_devices[0],
+		.data			= { NULL },
+		.nb_devs		= 0,
+		.max_devs		= RTE_COMPRESS_MAX_DEVS
+};
+
+struct rte_compressdev_global *rte_compressdev_globals = &compressdev_globals;
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_get_dev(uint8_t dev_id)
+{
+	return &rte_compressdev_globals->devs[dev_id];
+}
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_get_named_dev(const char *name)
+{
+	struct rte_compressdev *dev;
+	unsigned int i;
+
+	if (name == NULL)
+		return NULL;
+
+	for (i = 0; i < rte_compressdev_globals->max_devs; i++) {
+		dev = &rte_compressdev_globals->devs[i];
+
+		if ((dev->attached == RTE_COMPRESSDEV_ATTACHED) &&
+				(strcmp(dev->data->name, name) == 0))
+			return dev;
+	}
+
+	return NULL;
+}
+
+unsigned int __rte_experimental
+rte_compressdev_is_valid_dev(uint8_t dev_id)
+{
+	struct rte_compressdev *dev = NULL;
+
+	if (dev_id >= rte_compressdev_globals->nb_devs)
+		return 0;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+	if (dev->attached != RTE_COMPRESSDEV_ATTACHED)
+		return 0;
+	else
+		return 1;
+}
+
+
+uint8_t __rte_experimental
+rte_compressdev_count(void)
+{
+	return rte_compressdev_globals->nb_devs;
+}
+
+uint8_t __rte_experimental
+rte_compressdev_device_count_by_driver(uint8_t driver_id)
+{
+	uint8_t i, dev_count = 0;
+
+	for (i = 0; i < rte_compressdev_globals->max_devs; i++)
+		if (rte_compressdev_globals->devs[i].driver_id == driver_id &&
+			rte_compressdev_globals->devs[i].attached ==
+					RTE_COMPRESSDEV_ATTACHED)
+			dev_count++;
+
+	return dev_count;
+}
+
+uint8_t __rte_experimental
+rte_compressdev_devices_get(const char *driver_name, uint8_t *devices,
+	uint8_t nb_devices)
+{
+	uint8_t i, count = 0;
+	struct rte_compressdev *devs = rte_compressdev_globals->devs;
+	uint8_t max_devs = rte_compressdev_globals->max_devs;
+
+	for (i = 0; i < max_devs && count < nb_devices;	i++) {
+
+		if (devs[i].attached == RTE_COMPRESSDEV_ATTACHED) {
+			int cmp;
+
+			cmp = strncmp(devs[i].device->driver->name,
+					driver_name,
+					strlen(driver_name));
+
+			if (cmp == 0)
+				devices[count++] = devs[i].data->dev_id;
+		}
+	}
+
+	return count;
+}
+
+int __rte_experimental
+rte_compressdev_socket_id(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id))
+		return -1;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	return dev->data->socket_id;
+}
+
+static inline int
+rte_compressdev_data_alloc(uint8_t dev_id, struct rte_compressdev_data **data,
+		int socket_id)
+{
+	char mz_name[RTE_COMPRESSDEV_NAME_MAX_LEN];
+	const struct rte_memzone *mz;
+	int n;
+
+	/* generate memzone name */
+	n = snprintf(mz_name, sizeof(mz_name),
+			"rte_compressdev_data_%u", dev_id);
+	if (n >= (int)sizeof(mz_name))
+		return -EINVAL;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		mz = rte_memzone_reserve(mz_name,
+				sizeof(struct rte_compressdev_data),
+				socket_id, 0);
+	} else
+		mz = rte_memzone_lookup(mz_name);
+
+	if (mz == NULL)
+		return -ENOMEM;
+
+	*data = mz->addr;
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		memset(*data, 0, sizeof(struct rte_compressdev_data));
+
+	return 0;
+}
+
+static uint8_t
+rte_compressdev_find_free_device_index(void)
+{
+	uint8_t dev_id;
+
+	for (dev_id = 0; dev_id < RTE_COMPRESS_MAX_DEVS; dev_id++) {
+		if (rte_comp_devices[dev_id].attached ==
+				RTE_COMPRESSDEV_DETACHED)
+			return dev_id;
+	}
+	return RTE_COMPRESS_MAX_DEVS;
+}
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_allocate(const char *name, int socket_id)
+{
+	struct rte_compressdev *compressdev;
+	uint8_t dev_id;
+
+	if (rte_compressdev_pmd_get_named_dev(name) != NULL) {
+		COMPRESSDEV_LOG(ERR,
+			"comp device with name %s already allocated!", name);
+		return NULL;
+	}
+
+	dev_id = rte_compressdev_find_free_device_index();
+	if (dev_id == RTE_COMPRESS_MAX_DEVS) {
+		COMPRESSDEV_LOG(ERR, "Reached maximum number of comp devices");
+		return NULL;
+	}
+	compressdev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (compressdev->data == NULL) {
+		struct rte_compressdev_data *compressdev_data =
+				compressdev_globals.data[dev_id];
+
+		int retval = rte_compressdev_data_alloc(dev_id,
+				&compressdev_data, socket_id);
+
+		if (retval < 0 || compressdev_data == NULL)
+			return NULL;
+
+		compressdev->data = compressdev_data;
+
+		snprintf(compressdev->data->name, RTE_COMPRESSDEV_NAME_MAX_LEN,
+				"%s", name);
+
+		compressdev->data->dev_id = dev_id;
+		compressdev->data->socket_id = socket_id;
+		compressdev->data->dev_started = 0;
+
+		compressdev->attached = RTE_COMPRESSDEV_ATTACHED;
+
+		compressdev_globals.nb_devs++;
+	}
+
+	return compressdev;
+}
+
+int __rte_experimental
+rte_compressdev_pmd_release_device(struct rte_compressdev *compressdev)
+{
+	int ret;
+
+	if (compressdev == NULL)
+		return -EINVAL;
+
+	/* Close device only if device operations have been set */
+	if (compressdev->dev_ops) {
+		ret = rte_compressdev_close(compressdev->data->dev_id);
+		if (ret < 0)
+			return ret;
+	}
+
+	compressdev->attached = RTE_COMPRESSDEV_DETACHED;
+	compressdev_globals.nb_devs--;
+	return 0;
+}
+
+int __rte_experimental
+rte_compressdev_configure(uint8_t dev_id, struct rte_compressdev_config *config)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	if (dev->data->dev_started) {
+		COMPRESSDEV_LOG(ERR,
+		    "device %d must be stopped to allow configuration", dev_id);
+		return -EBUSY;
+	}
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_configure, -ENOTSUP);
+
+	return (*dev->dev_ops->dev_configure)(dev, config);
+}
+
+
+int __rte_experimental
+rte_compressdev_start(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+	int diag;
+
+	COMPRESSDEV_LOG(DEBUG, "Start dev_id=%" PRIu8, dev_id);
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_start, -ENOTSUP);
+
+	if (dev->data->dev_started != 0) {
+		COMPRESSDEV_LOG(ERR,
+		    "Device with dev_id=%" PRIu8 " already started", dev_id);
+		return 0;
+	}
+
+	diag = (*dev->dev_ops->dev_start)(dev);
+	if (diag == 0)
+		dev->data->dev_started = 1;
+	else
+		return diag;
+
+	return 0;
+}
+
+void __rte_experimental
+rte_compressdev_stop(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_stop);
+
+	if (dev->data->dev_started == 0) {
+		COMPRESSDEV_LOG(ERR,
+		    "Device with dev_id=%" PRIu8 " already stopped", dev_id);
+		return;
+	}
+
+	(*dev->dev_ops->dev_stop)(dev);
+	dev->data->dev_started = 0;
+}
+
+int __rte_experimental
+rte_compressdev_close(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+	int retval;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -1;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	/* Device must be stopped before it can be closed */
+	if (dev->data->dev_started == 1) {
+		COMPRESSDEV_LOG(ERR, "Device %u must be stopped before closing",
+				dev_id);
+		return -EBUSY;
+	}
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_close, -ENOTSUP);
+	retval = (*dev->dev_ops->dev_close)(dev);
+
+	if (retval < 0)
+		return retval;
+
+	return 0;
+}
+
+void __rte_experimental
+rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
+{
+	struct rte_compressdev *dev;
+
+	if (dev_id >= compressdev_globals.nb_devs) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%d", dev_id);
+		return;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	memset(dev_info, 0, sizeof(struct rte_compressdev_info));
+
+	RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
+	(*dev->dev_ops->dev_infos_get)(dev, dev_info);
+
+	dev_info->driver_name = dev->device->driver->name;
+}
+
+TAILQ_HEAD(compressdev_driver_list, compressdev_driver);
+
+static struct compressdev_driver_list compressdev_driver_list =
+	TAILQ_HEAD_INITIALIZER(compressdev_driver_list);
+
+int __rte_experimental
+rte_compressdev_driver_id_get(const char *name)
+{
+	struct compressdev_driver *driver;
+	const char *driver_name;
+
+	if (name == NULL) {
+		COMPRESSDEV_LOG(DEBUG, "name pointer NULL");
+		return -1;
+	}
+
+	TAILQ_FOREACH(driver, &compressdev_driver_list, next) {
+		driver_name = driver->driver->name;
+		if (strncmp(driver_name, name, strlen(driver_name)) == 0)
+			return driver->id;
+	}
+	return -1;
+}
+
+const char * __rte_experimental
+rte_compressdev_name_get(uint8_t dev_id)
+{
+	struct rte_compressdev *dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (dev == NULL)
+		return NULL;
+
+	return dev->data->name;
+}
+
+const char * __rte_experimental
+rte_compressdev_driver_name_get(uint8_t driver_id)
+{
+	struct compressdev_driver *driver;
+
+	TAILQ_FOREACH(driver, &compressdev_driver_list, next)
+		if (driver->id == driver_id)
+			return driver->driver->name;
+	return NULL;
+}
+
+uint8_t
+rte_compressdev_allocate_driver(struct compressdev_driver *comp_drv,
+		const struct rte_driver *drv)
+{
+	comp_drv->driver = drv;
+	comp_drv->id = nb_drivers;
+
+	TAILQ_INSERT_TAIL(&compressdev_driver_list, comp_drv, next);
+
+	return nb_drivers++;
+}
+
+RTE_INIT(rte_compressdev_log);
+
+static void
+rte_compressdev_log(void)
+{
+	compressdev_logtype = rte_log_register("lib.compressdev");
+	if (compressdev_logtype >= 0)
+		rte_log_set_level(compressdev_logtype, RTE_LOG_NOTICE);
+}
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
new file mode 100644
index 000000000..c5de61378
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -0,0 +1,267 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMPRESSDEV_H_
+#define _RTE_COMPRESSDEV_H_
+
+/**
+ * @file rte_compressdev.h
+ *
+ * RTE Compression Device APIs
+ *
+ * Defines comp device APIs for the provisioning of compression operations.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include "rte_kvargs.h"
+#include "rte_comp.h"
+#include "rte_dev.h"
+#include <rte_common.h>
+
+/* Logging Macros */
+extern int compressdev_logtype;
+#define COMPRESSDEV_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, compressdev_logtype, "%s(): "fmt "\n", \
+			__func__, ##args)
+
+#define RTE_COMPRESSDEV_DETACHED  (0)
+#define RTE_COMPRESSDEV_ATTACHED  (1)
+
+#define RTE_COMPRESSDEV_NAME_MAX_LEN	(64)
+/**< Max length of name of comp PMD */
+
+/**  comp device information */
+struct rte_compressdev_info {
+	const char *driver_name;		/**< Driver name. */
+	uint8_t driver_id;			/**< Driver identifier */
+};
+
+/**
+ * Get the compress device name given a device identifier.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - Returns compress device name.
+ *   - Returns NULL if compress device is not present.
+ */
+const char * __rte_experimental
+rte_compressdev_name_get(uint8_t dev_id);
+
+/**
+ * Get the total number of compress devices that have been successfully
+ * initialised.
+ *
+ * @return
+ *   - The total number of usable compress devices.
+ */
+uint8_t __rte_experimental
+rte_compressdev_count(void);
+
+/**
+ * Get number of comp device defined type.
+ *
+ * @param driver_id
+ *   Driver identifier
+ * @return
+ *   Returns number of comp device.
+ */
+uint8_t __rte_experimental
+rte_compressdev_device_count_by_driver(uint8_t driver_id);
+
+/**
+ * Get number and identifiers of attached comp devices that
+ * use the same compress driver.
+ *
+ * @param driver_name
+ *   Driver name
+ * @param devices
+ *   Output devices identifiers
+ * @param nb_devices
+ *   Maximal number of devices
+ *
+ * @return
+ *   Returns number of attached compress devices.
+ */
+uint8_t __rte_experimental
+rte_compressdev_devices_get(const char *driver_name, uint8_t *devices,
+		uint8_t nb_devices);
+
+/**
+ * Validate if the comp device index is valid attached comp device.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - If the device index is valid (1) or not (0).
+ */
+unsigned int __rte_experimental
+rte_compressdev_is_valid_dev(uint8_t dev_id);
+
+/*
+ * Return the NUMA socket to which a device is connected.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   The NUMA socket id to which the device is connected or
+ *   a default of zero if the socket could not be determined.
+ *   -1 if returned is the dev_id value is out of range.
+ */
+int __rte_experimental
+rte_compressdev_socket_id(uint8_t dev_id);
+
+/** Compress device configuration structure */
+struct rte_compressdev_config {
+	int socket_id;
+};
+
+/**
+ * Configure a device.
+ *
+ * This function must be invoked first before any other function in the
+ * API. This function can also be re-invoked when a device is in the
+ * stopped state.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param config
+ *   The compress device configuration
+ * @return
+ *   - 0: Success, device configured.
+ *   - <0: Error code returned by the driver configuration function.
+ */
+int __rte_experimental
+rte_compressdev_configure(uint8_t dev_id,
+			struct rte_compressdev_config *config);
+
+/**
+ * Start a device.
+ *
+ * The device start step is called after configuring the device and setting up
+ * its queue pairs.
+ * On success, data-path functions exported by the API (enqueue/dequeue, etc)
+ * can be invoked.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - 0: Success, device started.
+ *   - <0: Error code of the driver device start function.
+ */
+int __rte_experimental
+rte_compressdev_start(uint8_t dev_id);
+
+/**
+ * Stop a device. The device can be restarted with a call to
+ * rte_compressdev_start()
+ *
+ * @param dev_id
+ *   Compress device identifier
+ */
+void __rte_experimental
+rte_compressdev_stop(uint8_t dev_id);
+
+/**
+ * Close an device. The device cannot be restarted!
+ *
+ * @param dev_id
+ *   Compress device identifier
+ *
+ * @return
+ *  - 0 on successfully closing device
+ *  - <0 on failure to close device
+ */
+int __rte_experimental
+rte_compressdev_close(uint8_t dev_id);
+
+/**
+ * Retrieve the contextual information of a device.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param dev_info
+ *   A pointer to a structure of type *rte_compressdev_info*
+ *   to be filled with the contextual information of the device
+ *
+ * @note The capabilities field of dev_info is set to point to the first
+ * element of an array of struct rte_compressdev_capabilities.
+ * The element after the last valid element has it's op field set to
+ * RTE_COMP_ALGO_LIST_END.
+ */
+void __rte_experimental
+rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info);
+
+/** The data structure associated with each comp device. */
+struct rte_compressdev {
+	struct rte_compressdev_data *data;
+	/**< Pointer to device data */
+	struct rte_compressdev_ops *dev_ops;
+	/**< Functions exported by PMD */
+	struct rte_device *device;
+	/**< Backing device */
+
+	uint8_t driver_id;
+	/**< comp driver identifier*/
+
+	__extension__
+	uint8_t attached : 1;
+	/**< Flag indicating the device is attached */
+} __rte_cache_aligned;
+
+/**
+ *
+ * The data part, with no function pointers, associated with each device.
+ *
+ * This structure is safe to place in shared memory to be common among
+ * different processes in a multi-process configuration.
+ */
+struct rte_compressdev_data {
+	uint8_t dev_id;
+	/**< Compress device identifier */
+	uint8_t socket_id;
+	/**< Socket identifier where memory is allocated */
+	char name[RTE_COMPRESSDEV_NAME_MAX_LEN];
+	/**< Unique identifier name */
+
+	__extension__
+	uint8_t dev_started : 1;
+	/**< Device state: STARTED(1)/STOPPED(0) */
+
+	void *dev_private;
+	/**< PMD-specific private data */
+} __rte_cache_aligned;
+
+struct rte_compressdev *rte_compressdevs;
+
+/**
+ * Provide driver identifier.
+ *
+ * @param name
+ *   Compress driver name
+ * @return
+ *  The driver type identifier or -1 if no driver found
+ */
+int __rte_experimental
+rte_compressdev_driver_id_get(const char *name);
+
+/**
+ * Provide driver name.
+ *
+ * @param driver_id
+ *   The driver identifier
+ * @return
+ *  The driver name or null if no driver found
+ */
+const char * __rte_experimental
+rte_compressdev_driver_name_get(uint8_t driver_id);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_COMPRESSDEV_H_ */
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.c b/lib/librte_compressdev/rte_compressdev_pmd.c
new file mode 100644
index 000000000..148596e0b
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_pmd.c
@@ -0,0 +1,157 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#include <rte_malloc.h>
+#include <rte_kvargs.h>
+
+#include "rte_compressdev_pmd.h"
+
+int compressdev_logtype;
+
+/**
+ * Parse name from argument
+ */
+static int
+rte_compressdev_pmd_parse_name_arg(const char *key __rte_unused,
+		const char *value, void *extra_args)
+{
+	struct rte_compressdev_pmd_init_params *params = extra_args;
+	int n;
+
+	n = snprintf(params->name, RTE_COMPRESSDEV_NAME_MAX_LEN, "%s", value);
+	if (n >= RTE_COMPRESSDEV_NAME_MAX_LEN)
+		return -EINVAL;
+
+	return 0;
+}
+
+/**
+ * Parse unsigned integer from argument
+ */
+static int
+rte_compressdev_pmd_parse_uint_arg(const char *key __rte_unused,
+		const char *value, void *extra_args)
+{
+	int i;
+	char *end;
+
+	errno = 0;
+	i = strtol(value, &end, 10);
+	if (*end != 0 || errno != 0 || i < 0)
+		return -EINVAL;
+
+	*((uint32_t *)extra_args) = i;
+	return 0;
+}
+
+int
+rte_compressdev_pmd_parse_input_args(
+		struct rte_compressdev_pmd_init_params *params,
+		const char *args)
+{
+	struct rte_kvargs *kvlist = NULL;
+	int ret = 0;
+
+	if (params == NULL)
+		return -EINVAL;
+
+	if (args) {
+		kvlist = rte_kvargs_parse(args,	compressdev_pmd_valid_params);
+		if (kvlist == NULL)
+			return -EINVAL;
+
+		ret = rte_kvargs_process(kvlist,
+				RTE_COMPRESSDEV_PMD_SOCKET_ID_ARG,
+				&rte_compressdev_pmd_parse_uint_arg,
+				&params->socket_id);
+		if (ret < 0)
+			goto free_kvlist;
+
+		ret = rte_kvargs_process(kvlist,
+				RTE_COMPRESSDEV_PMD_NAME_ARG,
+				&rte_compressdev_pmd_parse_name_arg,
+				params);
+		if (ret < 0)
+			goto free_kvlist;
+	}
+
+free_kvlist:
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_create(const char *name,
+		struct rte_device *device,
+		struct rte_compressdev_pmd_init_params *params)
+{
+	struct rte_compressdev *compressdev;
+
+	if (params->name[0] != '\0') {
+		COMPRESSDEV_LOG(INFO, "[%s] User specified device name = %s\n",
+				device->driver->name, params->name);
+		name = params->name;
+	}
+
+	COMPRESSDEV_LOG(INFO, "[%s] - Creating compressdev %s\n",
+			device->driver->name, name);
+
+	COMPRESSDEV_LOG(INFO,
+	"[%s] - Init parameters - name: %s, socket id: %d",
+			device->driver->name, name,
+			params->socket_id);
+
+	/* allocate device structure */
+	compressdev = rte_compressdev_pmd_allocate(name, params->socket_id);
+	if (compressdev == NULL) {
+		COMPRESSDEV_LOG(ERR, "[%s] Failed to allocate comp device %s",
+				device->driver->name, name);
+		return NULL;
+	}
+
+	/* allocate private device structure */
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		compressdev->data->dev_private =
+				rte_zmalloc_socket("compressdev device private",
+						params->private_data_size,
+						RTE_CACHE_LINE_SIZE,
+						params->socket_id);
+
+		if (compressdev->data->dev_private == NULL) {
+			COMPRESSDEV_LOG(ERR,
+		"[%s] Cannot allocate memory for compressdev %s private data",
+					device->driver->name, name);
+
+			rte_compressdev_pmd_release_device(compressdev);
+			return NULL;
+		}
+	}
+
+	compressdev->device = device;
+
+	return compressdev;
+}
+
+int __rte_experimental
+rte_compressdev_pmd_destroy(struct rte_compressdev *compressdev)
+{
+	int retval;
+
+	COMPRESSDEV_LOG(INFO, "[%s] Closing comp device %s",
+			compressdev->device->driver->name,
+			compressdev->device->name);
+
+	/* free comp device */
+	retval = rte_compressdev_pmd_release_device(compressdev);
+	if (retval)
+		return retval;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		rte_free(compressdev->data->dev_private);
+
+	compressdev->device = NULL;
+	compressdev->data = NULL;
+
+	return 0;
+}
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
new file mode 100644
index 000000000..1b49b48a6
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -0,0 +1,283 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMPRESSDEV_PMD_H_
+#define _RTE_COMPRESSDEV_PMD_H_
+
+/** @file
+ * RTE comp PMD APIs
+ *
+ * @note
+ * These APIs are for comp PMDs only and user applications should not call
+ * them directly.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <string.h>
+
+#include <rte_dev.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_mempool.h>
+#include <rte_log.h>
+#include <rte_common.h>
+
+#include "rte_compressdev.h"
+
+#define RTE_COMPRESSDEV_PMD_NAME_ARG			("name")
+#define RTE_COMPRESSDEV_PMD_SOCKET_ID_ARG		("socket_id")
+
+static const char * const compressdev_pmd_valid_params[] = {
+	RTE_COMPRESSDEV_PMD_NAME_ARG,
+	RTE_COMPRESSDEV_PMD_SOCKET_ID_ARG
+};
+
+/**
+ * @internal
+ * Initialisation parameters for comp devices
+ */
+struct rte_compressdev_pmd_init_params {
+	char name[RTE_COMPRESSDEV_NAME_MAX_LEN];
+	size_t private_data_size;
+	int socket_id;
+};
+
+/** Global structure used for maintaining state of allocated comp devices */
+struct rte_compressdev_global {
+	struct rte_compressdev *devs;	/**< Device information array */
+	struct rte_compressdev_data *data[RTE_COMPRESS_MAX_DEVS];
+	/**< Device private data */
+	uint8_t nb_devs;		/**< Number of devices found */
+	uint8_t max_devs;		/**< Max number of devices */
+};
+
+/* compressdev driver, containing the driver identifier */
+struct compressdev_driver {
+	TAILQ_ENTRY(compressdev_driver) next; /**< Next in list. */
+	const struct rte_driver *driver;
+	uint8_t id;
+};
+
+/** Pointer to global array of comp devices */
+extern struct rte_compressdev *rte_compressdevs;
+/** Pointer to global comp devices data structure */
+extern struct rte_compressdev_global *rte_compressdev_globals;
+
+/**
+ * Get the rte_compressdev structure device pointer for the device. Assumes a
+ * valid device index.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - The rte_compressdev structure pointer for the given device identifier.
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_get_dev(uint8_t dev_id);
+
+/**
+ * Get the rte_compressdev structure device pointer for the named device.
+ *
+ * @param name
+ *   Compress device name
+ * @return
+ *   - The rte_compressdev structure pointer for the given device identifier.
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_get_named_dev(const char *name);
+
+
+/**
+ * Definitions of all functions exported by a driver through the
+ * the generic structure of type *comp_dev_ops* supplied in the
+ * *rte_compressdev* structure associated with a device.
+ */
+
+/**
+ * Function used to configure device.
+ *
+ * @param dev
+ *   Compress device
+ * @param config
+ *   Compress device configurations
+ * @return
+ *   Returns 0 on success
+ */
+typedef int (*compressdev_configure_t)(struct rte_compressdev *dev,
+		struct rte_compressdev_config *config);
+
+/**
+ * Function used to start a configured device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *   Returns 0 on success
+ */
+typedef int (*compressdev_start_t)(struct rte_compressdev *dev);
+
+/**
+ * Function used to stop a configured device.
+ *
+ * @param dev
+ *   Compress device
+ */
+typedef void (*compressdev_stop_t)(struct rte_compressdev *dev);
+
+/**
+ * Function used to close a configured device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ * - 0 on success.
+ * - EAGAIN if can't close as device is busy
+ */
+typedef int (*compressdev_close_t)(struct rte_compressdev *dev);
+
+
+/**
+ * Function used to get specific information of a device.
+ *
+ * @param dev
+ *   Compress device
+ */
+typedef void (*compressdev_info_get_t)(struct rte_compressdev *dev,
+				struct rte_compressdev_info *dev_info);
+
+/** comp device operations function pointer table */
+struct rte_compressdev_ops {
+	compressdev_configure_t dev_configure;	/**< Configure device. */
+	compressdev_start_t dev_start;		/**< Start device. */
+	compressdev_stop_t dev_stop;		/**< Stop device. */
+	compressdev_close_t dev_close;		/**< Close device. */
+
+	compressdev_info_get_t dev_infos_get;	/**< Get device info. */
+};
+
+/**
+ * @internal
+ *
+ * Function for internal use by dummy drivers primarily, e.g. ring-based
+ * driver.
+ * Allocates a new compressdev slot for an comp device and returns the pointer
+ * to that slot for the driver to use.
+ *
+ * @param name
+ *   Unique identifier name for each device
+ * @param socket_id
+ *   Socket to allocate resources on
+ * @return
+ *   - Slot in the rte_dev_devices array for a new device;
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_allocate(const char *name, int socket_id);
+
+/**
+ * @internal
+ *
+ * Function for internal use by dummy drivers primarily, e.g. ring-based
+ * driver.
+ * Release the specified compressdev device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *   - 0 on success, negative on error
+ */
+int __rte_experimental
+rte_compressdev_pmd_release_device(struct rte_compressdev *dev);
+
+
+/**
+ * @internal
+ *
+ * PMD assist function to parse initialisation arguments for comp driver
+ * when creating a new comp PMD device instance.
+ *
+ * PMD driver should set default values for that PMD before calling function,
+ * these default values will be over-written with successfully parsed values
+ * from args string.
+ *
+ * @param params
+ *   Parsed PMD initialisation parameters
+ * @param args
+ *   Input argument string to parse
+ * @return
+ *  - 0 on success
+ *  - errno on failure
+ */
+int __rte_experimental
+rte_compressdev_pmd_parse_input_args(
+		struct rte_compressdev_pmd_init_params *params,
+		const char *args);
+
+/**
+ * @internal
+ *
+ * PMD assist function to provide boiler plate code for comp driver to create
+ * and allocate resources for a new comp PMD device instance.
+ *
+ * @param name
+ *   Compress device name
+ * @param device
+ *   Base device instance
+ * @param params
+ *   PMD initialisation parameters
+ * @return
+ *  - comp device instance on success
+ *  - NULL on creation failure
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_create(const char *name,
+		struct rte_device *device,
+		struct rte_compressdev_pmd_init_params *params);
+
+/**
+ * @internal
+ *
+ * PMD assist function to provide boiler plate code for comp driver to
+ * destroy and free resources associated with a comp PMD device instance.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *  - 0 on success
+ *  - errno on failure
+ */
+int __rte_experimental
+rte_compressdev_pmd_destroy(struct rte_compressdev *dev);
+
+
+/**
+ * @internal
+ * Allocate compressdev driver.
+ *
+ * @param comp_drv
+ *   Compressdev driver
+ * @param drv
+ *   Rte_driver
+ * @return
+ *  The driver type identifier
+ */
+uint8_t __rte_experimental
+rte_compressdev_allocate_driver(struct compressdev_driver *comp_drv,
+		const struct rte_driver *drv);
+
+
+#define RTE_PMD_REGISTER_COMPRESSDEV_DRIVER(comp_drv, drv, driver_id)\
+RTE_INIT(init_ ##driver_id);\
+static void init_ ##driver_id(void)\
+{\
+	driver_id = rte_compressdev_allocate_driver(&comp_drv, &(drv).driver);\
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_COMPRESSDEV_PMD_H_ */
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
new file mode 100644
index 000000000..11bdc1317
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -0,0 +1,31 @@
+EXPERIMENTAL {
+        global:
+
+	rte_compressdevs;
+	rte_compressdev_allocate_driver;
+	rte_compressdev_callback_register;
+	rte_compressdev_callback_unregister;
+	rte_compressdev_close;
+	rte_compressdev_configure;
+	rte_compressdev_count;
+	rte_compressdev_device_count_by_driver;
+	rte_compressdev_devices_get;
+	rte_compressdev_driver_id_get;
+	rte_compressdev_driver_name_get;
+	rte_compressdev_info_get;
+	rte_compressdev_is_valid_dev;
+	rte_compressdev_name_get;
+	rte_compressdev_pmd_allocate;
+	rte_compressdev_pmd_callback_process;
+	rte_compressdev_pmd_create;
+	rte_compressdev_pmd_destroy;
+	rte_compressdev_pmd_get_dev;
+	rte_compressdev_pmd_get_named_dev;
+	rte_compressdev_pmd_parse_input_args;
+	rte_compressdev_pmd_release_device;
+	rte_compressdev_socket_id;
+	rte_compressdev_start;
+	rte_compressdev_stop;
+
+        local: *;
+};
diff --git a/lib/meson.build b/lib/meson.build
index ef6159170..ae718aa3f 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -14,7 +14,7 @@ libraries = [ 'compat', # just a header, used for versioning
 	'hash',    # efd depends on this
 	'kvargs',  # cryptodev depends on this
 	'acl', 'bbdev', 'bitratestats', 'cfgfile',
-	'cmdline', 'cryptodev',
+	'cmdline', 'compressdev', 'cryptodev',
 	'distributor', 'efd', 'eventdev',
 	'gro', 'gso', 'ip_frag', 'jobstats',
 	'kni', 'latencystats', 'lpm', 'member',
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index afd0aff8c..db89cbc16 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -94,6 +94,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF)           += -lrte_mbuf
 _LDLIBS-$(CONFIG_RTE_LIBRTE_NET)            += -lrte_net
 _LDLIBS-$(CONFIG_RTE_LIBRTE_ETHER)          += -lrte_ethdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_BBDEV)          += -lrte_bbdev
+_LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV)    += -lrte_compressdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_CRYPTODEV)      += -lrte_cryptodev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY)       += -lrte_security
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV)       += -lrte_eventdev
-- 
2.14.3

[dpdk-dev] [PATCH v4 02/13] compressdev: add queue pair management

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Add functions to manage device queue pairs.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 135 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           |  50 ++++++++
 lib/librte_compressdev/rte_compressdev_pmd.h       |  47 +++++++
 lib/librte_compressdev/rte_compressdev_version.map |   2 +
 4 files changed, 234 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index d635953b2..c90e4beaf 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -266,10 +266,100 @@ rte_compressdev_pmd_release_device(struct rte_compressdev *compressdev)
 	return 0;
 }
 
+uint16_t __rte_experimental
+rte_compressdev_queue_pair_count(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	dev = &rte_comp_devices[dev_id];
+	return dev->data->nb_queue_pairs;
+}
+
+static int
+rte_compressdev_queue_pairs_config(struct rte_compressdev *dev,
+		uint16_t nb_qpairs, int socket_id)
+{
+	struct rte_compressdev_info dev_info;
+	void **qp;
+	unsigned int i;
+
+	if ((dev == NULL) || (nb_qpairs < 1)) {
+		COMPRESSDEV_LOG(ERR, "invalid param: dev %p, nb_queues %u",
+							dev, nb_qpairs);
+		return -EINVAL;
+	}
+
+	COMPRESSDEV_LOG(DEBUG, "Setup %d queues pairs on device %u",
+			nb_qpairs, dev->data->dev_id);
+
+	memset(&dev_info, 0, sizeof(struct rte_compressdev_info));
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_infos_get, -ENOTSUP);
+	(*dev->dev_ops->dev_infos_get)(dev, &dev_info);
+
+	if ((dev_info.max_nb_queue_pairs != 0) &&
+			(nb_qpairs > dev_info.max_nb_queue_pairs)) {
+		COMPRESSDEV_LOG(ERR, "Invalid num queue_pairs (%u) for dev %u",
+				nb_qpairs, dev->data->dev_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->queue_pairs == NULL) { /* first time configuration */
+		dev->data->queue_pairs = rte_zmalloc_socket(
+				"compressdev->queue_pairs",
+				sizeof(dev->data->queue_pairs[0]) * nb_qpairs,
+				RTE_CACHE_LINE_SIZE, socket_id);
+
+		if (dev->data->queue_pairs == NULL) {
+			dev->data->nb_queue_pairs = 0;
+			COMPRESSDEV_LOG(ERR,
+			"failed to get memory for qp meta data, nb_queues %u",
+							nb_qpairs);
+			return -(ENOMEM);
+		}
+	} else { /* re-configure */
+		int ret;
+		uint16_t old_nb_queues = dev->data->nb_queue_pairs;
+
+		qp = dev->data->queue_pairs;
+
+		RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->queue_pair_release,
+				-ENOTSUP);
+
+		for (i = nb_qpairs; i < old_nb_queues; i++) {
+			ret = (*dev->dev_ops->queue_pair_release)(dev, i);
+			if (ret < 0)
+				return ret;
+		}
+
+		qp = rte_realloc(qp, sizeof(qp[0]) * nb_qpairs,
+				RTE_CACHE_LINE_SIZE);
+		if (qp == NULL) {
+			COMPRESSDEV_LOG(ERR,
+			"failed to realloc qp meta data, nb_queues %u",
+						nb_qpairs);
+			return -(ENOMEM);
+		}
+
+		if (nb_qpairs > old_nb_queues) {
+			uint16_t new_qs = nb_qpairs - old_nb_queues;
+
+			memset(qp + old_nb_queues, 0,
+				sizeof(qp[0]) * new_qs);
+		}
+
+		dev->data->queue_pairs = qp;
+
+	}
+	dev->data->nb_queue_pairs = nb_qpairs;
+	return 0;
+}
+
 int __rte_experimental
 rte_compressdev_configure(uint8_t dev_id, struct rte_compressdev_config *config)
 {
 	struct rte_compressdev *dev;
+	int diag;
 
 	if (!rte_compressdev_is_valid_dev(dev_id)) {
 		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
@@ -286,6 +376,16 @@ rte_compressdev_configure(uint8_t dev_id, struct rte_compressdev_config *config)
 
 	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_configure, -ENOTSUP);
 
+	/* Setup new number of queue pairs and reconfigure device. */
+	diag = rte_compressdev_queue_pairs_config(dev, config->nb_queue_pairs,
+			config->socket_id);
+	if (diag != 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev%d rte_comp_dev_queue_pairs_config = %d",
+				dev_id, diag);
+		return diag;
+	}
+
 	return (*dev->dev_ops->dev_configure)(dev, config);
 }
 
@@ -375,6 +475,41 @@ rte_compressdev_close(uint8_t dev_id)
 	return 0;
 }
 
+int __rte_experimental
+rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
+		uint32_t max_inflight_ops, int socket_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+	if (queue_pair_id >= dev->data->nb_queue_pairs) {
+		COMPRESSDEV_LOG(ERR, "Invalid queue_pair_id=%d", queue_pair_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->dev_started) {
+		COMPRESSDEV_LOG(ERR,
+		    "device %d must be stopped to allow configuration", dev_id);
+		return -EBUSY;
+	}
+
+	if (max_inflight_ops == 0) {
+		COMPRESSDEV_LOG(ERR,
+			"Invalid maximum number of inflight operations");
+		return -EINVAL;
+	}
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->queue_pair_setup, -ENOTSUP);
+
+	return (*dev->dev_ops->queue_pair_setup)(dev, queue_pair_id,
+			max_inflight_ops, socket_id);
+}
+
 void __rte_experimental
 rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index c5de61378..883a5c273 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -38,6 +38,10 @@ extern int compressdev_logtype;
 struct rte_compressdev_info {
 	const char *driver_name;		/**< Driver name. */
 	uint8_t driver_id;			/**< Driver identifier */
+	uint16_t max_nb_queue_pairs;
+	/**< Maximum number of queues pairs supported by device.
+	 * (If 0, there is no limit in maximum number of queue pairs)
+	 */
 };
 
 /**
@@ -118,6 +122,9 @@ rte_compressdev_socket_id(uint8_t dev_id);
 /** Compress device configuration structure */
 struct rte_compressdev_config {
 	int socket_id;
+	/**< Socket on which to allocate resources */
+	uint16_t nb_queue_pairs;
+	/**< Total number of queue pairs to configure on a device */
 };
 
 /**
@@ -179,6 +186,44 @@ rte_compressdev_stop(uint8_t dev_id);
 int __rte_experimental
 rte_compressdev_close(uint8_t dev_id);
 
+/**
+ * Allocate and set up a receive queue pair for a device.
+ * This should only be called when the device is stopped.
+ *
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param queue_pair_id
+ *   The index of the queue pairs to set up. The
+ *   value must be in the range [0, nb_queue_pair - 1]
+ *   previously supplied to rte_compressdev_configure()
+ * @param max_inflight_ops
+ *   Max number of ops which the qp will have to
+ *   accommodate simultaneously
+ * @param socket_id
+ *   The *socket_id* argument is the socket identifier
+ *   in case of NUMA. The value can be *SOCKET_ID_ANY*
+ *   if there is no NUMA constraint for the DMA memory
+ *   allocated for the receive queue pair
+ * @return
+ *   - 0: Success, queue pair correctly set up.
+ *   - <0: Queue pair configuration failed
+ */
+int __rte_experimental
+rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
+		uint32_t max_inflight_ops, int socket_id);
+
+/**
+ * Get the number of queue pairs on a specific comp device
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - The number of configured queue pairs.
+ */
+uint16_t __rte_experimental
+rte_compressdev_queue_pair_count(uint8_t dev_id);
+
 /**
  * Retrieve the contextual information of a device.
  *
@@ -232,6 +277,11 @@ struct rte_compressdev_data {
 	uint8_t dev_started : 1;
 	/**< Device state: STARTED(1)/STOPPED(0) */
 
+	void **queue_pairs;
+	/**< Array of pointers to queue pairs. */
+	uint16_t nb_queue_pairs;
+	/**< Number of device queue pairs */
+
 	void *dev_private;
 	/**< PMD-specific private data */
 } __rte_cache_aligned;
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
index 1b49b48a6..b71391659 100644
--- a/lib/librte_compressdev/rte_compressdev_pmd.h
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -149,6 +149,48 @@ typedef int (*compressdev_close_t)(struct rte_compressdev *dev);
 typedef void (*compressdev_info_get_t)(struct rte_compressdev *dev,
 				struct rte_compressdev_info *dev_info);
 
+/**
+ * Setup a queue pair for a device.
+ *
+ * @param dev
+ *   Compress device
+ * @param qp_id
+ *   Queue pair identifier
+ * @param max_inflight_ops
+ *   Max inflight ops which qp must accommodate
+ * @param socket_id
+ *   Socket identifier
+ * @return
+ *   Returns 0 on success.
+ */
+typedef int (*compressdev_queue_pair_setup_t)(struct rte_compressdev *dev,
+		uint16_t qp_id,	uint32_t max_inflight_ops, int socket_id);
+
+/**
+ * Release memory resources allocated by given queue pair.
+ *
+ * @param dev
+ *   Compress device
+ * @param qp_id
+ *   Queue pair identifier
+ * @return
+ * - 0 on success.
+ * - EAGAIN if can't close as device is busy
+ */
+typedef int (*compressdev_queue_pair_release_t)(struct rte_compressdev *dev,
+		uint16_t qp_id);
+
+/**
+ * Get number of available queue pairs of a device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *   Returns number of queue pairs on success.
+ */
+typedef uint32_t (*compressdev_queue_pair_count_t)(struct rte_compressdev *dev);
+
+
 /** comp device operations function pointer table */
 struct rte_compressdev_ops {
 	compressdev_configure_t dev_configure;	/**< Configure device. */
@@ -157,6 +199,11 @@ struct rte_compressdev_ops {
 	compressdev_close_t dev_close;		/**< Close device. */
 
 	compressdev_info_get_t dev_infos_get;	/**< Get device info. */
+
+	compressdev_queue_pair_setup_t queue_pair_setup;
+	/**< Set up a device queue pair. */
+	compressdev_queue_pair_release_t queue_pair_release;
+	/**< Release a queue pair. */
 };
 
 /**
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 11bdc1317..78ac8d904 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -23,6 +23,8 @@ EXPERIMENTAL {
 	rte_compressdev_pmd_get_named_dev;
 	rte_compressdev_pmd_parse_input_args;
 	rte_compressdev_pmd_release_device;
+	rte_compressdev_queue_pair_count;
+	rte_compressdev_queue_pair_setup;
 	rte_compressdev_socket_id;
 	rte_compressdev_start;
 	rte_compressdev_stop;
-- 
2.14.3

[dpdk-dev] [PATCH v4 03/13] compressdev: add compression specific data

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added structures and enums specific to compression,
including the compression operation structure and the
different supported algorithms, checksums and compression
levels.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/Makefile    |   1 +
 lib/librte_compressdev/meson.build |   3 +-
 lib/librte_compressdev/rte_comp.h  | 312 +++++++++++++++++++++++++++++++++++++
 3 files changed, 315 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_compressdev/rte_comp.h

diff --git a/lib/librte_compressdev/Makefile b/lib/librte_compressdev/Makefile
index 5f67ab817..6f1546afd 100644
--- a/lib/librte_compressdev/Makefile
+++ b/lib/librte_compressdev/Makefile
@@ -19,6 +19,7 @@ LDLIBS += -lrte_eal -lrte_mempool -lrte_kvargs
 SRCS-y += rte_compressdev.c rte_compressdev_pmd.c
 
 # export include files
+SYMLINK-y-include += rte_comp.h
 SYMLINK-y-include += rte_compressdev.h
 SYMLINK-y-include += rte_compressdev_pmd.h
 
diff --git a/lib/librte_compressdev/meson.build b/lib/librte_compressdev/meson.build
index fc5eaf009..a72d4cea5 100644
--- a/lib/librte_compressdev/meson.build
+++ b/lib/librte_compressdev/meson.build
@@ -4,5 +4,6 @@
 allow_experimental_apis = true
 sources = files('rte_compressdev.c', 'rte_compressdev_pmd.c')
 headers = files('rte_compressdev.h',
-	'rte_compressdev_pmd.h')
+	'rte_compressdev_pmd.h',
+	'rte_comp.h')
 deps += ['kvargs', 'mbuf']
diff --git a/lib/librte_compressdev/rte_comp.h b/lib/librte_compressdev/rte_comp.h
new file mode 100644
index 000000000..cf0f3c999
--- /dev/null
+++ b/lib/librte_compressdev/rte_comp.h
@@ -0,0 +1,312 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMP_H_
+#define _RTE_COMP_H_
+
+/**
+ * @file rte_comp.h
+ *
+ * RTE definitions for Data Compression Service
+ *
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <string.h>
+#include <rte_mempool.h>
+
+/** Status of comp operation */
+enum rte_comp_op_status {
+	RTE_COMP_OP_STATUS_SUCCESS = 0,
+	/**< Operation completed successfully */
+	RTE_COMP_OP_STATUS_NOT_PROCESSED,
+	/**< Operation has not yet been processed by the device */
+	RTE_COMP_OP_STATUS_INVALID_ARGS,
+	/**< Operation failed due to invalid arguments in request */
+	RTE_COMP_OP_STATUS_ERROR,
+	/**< Error handling operation */
+	RTE_COMP_OP_STATUS_INVALID_STATE,
+	/**< Operation is invoked in invalid state */
+	RTE_COMP_OP_STATUS_OUT_OF_SPACE,
+	/**< Output buffer ran out of space before operation completed */
+};
+
+/** Compression Algorithms */
+enum rte_comp_algorithm {
+	RTE_COMP_ALGO_UNSPECIFIED = 0,
+	/** No Compression algorithm */
+	RTE_COMP_ALGO_NULL,
+	/**< No compression.
+	 * Pass-through, data is copied unchanged from source buffer to
+	 * destination buffer.
+	 */
+	RTE_COMP_ALGO_DEFLATE,
+	/**< DEFLATE compression algorithm
+	 * https://tools.ietf.org/html/rfc1951
+	 */
+	RTE_COMP_ALGO_LZS,
+	/**< LZS compression algorithm
+	 * https://tools.ietf.org/html/rfc2395
+	 */
+	RTE_COMP_ALGO_LIST_END
+};
+
+/**< Compression Level.
+ * The number is interpreted by each PMD differently. However, lower numbers
+ * give fastest compression, at the expense of compression ratio while
+ * higher numbers may give better compression ratios but are likely slower.
+ */
+#define	RTE_COMP_LEVEL_PMD_DEFAULT	(-1)
+/** Use PMD Default */
+#define	RTE_COMP_LEVEL_NONE		(0)
+/** Output uncompressed blocks if supported by the specified algorithm */
+#define RTE_COMP_LEVEL_MIN		(1)
+/** Use minimum compression level supported by the PMD */
+#define RTE_COMP_LEVEL_MAX		(9)
+/** Use maximum compression level supported by the PMD */
+
+/** Compression checksum types */
+enum rte_comp_checksum_type {
+	RTE_COMP_CHECKSUM_NONE,
+	/**< No checksum generated */
+	RTE_COMP_CHECKSUM_CRC32,
+	/**< Generates a CRC32 checksum, as used by gzip */
+	RTE_COMP_CHECKSUM_ADLER32,
+	/**< Generates an Adler-32 checksum, as used by zlib */
+	RTE_COMP_CHECKSUM_CRC32_ADLER32,
+	/**< Generates both Adler-32 and CRC32 checksums, concatenated.
+	 * CRC32 is in the lower 32bits, Adler-32 in the upper 32 bits.
+	 */
+};
+
+
+/** Compression Huffman Type - used by DEFLATE algorithm */
+enum rte_comp_huffman {
+	RTE_COMP_HUFFMAN_DEFAULT,
+	/**< PMD may choose which Huffman codes to use */
+	RTE_COMP_HUFFMAN_FIXED,
+	/**< Use Fixed Huffman codes */
+	RTE_COMP_HUFFMAN_DYNAMIC,
+	/**< Use Dynamic Huffman codes */
+};
+
+enum rte_comp_flush_flag {
+	RTE_COMP_FLUSH_NONE,
+	/**< Data is not flushed. Output may remain in the compressor and be
+	 * processed during a following op. It may not be possible to decompress
+	 * output until a later op with some other flush flag has been sent.
+	 */
+	RTE_COMP_FLUSH_SYNC,
+	/**< All data should be flushed to output buffer. Output data can be
+	 * decompressed. However state and history is not cleared, so future
+	 * operations may use history from this operation.
+	 */
+	RTE_COMP_FLUSH_FULL,
+	/**< All data should be flushed to output buffer. Output data can be
+	 * decompressed. State and history data is cleared, so future
+	 * ops will be independent of ops processed before this.
+	 */
+	RTE_COMP_FLUSH_FINAL
+	/**< Same as RTE_COMP_FLUSH_FULL but if op.algo is RTE_COMP_ALGO_DEFLATE
+	 * then bfinal bit is set in the last block.
+	 */
+};
+
+/** Compression transform types */
+enum rte_comp_xform_type {
+	RTE_COMP_COMPRESS,
+	/**< Compression service - compress */
+	RTE_COMP_DECOMPRESS,
+	/**< Compression service - decompress */
+};
+
+enum rte_comp_op_type {
+	RTE_COMP_OP_STATELESS,
+	/**< All data to be processed is submitted in the op, no state or
+	 * history from previous ops is used and none will be stored for future
+	 * ops. Flush flag must be set to either FLUSH_FULL or FLUSH_FINAL.
+	 */
+	RTE_COMP_OP_STATEFUL
+	/**< There may be more data to be processed after this op, it's part of
+	 * a stream of data. State and history from previous ops can be used
+	 * and resulting state and history can be stored for future ops,
+	 * depending on flush flag.
+	 */
+};
+
+
+/** Parameters specific to the deflate algorithm */
+struct rte_comp_deflate_params {
+	enum rte_comp_huffman huffman;
+	/**< Compression huffman encoding type */
+};
+
+/** Setup Data for compression */
+struct rte_comp_compress_xform {
+	enum rte_comp_algorithm algo;
+	/**< Algorithm to use for compress operation */
+	union {
+		struct rte_comp_deflate_params deflate;
+		/**< Parameters specific to the deflate algorithm */
+	}; /**< Algorithm specific parameters */
+	int level;
+	/**< Compression level */
+	uint8_t window_size;
+	/**< Base two log value of sliding window to be used. If window size
+	 * can't be supported by the PMD then it may fall back to a smaller
+	 * size. This is likely to result in a worse compression ratio.
+	 */
+	enum rte_comp_checksum_type chksum;
+	/**< Type of checksum to generate on the uncompressed data */
+};
+
+/**
+ * Setup Data for decompression.
+ */
+struct rte_comp_decompress_xform {
+	enum rte_comp_algorithm algo;
+	/**< Algorithm to use for decompression */
+	enum rte_comp_checksum_type chksum;
+	/**< Type of checksum to generate on the decompressed data */
+	uint8_t window_size;
+	/**< Base two log value of sliding window which was used to generate
+	 * compressed data. If window size can't be supported by the PMD then
+	 * setup of stream or private_xform should fail.
+	 */
+};
+
+/**
+ * Compression transform structure.
+ *
+ * This is used to specify the compression transforms required.
+ * Each transform structure can hold a single transform, the type field is
+ * used to specify which transform is contained within the union.
+ * There are no chain cases currently supported, just single xforms of
+ *  - compress-only
+ *  - decompress-only
+ *
+ */
+struct rte_comp_xform {
+	struct rte_comp_xform *next;
+	/**< next xform in chain */
+	enum rte_comp_xform_type type;
+	/**< xform type */
+	union {
+		struct rte_comp_compress_xform compress;
+		/**< xform for compress operation */
+		struct rte_comp_decompress_xform decompress;
+		/**< decompress xform */
+	};
+};
+
+/**
+ * Compression Operation.
+ *
+ * This structure contains data relating to performing a compression
+ * operation on the referenced mbuf data buffers.
+ *
+ * Comp operations are enqueued and dequeued in comp PMDs using the
+ * rte_compressdev_enqueue_burst() / rte_compressdev_dequeue_burst() APIs
+ */
+struct rte_comp_op {
+	enum rte_comp_op_type op_type;
+	union {
+		void *private_xform;
+		/**< Stateless private PMD data derived from an rte_comp_xform.
+		 * A handle returned by rte_compressdev_private_xform_create()
+		 * must be attached to operations of op_type RTE_COMP_STATELESS.
+		 */
+		void *stream;
+		/**< Private PMD data derived initially from an rte_comp_xform,
+		 * which holds state and history data and evolves as operations
+		 * are processed. rte_compressdev_stream_create() must be called
+		 * on a device for all STATEFUL data streams and the resulting
+		 * stream attached to the one or more operations associated
+		 * with the data stream.
+		 * All operations in a stream must be sent to the same device.
+		 */
+	};
+
+	struct rte_mempool *mempool;
+	/**< Pool from which operation is allocated */
+	rte_iova_t iova_addr;
+	/**< IOVA address of this operation */
+	struct rte_mbuf *m_src;
+	/**< source mbuf
+	 * The total size of the input buffer(s) can be retrieved using
+	 * rte_pktmbuf_data_len(m_src)
+	 */
+	struct rte_mbuf *m_dst;
+	/**< destination mbuf
+	 * The total size of the output buffer(s) can be retrieved using
+	 * rte_pktmbuf_data_len(m_dst)
+	 */
+
+	struct {
+		uint32_t offset;
+		/**< Starting point for compression or decompression,
+		 * specified as number of bytes from start of packet in
+		 * source buffer.
+		 * Starting point for checksum generation in compress direction.
+		 */
+		uint32_t length;
+		/**< The length, in bytes, of the data in source buffer
+		 * to be compressed or decompressed.
+		 * Also the length of the data over which the checksum
+		 * should be generated in compress direction
+		 */
+	} src;
+	struct {
+		uint32_t offset;
+		/**< Starting point for writing output data, specified as
+		 * number of bytes from start of packet in dest
+		 * buffer. Starting point for checksum generation in
+		 * decompress direction.
+		 */
+	} dst;
+	enum rte_comp_flush_flag flush_flag;
+	/**< Defines flush characteristics for the output data.
+	 * Only applicable in compress direction
+	 */
+	uint64_t input_chksum;
+	/**< An input checksum can be provided to generate a
+	 * cumulative checksum across sequential blocks in a STATELESS stream.
+	 * Checksum type is as specified in xform chksum_type
+	 */
+	uint64_t output_chksum;
+	/**< If a checksum is generated it will be written in here.
+	 * Checksum type is as specified in xform chksum_type.
+	 */
+	uint32_t consumed;
+	/**< The number of bytes from the source buffer
+	 * which were compressed/decompressed.
+	 */
+	uint32_t produced;
+	/**< The number of bytes written to the destination buffer
+	 * which were compressed/decompressed.
+	 */
+	uint64_t debug_status;
+	/**<
+	 * Status of the operation is returned in the status param.
+	 * This field allows the PMD to pass back extra
+	 * pmd-specific debug information. Value is not defined on the API.
+	 */
+	uint8_t status;
+	/**<
+	 * Operation status - use values from enum rte_comp_status.
+	 * This is reset to
+	 * RTE_COMP_OP_STATUS_NOT_PROCESSED on allocation from mempool and
+	 * will be set to RTE_COMP_OP_STATUS_SUCCESS after operation
+	 * is successfully processed by a PMD
+	 */
+} __rte_cache_aligned;
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_COMP_H_ */
-- 
2.14.3

[dpdk-dev] [PATCH v4 04/13] compressdev: add enqueue/dequeue functions

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.h | 121 +++++++++++++++++++++++++++++++
 1 file changed, 121 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 883a5c273..06f9ee135 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -241,8 +241,23 @@ rte_compressdev_queue_pair_count(uint8_t dev_id);
 void __rte_experimental
 rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info);
 
+
+typedef uint16_t (*compress_dequeue_pkt_burst_t)(void *qp,
+		struct rte_comp_op **ops, uint16_t nb_ops);
+/**< Dequeue processed packets from queue pair of a device. */
+
+typedef uint16_t (*compress_enqueue_pkt_burst_t)(void *qp,
+		struct rte_comp_op **ops, uint16_t nb_ops);
+/**< Enqueue packets for processing on queue pair of a device. */
+
+
 /** The data structure associated with each comp device. */
 struct rte_compressdev {
+	compress_dequeue_pkt_burst_t dequeue_burst;
+	/**< Pointer to PMD receive function */
+	compress_enqueue_pkt_burst_t enqueue_burst;
+	/**< Pointer to PMD transmit function */
+
 	struct rte_compressdev_data *data;
 	/**< Pointer to device data */
 	struct rte_compressdev_ops *dev_ops;
@@ -288,6 +303,112 @@ struct rte_compressdev_data {
 
 struct rte_compressdev *rte_compressdevs;
 
+/**
+ *
+ * Dequeue a burst of processed compression operations from a queue on the comp
+ * device. The dequeued operation are stored in *rte_comp_op* structures
+ * whose pointers are supplied in the *ops* array.
+ *
+ * The rte_compressdev_dequeue_burst() function returns the number of ops
+ * actually dequeued, which is the number of *rte_comp_op* data structures
+ * effectively supplied into the *ops* array.
+ *
+ * A return value equal to *nb_ops* indicates that the queue contained
+ * at least *nb_ops* operations, and this is likely to signify that other
+ * processed operations remain in the devices output queue. Applications
+ * implementing a "retrieve as many processed operations as possible" policy
+ * can check this specific case and keep invoking the
+ * rte_compressdev_dequeue_burst() function until a value less than
+ * *nb_ops* is returned.
+ *
+ * The rte_compressdev_dequeue_burst() function does not provide any error
+ * notification to avoid the corresponding overhead.
+ *
+ * Note: operation ordering is not maintained within the queue pair.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param qp_id
+ *   The index of the queue pair from which to retrieve
+ *   processed operations. The value must be in the range
+ *   [0, nb_queue_pair - 1] previously supplied to
+ *   rte_compressdev_configure()
+ * @param ops
+ *   The address of an array of pointers to
+ *   *rte_comp_op* structures that must be
+ *   large enough to store *nb_ops* pointers in it
+ * @param nb_ops
+ *   The maximum number of operations to dequeue
+ * @return
+ *   - The number of operations actually dequeued, which is the number
+ *   of pointers to *rte_comp_op* structures effectively supplied to the
+ *   *ops* array.
+ */
+static inline uint16_t
+rte_compressdev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	struct rte_compressdev *dev = &rte_compressdevs[dev_id];
+
+	nb_ops = (*dev->dequeue_burst)
+			(dev->data->queue_pairs[qp_id], ops, nb_ops);
+
+	return nb_ops;
+}
+
+/**
+ * Enqueue a burst of operations for processing on a compression device.
+ *
+ * The rte_compressdev_enqueue_burst() function is invoked to place
+ * comp operations on the queue *qp_id* of the device designated by
+ * its *dev_id*.
+ *
+ * The *nb_ops* parameter is the number of operations to process which are
+ * supplied in the *ops* array of *rte_comp_op* structures.
+ *
+ * The rte_compressdev_enqueue_burst() function returns the number of
+ * operations it actually enqueued for processing. A return value equal to
+ * *nb_ops* means that all packets have been enqueued.
+ *
+ * @note All compression operations are Out-of-place (OOP) operations,
+ * as the size of the output data is different to the size of the input data.
+ *
+ * @note The flush flag only applies to operations which return SUCCESS.
+ * In OUT_OF_SPACE case whether STATEFUL or STATELESS, data in dest buffer
+ * is as if flush flag was FLUSH_NONE.
+ * @note flush flag only applies in compression direction. It has no meaning
+ * for decompression.
+ * @note: operation ordering is not maintained within the queue pair.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param qp_id
+ *   The index of the queue pair on which operations
+ *   are to be enqueued for processing. The value
+ *   must be in the range [0, nb_queue_pairs - 1]
+ *   previously supplied to *rte_compressdev_configure*
+ * @param ops
+ *   The address of an array of *nb_ops* pointers
+ *   to *rte_comp_op* structures which contain
+ *   the operations to be processed
+ * @param nb_ops
+ *   The number of operations to process
+ * @return
+ *   The number of operations actually enqueued on the device. The return
+ *   value can be less than the value of the *nb_ops* parameter when the
+ *   comp devices queue is full or if invalid parameters are specified in
+ *   a *rte_comp_op*.
+ */
+static inline uint16_t
+rte_compressdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	struct rte_compressdev *dev = &rte_compressdevs[dev_id];
+
+	return (*dev->enqueue_burst)(
+			dev->data->queue_pairs[qp_id], ops, nb_ops);
+}
+
 /**
  * Provide driver identifier.
  *
-- 
2.14.3

[dpdk-dev] [PATCH v4 05/13] compressdev: add operation management

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added functions to allocate and free compression operations.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_comp.h                  | 195 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.c           |  72 ++++++++
 lib/librte_compressdev/rte_compressdev_version.map |   1 +
 3 files changed, 268 insertions(+)

diff --git a/lib/librte_compressdev/rte_comp.h b/lib/librte_compressdev/rte_comp.h
index cf0f3c999..4cf84c5db 100644
--- a/lib/librte_compressdev/rte_comp.h
+++ b/lib/librte_compressdev/rte_comp.h
@@ -305,6 +305,201 @@ struct rte_comp_op {
 	 */
 } __rte_cache_aligned;
 
+
+/**
+ * Reset the fields of an operation to their default values.
+ *
+ * @note The private data associated with the operation is not zeroed.
+ *
+ * @param op
+ *   The operation to be reset
+ */
+static inline void
+__rte_comp_op_reset(struct rte_comp_op *op)
+{
+	struct rte_mempool *tmp_mp = op->mempool;
+	rte_iova_t tmp_iova_addr = op->iova_addr;
+
+	memset(op, 0, sizeof(struct rte_comp_op));
+	op->status = RTE_COMP_OP_STATUS_NOT_PROCESSED;
+	op->iova_addr = tmp_iova_addr;
+	op->mempool = tmp_mp;
+}
+
+/**
+ * Private data structure belonging to an operation pool.
+ */
+struct rte_comp_op_pool_private {
+	uint16_t user_size;
+	/**< Size of private user data with each operation. */
+};
+
+
+/**
+ * Returns the size of private user data allocated with each object in
+ * the mempool
+ *
+ * @param mempool
+ *   Mempool for operations
+ * @return
+ *   user data size
+ */
+static inline uint16_t
+__rte_comp_op_get_user_data_size(struct rte_mempool *mempool)
+{
+	struct rte_comp_op_pool_private *priv =
+	    (struct rte_comp_op_pool_private *)rte_mempool_get_priv(mempool);
+
+	return priv->user_size;
+}
+
+
+/**
+ * Creates an operation pool
+ *
+ * @param name
+ *   Compress pool name
+ * @param nb_elts
+ *   Number of elements in pool
+ * @param cache_size
+ *   Number of elements to cache on lcore, see
+ *   *rte_mempool_create* for further details about cache size
+ * @param user_size
+ *   Size of private data to allocate for user with each operation
+ * @param socket_id
+ *   Socket to identifier allocate memory on
+ * @return
+ *  - On success pointer to mempool
+ *  - On failure NULL
+ */
+struct rte_mempool *
+rte_comp_op_pool_create(const char *name,
+		unsigned int nb_elts, unsigned int cache_size,
+		uint16_t user_size, int socket_id);
+
+/**
+ * Bulk allocate raw element from mempool and return as comp operations
+ *
+ * @param mempool
+ *   Compress operation mempool
+ * @param ops
+ *   Array to place allocated operations
+ * @param nb_ops
+ *   Number of operations to allocate
+ * @return
+ * - On success returns  number of ops allocated
+ */
+static inline int
+__rte_comp_op_raw_bulk_alloc(struct rte_mempool *mempool,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	if (rte_mempool_get_bulk(mempool, (void **)ops, nb_ops) == 0)
+		return nb_ops;
+
+	return 0;
+}
+
+/**
+ * Allocate an operation from a mempool with default parameters set
+ *
+ * @param mempool
+ *   Compress operation mempool
+ *
+ * @return
+ * - On success returns a valid rte_comp_op structure
+ * - On failure returns NULL
+ */
+static inline struct rte_comp_op *
+rte_comp_op_alloc(struct rte_mempool *mempool)
+{
+	struct rte_comp_op *op = NULL;
+	int retval;
+
+	retval = __rte_comp_op_raw_bulk_alloc(mempool, &op, 1);
+	if (unlikely(retval != 1))
+		return NULL;
+
+	__rte_comp_op_reset(op);
+
+	return op;
+}
+
+
+/**
+ * Bulk allocate operations from a mempool with default parameters set
+ *
+ * @param mempool
+ *   Compress operation mempool
+ * @param ops
+ *   Array to place allocated operations
+ * @param nb_ops
+ *   Number of operations to allocate
+ * @return
+ * - nb_ops if the number of operations requested were allocated.
+ * - 0 if the requested number of ops are not available.
+ *   None are allocated in this case.
+ */
+static inline unsigned
+rte_comp_op_bulk_alloc(struct rte_mempool *mempool,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	int i;
+
+	if (unlikely(__rte_comp_op_raw_bulk_alloc(mempool, ops, nb_ops)
+			!= nb_ops))
+		return 0;
+
+	for (i = 0; i < nb_ops; i++)
+		__rte_comp_op_reset(ops[i]);
+
+	return nb_ops;
+}
+
+
+
+/**
+ * Returns a pointer to the private user data of an operation if
+ * that operation has enough capacity for requested size.
+ *
+ * @param op
+ *   Compress operation
+ * @param size
+ *   Size of space requested in private data
+ * @return
+ * - if sufficient space available returns pointer to start of user data
+ * - if insufficient space returns NULL
+ */
+static inline void *
+__rte_comp_op_get_user_data(struct rte_comp_op *op, uint32_t size)
+{
+	uint32_t user_size;
+
+	if (likely(op->mempool != NULL)) {
+		user_size = __rte_comp_op_get_user_data_size(op->mempool);
+
+		if (likely(user_size >= size))
+			return (void *)(op + 1);
+
+	}
+
+	return NULL;
+}
+
+/**
+ * free operation structure
+ * If operation has been allocate from a rte_mempool, then the operation will
+ * be returned to the mempool.
+ *
+ * @param op
+ *   Compress operation
+ */
+static inline void
+rte_comp_op_free(struct rte_comp_op *op)
+{
+	if (op != NULL && op->mempool != NULL)
+		rte_mempool_put(op->mempool, op);
+}
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index c90e4beaf..0dab92650 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -530,6 +530,78 @@ rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
 	dev_info->driver_name = dev->device->driver->name;
 }
 
+/** Initialise rte_comp_op mempool element */
+static void
+rte_comp_op_init(struct rte_mempool *mempool,
+		__rte_unused void *opaque_arg,
+		void *_op_data,
+		__rte_unused unsigned int i)
+{
+	struct rte_comp_op *op = _op_data;
+
+	memset(_op_data, 0, mempool->elt_size);
+
+	op->status = RTE_COMP_OP_STATUS_NOT_PROCESSED;
+	op->iova_addr = rte_mem_virt2iova(_op_data);
+	op->mempool = mempool;
+}
+
+
+struct rte_mempool * __rte_experimental
+rte_comp_op_pool_create(const char *name,
+		unsigned int nb_elts, unsigned int cache_size,
+		uint16_t user_size, int socket_id)
+{
+	struct rte_comp_op_pool_private *priv;
+
+	unsigned int elt_size = sizeof(struct rte_comp_op) + user_size;
+
+	/* lookup mempool in case already allocated */
+	struct rte_mempool *mp = rte_mempool_lookup(name);
+
+	if (mp != NULL) {
+		priv = (struct rte_comp_op_pool_private *)
+				rte_mempool_get_priv(mp);
+
+		if (mp->elt_size != elt_size ||
+				mp->cache_size < cache_size ||
+				mp->size < nb_elts ||
+				priv->user_size <  user_size) {
+			mp = NULL;
+			COMPRESSDEV_LOG(ERR,
+		"Mempool %s already exists but with incompatible parameters",
+					name);
+			return NULL;
+		}
+		return mp;
+	}
+
+	mp = rte_mempool_create(
+			name,
+			nb_elts,
+			elt_size,
+			cache_size,
+			sizeof(struct rte_comp_op_pool_private),
+			NULL,
+			NULL,
+			rte_comp_op_init,
+			NULL,
+			socket_id,
+			0);
+
+	if (mp == NULL) {
+		COMPRESSDEV_LOG(ERR, "Failed to create mempool %s", name);
+		return NULL;
+	}
+
+	priv = (struct rte_comp_op_pool_private *)
+			rte_mempool_get_priv(mp);
+
+	priv->user_size = user_size;
+
+	return mp;
+}
+
 TAILQ_HEAD(compressdev_driver_list, compressdev_driver);
 
 static struct compressdev_driver_list compressdev_driver_list =
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 78ac8d904..6a3596429 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -28,6 +28,7 @@ EXPERIMENTAL {
 	rte_compressdev_socket_id;
 	rte_compressdev_start;
 	rte_compressdev_stop;
+	rte_comp_op_pool_create;
 
         local: *;
 };
-- 
2.14.3

[dpdk-dev] [PATCH v4 06/13] compressdev: support stateless operations

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added private transform data (priv_xform) in compression
operation, which will contain the private data from each
PMD to support stateless operations.
Also, added functions to create/free this data.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 48 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 48 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 40 ++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  2 +
 4 files changed, 138 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 0dab92650..3843a6bbf 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -530,6 +530,54 @@ rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
 	dev_info->driver_name = dev->device->driver->name;
 }
 
+int __rte_experimental
+rte_compressdev_private_xform_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **priv_xform)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (xform == NULL || priv_xform == NULL || dev == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->private_xform_create, -ENOTSUP);
+	ret = (*dev->dev_ops->private_xform_create)(dev, xform, priv_xform);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to create private_xform: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
+int __rte_experimental
+rte_compressdev_private_xform_free(uint8_t dev_id, void *priv_xform)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (dev == NULL || priv_xform == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->private_xform_free, -ENOTSUP);
+	ret = dev->dev_ops->private_xform_free(dev, priv_xform);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to free private xform: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
 /** Initialise rte_comp_op mempool element */
 static void
 rte_comp_op_init(struct rte_mempool *mempool,
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 06f9ee135..917c0d764 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -125,6 +125,8 @@ struct rte_compressdev_config {
 	/**< Socket on which to allocate resources */
 	uint16_t nb_queue_pairs;
 	/**< Total number of queue pairs to configure on a device */
+	uint16_t max_nb_priv_xforms;
+	/**< Max number of private_xforms which will be created on the device */
 };
 
 /**
@@ -409,6 +411,52 @@ rte_compressdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 			dev->data->queue_pairs[qp_id], ops, nb_ops);
 }
 
+/**
+ * This should alloc a private_xform from the device's mempool and initialise
+ * it. The application should call this API when setting up for stateless
+ * processing on a device. If it returns non-shareable, then the appl cannot
+ * share this handle with multiple in-flight ops and should call this API again
+ * to get a separate handle for every in-flight op.
+ * The handle returned is only valid for use with ops of op_type STATELESS.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param xform
+ *   xform data
+ * @param private_xform
+ *   Pointer to where PMD's private_xform handle should be stored
+ *
+ * @return
+ *  - if successful returns 0
+ *    and valid private_xform handle
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private_xform could not be allocated.
+ */
+int __rte_experimental
+rte_compressdev_private_xform_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **private_xform);
+
+/**
+ * This should clear the private_xform and return it to the device's mempool.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ *
+ * @param private_xform
+ *   PMD's private_xform data
+ *
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -EBUSY if can't free private_xform due to inflight operations
+ */
+int __rte_experimental
+rte_compressdev_private_xform_free(uint8_t dev_id, void *private_xform);
+
 /**
  * Provide driver identifier.
  *
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
index b71391659..a8b868510 100644
--- a/lib/librte_compressdev/rte_compressdev_pmd.h
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -190,6 +190,41 @@ typedef int (*compressdev_queue_pair_release_t)(struct rte_compressdev *dev,
  */
 typedef uint32_t (*compressdev_queue_pair_count_t)(struct rte_compressdev *dev);
 
+/**
+ * Create driver private_xform data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param xform
+ *   xform data
+ * @param private_xform
+ *   ptr where handle of pmd's private_xform data should be stored
+ * @return
+ *  - if successful returns 0
+ *    and valid private_xform handle
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private_xform could not be allocated.
+ */
+typedef int (*compressdev_private_xform_create_t)(struct rte_compressdev *dev,
+		const struct rte_comp_xform *xform, void **private_xform);
+
+/**
+ * Free driver private_xform data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param private_xform
+ *   handle of pmd's private_xform data
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -EBUSY if can't free private_xform due to inflight operations
+ */
+typedef int (*compressdev_private_xform_free_t)(struct rte_compressdev *dev,
+		void *private_xform);
 
 /** comp device operations function pointer table */
 struct rte_compressdev_ops {
@@ -204,6 +239,11 @@ struct rte_compressdev_ops {
 	/**< Set up a device queue pair. */
 	compressdev_queue_pair_release_t queue_pair_release;
 	/**< Release a queue pair. */
+
+	compressdev_private_xform_create_t private_xform_create;
+	/**< Create a comp private_xform and initialise its private data. */
+	compressdev_private_xform_free_t private_xform_free;
+	/**< Free a comp private_xform's data. */
 };
 
 /**
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 6a3596429..6636decc2 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -23,6 +23,8 @@ EXPERIMENTAL {
 	rte_compressdev_pmd_get_named_dev;
 	rte_compressdev_pmd_parse_input_args;
 	rte_compressdev_pmd_release_device;
+	rte_compressdev_private_xform_create;
+	rte_compressdev_private_xform_free;
 	rte_compressdev_queue_pair_count;
 	rte_compressdev_queue_pair_setup;
 	rte_compressdev_socket_id;
-- 
2.14.3

[dpdk-dev] [PATCH v4 07/13] compressdev: support stateful operations

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added stream data (stream) in compression operation,
which will contain the private data from each PMD
to support stateful operations.
Also, added functions to create/free this data.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 49 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 50 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 41 ++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  2 +
 4 files changed, 142 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 3843a6bbf..433973aaa 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -578,6 +578,55 @@ rte_compressdev_private_xform_free(uint8_t dev_id, void *priv_xform)
 	return 0;
 }
 
+int __rte_experimental
+rte_compressdev_stream_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **stream)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (xform == NULL || dev == NULL || stream == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->stream_create, -ENOTSUP);
+	ret = (*dev->dev_ops->stream_create)(dev, xform, stream);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to create stream: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
+
+int __rte_experimental
+rte_compressdev_stream_free(uint8_t dev_id, void *stream)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (dev == NULL || stream == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->stream_free, -ENOTSUP);
+	ret = dev->dev_ops->stream_free(dev, stream);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to free stream: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
 /** Initialise rte_comp_op mempool element */
 static void
 rte_comp_op_init(struct rte_mempool *mempool,
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 917c0d764..0e148dd30 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -127,6 +127,8 @@ struct rte_compressdev_config {
 	/**< Total number of queue pairs to configure on a device */
 	uint16_t max_nb_priv_xforms;
 	/**< Max number of private_xforms which will be created on the device */
+	uint16_t max_nb_streams;
+	/**< Max number of streams which will be created on the device */
 };
 
 /**
@@ -411,6 +413,54 @@ rte_compressdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 			dev->data->queue_pairs[qp_id], ops, nb_ops);
 }
 
+/**
+ * This should alloc a stream from the device's mempool and initialise it.
+ * The application should call this API when setting up for the stateful
+ * processing of a set of data on a device. The API can be called multiple
+ * times to set up a stream for each data set. The handle returned is only for
+ * use with ops of op_type STATEFUL and must be passed to the PMD
+ * with every op in the data stream
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param xform
+ *   xform data
+ * @param stream
+ *   Pointer to where PMD's private stream handle should be stored
+ *
+ * @return
+ *  - 0 if successful and valid stream handle
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private stream could not be allocated.
+ *
+ */
+int __rte_experimental
+rte_compressdev_stream_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **stream);
+
+/**
+ * This should clear the stream and return it to the device's mempool.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ *
+ * @param stream
+ *   PMD's private stream data
+ *
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -EBUSY if can't free stream as there are inflight operations
+ */
+int __rte_experimental
+rte_compressdev_stream_free(uint8_t dev_id, void *stream);
+
 /**
  * This should alloc a private_xform from the device's mempool and initialise
  * it. The application should call this API when setting up for stateless
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
index a8b868510..59bcb688c 100644
--- a/lib/librte_compressdev/rte_compressdev_pmd.h
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -190,6 +190,42 @@ typedef int (*compressdev_queue_pair_release_t)(struct rte_compressdev *dev,
  */
 typedef uint32_t (*compressdev_queue_pair_count_t)(struct rte_compressdev *dev);
 
+/**
+ * Create driver private stream data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param xform
+ *   xform data
+ * @param stream
+ *   ptr where handle of pmd's private stream data should be stored
+ * @return
+ *  - Returns 0 if private stream structure has been created successfully.
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private stream could not be allocated.
+ */
+typedef int (*compressdev_stream_create_t)(struct rte_compressdev *dev,
+		const struct rte_comp_xform *xform, void **stream);
+
+/**
+ * Free driver private stream data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param stream
+ *   handle of pmd's private stream data
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -EBUSY if can't free stream as there are inflight operations
+ */
+typedef int (*compressdev_stream_free_t)(struct rte_compressdev *dev,
+		void *stream);
+
 /**
  * Create driver private_xform data.
  *
@@ -240,6 +276,11 @@ struct rte_compressdev_ops {
 	compressdev_queue_pair_release_t queue_pair_release;
 	/**< Release a queue pair. */
 
+	compressdev_stream_create_t stream_create;
+	/**< Create a comp stream and initialise its private data. */
+	compressdev_stream_free_t stream_free;
+	/**< Free a comp stream's private data. */
+
 	compressdev_private_xform_create_t private_xform_create;
 	/**< Create a comp private_xform and initialise its private data. */
 	compressdev_private_xform_free_t private_xform_free;
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 6636decc2..58cb5205a 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -30,6 +30,8 @@ EXPERIMENTAL {
 	rte_compressdev_socket_id;
 	rte_compressdev_start;
 	rte_compressdev_stop;
+	rte_compressdev_stream_create;
+	rte_compressdev_stream_free;
 	rte_comp_op_pool_create;
 
         local: *;
-- 
2.14.3

[dpdk-dev] [PATCH v4 08/13] compressdev: support hash operations

[Pablo de Lara]

From: Shally Verma <shally.verma@caviumnetworks.com>

- Added hash algo enumeration and params in xform and rte_comp_op
- Updated compress/decompress xform to input hash algorithm
- Updated struct rte_comp_op to input hash buffer

User in capability query will know about support hashes via
device info comp_feature_flag. If supported, application can initialize
desired algorithm enumeration in xform structure and pass valid hash
buffer during enqueue_burst().

Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Sunila Sahu <sunila.sahu@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_comp.h | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/lib/librte_compressdev/rte_comp.h b/lib/librte_compressdev/rte_comp.h
index 4cf84c5db..055f2ee75 100644
--- a/lib/librte_compressdev/rte_comp.h
+++ b/lib/librte_compressdev/rte_comp.h
@@ -55,6 +55,17 @@ enum rte_comp_algorithm {
 	RTE_COMP_ALGO_LIST_END
 };
 
+/** Compression Hash Algorithms */
+enum rte_comp_hash_algorithm {
+	RTE_COMP_HASH_ALGO_UNSPECIFIED = 0,
+	/**< No hash */
+	RTE_COMP_HASH_ALGO_SHA1,
+	/**< SHA1 hash algorithm */
+	RTE_COMP_HASH_ALGO_SHA2_256,
+	/**< SHA256 hash algorithm of SHA2 family */
+	RTE_COMP_HASH_ALGO_LIST_END
+};
+
 /**< Compression Level.
  * The number is interpreted by each PMD differently. However, lower numbers
  * give fastest compression, at the expense of compression ratio while
@@ -162,6 +173,10 @@ struct rte_comp_compress_xform {
 	 */
 	enum rte_comp_checksum_type chksum;
 	/**< Type of checksum to generate on the uncompressed data */
+	enum rte_comp_hash_algorithm hash_algo;
+	/**< Hash algorithm to be used with compress operation. Hash is always
+	 * done on plaintext.
+	 */
 };
 
 /**
@@ -177,6 +192,10 @@ struct rte_comp_decompress_xform {
 	 * compressed data. If window size can't be supported by the PMD then
 	 * setup of stream or private_xform should fail.
 	 */
+	enum rte_comp_hash_algorithm hash_algo;
+	/**< Hash algorithm to be used with decompress operation. Hash is always
+	 * done on plaintext.
+	 */
 };
 
 /**
@@ -268,6 +287,19 @@ struct rte_comp_op {
 		 * decompress direction.
 		 */
 	} dst;
+	struct {
+		uint8_t *digest;
+		/**< Output buffer to store hash output, if enabled in xform.
+		 * Buffer would contain valid value only after an op with
+		 * flush flag = RTE_COMP_FLUSH_FULL/FLUSH_FINAL is processed
+		 * successfully.
+		 *
+		 * Length of buffer should be contiguous and large enough to
+		 * accommodate digest produced by specific hash algo.
+		 */
+		rte_iova_t iova_addr;
+		/**< IO address of the buffer */
+	} hash;
 	enum rte_comp_flush_flag flush_flag;
 	/**< Defines flush characteristics for the output data.
 	 * Only applicable in compress direction
-- 
2.14.3

[dpdk-dev] [PATCH v4 09/13] compressdev: add device feature flags

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 21 ++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 33 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  1 +
 3 files changed, 55 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 433973aaa..b962ccf23 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -56,6 +56,27 @@ static struct rte_compressdev_global compressdev_globals = {
 
 struct rte_compressdev_global *rte_compressdev_globals = &compressdev_globals;
 
+const char * __rte_experimental
+rte_compressdev_get_feature_name(uint64_t flag)
+{
+	switch (flag) {
+	case RTE_COMPDEV_FF_HW_ACCELERATED:
+		return "HW_ACCELERATED";
+	case RTE_COMPDEV_FF_CPU_SSE:
+		return "CPU_SSE";
+	case RTE_COMPDEV_FF_CPU_AVX:
+		return "CPU_AVX";
+	case RTE_COMPDEV_FF_CPU_AVX2:
+		return "CPU_AVX2";
+	case RTE_COMPDEV_FF_CPU_AVX512:
+		return "CPU_AVX512";
+	case RTE_COMPDEV_FF_CPU_NEON:
+		return "CPU_NEON";
+	default:
+		return NULL;
+	}
+}
+
 struct rte_compressdev * __rte_experimental
 rte_compressdev_pmd_get_dev(uint8_t dev_id)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 0e148dd30..72c46dd96 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -34,10 +34,43 @@ extern int compressdev_logtype;
 #define RTE_COMPRESSDEV_NAME_MAX_LEN	(64)
 /**< Max length of name of comp PMD */
 
+/**
+ * compression device supported feature flags
+ *
+ * @note New features flags should be added to the end of the list
+ *
+ * Keep these flags synchronised with rte_compressdev_get_feature_name()
+ */
+#define	RTE_COMPDEV_FF_HW_ACCELERATED		(1ULL << 0)
+/**< Operations are off-loaded to an external hardware accelerator */
+#define	RTE_COMPDEV_FF_CPU_SSE			(1ULL << 1)
+/**< Utilises CPU SIMD SSE instructions */
+#define	RTE_COMPDEV_FF_CPU_AVX			(1ULL << 2)
+/**< Utilises CPU SIMD AVX instructions */
+#define	RTE_COMPDEV_FF_CPU_AVX2			(1ULL << 3)
+/**< Utilises CPU SIMD AVX2 instructions */
+#define	RTE_COMPDEV_FF_CPU_AVX512		(1ULL << 4)
+/**< Utilises CPU SIMD AVX512 instructions */
+#define	RTE_COMPDEV_FF_CPU_NEON			(1ULL << 5)
+/**< Utilises CPU NEON instructions */
+
+/**
+ * Get the name of a compress device feature flag.
+ *
+ * @param flag
+ *   The mask describing the flag
+ *
+ * @return
+ *   The name of this flag, or NULL if it's not a valid feature flag.
+ */
+const char * __rte_experimental
+rte_compressdev_get_feature_name(uint64_t flag);
+
 /**  comp device information */
 struct rte_compressdev_info {
 	const char *driver_name;		/**< Driver name. */
 	uint8_t driver_id;			/**< Driver identifier */
+	uint64_t feature_flags;			/**< Feature flags */
 	uint16_t max_nb_queue_pairs;
 	/**< Maximum number of queues pairs supported by device.
 	 * (If 0, there is no limit in maximum number of queue pairs)
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 58cb5205a..01ddc7a07 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -12,6 +12,7 @@ EXPERIMENTAL {
 	rte_compressdev_devices_get;
 	rte_compressdev_driver_id_get;
 	rte_compressdev_driver_name_get;
+	rte_compressdev_get_feature_name;
 	rte_compressdev_info_get;
 	rte_compressdev_is_valid_dev;
 	rte_compressdev_name_get;
-- 
2.14.3

[dpdk-dev] [PATCH v4 10/13] compressdev: add compression service feature flags

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 31 +++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 45 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  1 +
 3 files changed, 77 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index b962ccf23..43a26747b 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -77,6 +77,37 @@ rte_compressdev_get_feature_name(uint64_t flag)
 	}
 }
 
+const char * __rte_experimental
+rte_comp_get_feature_name(uint64_t flag)
+{
+	switch (flag) {
+	case RTE_COMP_FF_STATEFUL_COMPRESSION:
+		return "STATEFUL_COMPRESSION";
+	case RTE_COMP_FF_STATEFUL_DECOMPRESSION:
+		return "STATEFUL_DECOMPRESSION";
+	case RTE_COMP_FF_MBUF_SCATTER_GATHER:
+		return "MBUF_SCATTER_GATHER";
+	case RTE_COMP_FF_MULTI_PKT_CHECKSUM:
+		return "MULTI_PKT_CHECKSUM";
+	case RTE_COMP_FF_ADLER32_CHECKSUM:
+		return "ADLER32_CHECKSUM";
+	case RTE_COMP_FF_CRC32_CHECKSUM:
+		return "CRC32_CHECKSUM";
+	case RTE_COMP_FF_CRC32_ADLER32_CHECKSUM:
+		return "CRC32_ADLER32_CHECKSUM";
+	case RTE_COMP_FF_NONCOMPRESSED_BLOCKS:
+		return "NONCOMPRESSED_BLOCKS";
+	case RTE_COMP_FF_SHA1_HASH:
+		return "SHA1_HASH";
+	case RTE_COMP_FF_SHA2_SHA256_HASH:
+		return "SHA2_SHA256_HASH";
+	case RTE_COMP_FF_SHAREABLE_PRIV_XFORM:
+		return "SHAREABLE_PRIV_XFORM";
+	default:
+		return NULL;
+	}
+}
+
 struct rte_compressdev * __rte_experimental
 rte_compressdev_pmd_get_dev(uint8_t dev_id)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 72c46dd96..02a6c531c 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -54,6 +54,39 @@ extern int compressdev_logtype;
 #define	RTE_COMPDEV_FF_CPU_NEON			(1ULL << 5)
 /**< Utilises CPU NEON instructions */
 
+/**
+ * compression service feature flags
+ *
+ * @note New features flags should be added to the end of the list
+ *
+ * Keep these flags synchronised with rte_comp_get_feature_name()
+ */
+#define RTE_COMP_FF_STATEFUL_COMPRESSION	(1ULL << 0)
+/**< Stateful compression is supported */
+#define RTE_COMP_FF_STATEFUL_DECOMPRESSION	(1ULL << 1)
+/**< Stateful decompression is supported */
+#define	RTE_COMP_FF_MBUF_SCATTER_GATHER		(1ULL << 2)
+/**< Scatter-gather mbufs are supported */
+#define RTE_COMP_FF_ADLER32_CHECKSUM		(1ULL << 3)
+/**< Adler-32 Checksum is supported */
+#define RTE_COMP_FF_CRC32_CHECKSUM		(1ULL << 4)
+/**< CRC32 Checksum is supported */
+#define RTE_COMP_FF_CRC32_ADLER32_CHECKSUM	(1ULL << 5)
+/**< Adler-32/CRC32 Checksum is supported */
+#define RTE_COMP_FF_MULTI_PKT_CHECKSUM		(1ULL << 6)
+/**< Generation of checksum across multiple stateless packets is supported */
+#define RTE_COMP_FF_SHA1_HASH			(1ULL << 7)
+/**< SHA1 Hash is supported */
+#define RTE_COMP_FF_SHA2_SHA256_HASH		(1ULL << 8)
+/**< SHA256 Hash of SHA2 family is supported */
+#define RTE_COMP_FF_NONCOMPRESSED_BLOCKS	(1ULL << 9)
+/**< Creation of non-compressed blocks using RTE_COMP_LEVEL_NONE is supported */
+#define RTE_COMP_FF_SHAREABLE_PRIV_XFORM	(1ULL << 10)
+/**< Private xforms created by the PMD can be shared
+ * across multiple stateless operations. If not set, then app needs
+ * to create as many priv_xforms as many expected in flight.
+ */
+
 /**
  * Get the name of a compress device feature flag.
  *
@@ -66,6 +99,18 @@ extern int compressdev_logtype;
 const char * __rte_experimental
 rte_compressdev_get_feature_name(uint64_t flag);
 
+/**
+ * Get the name of a compress service feature flag
+ *
+ * @param flag
+ *   The mask describing the flag
+ *
+ * @return
+ *   The name of this flag, or NULL if it's not a valid feature flag.
+ */
+const char * __rte_experimental
+rte_comp_get_feature_name(uint64_t flag);
+
 /**  comp device information */
 struct rte_compressdev_info {
 	const char *driver_name;		/**< Driver name. */
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 01ddc7a07..f538b9f57 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -33,6 +33,7 @@ EXPERIMENTAL {
 	rte_compressdev_stop;
 	rte_compressdev_stream_create;
 	rte_compressdev_stream_free;
+	rte_comp_get_feature_name;
 	rte_comp_op_pool_create;
 
         local: *;
-- 
2.14.3

[dpdk-dev] [PATCH v4 11/13] compressdev: add device stats

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 41 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 39 ++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 26 ++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  2 ++
 4 files changed, 108 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 43a26747b..15811cd6e 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -562,6 +562,47 @@ rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
 			max_inflight_ops, socket_id);
 }
 
+
+int __rte_experimental
+rte_compressdev_stats_get(uint8_t dev_id, struct rte_compressdev_stats *stats)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%d", dev_id);
+		return -ENODEV;
+	}
+
+	if (stats == NULL) {
+		COMPRESSDEV_LOG(ERR, "Invalid stats ptr");
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+	memset(stats, 0, sizeof(*stats));
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->stats_get, -ENOTSUP);
+	(*dev->dev_ops->stats_get)(dev, stats);
+	return 0;
+}
+
+void __rte_experimental
+rte_compressdev_stats_reset(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	RTE_FUNC_PTR_OR_RET(*dev->dev_ops->stats_reset);
+	(*dev->dev_ops->stats_reset)(dev);
+}
+
+
 void __rte_experimental
 rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 02a6c531c..f8aab528c 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -122,6 +122,19 @@ struct rte_compressdev_info {
 	 */
 };
 
+/** comp device statistics */
+struct rte_compressdev_stats {
+	uint64_t enqueued_count;
+	/**< Count of all operations enqueued */
+	uint64_t dequeued_count;
+	/**< Count of all operations dequeued */
+
+	uint64_t enqueue_err_count;
+	/**< Total error count on operations enqueued */
+	uint64_t dequeue_err_count;
+	/**< Total error count on operations dequeued */
+};
+
 /**
  * Get the compress device name given a device identifier.
  *
@@ -306,6 +319,32 @@ rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
 uint16_t __rte_experimental
 rte_compressdev_queue_pair_count(uint8_t dev_id);
 
+
+/**
+ * Retrieve the general I/O statistics of a device.
+ *
+ * @param dev_id
+ *   The identifier of the device
+ * @param stats
+ *   A pointer to a structure of type
+ *   *rte_compressdev_stats* to be filled with the
+ *   values of device counters
+ * @return
+ *   - Zero if successful.
+ *   - Non-zero otherwise.
+ */
+int __rte_experimental
+rte_compressdev_stats_get(uint8_t dev_id, struct rte_compressdev_stats *stats);
+
+/**
+ * Reset the general I/O statistics of a device.
+ *
+ * @param dev_id
+ *   The identifier of the device.
+ */
+void __rte_experimental
+rte_compressdev_stats_reset(uint8_t dev_id);
+
 /**
  * Retrieve the contextual information of a device.
  *
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
index 59bcb688c..e08c69e36 100644
--- a/lib/librte_compressdev/rte_compressdev_pmd.h
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -140,6 +140,27 @@ typedef void (*compressdev_stop_t)(struct rte_compressdev *dev);
 typedef int (*compressdev_close_t)(struct rte_compressdev *dev);
 
 
+/**
+ * Function used to get statistics of a device.
+ *
+ * @param dev
+ *   Compress device
+ * @param stats
+ *   Compress device stats to populate
+ */
+typedef void (*compressdev_stats_get_t)(struct rte_compressdev *dev,
+				struct rte_compressdev_stats *stats);
+
+
+/**
+ * Function used to reset statistics of a device.
+ *
+ * @param dev
+ *   Compress device
+ */
+typedef void (*compressdev_stats_reset_t)(struct rte_compressdev *dev);
+
+
 /**
  * Function used to get specific information of a device.
  *
@@ -271,6 +292,11 @@ struct rte_compressdev_ops {
 
 	compressdev_info_get_t dev_infos_get;	/**< Get device info. */
 
+	compressdev_stats_get_t stats_get;
+	/**< Get device statistics. */
+	compressdev_stats_reset_t stats_reset;
+	/**< Reset device statistics. */
+
 	compressdev_queue_pair_setup_t queue_pair_setup;
 	/**< Set up a device queue pair. */
 	compressdev_queue_pair_release_t queue_pair_release;
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index f538b9f57..7bdc58a38 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -30,6 +30,8 @@ EXPERIMENTAL {
 	rte_compressdev_queue_pair_setup;
 	rte_compressdev_socket_id;
 	rte_compressdev_start;
+	rte_compressdev_stats_get;
+	rte_compressdev_stats_reset;
 	rte_compressdev_stop;
 	rte_compressdev_stream_create;
 	rte_compressdev_stream_free;
-- 
2.14.3

[dpdk-dev] [PATCH v4 12/13] compressdev: add device capabilities

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added structure which each PMD will fill out,
providing the capabilities of each driver
(containing mainly which compression services
it supports).

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 23 ++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 35 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  1 +
 3 files changed, 59 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 15811cd6e..4435801fd 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -56,6 +56,29 @@ static struct rte_compressdev_global compressdev_globals = {
 
 struct rte_compressdev_global *rte_compressdev_globals = &compressdev_globals;
 
+const struct rte_compressdev_capabilities * __rte_experimental
+rte_compressdev_capability_get(uint8_t dev_id,
+			enum rte_comp_algorithm algo)
+{
+	const struct rte_compressdev_capabilities *capability;
+	struct rte_compressdev_info dev_info;
+	int i = 0;
+
+	if (dev_id >= compressdev_globals.nb_devs) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%d", dev_id);
+		return NULL;
+	}
+	rte_compressdev_info_get(dev_id, &dev_info);
+
+	while ((capability = &dev_info.capabilities[i++])->algo !=
+			RTE_COMP_ALGO_UNSPECIFIED){
+		if (capability->algo == algo)
+			return capability;
+	}
+
+	return NULL;
+}
+
 const char * __rte_experimental
 rte_compressdev_get_feature_name(uint64_t flag)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index f8aab528c..e9cb212ec 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -34,6 +34,39 @@ extern int compressdev_logtype;
 #define RTE_COMPRESSDEV_NAME_MAX_LEN	(64)
 /**< Max length of name of comp PMD */
 
+/**
+ * Parameter log base 2 range description.
+ * Final value will be 2^value.
+ */
+struct rte_param_log2_range {
+	uint8_t min;	/**< Minimum log2 value */
+	uint8_t max;	/**< Maximum log2 value */
+	uint8_t increment;
+	/**< If a range of sizes are supported,
+	 * this parameter is used to indicate
+	 * increments in base 2 log byte value
+	 * that are supported between the minimum and maximum
+	 */
+};
+
+/** Structure used to capture a capability of a comp device */
+struct rte_compressdev_capabilities {
+	enum rte_comp_algorithm algo;
+	/* Compression algorithm */
+	uint64_t comp_feature_flags;
+	/**< Bitmask of flags for compression service features */
+	struct rte_param_log2_range window_size;
+	/**< Window size range in base two log byte values */
+};
+
+/** Macro used at end of comp PMD list */
+#define RTE_COMP_END_OF_CAPABILITIES_LIST() \
+	{ RTE_COMP_ALGO_UNSPECIFIED }
+
+const struct rte_compressdev_capabilities * __rte_experimental
+rte_compressdev_capability_get(uint8_t dev_id,
+			enum rte_comp_algorithm algo);
+
 /**
  * compression device supported feature flags
  *
@@ -116,6 +149,8 @@ struct rte_compressdev_info {
 	const char *driver_name;		/**< Driver name. */
 	uint8_t driver_id;			/**< Driver identifier */
 	uint64_t feature_flags;			/**< Feature flags */
+	const struct rte_compressdev_capabilities *capabilities;
+	/**< Array of devices supported capabilities */
 	uint16_t max_nb_queue_pairs;
 	/**< Maximum number of queues pairs supported by device.
 	 * (If 0, there is no limit in maximum number of queue pairs)
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 7bdc58a38..dec73fcff 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -5,6 +5,7 @@ EXPERIMENTAL {
 	rte_compressdev_allocate_driver;
 	rte_compressdev_callback_register;
 	rte_compressdev_callback_unregister;
+	rte_compressdev_capability_get;
 	rte_compressdev_close;
 	rte_compressdev_configure;
 	rte_compressdev_count;
-- 
2.14.3

[dpdk-dev] [PATCH v4 13/13] compressdev: get device id from name

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added API to retrieve the device id provided the device name.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 18 ++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 13 +++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  1 +
 3 files changed, 32 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 4435801fd..4ad13a0ec 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -173,6 +173,24 @@ rte_compressdev_is_valid_dev(uint8_t dev_id)
 }
 
 
+int __rte_experimental
+rte_compressdev_get_dev_id(const char *name)
+{
+	unsigned int i;
+
+	if (name == NULL)
+		return -1;
+
+	for (i = 0; i < rte_compressdev_globals->nb_devs; i++)
+		if ((strcmp(rte_compressdev_globals->devs[i].data->name, name)
+				== 0) &&
+				(rte_compressdev_globals->devs[i].attached ==
+						RTE_COMPRESSDEV_ATTACHED))
+			return i;
+
+	return -1;
+}
+
 uint8_t __rte_experimental
 rte_compressdev_count(void)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index e9cb212ec..1d1a072ef 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -170,6 +170,19 @@ struct rte_compressdev_stats {
 	/**< Total error count on operations dequeued */
 };
 
+
+/**
+ * Get the device identifier for the named compress device.
+ *
+ * @param name
+ *   Device name to select the device structure
+ * @return
+ *   - Returns compress device identifier on success.
+ *   - Return -1 on failure to find named compress device.
+ */
+int __rte_experimental
+rte_compressdev_get_dev_id(const char *name);
+
 /**
  * Get the compress device name given a device identifier.
  *
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index dec73fcff..46bdda88b 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -13,6 +13,7 @@ EXPERIMENTAL {
 	rte_compressdev_devices_get;
 	rte_compressdev_driver_id_get;
 	rte_compressdev_driver_name_get;
+	rte_compressdev_get_dev_id;
 	rte_compressdev_get_feature_name;
 	rte_compressdev_info_get;
 	rte_compressdev_is_valid_dev;
-- 
2.14.3

[dpdk-dev] [PATCH v2 1/3] compressdev: add structs and enum for compression service

[Fiona Trahe]

Introduce enums and structs to support the transforms and
operations needed for a compression service.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 MAINTAINERS                       |   7 +
 lib/librte_compressdev/rte_comp.h | 528 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 535 insertions(+)
 create mode 100644 lib/librte_compressdev/rte_comp.h

diff --git a/MAINTAINERS b/MAINTAINERS
index a646ca3..068f85e 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -325,6 +325,13 @@ F: drivers/raw/skeleton_rawdev/
 F: test/test/test_rawdev.c
 F: doc/guides/prog_guide/rawdev.rst
 
+Compression API - EXPERIMENTAL
+M: Fiona Trahe <fiona.trahe@intel.com>
+M: Pablo de Lara <pablo.de.lara.guarch@intel.com>
+M: Ashish Gupta <ashish.gupta@caviumnetworks.com>
+T: TBD
+F: lib/librte_compressdev/
+
 
 Bus Drivers
 -----------
diff --git a/lib/librte_compressdev/rte_comp.h b/lib/librte_compressdev/rte_comp.h
new file mode 100644
index 0000000..7d755ee
--- /dev/null
+++ b/lib/librte_compressdev/rte_comp.h
@@ -0,0 +1,528 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMP_H_
+#define _RTE_COMP_H_
+
+/**
+ * @file rte_comp.h
+ *
+ * RTE definitions for Data Compression Service
+ *
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <string.h>
+#include <rte_mempool.h>
+
+
+/** Status of comp operation */
+enum rte_comp_op_status {
+	RTE_COMP_OP_STATUS_SUCCESS = 0,
+	/**< Operation completed successfully */
+	RTE_COMP_OP_STATUS_NOT_PROCESSED,
+	/**< Operation has not yet been processed by the device */
+	RTE_COMP_OP_STATUS_INVALID_ARGS,
+	/**< Operation failed due to invalid arguments in request */
+	RTE_COMP_OP_STATUS_ERROR,
+	/**< Error handling operation */
+	RTE_COMP_OP_STATUS_INVALID_STATE,
+	/**< Operation is invoked in invalid state */
+	RTE_COMP_OP_STATUS_OUT_OF_SPACE_TERMINATED,
+	/**< Output buffer ran out of space before operation completed.
+	 * Error case. Application must resubmit all data with a larger
+	 * output buffer.
+	 */
+	RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE,
+	/**< Output buffer ran out of space before operation completed, but this
+	 * is not an error case. Output data up to op.produced can be used and
+	 * next op in the stream should continue on from op.consumed+1.
+	 */
+};
+
+
+/** Compression Algorithms */
+enum rte_comp_algorithm {
+	RTE_COMP_ALGO_UNSPECIFIED = 0,
+	/** No Compression algorithm */
+	RTE_COMP_ALGO_NULL,
+	/**< No compression.
+	 * Pass-through, data is copied unchanged from source buffer to
+	 * destination buffer.
+	 */
+	RTE_COMP_ALGO_DEFLATE,
+	/**< DEFLATE compression algorithm
+	 * https://tools.ietf.org/html/rfc1951
+	 */
+	RTE_COMP_ALGO_LZS,
+	/**< LZS compression algorithm
+	 * https://tools.ietf.org/html/rfc2395
+	 */
+	RTE_COMP_ALGO_LIST_END
+};
+
+/**< Compression Level.
+ * The number is interpreted by each PMD differently. However, lower numbers
+ * give fastest compression, at the expense of compression ratio while
+ * higher numbers may give better compression ratios but are likely slower.
+ */
+#define	RTE_COMP_LEVEL_PMD_DEFAULT	(-1)
+/** Use PMD Default */
+#define	RTE_COMP_LEVEL_NONE		(0)
+/** Output uncompressed blocks if supported by the specified algorithm */
+#define RTE_COMP_LEVEL_MIN		(1)
+/** Use minimum compression level supported by the PMD */
+#define RTE_COMP_LEVEL_MAX		(9)
+/** Use maximum compression level supported by the PMD */
+
+/** Compression checksum types */
+enum rte_comp_checksum_type {
+	RTE_COMP_CHECKSUM_NONE,
+	/**< No checksum generated */
+	RTE_COMP_CHECKSUM_CRC32,
+	/**< Generates a CRC32 checksum, as used by gzip */
+	RTE_COMP_CHECKSUM_ADLER32,
+	/**< Generates an Adler-32 checksum, as used by zlib */
+	RTE_COMP_CHECKSUM_CRC32_ADLER32,
+	/**< Generates both Adler-32 and CRC32 checksums, concatenated.
+	 * CRC32 is in the lower 32bits, Adler-32 in the upper 32 bits.
+	 */
+};
+
+
+/** Compression Huffman Type - used by DEFLATE algorithm */
+enum rte_comp_huffman {
+	RTE_COMP_HUFFMAN_DEFAULT,
+	/**< PMD may choose which Huffman codes to use */
+	RTE_COMP_HUFFMAN_FIXED,
+	/**< Use Fixed Huffman codes */
+	RTE_COMP_HUFFMAN_DYNAMIC,
+	/**< Use Dynamic Huffman codes */
+};
+
+enum rte_comp_flush_flag {
+	RTE_COMP_FLUSH_NONE,
+	/**< Data is not flushed. Output may remain in the compressor and be
+	 * processed during a following op. It may not be possible to decompress
+	 * output until a later op with some other flush flag has been sent.
+	 */
+	RTE_COMP_FLUSH_SYNC,
+	/**< All data should be flushed to output buffer. Output data can be
+	 * decompressed. However state and history is not cleared, so future
+	 * operations may use history from this operation.
+	 */
+	RTE_COMP_FLUSH_FULL,
+	/**< All data should be flushed to output buffer. Output data can be
+	 * decompressed. State and history data is cleared, so future
+	 * ops will be independent of ops processed before this.
+	 */
+	RTE_COMP_FLUSH_FINAL
+	/**< Same as RTE_COMP_FLUSH_FULL but if op.algo is RTE_COMP_ALGO_DEFLATE
+	 * then bfinal bit is set in the last block.
+	 */
+};
+
+/** Compression transform types */
+enum rte_comp_xform_type {
+	RTE_COMP_COMPRESS,
+	/**< Compression service - compress */
+	RTE_COMP_DECOMPRESS,
+	/**< Compression service - decompress */
+};
+
+enum rte_comp_private_xform_mode {
+	RTE_COMP_PRIV_XFORM_SHAREABLE,
+	/**< private_xform can be attached
+	 * to multiple ops inflight in the device simultaneously
+	 */
+	RTE_COMP_PRIV_XFORM_NOT_SHAREABLE
+	/**< private_xform can only be used on one inflight op at a time. */
+};
+
+enum rte_comp_op_type {
+	RTE_COMP_OP_STATELESS,
+	/**< All data to be processed is submitted in the op, no state or
+	 * history from previous ops is used and none will be stored for future
+	 * ops. Flush flag must be set to either FLUSH_FULL or FLUSH_FINAL.
+	 */
+	RTE_COMP_OP_STATEFUL
+	/**< There may be more data to be processed after this op, it's part of
+	 * a stream of data. State and history from previous ops can be used
+	 * and resulting state and history can be stored for future ops,
+	 * depending on flush flag.
+	 */
+};
+
+
+/** Parameters specific to the deflate algorithm */
+struct rte_comp_deflate_params {
+	enum rte_comp_huffman huffman;
+	/**< Compression huffman encoding type */
+};
+
+/** Setup Data for compression */
+struct rte_comp_compress_xform {
+	enum rte_comp_algorithm algo;
+	/**< Algorithm to use for compress operation */
+	union {
+		struct rte_comp_deflate_params deflate;
+		/**< Parameters specific to the deflate algorithm */
+	}; /**< Algorithm specific parameters */
+	int level;
+	/**< Compression level */
+	uint8_t window_size;
+	/**< Base two log value of sliding window to be used. If window size
+	 * can't be supported by the PMD then it may fall back to a smaller
+	 * size. This is likely to result in a worse compression ratio.
+	 */
+	enum rte_comp_checksum_type chksum;
+	/**< Type of checksum to generate on the uncompressed data */
+};
+
+/**
+ * Setup Data for decompression.
+ */
+struct rte_comp_decompress_xform {
+	enum rte_comp_algorithm algo;
+	/**< Algorithm to use for decompression */
+	enum rte_comp_checksum_type chksum;
+	/**< Type of checksum to generate on the decompressed data */
+	uint8_t window_size;
+	/**< Base two log value of sliding window which was used to generate
+	 * compressed data. If window size can't be supported by the PMD then
+	 * setup of stream or private_xform should fail.
+	 */
+};
+
+/**
+ * Compression transform structure.
+ *
+ * This is used to specify the compression transforms required.
+ * Each transform structure can hold a single transform, the type field is
+ * used to specify which transform is contained within the union.
+ * There are no chain cases currently supported, just single xforms of
+ *  - compress-only
+ *  - decompress-only
+ *
+ */
+struct rte_comp_xform {
+	struct rte_comp_xform *next;
+	/**< next xform in chain */
+	enum rte_comp_xform_type type;
+	/**< xform type */
+	union {
+		struct rte_comp_compress_xform compress;
+		/**< xform for compress operation */
+		struct rte_comp_decompress_xform decompress;
+		/**< decompress xform */
+	};
+};
+
+
+/**
+ * Compression Operation.
+ *
+ * This structure contains data relating to performing a compression
+ * operation on the referenced mbuf data buffers.
+ *
+ * Comp operations are enqueued and dequeued in comp PMDs using the
+ * rte_compressdev_enqueue_burst() / rte_compressdev_dequeue_burst() APIs
+ */
+struct rte_comp_op {
+
+	enum rte_comp_op_type op_type;
+	union {
+		void *private_xform;
+		/**< Stateless private PMD data derived from an rte_comp_xform.
+		 * A handle returned by rte_compressdev_private_xform_create()
+		 * must be attached to operations of op_type RTE_COMP_STATELESS.
+		 */
+		void *stream;
+		/**< Private PMD data derived initially from an rte_comp_xform,
+		 * which holds state and history data and evolves as operations
+		 * are processed. rte_comp_stream_create() must be called on a
+		 * device for all STATEFUL data streams and the resulting
+		 * stream attached to the one or more operations associated
+		 * with the data stream.
+		 * All operations in a stream must be sent to the same device.
+		 */
+	};
+
+	struct rte_mempool *mempool;
+	/**< Pool from which operation is allocated */
+	rte_iova_t phys_addr;
+	/**< Physical address of this operation */
+	struct rte_mbuf *m_src;
+	/**< source mbuf
+	 * The total size of the input buffer(s) can be retrieved using
+	 * rte_pktmbuf_data_len(m_src)
+	 */
+	struct rte_mbuf *m_dst;
+	/**< destination mbuf
+	 * The total size of the output buffer(s) can be retrieved using
+	 * rte_pktmbuf_data_len(m_dst)
+	 */
+
+	struct {
+		uint32_t offset;
+		/**< Starting point for compression or decompression,
+		 * specified as number of bytes from start of packet in
+		 * source buffer.
+		 * Starting point for checksum generation in compress direction.
+		 */
+		uint32_t length;
+		/**< The length, in bytes, of the data in source buffer
+		 * to be compressed or decompressed.
+		 * Also the length of the data over which the checksum
+		 * should be generated in compress direction
+		 */
+	} src;
+	struct {
+		uint32_t offset;
+		/**< Starting point for writing output data, specified as
+		 * number of bytes from start of packet in dest
+		 * buffer. Starting point for checksum generation in
+		 * decompress direction.
+		 */
+	} dst;
+	enum rte_comp_flush_flag flush_flag;
+	/**< Defines flush characteristics for the output data.
+	 * Only applicable in compress direction
+	 */
+	uint64_t input_chksum;
+	/**< An input checksum can be provided to generate a
+	 * cumulative checksum across sequential blocks in a STATELESS stream.
+	 * Checksum type is as specified in xform chksum_type
+	 */
+	uint64_t output_chksum;
+	/**< If a checksum is generated it will be written in here.
+	 * Checksum type is as specified in xform chksum_type.
+	 */
+	uint32_t consumed;
+	/**< The number of bytes from the source buffer
+	 * which were compressed/decompressed.
+	 */
+	uint32_t produced;
+	/**< The number of bytes written to the destination buffer
+	 * which were compressed/decompressed.
+	 */
+	uint64_t debug_status;
+	/**<
+	 * Status of the operation is returned in the status param.
+	 * This field allows the PMD to pass back extra
+	 * pmd-specific debug information. Value is not defined on the API.
+	 */
+	uint8_t status;
+	/**<
+	 * Operation status - use values from enum rte_comp_status.
+	 * This is reset to
+	 * RTE_COMP_OP_STATUS_NOT_PROCESSED on allocation from mempool and
+	 * will be set to RTE_COMP_OP_STATUS_SUCCESS after operation
+	 * is successfully processed by a PMD
+	 */
+} __rte_cache_aligned;
+
+
+/**
+ * Reset the fields of an operation to their default values.
+ *
+ * @note The private data associated with the operation is not zeroed.
+ *
+ * @param op
+ *   The operation to be reset
+ */
+static inline void
+__rte_comp_op_reset(struct rte_comp_op *op)
+{
+	struct rte_mempool *tmp_mp = op->mempool;
+	phys_addr_t tmp_phys_addr = op->phys_addr;
+
+	memset(op, 0, sizeof(struct rte_comp_op));
+	op->status = RTE_COMP_OP_STATUS_NOT_PROCESSED;
+	op->phys_addr = tmp_phys_addr;
+	op->mempool = tmp_mp;
+}
+
+/**
+ * Private data structure belonging to an operation pool.
+ */
+struct rte_comp_op_pool_private {
+	uint16_t user_size;
+	/**< Size of private user data with each operation. */
+};
+
+
+/**
+ * Returns the size of private user data allocated with each object in
+ * the mempool
+ *
+ * @param mempool
+ *   Mempool for operations
+ * @return
+ *   user data size
+ */
+static inline uint16_t
+__rte_comp_op_get_user_data_size(struct rte_mempool *mempool)
+{
+	struct rte_comp_op_pool_private *priv =
+	    (struct rte_comp_op_pool_private *)rte_mempool_get_priv(mempool);
+
+	return priv->user_size;
+}
+
+
+/**
+ * Creates an operation pool
+ *
+ * @param name
+ *   Compress pool name
+ * @param nb_elts
+ *   Number of elements in pool
+ * @param cache_size
+ *   Number of elements to cache on lcore, see
+ *   *rte_mempool_create* for further details about cache size
+ * @param user_size
+ *   Size of private data to allocate for user with each operation
+ * @param socket_id
+ *   Socket to identifier allocate memory on
+ * @return
+ *  - On success pointer to mempool
+ *  - On failure NULL
+ */
+struct rte_mempool *
+rte_comp_op_pool_create(const char *name,
+		unsigned int nb_elts, unsigned int cache_size,
+		uint16_t user_size, int socket_id);
+
+/**
+ * Bulk allocate raw element from mempool and return as comp operations
+ *
+ * @param mempool
+ *   Compress operation mempool
+ * @param ops
+ *   Array to place allocated operations
+ * @param nb_ops
+ *   Number of operations to allocate
+ * @return
+ * - On success returns  number of ops allocated
+ */
+static inline int
+__rte_comp_op_raw_bulk_alloc(struct rte_mempool *mempool,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	if (rte_mempool_get_bulk(mempool, (void **)ops, nb_ops) == 0)
+		return nb_ops;
+
+	return 0;
+}
+
+/**
+ * Allocate an operation from a mempool with default parameters set
+ *
+ * @param mempool
+ *   Compress operation mempool
+ *
+ * @return
+ * - On success returns a valid rte_comp_op structure
+ * - On failure returns NULL
+ */
+static inline struct rte_comp_op *
+rte_comp_op_alloc(struct rte_mempool *mempool)
+{
+	struct rte_comp_op *op = NULL;
+	int retval;
+
+	retval = __rte_comp_op_raw_bulk_alloc(mempool, &op, 1);
+	if (unlikely(retval != 1))
+		return NULL;
+
+	__rte_comp_op_reset(op);
+
+	return op;
+}
+
+
+/**
+ * Bulk allocate operations from a mempool with default parameters set
+ *
+ * @param mempool
+ *   Compress operation mempool
+ * @param ops
+ *   Array to place allocated operations
+ * @param nb_ops
+ *   Number of operations to allocate
+ * @return
+ * - nb_ops if the number of operations requested were allocated.
+ * - 0 if the requested number of ops are not available.
+ *   None are allocated in this case.
+ */
+static inline unsigned
+rte_comp_op_bulk_alloc(struct rte_mempool *mempool,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	int i;
+
+	if (unlikely(__rte_comp_op_raw_bulk_alloc(mempool, ops, nb_ops)
+			!= nb_ops))
+		return 0;
+
+	for (i = 0; i < nb_ops; i++)
+		__rte_comp_op_reset(ops[i]);
+
+	return nb_ops;
+}
+
+
+
+/**
+ * Returns a pointer to the private user data of an operation if
+ * that operation has enough capacity for requested size.
+ *
+ * @param op
+ *   Compress operation
+ * @param size
+ *   Size of space requested in private data
+ * @return
+ * - if sufficient space available returns pointer to start of user data
+ * - if insufficient space returns NULL
+ */
+static inline void *
+__rte_comp_op_get_user_data(struct rte_comp_op *op, uint32_t size)
+{
+	uint32_t user_size;
+
+	if (likely(op->mempool != NULL)) {
+		user_size = __rte_comp_op_get_user_data_size(op->mempool);
+
+		if (likely(user_size >= size))
+			return (void *)(op + 1);
+
+	}
+
+	return NULL;
+}
+
+/**
+ * free operation structure
+ * If operation has been allocate from a rte_mempool, then the operation will
+ * be returned to the mempool.
+ *
+ * @param op
+ *   Compress operation
+ */
+static inline void
+rte_comp_op_free(struct rte_comp_op *op)
+{
+	if (op != NULL && op->mempool != NULL)
+		rte_mempool_put(op->mempool, op);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_COMP_H_ */
-- 
2.7.4

[dpdk-dev] [PATCH v2 2/3] compressdev: implement API

[Fiona Trahe]

Introduce:
 - compressdev device management APIs
 - ompression service APIs
 - APIs for PMDs to plug in and offer the compression service
 - build systems for all above

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 config/common_base                                 |   6 +
 config/rte_config.h                                |   3 +
 lib/Makefile                                       |   3 +
 lib/librte_compressdev/Makefile                    |  29 +
 lib/librte_compressdev/meson.build                 |   9 +
 lib/librte_compressdev/rte_compressdev.c           | 904 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 727 +++++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.c       | 156 ++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 450 ++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  43 +
 lib/meson.build                                    |   2 +-
 mk/rte.app.mk                                      |   1 +
 12 files changed, 2332 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_compressdev/Makefile
 create mode 100644 lib/librte_compressdev/meson.build
 create mode 100644 lib/librte_compressdev/rte_compressdev.c
 create mode 100644 lib/librte_compressdev/rte_compressdev.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.c
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_version.map

diff --git a/config/common_base b/config/common_base
index ad03cf4..e0e5768 100644
--- a/config/common_base
+++ b/config/common_base
@@ -535,6 +535,12 @@ CONFIG_RTE_LIBRTE_PMD_MRVL_CRYPTO=n
 CONFIG_RTE_LIBRTE_PMD_MRVL_CRYPTO_DEBUG=n
 
 #
+# Compile generic compression device library
+#
+CONFIG_RTE_LIBRTE_COMPRESSDEV=y
+CONFIG_RTE_COMPRESS_MAX_DEVS=64
+
+#
 # Compile generic security library
 #
 CONFIG_RTE_LIBRTE_SECURITY=y
diff --git a/config/rte_config.h b/config/rte_config.h
index 699878a..6c2e60b 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -48,6 +48,9 @@
 #define RTE_ETHDEV_QUEUE_STAT_CNTRS 16
 #define RTE_ETHDEV_RXTX_CALLBACKS 1
 
+/* compressdev defines */
+#define RTE_COMPRESS_MAX_DEVS 64
+
 /* cryptodev defines */
 #define RTE_CRYPTO_MAX_DEVS 64
 #define RTE_CRYPTODEV_NAME_LEN 64
diff --git a/lib/Makefile b/lib/Makefile
index ec965a6..19396da 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -26,6 +26,9 @@ DEPDIRS-librte_bbdev := librte_eal librte_mempool librte_mbuf
 DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += librte_cryptodev
 DEPDIRS-librte_cryptodev := librte_eal librte_mempool librte_ring librte_mbuf
 DEPDIRS-librte_cryptodev += librte_kvargs
+DIRS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += librte_compressdev
+DEPDIRS-librte_compressdev := librte_eal librte_mempool librte_ring librte_mbuf
+DEPDIRS-librte_compressdev += librte_kvargs
 DIRS-$(CONFIG_RTE_LIBRTE_SECURITY) += librte_security
 DEPDIRS-librte_security := librte_eal librte_mempool librte_ring librte_mbuf
 DEPDIRS-librte_security += librte_ether
diff --git a/lib/librte_compressdev/Makefile b/lib/librte_compressdev/Makefile
new file mode 100644
index 0000000..6f1546a
--- /dev/null
+++ b/lib/librte_compressdev/Makefile
@@ -0,0 +1,29 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2017-2018 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_compressdev.a
+
+# library version
+LIBABIVER := 1
+
+# build flags
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+LDLIBS += -lrte_eal -lrte_mempool -lrte_kvargs
+
+# library source files
+SRCS-y += rte_compressdev.c rte_compressdev_pmd.c
+
+# export include files
+SYMLINK-y-include += rte_comp.h
+SYMLINK-y-include += rte_compressdev.h
+SYMLINK-y-include += rte_compressdev_pmd.h
+
+# versioning export map
+EXPORT_MAP := rte_compressdev_version.map
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_compressdev/meson.build b/lib/librte_compressdev/meson.build
new file mode 100644
index 0000000..a72d4ce
--- /dev/null
+++ b/lib/librte_compressdev/meson.build
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+allow_experimental_apis = true
+sources = files('rte_compressdev.c', 'rte_compressdev_pmd.c')
+headers = files('rte_compressdev.h',
+	'rte_compressdev_pmd.h',
+	'rte_comp.h')
+deps += ['kvargs', 'mbuf']
diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
new file mode 100644
index 0000000..66e5d26
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -0,0 +1,904 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#include <sys/types.h>
+#include <sys/queue.h>
+#include <ctype.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stdarg.h>
+#include <errno.h>
+#include <stdint.h>
+#include <inttypes.h>
+#include <netinet/in.h>
+
+#include <rte_byteorder.h>
+#include <rte_log.h>
+#include <rte_debug.h>
+#include <rte_dev.h>
+#include <rte_interrupts.h>
+#include <rte_memory.h>
+#include <rte_memcpy.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_tailq.h>
+#include <rte_eal.h>
+#include <rte_per_lcore.h>
+#include <rte_lcore.h>
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_mempool.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_errno.h>
+#include <rte_spinlock.h>
+#include <rte_string_fns.h>
+
+#include "rte_comp.h"
+#include "rte_compressdev.h"
+#include "rte_compressdev_pmd.h"
+
+static uint8_t nb_drivers;
+
+struct rte_compressdev rte_comp_devices[RTE_COMPRESS_MAX_DEVS];
+
+struct rte_compressdev *rte_compressdevs = &rte_comp_devices[0];
+
+static struct rte_compressdev_global compressdev_globals = {
+		.devs			= &rte_comp_devices[0],
+		.data			= { NULL },
+		.nb_devs		= 0,
+		.max_devs		= RTE_COMPRESS_MAX_DEVS
+};
+
+struct rte_compressdev_global *rte_compressdev_globals = &compressdev_globals;
+
+/**
+ * The compression algorithm strings identifiers.
+ * It could be used in application command line.
+ */
+const char *
+rte_comp_algorithm_strings[] = {
+	[RTE_COMP_ALGO_DEFLATE]		= "deflate",
+	[RTE_COMP_ALGO_LZS]		= "lzs"
+};
+
+
+#define param_range_check(x, y) \
+	(((x < y.min) || (x > y.max)) || \
+	(y.increment != 0 && (x % y.increment) != 0))
+
+const struct rte_compressdev_capabilities *__rte_experimental
+rte_compressdev_capability_get(uint8_t dev_id,
+			enum rte_comp_algorithm algo)
+{
+	const struct rte_compressdev_capabilities *capability;
+	struct rte_compressdev_info dev_info;
+	int i = 0;
+
+	memset(&dev_info, 0, sizeof(struct rte_compressdev_info));
+	rte_compressdev_info_get(dev_id, &dev_info);
+
+	while ((capability = &dev_info.capabilities[i++])->algo !=
+			RTE_COMP_ALGO_UNSPECIFIED){
+		if (capability->algo == algo)
+			return capability;
+	}
+
+	return NULL;
+}
+
+const char *__rte_experimental
+rte_compressdev_get_feature_name(uint64_t flag)
+{
+	switch (flag) {
+	case RTE_COMPDEV_FF_HW_ACCELERATED:
+		return "HW_ACCELERATED";
+	case RTE_COMPDEV_FF_CPU_SSE:
+		return "CPU_SSE";
+	case RTE_COMPDEV_FF_CPU_AVX:
+		return "CPU_AVX";
+	case RTE_COMPDEV_FF_CPU_AVX2:
+		return "CPU_AVX2";
+	case RTE_COMPDEV_FF_CPU_AVX512:
+		return "CPU_AVX512";
+	case RTE_COMPDEV_FF_CPU_NEON:
+		return "CPU_NEON";
+	default:
+		return NULL;
+	}
+}
+
+const char *__rte_experimental
+rte_comp_get_feature_name(uint64_t flag)
+{
+	switch (flag) {
+	case RTE_COMP_FF_STATEFUL_COMPRESSION:
+		return "STATEFUL_COMPRESSION";
+	case RTE_COMP_FF_STATEFUL_DECOMPRESSION:
+		return "STATEFUL_DECOMPRESSION";
+	case RTE_COMP_FF_MBUF_SCATTER_GATHER:
+		return "MBUF_SCATTER_GATHER";
+	case RTE_COMP_FF_MULTI_PKT_CHECKSUM:
+		return "MULTI_PKT_CHECKSUM";
+	case RTE_COMP_FF_ADLER32_CHECKSUM:
+		return "ADLER32_CHECKSUM";
+	case RTE_COMP_FF_CRC32_CHECKSUM:
+		return "CRC32_CHECKSUM";
+	case RTE_COMP_FF_CRC32_ADLER32_CHECKSUM:
+		return "CRC32_ADLER32_CHECKSUM";
+	case RTE_COMP_FF_NONCOMPRESSED_BLOCKS:
+		return "NONCOMPRESSED_BLOCKS";
+	default:
+		return NULL;
+	}
+}
+
+struct rte_compressdev *__rte_experimental
+rte_compressdev_pmd_get_dev(uint8_t dev_id)
+{
+	return &rte_compressdev_globals->devs[dev_id];
+}
+
+struct rte_compressdev *__rte_experimental
+rte_compressdev_pmd_get_named_dev(const char *name)
+{
+	struct rte_compressdev *dev;
+	unsigned int i;
+
+	if (name == NULL)
+		return NULL;
+
+	for (i = 0; i < rte_compressdev_globals->max_devs; i++) {
+		dev = &rte_compressdev_globals->devs[i];
+
+		if ((dev->attached == RTE_COMPRESSDEV_ATTACHED) &&
+				(strcmp(dev->data->name, name) == 0))
+			return dev;
+	}
+
+	return NULL;
+}
+
+unsigned int __rte_experimental
+rte_compressdev_pmd_is_valid_dev(uint8_t dev_id)
+{
+	struct rte_compressdev *dev = NULL;
+
+	if (dev_id >= rte_compressdev_globals->nb_devs)
+		return 0;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+	if (dev->attached != RTE_COMPRESSDEV_ATTACHED)
+		return 0;
+	else
+		return 1;
+}
+
+
+int __rte_experimental
+rte_compressdev_get_dev_id(const char *name)
+{
+	unsigned int i;
+
+	if (name == NULL)
+		return -1;
+
+	for (i = 0; i < rte_compressdev_globals->nb_devs; i++)
+		if ((strcmp(rte_compressdev_globals->devs[i].data->name, name)
+				== 0) &&
+				(rte_compressdev_globals->devs[i].attached ==
+						RTE_COMPRESSDEV_ATTACHED))
+			return i;
+
+	return -1;
+}
+
+uint8_t __rte_experimental
+rte_compressdev_count(void)
+{
+	return rte_compressdev_globals->nb_devs;
+}
+
+uint8_t __rte_experimental
+rte_compressdev_device_count_by_driver(uint8_t driver_id)
+{
+	uint8_t i, dev_count = 0;
+
+	for (i = 0; i < rte_compressdev_globals->max_devs; i++)
+		if (rte_compressdev_globals->devs[i].driver_id == driver_id &&
+			rte_compressdev_globals->devs[i].attached ==
+					RTE_COMPRESSDEV_ATTACHED)
+			dev_count++;
+
+	return dev_count;
+}
+
+uint8_t __rte_experimental
+rte_compressdev_devices_get(const char *driver_name, uint8_t *devices,
+	uint8_t nb_devices)
+{
+	uint8_t i, count = 0;
+	struct rte_compressdev *devs = rte_compressdev_globals->devs;
+	uint8_t max_devs = rte_compressdev_globals->max_devs;
+
+	for (i = 0; i < max_devs && count < nb_devices;	i++) {
+
+		if (devs[i].attached == RTE_COMPRESSDEV_ATTACHED) {
+			int cmp;
+
+			cmp = strncmp(devs[i].device->driver->name,
+					driver_name,
+					strlen(driver_name));
+
+			if (cmp == 0)
+				devices[count++] = devs[i].data->dev_id;
+		}
+	}
+
+	return count;
+}
+
+
+int __rte_experimental
+rte_compressdev_socket_id(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_pmd_is_valid_dev(dev_id))
+		return -1;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	return dev->data->socket_id;
+}
+
+static inline int
+rte_compressdev_data_alloc(uint8_t dev_id, struct rte_compressdev_data **data,
+		int socket_id)
+{
+	char mz_name[RTE_COMPRESSDEV_NAME_MAX_LEN];
+	const struct rte_memzone *mz;
+	int n;
+
+	/* generate memzone name */
+	n = snprintf(mz_name, sizeof(mz_name),
+			"rte_compressdev_data_%u", dev_id);
+	if (n >= (int)sizeof(mz_name))
+		return -EINVAL;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		mz = rte_memzone_reserve(mz_name,
+				sizeof(struct rte_compressdev_data),
+				socket_id, 0);
+	} else
+		mz = rte_memzone_lookup(mz_name);
+
+	if (mz == NULL)
+		return -ENOMEM;
+
+	*data = mz->addr;
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		memset(*data, 0, sizeof(struct rte_compressdev_data));
+
+	return 0;
+}
+
+static uint8_t
+rte_compressdev_find_free_device_index(void)
+{
+	uint8_t dev_id;
+
+	for (dev_id = 0; dev_id < RTE_COMPRESS_MAX_DEVS; dev_id++) {
+		if (rte_comp_devices[dev_id].attached ==
+				RTE_COMPRESSDEV_DETACHED)
+			return dev_id;
+	}
+	return RTE_COMPRESS_MAX_DEVS;
+}
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_allocate(const char *name, int socket_id)
+{
+	struct rte_compressdev *compressdev;
+	uint8_t dev_id;
+
+	if (rte_compressdev_pmd_get_named_dev(name) != NULL) {
+		COMPRESSDEV_LOG(ERR,
+			"comp device with name %s already allocated!", name);
+		return NULL;
+	}
+
+	dev_id = rte_compressdev_find_free_device_index();
+	if (dev_id == RTE_COMPRESS_MAX_DEVS) {
+		COMPRESSDEV_LOG(ERR, "Reached maximum number of comp devices");
+		return NULL;
+	}
+
+	compressdev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (compressdev->data == NULL) {
+		struct rte_compressdev_data *compressdev_data =
+				compressdev_globals.data[dev_id];
+
+		int retval = rte_compressdev_data_alloc(dev_id,
+				&compressdev_data, socket_id);
+
+		if (retval < 0 || compressdev_data == NULL)
+			return NULL;
+
+		compressdev->data = compressdev_data;
+
+		snprintf(compressdev->data->name, RTE_COMPRESSDEV_NAME_MAX_LEN,
+				"%s", name);
+
+		compressdev->data->dev_id = dev_id;
+		compressdev->data->socket_id = socket_id;
+		compressdev->data->dev_started = 0;
+
+		compressdev->attached = RTE_COMPRESSDEV_ATTACHED;
+
+		compressdev_globals.nb_devs++;
+	}
+
+	return compressdev;
+}
+
+int __rte_experimental
+rte_compressdev_pmd_release_device(struct rte_compressdev *compressdev)
+{
+	int ret;
+
+	if (compressdev == NULL)
+		return -EINVAL;
+
+	/* Close device only if device operations have been set */
+	if (compressdev->dev_ops) {
+		ret = rte_compressdev_close(compressdev->data->dev_id);
+		if (ret < 0)
+			return ret;
+	}
+
+	compressdev->attached = RTE_COMPRESSDEV_DETACHED;
+	compressdev_globals.nb_devs--;
+	return 0;
+}
+
+uint16_t __rte_experimental
+rte_compressdev_queue_pair_count(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	dev = &rte_comp_devices[dev_id];
+	return dev->data->nb_queue_pairs;
+}
+
+static int
+rte_compressdev_queue_pairs_config(struct rte_compressdev *dev,
+		uint16_t nb_qpairs, int socket_id)
+{
+	struct rte_compressdev_info dev_info;
+	void **qp;
+	unsigned int i;
+
+	if ((dev == NULL) || (nb_qpairs < 1)) {
+		COMPRESSDEV_LOG(ERR, "invalid param: dev %p, nb_queues %u",
+							dev, nb_qpairs);
+		return -EINVAL;
+	}
+
+	COMPRESSDEV_LOG(DEBUG, "Setup %d queues pairs on device %u",
+			nb_qpairs, dev->data->dev_id);
+
+	memset(&dev_info, 0, sizeof(struct rte_compressdev_info));
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_infos_get, -ENOTSUP);
+	(*dev->dev_ops->dev_infos_get)(dev, &dev_info);
+
+	if (nb_qpairs > (dev_info.max_nb_queue_pairs)) {
+		COMPRESSDEV_LOG(ERR, "Invalid num queue_pairs (%u) for dev %u",
+				nb_qpairs, dev->data->dev_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->queue_pairs == NULL) { /* first time configuration */
+		dev->data->queue_pairs = rte_zmalloc_socket(
+				"compressdev->queue_pairs",
+				sizeof(dev->data->queue_pairs[0]) * nb_qpairs,
+				RTE_CACHE_LINE_SIZE, socket_id);
+
+		if (dev->data->queue_pairs == NULL) {
+			dev->data->nb_queue_pairs = 0;
+			COMPRESSDEV_LOG(ERR,
+			"failed to get memory for qp meta data, nb_queues %u",
+							nb_qpairs);
+			return -(ENOMEM);
+		}
+	} else { /* re-configure */
+		int ret;
+		uint16_t old_nb_queues = dev->data->nb_queue_pairs;
+
+		qp = dev->data->queue_pairs;
+
+		RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->queue_pair_release,
+				-ENOTSUP);
+
+		for (i = nb_qpairs; i < old_nb_queues; i++) {
+			ret = (*dev->dev_ops->queue_pair_release)(dev, i);
+			if (ret < 0)
+				return ret;
+		}
+
+		qp = rte_realloc(qp, sizeof(qp[0]) * nb_qpairs,
+				RTE_CACHE_LINE_SIZE);
+		if (qp == NULL) {
+			COMPRESSDEV_LOG(ERR,
+			"failed to realloc qp meta data, nb_queues %u",
+						nb_qpairs);
+			return -(ENOMEM);
+		}
+
+		if (nb_qpairs > old_nb_queues) {
+			uint16_t new_qs = nb_qpairs - old_nb_queues;
+
+			memset(qp + old_nb_queues, 0,
+				sizeof(qp[0]) * new_qs);
+		}
+
+		dev->data->queue_pairs = qp;
+
+	}
+	dev->data->nb_queue_pairs = nb_qpairs;
+	return 0;
+}
+
+int __rte_experimental
+rte_compressdev_configure(uint8_t dev_id, struct rte_compressdev_config *config)
+{
+	struct rte_compressdev *dev;
+	int diag;
+
+	if (!rte_compressdev_pmd_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	if (dev->data->dev_started) {
+		COMPRESSDEV_LOG(ERR,
+		    "device %d must be stopped to allow configuration", dev_id);
+		return -EBUSY;
+	}
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_configure, -ENOTSUP);
+
+	/* Setup new number of queue pairs and reconfigure device. */
+	diag = rte_compressdev_queue_pairs_config(dev, config->nb_queue_pairs,
+			config->socket_id);
+	if (diag != 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev%d rte_comp_dev_queue_pairs_config = %d",
+				dev_id, diag);
+		return diag;
+	}
+
+	return (*dev->dev_ops->dev_configure)(dev, config);
+}
+
+
+int __rte_experimental
+rte_compressdev_start(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+	int diag;
+
+	COMPRESSDEV_LOG(DEBUG, "Start dev_id=%" PRIu8, dev_id);
+
+	if (!rte_compressdev_pmd_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_start, -ENOTSUP);
+
+	if (dev->data->dev_started != 0) {
+		COMPRESSDEV_LOG(ERR,
+		    "Device with dev_id=%" PRIu8 " already started", dev_id);
+		return 0;
+	}
+
+	diag = (*dev->dev_ops->dev_start)(dev);
+	if (diag == 0)
+		dev->data->dev_started = 1;
+	else
+		return diag;
+
+	return 0;
+}
+
+void __rte_experimental
+rte_compressdev_stop(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_pmd_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_stop);
+
+	if (dev->data->dev_started == 0) {
+		COMPRESSDEV_LOG(ERR,
+		    "Device with dev_id=%" PRIu8 " already stopped", dev_id);
+		return;
+	}
+
+	(*dev->dev_ops->dev_stop)(dev);
+	dev->data->dev_started = 0;
+}
+
+int __rte_experimental
+rte_compressdev_close(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+	int retval;
+
+	if (!rte_compressdev_pmd_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -1;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	/* Device must be stopped before it can be closed */
+	if (dev->data->dev_started == 1) {
+		COMPRESSDEV_LOG(ERR, "Device %u must be stopped before closing",
+				dev_id);
+		return -EBUSY;
+	}
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_close, -ENOTSUP);
+	retval = (*dev->dev_ops->dev_close)(dev);
+
+	if (retval < 0)
+		return retval;
+
+	return 0;
+}
+
+int __rte_experimental
+rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
+		uint32_t max_inflight_ops, int socket_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_pmd_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+	if (queue_pair_id >= dev->data->nb_queue_pairs) {
+		COMPRESSDEV_LOG(ERR, "Invalid queue_pair_id=%d", queue_pair_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->dev_started) {
+		COMPRESSDEV_LOG(ERR,
+		    "device %d must be stopped to allow configuration", dev_id);
+		return -EBUSY;
+	}
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->queue_pair_setup, -ENOTSUP);
+
+	return (*dev->dev_ops->queue_pair_setup)(dev, queue_pair_id,
+			max_inflight_ops, socket_id);
+}
+
+
+int __rte_experimental
+rte_compressdev_stats_get(uint8_t dev_id, struct rte_compressdev_stats *stats)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_pmd_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%d", dev_id);
+		return -ENODEV;
+	}
+
+	if (stats == NULL) {
+		COMPRESSDEV_LOG(ERR, "Invalid stats ptr");
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+	memset(stats, 0, sizeof(*stats));
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->stats_get, -ENOTSUP);
+	(*dev->dev_ops->stats_get)(dev, stats);
+	return 0;
+}
+
+void __rte_experimental
+rte_compressdev_stats_reset(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_pmd_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	RTE_FUNC_PTR_OR_RET(*dev->dev_ops->stats_reset);
+	(*dev->dev_ops->stats_reset)(dev);
+}
+
+
+void __rte_experimental
+rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
+{
+	struct rte_compressdev *dev;
+
+	if (dev_id >= compressdev_globals.nb_devs) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%d", dev_id);
+		return;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	memset(dev_info, 0, sizeof(struct rte_compressdev_info));
+
+	RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
+	(*dev->dev_ops->dev_infos_get)(dev, dev_info);
+
+	dev_info->driver_name = dev->device->driver->name;
+}
+
+int __rte_experimental
+rte_compressdev_private_xform_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **priv_xform)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (xform == NULL || priv_xform == NULL || dev == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->private_xform_create, -ENOTSUP);
+	ret = (*dev->dev_ops->private_xform_create)(dev, xform, priv_xform);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to create private_xform: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
+int __rte_experimental
+rte_compressdev_private_xform_free(uint8_t dev_id, void *priv_xform)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (dev == NULL || priv_xform == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->private_xform_free, -ENOTSUP);
+	ret = dev->dev_ops->private_xform_free(dev, priv_xform);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to free private xform: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
+int __rte_experimental
+rte_compressdev_stream_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **stream)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (xform == NULL || dev == NULL || stream == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->stream_create, -ENOTSUP);
+	ret = (*dev->dev_ops->stream_create)(dev, xform, stream);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to create stream: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
+
+int __rte_experimental
+rte_compressdev_stream_free(uint8_t dev_id, void *stream)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (dev == NULL || stream == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->stream_free, -ENOTSUP);
+	ret = dev->dev_ops->stream_free(dev, stream);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to free stream: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
+
+/** Initialise rte_comp_op mempool element */
+static void
+rte_comp_op_init(struct rte_mempool *mempool,
+		__rte_unused void *opaque_arg,
+		void *_op_data,
+		__rte_unused unsigned int i)
+{
+	struct rte_comp_op *op = _op_data;
+
+	memset(_op_data, 0, mempool->elt_size);
+
+	op->status = RTE_COMP_OP_STATUS_NOT_PROCESSED;
+	op->phys_addr = rte_mem_virt2iova(_op_data);
+	op->mempool = mempool;
+}
+
+
+struct rte_mempool * __rte_experimental
+rte_comp_op_pool_create(const char *name,
+		unsigned int nb_elts, unsigned int cache_size,
+		uint16_t user_size, int socket_id)
+{
+	struct rte_comp_op_pool_private *priv;
+
+	unsigned int elt_size = sizeof(struct rte_comp_op) + user_size;
+
+	/* lookup mempool in case already allocated */
+	struct rte_mempool *mp = rte_mempool_lookup(name);
+
+	if (mp != NULL) {
+		priv = (struct rte_comp_op_pool_private *)
+				rte_mempool_get_priv(mp);
+
+		if (mp->elt_size != elt_size ||
+				mp->cache_size < cache_size ||
+				mp->size < nb_elts ||
+				priv->user_size <  user_size) {
+			mp = NULL;
+			COMPRESSDEV_LOG(ERR,
+		"Mempool %s already exists but with incompatible parameters",
+					name);
+			return NULL;
+		}
+		return mp;
+	}
+
+	mp = rte_mempool_create(
+			name,
+			nb_elts,
+			elt_size,
+			cache_size,
+			sizeof(struct rte_comp_op_pool_private),
+			NULL,
+			NULL,
+			rte_comp_op_init,
+			NULL,
+			socket_id,
+			0);
+
+	if (mp == NULL) {
+		COMPRESSDEV_LOG(ERR, "Failed to create mempool %s", name);
+		return NULL;
+	}
+
+	priv = (struct rte_comp_op_pool_private *)
+			rte_mempool_get_priv(mp);
+
+	priv->user_size = user_size;
+
+	return mp;
+}
+
+TAILQ_HEAD(compressdev_driver_list, compressdev_driver);
+
+static struct compressdev_driver_list compressdev_driver_list =
+	TAILQ_HEAD_INITIALIZER(compressdev_driver_list);
+
+int __rte_experimental
+rte_compressdev_driver_id_get(const char *name)
+{
+	struct compressdev_driver *driver;
+	const char *driver_name;
+
+	if (name == NULL) {
+		COMPRESSDEV_LOG(DEBUG, "name pointer NULL");
+		return -1;
+	}
+
+	TAILQ_FOREACH(driver, &compressdev_driver_list, next) {
+		driver_name = driver->driver->name;
+		if (strncmp(driver_name, name, strlen(driver_name)) == 0)
+			return driver->id;
+	}
+	return -1;
+}
+
+const char * __rte_experimental
+rte_compressdev_name_get(uint8_t dev_id)
+{
+	struct rte_compressdev *dev = rte_compressdev_pmd_get_dev(dev_id);
+
+	if (dev == NULL)
+		return NULL;
+
+	return dev->data->name;
+}
+
+const char * __rte_experimental
+rte_compressdev_driver_name_get(uint8_t driver_id)
+{
+	struct compressdev_driver *driver;
+
+	TAILQ_FOREACH(driver, &compressdev_driver_list, next)
+		if (driver->id == driver_id)
+			return driver->driver->name;
+	return NULL;
+}
+
+uint8_t
+rte_compressdev_allocate_driver(struct compressdev_driver *comp_drv,
+		const struct rte_driver *drv)
+{
+	comp_drv->driver = drv;
+	comp_drv->id = nb_drivers;
+
+	TAILQ_INSERT_TAIL(&compressdev_driver_list, comp_drv, next);
+
+	return nb_drivers++;
+}
+
+RTE_INIT(rte_compressdev_log);
+
+static void
+rte_compressdev_log(void)
+{
+	compressdev_logtype = rte_log_register("librte.compressdev");
+	if (compressdev_logtype >= 0)
+		rte_log_set_level(compressdev_logtype, RTE_LOG_NOTICE);
+}
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
new file mode 100644
index 0000000..5c98d14
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -0,0 +1,727 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMPRESSDEV_H_
+#define _RTE_COMPRESSDEV_H_
+
+/**
+ * @file rte_compressdev.h
+ *
+ * RTE Compression Device APIs
+ *
+ * Defines comp device APIs for the provisioning of compression operations.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include "rte_kvargs.h"
+#include "rte_comp.h"
+#include "rte_dev.h"
+#include <rte_common.h>
+
+/* Logging Macros */
+extern int compressdev_logtype;
+#define COMPRESSDEV_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, compressdev_logtype, "%s(): "fmt "\n", \
+			__func__, ##args)
+
+#define RTE_COMPRESSDEV_DETACHED  (0)
+#define RTE_COMPRESSDEV_ATTACHED  (1)
+
+#define RTE_COMPRESSDEV_NAME_MAX_LEN	(64)
+
+/**< Max length of name of comp PMD */
+/**
+ * A macro that points to an offset from the start
+ * of the comp operation structure (rte_comp_op)
+ *
+ * The returned pointer is cast to type t.
+ *
+ * @param c
+ *   The comp operation
+ * @param o
+ *   The offset from the start of the comp operation
+ * @param t
+ *   The type to cast the result into
+ */
+#define rte_comp_op_ctod_offset(c, t, o)	\
+	((t)((char *)(c) + (o)))
+
+/**
+ * A macro that returns the physical address that points
+ * to an offset from the start of the comp operation
+ * (rte_comp_op).
+ *
+ * @param c
+ *   The comp operation
+ * @param o
+ *   The offset from the start of the comp operation
+ *   to calculate address from
+ */
+#define rte_comp_op_ctophys_offset(c, o)	\
+	(rte_iova_t)((c)->phys_addr + (o))
+
+/**
+ * Parameter log base 2 range description.
+ * Final value will be 2^value.
+ */
+struct rte_param_log2_range {
+	uint8_t min;	/**< Minimum log2 value */
+	uint8_t max;	/**< Maximum log2 value */
+	uint8_t increment;
+	/**< If a range of sizes are supported,
+	 * this parameter is used to indicate
+	 * increments in base 2 log byte value
+	 * that are supported between the minimum and maximum
+	 */
+};
+
+/** Structure used to capture a capability of a comp device */
+struct rte_compressdev_capabilities {
+	enum rte_comp_algorithm algo;
+	/* Compression algorithm */
+	uint64_t comp_feature_flags;
+	/**< Bitmask of flags for compression service features */
+	struct rte_param_log2_range window_size;
+	/**< Window size range in base two log byte values */
+};
+
+
+/** Macro used at end of comp PMD list */
+#define RTE_COMP_END_OF_CAPABILITIES_LIST() \
+	{ RTE_COMP_ALGO_UNSPECIFIED }
+
+const struct rte_compressdev_capabilities * __rte_experimental
+rte_compressdev_capability_get(uint8_t dev_id,
+			enum rte_comp_algorithm algo);
+
+/**
+ * compression device supported feature flags
+ *
+ * @note New features flags should be added to the end of the list
+ *
+ * Keep these flags synchronised with rte_compressdev_get_feature_name()
+ */
+
+#define	RTE_COMPDEV_FF_HW_ACCELERATED		(1ULL << 0)
+/**< Operations are off-loaded to an external hardware accelerator */
+#define	RTE_COMPDEV_FF_CPU_SSE			(1ULL << 1)
+/**< Utilises CPU SIMD SSE instructions */
+#define	RTE_COMPDEV_FF_CPU_AVX			(1ULL << 2)
+/**< Utilises CPU SIMD AVX instructions */
+#define	RTE_COMPDEV_FF_CPU_AVX2			(1ULL << 3)
+/**< Utilises CPU SIMD AVX2 instructions */
+#define	RTE_COMPDEV_FF_CPU_AVX512		(1ULL << 4)
+/**< Utilises CPU SIMD AVX512 instructions */
+#define	RTE_COMPDEV_FF_CPU_NEON			(1ULL << 5)
+/**< Utilises CPU NEON instructions */
+
+/**
+ * compression service feature flags
+ *
+ * @note New features flags should be added to the end of the list
+ *
+ * Keep these flags synchronised with rte_comp_get_feature_name()
+ */
+#define RTE_COMP_FF_STATEFUL_COMPRESSION	(1ULL << 0)
+/**< Stateful compression is supported */
+#define RTE_COMP_FF_STATEFUL_DECOMPRESSION	(1ULL << 1)
+/**< Stateful decompression is supported */
+#define	RTE_COMP_FF_MBUF_SCATTER_GATHER		(1ULL << 2)
+/**< Scatter-gather mbufs are supported */
+#define RTE_COMP_FF_ADLER32_CHECKSUM		(1ULL << 3)
+/**< Adler-32 Checksum is supported */
+#define RTE_COMP_FF_CRC32_CHECKSUM		(1ULL << 4)
+/**< CRC32 Checksum is supported */
+#define RTE_COMP_FF_CRC32_ADLER32_CHECKSUM	(1ULL << 5)
+/**< Adler-32/CRC32 Checksum is supported */
+#define RTE_COMP_FF_MULTI_PKT_CHECKSUM		(1ULL << 6)
+/**< Generation of checksum across multiple stateless packets is supported */
+#define RTE_COMP_FF_NONCOMPRESSED_BLOCKS	(1ULL << 7)
+/**< Creation of non-compressed blocks using RTE_COMP_LEVEL_NONE is supported */
+
+/**
+ * Get the name of a compress device feature flag.
+ *
+ * @param flag
+ *   The mask describing the flag
+ *
+ * @return
+ *   The name of this flag, or NULL if it's not a valid feature flag.
+ */
+const char * __rte_experimental
+rte_compressdev_get_feature_name(uint64_t flag);
+
+/**
+ * Get the name of a compress service feature flag
+ *
+ * @param flag
+ *   The mask describing the flag
+ *
+ * @return
+ *   The name of this flag, or NULL if it's not a valid feature flag.
+ */
+const char * __rte_experimental
+rte_comp_get_feature_name(uint64_t flag);
+
+/**  comp device information */
+struct rte_compressdev_info {
+	const char *driver_name;		/**< Driver name. */
+	uint8_t driver_id;			/**< Driver identifier */
+	uint64_t feature_flags;			/**< Feature flags */
+	const struct rte_compressdev_capabilities *capabilities;
+	/**< Array of devices supported capabilities */
+	unsigned int max_nb_queue_pairs;
+	/**< Maximum number of queues pairs supported by device. */
+};
+
+
+/** comp device statistics */
+struct rte_compressdev_stats {
+	uint64_t enqueued_count;
+	/**< Count of all operations enqueued */
+	uint64_t dequeued_count;
+	/**< Count of all operations dequeued */
+
+	uint64_t enqueue_err_count;
+	/**< Total error count on operations enqueued */
+	uint64_t dequeue_err_count;
+	/**< Total error count on operations dequeued */
+};
+
+
+/**
+ * Get the device identifier for the named compress device.
+ *
+ * @param name
+ *   Device name to select the device structure
+ * @return
+ *   - Returns compress device identifier on success.
+ *   - Return -1 on failure to find named compress device.
+ */
+int __rte_experimental
+rte_compressdev_get_dev_id(const char *name);
+
+/**
+ * Get the compress device name given a device identifier.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - Returns compress device name.
+ *   - Returns NULL if compress device is not present.
+ */
+const char * __rte_experimental
+rte_compressdev_name_get(uint8_t dev_id);
+
+/**
+ * Get the total number of compress devices that have been successfully
+ * initialised.
+ *
+ * @return
+ *   - The total number of usable compress devices.
+ */
+uint8_t __rte_experimental
+rte_compressdev_count(void);
+
+/**
+ * Get number of comp device defined type.
+ *
+ * @param driver_id
+ *   Driver identifier
+ * @return
+ *   Returns number of comp device.
+ */
+uint8_t __rte_experimental
+rte_compressdev_device_count_by_driver(uint8_t driver_id);
+
+/**
+ * Get number and identifiers of attached comp devices that
+ * use the same compress driver.
+ *
+ * @param driver_name
+ *   Driver name
+ * @param devices
+ *   Output devices identifiers
+ * @param nb_devices
+ *   Maximal number of devices
+ *
+ * @return
+ *   Returns number of attached compress devices.
+ */
+uint8_t __rte_experimental
+rte_compressdev_devices_get(const char *driver_name, uint8_t *devices,
+		uint8_t nb_devices);
+/*
+ * Return the NUMA socket to which a device is connected.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   The NUMA socket id to which the device is connected or
+ *   a default of zero if the socket could not be determined.
+ *   -1 if returned is the dev_id value is out of range.
+ */
+int __rte_experimental
+rte_compressdev_socket_id(uint8_t dev_id);
+
+/** Compress device configuration structure */
+struct rte_compressdev_config {
+	int socket_id;
+	/**< Socket on which to allocate resources */
+	uint16_t nb_queue_pairs;
+	/**< Total number of queue pairs to configure on a device */
+	uint16_t max_nb_priv_xforms;
+	/**< Max number of private_xforms which will be created on the device */
+	uint16_t max_nb_streams;
+	/**< Max number of streams which will be created on the device */
+};
+
+/**
+ * Configure a device.
+ *
+ * This function must be invoked first before any other function in the
+ * API. This function can also be re-invoked when a device is in the
+ * stopped state.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param config
+ *   The compress device configuration
+ * @return
+ *   - 0: Success, device configured.
+ *   - <0: Error code returned by the driver configuration function.
+ */
+int __rte_experimental
+rte_compressdev_configure(uint8_t dev_id,
+			struct rte_compressdev_config *config);
+
+/**
+ * Start a device.
+ *
+ * The device start step is called after configuring the device and setting up
+ * its queue_pairs.
+ * On success, data-path functions exported by the API (enqueue/dequeue, etc)
+ * can be invoked.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - 0: Success, device started.
+ *   - <0: Error code of the driver device start function.
+ */
+int __rte_experimental
+rte_compressdev_start(uint8_t dev_id);
+
+/**
+ * Stop a device. The device can be restarted with a call to
+ * rte_compressdev_start()
+ *
+ * @param dev_id
+ *   Compress device identifier
+ */
+void __rte_experimental
+rte_compressdev_stop(uint8_t dev_id);
+
+/**
+ * Close an device. The device cannot be restarted!
+ *
+ * @param dev_id
+ *   Compress device identifier
+ *
+ * @return
+ *  - 0 on successfully closing device
+ *  - <0 on failure to close device
+ */
+int __rte_experimental
+rte_compressdev_close(uint8_t dev_id);
+
+/**
+ * Allocate and set up a receive queue pair for a device.
+ * This should only be called when the device is stopped.
+ *
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param queue_pair_id
+ *   The index of the queue pairs to set up. The
+ *   value must be in the range [0, nb_queue_pair - 1]
+ *   previously supplied to rte_compressdev_configure()
+ * @param max_inflight_ops
+ *   Max number of ops which the qp will have to
+ *   accommodate simultaneously
+ * @param socket_id
+ *   The *socket_id* argument is the socket identifier
+ *   in case of NUMA. The value can be *SOCKET_ID_ANY*
+ *   if there is no NUMA constraint for the DMA memory
+ *   allocated for the receive queue pair
+ * @return
+ *   - 0: Success, queue pair correctly set up.
+ *   - <0: Queue pair configuration failed
+ */
+int __rte_experimental
+rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
+		uint32_t max_inflight_ops, int socket_id);
+
+/**
+ * Get the number of queue pairs on a specific comp device
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - The number of configured queue pairs.
+ */
+uint16_t __rte_experimental
+rte_compressdev_queue_pair_count(uint8_t dev_id);
+
+
+/**
+ * Retrieve the general I/O statistics of a device.
+ *
+ * @param dev_id
+ *   The identifier of the device
+ * @param stats
+ *   A pointer to a structure of type
+ *   *rte_compressdev_stats* to be filled with the
+ *   values of device counters
+ * @return
+ *   - Zero if successful.
+ *   - Non-zero otherwise.
+ */
+int __rte_experimental
+rte_compressdev_stats_get(uint8_t dev_id, struct rte_compressdev_stats *stats);
+
+/**
+ * Reset the general I/O statistics of a device.
+ *
+ * @param dev_id
+ *   The identifier of the device.
+ */
+void __rte_experimental
+rte_compressdev_stats_reset(uint8_t dev_id);
+
+/**
+ * Retrieve the contextual information of a device.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param dev_info
+ *   A pointer to a structure of type *rte_compressdev_info*
+ *   to be filled with the contextual information of the device
+ *
+ * @note The capabilities field of dev_info is set to point to the first
+ * element of an array of struct rte_compressdev_capabilities.
+ * The element after the last valid element has it's op field set to
+ * RTE_COMP_ALGO_LIST_END.
+ */
+void __rte_experimental
+rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info);
+
+
+typedef uint16_t (*compress_dequeue_pkt_burst_t)(void *qp,
+		struct rte_comp_op **ops, uint16_t nb_ops);
+/**< Dequeue processed packets from queue pair of a device. */
+
+typedef uint16_t (*compress_enqueue_pkt_burst_t)(void *qp,
+		struct rte_comp_op **ops, uint16_t nb_ops);
+/**< Enqueue packets for processing on queue pair of a device. */
+
+
+/** The data structure associated with each comp device. */
+struct rte_compressdev {
+	compress_dequeue_pkt_burst_t dequeue_burst;
+	/**< Pointer to PMD receive function */
+	compress_enqueue_pkt_burst_t enqueue_burst;
+	/**< Pointer to PMD transmit function */
+
+	struct rte_compressdev_data *data;
+	/**< Pointer to device data */
+	struct rte_compressdev_ops *dev_ops;
+	/**< Functions exported by PMD */
+	uint64_t feature_flags;
+	/**< Supported features */
+	struct rte_device *device;
+	/**< Backing device */
+
+	uint8_t driver_id;
+	/**< comp driver identifier*/
+
+	__extension__
+	uint8_t attached : 1;
+	/**< Flag indicating the device is attached */
+} __rte_cache_aligned;
+
+
+/**
+ *
+ * The data part, with no function pointers, associated with each device.
+ *
+ * This structure is safe to place in shared memory to be common among
+ * different processes in a multi-process configuration.
+ */
+struct rte_compressdev_data {
+	uint8_t dev_id;
+	/**< Compress device identifier */
+	uint8_t socket_id;
+	/**< Socket identifier where memory is allocated */
+	char name[RTE_COMPRESSDEV_NAME_MAX_LEN];
+	/**< Unique identifier name */
+
+	__extension__
+	uint8_t dev_started : 1;
+	/**< Device state: STARTED(1)/STOPPED(0) */
+
+	void **queue_pairs;
+	/**< Array of pointers to queue pairs. */
+	uint16_t nb_queue_pairs;
+	/**< Number of device queue pairs */
+
+	void *dev_private;
+	/**< PMD-specific private data */
+} __rte_cache_aligned;
+
+struct rte_compressdev *rte_compressdevs;
+/**
+ *
+ * Dequeue a burst of processed compression operations from a queue on the comp
+ * device. The dequeued operation are stored in *rte_comp_op* structures
+ * whose pointers are supplied in the *ops* array.
+ *
+ * The rte_compressdev_dequeue_burst() function returns the number of ops
+ * actually dequeued, which is the number of *rte_comp_op* data structures
+ * effectively supplied into the *ops* array.
+ *
+ * A return value equal to *nb_ops* indicates that the queue contained
+ * at least *nb_ops* operations, and this is likely to signify that other
+ * processed operations remain in the devices output queue. Applications
+ * implementing a "retrieve as many processed operations as possible" policy
+ * can check this specific case and keep invoking the
+ * rte_compressdev_dequeue_burst() function until a value less than
+ * *nb_ops* is returned.
+ *
+ * The rte_compressdev_dequeue_burst() function does not provide any error
+ * notification to avoid the corresponding overhead.
+ *
+ * @note: operation ordering is not maintained within the queue pair.
+ *
+ * @note: In case op status = OUT_OF_SPACE_TERMINATED, op.consumed=0 and the
+ * op must be resubmitted with the same input data and a larger output buffer.
+ * op.produced is usually 0, but in decompression cases a PMD may return > 0
+ * and the application may find it useful to inspect that data.
+ * This status is only returned on STATELESS ops.
+ *
+ * @note: In case op status = OUT_OF_SPACE_RECOVERABLE, op.produced can be used
+ * and next op in stream should continue on from op.consumed+1 with a fresh
+ * output buffer.
+ * Consumed=0, produced=0 is an unusual but allowed case. There may be useful
+ * state/history stored in the PMD, even though no output was produced yet.
+ *
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param qp_id
+ *   The index of the queue pair from which to retrieve
+ *   processed operations. The value must be in the range
+ *   [0, nb_queue_pair - 1] previously supplied to
+ *   rte_compressdev_configure()
+ * @param ops
+ *   The address of an array of pointers to
+ *   *rte_comp_op* structures that must be
+ *   large enough to store *nb_ops* pointers in it
+ * @param nb_ops
+ *   The maximum number of operations to dequeue
+ * @return
+ *   - The number of operations actually dequeued, which is the number
+ *   of pointers to *rte_comp_op* structures effectively supplied to the
+ *   *ops* array.
+ */
+static inline uint16_t
+rte_compressdev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	struct rte_compressdev *dev = &rte_compressdevs[dev_id];
+
+	nb_ops = (*dev->dequeue_burst)
+			(dev->data->queue_pairs[qp_id], ops, nb_ops);
+
+	return nb_ops;
+}
+
+/**
+ * Enqueue a burst of operations for processing on a compression device.
+ *
+ * The rte_compressdev_enqueue_burst() function is invoked to place
+ * comp operations on the queue *qp_id* of the device designated by
+ * its *dev_id*.
+ *
+ * The *nb_ops* parameter is the number of operations to process which are
+ * supplied in the *ops* array of *rte_comp_op* structures.
+ *
+ * The rte_compressdev_enqueue_burst() function returns the number of
+ * operations it actually enqueued for processing. A return value equal to
+ * *nb_ops* means that all packets have been enqueued.
+ *
+ * @note All compression operations are Out-of-place (OOP) operations,
+ * as the size of the output data is different to the size of the input data.
+ *
+ * @note The flush flag only applies to operations which return SUCCESS.
+ * In OUT_OF_SPACE cases whether STATEFUL or STATELESS, data in dest buffer
+ * is as if flush flag was FLUSH_NONE.
+ * @note flush flag only applies in compression direction. It has no meaning
+ * for decompression.
+ * @note: operation ordering is not maintained within the queue pair.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param qp_id
+ *   The index of the queue pair on which operations
+ *   are to be enqueued for processing. The value
+ *   must be in the range [0, nb_queue_pairs - 1]
+ *   previously supplied to *rte_compressdev_configure*
+ * @param ops
+ *   The address of an array of *nb_ops* pointers
+ *   to *rte_comp_op* structures which contain
+ *   the operations to be processed
+ * @param nb_ops
+ *   The number of operations to process
+ * @return
+ *   The number of operations actually enqueued on the device. The return
+ *   value can be less than the value of the *nb_ops* parameter when the
+ *   comp devices queue is full or if invalid parameters are specified in
+ *   a *rte_comp_op*.
+ */
+static inline uint16_t
+rte_compressdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	struct rte_compressdev *dev = &rte_compressdevs[dev_id];
+
+	return (*dev->enqueue_burst)(
+			dev->data->queue_pairs[qp_id], ops, nb_ops);
+}
+
+
+/**
+ * This should alloc a stream from the device's mempool and initialise it.
+ * The application should call this API when setting up for the stateful
+ * processing of a set of data on a device. The API can be called multiple
+ * times to set up a stream for each data set. The handle returned is only for
+ * use with ops of op_type STATEFUL and must be passed to the PMD
+ * with every op in the data stream
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param xform
+ *   xform data
+ * @param stream
+ *   Pointer to where PMD's private stream handle should be stored
+ *
+ * @return
+ *  - 0 if successful and valid stream handle
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private stream could not be allocated.
+ *
+ */
+int __rte_experimental
+rte_compressdev_stream_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **stream);
+
+/**
+ * This should clear the stream and return it to the device's mempool.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ *
+ * @param stream
+ *   PMD's private stream data
+ *
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -EBUSY if can't free stream as there are inflight operations
+ */
+int __rte_experimental
+rte_compressdev_stream_free(uint8_t dev_id, void *stream);
+
+/**
+ * This should alloc a private_xform from the device's mempool and initialise
+ * it. The application should call this API when setting up for stateless
+ * processing on a device. If it returns non-shareable, then the appl cannot
+ * share this handle with multiple in-flight ops and should call this API again
+ * to get a separate handle for every in-flight op.
+ * The handle returned is only valid for use with ops of op_type STATELESS.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param xform
+ *   xform data
+ * @param private_xform
+ *   Pointer to where PMD's private_xform handle should be stored
+ *
+ * @return
+ *  - if successful returns RTE_COMP_PRIV_XFORM_SHAREABLE/NOT_SHAREABLE
+ *    and valid private_xform handle
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private_xform could not be allocated.
+ */
+int __rte_experimental
+rte_compressdev_private_xform_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **private_xform);
+
+/**
+ * This should clear the private_xform and return it to the device's mempool.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ *
+ * @param private_xform
+ *   PMD's private_xform data
+ *
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -EBUSY if can't free private_xform due to inflight operations
+ */
+int __rte_experimental
+rte_compressdev_private_xform_free(uint8_t dev_id, void *private_xform);
+
+/**
+ * Provide driver identifier.
+ *
+ * @param name
+ *   Compress driver name
+ * @return
+ *  The driver type identifier or -1 if no driver found
+ */
+int __rte_experimental
+rte_compressdev_driver_id_get(const char *name);
+
+/**
+ * Provide driver name.
+ *
+ * @param driver_id
+ *   The driver identifier
+ * @return
+ *  The driver name or null if no driver found
+ */
+const char * __rte_experimental
+rte_compressdev_driver_name_get(uint8_t driver_id);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_COMPRESSDEV_H_ */
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.c b/lib/librte_compressdev/rte_compressdev_pmd.c
new file mode 100644
index 0000000..1e9731f
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_pmd.c
@@ -0,0 +1,156 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#include <rte_malloc.h>
+
+#include "rte_compressdev_pmd.h"
+
+int compressdev_logtype;
+
+/**
+ * Parse name from argument
+ */
+static int
+rte_compressdev_pmd_parse_name_arg(const char *key __rte_unused,
+		const char *value, void *extra_args)
+{
+	struct rte_compressdev_pmd_init_params *params = extra_args;
+	int n;
+
+	n = snprintf(params->name, RTE_COMPRESSDEV_NAME_MAX_LEN, "%s", value);
+	if (n >= RTE_COMPRESSDEV_NAME_MAX_LEN)
+		return -EINVAL;
+
+	return 0;
+}
+
+/**
+ * Parse unsigned integer from argument
+ */
+static int
+rte_compressdev_pmd_parse_uint_arg(const char *key __rte_unused,
+		const char *value, void *extra_args)
+{
+	int i;
+	char *end;
+
+	errno = 0;
+	i = strtol(value, &end, 10);
+	if (*end != 0 || errno != 0 || i < 0)
+		return -EINVAL;
+
+	*((uint32_t *)extra_args) = i;
+	return 0;
+}
+
+int
+rte_compressdev_pmd_parse_input_args(
+		struct rte_compressdev_pmd_init_params *params,
+		const char *args)
+{
+	struct rte_kvargs *kvlist = NULL;
+	int ret = 0;
+
+	if (params == NULL)
+		return -EINVAL;
+
+	if (args) {
+		kvlist = rte_kvargs_parse(args,	compressdev_pmd_valid_params);
+		if (kvlist == NULL)
+			return -EINVAL;
+
+		ret = rte_kvargs_process(kvlist,
+				RTE_COMPRESSDEV_PMD_SOCKET_ID_ARG,
+				&rte_compressdev_pmd_parse_uint_arg,
+				&params->socket_id);
+		if (ret < 0)
+			goto free_kvlist;
+
+		ret = rte_kvargs_process(kvlist,
+				RTE_COMPRESSDEV_PMD_NAME_ARG,
+				&rte_compressdev_pmd_parse_name_arg,
+				params);
+		if (ret < 0)
+			goto free_kvlist;
+	}
+
+free_kvlist:
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_create(const char *name,
+		struct rte_device *device,
+		struct rte_compressdev_pmd_init_params *params)
+{
+	struct rte_compressdev *compressdev;
+
+	if (params->name[0] != '\0') {
+		COMPRESSDEV_LOG(INFO, "[%s] User specified device name = %s\n",
+				device->driver->name, params->name);
+		name = params->name;
+	}
+
+	COMPRESSDEV_LOG(INFO, "[%s] - Creating compressdev %s\n",
+			device->driver->name, name);
+
+	COMPRESSDEV_LOG(INFO,
+	"[%s] - Init parameters - name: %s, socket id: %d",
+			device->driver->name, name,
+			params->socket_id);
+
+	/* allocate device structure */
+	compressdev = rte_compressdev_pmd_allocate(name, params->socket_id);
+	if (compressdev == NULL) {
+		COMPRESSDEV_LOG(ERR, "[%s] Failed to allocate comp device %s",
+				device->driver->name, name);
+		return NULL;
+	}
+
+	/* allocate private device structure */
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		compressdev->data->dev_private =
+				rte_zmalloc_socket("compressdev device private",
+						params->private_data_size,
+						RTE_CACHE_LINE_SIZE,
+						params->socket_id);
+
+		if (compressdev->data->dev_private == NULL) {
+			COMPRESSDEV_LOG(ERR,
+		"[%s] Cannot allocate memory for compressdev %s private data",
+					device->driver->name, name);
+
+			rte_compressdev_pmd_release_device(compressdev);
+			return NULL;
+		}
+	}
+
+	compressdev->device = device;
+
+	return compressdev;
+}
+
+int __rte_experimental
+rte_compressdev_pmd_destroy(struct rte_compressdev *compressdev)
+{
+	int retval;
+
+	COMPRESSDEV_LOG(INFO, "[%s] Closing comp device %s",
+			compressdev->device->driver->name,
+			compressdev->device->name);
+
+	/* free comp device */
+	retval = rte_compressdev_pmd_release_device(compressdev);
+	if (retval)
+		return retval;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		rte_free(compressdev->data->dev_private);
+
+	compressdev->device = NULL;
+	compressdev->data = NULL;
+
+	return 0;
+}
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
new file mode 100644
index 0000000..8b88b8e
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -0,0 +1,450 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMPRESSDEV_PMD_H_
+#define _RTE_COMPRESSDEV_PMD_H_
+
+/** @file
+ * RTE comp PMD APIs
+ *
+ * @note
+ * These APIs are for comp PMDs only and user applications should not call
+ * them directly.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <string.h>
+
+#include <rte_dev.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_mempool.h>
+#include <rte_log.h>
+#include <rte_common.h>
+
+#include "rte_comp.h"
+#include "rte_compressdev.h"
+
+#define RTE_COMPRESSDEV_PMD_NAME_ARG			("name")
+#define RTE_COMPRESSDEV_PMD_SOCKET_ID_ARG		("socket_id")
+
+static const char * const compressdev_pmd_valid_params[] = {
+	RTE_COMPRESSDEV_PMD_NAME_ARG,
+	RTE_COMPRESSDEV_PMD_SOCKET_ID_ARG
+};
+
+/**
+ * @internal
+ * Initialisation parameters for comp devices
+ */
+struct rte_compressdev_pmd_init_params {
+	char name[RTE_COMPRESSDEV_NAME_MAX_LEN];
+	size_t private_data_size;
+	int socket_id;
+};
+
+/** Global structure used for maintaining state of allocated comp devices */
+struct rte_compressdev_global {
+	struct rte_compressdev *devs;	/**< Device information array */
+	struct rte_compressdev_data *data[RTE_COMPRESS_MAX_DEVS];
+	/**< Device private data */
+	uint8_t nb_devs;		/**< Number of devices found */
+	uint8_t max_devs;		/**< Max number of devices */
+};
+
+/* compressdev driver, containing the driver identifier */
+struct compressdev_driver {
+	TAILQ_ENTRY(compressdev_driver) next; /**< Next in list. */
+	const struct rte_driver *driver;
+	uint8_t id;
+};
+
+/** Pointer to global array of comp devices */
+extern struct rte_compressdev *rte_compressdevs;
+/** Pointer to global comp devices data structure */
+extern struct rte_compressdev_global *rte_compressdev_globals;
+
+/**
+ * Get the rte_compressdev structure device pointer for the device. Assumes a
+ * valid device index.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - The rte_compressdev structure pointer for the given device identifier.
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_get_dev(uint8_t dev_id);
+
+/**
+ * Get the rte_compressdev structure device pointer for the named device.
+ *
+ * @param name
+ *   Compress device name
+ * @return
+ *   - The rte_compressdev structure pointer for the given device identifier.
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_get_named_dev(const char *name);
+
+/**
+ * Validate if the comp device index is valid attached comp device.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - If the device index is valid (1) or not (0).
+ */
+unsigned int __rte_experimental
+rte_compressdev_pmd_is_valid_dev(uint8_t dev_id);
+
+/**
+ * Definitions of all functions exported by a driver through the
+ * the generic structure of type *comp_dev_ops* supplied in the
+ * *rte_compressdev* structure associated with a device.
+ */
+
+/**
+ *	Function used to configure device.
+ *
+ * @param dev
+ *   Compress device
+ * @param config
+ *   Compress device configurations
+ * @return
+ *   Returns 0 on success
+ */
+typedef int (*compressdev_configure_t)(struct rte_compressdev *dev,
+		struct rte_compressdev_config *config);
+
+/**
+ * Function used to start a configured device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *   Returns 0 on success
+ */
+typedef int (*compressdev_start_t)(struct rte_compressdev *dev);
+
+/**
+ * Function used to stop a configured device.
+ *
+ * @param dev
+ *   Compress device
+ */
+typedef void (*compressdev_stop_t)(struct rte_compressdev *dev);
+
+/**
+ * Function used to close a configured device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ * - 0 on success.
+ * - EAGAIN if can't close as device is busy
+ */
+typedef int (*compressdev_close_t)(struct rte_compressdev *dev);
+
+
+/**
+ * Function used to get statistics of a device.
+ *
+ * @param dev
+ *   Compress device
+ * @param stats
+ *   Compress device stats to populate
+ */
+typedef void (*compressdev_stats_get_t)(struct rte_compressdev *dev,
+				struct rte_compressdev_stats *stats);
+
+
+/**
+ * Function used to reset statistics of a device.
+ *
+ * @param dev
+ *   Compress device
+ */
+typedef void (*compressdev_stats_reset_t)(struct rte_compressdev *dev);
+
+
+/**
+ * Function used to get specific information of a device.
+ *
+ * @param dev
+ *   Compress device
+ */
+typedef void (*compressdev_info_get_t)(struct rte_compressdev *dev,
+				struct rte_compressdev_info *dev_info);
+
+/**
+ * Setup a queue pair for a device.
+ *
+ * @param dev
+ *   Compress device
+ * @param qp_id
+ *   Queue pair identifier
+ * @param max_inflight_ops
+ *   Max inflight ops which qp must accommodate
+ * @param socket_id
+ *   Socket identifier
+ * @return
+ *   Returns 0 on success.
+ */
+typedef int (*compressdev_queue_pair_setup_t)(struct rte_compressdev *dev,
+		uint16_t qp_id,	uint32_t max_inflight_ops, int socket_id);
+
+/**
+ * Release memory resources allocated by given queue pair.
+ *
+ * @param dev
+ *   Compress device
+ * @param qp_id
+ *   Queue pair identifier
+ * @return
+ * - 0 on success.
+ * - EAGAIN if can't close as device is busy
+ */
+typedef int (*compressdev_queue_pair_release_t)(struct rte_compressdev *dev,
+		uint16_t qp_id);
+
+/**
+ * Get number of available queue pairs of a device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *   Returns number of queue pairs on success.
+ */
+typedef uint32_t (*compressdev_queue_pair_count_t)(struct rte_compressdev *dev);
+
+
+/**
+ * Create driver private stream data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param xform
+ *   xform data
+ * @param stream
+ *   ptr where handle of pmd's private stream data should be stored
+ * @return
+ *  - Returns 0 if private stream structure has been created successfully.
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private stream could not be allocated.
+ */
+typedef int (*compressdev_stream_create_t)(struct rte_compressdev *dev,
+		const struct rte_comp_xform *xform, void **stream);
+
+/**
+ * Free driver private stream data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param stream
+ *   handle of pmd's private stream data
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -EBUSY if can't free stream as there are inflight operations
+ */
+typedef int (*compressdev_stream_free_t)(struct rte_compressdev *dev,
+		void *stream);
+
+/**
+ * Create driver private_xform data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param xform
+ *   xform data
+ * @param private_xform
+ *   ptr where handle of pmd's private_xform data should be stored
+ * @return
+ *  - if successful returns RTE_COMP_PRIV_XFORM_SHAREABLE/NOT_SHAREABLE
+ *    and valid private_xform handle
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private_xform could not be allocated.
+ */
+typedef int (*compressdev_private_xform_create_t)(struct rte_compressdev *dev,
+		const struct rte_comp_xform *xform, void **private_xform);
+
+/**
+ * Free driver private_xform data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param private_xform
+ *   handle of pmd's private_xform data
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -EBUSY if can't free private_xform due to inflight operations
+ */
+typedef int (*compressdev_private_xform_free_t)(struct rte_compressdev *dev,
+		void *private_xform);
+
+/** comp device operations function pointer table */
+struct rte_compressdev_ops {
+	compressdev_configure_t dev_configure;	/**< Configure device. */
+	compressdev_start_t dev_start;		/**< Start device. */
+	compressdev_stop_t dev_stop;		/**< Stop device. */
+	compressdev_close_t dev_close;		/**< Close device. */
+
+	compressdev_info_get_t dev_infos_get;	/**< Get device info. */
+
+	compressdev_stats_get_t stats_get;
+	/**< Get device statistics. */
+	compressdev_stats_reset_t stats_reset;
+	/**< Reset device statistics. */
+
+	compressdev_queue_pair_setup_t queue_pair_setup;
+	/**< Set up a device queue pair. */
+	compressdev_queue_pair_release_t queue_pair_release;
+	/**< Release a queue pair. */
+
+	compressdev_stream_create_t stream_create;
+	/**< Create a comp stream and initialise its private data. */
+	compressdev_stream_free_t stream_free;
+	/**< Free a comp stream's private data. */
+
+	compressdev_private_xform_create_t private_xform_create;
+	/**< Create a comp private_xform and initialise its private data. */
+	compressdev_private_xform_free_t private_xform_free;
+	/**< Free a comp private_xform's data. */
+};
+
+
+/**
+ * @internal
+ *
+ * Function for internal use by dummy drivers primarily, e.g. ring-based
+ * driver.
+ * Allocates a new compressdev slot for an comp device and returns the pointer
+ * to that slot for the driver to use.
+ *
+ * @param name
+ *   Unique identifier name for each device
+ * @param socket_id
+ *   Socket to allocate resources on
+ * @return
+ *   - Slot in the rte_dev_devices array for a new device;
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_allocate(const char *name, int socket_id);
+
+/**
+ * @internal
+ *
+ * Function for internal use by dummy drivers primarily, e.g. ring-based
+ * driver.
+ * Release the specified compressdev device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *   - 0 on success, negative on error
+ */
+int __rte_experimental
+rte_compressdev_pmd_release_device(struct rte_compressdev *dev);
+
+
+/**
+ * @internal
+ *
+ * PMD assist function to parse initialisation arguments for comp driver
+ * when creating a new comp PMD device instance.
+ *
+ * PMD driver should set default values for that PMD before calling function,
+ * these default values will be over-written with successfully parsed values
+ * from args string.
+ *
+ * @param params
+ *   Parsed PMD initialisation parameters
+ * @param args
+ *   Input argument string to parse
+ * @return
+ *  - 0 on success
+ *  - errno on failure
+ */
+int __rte_experimental
+rte_compressdev_pmd_parse_input_args(
+		struct rte_compressdev_pmd_init_params *params,
+		const char *args);
+
+/**
+ * @internal
+ *
+ * PMD assist function to provide boiler plate code for comp driver to create
+ * and allocate resources for a new comp PMD device instance.
+ *
+ * @param name
+ *   Compress device name
+ * @param device
+ *   Base device instance
+ * @param params
+ *   PMD initialisation parameters
+ * @return
+ *  - comp device instance on success
+ *  - NULL on creation failure
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_create(const char *name,
+		struct rte_device *device,
+		struct rte_compressdev_pmd_init_params *params);
+
+/**
+ * @internal
+ *
+ * PMD assist function to provide boiler plate code for comp driver to
+ * destroy and free resources associated with a comp PMD device instance.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *  - 0 on success
+ *  - errno on failure
+ */
+int __rte_experimental
+rte_compressdev_pmd_destroy(struct rte_compressdev *dev);
+
+
+/**
+ * @internal
+ * Allocate compressdev driver.
+ *
+ * @param comp_drv
+ *   Compressdev driver
+ * @param drv
+ *   Rte_driver
+ * @return
+ *  The driver type identifier
+ */
+uint8_t __rte_experimental
+rte_compressdev_allocate_driver(struct compressdev_driver *comp_drv,
+		const struct rte_driver *drv);
+
+
+#define RTE_PMD_REGISTER_COMPRESSDEV_DRIVER(comp_drv, drv, driver_id)\
+RTE_INIT(init_ ##driver_id);\
+static void init_ ##driver_id(void)\
+{\
+	driver_id = rte_compressdev_allocate_driver(&comp_drv, &(drv).driver);\
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_COMPRESSDEV_PMD_H_ */
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
new file mode 100644
index 0000000..edf284b
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -0,0 +1,43 @@
+EXPERIMENTAL {
+        global:
+
+        rte_compressdevs;
+	rte_compressdev_allocate_driver;
+	rte_compressdev_callback_register;
+	rte_compressdev_callback_unregister;
+	rte_compressdev_close;
+	rte_compressdev_configure;
+	rte_compressdev_count;
+	rte_compressdev_device_count_by_driver;
+	rte_compressdev_devices_get;
+	rte_compressdev_driver_id_get;
+	rte_compressdev_driver_name_get;
+	rte_compressdev_get_dev_id;
+	rte_compressdev_get_feature_name;
+	rte_compressdev_info_get;
+	rte_compressdev_name_get;
+	rte_compressdev_pmd_allocate;
+	rte_compressdev_pmd_callback_process;
+	rte_compressdev_pmd_create;
+	rte_compressdev_pmd_destroy;
+	rte_compressdev_pmd_get_dev;
+	rte_compressdev_pmd_get_named_dev;
+	rte_compressdev_pmd_is_valid_dev;
+	rte_compressdev_pmd_parse_input_args;
+	rte_compressdev_pmd_release_device;
+	rte_compressdev_private_xform_create;
+	rte_compressdev_private_xform_free;
+	rte_compressdev_queue_pair_count;
+	rte_compressdev_queue_pair_setup;
+	rte_compressdev_socket_id;
+	rte_compressdev_start;
+	rte_compressdev_stats_get;
+	rte_compressdev_stats_reset;
+	rte_compressdev_stop;
+	rte_compressdev_stream_create;
+	rte_compressdev_stream_free;
+	rte_comp_get_feature_name;
+	rte_comp_op_pool_create;
+
+        local: *;
+};
diff --git a/lib/meson.build b/lib/meson.build
index ef61591..ae718aa 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -14,7 +14,7 @@ libraries = [ 'compat', # just a header, used for versioning
 	'hash',    # efd depends on this
 	'kvargs',  # cryptodev depends on this
 	'acl', 'bbdev', 'bitratestats', 'cfgfile',
-	'cmdline', 'cryptodev',
+	'cmdline', 'compressdev', 'cryptodev',
 	'distributor', 'efd', 'eventdev',
 	'gro', 'gso', 'ip_frag', 'jobstats',
 	'kni', 'latencystats', 'lpm', 'member',
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 3eb41d1..cff710a 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -94,6 +94,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF)           += -lrte_mbuf
 _LDLIBS-$(CONFIG_RTE_LIBRTE_NET)            += -lrte_net
 _LDLIBS-$(CONFIG_RTE_LIBRTE_ETHER)          += -lrte_ethdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_BBDEV)          += -lrte_bbdev
+_LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV)    += -lrte_compressdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_CRYPTODEV)      += -lrte_cryptodev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY)       += -lrte_security
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV)       += -lrte_eventdev
-- 
2.7.4

[dpdk-dev] [PATCH v2 3/3] doc: update doxy and release note for compressdev

[Fiona Trahe]

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
---
 doc/api/doxy-api-index.md              | 1 +
 doc/api/doxy-api.conf                  | 1 +
 doc/guides/rel_notes/release_18_05.rst | 6 ++++++
 3 files changed, 8 insertions(+)

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index d77f205..530808e 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -45,6 +45,7 @@ The public API headers are grouped by topics:
   [bbdev]              (@ref rte_bbdev.h),
   [cryptodev]          (@ref rte_cryptodev.h),
   [security]           (@ref rte_security.h),
+  [compressdev]        (@ref rte_compressdev.h),
   [eventdev]           (@ref rte_eventdev.h),
   [event_eth_rx_adapter]   (@ref rte_event_eth_rx_adapter.h),
   [rawdev]             (@ref rte_rawdev.h),
diff --git a/doc/api/doxy-api.conf b/doc/api/doxy-api.conf
index cda52fd..06432c3 100644
--- a/doc/api/doxy-api.conf
+++ b/doc/api/doxy-api.conf
@@ -45,6 +45,7 @@ INPUT                   = doc/api/doxy-api-index.md \
                           lib/librte_cfgfile \
                           lib/librte_cmdline \
                           lib/librte_compat \
+                          lib/librte_compressdev \
                           lib/librte_cryptodev \
                           lib/librte_distributor \
                           lib/librte_efd \
diff --git a/doc/guides/rel_notes/release_18_05.rst b/doc/guides/rel_notes/release_18_05.rst
index 85f4dc5..75c308d 100644
--- a/doc/guides/rel_notes/release_18_05.rst
+++ b/doc/guides/rel_notes/release_18_05.rst
@@ -41,6 +41,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **Added compressdev API, a generic compression service library.**
+
+  The compressdev library provides an API for offload of compression and
+  decompression operations to hardware or software accelerator devices.
+
 
 API Changes
 -----------
@@ -125,6 +130,7 @@ The libraries prepended with a plus sign were incremented in this version.
      librte_bus_vdev.so.1
      librte_cfgfile.so.2
      librte_cmdline.so.2
+   + librte_compressdev.so.1
      librte_cryptodev.so.4
      librte_distributor.so.1
      librte_eal.so.6
-- 
2.7.4

[dpdk-dev] [PATCH v5 00/13] Implement compression API

[Pablo de Lara]

With the vast amounts of data being transported around networks and stored in storage systems, reducing data size is becoming ever more important.

There are both software libraries and hardware devices available that provide compression, but no common API.
Such an API is proposed in this patchset, which supports the following features:

- Deflate Algorithm (https://tools.ietf.org/html/rfc1951)
- LZS algorithm (https://tools.ietf.org/html/rfc2395)
- Static and Dynamic Huffman encoding.
- Compression levels
- Checksum generation
- Asynchronous burst API
- private_xform - a place for PMDs to hold private data derived from
  a xform and used by stateless operations.
- stream - a place for PMDs to hold private data derived from
  a xform and also maintain state and history data. For
  stateful flows.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara pablo.de.lara.guarch@intel.com
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>

Changes in v5:
 - Make rte_compressdev_is_valid_dev, rte_compressdev_pmd_get_dev
   functions internal
 - Add rte_comp.h to API doc
 - Remove unused rte_comp_op_get_user_data,
   rte_comp_op_get_user_data_size functions
 - Fix comment
 - Move private data size arg out of PMD init params structure
 - Split out of space error status into two
 - Make rte_comp_op_alloc, rte_comp_op_bulk_alloc, rte_comp_free,
   rte_compressdev_enqueue_burst and rte_compressdev_dequeue_burst
   functions not inline.
 - Move rte_compressdev and rte_compressdev_data structures out of
   rte_compressdev.h, making them internal to the API
 - Remove unneeded driver registration (used in cryptodev, for multi PMD
   sessions)
 - Remove unneeded includes
 - rte_comp_op_bulk_alloc returns now 0 on success, instead of the
   number of allocated operations
 - Move internals macros to rte_compressdev_internals.h, to avoid
   exposing them unnecessarily

Changes in v4:
 - Fix build (missing ";")
 - Change order in common_base, so compression is placed after security

Changes in v3:
 - Remove rte_comp_op_ctod helper functions
 - Remove param_range_check macro
 - Rename from phys_addr to iova_addr
 - Remove rte_comp_algo_strings
 - Rename rte_compressdev_pmd_is_valid_dev to rte_compressdev_is_valid_dev
 - Remove feature flags from compressdev
 - Support hash operations
 - Add shareable priv xform in feature flags, instead of returnin it
   on xform creation
 - Allow max number of queue pairs to be 0, meaning that there is no
   limit.
 - Add invalid configuration checks
 - Add capability helper functions

Changes in v2:
 - Add stream APIs
 - Remove session
 - Add SHAREABLE / NON_SHAREABLE private_xform types
 - Add algo enum 'UNSPECIFIED' to fix warning in capabilities
 - Change one remaining log to use dynamic logging.
 - Add rte_cache_aligned keyword to op
 - Rename enums with better names _ALGO, __CHECKSUM, _HUFFMAN_
 - Use const keyword when passing xform
 - Remove qp_count fn from dev_ops as never used
 - Remove max_nb_queue-pairs from compressdev_init_param as never used
 - Clarify device configure and start sequence
 - Replace OUT_OF_SPACE with OUT_OF_SPACE_RECOVERABLE and TERMINATED
   and clarified usage. 
 - Add stream and private_xform sizes to device config for use in
   mempool creation
 - Add capability helper fn
 - Use Base2 log value for window size on xforms
 - Add Meson build
 - Update MAINTAINERS
 - Update Map file
 - Change order in doxy file
 - Update Release note 


Fiona Trahe (12):
  compressdev: add basic device management
  compressdev: add queue pair management
  compressdev: add compression specific data
  compressdev: add enqueue/dequeue functions
  compressdev: add operation management
  compressdev: support stateless operations
  compressdev: support stateful operations
  compressdev: add device feature flags
  compressdev: add compression service feature flags
  compressdev: add device stats
  compressdev: add device capabilities
  compressdev: get device id from name

Shally Verma (1):
  compressdev: support hash operations

 MAINTAINERS                                        |   7 +
 config/common_base                                 |   6 +
 config/rte_config.h                                |   3 +
 doc/api/doxy-api-index.md                          |   2 +
 doc/api/doxy-api.conf                              |   1 +
 doc/guides/rel_notes/release_18_05.rst             |   6 +
 lib/Makefile                                       |   3 +
 lib/librte_compressdev/Makefile                    |  31 +
 lib/librte_compressdev/meson.build                 |  12 +
 lib/librte_compressdev/rte_comp.c                  | 207 ++++++
 lib/librte_compressdev/rte_comp.h                  | 464 +++++++++++++
 lib/librte_compressdev/rte_compressdev.c           | 734 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 536 +++++++++++++++
 lib/librte_compressdev/rte_compressdev_internal.h  | 123 ++++
 lib/librte_compressdev/rte_compressdev_pmd.c       | 160 +++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 390 +++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  42 ++
 lib/meson.build                                    |   2 +-
 mk/rte.app.mk                                      |   1 +
 19 files changed, 2729 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_compressdev/Makefile
 create mode 100644 lib/librte_compressdev/meson.build
 create mode 100644 lib/librte_compressdev/rte_comp.c
 create mode 100644 lib/librte_compressdev/rte_comp.h
 create mode 100644 lib/librte_compressdev/rte_compressdev.c
 create mode 100644 lib/librte_compressdev/rte_compressdev.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_internal.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.c
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_version.map

-- 
2.14.3

[dpdk-dev] [PATCH v5 01/13] compressdev: add basic device management

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Add basic functions to manage compress devices,
including driver and device allocation, and the basic
interface with compressdev PMDs.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 MAINTAINERS                                        |   7 +
 config/common_base                                 |   6 +
 config/rte_config.h                                |   3 +
 doc/api/doxy-api-index.md                          |   1 +
 doc/api/doxy-api.conf                              |   1 +
 doc/guides/rel_notes/release_18_05.rst             |   6 +
 lib/Makefile                                       |   3 +
 lib/librte_compressdev/Makefile                    |  30 ++
 lib/librte_compressdev/meson.build                 |  10 +
 lib/librte_compressdev/rte_compressdev.c           | 378 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 165 +++++++++
 lib/librte_compressdev/rte_compressdev_internal.h  |  64 ++++
 lib/librte_compressdev/rte_compressdev_pmd.c       | 160 +++++++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 236 +++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  24 ++
 lib/meson.build                                    |   2 +-
 mk/rte.app.mk                                      |   1 +
 17 files changed, 1096 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_compressdev/Makefile
 create mode 100644 lib/librte_compressdev/meson.build
 create mode 100644 lib/librte_compressdev/rte_compressdev.c
 create mode 100644 lib/librte_compressdev/rte_compressdev.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_internal.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.c
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index e54c1f056..1e83bce71 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -357,6 +357,13 @@ F: drivers/raw/skeleton_rawdev/
 F: test/test/test_rawdev.c
 F: doc/guides/prog_guide/rawdev.rst
 
+Compression API - EXPERIMENTAL
+M: Fiona Trahe <fiona.trahe@intel.com>
+M: Pablo de Lara <pablo.de.lara.guarch@intel.com>
+M: Ashish Gupta <ashish.gupta@caviumnetworks.com>
+T: git://dpdk.org/next/dpdk-next-crypto
+F: lib/librte_compressdev/
+
 
 Bus Drivers
 -----------
diff --git a/config/common_base b/config/common_base
index ad4706267..f40354487 100644
--- a/config/common_base
+++ b/config/common_base
@@ -543,6 +543,12 @@ CONFIG_RTE_LIBRTE_PMD_MRVL_CRYPTO_DEBUG=n
 #
 CONFIG_RTE_LIBRTE_SECURITY=y
 
+#
+# Compile generic compression device library
+#
+CONFIG_RTE_LIBRTE_COMPRESSDEV=y
+CONFIG_RTE_COMPRESS_MAX_DEVS=64
+
 #
 # Compile generic event device library
 #
diff --git a/config/rte_config.h b/config/rte_config.h
index db6ceb6cd..949071f6e 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -51,6 +51,9 @@
 #define RTE_CRYPTO_MAX_DEVS 64
 #define RTE_CRYPTODEV_NAME_LEN 64
 
+/* compressdev defines */
+#define RTE_COMPRESS_MAX_DEVS 64
+
 /* eventdev defines */
 #define RTE_EVENT_MAX_DEVS 16
 #define RTE_EVENT_MAX_QUEUES_PER_DEV 64
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 2f04619cb..c076a27e6 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -45,6 +45,7 @@ The public API headers are grouped by topics:
   [bbdev]              (@ref rte_bbdev.h),
   [cryptodev]          (@ref rte_cryptodev.h),
   [security]           (@ref rte_security.h),
+  [compressdev]        (@ref rte_compressdev.h),
   [eventdev]           (@ref rte_eventdev.h),
   [event_eth_rx_adapter]   (@ref rte_event_eth_rx_adapter.h),
   [rawdev]             (@ref rte_rawdev.h),
diff --git a/doc/api/doxy-api.conf b/doc/api/doxy-api.conf
index cda52fdfb..06432c3aa 100644
--- a/doc/api/doxy-api.conf
+++ b/doc/api/doxy-api.conf
@@ -45,6 +45,7 @@ INPUT                   = doc/api/doxy-api-index.md \
                           lib/librte_cfgfile \
                           lib/librte_cmdline \
                           lib/librte_compat \
+                          lib/librte_compressdev \
                           lib/librte_cryptodev \
                           lib/librte_distributor \
                           lib/librte_efd \
diff --git a/doc/guides/rel_notes/release_18_05.rst b/doc/guides/rel_notes/release_18_05.rst
index a80611020..790b78688 100644
--- a/doc/guides/rel_notes/release_18_05.rst
+++ b/doc/guides/rel_notes/release_18_05.rst
@@ -70,6 +70,11 @@ New Features
 
   * AES-CMAC (128-bit key).
 
+* **Added compressdev API, a generic compression service library.**
+
+  The compressdev library provides an API for offload of compression and
+  decompression operations to hardware or software accelerator devices.
+
 
 API Changes
 -----------
@@ -166,6 +171,7 @@ The libraries prepended with a plus sign were incremented in this version.
      librte_cfgfile.so.2
      librte_cmdline.so.2
    + librte_common_octeontx.so.1
+   + librte_compressdev.so.1
      librte_cryptodev.so.4
      librte_distributor.so.1
    + librte_eal.so.7
diff --git a/lib/Makefile b/lib/Makefile
index ec965a606..19396daff 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -26,6 +26,9 @@ DEPDIRS-librte_bbdev := librte_eal librte_mempool librte_mbuf
 DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += librte_cryptodev
 DEPDIRS-librte_cryptodev := librte_eal librte_mempool librte_ring librte_mbuf
 DEPDIRS-librte_cryptodev += librte_kvargs
+DIRS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += librte_compressdev
+DEPDIRS-librte_compressdev := librte_eal librte_mempool librte_ring librte_mbuf
+DEPDIRS-librte_compressdev += librte_kvargs
 DIRS-$(CONFIG_RTE_LIBRTE_SECURITY) += librte_security
 DEPDIRS-librte_security := librte_eal librte_mempool librte_ring librte_mbuf
 DEPDIRS-librte_security += librte_ether
diff --git a/lib/librte_compressdev/Makefile b/lib/librte_compressdev/Makefile
new file mode 100644
index 000000000..b84769417
--- /dev/null
+++ b/lib/librte_compressdev/Makefile
@@ -0,0 +1,30 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2017-2018 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_compressdev.a
+
+# library version
+LIBABIVER := 1
+
+# build flags
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+LDLIBS += -lrte_eal -lrte_mempool -lrte_kvargs
+
+# library source files
+SRCS-y += rte_compressdev.c rte_compressdev_pmd.c
+
+# export include files
+SYMLINK-y-include += rte_compressdev.h
+# export include files (for PMDs)
+SYMLINK-y-include += rte_compressdev_pmd.h
+SYMLINK-y-include += rte_compressdev_internal.h
+
+# versioning export map
+EXPORT_MAP := rte_compressdev_version.map
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_compressdev/meson.build b/lib/librte_compressdev/meson.build
new file mode 100644
index 000000000..895e959d3
--- /dev/null
+++ b/lib/librte_compressdev/meson.build
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+allow_experimental_apis = true
+sources = files('rte_compressdev.c',
+	'rte_compressdev_pmd.c')
+headers = files('rte_compressdev.h',
+	'rte_compressdev_pmd.h',
+	'rte_compressdev_internal.h')
+deps += ['kvargs', 'mbuf']
diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
new file mode 100644
index 000000000..751517c3d
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -0,0 +1,378 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#include <string.h>
+#include <stdarg.h>
+#include <stdio.h>
+#include <inttypes.h>
+
+#include <rte_malloc.h>
+#include <rte_eal.h>
+#include <rte_memzone.h>
+
+#include "rte_compressdev.h"
+#include "rte_compressdev_internal.h"
+#include "rte_compressdev_pmd.h"
+
+#define RTE_COMPRESSDEV_DETACHED  (0)
+#define RTE_COMPRESSDEV_ATTACHED  (1)
+
+struct rte_compressdev rte_comp_devices[RTE_COMPRESS_MAX_DEVS];
+
+struct rte_compressdev *rte_compressdevs = &rte_comp_devices[0];
+
+static struct rte_compressdev_global compressdev_globals = {
+		.devs			= &rte_comp_devices[0],
+		.data			= { NULL },
+		.nb_devs		= 0,
+		.max_devs		= RTE_COMPRESS_MAX_DEVS
+};
+
+struct rte_compressdev_global *rte_compressdev_globals = &compressdev_globals;
+
+static struct rte_compressdev *
+rte_compressdev_get_dev(uint8_t dev_id)
+{
+	return &rte_compressdev_globals->devs[dev_id];
+}
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_get_named_dev(const char *name)
+{
+	struct rte_compressdev *dev;
+	unsigned int i;
+
+	if (name == NULL)
+		return NULL;
+
+	for (i = 0; i < rte_compressdev_globals->max_devs; i++) {
+		dev = &rte_compressdev_globals->devs[i];
+
+		if ((dev->attached == RTE_COMPRESSDEV_ATTACHED) &&
+				(strcmp(dev->data->name, name) == 0))
+			return dev;
+	}
+
+	return NULL;
+}
+
+static unsigned int
+rte_compressdev_is_valid_dev(uint8_t dev_id)
+{
+	struct rte_compressdev *dev = NULL;
+
+	if (dev_id >= rte_compressdev_globals->nb_devs)
+		return 0;
+
+	dev = rte_compressdev_get_dev(dev_id);
+	if (dev->attached != RTE_COMPRESSDEV_ATTACHED)
+		return 0;
+	else
+		return 1;
+}
+
+
+uint8_t __rte_experimental
+rte_compressdev_count(void)
+{
+	return rte_compressdev_globals->nb_devs;
+}
+
+uint8_t __rte_experimental
+rte_compressdev_devices_get(const char *driver_name, uint8_t *devices,
+	uint8_t nb_devices)
+{
+	uint8_t i, count = 0;
+	struct rte_compressdev *devs = rte_compressdev_globals->devs;
+	uint8_t max_devs = rte_compressdev_globals->max_devs;
+
+	for (i = 0; i < max_devs && count < nb_devices;	i++) {
+
+		if (devs[i].attached == RTE_COMPRESSDEV_ATTACHED) {
+			int cmp;
+
+			cmp = strncmp(devs[i].device->driver->name,
+					driver_name,
+					strlen(driver_name));
+
+			if (cmp == 0)
+				devices[count++] = devs[i].data->dev_id;
+		}
+	}
+
+	return count;
+}
+
+int __rte_experimental
+rte_compressdev_socket_id(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id))
+		return -1;
+
+	dev = rte_compressdev_get_dev(dev_id);
+
+	return dev->data->socket_id;
+}
+
+static inline int
+rte_compressdev_data_alloc(uint8_t dev_id, struct rte_compressdev_data **data,
+		int socket_id)
+{
+	char mz_name[RTE_COMPRESSDEV_NAME_MAX_LEN];
+	const struct rte_memzone *mz;
+	int n;
+
+	/* generate memzone name */
+	n = snprintf(mz_name, sizeof(mz_name),
+			"rte_compressdev_data_%u", dev_id);
+	if (n >= (int)sizeof(mz_name))
+		return -EINVAL;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		mz = rte_memzone_reserve(mz_name,
+				sizeof(struct rte_compressdev_data),
+				socket_id, 0);
+	} else
+		mz = rte_memzone_lookup(mz_name);
+
+	if (mz == NULL)
+		return -ENOMEM;
+
+	*data = mz->addr;
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		memset(*data, 0, sizeof(struct rte_compressdev_data));
+
+	return 0;
+}
+
+static uint8_t
+rte_compressdev_find_free_device_index(void)
+{
+	uint8_t dev_id;
+
+	for (dev_id = 0; dev_id < RTE_COMPRESS_MAX_DEVS; dev_id++) {
+		if (rte_comp_devices[dev_id].attached ==
+				RTE_COMPRESSDEV_DETACHED)
+			return dev_id;
+	}
+	return RTE_COMPRESS_MAX_DEVS;
+}
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_allocate(const char *name, int socket_id)
+{
+	struct rte_compressdev *compressdev;
+	uint8_t dev_id;
+
+	if (rte_compressdev_pmd_get_named_dev(name) != NULL) {
+		COMPRESSDEV_LOG(ERR,
+			"comp device with name %s already allocated!", name);
+		return NULL;
+	}
+
+	dev_id = rte_compressdev_find_free_device_index();
+	if (dev_id == RTE_COMPRESS_MAX_DEVS) {
+		COMPRESSDEV_LOG(ERR, "Reached maximum number of comp devices");
+		return NULL;
+	}
+	compressdev = rte_compressdev_get_dev(dev_id);
+
+	if (compressdev->data == NULL) {
+		struct rte_compressdev_data *compressdev_data =
+				compressdev_globals.data[dev_id];
+
+		int retval = rte_compressdev_data_alloc(dev_id,
+				&compressdev_data, socket_id);
+
+		if (retval < 0 || compressdev_data == NULL)
+			return NULL;
+
+		compressdev->data = compressdev_data;
+
+		snprintf(compressdev->data->name, RTE_COMPRESSDEV_NAME_MAX_LEN,
+				"%s", name);
+
+		compressdev->data->dev_id = dev_id;
+		compressdev->data->socket_id = socket_id;
+		compressdev->data->dev_started = 0;
+
+		compressdev->attached = RTE_COMPRESSDEV_ATTACHED;
+
+		compressdev_globals.nb_devs++;
+	}
+
+	return compressdev;
+}
+
+int __rte_experimental
+rte_compressdev_pmd_release_device(struct rte_compressdev *compressdev)
+{
+	int ret;
+
+	if (compressdev == NULL)
+		return -EINVAL;
+
+	/* Close device only if device operations have been set */
+	if (compressdev->dev_ops) {
+		ret = rte_compressdev_close(compressdev->data->dev_id);
+		if (ret < 0)
+			return ret;
+	}
+
+	compressdev->attached = RTE_COMPRESSDEV_DETACHED;
+	compressdev_globals.nb_devs--;
+	return 0;
+}
+
+int __rte_experimental
+rte_compressdev_configure(uint8_t dev_id, struct rte_compressdev_config *config)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	if (dev->data->dev_started) {
+		COMPRESSDEV_LOG(ERR,
+		    "device %d must be stopped to allow configuration", dev_id);
+		return -EBUSY;
+	}
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_configure, -ENOTSUP);
+
+	return (*dev->dev_ops->dev_configure)(dev, config);
+}
+
+
+int __rte_experimental
+rte_compressdev_start(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+	int diag;
+
+	COMPRESSDEV_LOG(DEBUG, "Start dev_id=%" PRIu8, dev_id);
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_start, -ENOTSUP);
+
+	if (dev->data->dev_started != 0) {
+		COMPRESSDEV_LOG(ERR,
+		    "Device with dev_id=%" PRIu8 " already started", dev_id);
+		return 0;
+	}
+
+	diag = (*dev->dev_ops->dev_start)(dev);
+	if (diag == 0)
+		dev->data->dev_started = 1;
+	else
+		return diag;
+
+	return 0;
+}
+
+void __rte_experimental
+rte_compressdev_stop(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_stop);
+
+	if (dev->data->dev_started == 0) {
+		COMPRESSDEV_LOG(ERR,
+		    "Device with dev_id=%" PRIu8 " already stopped", dev_id);
+		return;
+	}
+
+	(*dev->dev_ops->dev_stop)(dev);
+	dev->data->dev_started = 0;
+}
+
+int __rte_experimental
+rte_compressdev_close(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+	int retval;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -1;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	/* Device must be stopped before it can be closed */
+	if (dev->data->dev_started == 1) {
+		COMPRESSDEV_LOG(ERR, "Device %u must be stopped before closing",
+				dev_id);
+		return -EBUSY;
+	}
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_close, -ENOTSUP);
+	retval = (*dev->dev_ops->dev_close)(dev);
+
+	if (retval < 0)
+		return retval;
+
+	return 0;
+}
+
+void __rte_experimental
+rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
+{
+	struct rte_compressdev *dev;
+
+	if (dev_id >= compressdev_globals.nb_devs) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%d", dev_id);
+		return;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	memset(dev_info, 0, sizeof(struct rte_compressdev_info));
+
+	RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
+	(*dev->dev_ops->dev_infos_get)(dev, dev_info);
+
+	dev_info->driver_name = dev->device->driver->name;
+}
+
+const char * __rte_experimental
+rte_compressdev_name_get(uint8_t dev_id)
+{
+	struct rte_compressdev *dev = rte_compressdev_get_dev(dev_id);
+
+	if (dev == NULL)
+		return NULL;
+
+	return dev->data->name;
+}
+
+RTE_INIT(rte_compressdev_log);
+
+static void
+rte_compressdev_log(void)
+{
+	compressdev_logtype = rte_log_register("lib.compressdev");
+	if (compressdev_logtype >= 0)
+		rte_log_set_level(compressdev_logtype, RTE_LOG_NOTICE);
+}
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
new file mode 100644
index 000000000..6150850db
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -0,0 +1,165 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMPRESSDEV_H_
+#define _RTE_COMPRESSDEV_H_
+
+/**
+ * @file rte_compressdev.h
+ *
+ * RTE Compression Device APIs
+ *
+ * Defines comp device APIs for the provisioning of compression operations.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+
+/**  comp device information */
+struct rte_compressdev_info {
+	const char *driver_name;		/**< Driver name. */
+};
+
+/**
+ * Get the compress device name given a device identifier.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - Returns compress device name.
+ *   - Returns NULL if compress device is not present.
+ */
+const char * __rte_experimental
+rte_compressdev_name_get(uint8_t dev_id);
+
+/**
+ * Get the total number of compress devices that have been successfully
+ * initialised.
+ *
+ * @return
+ *   - The total number of usable compress devices.
+ */
+uint8_t __rte_experimental
+rte_compressdev_count(void);
+
+/**
+ * Get number and identifiers of attached comp devices that
+ * use the same compress driver.
+ *
+ * @param driver_name
+ *   Driver name
+ * @param devices
+ *   Output devices identifiers
+ * @param nb_devices
+ *   Maximal number of devices
+ *
+ * @return
+ *   Returns number of attached compress devices.
+ */
+uint8_t __rte_experimental
+rte_compressdev_devices_get(const char *driver_name, uint8_t *devices,
+		uint8_t nb_devices);
+
+/*
+ * Return the NUMA socket to which a device is connected.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   The NUMA socket id to which the device is connected or
+ *   a default of zero if the socket could not be determined.
+ *   -1 if returned is the dev_id value is out of range.
+ */
+int __rte_experimental
+rte_compressdev_socket_id(uint8_t dev_id);
+
+/** Compress device configuration structure */
+struct rte_compressdev_config {
+	int socket_id;
+};
+
+/**
+ * Configure a device.
+ *
+ * This function must be invoked first before any other function in the
+ * API. This function can also be re-invoked when a device is in the
+ * stopped state.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param config
+ *   The compress device configuration
+ * @return
+ *   - 0: Success, device configured.
+ *   - <0: Error code returned by the driver configuration function.
+ */
+int __rte_experimental
+rte_compressdev_configure(uint8_t dev_id,
+			struct rte_compressdev_config *config);
+
+/**
+ * Start a device.
+ *
+ * The device start step is called after configuring the device and setting up
+ * its queue pairs.
+ * On success, data-path functions exported by the API (enqueue/dequeue, etc)
+ * can be invoked.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - 0: Success, device started.
+ *   - <0: Error code of the driver device start function.
+ */
+int __rte_experimental
+rte_compressdev_start(uint8_t dev_id);
+
+/**
+ * Stop a device. The device can be restarted with a call to
+ * rte_compressdev_start()
+ *
+ * @param dev_id
+ *   Compress device identifier
+ */
+void __rte_experimental
+rte_compressdev_stop(uint8_t dev_id);
+
+/**
+ * Close an device. The device cannot be restarted!
+ *
+ * @param dev_id
+ *   Compress device identifier
+ *
+ * @return
+ *  - 0 on successfully closing device
+ *  - <0 on failure to close device
+ */
+int __rte_experimental
+rte_compressdev_close(uint8_t dev_id);
+
+/**
+ * Retrieve the contextual information of a device.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param dev_info
+ *   A pointer to a structure of type *rte_compressdev_info*
+ *   to be filled with the contextual information of the device
+ *
+ * @note The capabilities field of dev_info is set to point to the first
+ * element of an array of struct rte_compressdev_capabilities.
+ * The element after the last valid element has it's op field set to
+ * RTE_COMP_ALGO_LIST_END.
+ */
+void __rte_experimental
+rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_COMPRESSDEV_H_ */
diff --git a/lib/librte_compressdev/rte_compressdev_internal.h b/lib/librte_compressdev/rte_compressdev_internal.h
new file mode 100644
index 000000000..0a2ddcb2b
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_internal.h
@@ -0,0 +1,64 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMPRESSDEV_INTERNAL_H_
+#define _RTE_COMPRESSDEV_INTERNAL_H_
+
+/* rte_compressdev_internal.h
+ * This file holds Compressdev private data structures.
+ */
+#include <rte_log.h>
+
+#define RTE_COMPRESSDEV_NAME_MAX_LEN	(64)
+/**< Max length of name of comp PMD */
+
+/* Logging Macros */
+extern int compressdev_logtype;
+#define COMPRESSDEV_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, compressdev_logtype, "%s(): "fmt "\n", \
+			__func__, ##args)
+
+
+/** The data structure associated with each comp device. */
+struct rte_compressdev {
+	struct rte_compressdev_data *data;
+	/**< Pointer to device data */
+	struct rte_compressdev_ops *dev_ops;
+	/**< Functions exported by PMD */
+	uint64_t feature_flags;
+	/**< Supported features */
+	struct rte_device *device;
+	/**< Backing device */
+
+	uint8_t driver_id;
+	/**< comp driver identifier*/
+
+	__extension__
+	uint8_t attached : 1;
+	/**< Flag indicating the device is attached */
+} __rte_cache_aligned;
+
+/**
+ *
+ * The data part, with no function pointers, associated with each device.
+ *
+ * This structure is safe to place in shared memory to be common among
+ * different processes in a multi-process configuration.
+ */
+struct rte_compressdev_data {
+	uint8_t dev_id;
+	/**< Compress device identifier */
+	uint8_t socket_id;
+	/**< Socket identifier where memory is allocated */
+	char name[RTE_COMPRESSDEV_NAME_MAX_LEN];
+	/**< Unique identifier name */
+
+	__extension__
+	uint8_t dev_started : 1;
+	/**< Device state: STARTED(1)/STOPPED(0) */
+
+	void *dev_private;
+	/**< PMD-specific private data */
+} __rte_cache_aligned;
+#endif
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.c b/lib/librte_compressdev/rte_compressdev_pmd.c
new file mode 100644
index 000000000..7de4f339e
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_pmd.c
@@ -0,0 +1,160 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#include <rte_malloc.h>
+#include <rte_kvargs.h>
+#include <rte_eal.h>
+
+#include "rte_compressdev_internal.h"
+#include "rte_compressdev_pmd.h"
+
+int compressdev_logtype;
+
+/**
+ * Parse name from argument
+ */
+static int
+rte_compressdev_pmd_parse_name_arg(const char *key __rte_unused,
+		const char *value, void *extra_args)
+{
+	struct rte_compressdev_pmd_init_params *params = extra_args;
+	int n;
+
+	n = snprintf(params->name, RTE_COMPRESSDEV_NAME_MAX_LEN, "%s", value);
+	if (n >= RTE_COMPRESSDEV_NAME_MAX_LEN)
+		return -EINVAL;
+
+	return 0;
+}
+
+/**
+ * Parse unsigned integer from argument
+ */
+static int
+rte_compressdev_pmd_parse_uint_arg(const char *key __rte_unused,
+		const char *value, void *extra_args)
+{
+	int i;
+	char *end;
+
+	errno = 0;
+	i = strtol(value, &end, 10);
+	if (*end != 0 || errno != 0 || i < 0)
+		return -EINVAL;
+
+	*((uint32_t *)extra_args) = i;
+	return 0;
+}
+
+int __rte_experimental
+rte_compressdev_pmd_parse_input_args(
+		struct rte_compressdev_pmd_init_params *params,
+		const char *args)
+{
+	struct rte_kvargs *kvlist = NULL;
+	int ret = 0;
+
+	if (params == NULL)
+		return -EINVAL;
+
+	if (args) {
+		kvlist = rte_kvargs_parse(args,	compressdev_pmd_valid_params);
+		if (kvlist == NULL)
+			return -EINVAL;
+
+		ret = rte_kvargs_process(kvlist,
+				RTE_COMPRESSDEV_PMD_SOCKET_ID_ARG,
+				&rte_compressdev_pmd_parse_uint_arg,
+				&params->socket_id);
+		if (ret < 0)
+			goto free_kvlist;
+
+		ret = rte_kvargs_process(kvlist,
+				RTE_COMPRESSDEV_PMD_NAME_ARG,
+				&rte_compressdev_pmd_parse_name_arg,
+				params);
+		if (ret < 0)
+			goto free_kvlist;
+	}
+
+free_kvlist:
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_create(const char *name,
+		struct rte_device *device,
+		size_t private_data_size,
+		struct rte_compressdev_pmd_init_params *params)
+{
+	struct rte_compressdev *compressdev;
+
+	if (params->name[0] != '\0') {
+		COMPRESSDEV_LOG(INFO, "[%s] User specified device name = %s\n",
+				device->driver->name, params->name);
+		name = params->name;
+	}
+
+	COMPRESSDEV_LOG(INFO, "[%s] - Creating compressdev %s\n",
+			device->driver->name, name);
+
+	COMPRESSDEV_LOG(INFO,
+	"[%s] - Init parameters - name: %s, socket id: %d",
+			device->driver->name, name,
+			params->socket_id);
+
+	/* allocate device structure */
+	compressdev = rte_compressdev_pmd_allocate(name, params->socket_id);
+	if (compressdev == NULL) {
+		COMPRESSDEV_LOG(ERR, "[%s] Failed to allocate comp device %s",
+				device->driver->name, name);
+		return NULL;
+	}
+
+	/* allocate private device structure */
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		compressdev->data->dev_private =
+				rte_zmalloc_socket("compressdev device private",
+						private_data_size,
+						RTE_CACHE_LINE_SIZE,
+						params->socket_id);
+
+		if (compressdev->data->dev_private == NULL) {
+			COMPRESSDEV_LOG(ERR,
+		"[%s] Cannot allocate memory for compressdev %s private data",
+					device->driver->name, name);
+
+			rte_compressdev_pmd_release_device(compressdev);
+			return NULL;
+		}
+	}
+
+	compressdev->device = device;
+
+	return compressdev;
+}
+
+int __rte_experimental
+rte_compressdev_pmd_destroy(struct rte_compressdev *compressdev)
+{
+	int retval;
+
+	COMPRESSDEV_LOG(INFO, "[%s] Closing comp device %s",
+			compressdev->device->driver->name,
+			compressdev->device->name);
+
+	/* free comp device */
+	retval = rte_compressdev_pmd_release_device(compressdev);
+	if (retval)
+		return retval;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		rte_free(compressdev->data->dev_private);
+
+	compressdev->device = NULL;
+	compressdev->data = NULL;
+
+	return 0;
+}
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
new file mode 100644
index 000000000..43307ee8e
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -0,0 +1,236 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMPRESSDEV_PMD_H_
+#define _RTE_COMPRESSDEV_PMD_H_
+
+/** @file
+ * RTE comp PMD APIs
+ *
+ * @note
+ * These APIs are for comp PMDs only and user applications should not call
+ * them directly.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <string.h>
+
+#include <rte_dev.h>
+#include <rte_common.h>
+
+#include "rte_compressdev.h"
+#include "rte_compressdev_internal.h"
+
+#define RTE_COMPRESSDEV_PMD_NAME_ARG			("name")
+#define RTE_COMPRESSDEV_PMD_SOCKET_ID_ARG		("socket_id")
+
+static const char * const compressdev_pmd_valid_params[] = {
+	RTE_COMPRESSDEV_PMD_NAME_ARG,
+	RTE_COMPRESSDEV_PMD_SOCKET_ID_ARG
+};
+
+/**
+ * @internal
+ * Initialisation parameters for comp devices
+ */
+struct rte_compressdev_pmd_init_params {
+	char name[RTE_COMPRESSDEV_NAME_MAX_LEN];
+	int socket_id;
+};
+
+/** Global structure used for maintaining state of allocated comp devices */
+struct rte_compressdev_global {
+	struct rte_compressdev *devs;	/**< Device information array */
+	struct rte_compressdev_data *data[RTE_COMPRESS_MAX_DEVS];
+	/**< Device private data */
+	uint8_t nb_devs;		/**< Number of devices found */
+	uint8_t max_devs;		/**< Max number of devices */
+};
+
+/** Pointer to global array of comp devices */
+extern struct rte_compressdev *rte_compressdevs;
+/** Pointer to global comp devices data structure */
+extern struct rte_compressdev_global *rte_compressdev_globals;
+
+/**
+ * Get the rte_compressdev structure device pointer for the named device.
+ *
+ * @param name
+ *   Compress device name
+ * @return
+ *   - The rte_compressdev structure pointer for the given device identifier.
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_get_named_dev(const char *name);
+
+/**
+ * Definitions of all functions exported by a driver through the
+ * the generic structure of type *comp_dev_ops* supplied in the
+ * *rte_compressdev* structure associated with a device.
+ */
+
+/**
+ * Function used to configure device.
+ *
+ * @param dev
+ *   Compress device
+ * @param config
+ *   Compress device configurations
+ * @return
+ *   Returns 0 on success
+ */
+typedef int (*compressdev_configure_t)(struct rte_compressdev *dev,
+		struct rte_compressdev_config *config);
+
+/**
+ * Function used to start a configured device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *   Returns 0 on success
+ */
+typedef int (*compressdev_start_t)(struct rte_compressdev *dev);
+
+/**
+ * Function used to stop a configured device.
+ *
+ * @param dev
+ *   Compress device
+ */
+typedef void (*compressdev_stop_t)(struct rte_compressdev *dev);
+
+/**
+ * Function used to close a configured device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ * - 0 on success.
+ * - EAGAIN if can't close as device is busy
+ */
+typedef int (*compressdev_close_t)(struct rte_compressdev *dev);
+
+
+/**
+ * Function used to get specific information of a device.
+ *
+ * @param dev
+ *   Compress device
+ */
+typedef void (*compressdev_info_get_t)(struct rte_compressdev *dev,
+				struct rte_compressdev_info *dev_info);
+
+/** comp device operations function pointer table */
+struct rte_compressdev_ops {
+	compressdev_configure_t dev_configure;	/**< Configure device. */
+	compressdev_start_t dev_start;		/**< Start device. */
+	compressdev_stop_t dev_stop;		/**< Stop device. */
+	compressdev_close_t dev_close;		/**< Close device. */
+
+	compressdev_info_get_t dev_infos_get;	/**< Get device info. */
+};
+
+/**
+ * @internal
+ *
+ * Function for internal use by dummy drivers primarily, e.g. ring-based
+ * driver.
+ * Allocates a new compressdev slot for an comp device and returns the pointer
+ * to that slot for the driver to use.
+ *
+ * @param name
+ *   Unique identifier name for each device
+ * @param socket_id
+ *   Socket to allocate resources on
+ * @return
+ *   - Slot in the rte_dev_devices array for a new device;
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_allocate(const char *name, int socket_id);
+
+/**
+ * @internal
+ *
+ * Function for internal use by dummy drivers primarily, e.g. ring-based
+ * driver.
+ * Release the specified compressdev device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *   - 0 on success, negative on error
+ */
+int __rte_experimental
+rte_compressdev_pmd_release_device(struct rte_compressdev *dev);
+
+
+/**
+ * @internal
+ *
+ * PMD assist function to parse initialisation arguments for comp driver
+ * when creating a new comp PMD device instance.
+ *
+ * PMD driver should set default values for that PMD before calling function,
+ * these default values will be over-written with successfully parsed values
+ * from args string.
+ *
+ * @param params
+ *   Parsed PMD initialisation parameters
+ * @param args
+ *   Input argument string to parse
+ * @return
+ *  - 0 on success
+ *  - errno on failure
+ */
+int __rte_experimental
+rte_compressdev_pmd_parse_input_args(
+		struct rte_compressdev_pmd_init_params *params,
+		const char *args);
+
+/**
+ * @internal
+ *
+ * PMD assist function to provide boiler plate code for comp driver to create
+ * and allocate resources for a new comp PMD device instance.
+ *
+ * @param name
+ *   Compress device name
+ * @param device
+ *   Base device instance
+ * @param params
+ *   PMD initialisation parameters
+ * @return
+ *  - comp device instance on success
+ *  - NULL on creation failure
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_create(const char *name,
+		struct rte_device *device,
+		size_t private_data_size,
+		struct rte_compressdev_pmd_init_params *params);
+
+/**
+ * @internal
+ *
+ * PMD assist function to provide boiler plate code for comp driver to
+ * destroy and free resources associated with a comp PMD device instance.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *  - 0 on success
+ *  - errno on failure
+ */
+int __rte_experimental
+rte_compressdev_pmd_destroy(struct rte_compressdev *dev);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_COMPRESSDEV_PMD_H_ */
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
new file mode 100644
index 000000000..a996abcd2
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -0,0 +1,24 @@
+EXPERIMENTAL {
+        global:
+
+	rte_compressdev_callback_register;
+	rte_compressdev_callback_unregister;
+	rte_compressdev_close;
+	rte_compressdev_configure;
+	rte_compressdev_count;
+	rte_compressdev_devices_get;
+	rte_compressdev_info_get;
+	rte_compressdev_name_get;
+	rte_compressdev_pmd_allocate;
+	rte_compressdev_pmd_callback_process;
+	rte_compressdev_pmd_create;
+	rte_compressdev_pmd_destroy;
+	rte_compressdev_pmd_get_named_dev;
+	rte_compressdev_pmd_parse_input_args;
+	rte_compressdev_pmd_release_device;
+	rte_compressdev_socket_id;
+	rte_compressdev_start;
+	rte_compressdev_stop;
+
+        local: *;
+};
diff --git a/lib/meson.build b/lib/meson.build
index ef6159170..ae718aa3f 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -14,7 +14,7 @@ libraries = [ 'compat', # just a header, used for versioning
 	'hash',    # efd depends on this
 	'kvargs',  # cryptodev depends on this
 	'acl', 'bbdev', 'bitratestats', 'cfgfile',
-	'cmdline', 'cryptodev',
+	'cmdline', 'compressdev', 'cryptodev',
 	'distributor', 'efd', 'eventdev',
 	'gro', 'gso', 'ip_frag', 'jobstats',
 	'kni', 'latencystats', 'lpm', 'member',
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index afd0aff8c..db89cbc16 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -94,6 +94,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF)           += -lrte_mbuf
 _LDLIBS-$(CONFIG_RTE_LIBRTE_NET)            += -lrte_net
 _LDLIBS-$(CONFIG_RTE_LIBRTE_ETHER)          += -lrte_ethdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_BBDEV)          += -lrte_bbdev
+_LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV)    += -lrte_compressdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_CRYPTODEV)      += -lrte_cryptodev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY)       += -lrte_security
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV)       += -lrte_eventdev
-- 
2.14.3

[dpdk-dev] [PATCH v5 02/13] compressdev: add queue pair management

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Add functions to manage device queue pairs.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 135 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           |  45 +++++++
 lib/librte_compressdev/rte_compressdev_internal.h  |   5 +
 lib/librte_compressdev/rte_compressdev_pmd.h       |  47 +++++++
 lib/librte_compressdev/rte_compressdev_version.map |   2 +
 5 files changed, 234 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 751517c3d..67e7ad12f 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -227,10 +227,100 @@ rte_compressdev_pmd_release_device(struct rte_compressdev *compressdev)
 	return 0;
 }
 
+uint16_t __rte_experimental
+rte_compressdev_queue_pair_count(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	dev = &rte_comp_devices[dev_id];
+	return dev->data->nb_queue_pairs;
+}
+
+static int
+rte_compressdev_queue_pairs_config(struct rte_compressdev *dev,
+		uint16_t nb_qpairs, int socket_id)
+{
+	struct rte_compressdev_info dev_info;
+	void **qp;
+	unsigned int i;
+
+	if ((dev == NULL) || (nb_qpairs < 1)) {
+		COMPRESSDEV_LOG(ERR, "invalid param: dev %p, nb_queues %u",
+							dev, nb_qpairs);
+		return -EINVAL;
+	}
+
+	COMPRESSDEV_LOG(DEBUG, "Setup %d queues pairs on device %u",
+			nb_qpairs, dev->data->dev_id);
+
+	memset(&dev_info, 0, sizeof(struct rte_compressdev_info));
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_infos_get, -ENOTSUP);
+	(*dev->dev_ops->dev_infos_get)(dev, &dev_info);
+
+	if ((dev_info.max_nb_queue_pairs != 0) &&
+			(nb_qpairs > dev_info.max_nb_queue_pairs)) {
+		COMPRESSDEV_LOG(ERR, "Invalid num queue_pairs (%u) for dev %u",
+				nb_qpairs, dev->data->dev_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->queue_pairs == NULL) { /* first time configuration */
+		dev->data->queue_pairs = rte_zmalloc_socket(
+				"compressdev->queue_pairs",
+				sizeof(dev->data->queue_pairs[0]) * nb_qpairs,
+				RTE_CACHE_LINE_SIZE, socket_id);
+
+		if (dev->data->queue_pairs == NULL) {
+			dev->data->nb_queue_pairs = 0;
+			COMPRESSDEV_LOG(ERR,
+			"failed to get memory for qp meta data, nb_queues %u",
+							nb_qpairs);
+			return -(ENOMEM);
+		}
+	} else { /* re-configure */
+		int ret;
+		uint16_t old_nb_queues = dev->data->nb_queue_pairs;
+
+		qp = dev->data->queue_pairs;
+
+		RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->queue_pair_release,
+				-ENOTSUP);
+
+		for (i = nb_qpairs; i < old_nb_queues; i++) {
+			ret = (*dev->dev_ops->queue_pair_release)(dev, i);
+			if (ret < 0)
+				return ret;
+		}
+
+		qp = rte_realloc(qp, sizeof(qp[0]) * nb_qpairs,
+				RTE_CACHE_LINE_SIZE);
+		if (qp == NULL) {
+			COMPRESSDEV_LOG(ERR,
+			"failed to realloc qp meta data, nb_queues %u",
+						nb_qpairs);
+			return -(ENOMEM);
+		}
+
+		if (nb_qpairs > old_nb_queues) {
+			uint16_t new_qs = nb_qpairs - old_nb_queues;
+
+			memset(qp + old_nb_queues, 0,
+				sizeof(qp[0]) * new_qs);
+		}
+
+		dev->data->queue_pairs = qp;
+
+	}
+	dev->data->nb_queue_pairs = nb_qpairs;
+	return 0;
+}
+
 int __rte_experimental
 rte_compressdev_configure(uint8_t dev_id, struct rte_compressdev_config *config)
 {
 	struct rte_compressdev *dev;
+	int diag;
 
 	if (!rte_compressdev_is_valid_dev(dev_id)) {
 		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
@@ -247,6 +337,16 @@ rte_compressdev_configure(uint8_t dev_id, struct rte_compressdev_config *config)
 
 	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_configure, -ENOTSUP);
 
+	/* Setup new number of queue pairs and reconfigure device. */
+	diag = rte_compressdev_queue_pairs_config(dev, config->nb_queue_pairs,
+			config->socket_id);
+	if (diag != 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev%d rte_comp_dev_queue_pairs_config = %d",
+				dev_id, diag);
+		return diag;
+	}
+
 	return (*dev->dev_ops->dev_configure)(dev, config);
 }
 
@@ -336,6 +436,41 @@ rte_compressdev_close(uint8_t dev_id)
 	return 0;
 }
 
+int __rte_experimental
+rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
+		uint32_t max_inflight_ops, int socket_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+	if (queue_pair_id >= dev->data->nb_queue_pairs) {
+		COMPRESSDEV_LOG(ERR, "Invalid queue_pair_id=%d", queue_pair_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->dev_started) {
+		COMPRESSDEV_LOG(ERR,
+		    "device %d must be stopped to allow configuration", dev_id);
+		return -EBUSY;
+	}
+
+	if (max_inflight_ops == 0) {
+		COMPRESSDEV_LOG(ERR,
+			"Invalid maximum number of inflight operations");
+		return -EINVAL;
+	}
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->queue_pair_setup, -ENOTSUP);
+
+	return (*dev->dev_ops->queue_pair_setup)(dev, queue_pair_id,
+			max_inflight_ops, socket_id);
+}
+
 void __rte_experimental
 rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 6150850db..39b4c83cf 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -22,6 +22,10 @@ extern "C" {
 /**  comp device information */
 struct rte_compressdev_info {
 	const char *driver_name;		/**< Driver name. */
+	uint16_t max_nb_queue_pairs;
+	/**< Maximum number of queues pairs supported by device.
+	 * (If 0, there is no limit in maximum number of queue pairs)
+	 */
 };
 
 /**
@@ -80,6 +84,9 @@ rte_compressdev_socket_id(uint8_t dev_id);
 /** Compress device configuration structure */
 struct rte_compressdev_config {
 	int socket_id;
+	/**< Socket on which to allocate resources */
+	uint16_t nb_queue_pairs;
+	/**< Total number of queue pairs to configure on a device */
 };
 
 /**
@@ -141,6 +148,44 @@ rte_compressdev_stop(uint8_t dev_id);
 int __rte_experimental
 rte_compressdev_close(uint8_t dev_id);
 
+/**
+ * Allocate and set up a receive queue pair for a device.
+ * This should only be called when the device is stopped.
+ *
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param queue_pair_id
+ *   The index of the queue pairs to set up. The
+ *   value must be in the range [0, nb_queue_pair - 1]
+ *   previously supplied to rte_compressdev_configure()
+ * @param max_inflight_ops
+ *   Max number of ops which the qp will have to
+ *   accommodate simultaneously
+ * @param socket_id
+ *   The *socket_id* argument is the socket identifier
+ *   in case of NUMA. The value can be *SOCKET_ID_ANY*
+ *   if there is no NUMA constraint for the DMA memory
+ *   allocated for the receive queue pair
+ * @return
+ *   - 0: Success, queue pair correctly set up.
+ *   - <0: Queue pair configuration failed
+ */
+int __rte_experimental
+rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
+		uint32_t max_inflight_ops, int socket_id);
+
+/**
+ * Get the number of queue pairs on a specific comp device
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - The number of configured queue pairs.
+ */
+uint16_t __rte_experimental
+rte_compressdev_queue_pair_count(uint8_t dev_id);
+
 /**
  * Retrieve the contextual information of a device.
  *
diff --git a/lib/librte_compressdev/rte_compressdev_internal.h b/lib/librte_compressdev/rte_compressdev_internal.h
index 0a2ddcb2b..57af163c1 100644
--- a/lib/librte_compressdev/rte_compressdev_internal.h
+++ b/lib/librte_compressdev/rte_compressdev_internal.h
@@ -58,6 +58,11 @@ struct rte_compressdev_data {
 	uint8_t dev_started : 1;
 	/**< Device state: STARTED(1)/STOPPED(0) */
 
+	void **queue_pairs;
+	/**< Array of pointers to queue pairs. */
+	uint16_t nb_queue_pairs;
+	/**< Number of device queue pairs */
+
 	void *dev_private;
 	/**< PMD-specific private data */
 } __rte_cache_aligned;
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
index 43307ee8e..14ce76b44 100644
--- a/lib/librte_compressdev/rte_compressdev_pmd.h
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -125,6 +125,48 @@ typedef int (*compressdev_close_t)(struct rte_compressdev *dev);
 typedef void (*compressdev_info_get_t)(struct rte_compressdev *dev,
 				struct rte_compressdev_info *dev_info);
 
+/**
+ * Setup a queue pair for a device.
+ *
+ * @param dev
+ *   Compress device
+ * @param qp_id
+ *   Queue pair identifier
+ * @param max_inflight_ops
+ *   Max inflight ops which qp must accommodate
+ * @param socket_id
+ *   Socket identifier
+ * @return
+ *   Returns 0 on success.
+ */
+typedef int (*compressdev_queue_pair_setup_t)(struct rte_compressdev *dev,
+		uint16_t qp_id,	uint32_t max_inflight_ops, int socket_id);
+
+/**
+ * Release memory resources allocated by given queue pair.
+ *
+ * @param dev
+ *   Compress device
+ * @param qp_id
+ *   Queue pair identifier
+ * @return
+ * - 0 on success.
+ * - EAGAIN if can't close as device is busy
+ */
+typedef int (*compressdev_queue_pair_release_t)(struct rte_compressdev *dev,
+		uint16_t qp_id);
+
+/**
+ * Get number of available queue pairs of a device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *   Returns number of queue pairs on success.
+ */
+typedef uint32_t (*compressdev_queue_pair_count_t)(struct rte_compressdev *dev);
+
+
 /** comp device operations function pointer table */
 struct rte_compressdev_ops {
 	compressdev_configure_t dev_configure;	/**< Configure device. */
@@ -133,6 +175,11 @@ struct rte_compressdev_ops {
 	compressdev_close_t dev_close;		/**< Close device. */
 
 	compressdev_info_get_t dev_infos_get;	/**< Get device info. */
+
+	compressdev_queue_pair_setup_t queue_pair_setup;
+	/**< Set up a device queue pair. */
+	compressdev_queue_pair_release_t queue_pair_release;
+	/**< Release a queue pair. */
 };
 
 /**
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index a996abcd2..182a371d8 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -16,6 +16,8 @@ EXPERIMENTAL {
 	rte_compressdev_pmd_get_named_dev;
 	rte_compressdev_pmd_parse_input_args;
 	rte_compressdev_pmd_release_device;
+	rte_compressdev_queue_pair_count;
+	rte_compressdev_queue_pair_setup;
 	rte_compressdev_socket_id;
 	rte_compressdev_start;
 	rte_compressdev_stop;
-- 
2.14.3

[dpdk-dev] [PATCH v5 03/13] compressdev: add compression specific data

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added structures and enums specific to compression,
including the compression operation structure and the
different supported algorithms, checksums and compression
levels.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 doc/api/doxy-api-index.md          |   1 +
 lib/librte_compressdev/Makefile    |   1 +
 lib/librte_compressdev/meson.build |   3 +-
 lib/librte_compressdev/rte_comp.h  | 322 +++++++++++++++++++++++++++++++++++++
 4 files changed, 326 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_compressdev/rte_comp.h

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index c076a27e6..6331cddcc 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -46,6 +46,7 @@ The public API headers are grouped by topics:
   [cryptodev]          (@ref rte_cryptodev.h),
   [security]           (@ref rte_security.h),
   [compressdev]        (@ref rte_compressdev.h),
+  [compress]           (@ref rte_comp.h),
   [eventdev]           (@ref rte_eventdev.h),
   [event_eth_rx_adapter]   (@ref rte_event_eth_rx_adapter.h),
   [rawdev]             (@ref rte_rawdev.h),
diff --git a/lib/librte_compressdev/Makefile b/lib/librte_compressdev/Makefile
index b84769417..4dae6ca42 100644
--- a/lib/librte_compressdev/Makefile
+++ b/lib/librte_compressdev/Makefile
@@ -19,6 +19,7 @@ LDLIBS += -lrte_eal -lrte_mempool -lrte_kvargs
 SRCS-y += rte_compressdev.c rte_compressdev_pmd.c
 
 # export include files
+SYMLINK-y-include += rte_comp.h
 SYMLINK-y-include += rte_compressdev.h
 # export include files (for PMDs)
 SYMLINK-y-include += rte_compressdev_pmd.h
diff --git a/lib/librte_compressdev/meson.build b/lib/librte_compressdev/meson.build
index 895e959d3..2115961c2 100644
--- a/lib/librte_compressdev/meson.build
+++ b/lib/librte_compressdev/meson.build
@@ -6,5 +6,6 @@ sources = files('rte_compressdev.c',
 	'rte_compressdev_pmd.c')
 headers = files('rte_compressdev.h',
 	'rte_compressdev_pmd.h',
-	'rte_compressdev_internal.h')
+	'rte_compressdev_internal.h',
+	'rte_comp.h')
 deps += ['kvargs', 'mbuf']
diff --git a/lib/librte_compressdev/rte_comp.h b/lib/librte_compressdev/rte_comp.h
new file mode 100644
index 000000000..2b7fbb2e8
--- /dev/null
+++ b/lib/librte_compressdev/rte_comp.h
@@ -0,0 +1,322 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMP_H_
+#define _RTE_COMP_H_
+
+/**
+ * @file rte_comp.h
+ *
+ * RTE definitions for Data Compression Service
+ *
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <string.h>
+#include <rte_mempool.h>
+
+/** Status of comp operation */
+enum rte_comp_op_status {
+	RTE_COMP_OP_STATUS_SUCCESS = 0,
+	/**< Operation completed successfully */
+	RTE_COMP_OP_STATUS_NOT_PROCESSED,
+	/**< Operation has not yet been processed by the device */
+	RTE_COMP_OP_STATUS_INVALID_ARGS,
+	/**< Operation failed due to invalid arguments in request */
+	RTE_COMP_OP_STATUS_ERROR,
+	/**< Error handling operation */
+	RTE_COMP_OP_STATUS_INVALID_STATE,
+	/**< Operation is invoked in invalid state */
+	RTE_COMP_OP_STATUS_OUT_OF_SPACE_TERMINATED,
+	/**< Output buffer ran out of space before operation completed.
+	 * Error case. Application must resubmit all data with a larger
+	 * output buffer.
+	 */
+	RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE,
+	/**< Output buffer ran out of space before operation completed, but this
+	 * is not an error case. Output data up to op.produced can be used and
+	 * next op in the stream should continue on from op.consumed+1.
+	 */
+};
+
+/** Compression Algorithms */
+enum rte_comp_algorithm {
+	RTE_COMP_ALGO_UNSPECIFIED = 0,
+	/** No Compression algorithm */
+	RTE_COMP_ALGO_NULL,
+	/**< No compression.
+	 * Pass-through, data is copied unchanged from source buffer to
+	 * destination buffer.
+	 */
+	RTE_COMP_ALGO_DEFLATE,
+	/**< DEFLATE compression algorithm
+	 * https://tools.ietf.org/html/rfc1951
+	 */
+	RTE_COMP_ALGO_LZS,
+	/**< LZS compression algorithm
+	 * https://tools.ietf.org/html/rfc2395
+	 */
+	RTE_COMP_ALGO_LIST_END
+};
+
+/**< Compression Level.
+ * The number is interpreted by each PMD differently. However, lower numbers
+ * give fastest compression, at the expense of compression ratio while
+ * higher numbers may give better compression ratios but are likely slower.
+ */
+#define	RTE_COMP_LEVEL_PMD_DEFAULT	(-1)
+/** Use PMD Default */
+#define	RTE_COMP_LEVEL_NONE		(0)
+/** Output uncompressed blocks if supported by the specified algorithm */
+#define RTE_COMP_LEVEL_MIN		(1)
+/** Use minimum compression level supported by the PMD */
+#define RTE_COMP_LEVEL_MAX		(9)
+/** Use maximum compression level supported by the PMD */
+
+/** Compression checksum types */
+enum rte_comp_checksum_type {
+	RTE_COMP_CHECKSUM_NONE,
+	/**< No checksum generated */
+	RTE_COMP_CHECKSUM_CRC32,
+	/**< Generates a CRC32 checksum, as used by gzip */
+	RTE_COMP_CHECKSUM_ADLER32,
+	/**< Generates an Adler-32 checksum, as used by zlib */
+	RTE_COMP_CHECKSUM_CRC32_ADLER32,
+	/**< Generates both Adler-32 and CRC32 checksums, concatenated.
+	 * CRC32 is in the lower 32bits, Adler-32 in the upper 32 bits.
+	 */
+};
+
+
+/** Compression Huffman Type - used by DEFLATE algorithm */
+enum rte_comp_huffman {
+	RTE_COMP_HUFFMAN_DEFAULT,
+	/**< PMD may choose which Huffman codes to use */
+	RTE_COMP_HUFFMAN_FIXED,
+	/**< Use Fixed Huffman codes */
+	RTE_COMP_HUFFMAN_DYNAMIC,
+	/**< Use Dynamic Huffman codes */
+};
+
+/** Compression flush flags */
+enum rte_comp_flush_flag {
+	RTE_COMP_FLUSH_NONE,
+	/**< Data is not flushed. Output may remain in the compressor and be
+	 * processed during a following op. It may not be possible to decompress
+	 * output until a later op with some other flush flag has been sent.
+	 */
+	RTE_COMP_FLUSH_SYNC,
+	/**< All data should be flushed to output buffer. Output data can be
+	 * decompressed. However state and history is not cleared, so future
+	 * operations may use history from this operation.
+	 */
+	RTE_COMP_FLUSH_FULL,
+	/**< All data should be flushed to output buffer. Output data can be
+	 * decompressed. State and history data is cleared, so future
+	 * ops will be independent of ops processed before this.
+	 */
+	RTE_COMP_FLUSH_FINAL
+	/**< Same as RTE_COMP_FLUSH_FULL but if op.algo is RTE_COMP_ALGO_DEFLATE
+	 * then bfinal bit is set in the last block.
+	 */
+};
+
+/** Compression transform types */
+enum rte_comp_xform_type {
+	RTE_COMP_COMPRESS,
+	/**< Compression service - compress */
+	RTE_COMP_DECOMPRESS,
+	/**< Compression service - decompress */
+};
+
+/** Compression operation type */
+enum rte_comp_op_type {
+	RTE_COMP_OP_STATELESS,
+	/**< All data to be processed is submitted in the op, no state or
+	 * history from previous ops is used and none will be stored for future
+	 * ops. Flush flag must be set to either FLUSH_FULL or FLUSH_FINAL.
+	 */
+	RTE_COMP_OP_STATEFUL
+	/**< There may be more data to be processed after this op, it's part of
+	 * a stream of data. State and history from previous ops can be used
+	 * and resulting state and history can be stored for future ops,
+	 * depending on flush flag.
+	 */
+};
+
+
+/** Parameters specific to the deflate algorithm */
+struct rte_comp_deflate_params {
+	enum rte_comp_huffman huffman;
+	/**< Compression huffman encoding type */
+};
+
+/** Setup Data for compression */
+struct rte_comp_compress_xform {
+	enum rte_comp_algorithm algo;
+	/**< Algorithm to use for compress operation */
+	union {
+		struct rte_comp_deflate_params deflate;
+		/**< Parameters specific to the deflate algorithm */
+	}; /**< Algorithm specific parameters */
+	int level;
+	/**< Compression level */
+	uint8_t window_size;
+	/**< Base two log value of sliding window to be used. If window size
+	 * can't be supported by the PMD then it may fall back to a smaller
+	 * size. This is likely to result in a worse compression ratio.
+	 */
+	enum rte_comp_checksum_type chksum;
+	/**< Type of checksum to generate on the uncompressed data */
+};
+
+/**
+ * Setup Data for decompression.
+ */
+struct rte_comp_decompress_xform {
+	enum rte_comp_algorithm algo;
+	/**< Algorithm to use for decompression */
+	enum rte_comp_checksum_type chksum;
+	/**< Type of checksum to generate on the decompressed data */
+	uint8_t window_size;
+	/**< Base two log value of sliding window which was used to generate
+	 * compressed data. If window size can't be supported by the PMD then
+	 * setup of stream or private_xform should fail.
+	 */
+};
+
+/**
+ * Compression transform structure.
+ *
+ * This is used to specify the compression transforms required.
+ * Each transform structure can hold a single transform, the type field is
+ * used to specify which transform is contained within the union.
+ * There are no chain cases currently supported, just single xforms of
+ *  - compress-only
+ *  - decompress-only
+ *
+ */
+struct rte_comp_xform {
+	struct rte_comp_xform *next;
+	/**< next xform in chain */
+	enum rte_comp_xform_type type;
+	/**< xform type */
+	union {
+		struct rte_comp_compress_xform compress;
+		/**< xform for compress operation */
+		struct rte_comp_decompress_xform decompress;
+		/**< decompress xform */
+	};
+};
+
+/**
+ * Compression Operation.
+ *
+ * This structure contains data relating to performing a compression
+ * operation on the referenced mbuf data buffers.
+ *
+ * Comp operations are enqueued and dequeued in comp PMDs using the
+ * rte_compressdev_enqueue_burst() / rte_compressdev_dequeue_burst() APIs
+ */
+struct rte_comp_op {
+	enum rte_comp_op_type op_type;
+	union {
+		void *private_xform;
+		/**< Stateless private PMD data derived from an rte_comp_xform.
+		 * A handle returned by rte_compressdev_private_xform_create()
+		 * must be attached to operations of op_type RTE_COMP_STATELESS.
+		 */
+		void *stream;
+		/**< Private PMD data derived initially from an rte_comp_xform,
+		 * which holds state and history data and evolves as operations
+		 * are processed. rte_compressdev_stream_create() must be called
+		 * on a device for all STATEFUL data streams and the resulting
+		 * stream attached to the one or more operations associated
+		 * with the data stream.
+		 * All operations in a stream must be sent to the same device.
+		 */
+	};
+
+	struct rte_mempool *mempool;
+	/**< Pool from which operation is allocated */
+	rte_iova_t iova_addr;
+	/**< IOVA address of this operation */
+	struct rte_mbuf *m_src;
+	/**< source mbuf
+	 * The total size of the input buffer(s) can be retrieved using
+	 * rte_pktmbuf_data_len(m_src)
+	 */
+	struct rte_mbuf *m_dst;
+	/**< destination mbuf
+	 * The total size of the output buffer(s) can be retrieved using
+	 * rte_pktmbuf_data_len(m_dst)
+	 */
+
+	struct {
+		uint32_t offset;
+		/**< Starting point for compression or decompression,
+		 * specified as number of bytes from start of packet in
+		 * source buffer.
+		 * Starting point for checksum generation in compress direction.
+		 */
+		uint32_t length;
+		/**< The length, in bytes, of the data in source buffer
+		 * to be compressed or decompressed.
+		 * Also the length of the data over which the checksum
+		 * should be generated in compress direction
+		 */
+	} src;
+	struct {
+		uint32_t offset;
+		/**< Starting point for writing output data, specified as
+		 * number of bytes from start of packet in dest
+		 * buffer. Starting point for checksum generation in
+		 * decompress direction.
+		 */
+	} dst;
+	enum rte_comp_flush_flag flush_flag;
+	/**< Defines flush characteristics for the output data.
+	 * Only applicable in compress direction
+	 */
+	uint64_t input_chksum;
+	/**< An input checksum can be provided to generate a
+	 * cumulative checksum across sequential blocks in a STATELESS stream.
+	 * Checksum type is as specified in xform chksum_type
+	 */
+	uint64_t output_chksum;
+	/**< If a checksum is generated it will be written in here.
+	 * Checksum type is as specified in xform chksum_type.
+	 */
+	uint32_t consumed;
+	/**< The number of bytes from the source buffer
+	 * which were compressed/decompressed.
+	 */
+	uint32_t produced;
+	/**< The number of bytes written to the destination buffer
+	 * which were compressed/decompressed.
+	 */
+	uint64_t debug_status;
+	/**<
+	 * Status of the operation is returned in the status param.
+	 * This field allows the PMD to pass back extra
+	 * pmd-specific debug information. Value is not defined on the API.
+	 */
+	uint8_t status;
+	/**<
+	 * Operation status - use values from enum rte_comp_status.
+	 * This is reset to
+	 * RTE_COMP_OP_STATUS_NOT_PROCESSED on allocation from mempool and
+	 * will be set to RTE_COMP_OP_STATUS_SUCCESS after operation
+	 * is successfully processed by a PMD
+	 */
+} __rte_cache_aligned;
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_COMP_H_ */
-- 
2.14.3

[dpdk-dev] [PATCH v5 04/13] compressdev: add enqueue/dequeue functions

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           |  22 +++++
 lib/librte_compressdev/rte_compressdev.h           | 107 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |   2 +
 3 files changed, 131 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 67e7ad12f..5393da6d6 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -471,6 +471,28 @@ rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
 			max_inflight_ops, socket_id);
 }
 
+uint16_t __rte_experimental
+rte_compressdev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	struct rte_compressdev *dev = &rte_compressdevs[dev_id];
+
+	nb_ops = (*dev->dequeue_burst)
+			(dev->data->queue_pairs[qp_id], ops, nb_ops);
+
+	return nb_ops;
+}
+
+uint16_t __rte_experimental
+rte_compressdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	struct rte_compressdev *dev = &rte_compressdevs[dev_id];
+
+	return (*dev->enqueue_burst)(
+			dev->data->queue_pairs[qp_id], ops, nb_ops);
+}
+
 void __rte_experimental
 rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 39b4c83cf..0a4f657b0 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -19,6 +19,8 @@ extern "C" {
 
 #include <rte_common.h>
 
+#include "rte_comp.h"
+
 /**  comp device information */
 struct rte_compressdev_info {
 	const char *driver_name;		/**< Driver name. */
@@ -203,6 +205,111 @@ rte_compressdev_queue_pair_count(uint8_t dev_id);
 void __rte_experimental
 rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info);
 
+/**
+ *
+ * Dequeue a burst of processed compression operations from a queue on the comp
+ * device. The dequeued operation are stored in *rte_comp_op* structures
+ * whose pointers are supplied in the *ops* array.
+ *
+ * The rte_compressdev_dequeue_burst() function returns the number of ops
+ * actually dequeued, which is the number of *rte_comp_op* data structures
+ * effectively supplied into the *ops* array.
+ *
+ * A return value equal to *nb_ops* indicates that the queue contained
+ * at least *nb_ops* operations, and this is likely to signify that other
+ * processed operations remain in the devices output queue. Applications
+ * implementing a "retrieve as many processed operations as possible" policy
+ * can check this specific case and keep invoking the
+ * rte_compressdev_dequeue_burst() function until a value less than
+ * *nb_ops* is returned.
+ *
+ * The rte_compressdev_dequeue_burst() function does not provide any error
+ * notification to avoid the corresponding overhead.
+ *
+ * @note: operation ordering is not maintained within the queue pair.
+ *
+ * @note: In case op status = OUT_OF_SPACE_TERMINATED, op.consumed=0 and the
+ * op must be resubmitted with the same input data and a larger output buffer.
+ * op.produced is usually 0, but in decompression cases a PMD may return > 0
+ * and the application may find it useful to inspect that data.
+ * This status is only returned on STATELESS ops.
+ *
+ * @note: In case op status = OUT_OF_SPACE_RECOVERABLE, op.produced can be used
+ * and next op in stream should continue on from op.consumed+1 with a fresh
+ * output buffer.
+ * Consumed=0, produced=0 is an unusual but allowed case. There may be useful
+ * state/history stored in the PMD, even though no output was produced yet.
+ *
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param qp_id
+ *   The index of the queue pair from which to retrieve
+ *   processed operations. The value must be in the range
+ *   [0, nb_queue_pair - 1] previously supplied to
+ *   rte_compressdev_configure()
+ * @param ops
+ *   The address of an array of pointers to
+ *   *rte_comp_op* structures that must be
+ *   large enough to store *nb_ops* pointers in it
+ * @param nb_ops
+ *   The maximum number of operations to dequeue
+ * @return
+ *   - The number of operations actually dequeued, which is the number
+ *   of pointers to *rte_comp_op* structures effectively supplied to the
+ *   *ops* array.
+ */
+uint16_t __rte_experimental
+rte_compressdev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_comp_op **ops, uint16_t nb_ops);
+
+/**
+ * Enqueue a burst of operations for processing on a compression device.
+ *
+ * The rte_compressdev_enqueue_burst() function is invoked to place
+ * comp operations on the queue *qp_id* of the device designated by
+ * its *dev_id*.
+ *
+ * The *nb_ops* parameter is the number of operations to process which are
+ * supplied in the *ops* array of *rte_comp_op* structures.
+ *
+ * The rte_compressdev_enqueue_burst() function returns the number of
+ * operations it actually enqueued for processing. A return value equal to
+ * *nb_ops* means that all packets have been enqueued.
+ *
+ * @note All compression operations are Out-of-place (OOP) operations,
+ * as the size of the output data is different to the size of the input data.
+ *
+ * @note The flush flag only applies to operations which return SUCCESS.
+ * In OUT_OF_SPACE cases whether STATEFUL or STATELESS, data in dest buffer
+ * is as if flush flag was FLUSH_NONE.
+ * @note flush flag only applies in compression direction. It has no meaning
+ * for decompression.
+ * @note: operation ordering is not maintained within the queue pair.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param qp_id
+ *   The index of the queue pair on which operations
+ *   are to be enqueued for processing. The value
+ *   must be in the range [0, nb_queue_pairs - 1]
+ *   previously supplied to *rte_compressdev_configure*
+ * @param ops
+ *   The address of an array of *nb_ops* pointers
+ *   to *rte_comp_op* structures which contain
+ *   the operations to be processed
+ * @param nb_ops
+ *   The number of operations to process
+ * @return
+ *   The number of operations actually enqueued on the device. The return
+ *   value can be less than the value of the *nb_ops* parameter when the
+ *   comp devices queue is full or if invalid parameters are specified in
+ *   a *rte_comp_op*.
+ */
+uint16_t __rte_experimental
+rte_compressdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_comp_op **ops, uint16_t nb_ops);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 182a371d8..882395885 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -6,7 +6,9 @@ EXPERIMENTAL {
 	rte_compressdev_close;
 	rte_compressdev_configure;
 	rte_compressdev_count;
+	rte_compressdev_dequeue_burst;
 	rte_compressdev_devices_get;
+	rte_compressdev_enqueue_burst;
 	rte_compressdev_info_get;
 	rte_compressdev_name_get;
 	rte_compressdev_pmd_allocate;
-- 
2.14.3

[dpdk-dev] [PATCH v5 05/13] compressdev: add operation management

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added functions to allocate and free compression operations.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/Makefile                    |   2 +-
 lib/librte_compressdev/meson.build                 |   3 +-
 lib/librte_compressdev/rte_comp.c                  | 176 +++++++++++++++++++++
 lib/librte_compressdev/rte_comp.h                  |  64 ++++++++
 lib/librte_compressdev/rte_compressdev_internal.h  |  54 +++++++
 lib/librte_compressdev/rte_compressdev_version.map |   4 +
 6 files changed, 301 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_compressdev/rte_comp.c

diff --git a/lib/librte_compressdev/Makefile b/lib/librte_compressdev/Makefile
index 4dae6ca42..7ef89e613 100644
--- a/lib/librte_compressdev/Makefile
+++ b/lib/librte_compressdev/Makefile
@@ -16,7 +16,7 @@ CFLAGS += -DALLOW_EXPERIMENTAL_API
 LDLIBS += -lrte_eal -lrte_mempool -lrte_kvargs
 
 # library source files
-SRCS-y += rte_compressdev.c rte_compressdev_pmd.c
+SRCS-y += rte_compressdev.c rte_compressdev_pmd.c rte_comp.c
 
 # export include files
 SYMLINK-y-include += rte_comp.h
diff --git a/lib/librte_compressdev/meson.build b/lib/librte_compressdev/meson.build
index 2115961c2..5416571c9 100644
--- a/lib/librte_compressdev/meson.build
+++ b/lib/librte_compressdev/meson.build
@@ -3,7 +3,8 @@
 
 allow_experimental_apis = true
 sources = files('rte_compressdev.c',
-	'rte_compressdev_pmd.c')
+	'rte_compressdev_pmd.c',
+	'rte_comp.c')
 headers = files('rte_compressdev.h',
 	'rte_compressdev_pmd.h',
 	'rte_compressdev_internal.h',
diff --git a/lib/librte_compressdev/rte_comp.c b/lib/librte_compressdev/rte_comp.c
new file mode 100644
index 000000000..eb99e99d2
--- /dev/null
+++ b/lib/librte_compressdev/rte_comp.c
@@ -0,0 +1,176 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#include "rte_comp.h"
+#include "rte_compressdev.h"
+#include "rte_compressdev_internal.h"
+
+/**
+ * Reset the fields of an operation to their default values.
+ *
+ * @note The private data associated with the operation is not zeroed.
+ *
+ * @param op
+ *   The operation to be reset
+ */
+static inline void
+rte_comp_op_reset(struct rte_comp_op *op)
+{
+	struct rte_mempool *tmp_mp = op->mempool;
+	rte_iova_t tmp_iova_addr = op->iova_addr;
+
+	memset(op, 0, sizeof(struct rte_comp_op));
+	op->status = RTE_COMP_OP_STATUS_NOT_PROCESSED;
+	op->iova_addr = tmp_iova_addr;
+	op->mempool = tmp_mp;
+}
+
+/**
+ * Private data structure belonging to an operation pool.
+ */
+struct rte_comp_op_pool_private {
+	uint16_t user_size;
+	/**< Size of private user data with each operation. */
+};
+
+/**
+ * Bulk allocate raw element from mempool and return as comp operations
+ *
+ * @param mempool
+ *   Compress operation mempool
+ * @param ops
+ *   Array to place allocated operations
+ * @param nb_ops
+ *   Number of operations to allocate
+ * @return
+ *   - 0: Success
+ *   - -ENOENT: Not enough entries in the mempool; no ops are retrieved.
+ */
+static inline int
+rte_comp_op_raw_bulk_alloc(struct rte_mempool *mempool,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	if (rte_mempool_get_bulk(mempool, (void **)ops, nb_ops) == 0)
+		return nb_ops;
+
+	return 0;
+}
+
+/** Initialise rte_comp_op mempool element */
+static void
+rte_comp_op_init(struct rte_mempool *mempool,
+		__rte_unused void *opaque_arg,
+		void *_op_data,
+		__rte_unused unsigned int i)
+{
+	struct rte_comp_op *op = _op_data;
+
+	memset(_op_data, 0, mempool->elt_size);
+
+	op->status = RTE_COMP_OP_STATUS_NOT_PROCESSED;
+	op->iova_addr = rte_mem_virt2iova(_op_data);
+	op->mempool = mempool;
+}
+
+struct rte_mempool * __rte_experimental
+rte_comp_op_pool_create(const char *name,
+		unsigned int nb_elts, unsigned int cache_size,
+		uint16_t user_size, int socket_id)
+{
+	struct rte_comp_op_pool_private *priv;
+
+	unsigned int elt_size = sizeof(struct rte_comp_op) + user_size;
+
+	/* lookup mempool in case already allocated */
+	struct rte_mempool *mp = rte_mempool_lookup(name);
+
+	if (mp != NULL) {
+		priv = (struct rte_comp_op_pool_private *)
+				rte_mempool_get_priv(mp);
+
+		if (mp->elt_size != elt_size ||
+				mp->cache_size < cache_size ||
+				mp->size < nb_elts ||
+				priv->user_size <  user_size) {
+			mp = NULL;
+			COMPRESSDEV_LOG(ERR,
+		"Mempool %s already exists but with incompatible parameters",
+					name);
+			return NULL;
+		}
+		return mp;
+	}
+
+	mp = rte_mempool_create(
+			name,
+			nb_elts,
+			elt_size,
+			cache_size,
+			sizeof(struct rte_comp_op_pool_private),
+			NULL,
+			NULL,
+			rte_comp_op_init,
+			NULL,
+			socket_id,
+			0);
+
+	if (mp == NULL) {
+		COMPRESSDEV_LOG(ERR, "Failed to create mempool %s", name);
+		return NULL;
+	}
+
+	priv = (struct rte_comp_op_pool_private *)
+			rte_mempool_get_priv(mp);
+
+	priv->user_size = user_size;
+
+	return mp;
+}
+
+struct rte_comp_op * __rte_experimental
+rte_comp_op_alloc(struct rte_mempool *mempool)
+{
+	struct rte_comp_op *op = NULL;
+	int retval;
+
+	retval = rte_comp_op_raw_bulk_alloc(mempool, &op, 1);
+	if (unlikely(retval < 0))
+		return NULL;
+
+	rte_comp_op_reset(op);
+
+	return op;
+}
+
+int __rte_experimental
+rte_comp_op_bulk_alloc(struct rte_mempool *mempool,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	int ret;
+	uint16_t i;
+
+	ret = rte_comp_op_raw_bulk_alloc(mempool, ops, nb_ops);
+	if (unlikely(ret < nb_ops))
+		return ret;
+
+	for (i = 0; i < nb_ops; i++)
+		rte_comp_op_reset(ops[i]);
+
+	return nb_ops;
+}
+
+/**
+ * free operation structure
+ * If operation has been allocate from a rte_mempool, then the operation will
+ * be returned to the mempool.
+ *
+ * @param op
+ *   Compress operation
+ */
+void __rte_experimental
+rte_comp_op_free(struct rte_comp_op *op)
+{
+	if (op != NULL && op->mempool != NULL)
+		rte_mempool_put(op->mempool, op);
+}
diff --git a/lib/librte_compressdev/rte_comp.h b/lib/librte_compressdev/rte_comp.h
index 2b7fbb2e8..5020aed33 100644
--- a/lib/librte_compressdev/rte_comp.h
+++ b/lib/librte_compressdev/rte_comp.h
@@ -315,6 +315,70 @@ struct rte_comp_op {
 	 */
 } __rte_cache_aligned;
 
+/**
+ * Creates an operation pool
+ *
+ * @param name
+ *   Compress pool name
+ * @param nb_elts
+ *   Number of elements in pool
+ * @param cache_size
+ *   Number of elements to cache on lcore, see
+ *   *rte_mempool_create* for further details about cache size
+ * @param user_size
+ *   Size of private data to allocate for user with each operation
+ * @param socket_id
+ *   Socket to identifier allocate memory on
+ * @return
+ *  - On success pointer to mempool
+ *  - On failure NULL
+ */
+struct rte_mempool * __rte_experimental
+rte_comp_op_pool_create(const char *name,
+		unsigned int nb_elts, unsigned int cache_size,
+		uint16_t user_size, int socket_id);
+
+/**
+ * Allocate an operation from a mempool with default parameters set
+ *
+ * @param mempool
+ *   Compress operation mempool
+ *
+ * @return
+ * - On success returns a valid rte_comp_op structure
+ * - On failure returns NULL
+ */
+struct rte_comp_op * __rte_experimental
+rte_comp_op_alloc(struct rte_mempool *mempool);
+
+/**
+ * Bulk allocate operations from a mempool with default parameters set
+ *
+ * @param mempool
+ *   Compress operation mempool
+ * @param ops
+ *   Array to place allocated operations
+ * @param nb_ops
+ *   Number of operations to allocate
+ * @return
+ *   - 0: Success
+ *   - -ENOENT: Not enough entries in the mempool; no ops are retrieved.
+ */
+int __rte_experimental
+rte_comp_op_bulk_alloc(struct rte_mempool *mempool,
+		struct rte_comp_op **ops, uint16_t nb_ops);
+
+/**
+ * Free operation structure
+ * If operation has been allocate from a rte_mempool, then the operation will
+ * be returned to the mempool.
+ *
+ * @param op
+ *   Compress operation
+ */
+void __rte_experimental
+rte_comp_op_free(struct rte_comp_op *op);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_compressdev/rte_compressdev_internal.h b/lib/librte_compressdev/rte_compressdev_internal.h
index 57af163c1..f74c652d9 100644
--- a/lib/librte_compressdev/rte_compressdev_internal.h
+++ b/lib/librte_compressdev/rte_compressdev_internal.h
@@ -10,6 +10,8 @@
  */
 #include <rte_log.h>
 
+#include "rte_comp.h"
+
 #define RTE_COMPRESSDEV_NAME_MAX_LEN	(64)
 /**< Max length of name of comp PMD */
 
@@ -19,9 +21,61 @@ extern int compressdev_logtype;
 	rte_log(RTE_LOG_ ## level, compressdev_logtype, "%s(): "fmt "\n", \
 			__func__, ##args)
 
+/**
+ * Definitions of all functions exported by a driver through the
+ * the generic structure of type *comp_dev_ops* supplied in the
+ * *rte_compressdev* structure associated with a device.
+ */
+
+/**
+ * Dequeue processed packets from queue pair of a device.
+ *
+ * @param qp
+ *   The queue pair from which to retrieve
+ *   processed operations.
+ * @param ops
+ *   The address of an array of pointers to
+ *   *rte_comp_op* structures that must be
+ *   large enough to store *nb_ops* pointers in it
+ * @param nb_ops
+ *   The maximum number of operations to dequeue
+ * @return
+ *   - The number of operations actually dequeued, which is the number
+ *   of pointers to *rte_comp_op* structures effectively supplied to the
+ *   *ops* array.
+ */
+typedef uint16_t (*compressdev_dequeue_pkt_burst_t)(void *qp,
+		struct rte_comp_op **ops, uint16_t nb_ops);
+
+/**
+ * Enqueue a burst of operations for processing.
+ *
+ * @param qp
+ *   The queue pair on which operations
+ *   are to be enqueued for processing
+ * @param ops
+ *   The address of an array of *nb_ops* pointers
+ *   to *rte_comp_op* structures which contain
+ *   the operations to be processed
+ * @param nb_ops
+ *   The number of operations to process
+ * @return
+ *   The number of operations actually enqueued on the device. The return
+ *   value can be less than the value of the *nb_ops* parameter when the
+ *   comp devices queue is full or if invalid parameters are specified in
+ *   a *rte_comp_op*.
+ */
+
+typedef uint16_t (*compressdev_enqueue_pkt_burst_t)(void *qp,
+		struct rte_comp_op **ops, uint16_t nb_ops);
 
 /** The data structure associated with each comp device. */
 struct rte_compressdev {
+	compressdev_dequeue_pkt_burst_t dequeue_burst;
+	/**< Pointer to PMD receive function */
+	compressdev_enqueue_pkt_burst_t enqueue_burst;
+	/**< Pointer to PMD transmit function */
+
 	struct rte_compressdev_data *data;
 	/**< Pointer to device data */
 	struct rte_compressdev_ops *dev_ops;
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 882395885..292417fde 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -23,6 +23,10 @@ EXPERIMENTAL {
 	rte_compressdev_socket_id;
 	rte_compressdev_start;
 	rte_compressdev_stop;
+	rte_comp_op_alloc;
+	rte_comp_op_bulk_alloc;
+	rte_comp_op_free;
+	rte_comp_op_pool_create;
 
         local: *;
 };
-- 
2.14.3

[dpdk-dev] [PATCH v5 06/13] compressdev: support stateless operations

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added private transform data (priv_xform) in compression
operation, which will contain the private data from each
PMD to support stateless operations.
Also, added functions to create/free this data.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 48 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 49 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 40 ++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  2 +
 4 files changed, 139 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 5393da6d6..31f134224 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -513,6 +513,54 @@ rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
 	dev_info->driver_name = dev->device->driver->name;
 }
 
+int __rte_experimental
+rte_compressdev_private_xform_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **priv_xform)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_get_dev(dev_id);
+
+	if (xform == NULL || priv_xform == NULL || dev == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->private_xform_create, -ENOTSUP);
+	ret = (*dev->dev_ops->private_xform_create)(dev, xform, priv_xform);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to create private_xform: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
+int __rte_experimental
+rte_compressdev_private_xform_free(uint8_t dev_id, void *priv_xform)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_get_dev(dev_id);
+
+	if (dev == NULL || priv_xform == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->private_xform_free, -ENOTSUP);
+	ret = dev->dev_ops->private_xform_free(dev, priv_xform);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to free private xform: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
 const char * __rte_experimental
 rte_compressdev_name_get(uint8_t dev_id)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 0a4f657b0..a48b59d5e 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -89,6 +89,8 @@ struct rte_compressdev_config {
 	/**< Socket on which to allocate resources */
 	uint16_t nb_queue_pairs;
 	/**< Total number of queue pairs to configure on a device */
+	uint16_t max_nb_priv_xforms;
+	/**< Max number of private_xforms which will be created on the device */
 };
 
 /**
@@ -310,6 +312,53 @@ uint16_t __rte_experimental
 rte_compressdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 		struct rte_comp_op **ops, uint16_t nb_ops);
 
+
+/**
+ * This should alloc a private_xform from the device's mempool and initialise
+ * it. The application should call this API when setting up for stateless
+ * processing on a device. If it returns non-shareable, then the appl cannot
+ * share this handle with multiple in-flight ops and should call this API again
+ * to get a separate handle for every in-flight op.
+ * The handle returned is only valid for use with ops of op_type STATELESS.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param xform
+ *   xform data
+ * @param private_xform
+ *   Pointer to where PMD's private_xform handle should be stored
+ *
+ * @return
+ *  - if successful returns 0
+ *    and valid private_xform handle
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private_xform could not be allocated.
+ */
+int __rte_experimental
+rte_compressdev_private_xform_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **private_xform);
+
+/**
+ * This should clear the private_xform and return it to the device's mempool.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ *
+ * @param private_xform
+ *   PMD's private_xform data
+ *
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -EBUSY if can't free private_xform due to inflight operations
+ */
+int __rte_experimental
+rte_compressdev_private_xform_free(uint8_t dev_id, void *private_xform);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
index 14ce76b44..45ab61c25 100644
--- a/lib/librte_compressdev/rte_compressdev_pmd.h
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -166,6 +166,41 @@ typedef int (*compressdev_queue_pair_release_t)(struct rte_compressdev *dev,
  */
 typedef uint32_t (*compressdev_queue_pair_count_t)(struct rte_compressdev *dev);
 
+/**
+ * Create driver private_xform data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param xform
+ *   xform data
+ * @param private_xform
+ *   ptr where handle of pmd's private_xform data should be stored
+ * @return
+ *  - if successful returns 0
+ *    and valid private_xform handle
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private_xform could not be allocated.
+ */
+typedef int (*compressdev_private_xform_create_t)(struct rte_compressdev *dev,
+		const struct rte_comp_xform *xform, void **private_xform);
+
+/**
+ * Free driver private_xform data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param private_xform
+ *   handle of pmd's private_xform data
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -EBUSY if can't free private_xform due to inflight operations
+ */
+typedef int (*compressdev_private_xform_free_t)(struct rte_compressdev *dev,
+		void *private_xform);
 
 /** comp device operations function pointer table */
 struct rte_compressdev_ops {
@@ -180,6 +215,11 @@ struct rte_compressdev_ops {
 	/**< Set up a device queue pair. */
 	compressdev_queue_pair_release_t queue_pair_release;
 	/**< Release a queue pair. */
+
+	compressdev_private_xform_create_t private_xform_create;
+	/**< Create a comp private_xform and initialise its private data. */
+	compressdev_private_xform_free_t private_xform_free;
+	/**< Free a comp private_xform's data. */
 };
 
 /**
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 292417fde..d755bb65e 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -18,6 +18,8 @@ EXPERIMENTAL {
 	rte_compressdev_pmd_get_named_dev;
 	rte_compressdev_pmd_parse_input_args;
 	rte_compressdev_pmd_release_device;
+	rte_compressdev_private_xform_create;
+	rte_compressdev_private_xform_free;
 	rte_compressdev_queue_pair_count;
 	rte_compressdev_queue_pair_setup;
 	rte_compressdev_socket_id;
-- 
2.14.3

[dpdk-dev] [PATCH v5 07/13] compressdev: support stateful operations

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added stream data (stream) in compression operation,
which will contain the private data from each PMD
to support stateful operations.
Also, added functions to create/free this data.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 49 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 50 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 41 ++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  2 +
 4 files changed, 142 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 31f134224..79dc8f6e5 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -561,6 +561,55 @@ rte_compressdev_private_xform_free(uint8_t dev_id, void *priv_xform)
 	return 0;
 }
 
+int __rte_experimental
+rte_compressdev_stream_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **stream)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_get_dev(dev_id);
+
+	if (xform == NULL || dev == NULL || stream == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->stream_create, -ENOTSUP);
+	ret = (*dev->dev_ops->stream_create)(dev, xform, stream);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to create stream: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
+
+int __rte_experimental
+rte_compressdev_stream_free(uint8_t dev_id, void *stream)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_get_dev(dev_id);
+
+	if (dev == NULL || stream == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->stream_free, -ENOTSUP);
+	ret = dev->dev_ops->stream_free(dev, stream);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to free stream: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
 const char * __rte_experimental
 rte_compressdev_name_get(uint8_t dev_id)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index a48b59d5e..5d4218f8c 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -91,6 +91,8 @@ struct rte_compressdev_config {
 	/**< Total number of queue pairs to configure on a device */
 	uint16_t max_nb_priv_xforms;
 	/**< Max number of private_xforms which will be created on the device */
+	uint16_t max_nb_streams;
+	/**< Max number of streams which will be created on the device */
 };
 
 /**
@@ -313,6 +315,54 @@ rte_compressdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 		struct rte_comp_op **ops, uint16_t nb_ops);
 
 
+/**
+ * This should alloc a stream from the device's mempool and initialise it.
+ * The application should call this API when setting up for the stateful
+ * processing of a set of data on a device. The API can be called multiple
+ * times to set up a stream for each data set. The handle returned is only for
+ * use with ops of op_type STATEFUL and must be passed to the PMD
+ * with every op in the data stream
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param xform
+ *   xform data
+ * @param stream
+ *   Pointer to where PMD's private stream handle should be stored
+ *
+ * @return
+ *  - 0 if successful and valid stream handle
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private stream could not be allocated.
+ *
+ */
+int __rte_experimental
+rte_compressdev_stream_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **stream);
+
+/**
+ * This should clear the stream and return it to the device's mempool.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ *
+ * @param stream
+ *   PMD's private stream data
+ *
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -EBUSY if can't free stream as there are inflight operations
+ */
+int __rte_experimental
+rte_compressdev_stream_free(uint8_t dev_id, void *stream);
+
 /**
  * This should alloc a private_xform from the device's mempool and initialise
  * it. The application should call this API when setting up for stateless
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
index 45ab61c25..c08a3ee63 100644
--- a/lib/librte_compressdev/rte_compressdev_pmd.h
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -166,6 +166,42 @@ typedef int (*compressdev_queue_pair_release_t)(struct rte_compressdev *dev,
  */
 typedef uint32_t (*compressdev_queue_pair_count_t)(struct rte_compressdev *dev);
 
+/**
+ * Create driver private stream data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param xform
+ *   xform data
+ * @param stream
+ *   ptr where handle of pmd's private stream data should be stored
+ * @return
+ *  - Returns 0 if private stream structure has been created successfully.
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private stream could not be allocated.
+ */
+typedef int (*compressdev_stream_create_t)(struct rte_compressdev *dev,
+		const struct rte_comp_xform *xform, void **stream);
+
+/**
+ * Free driver private stream data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param stream
+ *   handle of pmd's private stream data
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -EBUSY if can't free stream as there are inflight operations
+ */
+typedef int (*compressdev_stream_free_t)(struct rte_compressdev *dev,
+		void *stream);
+
 /**
  * Create driver private_xform data.
  *
@@ -216,6 +252,11 @@ struct rte_compressdev_ops {
 	compressdev_queue_pair_release_t queue_pair_release;
 	/**< Release a queue pair. */
 
+	compressdev_stream_create_t stream_create;
+	/**< Create a comp stream and initialise its private data. */
+	compressdev_stream_free_t stream_free;
+	/**< Free a comp stream's private data. */
+
 	compressdev_private_xform_create_t private_xform_create;
 	/**< Create a comp private_xform and initialise its private data. */
 	compressdev_private_xform_free_t private_xform_free;
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index d755bb65e..ca4d9d3d8 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -25,6 +25,8 @@ EXPERIMENTAL {
 	rte_compressdev_socket_id;
 	rte_compressdev_start;
 	rte_compressdev_stop;
+	rte_compressdev_stream_create;
+	rte_compressdev_stream_free;
 	rte_comp_op_alloc;
 	rte_comp_op_bulk_alloc;
 	rte_comp_op_free;
-- 
2.14.3

[dpdk-dev] [PATCH v5 08/13] compressdev: support hash operations

[Pablo de Lara]

From: Shally Verma <shally.verma@caviumnetworks.com>

- Added hash algo enumeration and params in xform and rte_comp_op
- Updated compress/decompress xform to input hash algorithm
- Updated struct rte_comp_op to input hash buffer

User in capability query will know about support hashes via
device info comp_feature_flag. If supported, application can initialize
desired algorithm enumeration in xform structure and pass valid hash
buffer during enqueue_burst().

Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Sunila Sahu <sunila.sahu@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_comp.h | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/lib/librte_compressdev/rte_comp.h b/lib/librte_compressdev/rte_comp.h
index 5020aed33..639c577b6 100644
--- a/lib/librte_compressdev/rte_comp.h
+++ b/lib/librte_compressdev/rte_comp.h
@@ -63,6 +63,17 @@ enum rte_comp_algorithm {
 	RTE_COMP_ALGO_LIST_END
 };
 
+/** Compression Hash Algorithms */
+enum rte_comp_hash_algorithm {
+	RTE_COMP_HASH_ALGO_UNSPECIFIED = 0,
+	/**< No hash */
+	RTE_COMP_HASH_ALGO_SHA1,
+	/**< SHA1 hash algorithm */
+	RTE_COMP_HASH_ALGO_SHA2_256,
+	/**< SHA256 hash algorithm of SHA2 family */
+	RTE_COMP_HASH_ALGO_LIST_END
+};
+
 /**< Compression Level.
  * The number is interpreted by each PMD differently. However, lower numbers
  * give fastest compression, at the expense of compression ratio while
@@ -172,6 +183,10 @@ struct rte_comp_compress_xform {
 	 */
 	enum rte_comp_checksum_type chksum;
 	/**< Type of checksum to generate on the uncompressed data */
+	enum rte_comp_hash_algorithm hash_algo;
+	/**< Hash algorithm to be used with compress operation. Hash is always
+	 * done on plaintext.
+	 */
 };
 
 /**
@@ -187,6 +202,10 @@ struct rte_comp_decompress_xform {
 	 * compressed data. If window size can't be supported by the PMD then
 	 * setup of stream or private_xform should fail.
 	 */
+	enum rte_comp_hash_algorithm hash_algo;
+	/**< Hash algorithm to be used with decompress operation. Hash is always
+	 * done on plaintext.
+	 */
 };
 
 /**
@@ -278,6 +297,19 @@ struct rte_comp_op {
 		 * decompress direction.
 		 */
 	} dst;
+	struct {
+		uint8_t *digest;
+		/**< Output buffer to store hash output, if enabled in xform.
+		 * Buffer would contain valid value only after an op with
+		 * flush flag = RTE_COMP_FLUSH_FULL/FLUSH_FINAL is processed
+		 * successfully.
+		 *
+		 * Length of buffer should be contiguous and large enough to
+		 * accommodate digest produced by specific hash algo.
+		 */
+		rte_iova_t iova_addr;
+		/**< IO address of the buffer */
+	} hash;
 	enum rte_comp_flush_flag flush_flag;
 	/**< Defines flush characteristics for the output data.
 	 * Only applicable in compress direction
-- 
2.14.3

[dpdk-dev] [PATCH v5 09/13] compressdev: add device feature flags

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 21 ++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 33 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  1 +
 3 files changed, 55 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 79dc8f6e5..8e6b08f38 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -31,6 +31,27 @@ static struct rte_compressdev_global compressdev_globals = {
 
 struct rte_compressdev_global *rte_compressdev_globals = &compressdev_globals;
 
+const char * __rte_experimental
+rte_compressdev_get_feature_name(uint64_t flag)
+{
+	switch (flag) {
+	case RTE_COMPDEV_FF_HW_ACCELERATED:
+		return "HW_ACCELERATED";
+	case RTE_COMPDEV_FF_CPU_SSE:
+		return "CPU_SSE";
+	case RTE_COMPDEV_FF_CPU_AVX:
+		return "CPU_AVX";
+	case RTE_COMPDEV_FF_CPU_AVX2:
+		return "CPU_AVX2";
+	case RTE_COMPDEV_FF_CPU_AVX512:
+		return "CPU_AVX512";
+	case RTE_COMPDEV_FF_CPU_NEON:
+		return "CPU_NEON";
+	default:
+		return NULL;
+	}
+}
+
 static struct rte_compressdev *
 rte_compressdev_get_dev(uint8_t dev_id)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 5d4218f8c..1559d96ba 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -21,9 +21,42 @@ extern "C" {
 
 #include "rte_comp.h"
 
+/**
+ * compression device supported feature flags
+ *
+ * @note New features flags should be added to the end of the list
+ *
+ * Keep these flags synchronised with rte_compressdev_get_feature_name()
+ */
+#define	RTE_COMPDEV_FF_HW_ACCELERATED		(1ULL << 0)
+/**< Operations are off-loaded to an external hardware accelerator */
+#define	RTE_COMPDEV_FF_CPU_SSE			(1ULL << 1)
+/**< Utilises CPU SIMD SSE instructions */
+#define	RTE_COMPDEV_FF_CPU_AVX			(1ULL << 2)
+/**< Utilises CPU SIMD AVX instructions */
+#define	RTE_COMPDEV_FF_CPU_AVX2			(1ULL << 3)
+/**< Utilises CPU SIMD AVX2 instructions */
+#define	RTE_COMPDEV_FF_CPU_AVX512		(1ULL << 4)
+/**< Utilises CPU SIMD AVX512 instructions */
+#define	RTE_COMPDEV_FF_CPU_NEON			(1ULL << 5)
+/**< Utilises CPU NEON instructions */
+
+/**
+ * Get the name of a compress device feature flag.
+ *
+ * @param flag
+ *   The mask describing the flag
+ *
+ * @return
+ *   The name of this flag, or NULL if it's not a valid feature flag.
+ */
+const char * __rte_experimental
+rte_compressdev_get_feature_name(uint64_t flag);
+
 /**  comp device information */
 struct rte_compressdev_info {
 	const char *driver_name;		/**< Driver name. */
+	uint64_t feature_flags;			/**< Feature flags */
 	uint16_t max_nb_queue_pairs;
 	/**< Maximum number of queues pairs supported by device.
 	 * (If 0, there is no limit in maximum number of queue pairs)
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index ca4d9d3d8..f9223bcfe 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -9,6 +9,7 @@ EXPERIMENTAL {
 	rte_compressdev_dequeue_burst;
 	rte_compressdev_devices_get;
 	rte_compressdev_enqueue_burst;
+	rte_compressdev_get_feature_name;
 	rte_compressdev_info_get;
 	rte_compressdev_name_get;
 	rte_compressdev_pmd_allocate;
-- 
2.14.3

[dpdk-dev] [PATCH v5 10/13] compressdev: add compression service feature flags

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_comp.c                  | 31 +++++++++++++++
 lib/librte_compressdev/rte_comp.h                  | 46 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  1 +
 3 files changed, 78 insertions(+)

diff --git a/lib/librte_compressdev/rte_comp.c b/lib/librte_compressdev/rte_comp.c
index eb99e99d2..d596ba872 100644
--- a/lib/librte_compressdev/rte_comp.c
+++ b/lib/librte_compressdev/rte_comp.c
@@ -6,6 +6,37 @@
 #include "rte_compressdev.h"
 #include "rte_compressdev_internal.h"
 
+const char * __rte_experimental
+rte_comp_get_feature_name(uint64_t flag)
+{
+	switch (flag) {
+	case RTE_COMP_FF_STATEFUL_COMPRESSION:
+		return "STATEFUL_COMPRESSION";
+	case RTE_COMP_FF_STATEFUL_DECOMPRESSION:
+		return "STATEFUL_DECOMPRESSION";
+	case RTE_COMP_FF_MBUF_SCATTER_GATHER:
+		return "MBUF_SCATTER_GATHER";
+	case RTE_COMP_FF_MULTI_PKT_CHECKSUM:
+		return "MULTI_PKT_CHECKSUM";
+	case RTE_COMP_FF_ADLER32_CHECKSUM:
+		return "ADLER32_CHECKSUM";
+	case RTE_COMP_FF_CRC32_CHECKSUM:
+		return "CRC32_CHECKSUM";
+	case RTE_COMP_FF_CRC32_ADLER32_CHECKSUM:
+		return "CRC32_ADLER32_CHECKSUM";
+	case RTE_COMP_FF_NONCOMPRESSED_BLOCKS:
+		return "NONCOMPRESSED_BLOCKS";
+	case RTE_COMP_FF_SHA1_HASH:
+		return "SHA1_HASH";
+	case RTE_COMP_FF_SHA2_SHA256_HASH:
+		return "SHA2_SHA256_HASH";
+	case RTE_COMP_FF_SHAREABLE_PRIV_XFORM:
+		return "SHAREABLE_PRIV_XFORM";
+	default:
+		return NULL;
+	}
+}
+
 /**
  * Reset the fields of an operation to their default values.
  *
diff --git a/lib/librte_compressdev/rte_comp.h b/lib/librte_compressdev/rte_comp.h
index 639c577b6..870a5337e 100644
--- a/lib/librte_compressdev/rte_comp.h
+++ b/lib/librte_compressdev/rte_comp.h
@@ -19,6 +19,40 @@ extern "C" {
 #include <string.h>
 #include <rte_mempool.h>
 
+/**
+ * compression service feature flags
+ *
+ * @note New features flags should be added to the end of the list
+ *
+ * Keep these flags synchronised with rte_comp_get_feature_name()
+ */
+#define RTE_COMP_FF_STATEFUL_COMPRESSION	(1ULL << 0)
+/**< Stateful compression is supported */
+#define RTE_COMP_FF_STATEFUL_DECOMPRESSION	(1ULL << 1)
+/**< Stateful decompression is supported */
+#define	RTE_COMP_FF_MBUF_SCATTER_GATHER		(1ULL << 2)
+/**< Scatter-gather mbufs are supported */
+#define RTE_COMP_FF_ADLER32_CHECKSUM		(1ULL << 3)
+/**< Adler-32 Checksum is supported */
+#define RTE_COMP_FF_CRC32_CHECKSUM		(1ULL << 4)
+/**< CRC32 Checksum is supported */
+#define RTE_COMP_FF_CRC32_ADLER32_CHECKSUM	(1ULL << 5)
+/**< Adler-32/CRC32 Checksum is supported */
+#define RTE_COMP_FF_MULTI_PKT_CHECKSUM		(1ULL << 6)
+/**< Generation of checksum across multiple stateless packets is supported */
+#define RTE_COMP_FF_SHA1_HASH			(1ULL << 7)
+/**< SHA1 Hash is supported */
+#define RTE_COMP_FF_SHA2_SHA256_HASH		(1ULL << 8)
+/**< SHA256 Hash of SHA2 family is supported */
+#define RTE_COMP_FF_NONCOMPRESSED_BLOCKS	(1ULL << 9)
+/**< Creation of non-compressed blocks using RTE_COMP_LEVEL_NONE is supported */
+#define RTE_COMP_FF_SHAREABLE_PRIV_XFORM	(1ULL << 10)
+/**< Private xforms created by the PMD can be shared
+ * across multiple stateless operations. If not set, then app needs
+ * to create as many priv_xforms as it expects to have stateless
+ * operations in-flight.
+ */
+
 /** Status of comp operation */
 enum rte_comp_op_status {
 	RTE_COMP_OP_STATUS_SUCCESS = 0,
@@ -411,6 +445,18 @@ rte_comp_op_bulk_alloc(struct rte_mempool *mempool,
 void __rte_experimental
 rte_comp_op_free(struct rte_comp_op *op);
 
+/**
+ * Get the name of a compress service feature flag
+ *
+ * @param flag
+ *   The mask describing the flag
+ *
+ * @return
+ *   The name of this flag, or NULL if it's not a valid feature flag.
+ */
+const char * __rte_experimental
+rte_comp_get_feature_name(uint64_t flag);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index f9223bcfe..5ed3080b4 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -28,6 +28,7 @@ EXPERIMENTAL {
 	rte_compressdev_stop;
 	rte_compressdev_stream_create;
 	rte_compressdev_stream_free;
+	rte_comp_get_feature_name;
 	rte_comp_op_alloc;
 	rte_comp_op_bulk_alloc;
 	rte_comp_op_free;
-- 
2.14.3

[dpdk-dev] [PATCH v5 11/13] compressdev: add device stats

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 40 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 39 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 26 ++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  2 ++
 4 files changed, 107 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 8e6b08f38..8d141b798 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -514,6 +514,46 @@ rte_compressdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 			dev->data->queue_pairs[qp_id], ops, nb_ops);
 }
 
+int __rte_experimental
+rte_compressdev_stats_get(uint8_t dev_id, struct rte_compressdev_stats *stats)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%d", dev_id);
+		return -ENODEV;
+	}
+
+	if (stats == NULL) {
+		COMPRESSDEV_LOG(ERR, "Invalid stats ptr");
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+	memset(stats, 0, sizeof(*stats));
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->stats_get, -ENOTSUP);
+	(*dev->dev_ops->stats_get)(dev, stats);
+	return 0;
+}
+
+void __rte_experimental
+rte_compressdev_stats_reset(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	RTE_FUNC_PTR_OR_RET(*dev->dev_ops->stats_reset);
+	(*dev->dev_ops->stats_reset)(dev);
+}
+
+
 void __rte_experimental
 rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 1559d96ba..91c1792d7 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -63,6 +63,19 @@ struct rte_compressdev_info {
 	 */
 };
 
+/** comp device statistics */
+struct rte_compressdev_stats {
+	uint64_t enqueued_count;
+	/**< Count of all operations enqueued */
+	uint64_t dequeued_count;
+	/**< Count of all operations dequeued */
+
+	uint64_t enqueue_err_count;
+	/**< Total error count on operations enqueued */
+	uint64_t dequeue_err_count;
+	/**< Total error count on operations dequeued */
+};
+
 /**
  * Get the compress device name given a device identifier.
  *
@@ -225,6 +238,32 @@ rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
 uint16_t __rte_experimental
 rte_compressdev_queue_pair_count(uint8_t dev_id);
 
+
+/**
+ * Retrieve the general I/O statistics of a device.
+ *
+ * @param dev_id
+ *   The identifier of the device
+ * @param stats
+ *   A pointer to a structure of type
+ *   *rte_compressdev_stats* to be filled with the
+ *   values of device counters
+ * @return
+ *   - Zero if successful.
+ *   - Non-zero otherwise.
+ */
+int __rte_experimental
+rte_compressdev_stats_get(uint8_t dev_id, struct rte_compressdev_stats *stats);
+
+/**
+ * Reset the general I/O statistics of a device.
+ *
+ * @param dev_id
+ *   The identifier of the device.
+ */
+void __rte_experimental
+rte_compressdev_stats_reset(uint8_t dev_id);
+
 /**
  * Retrieve the contextual information of a device.
  *
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
index c08a3ee63..38e9ea02b 100644
--- a/lib/librte_compressdev/rte_compressdev_pmd.h
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -116,6 +116,27 @@ typedef void (*compressdev_stop_t)(struct rte_compressdev *dev);
 typedef int (*compressdev_close_t)(struct rte_compressdev *dev);
 
 
+/**
+ * Function used to get statistics of a device.
+ *
+ * @param dev
+ *   Compress device
+ * @param stats
+ *   Compress device stats to populate
+ */
+typedef void (*compressdev_stats_get_t)(struct rte_compressdev *dev,
+				struct rte_compressdev_stats *stats);
+
+
+/**
+ * Function used to reset statistics of a device.
+ *
+ * @param dev
+ *   Compress device
+ */
+typedef void (*compressdev_stats_reset_t)(struct rte_compressdev *dev);
+
+
 /**
  * Function used to get specific information of a device.
  *
@@ -247,6 +268,11 @@ struct rte_compressdev_ops {
 
 	compressdev_info_get_t dev_infos_get;	/**< Get device info. */
 
+	compressdev_stats_get_t stats_get;
+	/**< Get device statistics. */
+	compressdev_stats_reset_t stats_reset;
+	/**< Reset device statistics. */
+
 	compressdev_queue_pair_setup_t queue_pair_setup;
 	/**< Set up a device queue pair. */
 	compressdev_queue_pair_release_t queue_pair_release;
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 5ed3080b4..8e4004631 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -25,6 +25,8 @@ EXPERIMENTAL {
 	rte_compressdev_queue_pair_setup;
 	rte_compressdev_socket_id;
 	rte_compressdev_start;
+	rte_compressdev_stats_get;
+	rte_compressdev_stats_reset;
 	rte_compressdev_stop;
 	rte_compressdev_stream_create;
 	rte_compressdev_stream_free;
-- 
2.14.3

[dpdk-dev] [PATCH v5 12/13] compressdev: add device capabilities

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added structure which each PMD will fill out,
providing the capabilities of each driver
(containing mainly which compression services
it supports).

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 23 ++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 35 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  1 +
 3 files changed, 59 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 8d141b798..a5a1deaff 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -31,6 +31,29 @@ static struct rte_compressdev_global compressdev_globals = {
 
 struct rte_compressdev_global *rte_compressdev_globals = &compressdev_globals;
 
+const struct rte_compressdev_capabilities * __rte_experimental
+rte_compressdev_capability_get(uint8_t dev_id,
+			enum rte_comp_algorithm algo)
+{
+	const struct rte_compressdev_capabilities *capability;
+	struct rte_compressdev_info dev_info;
+	int i = 0;
+
+	if (dev_id >= compressdev_globals.nb_devs) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%d", dev_id);
+		return NULL;
+	}
+	rte_compressdev_info_get(dev_id, &dev_info);
+
+	while ((capability = &dev_info.capabilities[i++])->algo !=
+			RTE_COMP_ALGO_UNSPECIFIED){
+		if (capability->algo == algo)
+			return capability;
+	}
+
+	return NULL;
+}
+
 const char * __rte_experimental
 rte_compressdev_get_feature_name(uint64_t flag)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 91c1792d7..9e2fe95ca 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -21,6 +21,39 @@ extern "C" {
 
 #include "rte_comp.h"
 
+/**
+ * Parameter log base 2 range description.
+ * Final value will be 2^value.
+ */
+struct rte_param_log2_range {
+	uint8_t min;	/**< Minimum log2 value */
+	uint8_t max;	/**< Maximum log2 value */
+	uint8_t increment;
+	/**< If a range of sizes are supported,
+	 * this parameter is used to indicate
+	 * increments in base 2 log byte value
+	 * that are supported between the minimum and maximum
+	 */
+};
+
+/** Structure used to capture a capability of a comp device */
+struct rte_compressdev_capabilities {
+	enum rte_comp_algorithm algo;
+	/* Compression algorithm */
+	uint64_t comp_feature_flags;
+	/**< Bitmask of flags for compression service features */
+	struct rte_param_log2_range window_size;
+	/**< Window size range in base two log byte values */
+};
+
+/** Macro used at end of comp PMD list */
+#define RTE_COMP_END_OF_CAPABILITIES_LIST() \
+	{ RTE_COMP_ALGO_UNSPECIFIED }
+
+const struct rte_compressdev_capabilities * __rte_experimental
+rte_compressdev_capability_get(uint8_t dev_id,
+			enum rte_comp_algorithm algo);
+
 /**
  * compression device supported feature flags
  *
@@ -57,6 +90,8 @@ rte_compressdev_get_feature_name(uint64_t flag);
 struct rte_compressdev_info {
 	const char *driver_name;		/**< Driver name. */
 	uint64_t feature_flags;			/**< Feature flags */
+	const struct rte_compressdev_capabilities *capabilities;
+	/**< Array of devices supported capabilities */
 	uint16_t max_nb_queue_pairs;
 	/**< Maximum number of queues pairs supported by device.
 	 * (If 0, there is no limit in maximum number of queue pairs)
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 8e4004631..6c553341a 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -3,6 +3,7 @@ EXPERIMENTAL {
 
 	rte_compressdev_callback_register;
 	rte_compressdev_callback_unregister;
+	rte_compressdev_capability_get;
 	rte_compressdev_close;
 	rte_compressdev_configure;
 	rte_compressdev_count;
-- 
2.14.3

[dpdk-dev] [PATCH v5 13/13] compressdev: get device id from name

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added API to retrieve the device id provided the device name.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 18 ++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 13 +++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  1 +
 3 files changed, 32 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index a5a1deaff..e6f09fe06 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -117,6 +117,24 @@ rte_compressdev_is_valid_dev(uint8_t dev_id)
 }
 
 
+int __rte_experimental
+rte_compressdev_get_dev_id(const char *name)
+{
+	unsigned int i;
+
+	if (name == NULL)
+		return -1;
+
+	for (i = 0; i < rte_compressdev_globals->nb_devs; i++)
+		if ((strcmp(rte_compressdev_globals->devs[i].data->name, name)
+				== 0) &&
+				(rte_compressdev_globals->devs[i].attached ==
+						RTE_COMPRESSDEV_ATTACHED))
+			return i;
+
+	return -1;
+}
+
 uint8_t __rte_experimental
 rte_compressdev_count(void)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 9e2fe95ca..1164f4886 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -111,6 +111,19 @@ struct rte_compressdev_stats {
 	/**< Total error count on operations dequeued */
 };
 
+
+/**
+ * Get the device identifier for the named compress device.
+ *
+ * @param name
+ *   Device name to select the device structure
+ * @return
+ *   - Returns compress device identifier on success.
+ *   - Return -1 on failure to find named compress device.
+ */
+int __rte_experimental
+rte_compressdev_get_dev_id(const char *name);
+
 /**
  * Get the compress device name given a device identifier.
  *
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 6c553341a..e5cbc569e 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -10,6 +10,7 @@ EXPERIMENTAL {
 	rte_compressdev_dequeue_burst;
 	rte_compressdev_devices_get;
 	rte_compressdev_enqueue_burst;
+	rte_compressdev_get_dev_id;
 	rte_compressdev_get_feature_name;
 	rte_compressdev_info_get;
 	rte_compressdev_name_get;
-- 
2.14.3

[dpdk-dev] [PATCH v6 00/14] Implement compression API

[Pablo de Lara]

With the vast amounts of data being transported around networks
and stored in storage systems, reducing data size is becoming
ever more important.

There are both software libraries and hardware devices available
that provide compression, but no common API.
Such an API is proposed in this patchset, which supports
the following features:

- Deflate Algorithm (https://tools.ietf.org/html/rfc1951)
- LZS algorithm (https://tools.ietf.org/html/rfc2395)
- Static and Dynamic Huffman encoding.
- Compression levels
- Checksum generation
- Asynchronous burst API
- private_xform - a place for PMDs to hold private data derived from
  a xform and used by stateless operations.
- stream - a place for PMDs to hold private data derived from
  a xform and also maintain state and history data. For
  stateful flows.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>

Changes in v6:
 - Add programmer's guide section for library
 - Change macro for no hash
 - Fix compilation in patch 4
 - Remove next pointer in xform
 - Add queue pair release in device closure

Changes in v5:
 - Make rte_compressdev_is_valid_dev, rte_compressdev_pmd_get_dev
   functions internal
 - Add rte_comp.h to API doc
 - Remove unused rte_comp_op_get_user_data,
   rte_comp_op_get_user_data_size functions
 - Fix comment
 - Move private data size arg out of PMD init params structure
 - Split out of space error status into two
 - Make rte_comp_op_alloc, rte_comp_op_bulk_alloc, rte_comp_free,
   rte_compressdev_enqueue_burst and rte_compressdev_dequeue_burst
   functions not inline.
 - Move rte_compressdev and rte_compressdev_data structures out of
   rte_compressdev.h, making them internal to the API
 - Remove unneeded driver registration (used in cryptodev, for multi PMD
   sessions)
 - Remove unneeded includes
 - rte_comp_op_bulk_alloc returns now 0 on success, instead of the
   number of allocated operations
 - Move internals macros to rte_compressdev_internals.h, to avoid
   exposing them unnecessarily

Changes in v4:
 - Fix build (missing ";")
 - Change order in common_base, so compression is placed after security

Changes in v3:
 - Remove rte_comp_op_ctod helper functions
 - Remove param_range_check macro
 - Rename from phys_addr to iova_addr
 - Remove rte_comp_algo_strings
 - Rename rte_compressdev_pmd_is_valid_dev to rte_compressdev_is_valid_dev
 - Remove feature flags from compressdev
 - Support hash operations
 - Add shareable priv xform in feature flags, instead of returnin it
   on xform creation
 - Allow max number of queue pairs to be 0, meaning that there is no
   limit.
 - Add invalid configuration checks
 - Add capability helper functions

Changes in v2:
 - Add stream APIs
 - Remove session
 - Add SHAREABLE / NON_SHAREABLE private_xform types
 - Add algo enum 'UNSPECIFIED' to fix warning in capabilities
 - Change one remaining log to use dynamic logging.
 - Add rte_cache_aligned keyword to op
 - Rename enums with better names _ALGO, __CHECKSUM, _HUFFMAN_
 - Use const keyword when passing xform
 - Remove qp_count fn from dev_ops as never used
 - Remove max_nb_queue-pairs from compressdev_init_param as never used
 - Clarify device configure and start sequence
 - Replace OUT_OF_SPACE with OUT_OF_SPACE_RECOVERABLE and TERMINATED
   and clarified usage. 
 - Add stream and private_xform sizes to device config for use in
   mempool creation
 - Add capability helper fn
 - Use Base2 log value for window size on xforms
 - Add Meson build
 - Update MAINTAINERS
 - Update Map file
 - Change order in doxy file
 - Update Release note 

Ashish Gupta (1):
  doc: add compressdev library guide

Fiona Trahe (12):
  compressdev: add basic device management
  compressdev: add queue pair management
  compressdev: add compression specific data
  compressdev: add enqueue/dequeue functions
  compressdev: add operation management
  compressdev: support stateless operations
  compressdev: support stateful operations
  compressdev: add device feature flags
  compressdev: add compression service feature flags
  compressdev: add device stats
  compressdev: add device capabilities
  compressdev: get device id from name

Shally Verma (1):
  compressdev: support hash operations

 MAINTAINERS                                        |   8 +
 config/common_base                                 |   6 +
 config/rte_config.h                                |   3 +
 doc/api/doxy-api-index.md                          |   2 +
 doc/api/doxy-api.conf                              |   1 +
 doc/guides/prog_guide/compressdev_lib.rst          | 623 +++++++++++++++++
 doc/guides/prog_guide/img/stateful-op.svg          | 116 +++
 doc/guides/prog_guide/img/stateless-op-shared.svg  | 124 ++++
 doc/guides/prog_guide/img/stateless-op.svg         | 140 ++++
 doc/guides/prog_guide/index.rst                    |   1 +
 doc/guides/rel_notes/release_18_05.rst             |   6 +
 lib/Makefile                                       |   3 +
 lib/librte_compressdev/Makefile                    |  31 +
 lib/librte_compressdev/meson.build                 |  12 +
 lib/librte_compressdev/rte_comp.c                  | 207 ++++++
 lib/librte_compressdev/rte_comp.h                  | 458 ++++++++++++
 lib/librte_compressdev/rte_compressdev.c           | 775 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 539 ++++++++++++++
 lib/librte_compressdev/rte_compressdev_internal.h  | 117 ++++
 lib/librte_compressdev/rte_compressdev_pmd.c       | 160 +++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 390 +++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  42 ++
 lib/meson.build                                    |   2 +-
 mk/rte.app.mk                                      |   1 +
 24 files changed, 3766 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/prog_guide/compressdev_lib.rst
 create mode 100644 doc/guides/prog_guide/img/stateful-op.svg
 create mode 100644 doc/guides/prog_guide/img/stateless-op-shared.svg
 create mode 100644 doc/guides/prog_guide/img/stateless-op.svg
 create mode 100644 lib/librte_compressdev/Makefile
 create mode 100644 lib/librte_compressdev/meson.build
 create mode 100644 lib/librte_compressdev/rte_comp.c
 create mode 100644 lib/librte_compressdev/rte_comp.h
 create mode 100644 lib/librte_compressdev/rte_compressdev.c
 create mode 100644 lib/librte_compressdev/rte_compressdev.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_internal.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.c
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_version.map

-- 
2.14.3

[dpdk-dev] [PATCH v6 01/14] compressdev: add basic device management

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Add basic functions to manage compress devices,
including driver and device allocation, and the basic
interface with compressdev PMDs.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 MAINTAINERS                                        |   7 +
 config/common_base                                 |   6 +
 config/rte_config.h                                |   3 +
 doc/api/doxy-api-index.md                          |   1 +
 doc/api/doxy-api.conf                              |   1 +
 doc/guides/rel_notes/release_18_05.rst             |   6 +
 lib/Makefile                                       |   3 +
 lib/librte_compressdev/Makefile                    |  30 ++
 lib/librte_compressdev/meson.build                 |  10 +
 lib/librte_compressdev/rte_compressdev.c           | 378 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 169 +++++++++
 lib/librte_compressdev/rte_compressdev_internal.h  |  64 ++++
 lib/librte_compressdev/rte_compressdev_pmd.c       | 160 +++++++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 236 +++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  24 ++
 lib/meson.build                                    |   2 +-
 mk/rte.app.mk                                      |   1 +
 17 files changed, 1100 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_compressdev/Makefile
 create mode 100644 lib/librte_compressdev/meson.build
 create mode 100644 lib/librte_compressdev/rte_compressdev.c
 create mode 100644 lib/librte_compressdev/rte_compressdev.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_internal.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.c
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index b59fa0531..c1370258f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -363,6 +363,13 @@ F: drivers/raw/skeleton_rawdev/
 F: test/test/test_rawdev.c
 F: doc/guides/prog_guide/rawdev.rst
 
+Compression API - EXPERIMENTAL
+M: Fiona Trahe <fiona.trahe@intel.com>
+M: Pablo de Lara <pablo.de.lara.guarch@intel.com>
+M: Ashish Gupta <ashish.gupta@caviumnetworks.com>
+T: git://dpdk.org/next/dpdk-next-crypto
+F: lib/librte_compressdev/
+
 
 Memory Pool Drivers
 -------------------
diff --git a/config/common_base b/config/common_base
index f24417cb0..23cf00011 100644
--- a/config/common_base
+++ b/config/common_base
@@ -568,6 +568,12 @@ CONFIG_RTE_LIBRTE_PMD_MRVL_CRYPTO_DEBUG=n
 #
 CONFIG_RTE_LIBRTE_SECURITY=y
 
+#
+# Compile generic compression device library
+#
+CONFIG_RTE_LIBRTE_COMPRESSDEV=y
+CONFIG_RTE_COMPRESS_MAX_DEVS=64
+
 #
 # Compile generic event device library
 #
diff --git a/config/rte_config.h b/config/rte_config.h
index b1c0b3920..5db654534 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -56,6 +56,9 @@
 #define RTE_CRYPTO_MAX_DEVS 64
 #define RTE_CRYPTODEV_NAME_LEN 64
 
+/* compressdev defines */
+#define RTE_COMPRESS_MAX_DEVS 64
+
 /* eventdev defines */
 #define RTE_EVENT_MAX_DEVS 16
 #define RTE_EVENT_MAX_QUEUES_PER_DEV 64
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 26ce7b44b..a3dfc0899 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -18,6 +18,7 @@ The public API headers are grouped by topics:
   [bbdev]              (@ref rte_bbdev.h),
   [cryptodev]          (@ref rte_cryptodev.h),
   [security]           (@ref rte_security.h),
+  [compressdev]        (@ref rte_compressdev.h),
   [eventdev]           (@ref rte_eventdev.h),
   [event_eth_rx_adapter]   (@ref rte_event_eth_rx_adapter.h),
   [event_timer_adapter]    (@ref rte_event_timer_adapter.h),
diff --git a/doc/api/doxy-api.conf b/doc/api/doxy-api.conf
index ad8bdcf61..3a159eaed 100644
--- a/doc/api/doxy-api.conf
+++ b/doc/api/doxy-api.conf
@@ -45,6 +45,7 @@ INPUT                   = doc/api/doxy-api-index.md \
                           lib/librte_cfgfile \
                           lib/librte_cmdline \
                           lib/librte_compat \
+                          lib/librte_compressdev \
                           lib/librte_cryptodev \
                           lib/librte_distributor \
                           lib/librte_efd \
diff --git a/doc/guides/rel_notes/release_18_05.rst b/doc/guides/rel_notes/release_18_05.rst
index f93141b57..6290c759e 100644
--- a/doc/guides/rel_notes/release_18_05.rst
+++ b/doc/guides/rel_notes/release_18_05.rst
@@ -149,6 +149,11 @@ New Features
   stats/xstats on shared memory from secondary process, and also pdump packets on
   those virtual devices.
 
+* **Added Compressdev Library, a generic compression service library.**
+
+  The compressdev library provides an API for offload of compression and
+  decompression operations to hardware or software accelerator devices.
+
 
 API Changes
 -----------
@@ -353,6 +358,7 @@ The libraries prepended with a plus sign were incremented in this version.
      librte_cfgfile.so.2
      librte_cmdline.so.2
    + librte_common_octeontx.so.1
+   + librte_compressdev.so.1
      librte_cryptodev.so.4
      librte_distributor.so.1
    + librte_eal.so.7
diff --git a/lib/Makefile b/lib/Makefile
index 965be6c8d..96afa599e 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -26,6 +26,9 @@ DEPDIRS-librte_bbdev := librte_eal librte_mempool librte_mbuf
 DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += librte_cryptodev
 DEPDIRS-librte_cryptodev := librte_eal librte_mempool librte_ring librte_mbuf
 DEPDIRS-librte_cryptodev += librte_kvargs
+DIRS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += librte_compressdev
+DEPDIRS-librte_compressdev := librte_eal librte_mempool librte_ring librte_mbuf
+DEPDIRS-librte_compressdev += librte_kvargs
 DIRS-$(CONFIG_RTE_LIBRTE_SECURITY) += librte_security
 DEPDIRS-librte_security := librte_eal librte_mempool librte_ring librte_mbuf
 DEPDIRS-librte_security += librte_ether
diff --git a/lib/librte_compressdev/Makefile b/lib/librte_compressdev/Makefile
new file mode 100644
index 000000000..b84769417
--- /dev/null
+++ b/lib/librte_compressdev/Makefile
@@ -0,0 +1,30 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2017-2018 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_compressdev.a
+
+# library version
+LIBABIVER := 1
+
+# build flags
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+LDLIBS += -lrte_eal -lrte_mempool -lrte_kvargs
+
+# library source files
+SRCS-y += rte_compressdev.c rte_compressdev_pmd.c
+
+# export include files
+SYMLINK-y-include += rte_compressdev.h
+# export include files (for PMDs)
+SYMLINK-y-include += rte_compressdev_pmd.h
+SYMLINK-y-include += rte_compressdev_internal.h
+
+# versioning export map
+EXPORT_MAP := rte_compressdev_version.map
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_compressdev/meson.build b/lib/librte_compressdev/meson.build
new file mode 100644
index 000000000..895e959d3
--- /dev/null
+++ b/lib/librte_compressdev/meson.build
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+allow_experimental_apis = true
+sources = files('rte_compressdev.c',
+	'rte_compressdev_pmd.c')
+headers = files('rte_compressdev.h',
+	'rte_compressdev_pmd.h',
+	'rte_compressdev_internal.h')
+deps += ['kvargs', 'mbuf']
diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
new file mode 100644
index 000000000..751517c3d
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -0,0 +1,378 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#include <string.h>
+#include <stdarg.h>
+#include <stdio.h>
+#include <inttypes.h>
+
+#include <rte_malloc.h>
+#include <rte_eal.h>
+#include <rte_memzone.h>
+
+#include "rte_compressdev.h"
+#include "rte_compressdev_internal.h"
+#include "rte_compressdev_pmd.h"
+
+#define RTE_COMPRESSDEV_DETACHED  (0)
+#define RTE_COMPRESSDEV_ATTACHED  (1)
+
+struct rte_compressdev rte_comp_devices[RTE_COMPRESS_MAX_DEVS];
+
+struct rte_compressdev *rte_compressdevs = &rte_comp_devices[0];
+
+static struct rte_compressdev_global compressdev_globals = {
+		.devs			= &rte_comp_devices[0],
+		.data			= { NULL },
+		.nb_devs		= 0,
+		.max_devs		= RTE_COMPRESS_MAX_DEVS
+};
+
+struct rte_compressdev_global *rte_compressdev_globals = &compressdev_globals;
+
+static struct rte_compressdev *
+rte_compressdev_get_dev(uint8_t dev_id)
+{
+	return &rte_compressdev_globals->devs[dev_id];
+}
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_get_named_dev(const char *name)
+{
+	struct rte_compressdev *dev;
+	unsigned int i;
+
+	if (name == NULL)
+		return NULL;
+
+	for (i = 0; i < rte_compressdev_globals->max_devs; i++) {
+		dev = &rte_compressdev_globals->devs[i];
+
+		if ((dev->attached == RTE_COMPRESSDEV_ATTACHED) &&
+				(strcmp(dev->data->name, name) == 0))
+			return dev;
+	}
+
+	return NULL;
+}
+
+static unsigned int
+rte_compressdev_is_valid_dev(uint8_t dev_id)
+{
+	struct rte_compressdev *dev = NULL;
+
+	if (dev_id >= rte_compressdev_globals->nb_devs)
+		return 0;
+
+	dev = rte_compressdev_get_dev(dev_id);
+	if (dev->attached != RTE_COMPRESSDEV_ATTACHED)
+		return 0;
+	else
+		return 1;
+}
+
+
+uint8_t __rte_experimental
+rte_compressdev_count(void)
+{
+	return rte_compressdev_globals->nb_devs;
+}
+
+uint8_t __rte_experimental
+rte_compressdev_devices_get(const char *driver_name, uint8_t *devices,
+	uint8_t nb_devices)
+{
+	uint8_t i, count = 0;
+	struct rte_compressdev *devs = rte_compressdev_globals->devs;
+	uint8_t max_devs = rte_compressdev_globals->max_devs;
+
+	for (i = 0; i < max_devs && count < nb_devices;	i++) {
+
+		if (devs[i].attached == RTE_COMPRESSDEV_ATTACHED) {
+			int cmp;
+
+			cmp = strncmp(devs[i].device->driver->name,
+					driver_name,
+					strlen(driver_name));
+
+			if (cmp == 0)
+				devices[count++] = devs[i].data->dev_id;
+		}
+	}
+
+	return count;
+}
+
+int __rte_experimental
+rte_compressdev_socket_id(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id))
+		return -1;
+
+	dev = rte_compressdev_get_dev(dev_id);
+
+	return dev->data->socket_id;
+}
+
+static inline int
+rte_compressdev_data_alloc(uint8_t dev_id, struct rte_compressdev_data **data,
+		int socket_id)
+{
+	char mz_name[RTE_COMPRESSDEV_NAME_MAX_LEN];
+	const struct rte_memzone *mz;
+	int n;
+
+	/* generate memzone name */
+	n = snprintf(mz_name, sizeof(mz_name),
+			"rte_compressdev_data_%u", dev_id);
+	if (n >= (int)sizeof(mz_name))
+		return -EINVAL;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		mz = rte_memzone_reserve(mz_name,
+				sizeof(struct rte_compressdev_data),
+				socket_id, 0);
+	} else
+		mz = rte_memzone_lookup(mz_name);
+
+	if (mz == NULL)
+		return -ENOMEM;
+
+	*data = mz->addr;
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		memset(*data, 0, sizeof(struct rte_compressdev_data));
+
+	return 0;
+}
+
+static uint8_t
+rte_compressdev_find_free_device_index(void)
+{
+	uint8_t dev_id;
+
+	for (dev_id = 0; dev_id < RTE_COMPRESS_MAX_DEVS; dev_id++) {
+		if (rte_comp_devices[dev_id].attached ==
+				RTE_COMPRESSDEV_DETACHED)
+			return dev_id;
+	}
+	return RTE_COMPRESS_MAX_DEVS;
+}
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_allocate(const char *name, int socket_id)
+{
+	struct rte_compressdev *compressdev;
+	uint8_t dev_id;
+
+	if (rte_compressdev_pmd_get_named_dev(name) != NULL) {
+		COMPRESSDEV_LOG(ERR,
+			"comp device with name %s already allocated!", name);
+		return NULL;
+	}
+
+	dev_id = rte_compressdev_find_free_device_index();
+	if (dev_id == RTE_COMPRESS_MAX_DEVS) {
+		COMPRESSDEV_LOG(ERR, "Reached maximum number of comp devices");
+		return NULL;
+	}
+	compressdev = rte_compressdev_get_dev(dev_id);
+
+	if (compressdev->data == NULL) {
+		struct rte_compressdev_data *compressdev_data =
+				compressdev_globals.data[dev_id];
+
+		int retval = rte_compressdev_data_alloc(dev_id,
+				&compressdev_data, socket_id);
+
+		if (retval < 0 || compressdev_data == NULL)
+			return NULL;
+
+		compressdev->data = compressdev_data;
+
+		snprintf(compressdev->data->name, RTE_COMPRESSDEV_NAME_MAX_LEN,
+				"%s", name);
+
+		compressdev->data->dev_id = dev_id;
+		compressdev->data->socket_id = socket_id;
+		compressdev->data->dev_started = 0;
+
+		compressdev->attached = RTE_COMPRESSDEV_ATTACHED;
+
+		compressdev_globals.nb_devs++;
+	}
+
+	return compressdev;
+}
+
+int __rte_experimental
+rte_compressdev_pmd_release_device(struct rte_compressdev *compressdev)
+{
+	int ret;
+
+	if (compressdev == NULL)
+		return -EINVAL;
+
+	/* Close device only if device operations have been set */
+	if (compressdev->dev_ops) {
+		ret = rte_compressdev_close(compressdev->data->dev_id);
+		if (ret < 0)
+			return ret;
+	}
+
+	compressdev->attached = RTE_COMPRESSDEV_DETACHED;
+	compressdev_globals.nb_devs--;
+	return 0;
+}
+
+int __rte_experimental
+rte_compressdev_configure(uint8_t dev_id, struct rte_compressdev_config *config)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	if (dev->data->dev_started) {
+		COMPRESSDEV_LOG(ERR,
+		    "device %d must be stopped to allow configuration", dev_id);
+		return -EBUSY;
+	}
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_configure, -ENOTSUP);
+
+	return (*dev->dev_ops->dev_configure)(dev, config);
+}
+
+
+int __rte_experimental
+rte_compressdev_start(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+	int diag;
+
+	COMPRESSDEV_LOG(DEBUG, "Start dev_id=%" PRIu8, dev_id);
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_start, -ENOTSUP);
+
+	if (dev->data->dev_started != 0) {
+		COMPRESSDEV_LOG(ERR,
+		    "Device with dev_id=%" PRIu8 " already started", dev_id);
+		return 0;
+	}
+
+	diag = (*dev->dev_ops->dev_start)(dev);
+	if (diag == 0)
+		dev->data->dev_started = 1;
+	else
+		return diag;
+
+	return 0;
+}
+
+void __rte_experimental
+rte_compressdev_stop(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_stop);
+
+	if (dev->data->dev_started == 0) {
+		COMPRESSDEV_LOG(ERR,
+		    "Device with dev_id=%" PRIu8 " already stopped", dev_id);
+		return;
+	}
+
+	(*dev->dev_ops->dev_stop)(dev);
+	dev->data->dev_started = 0;
+}
+
+int __rte_experimental
+rte_compressdev_close(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+	int retval;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -1;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	/* Device must be stopped before it can be closed */
+	if (dev->data->dev_started == 1) {
+		COMPRESSDEV_LOG(ERR, "Device %u must be stopped before closing",
+				dev_id);
+		return -EBUSY;
+	}
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_close, -ENOTSUP);
+	retval = (*dev->dev_ops->dev_close)(dev);
+
+	if (retval < 0)
+		return retval;
+
+	return 0;
+}
+
+void __rte_experimental
+rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
+{
+	struct rte_compressdev *dev;
+
+	if (dev_id >= compressdev_globals.nb_devs) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%d", dev_id);
+		return;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	memset(dev_info, 0, sizeof(struct rte_compressdev_info));
+
+	RTE_FUNC_PTR_OR_RET(*dev->dev_ops->dev_infos_get);
+	(*dev->dev_ops->dev_infos_get)(dev, dev_info);
+
+	dev_info->driver_name = dev->device->driver->name;
+}
+
+const char * __rte_experimental
+rte_compressdev_name_get(uint8_t dev_id)
+{
+	struct rte_compressdev *dev = rte_compressdev_get_dev(dev_id);
+
+	if (dev == NULL)
+		return NULL;
+
+	return dev->data->name;
+}
+
+RTE_INIT(rte_compressdev_log);
+
+static void
+rte_compressdev_log(void)
+{
+	compressdev_logtype = rte_log_register("lib.compressdev");
+	if (compressdev_logtype >= 0)
+		rte_log_set_level(compressdev_logtype, RTE_LOG_NOTICE);
+}
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
new file mode 100644
index 000000000..5be5973bf
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -0,0 +1,169 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMPRESSDEV_H_
+#define _RTE_COMPRESSDEV_H_
+
+/**
+ * @file rte_compressdev.h
+ *
+ * RTE Compression Device APIs
+ *
+ * Defines comp device APIs for the provisioning of compression operations.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+
+/**  comp device information */
+struct rte_compressdev_info {
+	const char *driver_name;		/**< Driver name. */
+};
+
+/**
+ * Get the compress device name given a device identifier.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - Returns compress device name.
+ *   - Returns NULL if compress device is not present.
+ */
+const char * __rte_experimental
+rte_compressdev_name_get(uint8_t dev_id);
+
+/**
+ * Get the total number of compress devices that have been successfully
+ * initialised.
+ *
+ * @return
+ *   - The total number of usable compress devices.
+ */
+uint8_t __rte_experimental
+rte_compressdev_count(void);
+
+/**
+ * Get number and identifiers of attached comp devices that
+ * use the same compress driver.
+ *
+ * @param driver_name
+ *   Driver name
+ * @param devices
+ *   Output devices identifiers
+ * @param nb_devices
+ *   Maximal number of devices
+ *
+ * @return
+ *   Returns number of attached compress devices.
+ */
+uint8_t __rte_experimental
+rte_compressdev_devices_get(const char *driver_name, uint8_t *devices,
+		uint8_t nb_devices);
+
+/*
+ * Return the NUMA socket to which a device is connected.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   The NUMA socket id to which the device is connected or
+ *   a default of zero if the socket could not be determined.
+ *   -1 if returned is the dev_id value is out of range.
+ */
+int __rte_experimental
+rte_compressdev_socket_id(uint8_t dev_id);
+
+/** Compress device configuration structure */
+struct rte_compressdev_config {
+	int socket_id;
+};
+
+/**
+ * Configure a device.
+ *
+ * This function must be invoked first before any other function in the
+ * API. This function can also be re-invoked when a device is in the
+ * stopped state.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param config
+ *   The compress device configuration
+ * @return
+ *   - 0: Success, device configured.
+ *   - <0: Error code returned by the driver configuration function.
+ */
+int __rte_experimental
+rte_compressdev_configure(uint8_t dev_id,
+			struct rte_compressdev_config *config);
+
+/**
+ * Start a device.
+ *
+ * The device start step is called after configuring the device and setting up
+ * its queue pairs.
+ * On success, data-path functions exported by the API (enqueue/dequeue, etc)
+ * can be invoked.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - 0: Success, device started.
+ *   - <0: Error code of the driver device start function.
+ */
+int __rte_experimental
+rte_compressdev_start(uint8_t dev_id);
+
+/**
+ * Stop a device. The device can be restarted with a call to
+ * rte_compressdev_start()
+ *
+ * @param dev_id
+ *   Compress device identifier
+ */
+void __rte_experimental
+rte_compressdev_stop(uint8_t dev_id);
+
+/**
+ * Close an device.
+ * The memory allocated in the device gets freed.
+ * After calling this function, in order to use
+ * the device again, it is required to
+ * configure the device again.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ *
+ * @return
+ *  - 0 on successfully closing device
+ *  - <0 on failure to close device
+ */
+int __rte_experimental
+rte_compressdev_close(uint8_t dev_id);
+
+/**
+ * Retrieve the contextual information of a device.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param dev_info
+ *   A pointer to a structure of type *rte_compressdev_info*
+ *   to be filled with the contextual information of the device
+ *
+ * @note The capabilities field of dev_info is set to point to the first
+ * element of an array of struct rte_compressdev_capabilities.
+ * The element after the last valid element has it's op field set to
+ * RTE_COMP_ALGO_LIST_END.
+ */
+void __rte_experimental
+rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_COMPRESSDEV_H_ */
diff --git a/lib/librte_compressdev/rte_compressdev_internal.h b/lib/librte_compressdev/rte_compressdev_internal.h
new file mode 100644
index 000000000..0a2ddcb2b
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_internal.h
@@ -0,0 +1,64 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMPRESSDEV_INTERNAL_H_
+#define _RTE_COMPRESSDEV_INTERNAL_H_
+
+/* rte_compressdev_internal.h
+ * This file holds Compressdev private data structures.
+ */
+#include <rte_log.h>
+
+#define RTE_COMPRESSDEV_NAME_MAX_LEN	(64)
+/**< Max length of name of comp PMD */
+
+/* Logging Macros */
+extern int compressdev_logtype;
+#define COMPRESSDEV_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, compressdev_logtype, "%s(): "fmt "\n", \
+			__func__, ##args)
+
+
+/** The data structure associated with each comp device. */
+struct rte_compressdev {
+	struct rte_compressdev_data *data;
+	/**< Pointer to device data */
+	struct rte_compressdev_ops *dev_ops;
+	/**< Functions exported by PMD */
+	uint64_t feature_flags;
+	/**< Supported features */
+	struct rte_device *device;
+	/**< Backing device */
+
+	uint8_t driver_id;
+	/**< comp driver identifier*/
+
+	__extension__
+	uint8_t attached : 1;
+	/**< Flag indicating the device is attached */
+} __rte_cache_aligned;
+
+/**
+ *
+ * The data part, with no function pointers, associated with each device.
+ *
+ * This structure is safe to place in shared memory to be common among
+ * different processes in a multi-process configuration.
+ */
+struct rte_compressdev_data {
+	uint8_t dev_id;
+	/**< Compress device identifier */
+	uint8_t socket_id;
+	/**< Socket identifier where memory is allocated */
+	char name[RTE_COMPRESSDEV_NAME_MAX_LEN];
+	/**< Unique identifier name */
+
+	__extension__
+	uint8_t dev_started : 1;
+	/**< Device state: STARTED(1)/STOPPED(0) */
+
+	void *dev_private;
+	/**< PMD-specific private data */
+} __rte_cache_aligned;
+#endif
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.c b/lib/librte_compressdev/rte_compressdev_pmd.c
new file mode 100644
index 000000000..7de4f339e
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_pmd.c
@@ -0,0 +1,160 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#include <rte_malloc.h>
+#include <rte_kvargs.h>
+#include <rte_eal.h>
+
+#include "rte_compressdev_internal.h"
+#include "rte_compressdev_pmd.h"
+
+int compressdev_logtype;
+
+/**
+ * Parse name from argument
+ */
+static int
+rte_compressdev_pmd_parse_name_arg(const char *key __rte_unused,
+		const char *value, void *extra_args)
+{
+	struct rte_compressdev_pmd_init_params *params = extra_args;
+	int n;
+
+	n = snprintf(params->name, RTE_COMPRESSDEV_NAME_MAX_LEN, "%s", value);
+	if (n >= RTE_COMPRESSDEV_NAME_MAX_LEN)
+		return -EINVAL;
+
+	return 0;
+}
+
+/**
+ * Parse unsigned integer from argument
+ */
+static int
+rte_compressdev_pmd_parse_uint_arg(const char *key __rte_unused,
+		const char *value, void *extra_args)
+{
+	int i;
+	char *end;
+
+	errno = 0;
+	i = strtol(value, &end, 10);
+	if (*end != 0 || errno != 0 || i < 0)
+		return -EINVAL;
+
+	*((uint32_t *)extra_args) = i;
+	return 0;
+}
+
+int __rte_experimental
+rte_compressdev_pmd_parse_input_args(
+		struct rte_compressdev_pmd_init_params *params,
+		const char *args)
+{
+	struct rte_kvargs *kvlist = NULL;
+	int ret = 0;
+
+	if (params == NULL)
+		return -EINVAL;
+
+	if (args) {
+		kvlist = rte_kvargs_parse(args,	compressdev_pmd_valid_params);
+		if (kvlist == NULL)
+			return -EINVAL;
+
+		ret = rte_kvargs_process(kvlist,
+				RTE_COMPRESSDEV_PMD_SOCKET_ID_ARG,
+				&rte_compressdev_pmd_parse_uint_arg,
+				&params->socket_id);
+		if (ret < 0)
+			goto free_kvlist;
+
+		ret = rte_kvargs_process(kvlist,
+				RTE_COMPRESSDEV_PMD_NAME_ARG,
+				&rte_compressdev_pmd_parse_name_arg,
+				params);
+		if (ret < 0)
+			goto free_kvlist;
+	}
+
+free_kvlist:
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_create(const char *name,
+		struct rte_device *device,
+		size_t private_data_size,
+		struct rte_compressdev_pmd_init_params *params)
+{
+	struct rte_compressdev *compressdev;
+
+	if (params->name[0] != '\0') {
+		COMPRESSDEV_LOG(INFO, "[%s] User specified device name = %s\n",
+				device->driver->name, params->name);
+		name = params->name;
+	}
+
+	COMPRESSDEV_LOG(INFO, "[%s] - Creating compressdev %s\n",
+			device->driver->name, name);
+
+	COMPRESSDEV_LOG(INFO,
+	"[%s] - Init parameters - name: %s, socket id: %d",
+			device->driver->name, name,
+			params->socket_id);
+
+	/* allocate device structure */
+	compressdev = rte_compressdev_pmd_allocate(name, params->socket_id);
+	if (compressdev == NULL) {
+		COMPRESSDEV_LOG(ERR, "[%s] Failed to allocate comp device %s",
+				device->driver->name, name);
+		return NULL;
+	}
+
+	/* allocate private device structure */
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		compressdev->data->dev_private =
+				rte_zmalloc_socket("compressdev device private",
+						private_data_size,
+						RTE_CACHE_LINE_SIZE,
+						params->socket_id);
+
+		if (compressdev->data->dev_private == NULL) {
+			COMPRESSDEV_LOG(ERR,
+		"[%s] Cannot allocate memory for compressdev %s private data",
+					device->driver->name, name);
+
+			rte_compressdev_pmd_release_device(compressdev);
+			return NULL;
+		}
+	}
+
+	compressdev->device = device;
+
+	return compressdev;
+}
+
+int __rte_experimental
+rte_compressdev_pmd_destroy(struct rte_compressdev *compressdev)
+{
+	int retval;
+
+	COMPRESSDEV_LOG(INFO, "[%s] Closing comp device %s",
+			compressdev->device->driver->name,
+			compressdev->device->name);
+
+	/* free comp device */
+	retval = rte_compressdev_pmd_release_device(compressdev);
+	if (retval)
+		return retval;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		rte_free(compressdev->data->dev_private);
+
+	compressdev->device = NULL;
+	compressdev->data = NULL;
+
+	return 0;
+}
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
new file mode 100644
index 000000000..43307ee8e
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -0,0 +1,236 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMPRESSDEV_PMD_H_
+#define _RTE_COMPRESSDEV_PMD_H_
+
+/** @file
+ * RTE comp PMD APIs
+ *
+ * @note
+ * These APIs are for comp PMDs only and user applications should not call
+ * them directly.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <string.h>
+
+#include <rte_dev.h>
+#include <rte_common.h>
+
+#include "rte_compressdev.h"
+#include "rte_compressdev_internal.h"
+
+#define RTE_COMPRESSDEV_PMD_NAME_ARG			("name")
+#define RTE_COMPRESSDEV_PMD_SOCKET_ID_ARG		("socket_id")
+
+static const char * const compressdev_pmd_valid_params[] = {
+	RTE_COMPRESSDEV_PMD_NAME_ARG,
+	RTE_COMPRESSDEV_PMD_SOCKET_ID_ARG
+};
+
+/**
+ * @internal
+ * Initialisation parameters for comp devices
+ */
+struct rte_compressdev_pmd_init_params {
+	char name[RTE_COMPRESSDEV_NAME_MAX_LEN];
+	int socket_id;
+};
+
+/** Global structure used for maintaining state of allocated comp devices */
+struct rte_compressdev_global {
+	struct rte_compressdev *devs;	/**< Device information array */
+	struct rte_compressdev_data *data[RTE_COMPRESS_MAX_DEVS];
+	/**< Device private data */
+	uint8_t nb_devs;		/**< Number of devices found */
+	uint8_t max_devs;		/**< Max number of devices */
+};
+
+/** Pointer to global array of comp devices */
+extern struct rte_compressdev *rte_compressdevs;
+/** Pointer to global comp devices data structure */
+extern struct rte_compressdev_global *rte_compressdev_globals;
+
+/**
+ * Get the rte_compressdev structure device pointer for the named device.
+ *
+ * @param name
+ *   Compress device name
+ * @return
+ *   - The rte_compressdev structure pointer for the given device identifier.
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_get_named_dev(const char *name);
+
+/**
+ * Definitions of all functions exported by a driver through the
+ * the generic structure of type *comp_dev_ops* supplied in the
+ * *rte_compressdev* structure associated with a device.
+ */
+
+/**
+ * Function used to configure device.
+ *
+ * @param dev
+ *   Compress device
+ * @param config
+ *   Compress device configurations
+ * @return
+ *   Returns 0 on success
+ */
+typedef int (*compressdev_configure_t)(struct rte_compressdev *dev,
+		struct rte_compressdev_config *config);
+
+/**
+ * Function used to start a configured device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *   Returns 0 on success
+ */
+typedef int (*compressdev_start_t)(struct rte_compressdev *dev);
+
+/**
+ * Function used to stop a configured device.
+ *
+ * @param dev
+ *   Compress device
+ */
+typedef void (*compressdev_stop_t)(struct rte_compressdev *dev);
+
+/**
+ * Function used to close a configured device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ * - 0 on success.
+ * - EAGAIN if can't close as device is busy
+ */
+typedef int (*compressdev_close_t)(struct rte_compressdev *dev);
+
+
+/**
+ * Function used to get specific information of a device.
+ *
+ * @param dev
+ *   Compress device
+ */
+typedef void (*compressdev_info_get_t)(struct rte_compressdev *dev,
+				struct rte_compressdev_info *dev_info);
+
+/** comp device operations function pointer table */
+struct rte_compressdev_ops {
+	compressdev_configure_t dev_configure;	/**< Configure device. */
+	compressdev_start_t dev_start;		/**< Start device. */
+	compressdev_stop_t dev_stop;		/**< Stop device. */
+	compressdev_close_t dev_close;		/**< Close device. */
+
+	compressdev_info_get_t dev_infos_get;	/**< Get device info. */
+};
+
+/**
+ * @internal
+ *
+ * Function for internal use by dummy drivers primarily, e.g. ring-based
+ * driver.
+ * Allocates a new compressdev slot for an comp device and returns the pointer
+ * to that slot for the driver to use.
+ *
+ * @param name
+ *   Unique identifier name for each device
+ * @param socket_id
+ *   Socket to allocate resources on
+ * @return
+ *   - Slot in the rte_dev_devices array for a new device;
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_allocate(const char *name, int socket_id);
+
+/**
+ * @internal
+ *
+ * Function for internal use by dummy drivers primarily, e.g. ring-based
+ * driver.
+ * Release the specified compressdev device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *   - 0 on success, negative on error
+ */
+int __rte_experimental
+rte_compressdev_pmd_release_device(struct rte_compressdev *dev);
+
+
+/**
+ * @internal
+ *
+ * PMD assist function to parse initialisation arguments for comp driver
+ * when creating a new comp PMD device instance.
+ *
+ * PMD driver should set default values for that PMD before calling function,
+ * these default values will be over-written with successfully parsed values
+ * from args string.
+ *
+ * @param params
+ *   Parsed PMD initialisation parameters
+ * @param args
+ *   Input argument string to parse
+ * @return
+ *  - 0 on success
+ *  - errno on failure
+ */
+int __rte_experimental
+rte_compressdev_pmd_parse_input_args(
+		struct rte_compressdev_pmd_init_params *params,
+		const char *args);
+
+/**
+ * @internal
+ *
+ * PMD assist function to provide boiler plate code for comp driver to create
+ * and allocate resources for a new comp PMD device instance.
+ *
+ * @param name
+ *   Compress device name
+ * @param device
+ *   Base device instance
+ * @param params
+ *   PMD initialisation parameters
+ * @return
+ *  - comp device instance on success
+ *  - NULL on creation failure
+ */
+struct rte_compressdev * __rte_experimental
+rte_compressdev_pmd_create(const char *name,
+		struct rte_device *device,
+		size_t private_data_size,
+		struct rte_compressdev_pmd_init_params *params);
+
+/**
+ * @internal
+ *
+ * PMD assist function to provide boiler plate code for comp driver to
+ * destroy and free resources associated with a comp PMD device instance.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *  - 0 on success
+ *  - errno on failure
+ */
+int __rte_experimental
+rte_compressdev_pmd_destroy(struct rte_compressdev *dev);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_COMPRESSDEV_PMD_H_ */
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
new file mode 100644
index 000000000..a996abcd2
--- /dev/null
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -0,0 +1,24 @@
+EXPERIMENTAL {
+        global:
+
+	rte_compressdev_callback_register;
+	rte_compressdev_callback_unregister;
+	rte_compressdev_close;
+	rte_compressdev_configure;
+	rte_compressdev_count;
+	rte_compressdev_devices_get;
+	rte_compressdev_info_get;
+	rte_compressdev_name_get;
+	rte_compressdev_pmd_allocate;
+	rte_compressdev_pmd_callback_process;
+	rte_compressdev_pmd_create;
+	rte_compressdev_pmd_destroy;
+	rte_compressdev_pmd_get_named_dev;
+	rte_compressdev_pmd_parse_input_args;
+	rte_compressdev_pmd_release_device;
+	rte_compressdev_socket_id;
+	rte_compressdev_start;
+	rte_compressdev_stop;
+
+        local: *;
+};
diff --git a/lib/meson.build b/lib/meson.build
index 73d6f25c7..f52a0188c 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -15,7 +15,7 @@ libraries = [ 'compat', # just a header, used for versioning
 	'kvargs',  # cryptodev depends on this
 	'timer',   # eventdev depends on this
 	'acl', 'bbdev', 'bitratestats', 'cfgfile',
-	'cmdline', 'cryptodev',
+	'cmdline', 'compressdev', 'cryptodev',
 	'distributor', 'efd', 'eventdev',
 	'gro', 'gso', 'ip_frag', 'jobstats',
 	'kni', 'latencystats', 'lpm', 'member',
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 1324f19cc..97d1da195 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -93,6 +93,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF)           += -lrte_mbuf
 _LDLIBS-$(CONFIG_RTE_LIBRTE_NET)            += -lrte_net
 _LDLIBS-$(CONFIG_RTE_LIBRTE_ETHER)          += -lrte_ethdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_BBDEV)          += -lrte_bbdev
+_LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV)    += -lrte_compressdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_CRYPTODEV)      += -lrte_cryptodev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY)       += -lrte_security
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV)       += -lrte_eventdev
-- 
2.14.3

[dpdk-dev] [PATCH v6 02/14] compressdev: add queue pair management

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Add functions to manage device queue pairs.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 178 ++++++++++++++++++++-
 lib/librte_compressdev/rte_compressdev.h           |  45 ++++++
 lib/librte_compressdev/rte_compressdev_internal.h  |   5 +
 lib/librte_compressdev/rte_compressdev_pmd.h       |  47 ++++++
 lib/librte_compressdev/rte_compressdev_version.map |   2 +
 5 files changed, 276 insertions(+), 1 deletion(-)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 751517c3d..6667528b5 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -227,10 +227,136 @@ rte_compressdev_pmd_release_device(struct rte_compressdev *compressdev)
 	return 0;
 }
 
+uint16_t __rte_experimental
+rte_compressdev_queue_pair_count(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	dev = &rte_comp_devices[dev_id];
+	return dev->data->nb_queue_pairs;
+}
+
+static int
+rte_compressdev_queue_pairs_config(struct rte_compressdev *dev,
+		uint16_t nb_qpairs, int socket_id)
+{
+	struct rte_compressdev_info dev_info;
+	void **qp;
+	unsigned int i;
+
+	if ((dev == NULL) || (nb_qpairs < 1)) {
+		COMPRESSDEV_LOG(ERR, "invalid param: dev %p, nb_queues %u",
+							dev, nb_qpairs);
+		return -EINVAL;
+	}
+
+	COMPRESSDEV_LOG(DEBUG, "Setup %d queues pairs on device %u",
+			nb_qpairs, dev->data->dev_id);
+
+	memset(&dev_info, 0, sizeof(struct rte_compressdev_info));
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_infos_get, -ENOTSUP);
+	(*dev->dev_ops->dev_infos_get)(dev, &dev_info);
+
+	if ((dev_info.max_nb_queue_pairs != 0) &&
+			(nb_qpairs > dev_info.max_nb_queue_pairs)) {
+		COMPRESSDEV_LOG(ERR, "Invalid num queue_pairs (%u) for dev %u",
+				nb_qpairs, dev->data->dev_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->queue_pairs == NULL) { /* first time configuration */
+		dev->data->queue_pairs = rte_zmalloc_socket(
+				"compressdev->queue_pairs",
+				sizeof(dev->data->queue_pairs[0]) * nb_qpairs,
+				RTE_CACHE_LINE_SIZE, socket_id);
+
+		if (dev->data->queue_pairs == NULL) {
+			dev->data->nb_queue_pairs = 0;
+			COMPRESSDEV_LOG(ERR,
+			"failed to get memory for qp meta data, nb_queues %u",
+							nb_qpairs);
+			return -(ENOMEM);
+		}
+	} else { /* re-configure */
+		int ret;
+		uint16_t old_nb_queues = dev->data->nb_queue_pairs;
+
+		qp = dev->data->queue_pairs;
+
+		RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->queue_pair_release,
+				-ENOTSUP);
+
+		for (i = nb_qpairs; i < old_nb_queues; i++) {
+			ret = (*dev->dev_ops->queue_pair_release)(dev, i);
+			if (ret < 0)
+				return ret;
+		}
+
+		qp = rte_realloc(qp, sizeof(qp[0]) * nb_qpairs,
+				RTE_CACHE_LINE_SIZE);
+		if (qp == NULL) {
+			COMPRESSDEV_LOG(ERR,
+			"failed to realloc qp meta data, nb_queues %u",
+						nb_qpairs);
+			return -(ENOMEM);
+		}
+
+		if (nb_qpairs > old_nb_queues) {
+			uint16_t new_qs = nb_qpairs - old_nb_queues;
+
+			memset(qp + old_nb_queues, 0,
+				sizeof(qp[0]) * new_qs);
+		}
+
+		dev->data->queue_pairs = qp;
+
+	}
+	dev->data->nb_queue_pairs = nb_qpairs;
+	return 0;
+}
+
+static int
+rte_compressdev_queue_pairs_release(struct rte_compressdev *dev)
+{
+	uint16_t num_qps, i;
+	int ret;
+
+	if (dev == NULL) {
+		COMPRESSDEV_LOG(ERR, "invalid param: dev %p", dev);
+		return -EINVAL;
+	}
+
+	num_qps = dev->data->nb_queue_pairs;
+
+	if (num_qps == 0)
+		return 0;
+
+	COMPRESSDEV_LOG(DEBUG, "Free %d queues pairs on device %u",
+			dev->data->nb_queue_pairs, dev->data->dev_id);
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->queue_pair_release,
+				-ENOTSUP);
+
+	for (i = 0; i < num_qps; i++) {
+		ret = (*dev->dev_ops->queue_pair_release)(dev, i);
+		if (ret < 0)
+			return ret;
+	}
+
+	if (dev->data->queue_pairs != NULL)
+		rte_free(dev->data->queue_pairs);
+	dev->data->queue_pairs = NULL;
+	dev->data->nb_queue_pairs = 0;
+
+	return 0;
+}
+
 int __rte_experimental
 rte_compressdev_configure(uint8_t dev_id, struct rte_compressdev_config *config)
 {
 	struct rte_compressdev *dev;
+	int diag;
 
 	if (!rte_compressdev_is_valid_dev(dev_id)) {
 		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
@@ -247,10 +373,19 @@ rte_compressdev_configure(uint8_t dev_id, struct rte_compressdev_config *config)
 
 	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_configure, -ENOTSUP);
 
+	/* Setup new number of queue pairs and reconfigure device. */
+	diag = rte_compressdev_queue_pairs_config(dev, config->nb_queue_pairs,
+			config->socket_id);
+	if (diag != 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev%d rte_comp_dev_queue_pairs_config = %d",
+				dev_id, diag);
+		return diag;
+	}
+
 	return (*dev->dev_ops->dev_configure)(dev, config);
 }
 
-
 int __rte_experimental
 rte_compressdev_start(uint8_t dev_id)
 {
@@ -327,6 +462,12 @@ rte_compressdev_close(uint8_t dev_id)
 		return -EBUSY;
 	}
 
+	/* Free queue pairs memory */
+	retval = rte_compressdev_queue_pairs_release(dev);
+
+	if (retval < 0)
+		return retval;
+
 	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_close, -ENOTSUP);
 	retval = (*dev->dev_ops->dev_close)(dev);
 
@@ -336,6 +477,41 @@ rte_compressdev_close(uint8_t dev_id)
 	return 0;
 }
 
+int __rte_experimental
+rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
+		uint32_t max_inflight_ops, int socket_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+	if (queue_pair_id >= dev->data->nb_queue_pairs) {
+		COMPRESSDEV_LOG(ERR, "Invalid queue_pair_id=%d", queue_pair_id);
+		return -EINVAL;
+	}
+
+	if (dev->data->dev_started) {
+		COMPRESSDEV_LOG(ERR,
+		    "device %d must be stopped to allow configuration", dev_id);
+		return -EBUSY;
+	}
+
+	if (max_inflight_ops == 0) {
+		COMPRESSDEV_LOG(ERR,
+			"Invalid maximum number of inflight operations");
+		return -EINVAL;
+	}
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->queue_pair_setup, -ENOTSUP);
+
+	return (*dev->dev_ops->queue_pair_setup)(dev, queue_pair_id,
+			max_inflight_ops, socket_id);
+}
+
 void __rte_experimental
 rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 5be5973bf..81710bbea 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -22,6 +22,10 @@ extern "C" {
 /**  comp device information */
 struct rte_compressdev_info {
 	const char *driver_name;		/**< Driver name. */
+	uint16_t max_nb_queue_pairs;
+	/**< Maximum number of queues pairs supported by device.
+	 * (If 0, there is no limit in maximum number of queue pairs)
+	 */
 };
 
 /**
@@ -80,6 +84,9 @@ rte_compressdev_socket_id(uint8_t dev_id);
 /** Compress device configuration structure */
 struct rte_compressdev_config {
 	int socket_id;
+	/**< Socket on which to allocate resources */
+	uint16_t nb_queue_pairs;
+	/**< Total number of queue pairs to configure on a device */
 };
 
 /**
@@ -145,6 +152,44 @@ rte_compressdev_stop(uint8_t dev_id);
 int __rte_experimental
 rte_compressdev_close(uint8_t dev_id);
 
+/**
+ * Allocate and set up a receive queue pair for a device.
+ * This should only be called when the device is stopped.
+ *
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param queue_pair_id
+ *   The index of the queue pairs to set up. The
+ *   value must be in the range [0, nb_queue_pair - 1]
+ *   previously supplied to rte_compressdev_configure()
+ * @param max_inflight_ops
+ *   Max number of ops which the qp will have to
+ *   accommodate simultaneously
+ * @param socket_id
+ *   The *socket_id* argument is the socket identifier
+ *   in case of NUMA. The value can be *SOCKET_ID_ANY*
+ *   if there is no NUMA constraint for the DMA memory
+ *   allocated for the receive queue pair
+ * @return
+ *   - 0: Success, queue pair correctly set up.
+ *   - <0: Queue pair configuration failed
+ */
+int __rte_experimental
+rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
+		uint32_t max_inflight_ops, int socket_id);
+
+/**
+ * Get the number of queue pairs on a specific comp device
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @return
+ *   - The number of configured queue pairs.
+ */
+uint16_t __rte_experimental
+rte_compressdev_queue_pair_count(uint8_t dev_id);
+
 /**
  * Retrieve the contextual information of a device.
  *
diff --git a/lib/librte_compressdev/rte_compressdev_internal.h b/lib/librte_compressdev/rte_compressdev_internal.h
index 0a2ddcb2b..57af163c1 100644
--- a/lib/librte_compressdev/rte_compressdev_internal.h
+++ b/lib/librte_compressdev/rte_compressdev_internal.h
@@ -58,6 +58,11 @@ struct rte_compressdev_data {
 	uint8_t dev_started : 1;
 	/**< Device state: STARTED(1)/STOPPED(0) */
 
+	void **queue_pairs;
+	/**< Array of pointers to queue pairs. */
+	uint16_t nb_queue_pairs;
+	/**< Number of device queue pairs */
+
 	void *dev_private;
 	/**< PMD-specific private data */
 } __rte_cache_aligned;
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
index 43307ee8e..14ce76b44 100644
--- a/lib/librte_compressdev/rte_compressdev_pmd.h
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -125,6 +125,48 @@ typedef int (*compressdev_close_t)(struct rte_compressdev *dev);
 typedef void (*compressdev_info_get_t)(struct rte_compressdev *dev,
 				struct rte_compressdev_info *dev_info);
 
+/**
+ * Setup a queue pair for a device.
+ *
+ * @param dev
+ *   Compress device
+ * @param qp_id
+ *   Queue pair identifier
+ * @param max_inflight_ops
+ *   Max inflight ops which qp must accommodate
+ * @param socket_id
+ *   Socket identifier
+ * @return
+ *   Returns 0 on success.
+ */
+typedef int (*compressdev_queue_pair_setup_t)(struct rte_compressdev *dev,
+		uint16_t qp_id,	uint32_t max_inflight_ops, int socket_id);
+
+/**
+ * Release memory resources allocated by given queue pair.
+ *
+ * @param dev
+ *   Compress device
+ * @param qp_id
+ *   Queue pair identifier
+ * @return
+ * - 0 on success.
+ * - EAGAIN if can't close as device is busy
+ */
+typedef int (*compressdev_queue_pair_release_t)(struct rte_compressdev *dev,
+		uint16_t qp_id);
+
+/**
+ * Get number of available queue pairs of a device.
+ *
+ * @param dev
+ *   Compress device
+ * @return
+ *   Returns number of queue pairs on success.
+ */
+typedef uint32_t (*compressdev_queue_pair_count_t)(struct rte_compressdev *dev);
+
+
 /** comp device operations function pointer table */
 struct rte_compressdev_ops {
 	compressdev_configure_t dev_configure;	/**< Configure device. */
@@ -133,6 +175,11 @@ struct rte_compressdev_ops {
 	compressdev_close_t dev_close;		/**< Close device. */
 
 	compressdev_info_get_t dev_infos_get;	/**< Get device info. */
+
+	compressdev_queue_pair_setup_t queue_pair_setup;
+	/**< Set up a device queue pair. */
+	compressdev_queue_pair_release_t queue_pair_release;
+	/**< Release a queue pair. */
 };
 
 /**
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index a996abcd2..182a371d8 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -16,6 +16,8 @@ EXPERIMENTAL {
 	rte_compressdev_pmd_get_named_dev;
 	rte_compressdev_pmd_parse_input_args;
 	rte_compressdev_pmd_release_device;
+	rte_compressdev_queue_pair_count;
+	rte_compressdev_queue_pair_setup;
 	rte_compressdev_socket_id;
 	rte_compressdev_start;
 	rte_compressdev_stop;
-- 
2.14.3

Re: [dpdk-dev] [PATCH v6 02/14] compressdev: add queue pair management

[Verma, Shally]

One quick question

>-----Original Message-----
>From: Pablo de Lara [mailto:pablo.de.lara.guarch@intel.com]
>Sent: 27 April 2018 18:54
>To: dev@dpdk.org
>Cc: fiona.trahe@intel.com; Verma, Shally <Shally.Verma@cavium.com>; ahmed.mansour@nxp.com; Gupta, Ashish
><Ashish.Gupta@cavium.com>; Pablo de Lara <pablo.de.lara.guarch@intel.com>; Verma, Shally <Shally.Verma@cavium.com>; Gupta,
>Ashish <Ashish.Gupta@cavium.com>
>Subject: [PATCH v6 02/14] compressdev: add queue pair management
>
>From: Fiona Trahe <fiona.trahe@intel.com>
>
>Add functions to manage device queue pairs.
>
>Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
>Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
>Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
>Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
>---
> lib/librte_compressdev/rte_compressdev.c           | 178 ++++++++++++++++++++-
> lib/librte_compressdev/rte_compressdev.h           |  45 ++++++
> lib/librte_compressdev/rte_compressdev_internal.h  |   5 +
> lib/librte_compressdev/rte_compressdev_pmd.h       |  47 ++++++
> lib/librte_compressdev/rte_compressdev_version.map |   2 +
> 5 files changed, 276 insertions(+), 1 deletion(-)
>
>diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
>index 751517c3d..6667528b5 100644
>--- a/lib/librte_compressdev/rte_compressdev.c
>+++ b/lib/librte_compressdev/rte_compressdev.c
>@@ -227,10 +227,136 @@ rte_compressdev_pmd_release_device(struct rte_compressdev *compressdev)
> 	return 0;
> }
>
>+uint16_t __rte_experimental
>+rte_compressdev_queue_pair_count(uint8_t dev_id)
>+{
>+	struct rte_compressdev *dev;
>+
>+	dev = &rte_comp_devices[dev_id];
>+	return dev->data->nb_queue_pairs;
>+}
>+
>+static int
>+rte_compressdev_queue_pairs_config(struct rte_compressdev *dev,
>+		uint16_t nb_qpairs, int socket_id)
>+{
>+	struct rte_compressdev_info dev_info;
>+	void **qp;
>+	unsigned int i;
>+
>+	if ((dev == NULL) || (nb_qpairs < 1)) {
>+		COMPRESSDEV_LOG(ERR, "invalid param: dev %p, nb_queues %u",
>+							dev, nb_qpairs);
>+		return -EINVAL;
>+	}
>+
>+	COMPRESSDEV_LOG(DEBUG, "Setup %d queues pairs on device %u",
>+			nb_qpairs, dev->data->dev_id);
>+
>+	memset(&dev_info, 0, sizeof(struct rte_compressdev_info));
>+
>+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_infos_get, -ENOTSUP);
>+	(*dev->dev_ops->dev_infos_get)(dev, &dev_info);
>+
>+	if ((dev_info.max_nb_queue_pairs != 0) &&
>+			(nb_qpairs > dev_info.max_nb_queue_pairs)) {
>+		COMPRESSDEV_LOG(ERR, "Invalid num queue_pairs (%u) for dev %u",
>+				nb_qpairs, dev->data->dev_id);
>+		return -EINVAL;
>+	}
>+
>+	if (dev->data->queue_pairs == NULL) { /* first time configuration */
>+		dev->data->queue_pairs = rte_zmalloc_socket(
>+				"compressdev->queue_pairs",
>+				sizeof(dev->data->queue_pairs[0]) * nb_qpairs,
>+				RTE_CACHE_LINE_SIZE, socket_id);
>+
>+		if (dev->data->queue_pairs == NULL) {
>+			dev->data->nb_queue_pairs = 0;
>+			COMPRESSDEV_LOG(ERR,
>+			"failed to get memory for qp meta data, nb_queues %u",
>+							nb_qpairs);
>+			return -(ENOMEM);
>+		}
>+	} else { /* re-configure */
>+		int ret;
>+		uint16_t old_nb_queues = dev->data->nb_queue_pairs;
>+
[Shally] dev->data is shared entity among processes. So, does it mean if multiple processes call for device_configure, then every new process overrides qp allocations (and other resources) already done by any other process, leaving it unusable for them?

Thanks
Shally

>+		qp = dev->data->queue_pairs;
>+
>+		RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->queue_pair_release,
>+				-ENOTSUP);
>+
>+		for (i = nb_qpairs; i < old_nb_queues; i++) {
>+			ret = (*dev->dev_ops->queue_pair_release)(dev, i);
>+			if (ret < 0)
>+				return ret;
>+		}
>+
>+		qp = rte_realloc(qp, sizeof(qp[0]) * nb_qpairs,
>+				RTE_CACHE_LINE_SIZE);
>+		if (qp == NULL) {
>+			COMPRESSDEV_LOG(ERR,
>+			"failed to realloc qp meta data, nb_queues %u",
>+						nb_qpairs);
>+			return -(ENOMEM);
>+		}
>+
>+		if (nb_qpairs > old_nb_queues) {
>+			uint16_t new_qs = nb_qpairs - old_nb_queues;
>+
>+			memset(qp + old_nb_queues, 0,
>+				sizeof(qp[0]) * new_qs);
>+		}
>+
>+		dev->data->queue_pairs = qp;
>+
>+	}
>+	dev->data->nb_queue_pairs = nb_qpairs;
>+	return 0;
>+}
>+
>+static int
>+rte_compressdev_queue_pairs_release(struct rte_compressdev *dev)
>+{
>+	uint16_t num_qps, i;
>+	int ret;
>+
>+	if (dev == NULL) {
>+		COMPRESSDEV_LOG(ERR, "invalid param: dev %p", dev);
>+		return -EINVAL;
>+	}
>+
>+	num_qps = dev->data->nb_queue_pairs;
>+
>+	if (num_qps == 0)
>+		return 0;
>+
>+	COMPRESSDEV_LOG(DEBUG, "Free %d queues pairs on device %u",
>+			dev->data->nb_queue_pairs, dev->data->dev_id);
>+
>+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->queue_pair_release,
>+				-ENOTSUP);
>+
>+	for (i = 0; i < num_qps; i++) {
>+		ret = (*dev->dev_ops->queue_pair_release)(dev, i);
>+		if (ret < 0)
>+			return ret;
>+	}
>+
>+	if (dev->data->queue_pairs != NULL)
>+		rte_free(dev->data->queue_pairs);
>+	dev->data->queue_pairs = NULL;
>+	dev->data->nb_queue_pairs = 0;
>+
>+	return 0;
>+}
>+
> int __rte_experimental
> rte_compressdev_configure(uint8_t dev_id, struct rte_compressdev_config *config)
> {
> 	struct rte_compressdev *dev;
>+	int diag;
>
> 	if (!rte_compressdev_is_valid_dev(dev_id)) {
> 		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
>@@ -247,10 +373,19 @@ rte_compressdev_configure(uint8_t dev_id, struct rte_compressdev_config *config)
>
> 	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_configure, -ENOTSUP);
>
>+	/* Setup new number of queue pairs and reconfigure device. */
>+	diag = rte_compressdev_queue_pairs_config(dev, config->nb_queue_pairs,
>+			config->socket_id);
>+	if (diag != 0) {
>+		COMPRESSDEV_LOG(ERR,
>+			"dev%d rte_comp_dev_queue_pairs_config = %d",
>+				dev_id, diag);
>+		return diag;
>+	}
>+
> 	return (*dev->dev_ops->dev_configure)(dev, config);
> }
>
>-
> int __rte_experimental
> rte_compressdev_start(uint8_t dev_id)
> {
>@@ -327,6 +462,12 @@ rte_compressdev_close(uint8_t dev_id)
> 		return -EBUSY;
> 	}
>
>+	/* Free queue pairs memory */
>+	retval = rte_compressdev_queue_pairs_release(dev);
>+
>+	if (retval < 0)
>+		return retval;
>+
> 	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_close, -ENOTSUP);
> 	retval = (*dev->dev_ops->dev_close)(dev);
>
>@@ -336,6 +477,41 @@ rte_compressdev_close(uint8_t dev_id)
> 	return 0;
> }
>
>+int __rte_experimental
>+rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
>+		uint32_t max_inflight_ops, int socket_id)
>+{
>+	struct rte_compressdev *dev;
>+
>+	if (!rte_compressdev_is_valid_dev(dev_id)) {
>+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
>+		return -EINVAL;
>+	}
>+
>+	dev = &rte_comp_devices[dev_id];
>+	if (queue_pair_id >= dev->data->nb_queue_pairs) {
>+		COMPRESSDEV_LOG(ERR, "Invalid queue_pair_id=%d", queue_pair_id);
>+		return -EINVAL;
>+	}
>+
>+	if (dev->data->dev_started) {
>+		COMPRESSDEV_LOG(ERR,
>+		    "device %d must be stopped to allow configuration", dev_id);
>+		return -EBUSY;
>+	}
>+
>+	if (max_inflight_ops == 0) {
>+		COMPRESSDEV_LOG(ERR,
>+			"Invalid maximum number of inflight operations");
>+		return -EINVAL;
>+	}
>+
>+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->queue_pair_setup, -ENOTSUP);
>+
>+	return (*dev->dev_ops->queue_pair_setup)(dev, queue_pair_id,
>+			max_inflight_ops, socket_id);
>+}
>+
> void __rte_experimental
> rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
> {
>diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
>index 5be5973bf..81710bbea 100644
>--- a/lib/librte_compressdev/rte_compressdev.h
>+++ b/lib/librte_compressdev/rte_compressdev.h
>@@ -22,6 +22,10 @@ extern "C" {
> /**  comp device information */
> struct rte_compressdev_info {
> 	const char *driver_name;		/**< Driver name. */
>+	uint16_t max_nb_queue_pairs;
>+	/**< Maximum number of queues pairs supported by device.
>+	 * (If 0, there is no limit in maximum number of queue pairs)
>+	 */
> };
>
> /**
>@@ -80,6 +84,9 @@ rte_compressdev_socket_id(uint8_t dev_id);
> /** Compress device configuration structure */
> struct rte_compressdev_config {
> 	int socket_id;
>+	/**< Socket on which to allocate resources */
>+	uint16_t nb_queue_pairs;
>+	/**< Total number of queue pairs to configure on a device */
> };
>
> /**
>@@ -145,6 +152,44 @@ rte_compressdev_stop(uint8_t dev_id);
> int __rte_experimental
> rte_compressdev_close(uint8_t dev_id);
>
>+/**
>+ * Allocate and set up a receive queue pair for a device.
>+ * This should only be called when the device is stopped.
>+ *
>+ *
>+ * @param dev_id
>+ *   Compress device identifier
>+ * @param queue_pair_id
>+ *   The index of the queue pairs to set up. The
>+ *   value must be in the range [0, nb_queue_pair - 1]
>+ *   previously supplied to rte_compressdev_configure()
>+ * @param max_inflight_ops
>+ *   Max number of ops which the qp will have to
>+ *   accommodate simultaneously
>+ * @param socket_id
>+ *   The *socket_id* argument is the socket identifier
>+ *   in case of NUMA. The value can be *SOCKET_ID_ANY*
>+ *   if there is no NUMA constraint for the DMA memory
>+ *   allocated for the receive queue pair
>+ * @return
>+ *   - 0: Success, queue pair correctly set up.
>+ *   - <0: Queue pair configuration failed
>+ */
>+int __rte_experimental
>+rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
>+		uint32_t max_inflight_ops, int socket_id);
>+
>+/**
>+ * Get the number of queue pairs on a specific comp device
>+ *
>+ * @param dev_id
>+ *   Compress device identifier
>+ * @return
>+ *   - The number of configured queue pairs.
>+ */
>+uint16_t __rte_experimental
>+rte_compressdev_queue_pair_count(uint8_t dev_id);
>+
> /**
>  * Retrieve the contextual information of a device.
>  *
>diff --git a/lib/librte_compressdev/rte_compressdev_internal.h b/lib/librte_compressdev/rte_compressdev_internal.h
>index 0a2ddcb2b..57af163c1 100644
>--- a/lib/librte_compressdev/rte_compressdev_internal.h
>+++ b/lib/librte_compressdev/rte_compressdev_internal.h
>@@ -58,6 +58,11 @@ struct rte_compressdev_data {
> 	uint8_t dev_started : 1;
> 	/**< Device state: STARTED(1)/STOPPED(0) */
>
>+	void **queue_pairs;
>+	/**< Array of pointers to queue pairs. */
>+	uint16_t nb_queue_pairs;
>+	/**< Number of device queue pairs */
>+
> 	void *dev_private;
> 	/**< PMD-specific private data */
> } __rte_cache_aligned;
>diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
>index 43307ee8e..14ce76b44 100644
>--- a/lib/librte_compressdev/rte_compressdev_pmd.h
>+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
>@@ -125,6 +125,48 @@ typedef int (*compressdev_close_t)(struct rte_compressdev *dev);
> typedef void (*compressdev_info_get_t)(struct rte_compressdev *dev,
> 				struct rte_compressdev_info *dev_info);
>
>+/**
>+ * Setup a queue pair for a device.
>+ *
>+ * @param dev
>+ *   Compress device
>+ * @param qp_id
>+ *   Queue pair identifier
>+ * @param max_inflight_ops
>+ *   Max inflight ops which qp must accommodate
>+ * @param socket_id
>+ *   Socket identifier
>+ * @return
>+ *   Returns 0 on success.
>+ */
>+typedef int (*compressdev_queue_pair_setup_t)(struct rte_compressdev *dev,
>+		uint16_t qp_id,	uint32_t max_inflight_ops, int socket_id);
>+
>+/**
>+ * Release memory resources allocated by given queue pair.
>+ *
>+ * @param dev
>+ *   Compress device
>+ * @param qp_id
>+ *   Queue pair identifier
>+ * @return
>+ * - 0 on success.
>+ * - EAGAIN if can't close as device is busy
>+ */
>+typedef int (*compressdev_queue_pair_release_t)(struct rte_compressdev *dev,
>+		uint16_t qp_id);
>+
>+/**
>+ * Get number of available queue pairs of a device.
>+ *
>+ * @param dev
>+ *   Compress device
>+ * @return
>+ *   Returns number of queue pairs on success.
>+ */
>+typedef uint32_t (*compressdev_queue_pair_count_t)(struct rte_compressdev *dev);
>+
>+
> /** comp device operations function pointer table */
> struct rte_compressdev_ops {
> 	compressdev_configure_t dev_configure;	/**< Configure device. */
>@@ -133,6 +175,11 @@ struct rte_compressdev_ops {
> 	compressdev_close_t dev_close;		/**< Close device. */
>
> 	compressdev_info_get_t dev_infos_get;	/**< Get device info. */
>+
>+	compressdev_queue_pair_setup_t queue_pair_setup;
>+	/**< Set up a device queue pair. */
>+	compressdev_queue_pair_release_t queue_pair_release;
>+	/**< Release a queue pair. */
> };
>
> /**
>diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
>index a996abcd2..182a371d8 100644
>--- a/lib/librte_compressdev/rte_compressdev_version.map
>+++ b/lib/librte_compressdev/rte_compressdev_version.map
>@@ -16,6 +16,8 @@ EXPERIMENTAL {
> 	rte_compressdev_pmd_get_named_dev;
> 	rte_compressdev_pmd_parse_input_args;
> 	rte_compressdev_pmd_release_device;
>+	rte_compressdev_queue_pair_count;
>+	rte_compressdev_queue_pair_setup;
> 	rte_compressdev_socket_id;
> 	rte_compressdev_start;
> 	rte_compressdev_stop;
>--
>2.14.3

Re: [dpdk-dev] [PATCH v6 02/14] compressdev: add queue pair management

[Trahe, Fiona]

> -----Original Message-----
> From: Verma, Shally [mailto:Shally.Verma@cavium.com]
> Sent: Friday, May 4, 2018 3:49 AM
> To: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; dev@dpdk.org
> Cc: Trahe, Fiona <fiona.trahe@intel.com>; ahmed.mansour@nxp.com; Gupta, Ashish
> <Ashish.Gupta@cavium.com>; Sahu, Sunila <Sunila.Sahu@cavium.com>
> Subject: RE: [PATCH v6 02/14] compressdev: add queue pair management
> 
> One quick question
> 
> >-----Original Message-----
> >From: Pablo de Lara [mailto:pablo.de.lara.guarch@intel.com]
> >Sent: 27 April 2018 18:54
> >To: dev@dpdk.org
> >Cc: fiona.trahe@intel.com; Verma, Shally <Shally.Verma@cavium.com>; ahmed.mansour@nxp.com;
> Gupta, Ashish
> ><Ashish.Gupta@cavium.com>; Pablo de Lara <pablo.de.lara.guarch@intel.com>; Verma, Shally
> <Shally.Verma@cavium.com>; Gupta,
> >Ashish <Ashish.Gupta@cavium.com>
> >Subject: [PATCH v6 02/14] compressdev: add queue pair management
> >
> >From: Fiona Trahe <fiona.trahe@intel.com>
> >
> >Add functions to manage device queue pairs.
> >
> >Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
> >Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
> >Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
> >Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
> >---
> > lib/librte_compressdev/rte_compressdev.c           | 178 ++++++++++++++++++++-
> > lib/librte_compressdev/rte_compressdev.h           |  45 ++++++
> > lib/librte_compressdev/rte_compressdev_internal.h  |   5 +
> > lib/librte_compressdev/rte_compressdev_pmd.h       |  47 ++++++
> > lib/librte_compressdev/rte_compressdev_version.map |   2 +
> > 5 files changed, 276 insertions(+), 1 deletion(-)
> >
> >+
> >+	if (dev->data->queue_pairs == NULL) { /* first time configuration */
> >+		dev->data->queue_pairs = rte_zmalloc_socket(
> >+				"compressdev->queue_pairs",
> >+				sizeof(dev->data->queue_pairs[0]) * nb_qpairs,
> >+				RTE_CACHE_LINE_SIZE, socket_id);
> >+
> >+		if (dev->data->queue_pairs == NULL) {
> >+			dev->data->nb_queue_pairs = 0;
> >+			COMPRESSDEV_LOG(ERR,
> >+			"failed to get memory for qp meta data, nb_queues %u",
> >+							nb_qpairs);
> >+			return -(ENOMEM);
> >+		}
> >+	} else { /* re-configure */
> >+		int ret;
> >+		uint16_t old_nb_queues = dev->data->nb_queue_pairs;
> >+
> [Shally] dev->data is shared entity among processes. So, does it mean if multiple processes call for
> device_configure, then every new process overrides qp allocations (and other resources) already done
> by any other process, leaving it unusable for them?
> 
> Thanks
> Shally
[Fiona] Yes. If multiple processes want to share a device they'd need to ensure that a primary process
configures the device and agree between the processes which process uses which resource, e.g. each
process accesses only 1 queue_pair. 
Same is true of other device management APIs like start, stop and close.

Re: [dpdk-dev] [PATCH v6 02/14] compressdev: add queue pair management

[Verma, Shally]

HI Pablo

>-----Original Message-----
>From: Pablo de Lara [mailto:pablo.de.lara.guarch@intel.com]
>Sent: 27 April 2018 18:54
>To: dev@dpdk.org
>Cc: fiona.trahe@intel.com; Verma, Shally <Shally.Verma@cavium.com>; ahmed.mansour@nxp.com; Gupta, Ashish
><Ashish.Gupta@cavium.com>; Pablo de Lara <pablo.de.lara.guarch@intel.com>; Verma, Shally <Shally.Verma@cavium.com>; Gupta,
>Ashish <Ashish.Gupta@cavium.com>
>Subject: [PATCH v6 02/14] compressdev: add queue pair management
>
>From: Fiona Trahe <fiona.trahe@intel.com>
>
>Add functions to manage device queue pairs.
>
>Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
>Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
>Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
>Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
>---
> lib/librte_compressdev/rte_compressdev.c           | 178 ++++++++++++++++++++-
> lib/librte_compressdev/rte_compressdev.h           |  45 ++++++
> lib/librte_compressdev/rte_compressdev_internal.h  |   5 +
> lib/librte_compressdev/rte_compressdev_pmd.h       |  47 ++++++
> lib/librte_compressdev/rte_compressdev_version.map |   2 +
> 5 files changed, 276 insertions(+), 1 deletion(-)
>
>diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
>index 751517c3d..6667528b5 100644
>--- a/lib/librte_compressdev/rte_compressdev.c
>+++ b/lib/librte_compressdev/rte_compressdev.c
>@@ -227,10 +227,136 @@ rte_compressdev_pmd_release_device(struct rte_compressdev *compressdev)
> 	return 0;
> }
>
>+uint16_t __rte_experimental
>+rte_compressdev_queue_pair_count(uint8_t dev_id)
>+{
>+	struct rte_compressdev *dev;
>+
>+	dev = &rte_comp_devices[dev_id];
>+	return dev->data->nb_queue_pairs;
>+}
>+
>+static int
>+rte_compressdev_queue_pairs_config(struct rte_compressdev *dev,
>+		uint16_t nb_qpairs, int socket_id)
>+{
>+	struct rte_compressdev_info dev_info;
>+	void **qp;
>+	unsigned int i;
>+
>+	if ((dev == NULL) || (nb_qpairs < 1)) {
>+		COMPRESSDEV_LOG(ERR, "invalid param: dev %p, nb_queues %u",
>+							dev, nb_qpairs);
[Shally] shouldn't nb_qpairs < 0 as  nb_qpairs = 0 a valid input which imply number of supported qp is implementation defined and thus unlimited.

Thanks
Shally

//snip

Re: [dpdk-dev] [PATCH v6 02/14] compressdev: add queue pair management

[De Lara Guarch, Pablo]

Hi Shally,

> -----Original Message-----
> From: Verma, Shally [mailto:Shally.Verma@cavium.com]
> Sent: Monday, May 14, 2018 8:54 AM
> To: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; dev@dpdk.org
> Cc: Trahe, Fiona <fiona.trahe@intel.com>; ahmed.mansour@nxp.com; Gupta,
> Ashish <Ashish.Gupta@cavium.com>; Gupta, Ashish
> <Ashish.Gupta@cavium.com>; Sahu, Sunila <Sunila.Sahu@cavium.com>
> Subject: RE: [PATCH v6 02/14] compressdev: add queue pair management
> 
> HI Pablo
> 
> >-----Original Message-----
> >From: Pablo de Lara [mailto:pablo.de.lara.guarch@intel.com]
> >Sent: 27 April 2018 18:54
> >To: dev@dpdk.org
> >Cc: fiona.trahe@intel.com; Verma, Shally <Shally.Verma@cavium.com>;
> >ahmed.mansour@nxp.com; Gupta, Ashish <Ashish.Gupta@cavium.com>; Pablo
> >de Lara <pablo.de.lara.guarch@intel.com>; Verma, Shally
> ><Shally.Verma@cavium.com>; Gupta, Ashish <Ashish.Gupta@cavium.com>
> >Subject: [PATCH v6 02/14] compressdev: add queue pair management
> >
> >From: Fiona Trahe <fiona.trahe@intel.com>
> >
> >Add functions to manage device queue pairs.
> >
> >Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
> >Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
> >Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
> >Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
> >---
> > lib/librte_compressdev/rte_compressdev.c           | 178
> ++++++++++++++++++++-
> > lib/librte_compressdev/rte_compressdev.h           |  45 ++++++
> > lib/librte_compressdev/rte_compressdev_internal.h  |   5 +
> > lib/librte_compressdev/rte_compressdev_pmd.h       |  47 ++++++
> > lib/librte_compressdev/rte_compressdev_version.map |   2 +
> > 5 files changed, 276 insertions(+), 1 deletion(-)
> >
> >diff --git a/lib/librte_compressdev/rte_compressdev.c
> >b/lib/librte_compressdev/rte_compressdev.c
> >index 751517c3d..6667528b5 100644
> >--- a/lib/librte_compressdev/rte_compressdev.c
> >+++ b/lib/librte_compressdev/rte_compressdev.c
> >@@ -227,10 +227,136 @@ rte_compressdev_pmd_release_device(struct
> rte_compressdev *compressdev)
> > 	return 0;
> > }
> >
> >+uint16_t __rte_experimental
> >+rte_compressdev_queue_pair_count(uint8_t dev_id) {
> >+	struct rte_compressdev *dev;
> >+
> >+	dev = &rte_comp_devices[dev_id];
> >+	return dev->data->nb_queue_pairs;
> >+}
> >+
> >+static int
> >+rte_compressdev_queue_pairs_config(struct rte_compressdev *dev,
> >+		uint16_t nb_qpairs, int socket_id)
> >+{
> >+	struct rte_compressdev_info dev_info;
> >+	void **qp;
> >+	unsigned int i;
> >+
> >+	if ((dev == NULL) || (nb_qpairs < 1)) {
> >+		COMPRESSDEV_LOG(ERR, "invalid param: dev %p, nb_queues
> %u",
> >+							dev, nb_qpairs);
> [Shally] shouldn't nb_qpairs < 0 as  nb_qpairs = 0 a valid input which imply
> number of supported qp is implementation defined and thus unlimited.

You mean maximum number of queues. This is number of queues to configure,
which should be at least 1.

Thanks,
Pablo

> 
> Thanks
> Shally
> 
> //snip

Re: [dpdk-dev] [PATCH v6 02/14] compressdev: add queue pair management

[Verma, Shally]

>-----Original Message-----
>From: De Lara Guarch, Pablo [mailto:pablo.de.lara.guarch@intel.com]
>Sent: 14 May 2018 13:34
>To: Verma, Shally <Shally.Verma@cavium.com>; dev@dpdk.org
>Cc: Trahe, Fiona <fiona.trahe@intel.com>; ahmed.mansour@nxp.com; Gupta, Ashish <Ashish.Gupta@cavium.com>; Gupta, Ashish
><Ashish.Gupta@cavium.com>; Sahu, Sunila <Sunila.Sahu@cavium.com>
>Subject: RE: [PATCH v6 02/14] compressdev: add queue pair management
>
>Hi Shally,
>
>> -----Original Message-----
>> From: Verma, Shally [mailto:Shally.Verma@cavium.com]
>> Sent: Monday, May 14, 2018 8:54 AM
>> To: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; dev@dpdk.org
>> Cc: Trahe, Fiona <fiona.trahe@intel.com>; ahmed.mansour@nxp.com; Gupta,
>> Ashish <Ashish.Gupta@cavium.com>; Gupta, Ashish
>> <Ashish.Gupta@cavium.com>; Sahu, Sunila <Sunila.Sahu@cavium.com>
>> Subject: RE: [PATCH v6 02/14] compressdev: add queue pair management
>>
>> HI Pablo
>>
>> >-----Original Message-----
>> >From: Pablo de Lara [mailto:pablo.de.lara.guarch@intel.com]
>> >Sent: 27 April 2018 18:54
>> >To: dev@dpdk.org
>> >Cc: fiona.trahe@intel.com; Verma, Shally <Shally.Verma@cavium.com>;
>> >ahmed.mansour@nxp.com; Gupta, Ashish <Ashish.Gupta@cavium.com>; Pablo
>> >de Lara <pablo.de.lara.guarch@intel.com>; Verma, Shally
>> ><Shally.Verma@cavium.com>; Gupta, Ashish <Ashish.Gupta@cavium.com>
>> >Subject: [PATCH v6 02/14] compressdev: add queue pair management
>> >
>> >From: Fiona Trahe <fiona.trahe@intel.com>
>> >
>> >Add functions to manage device queue pairs.
>> >
>> >Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
>> >Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
>> >Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
>> >Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
>> >---
>> > lib/librte_compressdev/rte_compressdev.c           | 178
>> ++++++++++++++++++++-
>> > lib/librte_compressdev/rte_compressdev.h           |  45 ++++++
>> > lib/librte_compressdev/rte_compressdev_internal.h  |   5 +
>> > lib/librte_compressdev/rte_compressdev_pmd.h       |  47 ++++++
>> > lib/librte_compressdev/rte_compressdev_version.map |   2 +
>> > 5 files changed, 276 insertions(+), 1 deletion(-)
>> >
>> >diff --git a/lib/librte_compressdev/rte_compressdev.c
>> >b/lib/librte_compressdev/rte_compressdev.c
>> >index 751517c3d..6667528b5 100644
>> >--- a/lib/librte_compressdev/rte_compressdev.c
>> >+++ b/lib/librte_compressdev/rte_compressdev.c
>> >@@ -227,10 +227,136 @@ rte_compressdev_pmd_release_device(struct
>> rte_compressdev *compressdev)
>> > 	return 0;
>> > }
>> >
>> >+uint16_t __rte_experimental
>> >+rte_compressdev_queue_pair_count(uint8_t dev_id) {
>> >+	struct rte_compressdev *dev;
>> >+
>> >+	dev = &rte_comp_devices[dev_id];
>> >+	return dev->data->nb_queue_pairs;
>> >+}
>> >+
>> >+static int
>> >+rte_compressdev_queue_pairs_config(struct rte_compressdev *dev,
>> >+		uint16_t nb_qpairs, int socket_id)
>> >+{
>> >+	struct rte_compressdev_info dev_info;
>> >+	void **qp;
>> >+	unsigned int i;
>> >+
>> >+	if ((dev == NULL) || (nb_qpairs < 1)) {
>> >+		COMPRESSDEV_LOG(ERR, "invalid param: dev %p, nb_queues
>> %u",
>> >+							dev, nb_qpairs);
>> [Shally] shouldn't nb_qpairs < 0 as  nb_qpairs = 0 a valid input which imply
>> number of supported qp is implementation defined and thus unlimited.
>
>You mean maximum number of queues. This is number of queues to configure,
>which should be at least 1.
App can call dev_configure with max as returned by dev_info_get(), in case this value is 0 i.e.  implementation defined,
then app may choose to leave this value to 0 and configure device with its default setup. App, in any case, will call for qp_setup() 
to configure as many qp as it want. Isn't it so?
 
>
>Thanks,
>Pablo
>
>>
>> Thanks
>> Shally
>>
>> //snip

Re: [dpdk-dev] [PATCH v6 02/14] compressdev: add queue pair management

[De Lara Guarch, Pablo]

> -----Original Message-----
> From: Verma, Shally [mailto:Shally.Verma@cavium.com]
> Sent: Monday, May 14, 2018 9:17 AM
> To: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; dev@dpdk.org
> Cc: Trahe, Fiona <fiona.trahe@intel.com>; ahmed.mansour@nxp.com; Gupta,
> Ashish <Ashish.Gupta@cavium.com>; Gupta, Ashish
> <Ashish.Gupta@cavium.com>; Sahu, Sunila <Sunila.Sahu@cavium.com>
> Subject: RE: [PATCH v6 02/14] compressdev: add queue pair management
> 
> 
> 
> >-----Original Message-----
> >From: De Lara Guarch, Pablo [mailto:pablo.de.lara.guarch@intel.com]
> >Sent: 14 May 2018 13:34
> >To: Verma, Shally <Shally.Verma@cavium.com>; dev@dpdk.org
> >Cc: Trahe, Fiona <fiona.trahe@intel.com>; ahmed.mansour@nxp.com; Gupta,
> >Ashish <Ashish.Gupta@cavium.com>; Gupta, Ashish
> ><Ashish.Gupta@cavium.com>; Sahu, Sunila <Sunila.Sahu@cavium.com>
> >Subject: RE: [PATCH v6 02/14] compressdev: add queue pair management
> >
> >Hi Shally,
> >
> >> -----Original Message-----
> >> From: Verma, Shally [mailto:Shally.Verma@cavium.com]
> >> Sent: Monday, May 14, 2018 8:54 AM
> >> To: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>;
> >> dev@dpdk.org
> >> Cc: Trahe, Fiona <fiona.trahe@intel.com>; ahmed.mansour@nxp.com;
> >> Gupta, Ashish <Ashish.Gupta@cavium.com>; Gupta, Ashish
> >> <Ashish.Gupta@cavium.com>; Sahu, Sunila <Sunila.Sahu@cavium.com>
> >> Subject: RE: [PATCH v6 02/14] compressdev: add queue pair management
> >>
> >> HI Pablo
> >>
> >> >-----Original Message-----
> >> >From: Pablo de Lara [mailto:pablo.de.lara.guarch@intel.com]
> >> >Sent: 27 April 2018 18:54
> >> >To: dev@dpdk.org
> >> >Cc: fiona.trahe@intel.com; Verma, Shally <Shally.Verma@cavium.com>;
> >> >ahmed.mansour@nxp.com; Gupta, Ashish <Ashish.Gupta@cavium.com>;
> >> >Pablo de Lara <pablo.de.lara.guarch@intel.com>; Verma, Shally
> >> ><Shally.Verma@cavium.com>; Gupta, Ashish <Ashish.Gupta@cavium.com>
> >> >Subject: [PATCH v6 02/14] compressdev: add queue pair management
> >> >
> >> >From: Fiona Trahe <fiona.trahe@intel.com>
> >> >
> >> >Add functions to manage device queue pairs.
> >> >
> >> >Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
> >> >Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
> >> >Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
> >> >Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
> >> >---
> >> > lib/librte_compressdev/rte_compressdev.c           | 178
> >> ++++++++++++++++++++-
> >> > lib/librte_compressdev/rte_compressdev.h           |  45 ++++++
> >> > lib/librte_compressdev/rte_compressdev_internal.h  |   5 +
> >> > lib/librte_compressdev/rte_compressdev_pmd.h       |  47 ++++++
> >> > lib/librte_compressdev/rte_compressdev_version.map |   2 +
> >> > 5 files changed, 276 insertions(+), 1 deletion(-)
> >> >
> >> >diff --git a/lib/librte_compressdev/rte_compressdev.c
> >> >b/lib/librte_compressdev/rte_compressdev.c
> >> >index 751517c3d..6667528b5 100644
> >> >--- a/lib/librte_compressdev/rte_compressdev.c
> >> >+++ b/lib/librte_compressdev/rte_compressdev.c
> >> >@@ -227,10 +227,136 @@ rte_compressdev_pmd_release_device(struct
> >> rte_compressdev *compressdev)
> >> > 	return 0;
> >> > }
> >> >
> >> >+uint16_t __rte_experimental
> >> >+rte_compressdev_queue_pair_count(uint8_t dev_id) {
> >> >+	struct rte_compressdev *dev;
> >> >+
> >> >+	dev = &rte_comp_devices[dev_id];
> >> >+	return dev->data->nb_queue_pairs;
> >> >+}
> >> >+
> >> >+static int
> >> >+rte_compressdev_queue_pairs_config(struct rte_compressdev *dev,
> >> >+		uint16_t nb_qpairs, int socket_id) {
> >> >+	struct rte_compressdev_info dev_info;
> >> >+	void **qp;
> >> >+	unsigned int i;
> >> >+
> >> >+	if ((dev == NULL) || (nb_qpairs < 1)) {
> >> >+		COMPRESSDEV_LOG(ERR, "invalid param: dev %p, nb_queues
> >> %u",
> >> >+							dev, nb_qpairs);
> >> [Shally] shouldn't nb_qpairs < 0 as  nb_qpairs = 0 a valid input
> >> which imply number of supported qp is implementation defined and thus
> unlimited.
> >
> >You mean maximum number of queues. This is number of queues to
> >configure, which should be at least 1.
> App can call dev_configure with max as returned by dev_info_get(), in case this
> value is 0 i.e.  implementation defined, then app may choose to leave this value
> to 0 and configure device with its default setup. App, in any case, will call for
> qp_setup() to configure as many qp as it want. Isn't it so?

The number of queue pairs to be used/configured depends on the number of threads
to be used, so it is up to the app to set this value.
Also, the array storing the queue pairs is created when calling dev_configure,
so it needs to have a value different than 0.
Therefore, I think passing a 0 here is not useful.

Thanks,
Pablo

> 
> >
> >Thanks,
> >Pablo
> >
> >>
> >> Thanks
> >> Shally
> >>
> >> //snip

[dpdk-dev] [PATCH v6 03/14] compressdev: add compression specific data

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added structures and enums specific to compression,
including the compression operation structure and the
different supported algorithms, checksums and compression
levels.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 doc/api/doxy-api-index.md          |   1 +
 lib/librte_compressdev/Makefile    |   1 +
 lib/librte_compressdev/meson.build |   3 +-
 lib/librte_compressdev/rte_comp.h  | 316 +++++++++++++++++++++++++++++++++++++
 4 files changed, 320 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_compressdev/rte_comp.h

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index a3dfc0899..f53bd2431 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -19,6 +19,7 @@ The public API headers are grouped by topics:
   [cryptodev]          (@ref rte_cryptodev.h),
   [security]           (@ref rte_security.h),
   [compressdev]        (@ref rte_compressdev.h),
+  [compress]           (@ref rte_comp.h),
   [eventdev]           (@ref rte_eventdev.h),
   [event_eth_rx_adapter]   (@ref rte_event_eth_rx_adapter.h),
   [event_timer_adapter]    (@ref rte_event_timer_adapter.h),
diff --git a/lib/librte_compressdev/Makefile b/lib/librte_compressdev/Makefile
index b84769417..4dae6ca42 100644
--- a/lib/librte_compressdev/Makefile
+++ b/lib/librte_compressdev/Makefile
@@ -19,6 +19,7 @@ LDLIBS += -lrte_eal -lrte_mempool -lrte_kvargs
 SRCS-y += rte_compressdev.c rte_compressdev_pmd.c
 
 # export include files
+SYMLINK-y-include += rte_comp.h
 SYMLINK-y-include += rte_compressdev.h
 # export include files (for PMDs)
 SYMLINK-y-include += rte_compressdev_pmd.h
diff --git a/lib/librte_compressdev/meson.build b/lib/librte_compressdev/meson.build
index 895e959d3..2115961c2 100644
--- a/lib/librte_compressdev/meson.build
+++ b/lib/librte_compressdev/meson.build
@@ -6,5 +6,6 @@ sources = files('rte_compressdev.c',
 	'rte_compressdev_pmd.c')
 headers = files('rte_compressdev.h',
 	'rte_compressdev_pmd.h',
-	'rte_compressdev_internal.h')
+	'rte_compressdev_internal.h',
+	'rte_comp.h')
 deps += ['kvargs', 'mbuf']
diff --git a/lib/librte_compressdev/rte_comp.h b/lib/librte_compressdev/rte_comp.h
new file mode 100644
index 000000000..9f35c37c5
--- /dev/null
+++ b/lib/librte_compressdev/rte_comp.h
@@ -0,0 +1,316 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#ifndef _RTE_COMP_H_
+#define _RTE_COMP_H_
+
+/**
+ * @file rte_comp.h
+ *
+ * RTE definitions for Data Compression Service
+ *
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_mempool.h>
+#include <rte_mbuf.h>
+
+/** Status of comp operation */
+enum rte_comp_op_status {
+	RTE_COMP_OP_STATUS_SUCCESS = 0,
+	/**< Operation completed successfully */
+	RTE_COMP_OP_STATUS_NOT_PROCESSED,
+	/**< Operation has not yet been processed by the device */
+	RTE_COMP_OP_STATUS_INVALID_ARGS,
+	/**< Operation failed due to invalid arguments in request */
+	RTE_COMP_OP_STATUS_ERROR,
+	/**< Error handling operation */
+	RTE_COMP_OP_STATUS_INVALID_STATE,
+	/**< Operation is invoked in invalid state */
+	RTE_COMP_OP_STATUS_OUT_OF_SPACE_TERMINATED,
+	/**< Output buffer ran out of space before operation completed.
+	 * Error case. Application must resubmit all data with a larger
+	 * output buffer.
+	 */
+	RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE,
+	/**< Output buffer ran out of space before operation completed, but this
+	 * is not an error case. Output data up to op.produced can be used and
+	 * next op in the stream should continue on from op.consumed+1.
+	 */
+};
+
+/** Compression Algorithms */
+enum rte_comp_algorithm {
+	RTE_COMP_ALGO_UNSPECIFIED = 0,
+	/** No Compression algorithm */
+	RTE_COMP_ALGO_NULL,
+	/**< No compression.
+	 * Pass-through, data is copied unchanged from source buffer to
+	 * destination buffer.
+	 */
+	RTE_COMP_ALGO_DEFLATE,
+	/**< DEFLATE compression algorithm
+	 * https://tools.ietf.org/html/rfc1951
+	 */
+	RTE_COMP_ALGO_LZS,
+	/**< LZS compression algorithm
+	 * https://tools.ietf.org/html/rfc2395
+	 */
+	RTE_COMP_ALGO_LIST_END
+};
+
+/**< Compression Level.
+ * The number is interpreted by each PMD differently. However, lower numbers
+ * give fastest compression, at the expense of compression ratio while
+ * higher numbers may give better compression ratios but are likely slower.
+ */
+#define	RTE_COMP_LEVEL_PMD_DEFAULT	(-1)
+/** Use PMD Default */
+#define	RTE_COMP_LEVEL_NONE		(0)
+/** Output uncompressed blocks if supported by the specified algorithm */
+#define RTE_COMP_LEVEL_MIN		(1)
+/** Use minimum compression level supported by the PMD */
+#define RTE_COMP_LEVEL_MAX		(9)
+/** Use maximum compression level supported by the PMD */
+
+/** Compression checksum types */
+enum rte_comp_checksum_type {
+	RTE_COMP_CHECKSUM_NONE,
+	/**< No checksum generated */
+	RTE_COMP_CHECKSUM_CRC32,
+	/**< Generates a CRC32 checksum, as used by gzip */
+	RTE_COMP_CHECKSUM_ADLER32,
+	/**< Generates an Adler-32 checksum, as used by zlib */
+	RTE_COMP_CHECKSUM_CRC32_ADLER32,
+	/**< Generates both Adler-32 and CRC32 checksums, concatenated.
+	 * CRC32 is in the lower 32bits, Adler-32 in the upper 32 bits.
+	 */
+};
+
+
+/** Compression Huffman Type - used by DEFLATE algorithm */
+enum rte_comp_huffman {
+	RTE_COMP_HUFFMAN_DEFAULT,
+	/**< PMD may choose which Huffman codes to use */
+	RTE_COMP_HUFFMAN_FIXED,
+	/**< Use Fixed Huffman codes */
+	RTE_COMP_HUFFMAN_DYNAMIC,
+	/**< Use Dynamic Huffman codes */
+};
+
+/** Compression flush flags */
+enum rte_comp_flush_flag {
+	RTE_COMP_FLUSH_NONE,
+	/**< Data is not flushed. Output may remain in the compressor and be
+	 * processed during a following op. It may not be possible to decompress
+	 * output until a later op with some other flush flag has been sent.
+	 */
+	RTE_COMP_FLUSH_SYNC,
+	/**< All data should be flushed to output buffer. Output data can be
+	 * decompressed. However state and history is not cleared, so future
+	 * operations may use history from this operation.
+	 */
+	RTE_COMP_FLUSH_FULL,
+	/**< All data should be flushed to output buffer. Output data can be
+	 * decompressed. State and history data is cleared, so future
+	 * ops will be independent of ops processed before this.
+	 */
+	RTE_COMP_FLUSH_FINAL
+	/**< Same as RTE_COMP_FLUSH_FULL but if op.algo is RTE_COMP_ALGO_DEFLATE
+	 * then bfinal bit is set in the last block.
+	 */
+};
+
+/** Compression transform types */
+enum rte_comp_xform_type {
+	RTE_COMP_COMPRESS,
+	/**< Compression service - compress */
+	RTE_COMP_DECOMPRESS,
+	/**< Compression service - decompress */
+};
+
+/** Compression operation type */
+enum rte_comp_op_type {
+	RTE_COMP_OP_STATELESS,
+	/**< All data to be processed is submitted in the op, no state or
+	 * history from previous ops is used and none will be stored for future
+	 * ops. Flush flag must be set to either FLUSH_FULL or FLUSH_FINAL.
+	 */
+	RTE_COMP_OP_STATEFUL
+	/**< There may be more data to be processed after this op, it's part of
+	 * a stream of data. State and history from previous ops can be used
+	 * and resulting state and history can be stored for future ops,
+	 * depending on flush flag.
+	 */
+};
+
+
+/** Parameters specific to the deflate algorithm */
+struct rte_comp_deflate_params {
+	enum rte_comp_huffman huffman;
+	/**< Compression huffman encoding type */
+};
+
+/** Setup Data for compression */
+struct rte_comp_compress_xform {
+	enum rte_comp_algorithm algo;
+	/**< Algorithm to use for compress operation */
+	union {
+		struct rte_comp_deflate_params deflate;
+		/**< Parameters specific to the deflate algorithm */
+	}; /**< Algorithm specific parameters */
+	int level;
+	/**< Compression level */
+	uint8_t window_size;
+	/**< Base two log value of sliding window to be used. If window size
+	 * can't be supported by the PMD then it may fall back to a smaller
+	 * size. This is likely to result in a worse compression ratio.
+	 */
+	enum rte_comp_checksum_type chksum;
+	/**< Type of checksum to generate on the uncompressed data */
+};
+
+/**
+ * Setup Data for decompression.
+ */
+struct rte_comp_decompress_xform {
+	enum rte_comp_algorithm algo;
+	/**< Algorithm to use for decompression */
+	enum rte_comp_checksum_type chksum;
+	/**< Type of checksum to generate on the decompressed data */
+	uint8_t window_size;
+	/**< Base two log value of sliding window which was used to generate
+	 * compressed data. If window size can't be supported by the PMD then
+	 * setup of stream or private_xform should fail.
+	 */
+};
+
+/**
+ * Compression transform structure.
+ *
+ * This is used to specify the compression transforms required.
+ * Each transform structure can hold a single transform, the type field is
+ * used to specify which transform is contained within the union.
+ */
+struct rte_comp_xform {
+	enum rte_comp_xform_type type;
+	/**< xform type */
+	union {
+		struct rte_comp_compress_xform compress;
+		/**< xform for compress operation */
+		struct rte_comp_decompress_xform decompress;
+		/**< decompress xform */
+	};
+};
+
+/**
+ * Compression Operation.
+ *
+ * This structure contains data relating to performing a compression
+ * operation on the referenced mbuf data buffers.
+ *
+ * Comp operations are enqueued and dequeued in comp PMDs using the
+ * rte_compressdev_enqueue_burst() / rte_compressdev_dequeue_burst() APIs
+ */
+struct rte_comp_op {
+	enum rte_comp_op_type op_type;
+	union {
+		void *private_xform;
+		/**< Stateless private PMD data derived from an rte_comp_xform.
+		 * A handle returned by rte_compressdev_private_xform_create()
+		 * must be attached to operations of op_type RTE_COMP_STATELESS.
+		 */
+		void *stream;
+		/**< Private PMD data derived initially from an rte_comp_xform,
+		 * which holds state and history data and evolves as operations
+		 * are processed. rte_compressdev_stream_create() must be called
+		 * on a device for all STATEFUL data streams and the resulting
+		 * stream attached to the one or more operations associated
+		 * with the data stream.
+		 * All operations in a stream must be sent to the same device.
+		 */
+	};
+
+	struct rte_mempool *mempool;
+	/**< Pool from which operation is allocated */
+	rte_iova_t iova_addr;
+	/**< IOVA address of this operation */
+	struct rte_mbuf *m_src;
+	/**< source mbuf
+	 * The total size of the input buffer(s) can be retrieved using
+	 * rte_pktmbuf_data_len(m_src)
+	 */
+	struct rte_mbuf *m_dst;
+	/**< destination mbuf
+	 * The total size of the output buffer(s) can be retrieved using
+	 * rte_pktmbuf_data_len(m_dst)
+	 */
+
+	struct {
+		uint32_t offset;
+		/**< Starting point for compression or decompression,
+		 * specified as number of bytes from start of packet in
+		 * source buffer.
+		 * Starting point for checksum generation in compress direction.
+		 */
+		uint32_t length;
+		/**< The length, in bytes, of the data in source buffer
+		 * to be compressed or decompressed.
+		 * Also the length of the data over which the checksum
+		 * should be generated in compress direction
+		 */
+	} src;
+	struct {
+		uint32_t offset;
+		/**< Starting point for writing output data, specified as
+		 * number of bytes from start of packet in dest
+		 * buffer. Starting point for checksum generation in
+		 * decompress direction.
+		 */
+	} dst;
+	enum rte_comp_flush_flag flush_flag;
+	/**< Defines flush characteristics for the output data.
+	 * Only applicable in compress direction
+	 */
+	uint64_t input_chksum;
+	/**< An input checksum can be provided to generate a
+	 * cumulative checksum across sequential blocks in a STATELESS stream.
+	 * Checksum type is as specified in xform chksum_type
+	 */
+	uint64_t output_chksum;
+	/**< If a checksum is generated it will be written in here.
+	 * Checksum type is as specified in xform chksum_type.
+	 */
+	uint32_t consumed;
+	/**< The number of bytes from the source buffer
+	 * which were compressed/decompressed.
+	 */
+	uint32_t produced;
+	/**< The number of bytes written to the destination buffer
+	 * which were compressed/decompressed.
+	 */
+	uint64_t debug_status;
+	/**<
+	 * Status of the operation is returned in the status param.
+	 * This field allows the PMD to pass back extra
+	 * pmd-specific debug information. Value is not defined on the API.
+	 */
+	uint8_t status;
+	/**<
+	 * Operation status - use values from enum rte_comp_status.
+	 * This is reset to
+	 * RTE_COMP_OP_STATUS_NOT_PROCESSED on allocation from mempool and
+	 * will be set to RTE_COMP_OP_STATUS_SUCCESS after operation
+	 * is successfully processed by a PMD
+	 */
+} __rte_cache_aligned;
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_COMP_H_ */
-- 
2.14.3

[dpdk-dev] [PATCH v6 04/14] compressdev: add enqueue/dequeue functions

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           |  22 +++++
 lib/librte_compressdev/rte_compressdev.h           | 107 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_internal.h  |  49 ++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |   2 +
 4 files changed, 180 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 6667528b5..af0616980 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -512,6 +512,28 @@ rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
 			max_inflight_ops, socket_id);
 }
 
+uint16_t __rte_experimental
+rte_compressdev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	struct rte_compressdev *dev = &rte_compressdevs[dev_id];
+
+	nb_ops = (*dev->dequeue_burst)
+			(dev->data->queue_pairs[qp_id], ops, nb_ops);
+
+	return nb_ops;
+}
+
+uint16_t __rte_experimental
+rte_compressdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	struct rte_compressdev *dev = &rte_compressdevs[dev_id];
+
+	return (*dev->enqueue_burst)(
+			dev->data->queue_pairs[qp_id], ops, nb_ops);
+}
+
 void __rte_experimental
 rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 81710bbea..9660e5af6 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -19,6 +19,8 @@ extern "C" {
 
 #include <rte_common.h>
 
+#include "rte_comp.h"
+
 /**  comp device information */
 struct rte_compressdev_info {
 	const char *driver_name;		/**< Driver name. */
@@ -207,6 +209,111 @@ rte_compressdev_queue_pair_count(uint8_t dev_id);
 void __rte_experimental
 rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info);
 
+/**
+ *
+ * Dequeue a burst of processed compression operations from a queue on the comp
+ * device. The dequeued operation are stored in *rte_comp_op* structures
+ * whose pointers are supplied in the *ops* array.
+ *
+ * The rte_compressdev_dequeue_burst() function returns the number of ops
+ * actually dequeued, which is the number of *rte_comp_op* data structures
+ * effectively supplied into the *ops* array.
+ *
+ * A return value equal to *nb_ops* indicates that the queue contained
+ * at least *nb_ops* operations, and this is likely to signify that other
+ * processed operations remain in the devices output queue. Applications
+ * implementing a "retrieve as many processed operations as possible" policy
+ * can check this specific case and keep invoking the
+ * rte_compressdev_dequeue_burst() function until a value less than
+ * *nb_ops* is returned.
+ *
+ * The rte_compressdev_dequeue_burst() function does not provide any error
+ * notification to avoid the corresponding overhead.
+ *
+ * @note: operation ordering is not maintained within the queue pair.
+ *
+ * @note: In case op status = OUT_OF_SPACE_TERMINATED, op.consumed=0 and the
+ * op must be resubmitted with the same input data and a larger output buffer.
+ * op.produced is usually 0, but in decompression cases a PMD may return > 0
+ * and the application may find it useful to inspect that data.
+ * This status is only returned on STATELESS ops.
+ *
+ * @note: In case op status = OUT_OF_SPACE_RECOVERABLE, op.produced can be used
+ * and next op in stream should continue on from op.consumed+1 with a fresh
+ * output buffer.
+ * Consumed=0, produced=0 is an unusual but allowed case. There may be useful
+ * state/history stored in the PMD, even though no output was produced yet.
+ *
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param qp_id
+ *   The index of the queue pair from which to retrieve
+ *   processed operations. The value must be in the range
+ *   [0, nb_queue_pair - 1] previously supplied to
+ *   rte_compressdev_configure()
+ * @param ops
+ *   The address of an array of pointers to
+ *   *rte_comp_op* structures that must be
+ *   large enough to store *nb_ops* pointers in it
+ * @param nb_ops
+ *   The maximum number of operations to dequeue
+ * @return
+ *   - The number of operations actually dequeued, which is the number
+ *   of pointers to *rte_comp_op* structures effectively supplied to the
+ *   *ops* array.
+ */
+uint16_t __rte_experimental
+rte_compressdev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_comp_op **ops, uint16_t nb_ops);
+
+/**
+ * Enqueue a burst of operations for processing on a compression device.
+ *
+ * The rte_compressdev_enqueue_burst() function is invoked to place
+ * comp operations on the queue *qp_id* of the device designated by
+ * its *dev_id*.
+ *
+ * The *nb_ops* parameter is the number of operations to process which are
+ * supplied in the *ops* array of *rte_comp_op* structures.
+ *
+ * The rte_compressdev_enqueue_burst() function returns the number of
+ * operations it actually enqueued for processing. A return value equal to
+ * *nb_ops* means that all packets have been enqueued.
+ *
+ * @note All compression operations are Out-of-place (OOP) operations,
+ * as the size of the output data is different to the size of the input data.
+ *
+ * @note The flush flag only applies to operations which return SUCCESS.
+ * In OUT_OF_SPACE cases whether STATEFUL or STATELESS, data in dest buffer
+ * is as if flush flag was FLUSH_NONE.
+ * @note flush flag only applies in compression direction. It has no meaning
+ * for decompression.
+ * @note: operation ordering is not maintained within the queue pair.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param qp_id
+ *   The index of the queue pair on which operations
+ *   are to be enqueued for processing. The value
+ *   must be in the range [0, nb_queue_pairs - 1]
+ *   previously supplied to *rte_compressdev_configure*
+ * @param ops
+ *   The address of an array of *nb_ops* pointers
+ *   to *rte_comp_op* structures which contain
+ *   the operations to be processed
+ * @param nb_ops
+ *   The number of operations to process
+ * @return
+ *   The number of operations actually enqueued on the device. The return
+ *   value can be less than the value of the *nb_ops* parameter when the
+ *   comp devices queue is full or if invalid parameters are specified in
+ *   a *rte_comp_op*.
+ */
+uint16_t __rte_experimental
+rte_compressdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
+		struct rte_comp_op **ops, uint16_t nb_ops);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_compressdev/rte_compressdev_internal.h b/lib/librte_compressdev/rte_compressdev_internal.h
index 57af163c1..e33e75eb2 100644
--- a/lib/librte_compressdev/rte_compressdev_internal.h
+++ b/lib/librte_compressdev/rte_compressdev_internal.h
@@ -10,6 +10,8 @@
  */
 #include <rte_log.h>
 
+#include "rte_comp.h"
+
 #define RTE_COMPRESSDEV_NAME_MAX_LEN	(64)
 /**< Max length of name of comp PMD */
 
@@ -19,9 +21,56 @@ extern int compressdev_logtype;
 	rte_log(RTE_LOG_ ## level, compressdev_logtype, "%s(): "fmt "\n", \
 			__func__, ##args)
 
+/**
+ * Dequeue processed packets from queue pair of a device.
+ *
+ * @param qp
+ *   The queue pair from which to retrieve
+ *   processed operations.
+ * @param ops
+ *   The address of an array of pointers to
+ *   *rte_comp_op* structures that must be
+ *   large enough to store *nb_ops* pointers in it
+ * @param nb_ops
+ *   The maximum number of operations to dequeue
+ * @return
+ *   - The number of operations actually dequeued, which is the number
+ *   of pointers to *rte_comp_op* structures effectively supplied to the
+ *   *ops* array.
+ */
+typedef uint16_t (*compressdev_dequeue_pkt_burst_t)(void *qp,
+		struct rte_comp_op **ops, uint16_t nb_ops);
+
+/**
+ * Enqueue a burst of operations for processing.
+ *
+ * @param qp
+ *   The queue pair on which operations
+ *   are to be enqueued for processing
+ * @param ops
+ *   The address of an array of *nb_ops* pointers
+ *   to *rte_comp_op* structures which contain
+ *   the operations to be processed
+ * @param nb_ops
+ *   The number of operations to process
+ * @return
+ *   The number of operations actually enqueued on the device. The return
+ *   value can be less than the value of the *nb_ops* parameter when the
+ *   comp devices queue is full or if invalid parameters are specified in
+ *   a *rte_comp_op*.
+ */
+
+typedef uint16_t (*compressdev_enqueue_pkt_burst_t)(void *qp,
+		struct rte_comp_op **ops, uint16_t nb_ops);
+
 
 /** The data structure associated with each comp device. */
 struct rte_compressdev {
+	compressdev_dequeue_pkt_burst_t dequeue_burst;
+	/**< Pointer to PMD receive function */
+	compressdev_enqueue_pkt_burst_t enqueue_burst;
+	/**< Pointer to PMD transmit function */
+
 	struct rte_compressdev_data *data;
 	/**< Pointer to device data */
 	struct rte_compressdev_ops *dev_ops;
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 182a371d8..882395885 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -6,7 +6,9 @@ EXPERIMENTAL {
 	rte_compressdev_close;
 	rte_compressdev_configure;
 	rte_compressdev_count;
+	rte_compressdev_dequeue_burst;
 	rte_compressdev_devices_get;
+	rte_compressdev_enqueue_burst;
 	rte_compressdev_info_get;
 	rte_compressdev_name_get;
 	rte_compressdev_pmd_allocate;
-- 
2.14.3

[dpdk-dev] [PATCH v6 05/14] compressdev: add operation management

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added functions to allocate and free compression operations.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/Makefile                    |   2 +-
 lib/librte_compressdev/meson.build                 |   3 +-
 lib/librte_compressdev/rte_comp.c                  | 176 +++++++++++++++++++++
 lib/librte_compressdev/rte_comp.h                  |  64 ++++++++
 lib/librte_compressdev/rte_compressdev_internal.h  |   1 -
 lib/librte_compressdev/rte_compressdev_version.map |   4 +
 6 files changed, 247 insertions(+), 3 deletions(-)
 create mode 100644 lib/librte_compressdev/rte_comp.c

diff --git a/lib/librte_compressdev/Makefile b/lib/librte_compressdev/Makefile
index 4dae6ca42..7ef89e613 100644
--- a/lib/librte_compressdev/Makefile
+++ b/lib/librte_compressdev/Makefile
@@ -16,7 +16,7 @@ CFLAGS += -DALLOW_EXPERIMENTAL_API
 LDLIBS += -lrte_eal -lrte_mempool -lrte_kvargs
 
 # library source files
-SRCS-y += rte_compressdev.c rte_compressdev_pmd.c
+SRCS-y += rte_compressdev.c rte_compressdev_pmd.c rte_comp.c
 
 # export include files
 SYMLINK-y-include += rte_comp.h
diff --git a/lib/librte_compressdev/meson.build b/lib/librte_compressdev/meson.build
index 2115961c2..5416571c9 100644
--- a/lib/librte_compressdev/meson.build
+++ b/lib/librte_compressdev/meson.build
@@ -3,7 +3,8 @@
 
 allow_experimental_apis = true
 sources = files('rte_compressdev.c',
-	'rte_compressdev_pmd.c')
+	'rte_compressdev_pmd.c',
+	'rte_comp.c')
 headers = files('rte_compressdev.h',
 	'rte_compressdev_pmd.h',
 	'rte_compressdev_internal.h',
diff --git a/lib/librte_compressdev/rte_comp.c b/lib/librte_compressdev/rte_comp.c
new file mode 100644
index 000000000..eb99e99d2
--- /dev/null
+++ b/lib/librte_compressdev/rte_comp.c
@@ -0,0 +1,176 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017-2018 Intel Corporation
+ */
+
+#include "rte_comp.h"
+#include "rte_compressdev.h"
+#include "rte_compressdev_internal.h"
+
+/**
+ * Reset the fields of an operation to their default values.
+ *
+ * @note The private data associated with the operation is not zeroed.
+ *
+ * @param op
+ *   The operation to be reset
+ */
+static inline void
+rte_comp_op_reset(struct rte_comp_op *op)
+{
+	struct rte_mempool *tmp_mp = op->mempool;
+	rte_iova_t tmp_iova_addr = op->iova_addr;
+
+	memset(op, 0, sizeof(struct rte_comp_op));
+	op->status = RTE_COMP_OP_STATUS_NOT_PROCESSED;
+	op->iova_addr = tmp_iova_addr;
+	op->mempool = tmp_mp;
+}
+
+/**
+ * Private data structure belonging to an operation pool.
+ */
+struct rte_comp_op_pool_private {
+	uint16_t user_size;
+	/**< Size of private user data with each operation. */
+};
+
+/**
+ * Bulk allocate raw element from mempool and return as comp operations
+ *
+ * @param mempool
+ *   Compress operation mempool
+ * @param ops
+ *   Array to place allocated operations
+ * @param nb_ops
+ *   Number of operations to allocate
+ * @return
+ *   - 0: Success
+ *   - -ENOENT: Not enough entries in the mempool; no ops are retrieved.
+ */
+static inline int
+rte_comp_op_raw_bulk_alloc(struct rte_mempool *mempool,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	if (rte_mempool_get_bulk(mempool, (void **)ops, nb_ops) == 0)
+		return nb_ops;
+
+	return 0;
+}
+
+/** Initialise rte_comp_op mempool element */
+static void
+rte_comp_op_init(struct rte_mempool *mempool,
+		__rte_unused void *opaque_arg,
+		void *_op_data,
+		__rte_unused unsigned int i)
+{
+	struct rte_comp_op *op = _op_data;
+
+	memset(_op_data, 0, mempool->elt_size);
+
+	op->status = RTE_COMP_OP_STATUS_NOT_PROCESSED;
+	op->iova_addr = rte_mem_virt2iova(_op_data);
+	op->mempool = mempool;
+}
+
+struct rte_mempool * __rte_experimental
+rte_comp_op_pool_create(const char *name,
+		unsigned int nb_elts, unsigned int cache_size,
+		uint16_t user_size, int socket_id)
+{
+	struct rte_comp_op_pool_private *priv;
+
+	unsigned int elt_size = sizeof(struct rte_comp_op) + user_size;
+
+	/* lookup mempool in case already allocated */
+	struct rte_mempool *mp = rte_mempool_lookup(name);
+
+	if (mp != NULL) {
+		priv = (struct rte_comp_op_pool_private *)
+				rte_mempool_get_priv(mp);
+
+		if (mp->elt_size != elt_size ||
+				mp->cache_size < cache_size ||
+				mp->size < nb_elts ||
+				priv->user_size <  user_size) {
+			mp = NULL;
+			COMPRESSDEV_LOG(ERR,
+		"Mempool %s already exists but with incompatible parameters",
+					name);
+			return NULL;
+		}
+		return mp;
+	}
+
+	mp = rte_mempool_create(
+			name,
+			nb_elts,
+			elt_size,
+			cache_size,
+			sizeof(struct rte_comp_op_pool_private),
+			NULL,
+			NULL,
+			rte_comp_op_init,
+			NULL,
+			socket_id,
+			0);
+
+	if (mp == NULL) {
+		COMPRESSDEV_LOG(ERR, "Failed to create mempool %s", name);
+		return NULL;
+	}
+
+	priv = (struct rte_comp_op_pool_private *)
+			rte_mempool_get_priv(mp);
+
+	priv->user_size = user_size;
+
+	return mp;
+}
+
+struct rte_comp_op * __rte_experimental
+rte_comp_op_alloc(struct rte_mempool *mempool)
+{
+	struct rte_comp_op *op = NULL;
+	int retval;
+
+	retval = rte_comp_op_raw_bulk_alloc(mempool, &op, 1);
+	if (unlikely(retval < 0))
+		return NULL;
+
+	rte_comp_op_reset(op);
+
+	return op;
+}
+
+int __rte_experimental
+rte_comp_op_bulk_alloc(struct rte_mempool *mempool,
+		struct rte_comp_op **ops, uint16_t nb_ops)
+{
+	int ret;
+	uint16_t i;
+
+	ret = rte_comp_op_raw_bulk_alloc(mempool, ops, nb_ops);
+	if (unlikely(ret < nb_ops))
+		return ret;
+
+	for (i = 0; i < nb_ops; i++)
+		rte_comp_op_reset(ops[i]);
+
+	return nb_ops;
+}
+
+/**
+ * free operation structure
+ * If operation has been allocate from a rte_mempool, then the operation will
+ * be returned to the mempool.
+ *
+ * @param op
+ *   Compress operation
+ */
+void __rte_experimental
+rte_comp_op_free(struct rte_comp_op *op)
+{
+	if (op != NULL && op->mempool != NULL)
+		rte_mempool_put(op->mempool, op);
+}
diff --git a/lib/librte_compressdev/rte_comp.h b/lib/librte_compressdev/rte_comp.h
index 9f35c37c5..996b17871 100644
--- a/lib/librte_compressdev/rte_comp.h
+++ b/lib/librte_compressdev/rte_comp.h
@@ -309,6 +309,70 @@ struct rte_comp_op {
 	 */
 } __rte_cache_aligned;
 
+/**
+ * Creates an operation pool
+ *
+ * @param name
+ *   Compress pool name
+ * @param nb_elts
+ *   Number of elements in pool
+ * @param cache_size
+ *   Number of elements to cache on lcore, see
+ *   *rte_mempool_create* for further details about cache size
+ * @param user_size
+ *   Size of private data to allocate for user with each operation
+ * @param socket_id
+ *   Socket to identifier allocate memory on
+ * @return
+ *  - On success pointer to mempool
+ *  - On failure NULL
+ */
+struct rte_mempool * __rte_experimental
+rte_comp_op_pool_create(const char *name,
+		unsigned int nb_elts, unsigned int cache_size,
+		uint16_t user_size, int socket_id);
+
+/**
+ * Allocate an operation from a mempool with default parameters set
+ *
+ * @param mempool
+ *   Compress operation mempool
+ *
+ * @return
+ * - On success returns a valid rte_comp_op structure
+ * - On failure returns NULL
+ */
+struct rte_comp_op * __rte_experimental
+rte_comp_op_alloc(struct rte_mempool *mempool);
+
+/**
+ * Bulk allocate operations from a mempool with default parameters set
+ *
+ * @param mempool
+ *   Compress operation mempool
+ * @param ops
+ *   Array to place allocated operations
+ * @param nb_ops
+ *   Number of operations to allocate
+ * @return
+ *   - 0: Success
+ *   - -ENOENT: Not enough entries in the mempool; no ops are retrieved.
+ */
+int __rte_experimental
+rte_comp_op_bulk_alloc(struct rte_mempool *mempool,
+		struct rte_comp_op **ops, uint16_t nb_ops);
+
+/**
+ * Free operation structure
+ * If operation has been allocate from a rte_mempool, then the operation will
+ * be returned to the mempool.
+ *
+ * @param op
+ *   Compress operation
+ */
+void __rte_experimental
+rte_comp_op_free(struct rte_comp_op *op);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_compressdev/rte_compressdev_internal.h b/lib/librte_compressdev/rte_compressdev_internal.h
index e33e75eb2..9ec44f2c1 100644
--- a/lib/librte_compressdev/rte_compressdev_internal.h
+++ b/lib/librte_compressdev/rte_compressdev_internal.h
@@ -63,7 +63,6 @@ typedef uint16_t (*compressdev_dequeue_pkt_burst_t)(void *qp,
 typedef uint16_t (*compressdev_enqueue_pkt_burst_t)(void *qp,
 		struct rte_comp_op **ops, uint16_t nb_ops);
 
-
 /** The data structure associated with each comp device. */
 struct rte_compressdev {
 	compressdev_dequeue_pkt_burst_t dequeue_burst;
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 882395885..292417fde 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -23,6 +23,10 @@ EXPERIMENTAL {
 	rte_compressdev_socket_id;
 	rte_compressdev_start;
 	rte_compressdev_stop;
+	rte_comp_op_alloc;
+	rte_comp_op_bulk_alloc;
+	rte_comp_op_free;
+	rte_comp_op_pool_create;
 
         local: *;
 };
-- 
2.14.3

[dpdk-dev] [PATCH v6 06/14] compressdev: support stateless operations

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added private transform data (priv_xform) in compression
operation, which will contain the private data from each
PMD to support stateless operations.
Also, added functions to create/free this data.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 48 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 48 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 40 ++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  2 +
 4 files changed, 138 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index af0616980..0af1b4927 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -554,6 +554,54 @@ rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
 	dev_info->driver_name = dev->device->driver->name;
 }
 
+int __rte_experimental
+rte_compressdev_private_xform_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **priv_xform)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_get_dev(dev_id);
+
+	if (xform == NULL || priv_xform == NULL || dev == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->private_xform_create, -ENOTSUP);
+	ret = (*dev->dev_ops->private_xform_create)(dev, xform, priv_xform);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to create private_xform: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
+int __rte_experimental
+rte_compressdev_private_xform_free(uint8_t dev_id, void *priv_xform)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_get_dev(dev_id);
+
+	if (dev == NULL || priv_xform == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->private_xform_free, -ENOTSUP);
+	ret = dev->dev_ops->private_xform_free(dev, priv_xform);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to free private xform: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
 const char * __rte_experimental
 rte_compressdev_name_get(uint8_t dev_id)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 9660e5af6..705e9fe79 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -89,6 +89,8 @@ struct rte_compressdev_config {
 	/**< Socket on which to allocate resources */
 	uint16_t nb_queue_pairs;
 	/**< Total number of queue pairs to configure on a device */
+	uint16_t max_nb_priv_xforms;
+	/**< Max number of private_xforms which will be created on the device */
 };
 
 /**
@@ -314,6 +316,52 @@ uint16_t __rte_experimental
 rte_compressdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 		struct rte_comp_op **ops, uint16_t nb_ops);
 
+/**
+ * This should alloc a private_xform from the device's mempool and initialise
+ * it. The application should call this API when setting up for stateless
+ * processing on a device. If it returns non-shareable, then the appl cannot
+ * share this handle with multiple in-flight ops and should call this API again
+ * to get a separate handle for every in-flight op.
+ * The handle returned is only valid for use with ops of op_type STATELESS.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param xform
+ *   xform data
+ * @param private_xform
+ *   Pointer to where PMD's private_xform handle should be stored
+ *
+ * @return
+ *  - if successful returns 0
+ *    and valid private_xform handle
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private_xform could not be allocated.
+ */
+int __rte_experimental
+rte_compressdev_private_xform_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **private_xform);
+
+/**
+ * This should clear the private_xform and return it to the device's mempool.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ *
+ * @param private_xform
+ *   PMD's private_xform data
+ *
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -EBUSY if can't free private_xform due to inflight operations
+ */
+int __rte_experimental
+rte_compressdev_private_xform_free(uint8_t dev_id, void *private_xform);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
index 14ce76b44..45ab61c25 100644
--- a/lib/librte_compressdev/rte_compressdev_pmd.h
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -166,6 +166,41 @@ typedef int (*compressdev_queue_pair_release_t)(struct rte_compressdev *dev,
  */
 typedef uint32_t (*compressdev_queue_pair_count_t)(struct rte_compressdev *dev);
 
+/**
+ * Create driver private_xform data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param xform
+ *   xform data
+ * @param private_xform
+ *   ptr where handle of pmd's private_xform data should be stored
+ * @return
+ *  - if successful returns 0
+ *    and valid private_xform handle
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private_xform could not be allocated.
+ */
+typedef int (*compressdev_private_xform_create_t)(struct rte_compressdev *dev,
+		const struct rte_comp_xform *xform, void **private_xform);
+
+/**
+ * Free driver private_xform data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param private_xform
+ *   handle of pmd's private_xform data
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -EBUSY if can't free private_xform due to inflight operations
+ */
+typedef int (*compressdev_private_xform_free_t)(struct rte_compressdev *dev,
+		void *private_xform);
 
 /** comp device operations function pointer table */
 struct rte_compressdev_ops {
@@ -180,6 +215,11 @@ struct rte_compressdev_ops {
 	/**< Set up a device queue pair. */
 	compressdev_queue_pair_release_t queue_pair_release;
 	/**< Release a queue pair. */
+
+	compressdev_private_xform_create_t private_xform_create;
+	/**< Create a comp private_xform and initialise its private data. */
+	compressdev_private_xform_free_t private_xform_free;
+	/**< Free a comp private_xform's data. */
 };
 
 /**
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 292417fde..d755bb65e 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -18,6 +18,8 @@ EXPERIMENTAL {
 	rte_compressdev_pmd_get_named_dev;
 	rte_compressdev_pmd_parse_input_args;
 	rte_compressdev_pmd_release_device;
+	rte_compressdev_private_xform_create;
+	rte_compressdev_private_xform_free;
 	rte_compressdev_queue_pair_count;
 	rte_compressdev_queue_pair_setup;
 	rte_compressdev_socket_id;
-- 
2.14.3

[dpdk-dev] [PATCH v6 07/14] compressdev: support stateful operations

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added stream data (stream) in compression operation,
which will contain the private data from each PMD
to support stateful operations.
Also, added functions to create/free this data.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 49 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 50 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 41 ++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  2 +
 4 files changed, 142 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 0af1b4927..0ffc43bb9 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -602,6 +602,55 @@ rte_compressdev_private_xform_free(uint8_t dev_id, void *priv_xform)
 	return 0;
 }
 
+int __rte_experimental
+rte_compressdev_stream_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **stream)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_get_dev(dev_id);
+
+	if (xform == NULL || dev == NULL || stream == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->stream_create, -ENOTSUP);
+	ret = (*dev->dev_ops->stream_create)(dev, xform, stream);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to create stream: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
+
+int __rte_experimental
+rte_compressdev_stream_free(uint8_t dev_id, void *stream)
+{
+	struct rte_compressdev *dev;
+	int ret;
+
+	dev = rte_compressdev_get_dev(dev_id);
+
+	if (dev == NULL || stream == NULL)
+		return -EINVAL;
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->stream_free, -ENOTSUP);
+	ret = dev->dev_ops->stream_free(dev, stream);
+	if (ret < 0) {
+		COMPRESSDEV_LOG(ERR,
+			"dev_id %d failed to free stream: err=%d",
+			dev_id, ret);
+		return ret;
+	};
+
+	return 0;
+}
+
 const char * __rte_experimental
 rte_compressdev_name_get(uint8_t dev_id)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 705e9fe79..f18c7ccf6 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -91,6 +91,8 @@ struct rte_compressdev_config {
 	/**< Total number of queue pairs to configure on a device */
 	uint16_t max_nb_priv_xforms;
 	/**< Max number of private_xforms which will be created on the device */
+	uint16_t max_nb_streams;
+	/**< Max number of streams which will be created on the device */
 };
 
 /**
@@ -316,6 +318,54 @@ uint16_t __rte_experimental
 rte_compressdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 		struct rte_comp_op **ops, uint16_t nb_ops);
 
+/**
+ * This should alloc a stream from the device's mempool and initialise it.
+ * The application should call this API when setting up for the stateful
+ * processing of a set of data on a device. The API can be called multiple
+ * times to set up a stream for each data set. The handle returned is only for
+ * use with ops of op_type STATEFUL and must be passed to the PMD
+ * with every op in the data stream
+ *
+ * @param dev_id
+ *   Compress device identifier
+ * @param xform
+ *   xform data
+ * @param stream
+ *   Pointer to where PMD's private stream handle should be stored
+ *
+ * @return
+ *  - 0 if successful and valid stream handle
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private stream could not be allocated.
+ *
+ */
+int __rte_experimental
+rte_compressdev_stream_create(uint8_t dev_id,
+		const struct rte_comp_xform *xform,
+		void **stream);
+
+/**
+ * This should clear the stream and return it to the device's mempool.
+ *
+ * @param dev_id
+ *   Compress device identifier
+ *
+ * @param stream
+ *   PMD's private stream data
+ *
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -EBUSY if can't free stream as there are inflight operations
+ */
+int __rte_experimental
+rte_compressdev_stream_free(uint8_t dev_id, void *stream);
+
 /**
  * This should alloc a private_xform from the device's mempool and initialise
  * it. The application should call this API when setting up for stateless
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
index 45ab61c25..c08a3ee63 100644
--- a/lib/librte_compressdev/rte_compressdev_pmd.h
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -166,6 +166,42 @@ typedef int (*compressdev_queue_pair_release_t)(struct rte_compressdev *dev,
  */
 typedef uint32_t (*compressdev_queue_pair_count_t)(struct rte_compressdev *dev);
 
+/**
+ * Create driver private stream data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param xform
+ *   xform data
+ * @param stream
+ *   ptr where handle of pmd's private stream data should be stored
+ * @return
+ *  - Returns 0 if private stream structure has been created successfully.
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -ENOTSUP if comp device does not support the comp transform.
+ *  - Returns -ENOMEM if the private stream could not be allocated.
+ */
+typedef int (*compressdev_stream_create_t)(struct rte_compressdev *dev,
+		const struct rte_comp_xform *xform, void **stream);
+
+/**
+ * Free driver private stream data.
+ *
+ * @param dev
+ *   Compressdev device
+ * @param stream
+ *   handle of pmd's private stream data
+ * @return
+ *  - 0 if successful
+ *  - <0 in error cases
+ *  - Returns -EINVAL if input parameters are invalid.
+ *  - Returns -ENOTSUP if comp device does not support STATEFUL operations.
+ *  - Returns -EBUSY if can't free stream as there are inflight operations
+ */
+typedef int (*compressdev_stream_free_t)(struct rte_compressdev *dev,
+		void *stream);
+
 /**
  * Create driver private_xform data.
  *
@@ -216,6 +252,11 @@ struct rte_compressdev_ops {
 	compressdev_queue_pair_release_t queue_pair_release;
 	/**< Release a queue pair. */
 
+	compressdev_stream_create_t stream_create;
+	/**< Create a comp stream and initialise its private data. */
+	compressdev_stream_free_t stream_free;
+	/**< Free a comp stream's private data. */
+
 	compressdev_private_xform_create_t private_xform_create;
 	/**< Create a comp private_xform and initialise its private data. */
 	compressdev_private_xform_free_t private_xform_free;
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index d755bb65e..ca4d9d3d8 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -25,6 +25,8 @@ EXPERIMENTAL {
 	rte_compressdev_socket_id;
 	rte_compressdev_start;
 	rte_compressdev_stop;
+	rte_compressdev_stream_create;
+	rte_compressdev_stream_free;
 	rte_comp_op_alloc;
 	rte_comp_op_bulk_alloc;
 	rte_comp_op_free;
-- 
2.14.3

[dpdk-dev] [PATCH v6 08/14] compressdev: support hash operations

[Pablo de Lara]

From: Shally Verma <shally.verma@caviumnetworks.com>

- Added hash algo enumeration and params in xform and rte_comp_op
- Updated compress/decompress xform to input hash algorithm
- Updated struct rte_comp_op to input hash buffer

User in capability query will know about support hashes via
device info comp_feature_flag. If supported, application can initialize
desired algorithm enumeration in xform structure and pass valid hash
buffer during enqueue_burst().

Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Sunila Sahu <sunila.sahu@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_comp.h | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/lib/librte_compressdev/rte_comp.h b/lib/librte_compressdev/rte_comp.h
index 996b17871..6df1779af 100644
--- a/lib/librte_compressdev/rte_comp.h
+++ b/lib/librte_compressdev/rte_comp.h
@@ -63,6 +63,17 @@ enum rte_comp_algorithm {
 	RTE_COMP_ALGO_LIST_END
 };
 
+/** Compression Hash Algorithms */
+enum rte_comp_hash_algorithm {
+	RTE_COMP_HASH_ALGO_NONE = 0,
+	/**< No hash */
+	RTE_COMP_HASH_ALGO_SHA1,
+	/**< SHA1 hash algorithm */
+	RTE_COMP_HASH_ALGO_SHA2_256,
+	/**< SHA256 hash algorithm of SHA2 family */
+	RTE_COMP_HASH_ALGO_LIST_END
+};
+
 /**< Compression Level.
  * The number is interpreted by each PMD differently. However, lower numbers
  * give fastest compression, at the expense of compression ratio while
@@ -172,6 +183,10 @@ struct rte_comp_compress_xform {
 	 */
 	enum rte_comp_checksum_type chksum;
 	/**< Type of checksum to generate on the uncompressed data */
+	enum rte_comp_hash_algorithm hash_algo;
+	/**< Hash algorithm to be used with compress operation. Hash is always
+	 * done on plaintext.
+	 */
 };
 
 /**
@@ -187,6 +202,10 @@ struct rte_comp_decompress_xform {
 	 * compressed data. If window size can't be supported by the PMD then
 	 * setup of stream or private_xform should fail.
 	 */
+	enum rte_comp_hash_algorithm hash_algo;
+	/**< Hash algorithm to be used with decompress operation. Hash is always
+	 * done on plaintext.
+	 */
 };
 
 /**
@@ -272,6 +291,19 @@ struct rte_comp_op {
 		 * decompress direction.
 		 */
 	} dst;
+	struct {
+		uint8_t *digest;
+		/**< Output buffer to store hash output, if enabled in xform.
+		 * Buffer would contain valid value only after an op with
+		 * flush flag = RTE_COMP_FLUSH_FULL/FLUSH_FINAL is processed
+		 * successfully.
+		 *
+		 * Length of buffer should be contiguous and large enough to
+		 * accommodate digest produced by specific hash algo.
+		 */
+		rte_iova_t iova_addr;
+		/**< IO address of the buffer */
+	} hash;
 	enum rte_comp_flush_flag flush_flag;
 	/**< Defines flush characteristics for the output data.
 	 * Only applicable in compress direction
-- 
2.14.3

[dpdk-dev] [PATCH v6 09/14] compressdev: add device feature flags

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 21 ++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 33 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  1 +
 3 files changed, 55 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 0ffc43bb9..03f1ab8a7 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -31,6 +31,27 @@ static struct rte_compressdev_global compressdev_globals = {
 
 struct rte_compressdev_global *rte_compressdev_globals = &compressdev_globals;
 
+const char * __rte_experimental
+rte_compressdev_get_feature_name(uint64_t flag)
+{
+	switch (flag) {
+	case RTE_COMPDEV_FF_HW_ACCELERATED:
+		return "HW_ACCELERATED";
+	case RTE_COMPDEV_FF_CPU_SSE:
+		return "CPU_SSE";
+	case RTE_COMPDEV_FF_CPU_AVX:
+		return "CPU_AVX";
+	case RTE_COMPDEV_FF_CPU_AVX2:
+		return "CPU_AVX2";
+	case RTE_COMPDEV_FF_CPU_AVX512:
+		return "CPU_AVX512";
+	case RTE_COMPDEV_FF_CPU_NEON:
+		return "CPU_NEON";
+	default:
+		return NULL;
+	}
+}
+
 static struct rte_compressdev *
 rte_compressdev_get_dev(uint8_t dev_id)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index f18c7ccf6..30cf64c6e 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -21,9 +21,42 @@ extern "C" {
 
 #include "rte_comp.h"
 
+/**
+ * compression device supported feature flags
+ *
+ * @note New features flags should be added to the end of the list
+ *
+ * Keep these flags synchronised with rte_compressdev_get_feature_name()
+ */
+#define	RTE_COMPDEV_FF_HW_ACCELERATED		(1ULL << 0)
+/**< Operations are off-loaded to an external hardware accelerator */
+#define	RTE_COMPDEV_FF_CPU_SSE			(1ULL << 1)
+/**< Utilises CPU SIMD SSE instructions */
+#define	RTE_COMPDEV_FF_CPU_AVX			(1ULL << 2)
+/**< Utilises CPU SIMD AVX instructions */
+#define	RTE_COMPDEV_FF_CPU_AVX2			(1ULL << 3)
+/**< Utilises CPU SIMD AVX2 instructions */
+#define	RTE_COMPDEV_FF_CPU_AVX512		(1ULL << 4)
+/**< Utilises CPU SIMD AVX512 instructions */
+#define	RTE_COMPDEV_FF_CPU_NEON			(1ULL << 5)
+/**< Utilises CPU NEON instructions */
+
+/**
+ * Get the name of a compress device feature flag.
+ *
+ * @param flag
+ *   The mask describing the flag
+ *
+ * @return
+ *   The name of this flag, or NULL if it's not a valid feature flag.
+ */
+const char * __rte_experimental
+rte_compressdev_get_feature_name(uint64_t flag);
+
 /**  comp device information */
 struct rte_compressdev_info {
 	const char *driver_name;		/**< Driver name. */
+	uint64_t feature_flags;			/**< Feature flags */
 	uint16_t max_nb_queue_pairs;
 	/**< Maximum number of queues pairs supported by device.
 	 * (If 0, there is no limit in maximum number of queue pairs)
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index ca4d9d3d8..f9223bcfe 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -9,6 +9,7 @@ EXPERIMENTAL {
 	rte_compressdev_dequeue_burst;
 	rte_compressdev_devices_get;
 	rte_compressdev_enqueue_burst;
+	rte_compressdev_get_feature_name;
 	rte_compressdev_info_get;
 	rte_compressdev_name_get;
 	rte_compressdev_pmd_allocate;
-- 
2.14.3

[dpdk-dev] [PATCH v6 10/14] compressdev: add compression service feature flags

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_comp.c                  | 31 +++++++++++++++
 lib/librte_compressdev/rte_comp.h                  | 46 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  1 +
 3 files changed, 78 insertions(+)

diff --git a/lib/librte_compressdev/rte_comp.c b/lib/librte_compressdev/rte_comp.c
index eb99e99d2..d596ba872 100644
--- a/lib/librte_compressdev/rte_comp.c
+++ b/lib/librte_compressdev/rte_comp.c
@@ -6,6 +6,37 @@
 #include "rte_compressdev.h"
 #include "rte_compressdev_internal.h"
 
+const char * __rte_experimental
+rte_comp_get_feature_name(uint64_t flag)
+{
+	switch (flag) {
+	case RTE_COMP_FF_STATEFUL_COMPRESSION:
+		return "STATEFUL_COMPRESSION";
+	case RTE_COMP_FF_STATEFUL_DECOMPRESSION:
+		return "STATEFUL_DECOMPRESSION";
+	case RTE_COMP_FF_MBUF_SCATTER_GATHER:
+		return "MBUF_SCATTER_GATHER";
+	case RTE_COMP_FF_MULTI_PKT_CHECKSUM:
+		return "MULTI_PKT_CHECKSUM";
+	case RTE_COMP_FF_ADLER32_CHECKSUM:
+		return "ADLER32_CHECKSUM";
+	case RTE_COMP_FF_CRC32_CHECKSUM:
+		return "CRC32_CHECKSUM";
+	case RTE_COMP_FF_CRC32_ADLER32_CHECKSUM:
+		return "CRC32_ADLER32_CHECKSUM";
+	case RTE_COMP_FF_NONCOMPRESSED_BLOCKS:
+		return "NONCOMPRESSED_BLOCKS";
+	case RTE_COMP_FF_SHA1_HASH:
+		return "SHA1_HASH";
+	case RTE_COMP_FF_SHA2_SHA256_HASH:
+		return "SHA2_SHA256_HASH";
+	case RTE_COMP_FF_SHAREABLE_PRIV_XFORM:
+		return "SHAREABLE_PRIV_XFORM";
+	default:
+		return NULL;
+	}
+}
+
 /**
  * Reset the fields of an operation to their default values.
  *
diff --git a/lib/librte_compressdev/rte_comp.h b/lib/librte_compressdev/rte_comp.h
index 6df1779af..4c802195f 100644
--- a/lib/librte_compressdev/rte_comp.h
+++ b/lib/librte_compressdev/rte_comp.h
@@ -19,6 +19,40 @@ extern "C" {
 #include <rte_mempool.h>
 #include <rte_mbuf.h>
 
+/**
+ * compression service feature flags
+ *
+ * @note New features flags should be added to the end of the list
+ *
+ * Keep these flags synchronised with rte_comp_get_feature_name()
+ */
+#define RTE_COMP_FF_STATEFUL_COMPRESSION	(1ULL << 0)
+/**< Stateful compression is supported */
+#define RTE_COMP_FF_STATEFUL_DECOMPRESSION	(1ULL << 1)
+/**< Stateful decompression is supported */
+#define	RTE_COMP_FF_MBUF_SCATTER_GATHER		(1ULL << 2)
+/**< Scatter-gather mbufs are supported */
+#define RTE_COMP_FF_ADLER32_CHECKSUM		(1ULL << 3)
+/**< Adler-32 Checksum is supported */
+#define RTE_COMP_FF_CRC32_CHECKSUM		(1ULL << 4)
+/**< CRC32 Checksum is supported */
+#define RTE_COMP_FF_CRC32_ADLER32_CHECKSUM	(1ULL << 5)
+/**< Adler-32/CRC32 Checksum is supported */
+#define RTE_COMP_FF_MULTI_PKT_CHECKSUM		(1ULL << 6)
+/**< Generation of checksum across multiple stateless packets is supported */
+#define RTE_COMP_FF_SHA1_HASH			(1ULL << 7)
+/**< SHA1 Hash is supported */
+#define RTE_COMP_FF_SHA2_SHA256_HASH		(1ULL << 8)
+/**< SHA256 Hash of SHA2 family is supported */
+#define RTE_COMP_FF_NONCOMPRESSED_BLOCKS	(1ULL << 9)
+/**< Creation of non-compressed blocks using RTE_COMP_LEVEL_NONE is supported */
+#define RTE_COMP_FF_SHAREABLE_PRIV_XFORM	(1ULL << 10)
+/**< Private xforms created by the PMD can be shared
+ * across multiple stateless operations. If not set, then app needs
+ * to create as many priv_xforms as it expects to have stateless
+ * operations in-flight.
+ */
+
 /** Status of comp operation */
 enum rte_comp_op_status {
 	RTE_COMP_OP_STATUS_SUCCESS = 0,
@@ -405,6 +439,18 @@ rte_comp_op_bulk_alloc(struct rte_mempool *mempool,
 void __rte_experimental
 rte_comp_op_free(struct rte_comp_op *op);
 
+/**
+ * Get the name of a compress service feature flag
+ *
+ * @param flag
+ *   The mask describing the flag
+ *
+ * @return
+ *   The name of this flag, or NULL if it's not a valid feature flag.
+ */
+const char * __rte_experimental
+rte_comp_get_feature_name(uint64_t flag);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index f9223bcfe..5ed3080b4 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -28,6 +28,7 @@ EXPERIMENTAL {
 	rte_compressdev_stop;
 	rte_compressdev_stream_create;
 	rte_compressdev_stream_free;
+	rte_comp_get_feature_name;
 	rte_comp_op_alloc;
 	rte_comp_op_bulk_alloc;
 	rte_comp_op_free;
-- 
2.14.3

[dpdk-dev] [PATCH v6 11/14] compressdev: add device stats

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 40 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 39 +++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.h       | 26 ++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  2 ++
 4 files changed, 107 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 03f1ab8a7..855385e10 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -555,6 +555,46 @@ rte_compressdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
 			dev->data->queue_pairs[qp_id], ops, nb_ops);
 }
 
+int __rte_experimental
+rte_compressdev_stats_get(uint8_t dev_id, struct rte_compressdev_stats *stats)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%d", dev_id);
+		return -ENODEV;
+	}
+
+	if (stats == NULL) {
+		COMPRESSDEV_LOG(ERR, "Invalid stats ptr");
+		return -EINVAL;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+	memset(stats, 0, sizeof(*stats));
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->stats_get, -ENOTSUP);
+	(*dev->dev_ops->stats_get)(dev, stats);
+	return 0;
+}
+
+void __rte_experimental
+rte_compressdev_stats_reset(uint8_t dev_id)
+{
+	struct rte_compressdev *dev;
+
+	if (!rte_compressdev_is_valid_dev(dev_id)) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%" PRIu8, dev_id);
+		return;
+	}
+
+	dev = &rte_comp_devices[dev_id];
+
+	RTE_FUNC_PTR_OR_RET(*dev->dev_ops->stats_reset);
+	(*dev->dev_ops->stats_reset)(dev);
+}
+
+
 void __rte_experimental
 rte_compressdev_info_get(uint8_t dev_id, struct rte_compressdev_info *dev_info)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 30cf64c6e..5ad926976 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -63,6 +63,19 @@ struct rte_compressdev_info {
 	 */
 };
 
+/** comp device statistics */
+struct rte_compressdev_stats {
+	uint64_t enqueued_count;
+	/**< Count of all operations enqueued */
+	uint64_t dequeued_count;
+	/**< Count of all operations dequeued */
+
+	uint64_t enqueue_err_count;
+	/**< Total error count on operations enqueued */
+	uint64_t dequeue_err_count;
+	/**< Total error count on operations dequeued */
+};
+
 /**
  * Get the compress device name given a device identifier.
  *
@@ -229,6 +242,32 @@ rte_compressdev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
 uint16_t __rte_experimental
 rte_compressdev_queue_pair_count(uint8_t dev_id);
 
+
+/**
+ * Retrieve the general I/O statistics of a device.
+ *
+ * @param dev_id
+ *   The identifier of the device
+ * @param stats
+ *   A pointer to a structure of type
+ *   *rte_compressdev_stats* to be filled with the
+ *   values of device counters
+ * @return
+ *   - Zero if successful.
+ *   - Non-zero otherwise.
+ */
+int __rte_experimental
+rte_compressdev_stats_get(uint8_t dev_id, struct rte_compressdev_stats *stats);
+
+/**
+ * Reset the general I/O statistics of a device.
+ *
+ * @param dev_id
+ *   The identifier of the device.
+ */
+void __rte_experimental
+rte_compressdev_stats_reset(uint8_t dev_id);
+
 /**
  * Retrieve the contextual information of a device.
  *
diff --git a/lib/librte_compressdev/rte_compressdev_pmd.h b/lib/librte_compressdev/rte_compressdev_pmd.h
index c08a3ee63..38e9ea02b 100644
--- a/lib/librte_compressdev/rte_compressdev_pmd.h
+++ b/lib/librte_compressdev/rte_compressdev_pmd.h
@@ -116,6 +116,27 @@ typedef void (*compressdev_stop_t)(struct rte_compressdev *dev);
 typedef int (*compressdev_close_t)(struct rte_compressdev *dev);
 
 
+/**
+ * Function used to get statistics of a device.
+ *
+ * @param dev
+ *   Compress device
+ * @param stats
+ *   Compress device stats to populate
+ */
+typedef void (*compressdev_stats_get_t)(struct rte_compressdev *dev,
+				struct rte_compressdev_stats *stats);
+
+
+/**
+ * Function used to reset statistics of a device.
+ *
+ * @param dev
+ *   Compress device
+ */
+typedef void (*compressdev_stats_reset_t)(struct rte_compressdev *dev);
+
+
 /**
  * Function used to get specific information of a device.
  *
@@ -247,6 +268,11 @@ struct rte_compressdev_ops {
 
 	compressdev_info_get_t dev_infos_get;	/**< Get device info. */
 
+	compressdev_stats_get_t stats_get;
+	/**< Get device statistics. */
+	compressdev_stats_reset_t stats_reset;
+	/**< Reset device statistics. */
+
 	compressdev_queue_pair_setup_t queue_pair_setup;
 	/**< Set up a device queue pair. */
 	compressdev_queue_pair_release_t queue_pair_release;
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 5ed3080b4..8e4004631 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -25,6 +25,8 @@ EXPERIMENTAL {
 	rte_compressdev_queue_pair_setup;
 	rte_compressdev_socket_id;
 	rte_compressdev_start;
+	rte_compressdev_stats_get;
+	rte_compressdev_stats_reset;
 	rte_compressdev_stop;
 	rte_compressdev_stream_create;
 	rte_compressdev_stream_free;
-- 
2.14.3

[dpdk-dev] [PATCH v6 12/14] compressdev: add device capabilities

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added structure which each PMD will fill out,
providing the capabilities of each driver
(containing mainly which compression services
it supports).

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 23 ++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 35 ++++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  1 +
 3 files changed, 59 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 855385e10..2d7127ce2 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -31,6 +31,29 @@ static struct rte_compressdev_global compressdev_globals = {
 
 struct rte_compressdev_global *rte_compressdev_globals = &compressdev_globals;
 
+const struct rte_compressdev_capabilities * __rte_experimental
+rte_compressdev_capability_get(uint8_t dev_id,
+			enum rte_comp_algorithm algo)
+{
+	const struct rte_compressdev_capabilities *capability;
+	struct rte_compressdev_info dev_info;
+	int i = 0;
+
+	if (dev_id >= compressdev_globals.nb_devs) {
+		COMPRESSDEV_LOG(ERR, "Invalid dev_id=%d", dev_id);
+		return NULL;
+	}
+	rte_compressdev_info_get(dev_id, &dev_info);
+
+	while ((capability = &dev_info.capabilities[i++])->algo !=
+			RTE_COMP_ALGO_UNSPECIFIED){
+		if (capability->algo == algo)
+			return capability;
+	}
+
+	return NULL;
+}
+
 const char * __rte_experimental
 rte_compressdev_get_feature_name(uint64_t flag)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index 5ad926976..d52103e15 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -21,6 +21,39 @@ extern "C" {
 
 #include "rte_comp.h"
 
+/**
+ * Parameter log base 2 range description.
+ * Final value will be 2^value.
+ */
+struct rte_param_log2_range {
+	uint8_t min;	/**< Minimum log2 value */
+	uint8_t max;	/**< Maximum log2 value */
+	uint8_t increment;
+	/**< If a range of sizes are supported,
+	 * this parameter is used to indicate
+	 * increments in base 2 log byte value
+	 * that are supported between the minimum and maximum
+	 */
+};
+
+/** Structure used to capture a capability of a comp device */
+struct rte_compressdev_capabilities {
+	enum rte_comp_algorithm algo;
+	/* Compression algorithm */
+	uint64_t comp_feature_flags;
+	/**< Bitmask of flags for compression service features */
+	struct rte_param_log2_range window_size;
+	/**< Window size range in base two log byte values */
+};
+
+/** Macro used at end of comp PMD list */
+#define RTE_COMP_END_OF_CAPABILITIES_LIST() \
+	{ RTE_COMP_ALGO_UNSPECIFIED }
+
+const struct rte_compressdev_capabilities * __rte_experimental
+rte_compressdev_capability_get(uint8_t dev_id,
+			enum rte_comp_algorithm algo);
+
 /**
  * compression device supported feature flags
  *
@@ -57,6 +90,8 @@ rte_compressdev_get_feature_name(uint64_t flag);
 struct rte_compressdev_info {
 	const char *driver_name;		/**< Driver name. */
 	uint64_t feature_flags;			/**< Feature flags */
+	const struct rte_compressdev_capabilities *capabilities;
+	/**< Array of devices supported capabilities */
 	uint16_t max_nb_queue_pairs;
 	/**< Maximum number of queues pairs supported by device.
 	 * (If 0, there is no limit in maximum number of queue pairs)
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 8e4004631..6c553341a 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -3,6 +3,7 @@ EXPERIMENTAL {
 
 	rte_compressdev_callback_register;
 	rte_compressdev_callback_unregister;
+	rte_compressdev_capability_get;
 	rte_compressdev_close;
 	rte_compressdev_configure;
 	rte_compressdev_count;
-- 
2.14.3

[dpdk-dev] [PATCH v6 13/14] compressdev: get device id from name

[Pablo de Lara]

From: Fiona Trahe <fiona.trahe@intel.com>

Added API to retrieve the device id provided the device name.

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 lib/librte_compressdev/rte_compressdev.c           | 18 ++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           | 13 +++++++++++++
 lib/librte_compressdev/rte_compressdev_version.map |  1 +
 3 files changed, 32 insertions(+)

diff --git a/lib/librte_compressdev/rte_compressdev.c b/lib/librte_compressdev/rte_compressdev.c
index 2d7127ce2..6a38917dd 100644
--- a/lib/librte_compressdev/rte_compressdev.c
+++ b/lib/librte_compressdev/rte_compressdev.c
@@ -117,6 +117,24 @@ rte_compressdev_is_valid_dev(uint8_t dev_id)
 }
 
 
+int __rte_experimental
+rte_compressdev_get_dev_id(const char *name)
+{
+	unsigned int i;
+
+	if (name == NULL)
+		return -1;
+
+	for (i = 0; i < rte_compressdev_globals->nb_devs; i++)
+		if ((strcmp(rte_compressdev_globals->devs[i].data->name, name)
+				== 0) &&
+				(rte_compressdev_globals->devs[i].attached ==
+						RTE_COMPRESSDEV_ATTACHED))
+			return i;
+
+	return -1;
+}
+
 uint8_t __rte_experimental
 rte_compressdev_count(void)
 {
diff --git a/lib/librte_compressdev/rte_compressdev.h b/lib/librte_compressdev/rte_compressdev.h
index d52103e15..c86035f14 100644
--- a/lib/librte_compressdev/rte_compressdev.h
+++ b/lib/librte_compressdev/rte_compressdev.h
@@ -111,6 +111,19 @@ struct rte_compressdev_stats {
 	/**< Total error count on operations dequeued */
 };
 
+
+/**
+ * Get the device identifier for the named compress device.
+ *
+ * @param name
+ *   Device name to select the device structure
+ * @return
+ *   - Returns compress device identifier on success.
+ *   - Return -1 on failure to find named compress device.
+ */
+int __rte_experimental
+rte_compressdev_get_dev_id(const char *name);
+
 /**
  * Get the compress device name given a device identifier.
  *
diff --git a/lib/librte_compressdev/rte_compressdev_version.map b/lib/librte_compressdev/rte_compressdev_version.map
index 6c553341a..e5cbc569e 100644
--- a/lib/librte_compressdev/rte_compressdev_version.map
+++ b/lib/librte_compressdev/rte_compressdev_version.map
@@ -10,6 +10,7 @@ EXPERIMENTAL {
 	rte_compressdev_dequeue_burst;
 	rte_compressdev_devices_get;
 	rte_compressdev_enqueue_burst;
+	rte_compressdev_get_dev_id;
 	rte_compressdev_get_feature_name;
 	rte_compressdev_info_get;
 	rte_compressdev_name_get;
-- 
2.14.3

[dpdk-dev] [PATCH v6 14/14] doc: add compressdev library guide

[Pablo de Lara]

From: Ashish Gupta <Ashish.Gupta@caviumnetworks.com>

Add section in programmer’s guide for Compressdev library.

Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
Signed-off-by: Sunila Sahu <sunila.sahu@caviumnetworks.com>
Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>
---
 MAINTAINERS                                       |   1 +
 doc/guides/prog_guide/compressdev_lib.rst         | 623 ++++++++++++++++++++++
 doc/guides/prog_guide/img/stateful-op.svg         | 116 ++++
 doc/guides/prog_guide/img/stateless-op-shared.svg | 124 +++++
 doc/guides/prog_guide/img/stateless-op.svg        | 140 +++++
 doc/guides/prog_guide/index.rst                   |   1 +
 6 files changed, 1005 insertions(+)
 create mode 100644 doc/guides/prog_guide/compressdev_lib.rst
 create mode 100644 doc/guides/prog_guide/img/stateful-op.svg
 create mode 100644 doc/guides/prog_guide/img/stateless-op-shared.svg
 create mode 100644 doc/guides/prog_guide/img/stateless-op.svg

diff --git a/MAINTAINERS b/MAINTAINERS
index c1370258f..8de296866 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -369,6 +369,7 @@ M: Pablo de Lara <pablo.de.lara.guarch@intel.com>
 M: Ashish Gupta <ashish.gupta@caviumnetworks.com>
 T: git://dpdk.org/next/dpdk-next-crypto
 F: lib/librte_compressdev/
+F: doc/guides/prog_guide/compressdev_lib.rst
 
 
 Memory Pool Drivers
diff --git a/doc/guides/prog_guide/compressdev_lib.rst b/doc/guides/prog_guide/compressdev_lib.rst
new file mode 100644
index 000000000..87e264906
--- /dev/null
+++ b/doc/guides/prog_guide/compressdev_lib.rst
@@ -0,0 +1,623 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2017-2018 Cavium Networks.
+
+Compression Device Library
+===========================
+
+The compression framework provides a generic set of APIs to perform compression services
+as well as to query and configure compression devices both physical(hardware) and virtual(software)
+to perform those services. The framework currently only supports lossless compression schemes:
+Deflate and LZS.
+
+Device Management
+-----------------
+
+Device Creation
+~~~~~~~~~~~~~~~
+
+Physical compression devices are discovered during the bus probe of the EAL function
+which is executed at DPDK initialization, based on their unique device identifier.
+For eg. PCI devices can be identified using PCI BDF (bus/bridge, device, function).
+Specific physical compression devices, like other physical devices in DPDK can be
+white-listed or black-listed using the EAL command line options.
+
+Virtual devices can be created by two mechanisms, either using the EAL command
+line options or from within the application using an EAL API directly.
+
+From the command line using the --vdev EAL option
+
+.. code-block:: console
+
+   --vdev  '<pmd name>,socket_id=0'
+
+.. Note::
+
+   * If DPDK application requires multiple software compression PMD devices then required
+     number of ``--vdev`` with appropriate libraries are to be added.
+
+   * An Application with multiple compression device instances exposed by the same PMD must
+     specify a unique name for each device.
+
+   Example: ``--vdev  'pmd0' --vdev  'pmd1'``
+
+Or, by using the rte_vdev_init API within the application code.
+
+.. code-block:: c
+
+   rte_vdev_init("<pmd_name>","socket_id=0")
+
+All virtual compression devices support the following initialization parameters:
+
+* ``socket_id`` - socket on which to allocate the device resources on.
+
+Device Identification
+~~~~~~~~~~~~~~~~~~~~~
+
+Each device, whether virtual or physical is uniquely designated by two
+identifiers:
+
+- A unique device index used to designate the compression device in all functions
+  exported by the compressdev API.
+
+- A device name used to designate the compression device in console messages, for
+  administration or debugging purposes.
+
+Device Configuration
+~~~~~~~~~~~~~~~~~~~~
+
+The configuration of each compression device includes the following operations:
+
+- Allocation of resources, including hardware resources if a physical device.
+- Resetting the device into a well-known default state.
+- Initialization of statistics counters.
+
+The ``rte_compressdev_configure`` API is used to configure a compression device.
+
+The ``rte_compressdev_config`` structure is used to pass the configuration
+parameters.
+
+See *DPDK API Reference* for details.
+
+Configuration of Queue Pairs
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each compression device queue pair is individually configured through the
+``rte_compressdev_queue_pair_setup`` API.
+
+The ``max_inflight_ops`` is used to pass maximum number of
+rte_comp_op that could be present in a queue at-a-time.
+PMD then can allocate resources accordingly on a specified socket.
+
+See *DPDK API Reference* for details.
+
+Logical Cores, Memory and Queues Pair Relationships
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Library supports NUMA similarly as described in Cryptodev library section.
+
+A queue pair cannot be shared and should be exclusively used by a single processing
+context for enqueuing operations or dequeuing operations on the same compression device
+since sharing would require global locks and hinder performance. It is however possible
+to use a different logical core to dequeue an operation on a queue pair from the logical
+core on which it was enqueued. This means that a compression burst enqueue/dequeue
+APIs are a logical place to transition from one logical core to another in a
+data processing pipeline.
+
+Device Features and Capabilities
+---------------------------------
+
+Compression devices define their functionality through two mechanisms, global device
+features and algorithm features. Global devices features identify device
+wide level features which are applicable to the whole device such as supported hardware
+acceleration and CPU features. List of compression device features can be seen in the
+RTE_COMPDEV_FF_XXX macros.
+
+The algorithm features lists individual algo feature which device supports per-algorithm,
+such as a stateful compression/decompression, checksums operation etc. List of algorithm
+features can be seen in the RTE_COMP_FF_XXX macros.
+
+Capabilities
+~~~~~~~~~~~~
+Each PMD has a list of capabilities, including algorithms listed in
+enum ``rte_comp_algorithm`` and its associated feature flag and
+sliding window range in log base 2 value. Sliding window tells
+the minimum and maximum size of lookup window that algorithm uses
+to find duplicates.
+
+See *DPDK API Reference* for details.
+
+Each Compression poll mode driver defines its array of capabilities
+for each algorithm it supports. See PMD implementation for capability
+initialization.
+
+Capabilities Discovery
+~~~~~~~~~~~~~~~~~~~~~~
+
+PMD capability and features are discovered via ``rte_compressdev_info_get`` function.
+
+The ``rte_compressdev_info`` structure contains all the relevant information for the device.
+
+See *DPDK API Reference* for details.
+
+Compression Operation
+----------------------
+
+DPDK compression supports two types of compression methodologies:
+
+- Stateless, data associated to a compression operation is compressed without any reference
+  to another compression operation.
+
+- Stateful, data in each compression operation is compressed with reference to previous compression
+  operations in the same data stream i.e. history of data is maintained between the operations.
+
+For more explanation, please refer RFC https://www.ietf.org/rfc/rfc1951.txt
+
+Operation Representation
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Compression operation is described via ``struct rte_comp_op``, which contains both input and
+output data. The operation structure includes the operation type (stateless or stateful),
+the operation status and the priv_xform/stream handle, source, destination and checksum buffer
+pointers. It also contains the source mempool from which the operation is allocated.
+PMD updates consumed field with amount of data read from source buffer and produced
+field with amount of data of written into destination buffer along with status of
+operation. See section *Produced, Consumed And Operation Status* for more details.
+
+Compression operations mempool also has an ability to allocate private memory with the
+operation for application's purposes. Application software is responsible for specifying
+all the operation specific fields in the ``rte_comp_op`` structure which are then used
+by the compression PMD to process the requested operation.
+
+
+Operation Management and Allocation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The compressdev library provides an API set for managing compression operations which
+utilize the Mempool Library to allocate operation buffers. Therefore, it ensures
+that the compression operation is interleaved optimally across the channels and
+ranks for optimal processing.
+
+A ``rte_comp_op`` contains a field indicating the pool it originated from.
+
+``rte_comp_op_alloc()`` and ``rte_comp_op_bulk_alloc()`` are used to allocate
+compression operations from a given compression operation mempool.
+The operation gets reset before being returned to a user so that operation
+is always in a good known state before use by the application.
+
+``rte_comp_op_free()`` is called by the application to return an operation to
+its allocating pool.
+
+See *DPDK API Reference* for details.
+
+Passing source data as mbuf-chain
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+If input data is scattered across several different buffers, then
+Application can either parse through all such buffers and make one
+mbuf-chain and enqueue it for processing or, alternatively, it can
+make multiple sequential enqueue_burst() calls for each of them
+processing them statefully. See *Compression API Stateful Operation*
+for stateful processing of ops.
+
+Operation Status
+~~~~~~~~~~~~~~~~
+Each operation carries a status information updated by PMD after it is processed.
+following are currently supported status:
+
+- RTE_COMP_OP_STATUS_SUCCESS,
+    Operation is successfully completed
+
+- RTE_COMP_OP_STATUS_NOT_PROCESSED,
+    Operation has not yet been processed by the device
+
+- RTE_COMP_OP_STATUS_INVALID_ARGS,
+    Operation failed due to invalid arguments in request
+
+- RTE_COMP_OP_STATUS_ERROR,
+    Operation failed because of internal error
+
+- RTE_COMP_OP_STATUS_INVALID_STATE,
+    Operation is invoked in invalid state
+
+- RTE_COMP_OP_STATUS_OUT_OF_SPACE_TERMINATED,
+    Output buffer ran out of space during processing. Error case,
+    PMD cannot continue from here.
+
+- RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE,
+    Output buffer ran out of space before operation completed, but this
+    is not an error case. Output data up to op.produced can be used and
+    next op in the stream should continue on from op.consumed+1.
+
+Produced, Consumed And Operation Status
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- If status is RTE_COMP_OP_STATUS_SUCCESS,
+    consumed = amount of data read from input buffer, and
+    produced = amount of data written in destination buffer
+- If status is RTE_COMP_OP_STATUS_FAILURE,
+    consumed = produced = 0 or undefined
+- If status is RTE_COMP_OP_STATUS_OUT_OF_SPACE_TERMINATED,
+    consumed = 0 and
+    produced = usually 0, but in decompression cases a PMD may return > 0
+    i.e. amount of data successfully produced until out of space condition
+    hit. Application can consume output data in this case, if required.
+- If status is RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE,
+    consumed = amount of data read, and
+    produced = amount of data successfully produced until
+    out of space condition hit. PMD has ability to recover
+    from here, so application can submit next op from
+    consumed+1 and a destination buffer with available space.
+
+Transforms
+----------
+
+Compression transforms (``rte_comp_xform``) are the mechanism
+to specify the details of the compression operation such as algorithm,
+window size and checksum.
+
+Compression API Hash support
+----------------------------
+
+Compression API allows application to enable digest calculation
+alongside compression and decompression of data. A PMD reflects its
+support for hash algorithms via capability algo feature flags.
+If supported, PMD calculates digest always on plaintext i.e.
+before compression and after decompression.
+
+Currently supported list of hash algos are SHA-1 and SHA2 family
+SHA256.
+
+See *DPDK API Reference* for details.
+
+If required, application should set valid hash algo in compress
+or decompress xforms during ``rte_compressdev_stream_create()``
+or ``rte_compressdev_private_xform_create()`` and pass a valid
+output buffer in ``rte_comp_op`` hash field struct to store the
+resulting digest. Buffer passed should be contiguous and large
+enough to store digest which is 20 bytes for SHA-1 and
+32 bytes for SHA2-256.
+
+Compression API Stateless operation
+------------------------------------
+
+An op is processed stateless if it has
+- op_type set to RTE_COMP_OP_STATELESS
+- flush value set to RTE_FLUSH_FULL or RTE_FLUSH_FINAL
+(required only on compression side),
+- All required input in source buffer
+
+When all of the above conditions are met, PMD initiates stateless processing
+and releases acquired resources after processing of current operation is
+complete. Application can enqueue multiple stateless ops in a single burst
+and must attach priv_xform handle to such ops.
+
+priv_xform in Stateless operation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+priv_xform is PMD internally managed private data that it maintains to do stateless processing.
+priv_xforms are initialized provided a generic xform structure by an application via making call
+to ``rte_comp_private_xform_create``, at an output PMD returns an opaque priv_xform reference.
+If PMD support SHAREABLE priv_xform indicated via algorithm feature flag, then application can
+attach same priv_xform with many stateless ops at-a-time. If not, then application needs to
+create as many priv_xforms as it expects to have stateless operations in-flight.
+
+.. figure:: img/stateless-op.*
+
+   Stateless Ops using Non-Shareable priv_xform
+
+
+.. figure:: img/stateless-op-shared.*
+
+   Stateless Ops using Shareable priv_xform
+
+
+Application should call ``rte_compressdev_private_xform_create()`` and attach to stateless op before
+enqueuing them for processing and free via ``rte_compressdev_private_xform_free()`` during termination.
+
+An example pseudocode to setup and process NUM_OPS stateless ops with each of length OP_LEN
+using priv_xform would look like:
+
+.. code-block:: c
+
+    /*
+     * pseudocode for stateless compression
+     */
+
+    uint8_t cdev_id = rte_compdev_get_dev_id(<pmd name>);
+
+    /* configure the device. */
+    if (rte_compressdev_configure(cdev_id, &conf) < 0)
+        rte_exit(EXIT_FAILURE, "Failed to configure compressdev %u", cdev_id);
+
+    if (rte_compressdev_queue_pair_setup(cdev_id, 0, NUM_MAX_INFLIGHT_OPS,
+                            socket_id()) < 0)
+        rte_exit(EXIT_FAILURE, "Failed to setup queue pair\n");
+
+    if (rte_compressdev_start(cdev_id) < 0)
+        rte_exit(EXIT_FAILURE, "Failed to start device\n");
+
+    /* setup compress transform */
+    struct rte_compress_compress_xform compress_xform = {
+        .type = RTE_COMP_COMPRESS,
+        .compress = {
+            .algo = RTE_COMP_ALGO_DEFLATE,
+            .deflate = {
+                .huffman = RTE_COMP_HUFFMAN_DEFAULT
+            },
+            .level = RTE_COMP_LEVEL_PMD_DEFAULT,
+            .chksum = RTE_COMP_CHECKSUM_NONE,
+            .window_size = DEFAULT_WINDOW_SIZE,
+            .hash_algo = RTE_COMP_HASH_ALGO_NONE
+        }
+    };
+
+    /* create priv_xform and initialize it for the compression device. */
+    void *priv_xform = NULL;
+    rte_compressdev_info_get(cdev_id, &dev_info);
+    if(dev_info.capability->comps_feature_flag & RTE_COMP_FF_SHAREABLE_PRIV_XFORM) {
+        rte_comp_priv_xform_create(cdev_id, &compress_xform, &priv_xform);
+    } else {
+        shareable = 0;
+    }
+
+    /* create operation pool via call to rte_comp_op_pool_create and alloc ops */
+    rte_comp_op_bulk_alloc(op_pool, comp_ops, NUM_OPS);
+
+    /* prepare ops for compression operations */
+    for (i = 0; i < NUM_OPS; i++) {
+        struct rte_comp_op *op = comp_ops[i];
+        if (!shareable)
+            rte_priv_xform_create(cdev_id, &compress_xform, &op->priv_xform)
+        else
+            op->priv_xform = priv_xform;
+        op->type = RTE_COMP_OP_STATELESS;
+        op->flush = RTE_COMP_FLUSH_FINAL;
+
+        op->src.offset = 0;
+        op->dst.offset = 0;
+        op->src.length = OP_LEN;
+        op->input_chksum = 0;
+        setup op->m_src and op->m_dst;
+    }
+    num_enqd = rte_compressdev_enqueue_burst(cdev_id, 0, comp_ops, NUM_OPS);
+    /* wait for this to complete before enqueing next*/
+    do {
+        num_deque = rte_compressdev_dequeue_burst(cdev_id, 0 , &processed_ops, NUM_OPS);
+    } while (num_dqud < num_enqd);
+
+
+Stateless and OUT_OF_SPACE
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+OUT_OF_SPACE is a condition when output buffer runs out of space and where PMD
+still has more data to produce. If PMD runs into such condition, then PMD returns
+RTE_COMP_OP_OUT_OF_SPACE_TERMINATED error. In such case, PMD resets itself and can set
+consumed=0 and produced=amount of output it could produce before hitting out_of_space.
+Application would need to resubmit the whole input with a larger output buffer, if it
+wants the operation to be completed.
+
+Hash in Stateless
+~~~~~~~~~~~~~~~~~
+If hash is enabled, digest buffer will contain valid data after op is successfully
+processed i.e. dequeued with status = RTE_COMP_OP_STATUS_SUCCESS.
+
+Checksum in Stateless
+~~~~~~~~~~~~~~~~~~~~~
+If checksum is enabled, checksum will only be available after op is successfully
+processed i.e. dequeued with status = RTE_COMP_OP_STATUS_SUCCESS.
+
+Compression API Stateful operation
+-----------------------------------
+
+Compression API provide RTE_COMP_FF_STATEFUL_COMPRESSION and
+RTE_COMP_FF_STATEFUL_DECOMPRESSION feature flag for PMD to reflect
+its support for Stateful operations.
+
+A Stateful operation in DPDK compression means application invokes enqueue
+burst() multiple times to process related chunk of data because
+application broke data into several ops.
+
+In such case
+- ops are setup with op_type RTE_COMP_OP_STATEFUL,
+- all ops except last set to flush value = RTE_COMP_NO/SYNC_FLUSH
+and last set to flush value RTE_COMP_FULL/FINAL_FLUSH.
+
+In case of either one or all of the above conditions, PMD initiates
+stateful processing and releases acquired resources after processing
+operation with flush value = RTE_COMP_FLUSH_FULL/FINAL is complete.
+Unlike stateless, application can enqueue only one stateful op from
+a particular stream at a time and must attach stream handle
+to each op.
+
+Stream in Stateful operation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`stream` in DPDK compression is a logical entity which identifies related set of ops, say, a one large
+file broken into multiple chunks then file is represented by a stream and each chunk of that file is
+represented by compression op `rte_comp_op`. Whenever application wants a stateful processing of such
+data, then it must get a stream handle via making call to ``rte_comp_stream_create()``
+with xform, at an output the target PMD will return an opaque stream handle to application which
+it must attach to all of the ops carrying data of that stream. In stateful processing, every op
+requires previous op data for compression/decompression. A PMD allocates and set up resources such
+as history, states, etc. within a stream, which are maintained during the processing of the related ops.
+
+Unlike priv_xforms, stream is always a NON_SHAREABLE entity. One stream handle must be attached to only
+one set of related ops and cannot be reused until all of them are processed with status Success or failure.
+
+.. figure:: img/stateful-op.*
+
+   Stateful Ops
+
+
+Application should call ``rte_comp_stream_create()`` and attach to op before
+enqueuing them for processing and free via ``rte_comp_stream_free()`` during
+termination. All ops that are to be processed statefully should carry *same* stream.
+
+See *DPDK API Reference* document for details.
+
+An example pseudocode to set up and process a stream having NUM_CHUNKS with each chunk size of CHUNK_LEN would look like:
+
+.. code-block:: c
+
+    /*
+     * pseudocode for stateful compression
+     */
+
+    uint8_t cdev_id = rte_compdev_get_dev_id(<pmd name>);
+
+    /* configure the  device. */
+    if (rte_compressdev_configure(cdev_id, &conf) < 0)
+        rte_exit(EXIT_FAILURE, "Failed to configure compressdev %u", cdev_id);
+
+    if (rte_compressdev_queue_pair_setup(cdev_id, 0, NUM_MAX_INFLIGHT_OPS,
+                                    socket_id()) < 0)
+        rte_exit(EXIT_FAILURE, "Failed to setup queue pair\n");
+
+    if (rte_compressdev_start(cdev_id) < 0)
+        rte_exit(EXIT_FAILURE, "Failed to start device\n");
+
+    /* setup compress transform. */
+    struct rte_compress_compress_xform compress_xform = {
+        .type = RTE_COMP_COMPRESS,
+        .compress = {
+            .algo = RTE_COMP_ALGO_DEFLATE,
+            .deflate = {
+                .huffman = RTE_COMP_HUFFMAN_DEFAULT
+            },
+            .level = RTE_COMP_LEVEL_PMD_DEFAULT,
+            .chksum = RTE_COMP_CHECKSUM_NONE,
+            .window_size = DEFAULT_WINDOW_SIZE,
+                        .hash_algo = RTE_COMP_HASH_ALGO_NONE
+        }
+    };
+
+    /* create stream */
+    rte_comp_stream_create(cdev_id, &compress_xform, &stream);
+
+    /* create an op pool and allocate ops */
+    rte_comp_op_bulk_alloc(op_pool, comp_ops, NUM_CHUNKS);
+
+    /* Prepare source and destination mbufs for compression operations */
+    unsigned int i;
+    for (i = 0; i < NUM_CHUNKS; i++) {
+        if (rte_pktmbuf_append(mbufs[i], CHUNK_LEN) == NULL)
+            rte_exit(EXIT_FAILURE, "Not enough room in the mbuf\n");
+        comp_ops[i]->m_src = mbufs[i];
+        if (rte_pktmbuf_append(dst_mbufs[i], CHUNK_LEN) == NULL)
+            rte_exit(EXIT_FAILURE, "Not enough room in the mbuf\n");
+        comp_ops[i]->m_dst = dst_mbufs[i];
+    }
+
+    /* Set up the compress operations. */
+    for (i = 0; i < NUM_CHUNKS; i++) {
+        struct rte_comp_op *op = comp_ops[i];
+        op->stream = stream;
+        op->m_src = src_buf[i];
+        op->m_dst = dst_buf[i];
+        op->type = RTE_COMP_OP_STATEFUL;
+        if(i == NUM_CHUNKS-1) {
+            /* set to final, if last chunk*/
+            op->flush = RTE_COMP_FLUSH_FINAL;
+        } else {
+            /* set to NONE, for all intermediary ops */
+            op->flush = RTE_COMP_FLUSH_NONE;
+        }
+        op->src.offset = 0;
+        op->dst.offset = 0;
+        op->src.length = CHUNK_LEN;
+        op->input_chksum = 0;
+        num_enqd = rte_compressdev_enqueue_burst(cdev_id, 0, &op[i], 1);
+        /* wait for this to complete before enqueing next*/
+        do {
+            num_deqd = rte_compressdev_dequeue_burst(cdev_id, 0 , &processed_ops, 1);
+        } while (num_deqd < num_enqd);
+        /* push next op*/
+    }
+
+
+Stateful and OUT_OF_SPACE
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If PMD supports stateful operation, then OUT_OF_SPACE status is not an actual
+error for the PMD. In such case, PMD returns with status
+RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE with consumed = number of input bytes
+read and produced = length of complete output buffer.
+Application should enqueue next op with source starting at consumed+1 and an
+output buffer with available space.
+
+Hash in Stateful
+~~~~~~~~~~~~~~~~
+If enabled, digest buffer will contain valid digest after last op in stream
+(having flush = RTE_COMP_OP_FLUSH_FINAL) is successfully processed i.e. dequeued
+with status = RTE_COMP_OP_STATUS_SUCCESS.
+
+Checksum in Stateful
+~~~~~~~~~~~~~~~~~~~~
+If enabled, checksum will only be available after last op in stream
+(having flush = RTE_COMP_OP_FLUSH_FINAL) is successfully processed i.e. dequeued
+with status = RTE_COMP_OP_STATUS_SUCCESS.
+
+Burst in compression API
+-------------------------
+
+Scheduling of compression operations on DPDK's application data path is
+performed using a burst oriented asynchronous API set. A queue pair on a compression
+device accepts a burst of compression operations using enqueue burst API. On physical
+devices the enqueue burst API will place the operations to be processed
+on the device's hardware input queue, for virtual devices the processing of the
+operations is usually completed during the enqueue call to the compression
+device. The dequeue burst API will retrieve any processed operations available
+from the queue pair on the compression device, from physical devices this is usually
+directly from the devices processed queue, and for virtual device's from a
+``rte_ring`` where processed operations are place after being processed on the
+enqueue call.
+
+A burst in DPDK compression can be a combination of stateless and stateful operations with a condition
+that for stateful ops only one op at-a-time should be enqueued from a particular stream i.e. no-two ops
+should belong to same stream in a single burst. However a burst may contain multiple stateful ops as long
+as each op is attached to a different stream i.e. a burst can look like:
+
++---------------+--------------+--------------+-----------------+--------------+--------------+
+| enqueue_burst | op1.no_flush | op2.no_flush | op3.flush_final | op4.no_flush | op5.no_flush |
++---------------+--------------+--------------+-----------------+--------------+--------------+
+
+Where, op1 .. op5 all belong to different independent data units. op1, op2, op4, op5 must be stateful
+as stateless ops can only use flush full or final and op3 can be of type stateless or stateful.
+Every op with type set to RTE_COMP_OP_TYPE_STATELESS must be attached to priv_xform and
+Every op with type set to RTE_COMP_OP_TYPE_STATEFUL *must* be attached to stream.
+
+Since each operation in a burst is independent and thus can be completed
+out-of-order,  applications which need ordering, should setup per-op user data
+area with reordering information so that it can determine enqueue order at
+dequeue.
+
+Also if multiple threads calls enqueue_burst() on same queue pair then it’s
+application onus to use proper locking mechanism to ensure exclusive enqueuing
+of operations.
+
+Enqueue / Dequeue Burst APIs
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The burst enqueue API uses a compression device identifier and a queue pair
+identifier to specify the compression device queue pair to schedule the processing on.
+The ``nb_ops`` parameter is the number of operations to process which are
+supplied in the ``ops`` array of ``rte_comp_op`` structures.
+The enqueue function returns the number of operations it actually enqueued for
+processing, a return value equal to ``nb_ops`` means that all packets have been
+enqueued.
+
+The dequeue API uses the same format as the enqueue API but
+the ``nb_ops`` and ``ops`` parameters are now used to specify the max processed
+operations the user wishes to retrieve and the location in which to store them.
+The API call returns the actual number of processed operations returned, this
+can never be larger than ``nb_ops``.
+
+Sample code
+-----------
+
+There are unit test applications that show how to use the compressdev library inside
+test/test/test_compressdev.c
+
+Compression Device API
+~~~~~~~~~~~~~~~~~~~~~~
+
+The compressdev Library API is described in the *DPDK API Reference* document.
diff --git a/doc/guides/prog_guide/img/stateful-op.svg b/doc/guides/prog_guide/img/stateful-op.svg
new file mode 100644
index 000000000..e6ef6353d
--- /dev/null
+++ b/doc/guides/prog_guide/img/stateful-op.svg
@@ -0,0 +1,116 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<!-- Generated by Microsoft Visio, SVG Export stateful-ops.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+		xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="1.49139in" height="1.35359in"
+		viewBox="0 0 107.38 97.4587" xml:space="preserve" color-interpolation-filters="sRGB" class="st6">
+	<v:documentProperties v:langID="16393" v:metric="true" v:viewMarkup="false"/>
+
+	<style type="text/css">
+	<![CDATA[
+		.st1 {visibility:visible}
+		.st2 {fill:#5b9bd5;fill-opacity:0.22;filter:url(#filter_2);stroke:#5b9bd5;stroke-opacity:0.22}
+		.st3 {fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25}
+		.st4 {fill:#feffff;font-family:Calibri;font-size:0.833336em}
+		.st5 {stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+		.st6 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+	]]>
+	</style>
+
+	<defs id="Filters">
+		<filter id="filter_2">
+			<feGaussianBlur stdDeviation="2"/>
+		</filter>
+	</defs>
+	<g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+		<title>Page-1</title>
+		<v:pageProperties v:drawingScale="0.0393701" v:pageScale="0.0393701" v:drawingUnits="24" v:shadowOffsetX="8.50394"
+				v:shadowOffsetY="-8.50394"/>
+		<g id="shape38-1" v:mID="38" v:groupContext="shape" transform="translate(58.305,-28.025)">
+			<title>Circle</title>
+			<desc>stream</desc>
+			<v:userDefs>
+				<v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+			</v:userDefs>
+			<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
+			<v:textRect cx="22.6772" cy="74.7815" width="39.69" height="34.0157"/>
+			<g id="shadow38-2" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
+					transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+				<path d="M0 74.78 A22.6772 22.6772 0 0 1 45.35 74.78 A22.6772 22.6772 0 1 1 0 74.78 Z" class="st2"/>
+			</g>
+			<path d="M0 74.78 A22.6772 22.6772 0 0 1 45.35 74.78 A22.6772 22.6772 0 1 1 0 74.78 Z" class="st3"/>
+			<text x="8.43" y="77.78" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>stream</text>		</g>
+		<g id="shape39-7" v:mID="39" v:groupContext="shape" transform="translate(3.0294,-73.3793)">
+			<title>Circle.39</title>
+			<desc>op</desc>
+			<v:userDefs>
+				<v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+			</v:userDefs>
+			<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
+			<v:textRect cx="11.3386" cy="86.1201" width="19.85" height="17.0079"/>
+			<g id="shadow39-8" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
+					transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+				<path d="M0 86.12 A11.3386 11.3386 0 0 1 22.68 86.12 A11.3386 11.3386 0 1 1 0 86.12 Z" class="st2"/>
+			</g>
+			<path d="M0 86.12 A11.3386 11.3386 0 0 1 22.68 86.12 A11.3386 11.3386 0 1 1 0 86.12 Z" class="st3"/>
+			<text x="6.07" y="89.12" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text>		</g>
+		<g id="shape40-13" v:mID="40" v:groupContext="shape" transform="translate(3.0294,-50.7021)">
+			<title>Circle.40</title>
+			<desc>op</desc>
+			<v:userDefs>
+				<v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+			</v:userDefs>
+			<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
+			<v:textRect cx="11.3386" cy="86.1201" width="19.85" height="17.0079"/>
+			<g id="shadow40-14" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
+					transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+				<path d="M0 86.12 A11.3386 11.3386 0 0 1 22.68 86.12 A11.3386 11.3386 0 1 1 0 86.12 Z" class="st2"/>
+			</g>
+			<path d="M0 86.12 A11.3386 11.3386 0 0 1 22.68 86.12 A11.3386 11.3386 0 1 1 0 86.12 Z" class="st3"/>
+			<text x="6.07" y="89.12" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text>		</g>
+		<g id="shape41-19" v:mID="41" v:groupContext="shape" transform="translate(3.0294,-28.025)">
+			<title>Circle.41</title>
+			<desc>op</desc>
+			<v:userDefs>
+				<v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+			</v:userDefs>
+			<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
+			<v:textRect cx="11.3386" cy="86.1201" width="19.85" height="17.0079"/>
+			<g id="shadow41-20" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
+					transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+				<path d="M0 86.12 A11.3386 11.3386 0 0 1 22.68 86.12 A11.3386 11.3386 0 1 1 0 86.12 Z" class="st2"/>
+			</g>
+			<path d="M0 86.12 A11.3386 11.3386 0 0 1 22.68 86.12 A11.3386 11.3386 0 1 1 0 86.12 Z" class="st3"/>
+			<text x="6.07" y="89.12" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text>		</g>
+		<g id="shape42-25" v:mID="42" v:groupContext="shape" transform="translate(3.0294,-5.34779)">
+			<title>Circle.249</title>
+			<desc>op</desc>
+			<v:userDefs>
+				<v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+			</v:userDefs>
+			<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
+			<v:textRect cx="11.3386" cy="86.1201" width="19.85" height="17.0079"/>
+			<g id="shadow42-26" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
+					transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+				<path d="M0 86.12 A11.3386 11.3386 0 0 1 22.68 86.12 A11.3386 11.3386 0 1 1 0 86.12 Z" class="st2"/>
+			</g>
+			<path d="M0 86.12 A11.3386 11.3386 0 0 1 22.68 86.12 A11.3386 11.3386 0 1 1 0 86.12 Z" class="st3"/>
+			<text x="6.07" y="89.12" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text>		</g>
+		<g id="shape43-31" v:mID="43" v:groupContext="shape" transform="translate(66.3024,-75.8604) rotate(24.6166)">
+			<title>Sheet.43</title>
+			<path d="M0 97.46 L43.16 97.46" class="st5"/>
+		</g>
+		<g id="shape44-34" v:mID="44" v:groupContext="shape" transform="translate(37.2064,-61.3598) rotate(6.77654)">
+			<title>Sheet.44</title>
+			<path d="M0 97.46 L34.05 97.46" class="st5"/>
+		</g>
+		<g id="shape45-37" v:mID="45" v:groupContext="shape" transform="translate(-6.31062,-33.9543) rotate(-19.179)">
+			<title>Sheet.45</title>
+			<path d="M0 97.46 L34.51 97.46" class="st5"/>
+		</g>
+		<g id="shape46-40" v:mID="46" v:groupContext="shape" transform="translate(-14.8893,-7.82888) rotate(-24.6166)">
+			<title>Sheet.46</title>
+			<path d="M0 97.46 L43.16 97.46" class="st5"/>
+		</g>
+	</g>
+</svg>
diff --git a/doc/guides/prog_guide/img/stateless-op-shared.svg b/doc/guides/prog_guide/img/stateless-op-shared.svg
new file mode 100644
index 000000000..257a69a52
--- /dev/null
+++ b/doc/guides/prog_guide/img/stateless-op-shared.svg
@@ -0,0 +1,124 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<!-- Generated by Microsoft Visio, SVG Export Drawing5.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+		xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="1.89687in" height="1.60662in"
+		viewBox="0 0 136.575 115.676" xml:space="preserve" color-interpolation-filters="sRGB" class="st7">
+	<v:documentProperties v:langID="16393" v:metric="true" v:viewMarkup="false"/>
+
+	<style type="text/css">
+	<![CDATA[
+		.st1 {visibility:visible}
+		.st2 {fill:#5b9bd5;fill-opacity:0.22;filter:url(#filter_2);stroke:#5b9bd5;stroke-opacity:0.22}
+		.st3 {fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25}
+		.st4 {fill:#feffff;font-family:Calibri;font-size:0.833336em}
+		.st5 {fill:none}
+		.st6 {stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.999999}
+		.st7 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+	]]>
+	</style>
+
+	<defs id="Filters">
+		<filter id="filter_2">
+			<feGaussianBlur stdDeviation="2"/>
+		</filter>
+	</defs>
+	<g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+		<title>Page-1</title>
+		<v:pageProperties v:drawingScale="0.0393701" v:pageScale="0.0393701" v:drawingUnits="24" v:shadowOffsetX="8.50394"
+				v:shadowOffsetY="-8.50394"/>
+		<g id="group47-1" transform="translate(3.02997,-5.34779)" v:mID="47" v:groupContext="group">
+			<title>Sheet.47</title>
+			<g id="shape36-2" v:mID="36" v:groupContext="shape" transform="translate(66.2255,-27.0553)">
+				<title>Circle</title>
+				<desc>priv_xform</desc>
+				<v:userDefs>
+					<v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+				</v:userDefs>
+				<v:textBlock v:margins="rect(3.99999,3.99999,3.99999,3.99999)" v:tabSpace="42.5196"/>
+				<v:textRect cx="31.7998" cy="88.2699" width="55.66" height="40.7542"/>
+				<g id="shadow36-3" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
+						transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+					<path d="M0 88.27 A31.7998 27.1694 0 1 1 63.6 88.27 A31.7998 27.1694 0 1 1 0 88.27 Z" class="st2"/>
+				</g>
+				<path d="M0 88.27 A31.7998 27.1694 0 1 1 63.6 88.27 A31.7998 27.1694 0 1 1 0 88.27 Z" class="st3"/>
+				<text x="9.47" y="91.27" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>priv_xform</text>			</g>
+			<g id="shape39-8" v:mID="39" v:groupContext="shape" transform="translate(-5.9952E-015,-81.5083)">
+				<title>Circle.40</title>
+				<desc>op</desc>
+				<v:userDefs>
+					<v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+				</v:userDefs>
+				<v:textBlock v:margins="rect(3.99999,3.99999,3.99999,3.99999)" v:tabSpace="42.5196"/>
+				<v:textRect cx="13.5848" cy="101.968" width="23.78" height="20.3771"/>
+				<g id="shadow39-9" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
+						transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+					<path d="M0 101.97 A13.5848 13.5848 0 1 1 27.17 101.97 A13.5848 13.5848 0 1 1 0 101.97 Z" class="st2"/>
+				</g>
+				<path d="M0 101.97 A13.5848 13.5848 0 1 1 27.17 101.97 A13.5848 13.5848 0 1 1 0 101.97 Z" class="st3"/>
+				<text x="8.32" y="104.97" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text>			</g>
+			<g id="shape40-14" v:mID="40" v:groupContext="shape" transform="translate(-5.9952E-015,-54.3389)">
+				<title>Circle.41</title>
+				<desc>op</desc>
+				<v:userDefs>
+					<v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+				</v:userDefs>
+				<v:textBlock v:margins="rect(3.99999,3.99999,3.99999,3.99999)" v:tabSpace="42.5196"/>
+				<v:textRect cx="13.5848" cy="101.968" width="23.78" height="20.3771"/>
+				<g id="shadow40-15" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
+						transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+					<path d="M0 101.97 A13.5848 13.5848 0 1 1 27.17 101.97 A13.5848 13.5848 0 1 1 0 101.97 Z" class="st2"/>
+				</g>
+				<path d="M0 101.97 A13.5848 13.5848 0 1 1 27.17 101.97 A13.5848 13.5848 0 1 1 0 101.97 Z" class="st3"/>
+				<text x="8.32" y="104.97" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text>			</g>
+			<g id="shape41-20" v:mID="41" v:groupContext="shape" transform="translate(-5.9952E-015,-27.1694)">
+				<title>Circle.42</title>
+				<desc>op</desc>
+				<v:userDefs>
+					<v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+				</v:userDefs>
+				<v:textBlock v:margins="rect(3.99999,3.99999,3.99999,3.99999)" v:tabSpace="42.5196"/>
+				<v:textRect cx="13.5848" cy="101.968" width="23.78" height="20.3771"/>
+				<g id="shadow41-21" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
+						transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+					<path d="M0 101.97 A13.5848 13.5848 0 1 1 27.17 101.97 A13.5848 13.5848 0 1 1 0 101.97 Z" class="st2"/>
+				</g>
+				<path d="M0 101.97 A13.5848 13.5848 0 1 1 27.17 101.97 A13.5848 13.5848 0 1 1 0 101.97 Z" class="st3"/>
+				<text x="8.32" y="104.97" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text>			</g>
+			<g id="shape42-26" v:mID="42" v:groupContext="shape">
+				<title>Circle.249</title>
+				<desc>op</desc>
+				<v:userDefs>
+					<v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+				</v:userDefs>
+				<v:textBlock v:margins="rect(3.99999,3.99999,3.99999,3.99999)" v:tabSpace="42.5196"/>
+				<v:textRect cx="13.5848" cy="101.968" width="23.78" height="20.3771"/>
+				<g id="shadow42-27" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
+						transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+					<path d="M0 101.97 A13.5848 13.5848 0 1 1 27.17 101.97 A13.5848 13.5848 0 1 1 0 101.97 Z" class="st2"/>
+				</g>
+				<path d="M0 101.97 A13.5848 13.5848 0 1 1 27.17 101.97 A13.5848 13.5848 0 1 1 0 101.97 Z" class="st3"/>
+				<text x="8.32" y="104.97" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text>			</g>
+			<g id="shape43-32" v:mID="43" v:groupContext="shape" transform="translate(75.3544,-84.7046) rotate(24.6166)">
+				<title>Sheet.43</title>
+				<path d="M0 115.68 L51.71 115.68 L0 115.68 Z" class="st5"/>
+				<path d="M0 115.68 L51.71 115.68" class="st6"/>
+			</g>
+			<g id="shape44-35" v:mID="44" v:groupContext="shape" transform="translate(40.8189,-67.2403) rotate(6.77654)">
+				<title>Sheet.44</title>
+				<path d="M0 115.68 L40.8 115.68 L0 115.68 Z" class="st5"/>
+				<path d="M0 115.68 L40.8 115.68" class="st6"/>
+			</g>
+			<g id="shape45-38" v:mID="45" v:groupContext="shape" transform="translate(-10.8336,-34.4585) rotate(-19.179)">
+				<title>Sheet.45</title>
+				<path d="M0 115.68 L41.35 115.68 L0 115.68 Z" class="st5"/>
+				<path d="M0 115.68 L41.35 115.68" class="st6"/>
+			</g>
+			<g id="shape46-41" v:mID="46" v:groupContext="shape" transform="translate(-21.0159,-3.19618) rotate(-24.6166)">
+				<title>Sheet.46</title>
+				<path d="M0 115.68 L51.71 115.68 L0 115.68 Z" class="st5"/>
+				<path d="M0 115.68 L51.71 115.68" class="st6"/>
+			</g>
+		</g>
+	</g>
+</svg>
diff --git a/doc/guides/prog_guide/img/stateless-op.svg b/doc/guides/prog_guide/img/stateless-op.svg
new file mode 100644
index 000000000..fd951b7ad
--- /dev/null
+++ b/doc/guides/prog_guide/img/stateless-op.svg
@@ -0,0 +1,140 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<!-- Generated by Microsoft Visio, SVG Export stateless-ops.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+		xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="2.24024in" height="2.70592in"
+		viewBox="0 0 161.298 194.826" xml:space="preserve" color-interpolation-filters="sRGB" class="st8">
+	<v:documentProperties v:langID="16393" v:metric="true" v:viewMarkup="false"/>
+
+	<style type="text/css">
+	<![CDATA[
+		.st1 {visibility:visible}
+		.st2 {fill:#5b9bd5;fill-opacity:0.22;filter:url(#filter_2);stroke:#5b9bd5;stroke-opacity:0.22}
+		.st3 {fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25}
+		.st4 {fill:#feffff;font-family:Calibri;font-size:0.833336em}
+		.st5 {fill:#feffff;font-family:Calibri;font-size:0.75em}
+		.st6 {marker-start:url(#mrkr13-19);stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+		.st7 {fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-opacity:1;stroke-width:0.28409094308259}
+		.st8 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+	]]>
+	</style>
+
+	<defs id="Markers">
+		<g id="lend13">
+			<path d="M 3 1 L 0 0 L 3 -1 L 3 1 " style="stroke:none"/>
+		</g>
+		<marker id="mrkr13-19" class="st7" v:arrowType="13" v:arrowSize="2" v:setback="0" refX="0" orient="auto"
+				markerUnits="strokeWidth" overflow="visible">
+			<use xlink:href="#lend13" transform="scale(3.5199995788296) "/>
+		</marker>
+	</defs>
+	<defs id="Filters">
+		<filter id="filter_2">
+			<feGaussianBlur stdDeviation="2"/>
+		</filter>
+	</defs>
+	<g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+		<title>Page-1</title>
+		<v:pageProperties v:drawingScale="0.0393701" v:pageScale="0.0393701" v:drawingUnits="24" v:shadowOffsetX="9"
+				v:shadowOffsetY="-9"/>
+		<g id="group61-1" transform="translate(3.02943,-5.34782)" v:mID="61" v:groupContext="group">
+			<title>Sheet.61</title>
+			<g id="shape52-2" v:mID="52" v:groupContext="shape" transform="translate(97.856,-133.39)">
+				<title>Circle.40</title>
+				<desc>op</desc>
+				<v:userDefs>
+					<v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+				</v:userDefs>
+				<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
+				<v:textRect cx="28.3463" cy="167.479" width="49.62" height="41.4408"/>
+				<g id="shadow52-3" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
+						transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+					<path d="M-0 167.48 A28.3465 27.6272 180 0 1 56.69 167.48 A28.3465 27.6272 180 0 1 -0 167.48 Z" class="st2"/>
+				</g>
+				<path d="M-0 167.48 A28.3465 27.6272 180 0 1 56.69 167.48 A28.3465 27.6272 180 0 1 -0 167.48 Z" class="st3"/>
+				<text x="23.08" y="170.48" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text>			</g>
+			<g id="shape53-8" v:mID="53" v:groupContext="shape" transform="translate(-3.9968E-015,-133.39)">
+				<title>Circle.299</title>
+				<desc>priv_xform</desc>
+				<v:userDefs>
+					<v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+				</v:userDefs>
+				<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
+				<v:textRect cx="28.3463" cy="167.479" width="49.62" height="41.4408"/>
+				<g id="shadow53-9" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
+						transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+					<path d="M-0 167.48 A28.3465 27.6272 180 0 1 56.69 167.48 A28.3465 27.6272 180 0 1 -0 167.48 Z" class="st2"/>
+				</g>
+				<path d="M-0 167.48 A28.3465 27.6272 180 0 1 56.69 167.48 A28.3465 27.6272 180 0 1 -0 167.48 Z" class="st3"/>
+				<text x="8.25" y="170.18" class="st5" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>priv_xform</text>			</g>
+			<g id="shape54-14" v:mID="54" v:groupContext="shape" transform="translate(56.693,-160.74)">
+				<title>Sheet.54</title>
+				<path d="M0 194.83 L10.2 194.83 L10.56 194.83 L41.16 194.83" class="st6"/>
+			</g>
+			<g id="shape55-20" v:mID="55" v:groupContext="shape" transform="translate(97.856,-65.1969)">
+				<title>Circle.479</title>
+				<desc>op</desc>
+				<v:userDefs>
+					<v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+				</v:userDefs>
+				<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
+				<v:textRect cx="28.3463" cy="166.185" width="49.62" height="42.5197"/>
+				<g id="shadow55-21" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
+						transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+					<path d="M0 166.19 A28.3465 28.3465 0 1 1 56.69 166.19 A28.3465 28.3465 0 1 1 0 166.19 Z" class="st2"/>
+				</g>
+				<path d="M0 166.19 A28.3465 28.3465 0 1 1 56.69 166.19 A28.3465 28.3465 0 1 1 0 166.19 Z" class="st3"/>
+				<text x="23.08" y="169.19" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text>			</g>
+			<g id="shape56-26" v:mID="56" v:groupContext="shape" transform="translate(-3.9968E-015,-65.7801)">
+				<title>Circle.480</title>
+				<desc>priv_xform</desc>
+				<v:userDefs>
+					<v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+				</v:userDefs>
+				<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
+				<v:textRect cx="28.3463" cy="166.768" width="49.62" height="42.5197"/>
+				<g id="shadow56-27" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
+						transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+					<path d="M0 166.77 A28.3465 28.3465 0 0 1 56.69 166.77 A28.3465 28.3465 0 0 1 0 166.77 Z" class="st2"/>
+				</g>
+				<path d="M0 166.77 A28.3465 28.3465 0 0 1 56.69 166.77 A28.3465 28.3465 0 0 1 0 166.77 Z" class="st3"/>
+				<text x="8.25" y="169.47" class="st5" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>priv_xform</text>			</g>
+			<g id="shape57-32" v:mID="57" v:groupContext="shape" transform="translate(56.693,-93.8414)">
+				<title>Sheet.57</title>
+				<path d="M0 194.83 L10.2 194.83 L10.56 194.83 L41.16 194.83" class="st6"/>
+			</g>
+			<g id="shape58-37" v:mID="58" v:groupContext="shape" transform="translate(97.856,0)">
+				<title>Circle.482</title>
+				<desc>op</desc>
+				<v:userDefs>
+					<v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+				</v:userDefs>
+				<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
+				<v:textRect cx="28.3463" cy="166.185" width="49.62" height="42.5197"/>
+				<g id="shadow58-38" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
+						transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+					<path d="M0 166.19 A28.3465 28.3465 0 1 1 56.69 166.19 A28.3465 28.3465 0 1 1 0 166.19 Z" class="st2"/>
+				</g>
+				<path d="M0 166.19 A28.3465 28.3465 0 1 1 56.69 166.19 A28.3465 28.3465 0 1 1 0 166.19 Z" class="st3"/>
+				<text x="23.08" y="169.19" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text>			</g>
+			<g id="shape59-43" v:mID="59" v:groupContext="shape" transform="translate(-3.9968E-015,-0.583223)">
+				<title>Circle.483</title>
+				<desc>priv_xform</desc>
+				<v:userDefs>
+					<v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+				</v:userDefs>
+				<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
+				<v:textRect cx="28.3463" cy="166.768" width="49.62" height="42.5197"/>
+				<g id="shadow59-44" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
+						transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+					<path d="M0 166.77 A28.3465 28.3465 0 0 1 56.69 166.77 A28.3465 28.3465 0 0 1 0 166.77 Z" class="st2"/>
+				</g>
+				<path d="M0 166.77 A28.3465 28.3465 0 0 1 56.69 166.77 A28.3465 28.3465 0 0 1 0 166.77 Z" class="st3"/>
+				<text x="8.25" y="169.47" class="st5" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>priv_xform</text>			</g>
+			<g id="shape60-49" v:mID="60" v:groupContext="shape" transform="translate(56.693,-28.6446)">
+				<title>Sheet.60</title>
+				<path d="M0 194.83 L10.2 194.83 L10.56 194.83 L41.16 194.83" class="st6"/>
+			</g>
+		</g>
+	</g>
+</svg>
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index 589c05d96..553d56ef0 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -21,6 +21,7 @@ Programmer's Guide
     traffic_management
     bbdev
     cryptodev_lib
+    compressdev_lib
     rte_security
     rawdev
     link_bonding_poll_mode_drv_lib
-- 
2.14.3

Re: [dpdk-dev] [PATCH v6 00/14] Implement compression API

[De Lara Guarch, Pablo]

> -----Original Message-----
> From: De Lara Guarch, Pablo
> Sent: Friday, April 27, 2018 2:24 PM
> To: dev@dpdk.org
> Cc: Trahe, Fiona <fiona.trahe@intel.com>; shally.verma@cavium.com;
> ahmed.mansour@nxp.com; Ashish.Gupta@cavium.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Shally Verma
> <shally.verma@caviumnetworks.com>; Ashish Gupta
> <ashish.gupta@caviumnetworks.com>
> Subject: [PATCH v6 00/14] Implement compression API
> 
> With the vast amounts of data being transported around networks and stored in
> storage systems, reducing data size is becoming ever more important.
> 
> There are both software libraries and hardware devices available that provide
> compression, but no common API.
> Such an API is proposed in this patchset, which supports the following features:
> 
> - Deflate Algorithm (https://tools.ietf.org/html/rfc1951)
> - LZS algorithm (https://tools.ietf.org/html/rfc2395)
> - Static and Dynamic Huffman encoding.
> - Compression levels
> - Checksum generation
> - Asynchronous burst API
> - private_xform - a place for PMDs to hold private data derived from
>   a xform and used by stateless operations.
> - stream - a place for PMDs to hold private data derived from
>   a xform and also maintain state and history data. For
>   stateful flows.
> 
> Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>
> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
> Signed-off-by: Shally Verma <shally.verma@caviumnetworks.com>
> Signed-off-by: Ashish Gupta <ashish.gupta@caviumnetworks.com>

Applied to dpdk-next-crypto.
Thanks,

Pablo