From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id C16401B010 for ; Fri, 15 Dec 2017 18:46:09 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 15 Dec 2017 09:46:07 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.45,406,1508828400"; d="scan'208";a="3030562" Received: from sivswdev01.ir.intel.com (HELO localhost.localdomain) ([10.237.217.45]) by orsmga007.jf.intel.com with ESMTP; 15 Dec 2017 09:46:06 -0800 From: "Trahe, Fiona" To: dev@dpdk.org, Shally.Verma@cavium.com Cc: Mahipal.Challa@cavium.com, NarayanaPrasad.Athreya@cavium.com, pablo.de.lara.guarch@intel.com, fiona.trahe@intel.com Date: Fri, 15 Dec 2017 17:46:03 +0000 Message-Id: <1513359963-13817-1-git-send-email-fiona.trahe@intel.com> X-Mailer: git-send-email 1.7.0.7 In-Reply-To: <1511542480-389-1-git-send-email-fiona.trahe@intel.com> References: <1511542480-389-1-git-send-email-fiona.trahe@intel.com> Subject: [dpdk-dev] [RFC v3 0/1] Compression API in DPDK X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Dec 2017 17:46:10 -0000 With the vast amounts of data being transported around networks and stored in storage systems, reducing data size is becoming ever more important. There are both software libraries and hardware devices available that provide compression, but no common API. This RFC proposes a compression API for DPDK to address this need. Features: • Deflate Algorithm (https://tools.ietf.org/html/rfc1951) • LZS algorithm (https://tools.ietf.org/html/rfc2395) • Static and Dynamic Huffman encoding. • Compression levels • Checksum generation • Asynchronous burst API • Session-based (a session contains immutable data only and is useable across devices) • stream-based to maintain state and history data for stateful flows. Note 1: Split of functionality above/below API When considering whether features should be supported on the API or not, the decision was based on the following: The purpose of the API is to decouple the application from the compute-intensive functions needed for compression by abstracting them under a common API. These can then be implemented by either hardware accelerators or optimised software libraries. Where features are not compute-intensive and unlikely to be offloaded or optimised, there’s nothing to be gained by each PMD having to separately implement them, and it makes more sense for them to be done above the API. So the following are not handled on the API and can be done above. • Prepending/appending protocol headers (gzip, zlib) • File-handling, breaking files up into packets, reassembling. • Synchronous API • Serialisation of stateful requests Note 2: The tricky question of where the API belongs We considered 1. Extending cryptodev 2. New acceldev APIs for device handling + compressdev APIs for data path 3. New acceldev for all APIs 4. New compressdev API We've gone with option 4, a compressdev API. See original RFC [1] for reasons. We explored wrapping this around a generic acceldev that would be hidden from the API but could be common to cryptodev, compressdev and other accelerators on the PMD interface, but this added complexity and indirection and didn't add enough value, so we've abandoned it. Opens: - Define structures and API for proposed hash functionality - Agree on stateful behaviour - Complete capability APIs Changes in RFC v3: - Fixed incorrect op pool element size - removed unnecessary rte_compressdev_qp_conf and renamed nb_descriptors to max_inflight_ops - Removed session_pool from qp setup as not needed - Removed session_pool from rte_compressdev_data as not needed - renamed CDEV_ to COMPDEV_ to avoid build clash with cryptodev log - fixed rte_comp_op_reset() so phys_addr and mempool not deleted. - added window size to xforms - added lib dependency in rte.app.mk - returned PMD value directly in rte_compressdev_get_private_session_size() as no need to adjust for hdr size as not supporting sessionless - renamed rte_compressdev_session_free()->..terminate() - added capability structs - added stateless/stateful enum, stream APIs and text about flush flag and stateful behaviour - removed intermediate_buffer_size from device info and config structs as not generic. Can be handled by PMD-specific config in the config file instead. - added max_nb_streams_per_qp to info struct. - Fix checkpatch errors. Changes in RFC v2: - rebased off 17.11 and extended to include all device APIs and to compile. - Added LZS algorithm - Clarified NULL algorithm - Use FIXED terminology rather than STATIC for Huffman type - removed proposed data verification functionality - quashed xform chain into a single xform for each direction - added FLUSH_SYNC - renamed OVERFLOW to OUT_OF_SPACE and clarified comment - added #defines for Level values - clarified how src and dst buffer lengths are passed on API [1] http://dpdk.org/ml/archives/dev/2017-October/078944.html Trahe, Fiona (1): lib: Add compressdev API config/common_base | 7 + lib/Makefile | 3 + lib/librte_compressdev/Makefile | 54 + lib/librte_compressdev/rte_comp.h | 608 ++++++++++ lib/librte_compressdev/rte_compressdev.c | 1167 ++++++++++++++++++++ lib/librte_compressdev/rte_compressdev.h | 892 +++++++++++++++ lib/librte_compressdev/rte_compressdev_pmd.c | 194 ++++ lib/librte_compressdev/rte_compressdev_pmd.h | 533 +++++++++ lib/librte_compressdev/rte_compressdev_version.map | 50 + lib/librte_eal/common/include/rte_log.h | 1 + mk/rte.app.mk | 1 + 11 files changed, 3510 insertions(+), 0 deletions(-) create mode 100644 lib/librte_compressdev/Makefile create mode 100644 lib/librte_compressdev/rte_comp.h create mode 100644 lib/librte_compressdev/rte_compressdev.c create mode 100644 lib/librte_compressdev/rte_compressdev.h create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.c create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.h create mode 100644 lib/librte_compressdev/rte_compressdev_version.map