From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 0E547235 for ; Fri, 24 Nov 2017 17:54:58 +0100 (CET) Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 24 Nov 2017 08:54:57 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.44,448,1505804400"; d="scan'208";a="9137936" Received: from sivswdev01.ir.intel.com (HELO localhost.localdomain) ([10.237.217.45]) by orsmga001.jf.intel.com with ESMTP; 24 Nov 2017 08:54:55 -0800 From: "Trahe, Fiona" To: dev@dpdk.org, Shally.Verma@cavium.com Cc: Mahipal.Challa@cavium.com, NarayanaPrasad.Athreya@cavium.com, pablo.de.lara.guarch@intel.com, fiona.trahe@intel.com Date: Fri, 24 Nov 2017 16:54:40 +0000 Message-Id: <1511542480-389-1-git-send-email-fiona.trahe@intel.com> X-Mailer: git-send-email 1.7.0.7 Subject: [dpdk-dev] [RFC v2] Compression API in DPDK X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 16:54:59 -0000 With the vast amounts of data being transported around networks and stored in storage systems, reducing data size is becoming ever more important. There are both software libraries and hardware devices available that provide compression, but no common API. This RFC proposes a compression API for DPDK to address this need. Features: • Deflate Algorithm (https://tools.ietf.org/html/rfc1951) • LZS algorithm (https://tools.ietf.org/html/rfc2395) • Static and Dynamic Huffman encoding. • Compression levels • Checksum generation • Asynchronous burst API • Session-based (for stateless a session may be useable across devices, for stateful it wouldn’t be.) Note 1: Split of functionality above/below API When considering whether features should be supported on the API or not, the decision was based on the following: The purpose of the API is to decouple the application from the compute-intensive functions needed for compression by abstracting them under a common API. These can then be implemented by either hardware accelerators or optimised software libraries. Where features are not compute-intensive and unlikely to be offloaded or optimised, there’s nothing to be gained by each PMD having to separately implement them, and it makes more sense for them to be done above the API. So the following are not handled on the API and can be done above. • Prepending/appending protocol headers (gzip, zlib) • File-handling, breaking files up into packets, reassembling. • Synchronous API Note 2: Stateful compression. A better compression ratio can be achieved by using stateful compression, allowing pattern matching across multiple sequential packets. This will be supported on the API, but in the interests of getting an initial RFC out for feedback, there are placeholder parameters on the API. Feedback welcome on what parameters are needed for this feature. Note 3: The tricky question of where the API belongs We considered 1. Extending cryptodev 2. New acceldev APIs for device handling + compressdev APIs for data path 3. New acceldev for all APIs 4. New compressdev API We've gone with option 4, a compressdev API. See original RFC for reasons. We explored wrapping this around a generic acceldev that would be hidden from the API but could be common to cryptodev, compressdev and other accelerators on the PMD interface, but this added complexity and indirection and didn't add enough value, so we've abandoned it. Opens: - Clarify stateful behaviour - Define Capability structures and API - Define structures and API for proposed hash functionality - simplifications possible in code derived from cryptodev, but maybe over-engineered for compressdev, e.g. - session architected to be useable across PMDs - xform chains Changes from RFC v1: - rebased off 17.11 and extended to include all device APIs and to compile. - Added LZS algorithm - Clarified NULL algorithm - Use FIXED terminology rather than STATIC for Huffman type - removed proposed data verification functionality - quashed xform chain into a single xform for each direction - added FLUSH_SYNC - renamed OVERFLOW to OUT_OF_SPACE and clarified comment - added #defines for Level values - clarified how src and dst buffer lengths are passed on API http://dpdk.org/ml/archives/dev/2017-October/078944.html Trahe, Fiona (1): [RFC v2] lib: add compressdev API config/common_base | 7 + lib/Makefile | 3 + lib/librte_compressdev/Makefile | 54 + lib/librte_compressdev/rte_comp.h | 565 ++++++++++ lib/librte_compressdev/rte_compressdev.c | 1181 ++++++++++++++++++++ lib/librte_compressdev/rte_compressdev.h | 863 ++++++++++++++ lib/librte_compressdev/rte_compressdev_pmd.c | 193 ++++ lib/librte_compressdev/rte_compressdev_pmd.h | 535 +++++++++ lib/librte_compressdev/rte_compressdev_version.map | 58 + lib/librte_eal/common/include/rte_log.h | 1 + 10 files changed, 3460 insertions(+), 0 deletions(-) create mode 100644 lib/librte_compressdev/Makefile create mode 100644 lib/librte_compressdev/rte_comp.h create mode 100644 lib/librte_compressdev/rte_compressdev.c create mode 100644 lib/librte_compressdev/rte_compressdev.h create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.c create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.h create mode 100644 lib/librte_compressdev/rte_compressdev_version.map