Re: [dpdk-dev] [RFC v3 0/1] Compression API in DPDK

["Verma, Shally" <Shally.Verma@cavium.com>]

> -----Original Message-----
> From: Ahmed Mansour [mailto:ahmed.mansour@nxp.com]
> Sent: 19 December 2017 02:37
> To: dev@dpdk.org; Verma, Shally <Shally.Verma@cavium.com>; Hemant
> Agrawal <hemant.agrawal@nxp.com>
> Cc: Hemant Agrawal <hemant.agrawal@nxp.com>; Challa, Mahipal
> <Mahipal.Challa@cavium.com>; fiona.trahe@intel.com; Athreya, Narayana
> Prasad <NarayanaPrasad.Athreya@cavium.com>;
> pablo.de.lara.guarch@intel.com; Roy Pledge <roy.pledge@nxp.com>; Youri
> Querry <youri.querry_1@nxp.com>
> Subject: Re: [RFC v3 0/1] Compression API in DPDK
> 
> Hi Fiona,
> 
> On 12/15/2017 11:16 PM, Trahe, Fiona wrote:
> 
> > With the vast amounts of data being transported around networks and
> stored in
> > storage systems, reducing data size is becoming ever more important.
> There
> > are both software libraries and hardware devices available that provide
> > compression, but no common API. This RFC proposes a compression API for
> > DPDK to address this need.
> >
> > Features:
> > • Deflate Algorithm
> (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fto
> ols.ietf.org%2Fhtml%2Frfc1951&data=02%7C01%7Cahmed.mansour%40nxp.
> com%7C76241a2796db4701ef0108d54634a70f%7C686ea1d3bc2b4c6fa92cd99c
> 5c301635%7C0%7C0%7C636492114308805852&sdata=yglh48%2F8IuEn%2F7YC
> L49FlyhGyCnNRX4g4xx2WJQesFs%3D&reserved=0)
> > • LZS algorithm
> (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fto
> ols.ietf.org%2Fhtml%2Frfc2395&data=02%7C01%7Cahmed.mansour%40nxp.
> com%7C76241a2796db4701ef0108d54634a70f%7C686ea1d3bc2b4c6fa92cd99c
> 5c301635%7C0%7C0%7C636492114308805852&sdata=BlfNe3pRXpEFJE4KS8Sp
> k8bJ5GWPDqADhZYoD7SKvjk%3D&reserved=0)
> > • Static and Dynamic Huffman encoding.
> > • Compression levels
> > • Checksum generation
> > • Asynchronous burst API
> > • Session-based (a session contains immutable data only and is useable
> across devices)
> > • stream-based to maintain state and history data for stateful flows.
> >
> > Note 1: Split of functionality above/below API
> > When considering whether features should be supported on the API or
> not, the
> > decision was based on the following:
> > The purpose of the API is to decouple the application from the compute-
> intensive
> > functions needed for compression by abstracting them under a common
> API. These
> > can then be implemented by either hardware accelerators or optimised
> software
> > libraries. Where features are not compute-intensive and unlikely to be
> > offloaded or optimised, there’s nothing to be gained by each PMD
> having
> > to separately implement them, and it makes more sense for them to be
> done
> > above the API. So the following are not handled on the API and can be
> done above.
> > • Prepending/appending protocol headers (gzip, zlib)
> 
> Agreed with the notion, however the header and footer handling can be
> added as an option. PMDs can support or not support each format. We
> (NXP) support auto padding of gzip and zlib headers as well as DEFLATE only.
> 

[Shally] Just my thoughts on this. For such case, ZLIB and GZIP should be added into algorithm lists. PMD which support them can generate
ZLIB/GZIP styled compressed streams and app can invoke compression with algo such as ZLIB/GZIP. 

Thanks
Shally

> > • File-handling, breaking files up into packets, reassembling.
> > • Synchronous API
> > • Serialisation of stateful requests
> >
> >
> Is stateful planned for next phase? During design discussions we uncovered
> many API design
> considerations necessary for stateful use. Chained stateful support in the
> future might not be
> possible without compatibility breaking
> 
> > Note 2: The tricky question of where the API belongs
> > We considered
> > 1. Extending cryptodev
> > 2. New acceldev APIs for device handling + compressdev APIs for data path
> > 3. New acceldev for all APIs
> > 4. New compressdev API
> > We've gone with option 4, a compressdev API.  See original RFC [1] for
> reasons.
> > We explored wrapping this around a generic acceldev that would be hidden
> from the API
> > but could be common to cryptodev, compressdev and other accelerators
> on the PMD interface,
> > but this added complexity and indirection and didn't add enough value, so
> we've abandoned it.
> >
> Makes sense. compression is common enough to be attempted to be a
> different device category.
> 
> 
> > Opens:
> >  - Define structures and API for proposed hash functionality
> >  - Agree on stateful behaviour
> 
> What are the the current thoughts for stateful behavior?
> 
> >  - Complete capability APIs
> 
> A capability API is very much required as different HW/SW can have
> different capabilities.
> 
> 
> regards,
> Ahmed
>