[dpdk-dev] [RFC v2] Compression API in DPDK

[dpdk-dev] [RFC v2] Compression API in DPDK

[Trahe, Fiona]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 4540 bytes --]

With the vast amounts of data being transported around networks and stored in storage systems, reducing data size is becoming ever more important. There are both software libraries and hardware devices available that provide compression, but no common API. This RFC proposes a compression API for DPDK to address this need.

Features:
• Deflate Algorithm (https://tools.ietf.org/html/rfc1951)
• LZS algorithm (https://tools.ietf.org/html/rfc2395)
• Static and Dynamic Huffman encoding.
• Compression levels
• Checksum generation
• Asynchronous burst API
• Session-based (for stateless a session may be useable across devices, for stateful it wouldn’t be.) 


Note 1: Split of functionality above/below API
When considering whether features should be supported on the API or not, the decision was based on the following:
The purpose of the API is to decouple the application from the compute-intensive functions needed for compression by abstracting them under a common API. These can then be implemented by either hardware accelerators or optimised software libraries. Where features are not compute-intensive and unlikely to be offloaded or optimised, there’s nothing to be gained by each PMD having to separately implement them, and it makes more sense for them to be done above the API. So the following are not handled on the API and can be done above. 
• Prepending/appending protocol headers (gzip, zlib) 
• File-handling, breaking files up into packets, reassembling.
• Synchronous API


Note 2: Stateful compression.
A better compression ratio can be achieved by using stateful compression, allowing pattern matching across multiple sequential packets. This will be supported on the API, but in the interests of getting an initial RFC out for feedback, there are placeholder parameters on the API. Feedback welcome on what parameters are needed for this feature.


Note 3: The tricky question of where the API belongs
We considered 
1. Extending cryptodev
2. New acceldev APIs for device handling + compressdev APIs for data path 
3. New acceldev for all APIs
4. New compressdev API
We've gone with option 4, a compressdev API.  See original RFC for reasons.
We explored wrapping this around a generic acceldev that would be hidden from the API 
but could be common to cryptodev, compressdev and other accelerators on the PMD interface,
but this added complexity and indirection and didn't add enough value, so we've abandoned it.

Opens:
 - Clarify stateful behaviour
 - Define Capability structures and API
 - Define structures and API for proposed hash functionality
 - simplifications possible in code derived from cryptodev, but maybe over-engineered for compressdev, e.g.
   - session architected to be useable across PMDs
   - xform chains


Changes from RFC v1:
 - rebased off 17.11 and extended to include all device APIs and to compile.
 - Added LZS algorithm
 - Clarified NULL algorithm
 - Use FIXED terminology rather than STATIC for Huffman type
 - removed proposed data verification functionality
 - quashed xform chain into a single xform for each direction
 - added FLUSH_SYNC
 - renamed OVERFLOW to OUT_OF_SPACE and clarified comment
 - added #defines for Level values
 - clarified how src and dst buffer lengths are passed on API

http://dpdk.org/ml/archives/dev/2017-October/078944.html


Trahe, Fiona (1):
  [RFC v2] lib: add compressdev API

 config/common_base                                 |    7 +
 lib/Makefile                                       |    3 +
 lib/librte_compressdev/Makefile                    |   54 +
 lib/librte_compressdev/rte_comp.h                  |  565 ++++++++++
 lib/librte_compressdev/rte_compressdev.c           | 1181 ++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           |  863 ++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.c       |  193 ++++
 lib/librte_compressdev/rte_compressdev_pmd.h       |  535 +++++++++
 lib/librte_compressdev/rte_compressdev_version.map |   58 +
 lib/librte_eal/common/include/rte_log.h            |    1 +
 10 files changed, 3460 insertions(+), 0 deletions(-)
 create mode 100644 lib/librte_compressdev/Makefile
 create mode 100644 lib/librte_compressdev/rte_comp.h
 create mode 100644 lib/librte_compressdev/rte_compressdev.c
 create mode 100644 lib/librte_compressdev/rte_compressdev.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.c
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_version.map

[dpdk-dev] [RFC v3 0/1] Compression API in DPDK

[Trahe, Fiona]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 5159 bytes --]

With the vast amounts of data being transported around networks and stored in
storage systems, reducing data size is becoming ever more important. There
are both software libraries and hardware devices available that provide
compression, but no common API. This RFC proposes a compression API for
DPDK to address this need.

Features:
• Deflate Algorithm (https://tools.ietf.org/html/rfc1951)
• LZS algorithm (https://tools.ietf.org/html/rfc2395)
• Static and Dynamic Huffman encoding.
• Compression levels
• Checksum generation
• Asynchronous burst API
• Session-based (a session contains immutable data only and is useable across devices) 
• stream-based to maintain state and history data for stateful flows.

Note 1: Split of functionality above/below API
When considering whether features should be supported on the API or not, the
decision was based on the following:
The purpose of the API is to decouple the application from the compute-intensive
functions needed for compression by abstracting them under a common API. These
can then be implemented by either hardware accelerators or optimised software
libraries. Where features are not compute-intensive and unlikely to be
offloaded or optimised, there’s nothing to be gained by each PMD having
to separately implement them, and it makes more sense for them to be done
above the API. So the following are not handled on the API and can be done above. 
• Prepending/appending protocol headers (gzip, zlib) 
• File-handling, breaking files up into packets, reassembling.
• Synchronous API
• Serialisation of stateful requests


Note 2: The tricky question of where the API belongs
We considered 
1. Extending cryptodev
2. New acceldev APIs for device handling + compressdev APIs for data path 
3. New acceldev for all APIs
4. New compressdev API
We've gone with option 4, a compressdev API.  See original RFC [1] for reasons.
We explored wrapping this around a generic acceldev that would be hidden from the API 
but could be common to cryptodev, compressdev and other accelerators on the PMD interface,
but this added complexity and indirection and didn't add enough value, so we've abandoned it.

Opens:
 - Define structures and API for proposed hash functionality
 - Agree on stateful behaviour
 - Complete capability APIs 

Changes in RFC v3:
 - Fixed incorrect op pool element size
 - removed unnecessary rte_compressdev_qp_conf
   and renamed nb_descriptors to max_inflight_ops
 - Removed session_pool from qp setup as not needed
 - Removed session_pool from rte_compressdev_data as not needed
 - renamed CDEV_ to COMPDEV_ to avoid build clash with cryptodev log
 - fixed rte_comp_op_reset() so phys_addr and mempool not deleted.
 - added window size to xforms
 - added lib dependency in rte.app.mk
 - returned PMD value directly in rte_compressdev_get_private_session_size()
   as no need to adjust for hdr size as not supporting sessionless
 - renamed rte_compressdev_session_free()->..terminate()
 - added capability structs
 - added stateless/stateful enum, stream APIs and text about flush flag and stateful behaviour
 - removed intermediate_buffer_size from device info and config structs as not generic.
   Can be handled by PMD-specific config in the config file instead.
 - added max_nb_streams_per_qp to info struct.
 - Fix checkpatch errors.


Changes in RFC v2:
 - rebased off 17.11 and extended to include all device APIs and to compile.
 - Added LZS algorithm
 - Clarified NULL algorithm
 - Use FIXED terminology rather than STATIC for Huffman type
 - removed proposed data verification functionality
 - quashed xform chain into a single xform for each direction
 - added FLUSH_SYNC
 - renamed OVERFLOW to OUT_OF_SPACE and clarified comment
 - added #defines for Level values
 - clarified how src and dst buffer lengths are passed on API

[1] http://dpdk.org/ml/archives/dev/2017-October/078944.html


Trahe, Fiona (1):
  lib: Add compressdev API

 config/common_base                                 |    7 +
 lib/Makefile                                       |    3 +
 lib/librte_compressdev/Makefile                    |   54 +
 lib/librte_compressdev/rte_comp.h                  |  608 ++++++++++
 lib/librte_compressdev/rte_compressdev.c           | 1167 ++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           |  892 +++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.c       |  194 ++++
 lib/librte_compressdev/rte_compressdev_pmd.h       |  533 +++++++++
 lib/librte_compressdev/rte_compressdev_version.map |   50 +
 lib/librte_eal/common/include/rte_log.h            |    1 +
 mk/rte.app.mk                                      |    1 +
 11 files changed, 3510 insertions(+), 0 deletions(-)
 create mode 100644 lib/librte_compressdev/Makefile
 create mode 100644 lib/librte_compressdev/rte_comp.h
 create mode 100644 lib/librte_compressdev/rte_compressdev.c
 create mode 100644 lib/librte_compressdev/rte_compressdev.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.c
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_version.map

Re: [dpdk-dev] [RFC v3 0/1] Compression API in DPDK

[Ahmed Mansour]

Hi Fiona,

On 12/15/2017 11:16 PM, Trahe, Fiona wrote:

> With the vast amounts of data being transported around networks and stored in
> storage systems, reducing data size is becoming ever more important. There
> are both software libraries and hardware devices available that provide
> compression, but no common API. This RFC proposes a compression API for
> DPDK to address this need.
>
> Features:
> • Deflate Algorithm (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc1951&data=02%7C01%7Cahmed.mansour%40nxp.com%7C76241a2796db4701ef0108d54634a70f%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636492114308805852&sdata=yglh48%2F8IuEn%2F7YCL49FlyhGyCnNRX4g4xx2WJQesFs%3D&reserved=0)
> • LZS algorithm (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc2395&data=02%7C01%7Cahmed.mansour%40nxp.com%7C76241a2796db4701ef0108d54634a70f%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636492114308805852&sdata=BlfNe3pRXpEFJE4KS8Spk8bJ5GWPDqADhZYoD7SKvjk%3D&reserved=0)
> • Static and Dynamic Huffman encoding.
> • Compression levels
> • Checksum generation
> • Asynchronous burst API
> • Session-based (a session contains immutable data only and is useable across devices)
> • stream-based to maintain state and history data for stateful flows.
>
> Note 1: Split of functionality above/below API
> When considering whether features should be supported on the API or not, the
> decision was based on the following:
> The purpose of the API is to decouple the application from the compute-intensive
> functions needed for compression by abstracting them under a common API. These
> can then be implemented by either hardware accelerators or optimised software
> libraries. Where features are not compute-intensive and unlikely to be
> offloaded or optimised, there’s nothing to be gained by each PMD having
> to separately implement them, and it makes more sense for them to be done
> above the API. So the following are not handled on the API and can be done above.
> • Prepending/appending protocol headers (gzip, zlib)

Agreed with the notion, however the header and footer handling can be 
added as an option. PMDs can support or not support each format. We 
(NXP) support auto padding of gzip and zlib headers as well as DEFLATE only.

> • File-handling, breaking files up into packets, reassembling.
> • Synchronous API
> • Serialisation of stateful requests
>
>
Is stateful planned for next phase? During design discussions we uncovered many API design
considerations necessary for stateful use. Chained stateful support in the future might not be
possible without compatibility breaking  

> Note 2: The tricky question of where the API belongs
> We considered
> 1. Extending cryptodev
> 2. New acceldev APIs for device handling + compressdev APIs for data path
> 3. New acceldev for all APIs
> 4. New compressdev API
> We've gone with option 4, a compressdev API.  See original RFC [1] for reasons.
> We explored wrapping this around a generic acceldev that would be hidden from the API
> but could be common to cryptodev, compressdev and other accelerators on the PMD interface,
> but this added complexity and indirection and didn't add enough value, so we've abandoned it.
>
Makes sense. compression is common enough to be attempted to be a 
different device category.


> Opens:
>  - Define structures and API for proposed hash functionality
>  - Agree on stateful behaviour

What are the the current thoughts for stateful behavior?

>  - Complete capability APIs

A capability API is very much required as different HW/SW can have 
different capabilities.


regards,
Ahmed

Re: [dpdk-dev] [RFC v3 0/1] Compression API in DPDK

[Verma, Shally]

> -----Original Message-----
> From: Ahmed Mansour [mailto:ahmed.mansour@nxp.com]
> Sent: 19 December 2017 02:37
> To: dev@dpdk.org; Verma, Shally <Shally.Verma@cavium.com>; Hemant
> Agrawal <hemant.agrawal@nxp.com>
> Cc: Hemant Agrawal <hemant.agrawal@nxp.com>; Challa, Mahipal
> <Mahipal.Challa@cavium.com>; fiona.trahe@intel.com; Athreya, Narayana
> Prasad <NarayanaPrasad.Athreya@cavium.com>;
> pablo.de.lara.guarch@intel.com; Roy Pledge <roy.pledge@nxp.com>; Youri
> Querry <youri.querry_1@nxp.com>
> Subject: Re: [RFC v3 0/1] Compression API in DPDK
> 
> Hi Fiona,
> 
> On 12/15/2017 11:16 PM, Trahe, Fiona wrote:
> 
> > With the vast amounts of data being transported around networks and
> stored in
> > storage systems, reducing data size is becoming ever more important.
> There
> > are both software libraries and hardware devices available that provide
> > compression, but no common API. This RFC proposes a compression API for
> > DPDK to address this need.
> >
> > Features:
> > • Deflate Algorithm
> (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fto
> ols.ietf.org%2Fhtml%2Frfc1951&data=02%7C01%7Cahmed.mansour%40nxp.
> com%7C76241a2796db4701ef0108d54634a70f%7C686ea1d3bc2b4c6fa92cd99c
> 5c301635%7C0%7C0%7C636492114308805852&sdata=yglh48%2F8IuEn%2F7YC
> L49FlyhGyCnNRX4g4xx2WJQesFs%3D&reserved=0)
> > • LZS algorithm
> (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fto
> ols.ietf.org%2Fhtml%2Frfc2395&data=02%7C01%7Cahmed.mansour%40nxp.
> com%7C76241a2796db4701ef0108d54634a70f%7C686ea1d3bc2b4c6fa92cd99c
> 5c301635%7C0%7C0%7C636492114308805852&sdata=BlfNe3pRXpEFJE4KS8Sp
> k8bJ5GWPDqADhZYoD7SKvjk%3D&reserved=0)
> > • Static and Dynamic Huffman encoding.
> > • Compression levels
> > • Checksum generation
> > • Asynchronous burst API
> > • Session-based (a session contains immutable data only and is useable
> across devices)
> > • stream-based to maintain state and history data for stateful flows.
> >
> > Note 1: Split of functionality above/below API
> > When considering whether features should be supported on the API or
> not, the
> > decision was based on the following:
> > The purpose of the API is to decouple the application from the compute-
> intensive
> > functions needed for compression by abstracting them under a common
> API. These
> > can then be implemented by either hardware accelerators or optimised
> software
> > libraries. Where features are not compute-intensive and unlikely to be
> > offloaded or optimised, there’s nothing to be gained by each PMD
> having
> > to separately implement them, and it makes more sense for them to be
> done
> > above the API. So the following are not handled on the API and can be
> done above.
> > • Prepending/appending protocol headers (gzip, zlib)
> 
> Agreed with the notion, however the header and footer handling can be
> added as an option. PMDs can support or not support each format. We
> (NXP) support auto padding of gzip and zlib headers as well as DEFLATE only.
> 

[Shally] Just my thoughts on this. For such case, ZLIB and GZIP should be added into algorithm lists. PMD which support them can generate
ZLIB/GZIP styled compressed streams and app can invoke compression with algo such as ZLIB/GZIP. 

Thanks
Shally

> > • File-handling, breaking files up into packets, reassembling.
> > • Synchronous API
> > • Serialisation of stateful requests
> >
> >
> Is stateful planned for next phase? During design discussions we uncovered
> many API design
> considerations necessary for stateful use. Chained stateful support in the
> future might not be
> possible without compatibility breaking
> 
> > Note 2: The tricky question of where the API belongs
> > We considered
> > 1. Extending cryptodev
> > 2. New acceldev APIs for device handling + compressdev APIs for data path
> > 3. New acceldev for all APIs
> > 4. New compressdev API
> > We've gone with option 4, a compressdev API.  See original RFC [1] for
> reasons.
> > We explored wrapping this around a generic acceldev that would be hidden
> from the API
> > but could be common to cryptodev, compressdev and other accelerators
> on the PMD interface,
> > but this added complexity and indirection and didn't add enough value, so
> we've abandoned it.
> >
> Makes sense. compression is common enough to be attempted to be a
> different device category.
> 
> 
> > Opens:
> >  - Define structures and API for proposed hash functionality
> >  - Agree on stateful behaviour
> 
> What are the the current thoughts for stateful behavior?
> 
> >  - Complete capability APIs
> 
> A capability API is very much required as different HW/SW can have
> different capabilities.
> 
> 
> regards,
> Ahmed
> 

Re: [dpdk-dev] [RFC v3 0/1] Compression API in DPDK

[Trahe, Fiona]

Hi Ahmed, 
thanks for your feedback and sorry for the slow response.
Comments below.

> -----Original Message-----
> From: Ahmed Mansour [mailto:ahmed.mansour@nxp.com]
> Sent: Monday, December 18, 2017 9:07 PM
> To: dev@dpdk.org; Shally.Verma@cavium.com; Hemant Agrawal <hemant.agrawal@nxp.com>
> Cc: Hemant Agrawal <hemant.agrawal@nxp.com>; Mahipal.Challa@cavium.com; Trahe, Fiona
> <fiona.trahe@intel.com>; NarayanaPrasad.Athreya@cavium.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; Roy Pledge <roy.pledge@nxp.com>; Youri Querry
> <youri.querry_1@nxp.com>
> Subject: Re: [RFC v3 0/1] Compression API in DPDK
> 
> Hi Fiona,
> 
> On 12/15/2017 11:16 PM, Trahe, Fiona wrote:
> 
> > With the vast amounts of data being transported around networks and stored in
> > storage systems, reducing data size is becoming ever more important. There
> > are both software libraries and hardware devices available that provide
> > compression, but no common API. This RFC proposes a compression API for
> > DPDK to address this need.
> >
> > Features:
> > • Deflate Algorithm
> (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc195
> 1&data=02%7C01%7Cahmed.mansour%40nxp.com%7C76241a2796db4701ef0108d54634a70f%7C686ea
> 1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636492114308805852&sdata=yglh48%2F8IuEn%2F7YCL4
> 9FlyhGyCnNRX4g4xx2WJQesFs%3D&reserved=0)
> > • LZS algorithm
> (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc239
> 5&data=02%7C01%7Cahmed.mansour%40nxp.com%7C76241a2796db4701ef0108d54634a70f%7C686ea
> 1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636492114308805852&sdata=BlfNe3pRXpEFJE4KS8Spk8
> bJ5GWPDqADhZYoD7SKvjk%3D&reserved=0)
> > • Static and Dynamic Huffman encoding.
> > • Compression levels
> > • Checksum generation
> > • Asynchronous burst API
> > • Session-based (a session contains immutable data only and is useable across devices)
> > • stream-based to maintain state and history data for stateful flows.
> >
> > Note 1: Split of functionality above/below API
> > When considering whether features should be supported on the API or not, the
> > decision was based on the following:
> > The purpose of the API is to decouple the application from the compute-intensive
> > functions needed for compression by abstracting them under a common API. These
> > can then be implemented by either hardware accelerators or optimised software
> > libraries. Where features are not compute-intensive and unlikely to be
> > offloaded or optimised, there’s nothing to be gained by each PMD having
> > to separately implement them, and it makes more sense for them to be done
> > above the API. So the following are not handled on the API and can be done above.
> > • Prepending/appending protocol headers (gzip, zlib)
> 
> Agreed with the notion, however the header and footer handling can be
> added as an option. PMDs can support or not support each format. We
> (NXP) support auto padding of gzip and zlib headers as well as DEFLATE only.
[Fiona] We'd like to stabilise the API with the current functionality before adding new features. Our focus now is on delivering a v1 of the full code rather than another iteration of the RFC.
The API will be experimental initially, so I think it should be easy add these later.
 
> > • File-handling, breaking files up into packets, reassembling.
> > • Synchronous API
> > • Serialisation of stateful requests
> >
> >
> Is stateful planned for next phase? During design discussions we uncovered many API design
> considerations necessary for stateful use. Chained stateful support in the future might not be
> possible without compatibility breaking
> 
[Fiona] Stateful is covered in the v3 version. See the thread 
http://dpdk.org/ml/archives/dev/2017-December/084713.html
Have a look at this, though maybe it would be better to hold off on a reply until the v2 version of this doc is posted by Shally.


> > Note 2: The tricky question of where the API belongs
> > We considered
> > 1. Extending cryptodev
> > 2. New acceldev APIs for device handling + compressdev APIs for data path
> > 3. New acceldev for all APIs
> > 4. New compressdev API
> > We've gone with option 4, a compressdev API.  See original RFC [1] for reasons.
> > We explored wrapping this around a generic acceldev that would be hidden from the API
> > but could be common to cryptodev, compressdev and other accelerators on the PMD interface,
> > but this added complexity and indirection and didn't add enough value, so we've abandoned it.
> >
> Makes sense. compression is common enough to be attempted to be a
> different device category.
> 
> 
> > Opens:
> >  - Define structures and API for proposed hash functionality
> >  - Agree on stateful behaviour
> 
> What are the the current thoughts for stateful behavior?
> 
> >  - Complete capability APIs
> 
> A capability API is very much required as different HW/SW can have
> different capabilities.
[Fiona] Agreed. We have work to do to flesh this out  - but expect to do in v1 of actual code
rather than a further RFC iteration

> 
> regards,
> Ahmed
> 

Re: [dpdk-dev] [RFC v3 0/1] Compression API in DPDK

[Ahmed Mansour]

Hi Fiona,

Thanks for your response. I added comments inline

On 12/22/2017 8:44 AM, Trahe, Fiona wrote:
> Hi Ahmed, 
> thanks for your feedback and sorry for the slow response.
> Comments below.
>
>> -----Original Message-----
>> From: Ahmed Mansour [mailto:ahmed.mansour@nxp.com]
>> Sent: Monday, December 18, 2017 9:07 PM
>> To: dev@dpdk.org; Shally.Verma@cavium.com; Hemant Agrawal <hemant.agrawal@nxp.com>
>> Cc: Hemant Agrawal <hemant.agrawal@nxp.com>; Mahipal.Challa@cavium.com; Trahe, Fiona
>> <fiona.trahe@intel.com>; NarayanaPrasad.Athreya@cavium.com; De Lara Guarch, Pablo
>> <pablo.de.lara.guarch@intel.com>; Roy Pledge <roy.pledge@nxp.com>; Youri Querry
>> <youri.querry_1@nxp.com>
>> Subject: Re: [RFC v3 0/1] Compression API in DPDK
>>
>> Hi Fiona,
>>
>> On 12/15/2017 11:16 PM, Trahe, Fiona wrote:
>>
>>> With the vast amounts of data being transported around networks and stored in
>>> storage systems, reducing data size is becoming ever more important. There
>>> are both software libraries and hardware devices available that provide
>>> compression, but no common API. This RFC proposes a compression API for
>>> DPDK to address this need.
>>>
>>> Features:
>>> • Deflate Algorithm
>> (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc195
>> 1&data=02%7C01%7Cahmed.mansour%40nxp.com%7C76241a2796db4701ef0108d54634a70f%7C686ea
>> 1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636492114308805852&sdata=yglh48%2F8IuEn%2F7YCL4
>> 9FlyhGyCnNRX4g4xx2WJQesFs%3D&reserved=0)
>>> • LZS algorithm
>> (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc239
>> 5&data=02%7C01%7Cahmed.mansour%40nxp.com%7C76241a2796db4701ef0108d54634a70f%7C686ea
>> 1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636492114308805852&sdata=BlfNe3pRXpEFJE4KS8Spk8
>> bJ5GWPDqADhZYoD7SKvjk%3D&reserved=0)
>>> • Static and Dynamic Huffman encoding.
>>> • Compression levels
>>> • Checksum generation
>>> • Asynchronous burst API
>>> • Session-based (a session contains immutable data only and is useable across devices)
>>> • stream-based to maintain state and history data for stateful flows.
>>>
>>> Note 1: Split of functionality above/below API
>>> When considering whether features should be supported on the API or not, the
>>> decision was based on the following:
>>> The purpose of the API is to decouple the application from the compute-intensive
>>> functions needed for compression by abstracting them under a common API. These
>>> can then be implemented by either hardware accelerators or optimised software
>>> libraries. Where features are not compute-intensive and unlikely to be
>>> offloaded or optimised, there’s nothing to be gained by each PMD having
>>> to separately implement them, and it makes more sense for them to be done
>>> above the API. So the following are not handled on the API and can be done above.
>>> • Prepending/appending protocol headers (gzip, zlib)
>> Agreed with the notion, however the header and footer handling can be
>> added as an option. PMDs can support or not support each format. We
>> (NXP) support auto padding of gzip and zlib headers as well as DEFLATE only.
> [Fiona] We'd like to stabilise the API with the current functionality before adding new features. Our focus now is on delivering a v1 of the full code rather than another iteration of the RFC.
> The API will be experimental initially, so I think it should be easy add these later.

[Ahmed] The two modes of using the API conflict. If in this API we
assume that the application will package the compressed data by
themselves then maybe we should not provide that service now or in the
future.
On the other hand, most users are familiar with the zlib.net interface
which does the packaging of the headers and footers for the user. I
think most applications will have to write zlib/gzip packagining
software to use this API.
Therefore, It makes sense for DPDK to own that portion of the packaging
I think. I am not sure how to provide that service and simultaneously
provide both a gzip and a zlib packaged output. I guess for the subset
of users who
need both gzip and zlib packaging for the same data, they will have to
do their own packaging. Most other customers will just use the API to
select what format they want right away

>  
>>> • File-handling, breaking files up into packets, reassembling.
>>> • Synchronous API
>>> • Serialisation of stateful requests
>>>
>>>
>> Is stateful planned for next phase? During design discussions we uncovered many API design
>> considerations necessary for stateful use. Chained stateful support in the future might not be
>> possible without compatibility breaking
>>
> [Fiona] Stateful is covered in the v3 version. See the thread 
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdpdk.org%2Fml%2Farchives%2Fdev%2F2017-December%2F084713.html&data=02%7C01%7Cahmed.mansour%40nxp.com%7Cc79921b3d15d468da38e08d549421eb5%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636495470754303375&sdata=xlQdD1hk4N%2BENbB6nNCmBXdR6ycgwyKdrckPpYR2DjE%3D&reserved=0
> Have a look at this, though maybe it would be better to hold off on a reply until the v2 version of this doc is posted by Shally.

[Ahmed] The proposed implementation only allows one op to be in flight
at a time for a particular stream. We see use cases where multiple
stateful in flight pieces of info are sent sequentially

>
>>> Note 2: The tricky question of where the API belongs
>>> We considered
>>> 1. Extending cryptodev
>>> 2. New acceldev APIs for device handling + compressdev APIs for data path
>>> 3. New acceldev for all APIs
>>> 4. New compressdev API
>>> We've gone with option 4, a compressdev API.  See original RFC [1] for reasons.
>>> We explored wrapping this around a generic acceldev that would be hidden from the API
>>> but could be common to cryptodev, compressdev and other accelerators on the PMD interface,
>>> but this added complexity and indirection and didn't add enough value, so we've abandoned it.
>>>
>> Makes sense. compression is common enough to be attempted to be a
>> different device category.
>>
>>
>>> Opens:
>>>  - Define structures and API for proposed hash functionality
>>>  - Agree on stateful behaviour
>> What are the the current thoughts for stateful behavior?
>>
>>>  - Complete capability APIs
>> A capability API is very much required as different HW/SW can have
>> different capabilities.
> [Fiona] Agreed. We have work to do to flesh this out  - but expect to do in v1 of actual code
> rather than a further RFC iteration

[Ahmed] How far apart are we going to allow the different DPDK
implementations to drift? I understood that DPDK is meant to improve
portability.The compatibility API moves the design complextiy one layer up.
I think this will make it harder for application designers which may impact adoption

>> regards,
>> Ahmed
>>
>