From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR01-HE1-obe.outbound.protection.outlook.com (mail-he1eur01on0081.outbound.protection.outlook.com [104.47.0.81]) by dpdk.org (Postfix) with ESMTP id 7A9A21B1E1 for ; Tue, 9 Jan 2018 18:36:14 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=hMPXOLwlijwYS/od231woHAEDZ2+0PvT0P1KWXGwKwQ=; b=JCB0csdOBPwZFq2kdc1SlPJfFrt1QDPq44R0xooeM5/FyT6t2r9xBLjPdpHgh1JQW+vYk7KVAlw66UsU6C50nnxupsixK2UOAHecwEiQmdTxBBPOSJefC3BxO4MH5mnqikzj04+UpsWCe/spxCZVtMYp20FOqWVg54ZBj4lZOyk= Received: from AM0PR0402MB3842.eurprd04.prod.outlook.com (52.133.39.138) by AM3PR04MB0760.eurprd04.prod.outlook.com (10.160.6.146) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.386.5; Tue, 9 Jan 2018 17:36:13 +0000 Received: from AM0PR0402MB3842.eurprd04.prod.outlook.com ([fe80::dce3:8486:f9d6:d566]) by AM0PR0402MB3842.eurprd04.prod.outlook.com ([fe80::dce3:8486:f9d6:d566%13]) with mapi id 15.20.0386.006; Tue, 9 Jan 2018 17:36:12 +0000 From: Ahmed Mansour To: "Trahe, Fiona" , "dev@dpdk.org" , "Shally.Verma@cavium.com" , Hemant Agrawal CC: "Mahipal.Challa@cavium.com" , "NarayanaPrasad.Athreya@cavium.com" , "De Lara Guarch, Pablo" , Roy Pledge , Youri Querry Thread-Topic: [RFC v3 0/1] Compression API in DPDK Thread-Index: AQHTeEQmcjdXftTxmkKPJr/pHav7/g== Date: Tue, 9 Jan 2018 17:36:11 +0000 Message-ID: References: <1511542480-389-1-git-send-email-fiona.trahe@intel.com> <1513359963-13817-1-git-send-email-fiona.trahe@intel.com> <1513359963-13817-1-git-send-email-fiona.trahe@intel.com> <348A99DA5F5B7549AA880327E580B435892E98F4@IRSMSX101.ger.corp.intel.com> Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=ahmed.mansour@nxp.com; x-originating-ip: [192.88.168.1] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; AM3PR04MB0760; 7:yhn3Rk1JvWADKJ9h8/9XQH0qu5sGheDVCeeTJCg8i/u6lKOHOifyXJ1bO/xbNvUC46in1DmyQSUsKUcDMEPmcI/UQ4kmauoBnKnfpSS9c5lVp/XP00yTnp29TJ2Qd8cgCCiyI9V5T8tOl788qxPRhV00ORBD73MWbGk0CEZLr99JqaXVK0qvOu4EbB6g0yk1GTqjmLaw2wYZFB6Vk4Dlc8k4W5yiCpFn61Tmw6Ik160tOPHTiUuz0+7w4tiTJkLS x-ms-exchange-antispam-srfa-diagnostics: SSOS;SSOR; x-forefront-antispam-report: SFV:SKI; SCL:-1; SFV:NSPM; SFS:(10009020)(366004)(346002)(396003)(376002)(39860400002)(39380400002)(24454002)(13464003)(189003)(199004)(66066001)(478600001)(68736007)(5250100002)(6246003)(6636002)(3280700002)(3660700001)(2201001)(966005)(81166006)(2900100001)(53546011)(7696005)(59450400001)(4326008)(86362001)(8936002)(53936002)(76176011)(2906002)(6116002)(3846002)(6506007)(5660300001)(25786009)(14454004)(45080400002)(99286004)(6436002)(106356001)(102836004)(74316002)(54906003)(110136005)(8676002)(7736002)(33656002)(316002)(305945005)(55016002)(9686003)(81156014)(93886005)(105586002)(97736004)(6306002)(229853002)(2501003); DIR:OUT; SFP:1101; SCL:1; SRVR:AM3PR04MB0760; H:AM0PR0402MB3842.eurprd04.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 97d41305-a676-4c14-93b2-08d5578779e7 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(48565401081)(4534020)(4602075)(4627115)(201703031133081)(201702281549075)(5600026)(4604075)(3008032)(2017052603307)(7153060)(7193020); SRVR:AM3PR04MB0760; x-ms-traffictypediagnostic: AM3PR04MB0760: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(278428928389397)(189930954265078)(185117386973197)(45079756050767)(228905959029699); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040470)(2401047)(5005006)(8121501046)(93006095)(93001095)(3002001)(10201501046)(3231023)(944501110)(6055026)(6041268)(20161123562045)(20161123558120)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(6072148)(201708071742011); SRVR:AM3PR04MB0760; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:AM3PR04MB0760; x-forefront-prvs: 0547116B72 received-spf: None (protection.outlook.com: nxp.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: J1rHkKwM81bp3Kum6cbENoGtgI82a3429rm/7zCGKExI2tgkfeYfwVIjb4GqHrtQf/6XCzU9C7m5vQVbixOiPw== spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: 97d41305-a676-4c14-93b2-08d5578779e7 X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Jan 2018 17:36:11.8964 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM3PR04MB0760 Subject: Re: [dpdk-dev] [RFC v3 0/1] Compression API in DPDK X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Jan 2018 17:36:14 -0000 Hi Fiona,=0A= =0A= Thanks for your response. I added comments inline=0A= =0A= On 12/22/2017 8:44 AM, Trahe, Fiona wrote:=0A= > Hi Ahmed, =0A= > thanks for your feedback and sorry for the slow response.=0A= > Comments below.=0A= >=0A= >> -----Original Message-----=0A= >> From: Ahmed Mansour [mailto:ahmed.mansour@nxp.com]=0A= >> Sent: Monday, December 18, 2017 9:07 PM=0A= >> To: dev@dpdk.org; Shally.Verma@cavium.com; Hemant Agrawal =0A= >> Cc: Hemant Agrawal ; Mahipal.Challa@cavium.com; = Trahe, Fiona=0A= >> ; NarayanaPrasad.Athreya@cavium.com; De Lara Guar= ch, Pablo=0A= >> ; Roy Pledge ; Youri= Querry=0A= >> =0A= >> Subject: Re: [RFC v3 0/1] Compression API in DPDK=0A= >>=0A= >> Hi Fiona,=0A= >>=0A= >> On 12/15/2017 11:16 PM, Trahe, Fiona wrote:=0A= >>=0A= >>> With the vast amounts of data being transported around networks and sto= red in=0A= >>> storage systems, reducing data size is becoming ever more important. Th= ere=0A= >>> are both software libraries and hardware devices available that provide= =0A= >>> compression, but no common API. This RFC proposes a compression API for= =0A= >>> DPDK to address this need.=0A= >>>=0A= >>> Features:=0A= >>> =E2=80=A2 Deflate Algorithm=0A= >> (https://emea01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fto= ols.ietf.org%2Fhtml%2Frfc195=0A= >> 1&data=3D02%7C01%7Cahmed.mansour%40nxp.com%7C76241a2796db4701ef0108d5463= 4a70f%7C686ea=0A= >> 1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636492114308805852&sdata=3Dyglh48%= 2F8IuEn%2F7YCL4=0A= >> 9FlyhGyCnNRX4g4xx2WJQesFs%3D&reserved=3D0)=0A= >>> =E2=80=A2 LZS algorithm=0A= >> (https://emea01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fto= ols.ietf.org%2Fhtml%2Frfc239=0A= >> 5&data=3D02%7C01%7Cahmed.mansour%40nxp.com%7C76241a2796db4701ef0108d5463= 4a70f%7C686ea=0A= >> 1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636492114308805852&sdata=3DBlfNe3p= RXpEFJE4KS8Spk8=0A= >> bJ5GWPDqADhZYoD7SKvjk%3D&reserved=3D0)=0A= >>> =E2=80=A2 Static and Dynamic Huffman encoding.=0A= >>> =E2=80=A2 Compression levels=0A= >>> =E2=80=A2 Checksum generation=0A= >>> =E2=80=A2 Asynchronous burst API=0A= >>> =E2=80=A2 Session-based (a session contains immutable data only and is = useable across devices)=0A= >>> =E2=80=A2 stream-based to maintain state and history data for stateful = flows.=0A= >>>=0A= >>> Note 1: Split of functionality above/below API=0A= >>> When considering whether features should be supported on the API or not= , the=0A= >>> decision was based on the following:=0A= >>> The purpose of the API is to decouple the application from the compute-= intensive=0A= >>> functions needed for compression by abstracting them under a common API= . These=0A= >>> can then be implemented by either hardware accelerators or optimised so= ftware=0A= >>> libraries. Where features are not compute-intensive and unlikely to be= =0A= >>> offloaded or optimised, there=E2=80=99s nothing to be gained by each PM= D having=0A= >>> to separately implement them, and it makes more sense for them to be do= ne=0A= >>> above the API. So the following are not handled on the API and can be d= one above.=0A= >>> =E2=80=A2 Prepending/appending protocol headers (gzip, zlib)=0A= >> Agreed with the notion, however the header and footer handling can be=0A= >> added as an option. PMDs can support or not support each format. We=0A= >> (NXP) support auto padding of gzip and zlib headers as well as DEFLATE o= nly.=0A= > [Fiona] We'd like to stabilise the API with the current functionality bef= ore adding new features. Our focus now is on delivering a v1 of the full co= de rather than another iteration of the RFC.=0A= > The API will be experimental initially, so I think it should be easy add = these later.=0A= =0A= [Ahmed] The two modes of using the API conflict. If in this API we=0A= assume that the application will package the compressed data by=0A= themselves then maybe we should not provide that service now or in the=0A= future.=0A= On the other hand, most users are familiar with the zlib.net interface=0A= which does the packaging of the headers and footers for the user. I=0A= think most applications will have to write zlib/gzip packagining=0A= software to use this API.=0A= Therefore, It makes sense for DPDK to own that portion of the packaging=0A= I think. I am not sure how to provide that service and simultaneously=0A= provide both a gzip and a zlib packaged output. I guess for the subset=0A= of users who=0A= need both gzip and zlib packaging for the same data, they will have to=0A= do their own packaging. Most other customers will just use the API to=0A= select what format they want right away=0A= =0A= > =0A= >>> =E2=80=A2 File-handling, breaking files up into packets, reassembling.= =0A= >>> =E2=80=A2 Synchronous API=0A= >>> =E2=80=A2 Serialisation of stateful requests=0A= >>>=0A= >>>=0A= >> Is stateful planned for next phase? During design discussions we uncover= ed many API design=0A= >> considerations necessary for stateful use. Chained stateful support in t= he future might not be=0A= >> possible without compatibility breaking=0A= >>=0A= > [Fiona] Stateful is covered in the v3 version. See the thread =0A= > https://emea01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fdpdk.= org%2Fml%2Farchives%2Fdev%2F2017-December%2F084713.html&data=3D02%7C01%7Cah= med.mansour%40nxp.com%7Cc79921b3d15d468da38e08d549421eb5%7C686ea1d3bc2b4c6f= a92cd99c5c301635%7C0%7C0%7C636495470754303375&sdata=3DxlQdD1hk4N%2BENbB6nNC= mBXdR6ycgwyKdrckPpYR2DjE%3D&reserved=3D0=0A= > Have a look at this, though maybe it would be better to hold off on a rep= ly until the v2 version of this doc is posted by Shally.=0A= =0A= [Ahmed] The proposed implementation only allows one op to be in flight=0A= at a time for a particular stream. We see use cases where multiple=0A= stateful in flight pieces of info are sent sequentially=0A= =0A= >=0A= >>> Note 2: The tricky question of where the API belongs=0A= >>> We considered=0A= >>> 1. Extending cryptodev=0A= >>> 2. New acceldev APIs for device handling + compressdev APIs for data pa= th=0A= >>> 3. New acceldev for all APIs=0A= >>> 4. New compressdev API=0A= >>> We've gone with option 4, a compressdev API. See original RFC [1] for = reasons.=0A= >>> We explored wrapping this around a generic acceldev that would be hidde= n from the API=0A= >>> but could be common to cryptodev, compressdev and other accelerators on= the PMD interface,=0A= >>> but this added complexity and indirection and didn't add enough value, = so we've abandoned it.=0A= >>>=0A= >> Makes sense. compression is common enough to be attempted to be a=0A= >> different device category.=0A= >>=0A= >>=0A= >>> Opens:=0A= >>> - Define structures and API for proposed hash functionality=0A= >>> - Agree on stateful behaviour=0A= >> What are the the current thoughts for stateful behavior?=0A= >>=0A= >>> - Complete capability APIs=0A= >> A capability API is very much required as different HW/SW can have=0A= >> different capabilities.=0A= > [Fiona] Agreed. We have work to do to flesh this out - but expect to do = in v1 of actual code=0A= > rather than a further RFC iteration=0A= =0A= [Ahmed] How far apart are we going to allow the different DPDK=0A= implementations to drift? I understood that DPDK is meant to improve=0A= portability.The compatibility API moves the design complextiy one layer up.= =0A= I think this will make it harder for application designers which may impact= adoption=0A= =0A= >> regards,=0A= >> Ahmed=0A= >>=0A= >=0A= =0A=