From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR01-HE1-obe.outbound.protection.outlook.com (mail-he1eur01on0065.outbound.protection.outlook.com [104.47.0.65]) by dpdk.org (Postfix) with ESMTP id D65A91B2BD for ; Thu, 15 Feb 2018 20:51:24 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=GQrAEV398sH0ZBOhzHiTIOsbpePV1phrQ9dN1mwbPek=; b=QI8341pjkyhAhar4mV49wWVfodD13oGDx1k7q9YJDQ6qe7XTHo1Bu8UA5NS7ZgYpXgxS5hxLfkzoMo0JfpIeK+sRDiKjr2brDh0uCDVXcSiafurVagv3rFRdui9D9ETPLMHzKA1IcdPC88o399mBI/68BPIhp5yPLJHtPTSWuqc= Received: from AM0PR0402MB3842.eurprd04.prod.outlook.com (52.133.39.138) by AM0PR0402MB3716.eurprd04.prod.outlook.com (52.133.38.156) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.485.10; Thu, 15 Feb 2018 19:51:23 +0000 Received: from AM0PR0402MB3842.eurprd04.prod.outlook.com ([fe80::28a2:ee3e:4f18:5f86]) by AM0PR0402MB3842.eurprd04.prod.outlook.com ([fe80::28a2:ee3e:4f18:5f86%13]) with mapi id 15.20.0485.015; Thu, 15 Feb 2018 19:51:21 +0000 From: Ahmed Mansour To: "Trahe, Fiona" , "Verma, Shally" , "dev@dpdk.org" CC: "Athreya, Narayana Prasad" , "Gupta, Ashish" , "Sahu, Sunila" , "De Lara Guarch, Pablo" , "Challa, Mahipal" , "Jain, Deepak K" , Hemant Agrawal , Roy Pledge , Youri Querry Thread-Topic: [RFC v2] doc compression API for DPDK Thread-Index: AdOFUW8Wdt99b3u6RKydGSrxJwvtHg== Date: Thu, 15 Feb 2018 19:51:21 +0000 Message-ID: References: <348A99DA5F5B7549AA880327E580B435892F589D@IRSMSX101.ger.corp.intel.com> <348A99DA5F5B7549AA880327E580B43589315232@IRSMSX101.ger.corp.intel.com> <348A99DA5F5B7549AA880327E580B43589315AF3@IRSMSX101.ger.corp.intel.com> <348A99DA5F5B7549AA880327E580B4358931F4E3@IRSMSX101.ger.corp.intel.com> Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=ahmed.mansour@nxp.com; x-originating-ip: [192.88.168.1] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; AM0PR0402MB3716; 7:Mdyot8rng+b1bG+Krf++lM6x1msDiKo1zXnMa/Nw96V9JozaQ5WLqss9QdZ6eFz0C80YwVu0j1T+ei7zWLUndNPBf0HsEe5t0QA2pNfUsnHc2m5Fj+1G8yGh95WlDIvP9m82R3gFxuc5SN6Llu/Gpj5pHCP5DaddOTgDE0eRX+72WTg6x4L5jbyf5IDWVbiuZ6dcFF+s2JzWv442ovG69WjQRGnc41gpOCW1TPArSP1IH1LCa0corhmhFtKllPUi x-ms-exchange-antispam-srfa-diagnostics: SSOS;SSOR; x-forefront-antispam-report: SFV:SKI; SCL:-1; SFV:NSPM; SFS:(10009020)(366004)(396003)(346002)(39380400002)(376002)(39860400002)(189003)(199004)(57704003)(81166006)(59450400001)(478600001)(26005)(6246003)(8676002)(102836004)(105586002)(97736004)(76176011)(4326008)(33656002)(6506007)(6436002)(316002)(53936002)(99286004)(5250100002)(110136005)(93886005)(106356001)(7696005)(186003)(54906003)(25786009)(66066001)(2906002)(2900100001)(9686003)(68736007)(55016002)(86362001)(305945005)(3280700002)(14454004)(2501003)(7736002)(5660300001)(6116002)(81156014)(3660700001)(74316002)(3846002)(229853002)(8936002); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR0402MB3716; H:AM0PR0402MB3842.eurprd04.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: e9be44df-874e-472a-163e-08d574ad7d12 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(48565401081)(5600026)(4604075)(3008032)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603307)(7153060)(7193020); SRVR:AM0PR0402MB3716; x-ms-traffictypediagnostic: AM0PR0402MB3716: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(278428928389397); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040501)(2401047)(5005006)(8121501046)(3231101)(2400082)(944501161)(3002001)(10201501046)(93006095)(93001095)(6055026)(6041288)(20161123558120)(20161123560045)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(6072148)(201708071742011); SRVR:AM0PR0402MB3716; BCL:0; PCL:0; RULEID:; SRVR:AM0PR0402MB3716; x-forefront-prvs: 058441C12A received-spf: None (protection.outlook.com: nxp.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: aTbVsXkX/8yvAN0l9XCJnLe8EZQJsuIYrzz3QW+Wf5JFahEvBbhuZ+REKE+RqLglC/og001eovnfjLgrpLk29w== spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: e9be44df-874e-472a-163e-08d574ad7d12 X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Feb 2018 19:51:21.8513 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR0402MB3716 Subject: Re: [dpdk-dev] [RFC v2] doc compression API for DPDK X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2018 19:51:25 -0000 /// snip ///=0A= >>>>>=0A= >>>>>>>> [Fiona] I propose if BFINAL bit is detected before end of input=0A= >>>>>>>> the decompression should stop. In this case consumed will be < src= .length.=0A= >>>>>>>> produced will be < dst buffer size. Do we need an extra STATUS res= ponse?=0A= >>>>>>>> STATUS_BFINAL_DETECTED ?=0A= >>>>>>> [Shally] @fiona, I assume you mean here decompressor stop after pro= cessing Final block right?=0A= >>>>>> [Fiona] Yes.=0A= >>>>>>=0A= >>>>>> And if yes,=0A= >>>>>>> and if it can process that final block successfully/unsuccessfully,= then status could simply be=0A= >>>>>>> SUCCESS/FAILED.=0A= >>>>>>> I don't see need of specific return code for this use case. Just to= share, in past, we have practically=0A= >> run into=0A= >>>>>>> such cases with boost lib, and decompressor has simply worked this = way.=0A= >>>>>> [Fiona] I'm ok with this.=0A= >>>>>>=0A= >>>>>>>> Only thing I don't like this is it can impact on performance, as n= ormally=0A= >>>>>>>> we can just look for STATUS =3D=3D SUCCESS. Anything else should b= e an exception.=0A= >>>>>>>> Now the application would have to check for SUCCESS || BFINAL_DETE= CTED every time.=0A= >>>>>>>> Do you have a suggestion on how we should handle this?=0A= >>>>>>>>=0A= >>>>> [Ahmed] This makes sense. So in all cases the PMD should assume that = it=0A= >>>>> should stop as soon as a BFINAL is observed.=0A= >>>>>=0A= >>>>> A question. What happens ins stateful vs stateless modes when=0A= >>>>> decompressing an op that encompasses multiple BFINALs. I assume the= =0A= >>>>> caller in that case will use the consumed=3Dx bytes to find out how f= ar in=0A= >>>>> to the input is the end of the first stream and start from the next= =0A= >>>>> byte. Is this correct?=0A= >>>> [Shally] As per my understanding, each op can be tied up to only one = stream as we have only one=0A= >> stream pointer per op and one=0A= >>> stream can have only one BFINAL (as stream is one complete compressed d= ata) but looks like you're=0A= >> suggesting a case where one op=0A= >>> can carry multiple independent streams? and thus multiple BFINAL?! , su= ch as, below here is op=0A= >> pointing to more than one streams=0A= >>>> --------------------------------------------=0A= >>>> op --> |stream1|stream2| |stream3|=0A= >>>> --------------------------------------------=0A= >>>>=0A= >>>> Could you confirm if I understand your question correct?=0A= >>> [Ahmed] Correct. We found that in some storage applications the user=0A= >>> does not know where exactly the BFINAL is. They rely on zlib software= =0A= >>> today. zlib.net software halts at the first BFINAL. Users put multiple= =0A= >>> streams in one op and rely on zlib to stop and inform them of the end= =0A= >>> location of the first stream.=0A= >> [Shally] Then this is practically case possible on decompressor and deco= mpressor doesn't regard flush=0A= >> flag. So in that case, I expect PMD to internally reset themselves (say = in case of zlib going through cycle=0A= >> of deflateEnd and deflateInit or deflateReset) and return with status = =3D SUCCESS with updated produced=0A= >> and consumed. Now in such case, if previous stream also has some footer = followed by start of next=0A= >> stream, then I am not sure how PMD / lib can support that case. Have you= had practically run of such=0A= >> use-case on zlib? If yes, how then such application handle it in your ex= perience?=0A= >> I can imagine for such input zlib would return with Z_FLUSH_END after 1s= t BFINAL is processed to the=0A= >> user. Then application doing deflateReset() or Init-End() cycle before s= tarting with next. But if it starts=0A= >> with input that doesn't have valid zlib header, then likely it will thro= w an error.=0A= >>=0A= > [Fiona] The consumed and produced tell the Application hw much data was p= rocessed up to =0A= > the end of the first deflate block encountered with a bfinal set.=0A= > If there is data, e.g. footer after the block with bfinal, then I think i= t must be the responsibility of=0A= > the application to know this, the PMD can't have any responsibility for t= his.=0A= > The next op sent to the PMD must start with a valid deflate block.=0A= [Ahmed] Agreed. This is exactly what I expected. In our case we support=0A= gzip and zlib header/footer processing, but that does not fundamentally=0A= change the setup. The user may have other meta data after the footer=0A= which the PMD is not responsible for. The PMD should stop processing=0A= depending on the mode. In raw DEFLATE, it should stop immediately. In=0A= other modes it should stop after the footer. We also have a mode in our=0A= PMD to simply continue decompression. In that case there cannot be=0A= header/footer between streams in raw DEFLATE. That mode can be enabled=0A= perhaps at the session level in the future with a session parameter at=0A= setup time. We call it "member continue". In this mode the PMD plows=0A= through as much of the op as possible. If it hits incorrectly setup data=0A= then it returns what it did decompress successfully and the error code=0A= in decompressing the data afterwards.=0A= >=0A= >=0A= >>>> Thanks=0A= >>>> Shally=0A= >>>>=0A= >=0A= =0A=