From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM03-DM3-obe.outbound.protection.outlook.com (mail-dm3nam03on0073.outbound.protection.outlook.com [104.47.41.73]) by dpdk.org (Postfix) with ESMTP id 282A12BC7 for ; Mon, 26 Feb 2018 12:25:57 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=z5lyimx8lyXMW3WmCUrjjy8GSo3kRbtIBbxDQWoVH3I=; b=AmOyMCBf6L4P3i5MREDtEpnll82kOv82X7lILHniKerTSSnpBmt8u2/Yf9MUeAJ2/cDiW/Z9DxT9YOA/bj9ZBdwLY3VPn3zJ1eiJgs50aJKTT3D3iSx5GIi+M2+IPDKW/VPLFNoUuG1VaK6G0axDYTltgOph9t9pIzuQrQV4aBA= Received: from CY4PR0701MB3634.namprd07.prod.outlook.com (52.132.101.164) by CY4PR0701MB3620.namprd07.prod.outlook.com (52.132.101.162) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.527.15; Mon, 26 Feb 2018 11:25:55 +0000 Received: from CY4PR0701MB3634.namprd07.prod.outlook.com ([fe80::e842:2eb2:2ff5:5dd6]) by CY4PR0701MB3634.namprd07.prod.outlook.com ([fe80::e842:2eb2:2ff5:5dd6%13]) with mapi id 15.20.0527.021; Mon, 26 Feb 2018 11:25:54 +0000 From: "Verma, Shally" To: Ahmed Mansour , "Trahe, Fiona" , "dev@dpdk.org" CC: "De Lara Guarch, Pablo" , "Athreya, Narayana Prasad" , "Gupta, Ashish" , "Sahu, Sunila" , "Challa, Mahipal" , "Jain, Deepak K" , Hemant Agrawal , Roy Pledge , Youri Querry Thread-Topic: [dpdk-dev] [PATCH] compressdev: implement API Thread-Index: AQHTnFM4Vpm7OG1Ltk6wDJVFHhqAyaO2Xauw Date: Mon, 26 Feb 2018 11:25:54 +0000 Message-ID: References: <1517595924-25963-1-git-send-email-fiona.trahe@intel.com> <12544144.czVLKRyaz4@xps> <348A99DA5F5B7549AA880327E580B43589325187@IRSMSX101.ger.corp.intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Shally.Verma@cavium.com; x-originating-ip: [115.113.156.2] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; CY4PR0701MB3620; 7:cPwdy24zHozm4aD1dU/KLZpuFfK7R7dA8gM2hJ/xv0YnFJgMK2ASxSD6VFifcfqWS0aAM17wY1XKfvfwAM0uIMEA1JtHvE9K/MEwSpQmsUoVcyEtK5y51A8XvINANrtxGhvWrjreOjoGkpD2HQUN9WrnE4CYYksN0/T2pXEHGpjXfu4SSxJIjniaXS0a7MLVkt16rxk5X5+yRUTy7Fpn/KKPzMVmbNu20phOyAQXaX/YCC7hWbz/FHT49BxstXaD x-ms-exchange-antispam-srfa-diagnostics: SSOS;SSOR; x-forefront-antispam-report: SFV:SKI; SCL:-1; SFV:NSPM; SFS:(10009020)(39850400004)(39380400002)(396003)(346002)(366004)(376002)(199004)(189003)(13464003)(25786009)(99286004)(2900100001)(4326008)(54906003)(110136005)(229853002)(5250100002)(105586002)(5890100001)(2501003)(3280700002)(7696005)(3660700001)(2906002)(93886005)(8656006)(97736004)(6246003)(86362001)(8936002)(102836004)(14454004)(55236004)(76176011)(72206003)(8676002)(26005)(66066001)(68736007)(561944003)(186003)(6506007)(33656002)(81166006)(7736002)(2950100002)(6436002)(81156014)(305945005)(59450400001)(55016002)(74316002)(3846002)(9686003)(106356001)(6116002)(53936002)(478600001)(316002)(5660300001); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR0701MB3620; H:CY4PR0701MB3634.namprd07.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; x-ms-office365-filtering-correlation-id: 8a9421aa-b3e8-4395-cf45-08d57d0bb304 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(4534165)(7168020)(4627221)(201703031133081)(201702281549075)(5600026)(4604075)(3008032)(2017052603307)(7153060)(7193020); SRVR:CY4PR0701MB3620; x-ms-traffictypediagnostic: CY4PR0701MB3620: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(192374486261705)(131327999870524)(185117386973197)(228905959029699); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040501)(2401047)(8121501046)(5005006)(3002001)(10201501046)(3231220)(944501161)(52105095)(93006095)(93001095)(6041288)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(20161123560045)(20161123564045)(6072148)(201708071742011); SRVR:CY4PR0701MB3620; BCL:0; PCL:0; RULEID:; SRVR:CY4PR0701MB3620; x-forefront-prvs: 05954A7C45 received-spf: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: Bt+lIXqtit2lMxQ3AKYMndVSz39gw9Lb0GxTSdzp0GmrVTfqT1XrTc/eHF1GHnD9efCnF0Uu83fHZfBd+VlUWXidD+WOy7ExjFn3miQDmH+g7Y+toBbK9y/wzUMze2dEUz8ZkUjf13sagzUV4gjNDQvThWXj3EvkbtDwM5DI8yE= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: cavium.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8a9421aa-b3e8-4395-cf45-08d57d0bb304 X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Feb 2018 11:25:54.2623 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR0701MB3620 Subject: Re: [dpdk-dev] [PATCH] compressdev: implement API X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Feb 2018 11:25:57 -0000 Hi Fiona, Ahmed >-----Original Message----- >From: Ahmed Mansour [mailto:ahmed.mansour@nxp.com] >Sent: 24 February 2018 06:47 >To: Trahe, Fiona ; dev@dpdk.org; Verma, Shally >Cc: De Lara Guarch, Pablo ; Athreya, Naray= ana Prasad ; >Gupta, Ashish ; Sahu, Sunila ; Challa, Mahipal >; Jain, Deepak K ; Hem= ant Agrawal ; Roy >Pledge ; Youri Querry >Subject: Re: [dpdk-dev] [PATCH] compressdev: implement API > >Hi Fiona, > >Thanks for starting this discussion. In the current API the user must >make 12 API calls just to get information to compress. Maybe there is a >way to simplify. At least for some use cases (stateless). I think a call >sometime next week would be good to help clarify coalesce some of the >complexity. > >I added specific comments inline. > >Thanks, > >Ahmed > >On 2/21/2018 2:12 PM, Trahe, Fiona wrote: >> We've been struggling with the idea of session in compressdev. >> >> Is it really a session? >> - It's not in the same sense as cryptodev where it's used to hold a key= , and maps to a Security Association. >> - It's a set of immutable data that is needed with the op and stream to= perform the operation. >> - It inherited from cryptodev the ability to be set up for multiple dri= ver types and used across any >> devices of those types. For stateful ops this facility can't be used= . >> For stateless we don't think it's important, and think it's unlikely= to be used. >> - Drivers use it to prepare private data, set up resources, do pre-work= , so there's >> less work to be done on the data path. Initially we didn't have a st= ream, we do now, >> this may be a better alternative place for that work. >> So we've been toying with the idea of getting rid of the session. >[Ahmed] In our proprietary API the stream and session are one. A session >holds many properties like the op-type, instead of having this >information in the op itself. This way we lower the per op setup cost. >This also allows rapid reuse of stateful infrastructure, once a stream >is closed on a stateful session, the next op (stream) on this session >reuses the stateful storage. Obviously if a stream is in "pause mode" on >a session, all following ops that may be unrelated to this >stream/session must also wait until this current stream is closed or >aborted before the infrastructure can be reused. >> >> We also struggle with the idea of setting up a stream for stateless ops. >> - Well, really I just think the name is misleading, i.e. there's no pr= oblem with setting >> up some private PMD data to use with stateless operations, just call= ing it a >> stream doesn't seem right. >[Ahmed] I agree. The op has all the necessary information to process it >in the current API? Both the stream and the op are one time use. We >can't attach multiple similar ops to a single stream/session and rely on >their properties to simplify op setup, so why the hassle. [Shally] As per my knowledge, session came with idea in DPDK, if system ha= s multiple devices setup to do similar jobs then application can fan out op= s to any of them for load-balancing. Though it is not possible for stateful= ops but it still can work for stateless. If there's an application which o= nly have stateless ops to process then I see this is still useful feature t= o support. In current proposal, stream logically represent data and hold its specific = information and session is generic information that can be applied on multi= ple data. If we want to combine stream and session. Then one way to look at= this is: "let application only allocate and initialize session with rte_comp_xform (= and possibly op type) information so that PMD can do one-time setup and all= ocate enough resources. Once attached to op, cannot be reused until that op= is fully processed. So, if app has 16 data elements to process in a burst,= it will setup 16 sessions." This is same as what Ahmed suggested. For a particular load-balancing case = suggested above, If application want, can initialize different sessions on = multiple devices with same xform so that each is prepared to process ops. = Application can then fanout stateless ops to multiple devices for load-bala= ncing but then it would need to keep map of device & a session map. If this sound feasible, then I too believe we can rather get rid of either = and keep one (possibly session but am open with stream as well).=20 However, regardless of case whether we live with name stream or session, I = don't see much deviation from current API spec except description and few m= odifications/additions as identified. So, then I see it as: - A stream(or session whichever name is chosen) can be used with only one-o= p at-a-time - It can be re-used when previously attached op is processed - if it is stream then currently it is allocated from PMD managed pool whe= reas Sessions are allocated from application created mempool. In either of case, I would expect to review pool management API With this in mind, below are few of my comments >> >> So putting above thoughts together I want to propose: >> - Removal of the session and all associated APIs. >> - Passing in one of three data types in the rte_comp_op >> >> union { >> struct rte_comp_xform *xform; >> /**< Immutable compress/decompress params */ >> void *pmd_stateless_data; >> /**< Stateless private PMD data derived from an rte_comp_xform >> * rte_comp_stateless_data_init() must be called on a device >> * before sending any STATELESS operations. If the PMD returns a= non-NULL >> * value the handle must be attached to subsequent STATELESS ope= rations. >> * If a PMD returns NULL, then the xform should be passed direct= ly to each op >> */ [Shally] It sounds like stateless_data_init() nothing more than a replaceme= nt of session_init().=20 So, this is needed neither if we retain session concept nor if we retain s= tream concept ( rte_comp_stream_create() with op_type: stateless can serve = same purpose).=20 It should be sufficient to provide either stream (or session) pointer. >> void *stream; >> /* Private PMD data derived initially from an rte_comp_xform, wh= ich holds state >> * and history data and evolves as operations are processed. >> * rte_comp_stream_create() must be called on a device for all S= TATEFUL >> * data streams and the resulting stream attached >> * to the one or more operations associated with the data stream= . >> * All operations in a stream must be sent to the same device. >> */ >> } >[Ahmed] I like this setup, but I am not sure in what cases the xform >immutable would be used. I understand the other two. [Shally] my understanding is xform will be mapped by PMD to its internally = managed stream(or session data structure). And then we can remove STATEFUL= reference here and just say stream(or session) it belongs to. However, Thi= s condition still apply: *All operations that belong to same stream must be sent to the same = device.* >> Notes: >> 1. Internally if a PMD wants to use the exact same data structure for bo= th it can do, >> just on the API I think it's better if they're named differently wi= th >> different comments. >> 2. I'm not clear of the constraints if any, which attach to the pmd_stat= eless_data >> For our PMD it would only hold immutable data as the session did, a= nd so >> could be attached to many ops in parallel. >> Is this true for all PMDs or are there constraints which should be = called out? >> Is it limited to a specific device, qp, or to be used on one op at = a time? >> 3. Am open to other naming suggestions, just trying to capture the essen= ce >> of these data structs better than our current API does. >> >> We would put some more helper fns and structure around the above code if= people >> are in agreement, just want to see if the concept flies before going fur= ther? >> >> Fiona >> >> >>