From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by dpdk.org (Postfix) with ESMTP id 5F9C110A3 for ; Fri, 18 Jan 2019 16:44:53 +0100 (CET) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id x0IFeqhq120125; Fri, 18 Jan 2019 15:44:52 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=mime-version : message-id : date : from : sender : to : cc : subject : references : in-reply-to : content-type : content-transfer-encoding; s=corp-2018-07-02; bh=rY1VmE0gMsBFvL3MXoGfJgQbuO1Utdl5konHru2xAEk=; b=ndsB8Vnfv/mQFVtGRdhXwMMgKxbWu4ysUYdx5W3p/AuNlwigbl53DgAHi6gxyJahQcTJ pDPqvH9dOu3IQXt3hlypPzb9j8N8ZRdJB4icU3XDCCCrbfSf/oxq2J+ej7PUMSxPKjE8 Gzwn2zdTp4Scrxs5B/ZlBg3uwLVaLegxql1RIGmQmw152RobR1cECE4SOMj4bUDs5wCu YYAVh4zQTSGFS+v/0AV9FQFVb+u32fDLnZL7MbpF7tR74IRThHyXKV7CTJsppG12KkTR YIuUiqgMrKDIbAw5KP9DdqdtTAJ5EBwLDLMdtKNKhrMNNoQ6e6qlTwYf13IXB8i1rYfA /w== Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2120.oracle.com with ESMTP id 2pybjspa1a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 18 Jan 2019 15:44:52 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x0IFipUS029213 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 18 Jan 2019 15:44:51 GMT Received: from abhmp0018.oracle.com (abhmp0018.oracle.com [141.146.116.24]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x0IFipM4017770; Fri, 18 Jan 2019 15:44:51 GMT MIME-Version: 1.0 Message-ID: <5e87ae90-94b9-4e85-9172-46b95365ec36@default> Date: Fri, 18 Jan 2019 07:44:47 -0800 (PST) From: Changchun Zhang Sender: Changchun Zhang To: "Pathak, Pravin" , "Trahe, Fiona" , users@dpdk.org Cc: "Trahe, Fiona" References: <03fd164b-112b-4e44-a5b0-15c6e3703662@default> <348A99DA5F5B7549AA880327E580B435896CD08F@IRSMSX101.ger.corp.intel.com> <168A68C163D584429EF02A476D5274424DEA9B7C@FMSMSX108.amr.corp.intel.com> In-Reply-To: <168A68C163D584429EF02A476D5274424DEA9B7C@FMSMSX108.amr.corp.intel.com> X-Priority: 3 X-Mailer: Oracle Beehive Extensions for Outlook 2.0.1.9.1 (1003210) [OL 15.0.5093.0 (x86)] Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9140 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901180112 Subject: Re: [dpdk-users] Run-to-completion or Pipe-line for QAT PMD in DPDK X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jan 2019 15:44:53 -0000 Hi Fiona and Pathak, Thanks! Changchun (Alex) -----Original Message----- From: Pathak, Pravin [mailto:pravin.pathak@intel.com]=20 Sent: Friday, January 18, 2019 9:29 AM To: Trahe, Fiona ; Changchun Zhang ; users@dpdk.org Cc: Trahe, Fiona Subject: RE: [dpdk-users] Run-to-completion or Pipe-line for QAT PMD in DPD= K Hi Alex - -----Original Message----- From: users [mailto:users-bounces@dpdk.org] On Behalf Of Trahe, Fiona Sent: Friday, January 18, 2019 8:14 AM To: Changchun Zhang ; users@dpdk.org Cc: Trahe, Fiona Subject: Re: [dpdk-users] Run-to-completion or Pipe-line for QAT PMD in DPD= K Hi Alex, > -----Original Message----- > From: users [mailto:users-bounces@dpdk.org] On Behalf Of Changchun=20 > Zhang > Sent: Thursday, January 17, 2019 11:01 PM > To: users@dpdk.org > Subject: [dpdk-users] Run-to-completion or Pipe-line for QAT PMD in=20 > DPDK >=20 > Hi, >=20 >=20 >=20 > I have user question on using the QAT device in the DPDK. >=20 > In the real design, after calling enqueuer_burst() on the specified=20 > queue pair at one of the lcore, usually which one is usually done? >=20 > 1. should we do run-to-completion to call dequeuer_burst() waiting fo= r the device finishing the > crypto operation, >=20 > 2. or should we do pipe-line, in which we return right after enqueuer= _burst() and release the CPU. > And call dequeuer_burst() on other thread function? >=20 > Option 1 is more like synchronous and can be seen on all the DPDK=20 > crypto examples, while option 2 is asynchronous which I have never seen i= n any reference design if I missed anything. [Fiona] Option 2 is not possible with QAT - the dequeue must be called in the same = thread as the enqueue. This is optimised without atomics for best performan= ce - if this is a problem let us know.=20 However best performance is not quite using option 1 and not a synchronous = blocking method.=20 If you enqueue and then go straight to dequeue, you're not getting the best= advantage from the cycles freed up by offloading.=20 i.e. best to enqueue a burst, then go do some other work, like maybe collec= ting more requests for next enqueue or other processing, then dequeue. Take= and process whatever ops are dequeued - this will not necessarily match up= with the number you've enqueued - depends on how quickly you call the dequ= eue. Don't wait until all the enqueued ops are dequeued before enqueuing the nex= t batch. SO it's asynchronous. But in the same thread. [changchun] In the same thread, but how about to dequeuer at the beginning = of the thread each time, if data presents then processing them, if no data = just do other work, and equeue the packets at some time but does not wait. For example: While(1) { =09Nb_ops =3D dequeuer(); =09If(nb_ops > )=20 { process_dequeued_data(); continue;=09 } =20 Other_work(); If(ipsec) Enqueuer();=20 } Does it make sense? You'll get best throughput when you keep the input filled up so the device = has operations to work on and regularly dequeue a burst. Dequeuing too ofte= n will waste cycles in the overhead calling the API, dequeuing too slowly w= ill cause the device to back up. Ideally tune for your application to find = the sweet spot in between these 2 extremes. =20 [Pravin] I faced exact same issue while moving from software crypto to HW. I impleme= nted option Fiona suggested. =20 Thread enqueues to crypto engine and goes back to other work. It periodical= ly polls crypto to see if work is finished. As we have a single thread running, it keeps doing queuing as work arrives = and de-queuing as results are ready while in between doing other stuff. To keep track of packets, I put some ID into crypto operation private data.