Re: [dpdk-users] Run-to-completion or Pipe-line for QAT PMD in DPDK

[Changchun Zhang <changchun.zhang@oracle.com>]

Thanks!
Changchun (Alex)


-----Original Message-----
From: Trahe, Fiona [mailto:fiona.trahe@intel.com] 
Sent: Friday, January 18, 2019 11:58 AM
To: Changchun Zhang <changchun.zhang@oracle.com>; Pathak, Pravin <pravin.pathak@intel.com>; users@dpdk.org
Cc: Trahe, Fiona <fiona.trahe@intel.com>
Subject: RE: [dpdk-users] Run-to-completion or Pipe-line for QAT PMD in DPDK

Hi Alex,

> -----Original Message-----
> From: Changchun Zhang [mailto:changchun.zhang@oracle.com]
> Sent: Friday, January 18, 2019 4:42 PM
> To: Trahe, Fiona <fiona.trahe@intel.com>; Pathak, Pravin 
> <pravin.pathak@intel.com>; users@dpdk.org
> Subject: RE: [dpdk-users] Run-to-completion or Pipe-line for QAT PMD 
> in DPDK
> 
> 
> 
> Thanks!
> Changchun (Alex)
> 
> 
> -----Original Message-----
> From: Trahe, Fiona [mailto:fiona.trahe@intel.com]
> Sent: Friday, January 18, 2019 11:26 AM
> To: Changchun Zhang <changchun.zhang@oracle.com>; Pathak, Pravin 
> <pravin.pathak@intel.com>; users@dpdk.org
> Cc: Trahe, Fiona <fiona.trahe@intel.com>
> Subject: RE: [dpdk-users] Run-to-completion or Pipe-line for QAT PMD 
> in DPDK
> 
> Hi Alex,
> > [changchun] In the same thread, but how about to dequeuer at the 
> > beginning of the thread each time, if data presents then processing 
> > them, if no data just do other work, and equeue the packets at some time but does not wait.
> > For example:
> > While(1)
> > {
> > 	Nb_ops = dequeuer();
> > 	If(nb_ops > )
> >              {
> >                  process_dequeued_data();
> >                  continue;
> >              }
> >
> >               Other_work();
> >               If(ipsec)
> >                   Enqueuer();
> > }
> > Does it make sense?
> [Fiona] It can, though on the first loop ro after a queit time youll 
> proably get very few back on first and second dequeue as It'll be 
> called immediately after the enqueue. Once it gets busy that could be 
> ok though [changchun] Thank you Fiona. One more question, as you said, 
> enqueuer/dequeue should be called within the same thread. Why? Is it 
> because the other thread(lcore 1) cannot dequeuer the processed data 
> from other thread(lcore 2)? But as the cryptograph device lib doc 
> says, "it is howerver possible to use a different logical core to dequeuer an operation on a queue pair from the logical core which it was enqueued on". Looking forward to more details.
It's because the QAT inflight counter would be incremented and decremented by both threads so would need to be an atomic. It used to be atomic until 17.11 release but we got a good reduction in offload cycle-count by replacing this with a normal variable and as all the feedback we got was that applications were not using in pipeline mode we decided to trade off this limitation for the added performance. The limitation is documented here:
https://urldefense.proofpoint.com/v2/url?u=http-3A__doc.dpdk.org_guides_cryptodevs_qat.html&d=DwIFAg&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=VCO7jqQFACS5pfBnqRHRln4bZ3htFERYZ3PkX0ytpns&m=Bp2X1mxAJ_Eg1kB1t91mgFDUfIrr8Vl-2Nxq_9RLxrI&s=2MPxBDAAKTRMsgE6mrw0cbGX8C4oFcFstJWzpEp8opI&e=
You can look at code before 17.11 release to see the difference.

 [changchun] Many thanks! So from this limitation, we can conclude that Lcore can only dequeue the QAT queue which was enqueued by itself, right. If so, then the Crypto device lib doc may be a little misleading, at least some notes should be put there.