Re: Intel QAT 8970 accel card on ARM Ampere Server

[Patrick Robb <probb@iol.unh.edu>]

Hi all,

I have run the crypto_perf_cryptodev_perf DTS testsuite for the QAT
card on the Ampere server, and have some updates below:

On Wed, Feb 28, 2024 at 3:40 PM Honnappa Nagarahalli
<Honnappa.Nagarahalli@arm.com> wrote:
>
>
>
> > On Feb 28, 2024, at 2:00 PM, Patrick Robb <probb@iol.unh.edu> wrote:
> >
> > quick update:
> >
> > I could bind the QAT VFs to vfio-pci after using the module loading options Dharmik mentioned.
> >
> > First I tested SYM QAT pmd from dpdk test on the VF and got:
> >
> >  + Tests Total :       751
> >  + Tests Skipped :     257
> >  + Tests Executed :    659
> >  + Tests Unsupported:   0
> >  + Tests Passed :      494
> >  + Tests Failed :       0
> >  + ------------------------------------------------------- +
> > Test OK
> >
> > I can try the crypto performance DTS testsuite next. Let me know if you have any thoughts.
> Please go ahead and try. We have not worked on the performance, but it is fine to try.

First, two tiny change are needed in DTS to make it work:

1. As Dharmik and David discussed, there are some QAT devices that
need VFIO denylist=1. To account for this, in cryptodev_common.py
(which the crypto perf testsuite imports), we need to add:

given the c62x device id is 37c8

if dev_id in ["37c8", "435", "19e2"]:
    test_case.dut.send_expect('modprobe -r vfio_iommu_type1; modprobe
-r vfio_pci; modprobe -r vfio_virqfd; modprobe -r vfio', '# ', 5)
    test_case.dut.send_expect('modprobe vfio-pci disable_denylist=1
enable_sriov=1 vfio-pci.ids=8086:37c9', '# ', 5)
    test_case.dut.send_expect('echo "1" | tee
/sys/module/vfio/parameters/enable_unsafe_noiommu_mode', '# ', 5)

In order to maintain the custom vfio loading Dharmik recommended. The
latter two dev ids in that list are for DH895XCC and C3XXX, since they
are also included in
https://github.com/torvalds/linux/commit/50173329c8cc0c892eaa7a9d0f0692ac39cd7b04

David and Dharmik, I think this is correct, but please chime in if it isn't.

2. For this testsuite we need to add some whitespace stripping on the
lscpu output for ARM systems. For some reason on some systems there is
no leading whitespace before "Core(s) per socket" in lscpu, but in
others (the arm servers we have at the lab) there is.

So, as long as this is all fine, I can submit a patch to DTS for these items.

And from there we can run the testsuite and all QAT testcases are
passing. It will give some results like:

            PerfTestsCryptodev: Test Case test_qat_zuc Begin
dut.arm-ampere-dut.dpdklab.iol.unh.edu: lscpu
dut.arm-ampere-dut.dpdklab.iol.unh.edu:
x86_64-native-linux-gcc/app/dpdk-test-crypto-perf  -l 9,10 -a
0000:03:01.0 --socket-mem 2048,0 -n 6  -- --ptest throughput --silent
--total-
CRYPTODEV: Initialisation parameters - name:
0000:03:01.0_qat_sym,socket id: 0, max queue pairs: 0
Allocated pool "sess_mp_0" on socket 0
    lcore id    Buf Size  Burst Size    Enqueued    Dequeued  Failed
Enq  Failed Deq        MOps        Gbps  Cycles/Buf

          10          64          32    30000000    30000000
39393954    33424660      5.5361      2.8345        4.52
          10         128          32    30000000    30000000
40170307    34256181      5.4867      5.6184        4.56
          10         256          32    30000000    30000000
42119414    36231215      5.3883     11.0352        4.64
          10         512          32    30000000    30000000
44557481    38555569      5.2235     21.3955        4.79
          10        1024          32    30000000    30000000
55097817    48193496      4.6161     37.8149        5.42
          10        2048          32    30000000    30000000
126698128   118908347      3.0483     49.9439        8.20

I will let you folks who are working on this to assess the performance
metrics. I assume this is useful, and if/when we bring this to CI, all
these results will be stored as artifacts and viewable for any new
series which come in.

Happy to discuss further tomorrow at the CI meeting. If there are no
issues here, I think we can write up the jenkins scripts pretty
quickly and get this online tomorrow or early next week.