From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1E46942FA8; Mon, 31 Jul 2023 19:14:01 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1901840A89; Mon, 31 Jul 2023 19:14:01 +0200 (CEST) Received: from mail-oa1-f44.google.com (mail-oa1-f44.google.com [209.85.160.44]) by mails.dpdk.org (Postfix) with ESMTP id 95E234067B for ; Mon, 31 Jul 2023 19:14:00 +0200 (CEST) Received: by mail-oa1-f44.google.com with SMTP id 586e51a60fabf-1bb75afeee3so3235309fac.0 for ; Mon, 31 Jul 2023 10:14:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1690823640; x=1691428440; h=cc:to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=yDCXQxbfoJ+sHSMkSHUeSsLm4znDleY9+wUJJXX3vtg=; b=HGbEdtplOEjl7C0T+mM8B3rumaWL19onm6b5t5UHy7QQssIMhSTjeUpemo4vtm++ZX 6vnOe2sK/AgX+TufnUvI1EGYyDQR87TroLBlhQJe0f50bzm1kiHEV1cYzoNETNeTeqhr 1sFBXQ+FFCfuNCfIWY3GuA1vpXyWBK8jHRL6E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690823640; x=1691428440; h=cc:to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=yDCXQxbfoJ+sHSMkSHUeSsLm4znDleY9+wUJJXX3vtg=; b=WbdiYZkT9X5TdLC7pWStDSAqjIRAcHCb4+pj/clkw9fg5hhups/bsTooI0CZoi/CSH 7NxPDd7hNNrjFRDFakk0Si4u1yteJ5SOBelevy4mg4L9w3IQeZ+WQdOmZvNqge7CB5Fp bouw4XT6OFukVAhTjC7khhdGGilh+CBDU028IBHC91FCzmxk7Zc1Dwx4PwOsSnnxs0yj evXAdVxUAiZH5BBS+l13kLWtBwtRapYSrsix/sC2VIB7GmZB3duhVdBMYcu2vzAhxf4Y 7DOWcTf0sLFJPdleeVZSnBzBz94kQUwGPxlikTTLVv5AM+bXYEstRSZg5Mkoq56/2TGf +JiQ== X-Gm-Message-State: ABy/qLZctacxxLbmAaw7XsbLgAkKG6TZqhAjxv3S5ZqAMw/m9vYx00SP Sht34ARjU0g6aS2Ha3pfIzMIF4Wx32qTckhFkJNDSw== X-Google-Smtp-Source: APBJJlFU1p1LpneSRTHMylfckC5fWivPAR6+ACiMCFnrZjShQPD6JY1VvakasRsKnjPz1nM2vrywlFbdt7hKWA9iGRI= X-Received: by 2002:a05:6870:b30e:b0:1b0:4fe8:5e58 with SMTP id a14-20020a056870b30e00b001b04fe85e58mr11275833oao.19.1690823639709; Mon, 31 Jul 2023 10:13:59 -0700 (PDT) MIME-Version: 1.0 From: Patrick Robb Date: Mon, 31 Jul 2023 13:13:49 -0400 Message-ID: Subject: Intel QAT 8970 accel card on ARM Ampere Server To: Ruifeng Wang , Honnappa Nagarahalli , =?UTF-8?Q?Juraj_Linke=C5=A1?= Cc: dharmikjayesh.thakkar@arm.com, ci@dpdk.org Content-Type: multipart/alternative; boundary="0000000000006c50140601cb8ebb" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org --0000000000006c50140601cb8ebb Content-Type: text/plain; charset="UTF-8" Hi Ruifeng, Honnappa, Juraj, The Intel QAT 8970 accelerator card has arrived to the Community Lab, and we've installed it on the Ampere server. Presumably, we should test both crypto and compress operations (and their respective performance metrics). To that end, there are also DTS testsuites for testing QAT crypto/compress functions. These testsuites make use of the crypto perf dpdk app and the compress perf dpdk app. If you want, you can setup the DTS stuff yourself, both on the system side, and the Jenkins side (you are allowed to submit PRs on our gitlab now), but we can also do that on the lab side as we probably have more experience. I do, however, have a question about the QAT kernel driver and corresponding PMDs. compress suite: https://git.dpdk.org/tools/dts/tree/test_plans/compressdev_qat_pmd_test_plan.rst crypto suite: https://git.dpdk.org/tools/dts/tree/test_plans/crypto_perf_cryptodev_perf_test_plan.rst For reference, the DPDK docs page explaining QAT driver capabilities and building the QAT PMDs (crypto sym, crypto asym, and compress) is here: https://doc.dpdk.org/guides/cryptodevs/qat.html#building-qat Some notes before I get to my main question: -The 8970 is a C62x device -OpenSSL (arm requires it for QAT) is installed -3 PFs are visible from lspci (expected) -SRIOV is enabled However, although the system is on a valid kernel version for the QAT driver, the kernel module for QAT is not loaded, so in trying to set up testing, I am unable to create the 16 VFs for the 3 PFs respectively, like the example below: echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs There is also an option to download the firmware from the kernel firmware repo and copy the qat binaries to /lib/firmware and start the qat modules from there. I wasn't able to resolve the situation with this method, but it also could have been user error on my part. There is an option to install using the IDZ QAT Driver , but it should not be required given the kernel version the Ampere server is on, and I don't want to go down the road of relying on this "fall back" method without consulting you first. Is it possible that there is anything specific to running a QAT device on ARM specifically which I am missing here? The DTS testsuite testplans actually seem to recommend going down this road in general, but the DPDK docs say to use the kernel driver, so I don't know. In any case, one of you should be able to login to the Ampere server in situations like this, or just in general. Ruifeng/Juraj I see you both have accounts on our IdM system, so you should have access. Please let me know if you need renewed vpn cert configs and I will send you one. If you do login, know this system could be running CI testing at any time. I can always schedule time for it to be offline and available for maintenance if you want to do anything which could be disruptive to testing. I also CC'd Dharmik on this as I see he sent an email regarding QAT support on aarch64 in June. Let me know if you have any thoughts on the QAT kernel driver part. Thanks, Patrick --0000000000006c50140601cb8ebb Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Ruifeng, Honnappa, Juraj,

The Intel = QAT 8970 accelerator card has arrived to the Community Lab, and we've i= nstalled it on the Ampere server. Presumably, we should test both crypto an= d compress operations (and their respective performance metrics). To that e= nd, there are also DTS testsuites for testing QAT crypto/compress functions= . These testsuites make use of the crypto perf dpdk app and the compress pe= rf dpdk app. If you want, you can setup the DTS stuff yourself, both on the= system=C2=A0side, and the Jenkins side (you are allowed=C2=A0to submit PRs= on our gitlab now), but we can also do that on the lab side as we probably= have more experience. I do, however, have a question about the QAT kernel = driver and corresponding PMDs.


For reference, the DPDK do= cs page explaining QAT driver capabilities and building the QAT PMDs (crypt= o sym, crypto asym, and compress) is here:=C2=A0https://doc.dpdk.org/guides/c= ryptodevs/qat.html#building-qat

Some notes before I get to my ma= in question:
-The 8970 is a C62x device
-OpenSSL (arm r= equires it for QAT) is installed
-3 PFs are visible from lspci (e= xpected)
-SRIOV is enabled

However,= although the system is on a valid kernel version for the QAT driver, the k= ernel module for QAT is not loaded, so in trying to set up testing, I am un= able to create the 16 VFs for the 3 PFs respectively, like the example belo= w:=C2=A0

echo 16 > /sys/bus/pci/drivers/c6xx/(p= ci address)/sriov_numvfs
echo 16 > /sys/bus/pci/drivers/c6xx/(pci add= ress)/sriov_numvfs
echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/= sriov_numvfs

There is also an option to downlo= ad the firmware from the kernel firmware repo and copy the qat binaries to = /lib/firmware and start the qat modules from there. I wasn't able to re= solve the situation with this method, but it also could have been user erro= r on my part.=C2=A0

There is an option to install = using the IDZ QAT Driver= , but it should not be required given the kernel version the Ampere server = is on, and I don't want to go down the road of relying on this "fa= ll back" method without consulting you first. Is it possible that ther= e is anything specific to running a QAT device on ARM specifically which I = am missing here? The DTS testsuite testplans actually seem to recommend goi= ng down this road in general, but the DPDK docs say to use the kernel drive= r, so I don't know.

In any case, one of you sh= ould be able to login to the Ampere server in situations like this, or just= in general. Ruifeng/Juraj I see you both have accounts on our IdM system, = so you should have access. Please let me know if you need renewed vpn cert = configs and I will send you one. If you do login, know this system could be= running CI testing at any time. I can always schedule time for it to be of= fline and available for maintenance if you want to do anything which could = be disruptive to testing.=C2=A0

I also CC'd Dh= armik on this as I see he sent an email regarding QAT support on aarch64 in= June.

Let me know if you have any thoughts on the= QAT kernel driver part.=C2=A0

Thanks,
P= atrick=C2=A0
--0000000000006c50140601cb8ebb--