From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A457543B79; Thu, 7 Mar 2024 06:27:50 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9D110402F2; Thu, 7 Mar 2024 06:27:50 +0100 (CET) Received: from mail-ot1-f49.google.com (mail-ot1-f49.google.com [209.85.210.49]) by mails.dpdk.org (Postfix) with ESMTP id 3C8E7402EA for ; Thu, 7 Mar 2024 06:27:49 +0100 (CET) Received: by mail-ot1-f49.google.com with SMTP id 46e09a7af769-6e4efdf31c9so190230a34.1 for ; Wed, 06 Mar 2024 21:27:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1709789268; x=1710394068; darn=dpdk.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=ik2QXV6LIomY1Gxt+k8ZjlJxAPpxkaGkP8lxy0IdPY0=; b=jXVSwygiqr8locA5lg3FNkrpN21brFySddkF85XwXNmiUqIreH2RqCs0I2icjuwKkl HbsQrGcfyOtlOAd3CuHnMIs7Qh/dt0uKDnwVAKDLBr9MRwf0oEpWkbS+OvsgMCuIfgxW 4h6B3rvlRRZZpwiAxWGOq3z0VCODc1URUUCxc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709789268; x=1710394068; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ik2QXV6LIomY1Gxt+k8ZjlJxAPpxkaGkP8lxy0IdPY0=; b=aeP77chOsY1N8IlxrQg+wxjStmOc7oXV7a3+/6YBbP0NfpMFlEkWs1QV5MBDOh6/mo cURqYTzaV38uvDojOMHtACSYRjKEgfEMk2Jh6vO58M+B+n8cGtoUqCT7SNFsk85jgrMT YVWOtFF9ekuNw+n2kPEWT4o9KuxnoyMaXK6EdkEBlV1//f12LeWIxnTqJ0Oe+SCBvVtb UDWVkp46lHQRhc2qNITHSws2ehVUI9M4WASq9L2wD7V4TlhonDZTuNgbSEmc5fRaPTza Qr1N9HRGguXS/rqix+qxa0wIUVldKHsTOBN3kA2mybI1WLjBi5zf8/vhM8/x7TPmO1qM E0ng== X-Forwarded-Encrypted: i=1; AJvYcCVy1KJKWc82TTgf8HjHQvuJcRPF5GAP3isAW4Tvt/AeCvqoebPymDoLv69tbZlZdQkcHWWv2vI4TuZnkQ== X-Gm-Message-State: AOJu0YyFcl114gkpTODB0OUS3QQsC/4b8s45UkEg4yoXo5grMfJ4Niak kdKfmpWa8t+atbBq4XV2FqoMCMJQMy42eqUx6wSyvIklz+gmahV+suKnlCsIaXOpUQFVKGjJ38L shdkhU4wu2GcDTOC9v+J+3vwaGOPmjBxVx4IkQQ== X-Google-Smtp-Source: AGHT+IHR3xFdr0ZK+H+aWUveilG2jdjCvSFsQMGJUmVAJu7BLWzEtgN7Z30WYP2/IOnulUIzLHWGqVeBqg7BBue3nds= X-Received: by 2002:a05:6830:1552:b0:6e4:e328:4d49 with SMTP id l18-20020a056830155200b006e4e3284d49mr173926otp.16.1709789268399; Wed, 06 Mar 2024 21:27:48 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Patrick Robb Date: Thu, 7 Mar 2024 00:27:37 -0500 Message-ID: Subject: Re: Intel QAT 8970 accel card on ARM Ampere Server To: Honnappa Nagarahalli Cc: Ruifeng Wang , =?UTF-8?Q?Juraj_Linke=C5=A1?= , Dharmik Jayesh Thakkar , David Marchand , "ci@dpdk.org" , nd , Wathsala Wathawana Vithanage , Paul Szczepanek , Dhruv Tripathi Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Hi all, I have run the crypto_perf_cryptodev_perf DTS testsuite for the QAT card on the Ampere server, and have some updates below: On Wed, Feb 28, 2024 at 3:40=E2=80=AFPM Honnappa Nagarahalli wrote: > > > > > On Feb 28, 2024, at 2:00=E2=80=AFPM, Patrick Robb w= rote: > > > > quick update: > > > > I could bind the QAT VFs to vfio-pci after using the module loading opt= ions Dharmik mentioned. > > > > First I tested SYM QAT pmd from dpdk test on the VF and got: > > > > + Tests Total : 751 > > + Tests Skipped : 257 > > + Tests Executed : 659 > > + Tests Unsupported: 0 > > + Tests Passed : 494 > > + Tests Failed : 0 > > + ------------------------------------------------------- + > > Test OK > > > > I can try the crypto performance DTS testsuite next. Let me know if you= have any thoughts. > Please go ahead and try. We have not worked on the performance, but it is= fine to try. First, two tiny change are needed in DTS to make it work: 1. As Dharmik and David discussed, there are some QAT devices that need VFIO denylist=3D1. To account for this, in cryptodev_common.py (which the crypto perf testsuite imports), we need to add: given the c62x device id is 37c8 if dev_id in ["37c8", "435", "19e2"]: test_case.dut.send_expect('modprobe -r vfio_iommu_type1; modprobe -r vfio_pci; modprobe -r vfio_virqfd; modprobe -r vfio', '# ', 5) test_case.dut.send_expect('modprobe vfio-pci disable_denylist=3D1 enable_sriov=3D1 vfio-pci.ids=3D8086:37c9', '# ', 5) test_case.dut.send_expect('echo "1" | tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode', '# ', 5) In order to maintain the custom vfio loading Dharmik recommended. The latter two dev ids in that list are for DH895XCC and C3XXX, since they are also included in https://github.com/torvalds/linux/commit/50173329c8cc0c892eaa7a9d0f0692ac39= cd7b04 David and Dharmik, I think this is correct, but please chime in if it isn't= . 2. For this testsuite we need to add some whitespace stripping on the lscpu output for ARM systems. For some reason on some systems there is no leading whitespace before "Core(s) per socket" in lscpu, but in others (the arm servers we have at the lab) there is. So, as long as this is all fine, I can submit a patch to DTS for these item= s. And from there we can run the testsuite and all QAT testcases are passing. It will give some results like: PerfTestsCryptodev: Test Case test_qat_zuc Begin dut.arm-ampere-dut.dpdklab.iol.unh.edu: lscpu dut.arm-ampere-dut.dpdklab.iol.unh.edu: x86_64-native-linux-gcc/app/dpdk-test-crypto-perf -l 9,10 -a 0000:03:01.0 --socket-mem 2048,0 -n 6 -- --ptest throughput --silent --total- CRYPTODEV: Initialisation parameters - name: 0000:03:01.0_qat_sym,socket id: 0, max queue pairs: 0 Allocated pool "sess_mp_0" on socket 0 lcore id Buf Size Burst Size Enqueued Dequeued Failed Enq Failed Deq MOps Gbps Cycles/Buf 10 64 32 30000000 30000000 39393954 33424660 5.5361 2.8345 4.52 10 128 32 30000000 30000000 40170307 34256181 5.4867 5.6184 4.56 10 256 32 30000000 30000000 42119414 36231215 5.3883 11.0352 4.64 10 512 32 30000000 30000000 44557481 38555569 5.2235 21.3955 4.79 10 1024 32 30000000 30000000 55097817 48193496 4.6161 37.8149 5.42 10 2048 32 30000000 30000000 126698128 118908347 3.0483 49.9439 8.20 I will let you folks who are working on this to assess the performance metrics. I assume this is useful, and if/when we bring this to CI, all these results will be stored as artifacts and viewable for any new series which come in. Happy to discuss further tomorrow at the CI meeting. If there are no issues here, I think we can write up the jenkins scripts pretty quickly and get this online tomorrow or early next week.