From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id ED9BF42367; Wed, 11 Oct 2023 22:13:52 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CB206400EF; Wed, 11 Oct 2023 22:13:52 +0200 (CEST) Received: from mail-oo1-f50.google.com (mail-oo1-f50.google.com [209.85.161.50]) by mails.dpdk.org (Postfix) with ESMTP id 2F89F400D7 for ; Wed, 11 Oct 2023 22:13:52 +0200 (CEST) Received: by mail-oo1-f50.google.com with SMTP id 006d021491bc7-57b706f604aso144020eaf.0 for ; Wed, 11 Oct 2023 13:13:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1697055231; x=1697660031; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=dt2FLPtwbLAmYOgW1u7gepDtr8X/Xk6nfjSPlwYpuBg=; b=FTfNcBAmFuAhA/HSzfJp3A7Wj8Rds5xoYuiohpefZRTda/2CW2ixZ1Sy9fZAoJDZVY Ft8ACSY7VG5z+E77WTKvoV6Jza8ZQkGiW8JzFxyHP8/JgnbnyhQGWyd2ya5m6t3T/ebw 5g0K+I/JsL1o8ZNRkB/+vV8pE8xF3o8hZtVew= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697055231; x=1697660031; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=dt2FLPtwbLAmYOgW1u7gepDtr8X/Xk6nfjSPlwYpuBg=; b=vz688CjLi0bCUfU64uaq2jsBLnrPXXgibXpcYOh22hl1lsGfYLLQOTgMYgI1AFVUSt 6KxHBCr9J+lsXzlhXp/b8pDr8y8hKcEq7uCUnFg3AGzk0cBQP4MySBKyeCN9hJAIXI/r /NF4+xfnsUrIru6A6KbELB5VTPchjmLTPmTbiGeENxAdm9fU3KoYnIKey0OmtVRR1ii3 16dtGF8OQrWKzL4s2do56qxLdqESFfUEA0t8KNrRpyZquoCi5Ix4HX8rSLfZkzO0vzow ae9SU+OoqbYY/9CS2Dc247i1ekaFzW9BeqPZm7UGs3DC3iyxIi9jIIG8ZIf4X5TMLyqi XpKA== X-Gm-Message-State: AOJu0YzvpP2l9K+T8S2AqHYUZDdGC7UGk+svmm/LcRDq+t6Q+4O58q4W 52CpEJyKUp5kAB5u7cOGqLS88TajleLlPP86OToJZQ== X-Google-Smtp-Source: AGHT+IGk9Xv6UXXqDVClkgBK/k6Eh8GeDAH/7GNKxoX37jrzItfVkzJtZ6GXS+nOMDMao3sfYA8F6U8Xm3FWEAlSHrE= X-Received: by 2002:a4a:8447:0:b0:571:28d5:2c71 with SMTP id m7-20020a4a8447000000b0057128d52c71mr19100395oog.2.1697055231318; Wed, 11 Oct 2023 13:13:51 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Patrick Robb Date: Wed, 11 Oct 2023 16:13:40 -0400 Message-ID: Subject: Re: Intel QAT 8970 accel card on ARM Ampere Server To: =?UTF-8?Q?Juraj_Linke=C5=A1?= Cc: Dharmik Jayesh Thakkar , David Marchand , Ruifeng Wang , Honnappa Nagarahalli , "ci@dpdk.org" , nd Content-Type: multipart/alternative; boundary="0000000000003a369706077676cc" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org --0000000000003a369706077676cc Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Oct 11, 2023 at 4:14=E2=80=AFAM Juraj Linke=C5=A1 wrote: > > > On Tue, Oct 10, 2023 at 5:59=E2=80=AFPM Patrick Robb = wrote: > >> >> >> On Mon, Oct 9, 2023 at 11:56=E2=80=AFPM Dharmik Jayesh Thakkar < >> DharmikJayesh.Thakkar@arm.com> wrote: >> >>> Hi Patrick, >>> >>> >>> >>> Can you provide the grub settings? Is iommu.passthrough=3D1 included? >>> >> >> Sure. I'm not sure if you just wanted the kernel cmdline options or the >> whole grub config, but I assume you just meant kernel cmdline. Let me kn= ow >> if you meant more. >> >> GRUB_CMDLINE_LINUX_DEFAULT=3D"default_hugepagesz=3D1G hugepagesz=3D1G >> hugepages=3D32 iommu=3Dpt intel_iommu=3Don isolcpus=3D39-79 nohz_full=3D= 39-79 >> rcu_nocbs=3D39-79 processor.max_cstate=3D1 intel_pstate=3Ddisable >> console=3DttyS0,115200 console=3Dtty0" >> >> But, iommu.passthrough=3D1 is not included, so I can add that if we need >> to. Do you know that this won't have any bad implications for the (intel= , >> nvidia, broadcom) NICs which we test on this server? >> >> > > Just a note here, Patrick. The iommu kernel and intel_pstate parameters > aren't supported on arm, so you can remove those. And when > iommu.passthrouh=3D1, IOMMU is bypassed and intel_iommu doesn't do anythi= ng > (and maybe isn't supported on arm, but that's not clear from the docs > ), > so that can be removed as well. > Thanks Dharmik and Juraj. Updated kernel cmdline args: BOOT_IMAGE=3D/vmlinuz-5.15.82+ root=3D/dev/mapper/ubuntu--vg--1-ubuntu--lv = ro default_hugepagesz=3D1G hugepagesz=3D1G hugepages=3D32 isolcpus=3D39-79 nohz_full=3D39-79 rcu_nocbs=3D39-79 processor.max_cstate=3D1 iommu.passthro= ugh=3D1 console=3DttyS0,115200 console=3Dtty0 I added the iommu.passthrough option and tried again, to no avail. FYI I am still using the guidance here: https://doc.dpdk.org/guides/cryptodevs/qat.html along with your added steps= . root@arm-ampere-dut:~# echo 16 > /sys/bus/pci/drivers/c6xx/0000:03:00.0/sriov_numvfs Segmentation fault (core dumped) As you know the above setting of the 48 VFs is a prerequisite to binding the VFs to vfio-pci. But, I did run through loading the custom vfio and there were no issues, so once we clear this initial hurdle we should be fine. --0000000000003a369706077676cc Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Wed, Oct 11, 2023 at 4:14=E2=80=AFAM J= uraj Linke=C5=A1 <juraj.linkes@pantheon.tech> wrote:


On Tue, Oct 10, 2023 at 5:59=E2=80=AFPM Patrick Robb <probb@iol.unh.edu> wrote:=


On Mon, Oct 9, 2023 at 11:56=E2=80=AFPM Dharmik Jaye= sh Thakkar <DharmikJayesh.Thakkar@arm.com> wrote:

Hi Patrick,

=C2=A0

Can you provide the grub settings? Is iommu.passthro= ugh=3D1 included?


Sur= e. I'm not sure if you just wanted the kernel cmdline options or the wh= ole grub config,=C2=A0but I assume you just meant kernel cmdline. Let me kn= ow if you meant more.

GRUB_CMDLINE_LINUX_DEFAULT= =3D"default_hugepagesz=3D1G hugepagesz=3D1G hugepages=3D32 iommu=3Dpt = intel_iommu=3Don isolcpus=3D39-79 nohz_full=3D39-79 rcu_nocbs=3D39-79 proce= ssor.max_cstate=3D1 intel_pstate=3Ddisable console=3DttyS0,115200 console= =3Dtty0"

But, iommu.passthrough=3D1 is not included, so I can a= dd that if we need to. Do you know that this won't have any bad implica= tions for the (intel, nvidia, broadcom) NICs which we test on this server?<= /div>
=C2=A0

Just a n= ote here, Patrick. The iommu kernel and intel_pstate parameters aren't = supported on arm, so you can remove those. And when iommu.passthrouh=3D1, I= OMMU is bypassed and intel_iommu doesn't do anything (and maybe isn'= ;t supported on arm, but that's not clear from the docs), so that can be removed as well.
<= /blockquote>

Thanks Dharmik and Juraj. Updated kernel cm= dline args:=C2=A0
BOOT_IMAGE=3D/vmlinuz-5.15.82+ root=3D/dev/mapp= er/ubuntu--vg--1-ubuntu--lv ro default_hugepagesz=3D1G hugepagesz=3D1G huge= pages=3D32 isolcpus=3D39-79 nohz_full=3D39-79 rcu_nocbs=3D39-79 processor.m= ax_cstate=3D1 iommu.passthrough=3D1 console=3DttyS0,115200 console=3Dtty0

I added the iommu.passthrough option and tried agai= n,=C2=A0to no avail. FYI I am still using the guidance here:=C2=A0https://doc.dpdk.org/g= uides/cryptodevs/qat.html along with your added steps.

root@arm-ampere-dut:~# echo 16 > /sys/bus/pci/drivers/c6xx/0000= :03:00.0/sriov_numvfs
Segmentation fault (core dumped)
As you know the above setting of the 48 VFs is a prerequisite t= o binding the VFs to vfio-pci. But, I did run through loading the=C2=A0cust= om vfio and there were no issues,=C2=A0so once we clear this initial hurdle= we should be fine.
--0000000000003a369706077676cc--