From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 902944234D; Tue, 10 Oct 2023 17:59:58 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 88346402CD; Tue, 10 Oct 2023 17:59:58 +0200 (CEST) Received: from mail-oo1-f47.google.com (mail-oo1-f47.google.com [209.85.161.47]) by mails.dpdk.org (Postfix) with ESMTP id 9A5B640297 for ; Tue, 10 Oct 2023 17:59:56 +0200 (CEST) Received: by mail-oo1-f47.google.com with SMTP id 006d021491bc7-57b6c7d0cabso3496731eaf.1 for ; Tue, 10 Oct 2023 08:59:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1696953596; x=1697558396; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=nj+4AOjp3NxW/4+Z1+FM61lY7g7dBdXni4Dx4c3+ILA=; b=XDBwDY1YIEGeSzDDQNlhlkRxkw3C1sTZUrntjSEp2Xm7ofCYiXHZ9OtDxL03EdbhP5 4X98O5yqoYVD9bs+wTX+RikimGay6LNaV3pnf8jkUK8Yei7rSdTbVTV45TZVH+59Nrc6 H6z6RDz08VyKfujl1xEWZx3hJ6sLTcXIkw+7A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696953596; x=1697558396; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=nj+4AOjp3NxW/4+Z1+FM61lY7g7dBdXni4Dx4c3+ILA=; b=buPXbCrLUzj91DuGXUeafCnIMpB6m9CzyxaOucv2glqJwhhNEZiIb1q3guDHgXevYH mDv9AlnKYJ2pFWxLa8lN6/fgIq5BkZ2r2kdogdlvnrWkvM6tyf+fG/629pjSaz6u2OB5 4GF0yL/gbrem1kzczBTi56AXdKYbw9BAzbhoEeIA8Yo1WpNZOfRlMRy/nQPzalZ8RF77 nqbvLOurhsw5q8BY8ZI3gvj7YvZLJiD1CU2D3zL8UVVMb1asQSFgTnJLH1FyaYNqmA60 ApQCbuax+Hy40m3givVsuyBXlBFCeADbHHh/Da/gCpDnduan7d8foMDYPw+T0654G8Ts c/0Q== X-Gm-Message-State: AOJu0YyPzpifgEjcuM5OEjSj0IPyZWmVGW3UXCf8JPQWWP+lBObzOxcQ /uJ0js3oAHlYlRSh956JEgXEzShIFgiW6/hcFWeC+g== X-Google-Smtp-Source: AGHT+IEoLXjvdMxV1Jzeb/03AzGShJQUZVp7xWgeGhPVv3hA8J4oe+89LeKDMVkIyY+NmASTmRUi1MNNWcnFIlelynA= X-Received: by 2002:a4a:6c57:0:b0:57b:82d2:8254 with SMTP id u23-20020a4a6c57000000b0057b82d28254mr16861332oof.3.1696953595863; Tue, 10 Oct 2023 08:59:55 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Patrick Robb Date: Tue, 10 Oct 2023 11:59:45 -0400 Message-ID: Subject: Re: Intel QAT 8970 accel card on ARM Ampere Server To: Dharmik Jayesh Thakkar , David Marchand Cc: Ruifeng Wang , =?UTF-8?Q?Juraj_Linke=C5=A1?= , Honnappa Nagarahalli , "ci@dpdk.org" , nd Content-Type: multipart/alternative; boundary="000000000000483b8606075ecc22" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org --000000000000483b8606075ecc22 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Oct 9, 2023 at 11:56=E2=80=AFPM Dharmik Jayesh Thakkar < DharmikJayesh.Thakkar@arm.com> wrote: > Hi Patrick, > > > > Can you provide the grub settings? Is iommu.passthrough=3D1 included? > Sure. I'm not sure if you just wanted the kernel cmdline options or the whole grub config, but I assume you just meant kernel cmdline. Let me know if you meant more. GRUB_CMDLINE_LINUX_DEFAULT=3D"default_hugepagesz=3D1G hugepagesz=3D1G hugepages=3D32 iommu=3Dpt intel_iommu=3Don isolcpus=3D39-79 nohz_full=3D39-= 79 rcu_nocbs=3D39-79 processor.max_cstate=3D1 intel_pstate=3Ddisable console=3DttyS0,115200 console=3Dtty0" But, iommu.passthrough=3D1 is not included, so I can add that if we need to= . Do you know that this won't have any bad implications for the (intel, nvidia, broadcom) NICs which we test on this server? > > > Also, is qat_c62xvf loaded as well? > qat_c62xvf is built in to the kernel also. > > > > Finally, a few guidelines on the vfio driver: > > At times, we need to configure the vfio driver. > > On kernel vers. 5.9+ we need to load the vfio-pci driver with the > additional parameter *disable_denylist=3D1* > > Unload the vfio-pci driver if it is already loaded so that we can reload > it with the correct parameters : > *sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo > modprobe -r vfio_virqfd; sudo modprobe -r vfio* > > If you can't unload the vfio driver because it's been built into the > kernel, you'll have to find another way to change VFIO parameters, or to > rebuild your kernel with VFIO_PCI set as a module. Failing to do that, yo= u > might encounter issues later on when you try to bind the VFs to VFIO. > > Load the vfio-pci driver and bind it to QAT VFs device ids: > *sudo modprobe vfio-pci disable_denylist=3D1 enable_sriov=3D1 > vfio-pci.ids=3D8086:37c9* > > Enable no-iommu-mode: > *echo "1" | sudo tee > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode* > > /sys/module/vfio/parameter is missing ? > > If /sys/module/vfio/parameters does not exist, you might be missing the > kernel module VFIO_NOIOMMU > > > > *Automatically set VFIO params on boot* > > It's possible to set these parameters automatically on boot by creating a > */etc/modprobe.d/vfio-pci.conf *file with the parameters : > *cat /etc/modprobe.d/vfio-pci.conf* > *options vfio enable_unsafe_noiommu_mode=3D1* > *options vfio-pci disable_denylist=3D1 enable_sriov=3D1 vfio-pci.ids=3D80= 86:37c9* > > > > We haven=E2=80=99t encountered this issue in the past, so just making sur= e the > configuration is correct. I don=E2=80=99t think having the driver static/= loadable > should make a difference, I will try with building statically on my setup= . > > > > Thank you! > > > Okay, this should be fine. Like I said, we are also running tests on NICs on this server. So, in our Jenkinsfiles scripts for running the testing, I will add a preliminary step only for QAT tests which runs: *sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo modprobe -r vfio_virqfd; sudo modprobe -r vfio* *sudo modprobe vfio-pci disable_denylist=3D1 enable_sriov=3D1 vfio-pci.ids=3D8086:37c9* *echo "1" | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode= * (then run QAT tests) And if running on NICs, have a preliminary step which runs *sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo modprobe -r vfio_virqfd; sudo modprobe -r vfio* *sudo modprobe vfio* David does this also sound reasonable to you, per your comment about isolating this setting to QAT card testing? Dharmik if this all sounds okay and you can confirm the iommu.passthrough change is fine, I will proceed. Thank you for providing the assistance. --000000000000483b8606075ecc22 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Mon, Oct 9, 2023 at 11:56=E2=80=AF= PM Dharmik Jayesh Thakkar <DharmikJayesh.Thakkar@arm.com> wrote:

Hi Patrick,

=C2=A0

Can you provide the grub settings? Is iommu.passthro= ugh=3D1 included?


Sur= e. I'm not sure if you just wanted the kernel cmdline options or the wh= ole grub config,=C2=A0but I assume you just meant kernel cmdline. Let me kn= ow if you meant more.

GRUB_CMDLINE_LINUX_DEFAULT= =3D"default_hugepagesz=3D1G hugepagesz=3D1G hugepages=3D32 iommu=3Dpt = intel_iommu=3Don isolcpus=3D39-79 nohz_full=3D39-79 rcu_nocbs=3D39-79 proce= ssor.max_cstate=3D1 intel_pstate=3Ddisable console=3DttyS0,115200 console= =3Dtty0"

But, iommu.passthrough=3D1 is not included, so I can a= dd that if we need to. Do you know that this won't have any bad implica= tions for the (intel, nvidia, broadcom) NICs which we test on this server?<= /div>
=C2=A0

=C2=A0

Also, is qat_c62xvf loaded as well?

<= /div>
qat_c62xvf is built in to the kernel also.=C2=A0
=C2=A0

=C2=A0

=C2=A0

Finally, a few guidelines on the vfio driver:=

At times, we need to configure the vfio driver.

On kernel vers. 5.9+ we need to load the vfio-pci dr= iver with the additional parameter=C2=A0disable_denylist=3D1<= u>

Unload the vfio-pci driver if it is already loaded s= o that we can reload it with the correct parameters :
sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo modpr= obe -r vfio_virqfd; sudo modprobe -r vfio

If you can't unload the vfio driver because it&#= 39;s been built into the kernel, you'll have to find another way to cha= nge VFIO parameters, or to rebuild your kernel with VFIO_PCI set as a modul= e. Failing to do that, you might encounter issues later on when you try to bind the VFs to VFIO.
=C2=A0
Load the vfio-pci driver and bind it to QAT VFs device ids:
sudo modprobe vfio-pci disable_denylist=3D1 enable_sriov=3D1 vfio-pci.id= s=3D8086:37c9
=C2=A0
Enable no-iommu-mode:
echo "1" | sudo tee /sys/module/vfio/parameters/enable_unsafe_= noiommu_mode

=C2=A0/sys/module/vfio/parameter is missing ?=

If /sys/module/vfio/parameters does not exist, you m= ight be missing the kernel module VFIO_NOIOMMU

=C2=A0

Automatically set VFIO params on boot

It's possible to set these parameters automatica= lly on boot by creating a=C2=A0/etc/modprobe.d/vfio-pci.conf=C2=A0fi= le with the parameters :
cat /etc/modprobe.d/vfio-pci.conf
options vfio enable_unsafe_noiommu_mode=3D1
options vfio-pci disable_denylist=3D1 enable_sriov=3D1 vfio-pci.ids=3D80= 86:37c9

=C2=A0

We haven=E2=80=99t encountered this issue in the pas= t, so just making sure the configuration is correct. I don=E2=80=99t think = having the driver static/loadable should make a difference, I will try with= building statically on my setup.

=C2=A0

Thank you!


Okay, this should be fine. Like I said, we ar= e also running tests on NICs on this server. So, in our Jenkinsfiles script= s for running the testing, I will add a preliminary step only for QAT tests= which runs:=C2=A0
sudo modprobe -r vfio_iommu_type1; sudo modprobe -= r vfio_pci; sudo modprobe -r vfio_virqfd; sudo modprobe -r vfio
sudo modprobe vfio-pci disable_denylist=3D1 enable_sriov=3D1 vfio-pci= .ids=3D8086:37c9
echo "1" | sudo tee /sys/module= /vfio/parameters/enable_unsafe_noiommu_mode
(then run QAT tes= ts)

And if running on NICs, have a preliminary ste= p which runs=C2=A0
sudo modprobe -r vfio_iommu_type1; sudo mod= probe -r vfio_pci; sudo modprobe -r vfio_virqfd; sudo modprobe -r vfio<= br>
sudo modprobe vfio

Dav= id does this also sound reasonable to you, per your comment about isolating= this setting to QAT card testing?

Dharmik if this= all sounds okay and you can confirm the iommu.passthrough change is fine, = I will proceed. Thank you for providing the assistance.=C2=A0

--000000000000483b8606075ecc22--