From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 03EDA4235E; Wed, 11 Oct 2023 10:14:31 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id F07B8402F1; Wed, 11 Oct 2023 10:14:30 +0200 (CEST) Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) by mails.dpdk.org (Postfix) with ESMTP id 29A7240279 for ; Wed, 11 Oct 2023 10:14:29 +0200 (CEST) Received: by mail-ej1-f49.google.com with SMTP id a640c23a62f3a-9936b3d0286so1172275566b.0 for ; Wed, 11 Oct 2023 01:14:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1697012069; x=1697616869; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=0u9XKNPV2ipnII0RXLMyl3CpYl/zdUrDaeQGQD8poQo=; b=smACdeEY2ofCPS4z7t7h/8arA36hCl5l2JMMyZMF4el5PcXAnQpr1l9SdLH8b409zE yyidHe4ZqLm+mqIch6CY8JHrC2dF4173BSu4egxk7lYyaIw/5zpYnauOzf051gldRE+J /kNxvnWPGsJiQcrmuaU34tjoDIldQO3laNJQnhL88T77N83EDM4OxsTkEJbY/6M13kXW p+IKB6HORSWAY75TFFIlq/Z3UfSDiIdhZgZ2iEvBh1ap2+LSJPgfqknfPHIzkw2e92Tu VO+8SkINlcOELYFDNLrQw20MwX1LSEyIl0Imp9ffVBSVO8MprvYSr/jkICXghzFPl5b7 0Whg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697012069; x=1697616869; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=0u9XKNPV2ipnII0RXLMyl3CpYl/zdUrDaeQGQD8poQo=; b=CFBBbbGu4BqT1Ws6PxQrSvvZWudNISxDzT34R4swq3vs9zz5iWloQRqoUYGLnVIpd2 JfFyBIJLyFQWK4/2SdPpNh0VOegMC0z3u3g7ll9OC5dmDXGZ1vBuAwgNRKWsu4NH1vc4 uUDX0g4p0p3yXTEHlz+9PspkiTHPRVV4W3KTHrPTzzfgFbwhRmdUZF0gi/vUDBF8v12j L1zhjZ+Dn8RuP7h1/Duzj9Aa3hhqE+Ojd/SGA37VD2mHeizpz2yP7Ja3jHsByN4JZHYd JMd0eimcDvJDhPp1VjFa2FB8A76D6CxAWAWLp1AVRPXHYAtWcXVclS7rfxLamZ6AGr73 KWTg== X-Gm-Message-State: AOJu0YzrJmS2y90LU8HqULuy5hcTxnanenpmUJz5b7SoDXI1amF72awd qN3EF8K5M8b2C5uQ2b/EAPR1t+/gaD6S5347NZ0rhw== X-Google-Smtp-Source: AGHT+IHbGmj0y2vlXajH5rwj+HuwZkyHeeZFELRY0MOyvQoV0xnqUdB5TQ9h6r503KLAubxOmK5UJn3dYQv3r3q+JgU= X-Received: by 2002:a17:906:7492:b0:9b2:d018:20b2 with SMTP id e18-20020a170906749200b009b2d01820b2mr18711870ejl.39.1697012068786; Wed, 11 Oct 2023 01:14:28 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?Q?Juraj_Linke=C5=A1?= Date: Wed, 11 Oct 2023 10:14:17 +0200 Message-ID: Subject: Re: Intel QAT 8970 accel card on ARM Ampere Server To: Patrick Robb Cc: Dharmik Jayesh Thakkar , David Marchand , Ruifeng Wang , Honnappa Nagarahalli , "ci@dpdk.org" , nd Content-Type: multipart/alternative; boundary="0000000000008a3e9906076c695e" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org --0000000000008a3e9906076c695e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Oct 10, 2023 at 5:59=E2=80=AFPM Patrick Robb wr= ote: > > > On Mon, Oct 9, 2023 at 11:56=E2=80=AFPM Dharmik Jayesh Thakkar < > DharmikJayesh.Thakkar@arm.com> wrote: > >> Hi Patrick, >> >> >> >> Can you provide the grub settings? Is iommu.passthrough=3D1 included? >> > > Sure. I'm not sure if you just wanted the kernel cmdline options or the > whole grub config, but I assume you just meant kernel cmdline. Let me kno= w > if you meant more. > > GRUB_CMDLINE_LINUX_DEFAULT=3D"default_hugepagesz=3D1G hugepagesz=3D1G > hugepages=3D32 iommu=3Dpt intel_iommu=3Don isolcpus=3D39-79 nohz_full=3D3= 9-79 > rcu_nocbs=3D39-79 processor.max_cstate=3D1 intel_pstate=3Ddisable > console=3DttyS0,115200 console=3Dtty0" > > But, iommu.passthrough=3D1 is not included, so I can add that if we need = to. > Do you know that this won't have any bad implications for the (intel, > nvidia, broadcom) NICs which we test on this server? > > Just a note here, Patrick. The iommu kernel and intel_pstate parameters aren't supported on arm, so you can remove those. And when iommu.passthrouh=3D1, IOMMU is bypassed and intel_iommu doesn't do anything (and maybe isn't supported on arm, but that's not clear from the docs ), so that can be removed as well. >From what I can tell, using iommu.passthrough=3D1 is the standard, so if there are any negative implications, we should investigate them, but there shouldn't be anything major. > >> >> Also, is qat_c62xvf loaded as well? >> > qat_c62xvf is built in to the kernel also. > > >> >> > >> >> Finally, a few guidelines on the vfio driver: >> >> At times, we need to configure the vfio driver. >> >> On kernel vers. 5.9+ we need to load the vfio-pci driver with the >> additional parameter *disable_denylist=3D1* >> >> Unload the vfio-pci driver if it is already loaded so that we can reload >> it with the correct parameters : >> *sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo >> modprobe -r vfio_virqfd; sudo modprobe -r vfio* >> >> If you can't unload the vfio driver because it's been built into the >> kernel, you'll have to find another way to change VFIO parameters, or to >> rebuild your kernel with VFIO_PCI set as a module. Failing to do that, y= ou >> might encounter issues later on when you try to bind the VFs to VFIO. >> >> Load the vfio-pci driver and bind it to QAT VFs device ids: >> *sudo modprobe vfio-pci disable_denylist=3D1 enable_sriov=3D1 >> vfio-pci.ids=3D8086:37c9* >> >> Enable no-iommu-mode: >> *echo "1" | sudo tee >> /sys/module/vfio/parameters/enable_unsafe_noiommu_mode* >> >> /sys/module/vfio/parameter is missing ? >> >> If /sys/module/vfio/parameters does not exist, you might be missing the >> kernel module VFIO_NOIOMMU >> >> >> >> *Automatically set VFIO params on boot* >> >> It's possible to set these parameters automatically on boot by creating = a >> */etc/modprobe.d/vfio-pci.conf *file with the parameters : >> *cat /etc/modprobe.d/vfio-pci.conf* >> *options vfio enable_unsafe_noiommu_mode=3D1* >> *options vfio-pci disable_denylist=3D1 enable_sriov=3D1 >> vfio-pci.ids=3D8086:37c9* >> >> >> >> We haven=E2=80=99t encountered this issue in the past, so just making su= re the >> configuration is correct. I don=E2=80=99t think having the driver static= /loadable >> should make a difference, I will try with building statically on my setu= p. >> >> >> >> Thank you! >> >> >> Okay, this should be fine. Like I said, we are also running tests on NIC= s > on this server. So, in our Jenkinsfiles scripts for running the testing, = I > will add a preliminary step only for QAT tests which runs: > *sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo > modprobe -r vfio_virqfd; sudo modprobe -r vfio* > *sudo modprobe vfio-pci disable_denylist=3D1 enable_sriov=3D1 > vfio-pci.ids=3D8086:37c9* > *echo "1" | sudo tee > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode* > (then run QAT tests) > > And if running on NICs, have a preliminary step which runs > *sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo > modprobe -r vfio_virqfd; sudo modprobe -r vfio* > *sudo modprobe vfio* > > David does this also sound reasonable to you, per your comment about > isolating this setting to QAT card testing? > > Dharmik if this all sounds okay and you can confirm the iommu.passthrough > change is fine, I will proceed. Thank you for providing the assistance. > > --0000000000008a3e9906076c695e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Tue, Oct 10, 2023 at 5:59=E2=80=AF= PM Patrick Robb <probb@iol.unh.edu<= /a>> wrote:
<= div dir=3D"ltr">


<= div dir=3D"ltr" class=3D"gmail_attr">On Mon, Oct 9, 2023 at 11:56=E2=80=AFP= M Dharmik Jayesh Thakkar <DharmikJayesh.Thakkar@arm.com> wrote:
=

Hi Patrick,

=C2=A0

Can you provide the grub settings? Is iommu.passthro= ugh=3D1 included?


Sur= e. I'm not sure if you just wanted the kernel cmdline options or the wh= ole grub config,=C2=A0but I assume you just meant kernel cmdline. Let me kn= ow if you meant more.

GRUB_CMDLINE_LINUX_DEFAULT= =3D"default_hugepagesz=3D1G hugepagesz=3D1G hugepages=3D32 iommu=3Dpt = intel_iommu=3Don isolcpus=3D39-79 nohz_full=3D39-79 rcu_nocbs=3D39-79 proce= ssor.max_cstate=3D1 intel_pstate=3Ddisable console=3DttyS0,115200 console= =3Dtty0"

But, iommu.passthrough=3D1 is not included, so I can a= dd that if we need to. Do you know that this won't have any bad implica= tions for the (intel, nvidia, broadcom) NICs which we test on this server?<= /div>
=C2=A0

Just a n= ote here, Patrick. The iommu kernel and intel_pstate parameters aren't = supported on arm, so you can remove those. And when iommu.passthrouh=3D1, I= OMMU is bypassed and intel_iommu doesn't do anything (and maybe isn'= ;t supported on arm, but that's not clear from the docs= ), so that can be removed as well.

From what I= can tell, using iommu.passthrough=3D1 is the standard, so if there are any= negative implications, we should investigate them, but there shouldn't= be anything major.
=C2=A0

=C2=A0

Also, is qat_c62xvf loaded as well?

<= /div>
qat_c62xvf is built in to the kernel also.=C2=A0
=C2=A0
<= div lang=3D"EN-US">

=C2=A0

=

=C2=A0

Finally, a few guidelines on the vfio driver:=

At times, we need to configure the vfio driver.

On kernel vers. 5.9+ we need to load the vfio-pci dr= iver with the additional parameter=C2=A0disable_denylist=3D1<= u>

Unload the vfio-pci driver if it is already loaded s= o that we can reload it with the correct parameters :
sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo modpr= obe -r vfio_virqfd; sudo modprobe -r vfio

If you can't unload the vfio driver because it&#= 39;s been built into the kernel, you'll have to find another way to cha= nge VFIO parameters, or to rebuild your kernel with VFIO_PCI set as a modul= e. Failing to do that, you might encounter issues later on when you try to bind the VFs to VFIO.
=C2=A0
Load the vfio-pci driver and bind it to QAT VFs device ids:
sudo modprobe vfio-pci disable_denylist=3D1 enable_sriov=3D1 vfio-pci.id= s=3D8086:37c9
=C2=A0
Enable no-iommu-mode:
echo "1" | sudo tee /sys/module/vfio/parameters/enable_unsafe_= noiommu_mode

=C2=A0/sys/module/vfio/parameter is missing ?=

If /sys/module/vfio/parameters does not exist, you m= ight be missing the kernel module VFIO_NOIOMMU

=C2=A0

Automatically set VFIO params on boot

It's possible to set these parameters automatica= lly on boot by creating a=C2=A0/etc/modprobe.d/vfio-pci.conf=C2=A0fi= le with the parameters :
cat /etc/modprobe.d/vfio-pci.conf
options vfio enable_unsafe_noiommu_mode=3D1
options vfio-pci disable_denylist=3D1 enable_sriov=3D1 vfio-pci.ids=3D80= 86:37c9

=C2=A0

We haven=E2=80=99t encountered this issue in the pas= t, so just making sure the configuration is correct. I don=E2=80=99t think = having the driver static/loadable should make a difference, I will try with= building statically on my setup.

=C2=A0

Thank you!


Okay, this should be fine. Like I said, we ar= e also running tests on NICs on this server. So, in our Jenkinsfiles script= s for running the testing, I will add a preliminary step only for QAT tests= which runs:=C2=A0
sudo modprobe -r vfio_iommu_type1; sudo modprobe -= r vfio_pci; sudo modprobe -r vfio_virqfd; sudo modprobe -r vfio
sudo modprobe vfio-pci disable_denylist=3D1 enable_sriov=3D1 vfio-pci= .ids=3D8086:37c9
echo "1" | sudo tee /sys/module= /vfio/parameters/enable_unsafe_noiommu_mode
(then run QAT tes= ts)

And if running on NICs, have a preliminary ste= p which runs=C2=A0
sudo modprobe -r vfio_iommu_type1; sudo mod= probe -r vfio_pci; sudo modprobe -r vfio_virqfd; sudo modprobe -r vfio<= br>
sudo modprobe vfio

Dav= id does this also sound reasonable to you, per your comment about isolating= this setting to QAT card testing?

Dharmik if this= all sounds okay and you can confirm the iommu.passthrough change is fine, = I will proceed. Thank you for providing the assistance.=C2=A0

--0000000000008a3e9906076c695e--