From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 36CC842360; Wed, 11 Oct 2023 13:51:50 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2DB8E40648; Wed, 11 Oct 2023 13:51:50 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 4ABB940266 for ; Wed, 11 Oct 2023 13:51:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1697025107; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i9UfnLjd3OVq/T1QZy9FmU4Da/GU48XNsX88HuIJ+iY=; b=LxP4AhLT2f3b16nMP2OURJBtd2781vwYCfC9KCp7hKoi6Y45QFJGgf5mJb+df9lfbw+bnj s+K8BnT1zWqFmDfDIPHXmBRVC8CU5eeCTZA9W02SRls+7L2smgua25PKZrwVPlIxzah1P/ sNN/fhbHdSIjQDtQsWedJ9CC+O1svmw= Received: from mail-lf1-f71.google.com (mail-lf1-f71.google.com [209.85.167.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-106-FZi5VxkjM2ubxu5DLcwZOA-1; Wed, 11 Oct 2023 07:51:46 -0400 X-MC-Unique: FZi5VxkjM2ubxu5DLcwZOA-1 Received: by mail-lf1-f71.google.com with SMTP id 2adb3069b0e04-5043eb2c436so6067322e87.3 for ; Wed, 11 Oct 2023 04:51:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697025105; x=1697629905; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i9UfnLjd3OVq/T1QZy9FmU4Da/GU48XNsX88HuIJ+iY=; b=WIO67A++7mnPwAtn3Q457sYmbNN1tw2hMUoN0gKw4WrUDuaKnNOudRmU7pyXnxzxhZ /x+HDSr+7Ak3vbbCASGe+HOZzqC1ubGkGYoy5rwYNGdzyBfrRMOrqPYAZUc/nrFFUB7C ekCkrf/45/ZaNNqB2WB4xPNLDjsqtIig1T9rheKzejO5s/XKjc413aXIEkXVnrMBSWZu yr2NWRC1drLKY1XmJuyXug7+QoV9xH9LW1IZ0bscjSO7RWq32sBx4DwS+nWY7POJZycO 3hJw2D0yAunnrRgNvs+5v+dnvvEWFsyGDBojrQGSeVTFT60PeZr1YIOoR+ARitFE4cHU 0caA== X-Gm-Message-State: AOJu0YznTmpgCTsajJso6PjGuX8IEbeMUodjriJorajPhLL0Xg9Y2qz9 xKvE4fCJ2xJ+xaBy/+uC0T/j9/VWYNtrK85ptzHO6v+FJ3FIkspiGV66TwaNlrXgZ2Jh+2pGfYB u4cpEHfmHUaVcK6I97w== X-Received: by 2002:a19:6556:0:b0:503:17d6:7dac with SMTP id c22-20020a196556000000b0050317d67dacmr18246238lfj.42.1697025104773; Wed, 11 Oct 2023 04:51:44 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGN4rnP8U4yG4gZwh6sBOr4F295gVACP7AAHeXrZBEUqArbBd9c6yB/VOM0sk9W+DTFRD1UQp4zwiF6p5z7btk= X-Received: by 2002:a19:6556:0:b0:503:17d6:7dac with SMTP id c22-20020a196556000000b0050317d67dacmr18246217lfj.42.1697025104286; Wed, 11 Oct 2023 04:51:44 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: David Marchand Date: Wed, 11 Oct 2023 13:51:32 +0200 Message-ID: Subject: Re: Intel QAT 8970 accel card on ARM Ampere Server To: Patrick Robb Cc: Dharmik Jayesh Thakkar , Ruifeng Wang , =?UTF-8?Q?Juraj_Linke=C5=A1?= , Honnappa Nagarahalli , "ci@dpdk.org" , nd X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org On Tue, Oct 10, 2023 at 6:00=E2=80=AFPM Patrick Robb wr= ote: > On Mon, Oct 9, 2023 at 11:56=E2=80=AFPM Dharmik Jayesh Thakkar wrote: >> >> Hi Patrick, >> >> >> >> Can you provide the grub settings? Is iommu.passthrough=3D1 included? > > > Sure. I'm not sure if you just wanted the kernel cmdline options or the w= hole grub config, but I assume you just meant kernel cmdline. Let me know i= f you meant more. > > GRUB_CMDLINE_LINUX_DEFAULT=3D"default_hugepagesz=3D1G hugepagesz=3D1G hug= epages=3D32 iommu=3Dpt intel_iommu=3Don isolcpus=3D39-79 nohz_full=3D39-79 = rcu_nocbs=3D39-79 processor.max_cstate=3D1 intel_pstate=3Ddisable console= =3DttyS0,115200 console=3Dtty0" > > But, iommu.passthrough=3D1 is not included, so I can add that if we need = to. Do you know that this won't have any bad implications for the (intel, n= vidia, broadcom) NICs which we test on this server? > >> >> >> >> Also, is qat_c62xvf loaded as well? > > qat_c62xvf is built in to the kernel also. > >> >> >> >> >> >> Finally, a few guidelines on the vfio driver: >> >> At times, we need to configure the vfio driver. >> >> On kernel vers. 5.9+ we need to load the vfio-pci driver with the additi= onal parameter disable_denylist=3D1 >> >> Unload the vfio-pci driver if it is already loaded so that we can reload= it with the correct parameters : >> sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo modpr= obe -r vfio_virqfd; sudo modprobe -r vfio >> >> If you can't unload the vfio driver because it's been built into the ker= nel, you'll have to find another way to change VFIO parameters, or to rebui= ld your kernel with VFIO_PCI set as a module. Failing to do that, you might= encounter issues later on when you try to bind the VFs to VFIO. >> >> Load the vfio-pci driver and bind it to QAT VFs device ids: >> sudo modprobe vfio-pci disable_denylist=3D1 enable_sriov=3D1 vfio-pci.id= s=3D8086:37c9 >> >> Enable no-iommu-mode: >> echo "1" | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mo= de >> >> /sys/module/vfio/parameter is missing ? >> >> If /sys/module/vfio/parameters does not exist, you might be missing the = kernel module VFIO_NOIOMMU >> >> >> >> Automatically set VFIO params on boot >> >> It's possible to set these parameters automatically on boot by creating = a /etc/modprobe.d/vfio-pci.conf file with the parameters : >> cat /etc/modprobe.d/vfio-pci.conf >> options vfio enable_unsafe_noiommu_mode=3D1 >> options vfio-pci disable_denylist=3D1 enable_sriov=3D1 vfio-pci.ids=3D80= 86:37c9 >> >> >> >> We haven=E2=80=99t encountered this issue in the past, so just making su= re the configuration is correct. I don=E2=80=99t think having the driver st= atic/loadable should make a difference, I will try with building statically= on my setup. >> >> >> >> Thank you! >> >> > Okay, this should be fine. Like I said, we are also running tests on NICs= on this server. So, in our Jenkinsfiles scripts for running the testing, I= will add a preliminary step only for QAT tests which runs: > sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo modpro= be -r vfio_virqfd; sudo modprobe -r vfio - I thought vfio_iommu_type1 was a x86 thing. So it would work for x86 (Intel/AMD) systems, but fail on other arches.. ? If you tested this on ARM, it is probably ok as is. > sudo modprobe vfio-pci disable_denylist=3D1 enable_sriov=3D1 vfio-pci.ids= =3D8086:37c9 - Speaking to myself, too bad the disable_denylist param value is only read once, when loading the vfio-pci kernel module... So ok, I get why you need to reload the whole chain of kmods. However, I don't think the vfio-pci.ids syntax works for passing parameters= . And in any case, why do you need to set this initial list? Binding devices (using either driverctl or dpdk-devbind.py) to vfio-pci should be done the "usual" way, or is there some special case again for QAT? - Besides, from what I understood so far, there are two parts specific to this QAT test: * enabling SRIOV so that creating VF is possible with a PF bound to vfio-pci (option enable_sriov=3D1), * for a list of PCI QAT cards, forcing the disable_denylist is needed (option disable_denylist=3D1), For the latter point, at this step of the test setup, do you know which QAT devices will be used? If so, the commandline params could be constructed to enable disable_denylist only for known-broken QAT devices (the list is available in the kernel commit Dharmik provided earlier). > echo "1" | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mod= e > (then run QAT tests) > > And if running on NICs, have a preliminary step which runs > sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo modpro= be -r vfio_virqfd; sudo modprobe -r vfio > sudo modprobe vfio Given that vfio_iommu_type1 is ok on other arch, this lgtm. --=20 David Marchand