From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0EE66459B5; Tue, 17 Sep 2024 06:22:20 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D05D140267; Tue, 17 Sep 2024 06:22:19 +0200 (CEST) Received: from mail-oa1-f44.google.com (mail-oa1-f44.google.com [209.85.160.44]) by mails.dpdk.org (Postfix) with ESMTP id 4BBC140261 for ; Tue, 17 Sep 2024 06:22:18 +0200 (CEST) Received: by mail-oa1-f44.google.com with SMTP id 586e51a60fabf-27806115eafso2691204fac.3 for ; Mon, 16 Sep 2024 21:22:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1726546937; x=1727151737; darn=dpdk.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=DiOA5IbgmmBJLH6A8R3T6DARdVumLUu/2N59mzTpvb4=; b=bJcGyWNu8dGK4ya7ebPj9qfvrr3jNSOOpuE/m1KW8JykXZ2SRrkR1xFLcAdq9hae3Y 9IQzljkOYjSpRWL6IiZ1FYJ+NxZbF8A9aGtYWVP4KONyzFUa3kTsAuSI6dVO3h1VMtq7 2kSlmsAUQXoAntNthuJAbLRXlfp+zRoBp0l78= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726546937; x=1727151737; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DiOA5IbgmmBJLH6A8R3T6DARdVumLUu/2N59mzTpvb4=; b=YSzyjBtpaaMihdWzMnj+xCE8Rsj8YX2/Lmnk6w8GQT7gPU1ZWdmxPMl3TGrRz07d9u 49wtzS62o74SVOx0GUeaHsuizkuLZk0pdIFw7/TeNU+ASR/9PkHdAtR0sAKinRLMoWN9 nl+9U8S9eNNw8wZvWzQju2PtWUUIdpPwFhKjgid6Hn0oWVjM294MpxUsKYHmifwBYG8W +OydQ3JuEfCG3vvbwjNiHmF9uOQNwbha1gzStUDKY+z/Fsx+uKyL6C7M0Xz6qKi4wTam 2QVpQs0MxOuBmVjfy6xqVjRozGBcpq7vZXbFZ73qItnRprhF1/ExzmW48018hbk1+RV+ 2R4Q== X-Gm-Message-State: AOJu0YzKvI2XMucFvQG7WtuHRplorcprDfAOLyv09WI+BqN9rR8UGN8S wFK2VWi3iv+BOz3H/dR5LXEQl0fXKX1BFMTcIwDfaPWJeD2nKI9qDHGfw/mfeFGk2G1N8FYFmQ9 lcnx7EaIo/D4u1Kla5eF/rMlHRxVCOyCRhEZoqg== X-Google-Smtp-Source: AGHT+IGWN52CMXdnfAZJ0+q+yLwHWOPOrgsavykymv3Cb+jGDLlOp/LXytiZT3JFEn+SaAgaSIxCAN3rzHaFANx75bY= X-Received: by 2002:a05:6870:548b:b0:278:222c:98c4 with SMTP id 586e51a60fabf-27c3f410605mr12336672fac.21.1726546937391; Mon, 16 Sep 2024 21:22:17 -0700 (PDT) MIME-Version: 1.0 References: <20240916041409.181259-1-probb@iol.unh.edu> <20240916041409.181259-2-probb@iol.unh.edu> In-Reply-To: From: Patrick Robb Date: Tue, 17 Sep 2024 00:21:22 -0400 Message-ID: Subject: Re: [PATCH 1/1] tests/cryptodev_common.py Supporting vfio denylist for QAT To: David Marchand Cc: dts@dpdk.org, Dharmik Jayesh Thakkar Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dts@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: test suite reviews and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dts-bounces@dpdk.org On Mon, Sep 16, 2024 at 5:30=E2=80=AFAM David Marchand wrote: > > On Mon, Sep 16, 2024 at 6:15=E2=80=AFAM Patrick Robb = wrote: > > > > > > diff --git a/tests/cryptodev_common.py b/tests/cryptodev_common.py > > index b550b46869df..37483c51e3e7 100644 > > --- a/tests/cryptodev_common.py > > +++ b/tests/cryptodev_common.py > > @@ -15,6 +15,10 @@ def bind_qat_device(test_case, driver=3D"igb_uio"): > > > > if "crypto_dev_id" in conf.suite_cfg: > > dev_id =3D conf.suite_cfg["crypto_dev_id"] > > + if dev_id in ["37c8", "435", "19e2"]: > > Usually, PCI ids are represented on 4 chars, leading 0 included, so I > would expect 0435. > Do you have such hw and did you test with it? I agree that 3 char id is strange. This was some months ago, but I believe I grabbed the dev ids from https://doc.dpdk.org/guides/cryptodevs/qat.html#available-kernel-drivers based on the kernel commit you posted above. This website, which I usually use when adding a new PCI device to legacy dts, https://devicehunt.com/view/type/pci/vendor/8086/device/0435 suggests that the correct id is 0435. If this sounds right to you, I can submit a new version of this series with the 0 added, and a dpdk/doc patch for the page I included above (presuming that the id in the table is actually wrong). In terms of testing, yes we do have a c62x / 37c8 card (qat 8970), and this commit did "fix" the testsuite for that card. I included the other two cards because even though I didn't have the hardware to test the change, I felt that my reading of the linux kernel commit above indicated it was appropriate to sumit this change, even without having run it on the specific hardware. Please let me know if this is inappropriate. > > > > + test_case.dut.send_expect('modprobe -r vfio_iommu_type1; m= odprobe -r vfio_pci; modprobe -r vfio_virqfd; modprobe -r vfio', '# ', 5) > > + test_case.dut.send_expect('modprobe vfio-pci disable_denyl= ist=3D1 enable_sriov=3D1', '# ', 5) > > While I do understand the disable_denylist option, the enable_sriov=3D > part seems a different topic... > Why is sriov needed in this test? > I believe the enable_sriov requirement comes from the fact that this testsuite is running from virtual functons on the QAT card. You can find an example similar to how we set those up here: https://doc.dpdk.org/guides/cryptodevs/qat.html#binding-the-available-vfs-t= o-the-vfio-pci-driver However, these specific options for the QAT cards in question came from Dharmik Thakkar (ARM) so I'm CCing him here as I think he is more of an authority on the subject than I am. https://inbox.dpdk.org/ci/AS4PR08MB755371CDAF5C105050D64850F7CDA@AS4PR08MB7= 553.eurprd08.prod.outlook.com/ > > > + test_case.dut.send_expect('echo "1" | tee /sys/module/vfio= /parameters/enable_unsafe_noiommu_mode', '# ', 5) > > This helps in systems that do not have a IOMMU (or an emulated one for > virtual machines). > I suspect forcing noiommu will break setups with a hw iommu as DPDK > will force PA when noiommu is detected. This is an ARM Neoverse N-1 Ampere Mt. Jade system, which appears to have I= OMMU: ``` probb@arm-ampere-dut:~$ sudo dmesg | grep -i IOMMU [31804.426192] c6xxvf 0000:03:01.0: Adding to iommu group 59 [31804.473157] c6xxvf 0000:03:01.1: Adding to iommu group 60 [31804.533436] c6xxvf 0000:03:01.2: Adding to iommu group 61 [31804.577145] c6xxvf 0000:03:01.3: Adding to iommu group 62 [31804.621197] c6xxvf 0000:03:01.4: Adding to iommu group 63 [31804.665133] c6xxvf 0000:03:01.5: Adding to iommu group 64 ``` But, I guess this is another question for Dharmik I think, who suggested the enable_unsafe_no_iommu_mode setting and again is more knowledgeable on the subject than I am. However, I am happy to test this with enable_unsafe_noiommu_mode=3D0. I'll give it a whirl now and report back.