From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <users-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id 8A10943033
for <public@inbox.dpdk.org>; Fri, 11 Aug 2023 14:37:36 +0200 (CEST)
Received: from mails.dpdk.org (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id CD7EA42D55;
Fri, 11 Aug 2023 14:37:35 +0200 (CEST)
Received: from mail-yb1-f171.google.com (mail-yb1-f171.google.com
[209.85.219.171])
by mails.dpdk.org (Postfix) with ESMTP id 36E5F40E03
for <users@dpdk.org>; Fri, 11 Aug 2023 14:37:34 +0200 (CEST)
Received: by mail-yb1-f171.google.com with SMTP id
3f1490d57ef6-d63c0a6568fso1581201276.0
for <users@dpdk.org>; Fri, 11 Aug 2023 05:37:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20221208; t=1691757453; x=1692362253;
h=cc:to:subject:message-id:date:from:in-reply-to:references
:mime-version:from:to:cc:subject:date:message-id:reply-to;
bh=CP++u263TJAGU5YjriEn8bfDQmP/PYdxHhv6DDD4qbA=;
b=aZpHPeBtlFOdK7mUTu1NWtWgvAzxufXixlIa0ZoWAumw5l8QBpLhf+EYxGCE0fJx6m
uTFNWi5iCo/gAJ3IAnLZjN8Y9UzyDzoFOUKZrixad1L0AVIb04z+jHFCI7IeCBz1o8QJ
jQ5Rsqpra/+A/DQQgzsraAt9m2ETF7lAazWK5cxIZEqEcstMg1iEca5SY0hvRjHO7nZW
0ywCsWm5oGgbXR0EvhenKe3BKijnFa+KJWFy5LYJjmBq5F3IWiDCFXRBCTq0KNB3SJRG
xwv+zYEytLMigoQAaSSFOX0RbFvQOrjhJPUoSjDgwk3yn1Pkca32lgRofthEoSJpzD19
Ntig==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20221208; t=1691757453; x=1692362253;
h=cc:to:subject:message-id:date:from:in-reply-to:references
:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id
:reply-to;
bh=CP++u263TJAGU5YjriEn8bfDQmP/PYdxHhv6DDD4qbA=;
b=dmBzU/Xd6PibITrbdjaRZeHAkCBOmbDxtyeiSbHDjPP7bf223XW4tlayT5vCuDNmN6
NBRnoIEC4iDCB/2N+6r2iAIP75u82GTK0MfDLRWu7PDX0MlnUSbwdJDhIX6f/Wa6Iqov
MKOMAtNXzBq17CRWbm6GXSO3KL5yvHaCr597bx0qL/qIqWgwNxUYQZ4hHJjHAFJg3Nx3
ZcLaQ1rrOw4tUY1O0Kp7QcjleNP2l8K/UXFn4OdOW94vhEdd0wNq5XxY7cZCwc+NJAh4
+ZE34Fq4G75m5EftgtqgXIGPbplkp5jqCe2k6lNFA8Mfh55B/oU/FtdfPUtdt+55lJGC
wqxA==
X-Gm-Message-State: AOJu0Yx4TCX0sTogNGxbZQcjvR26KdiHTeqZ6yaP2XAoDmyAnvTFqh9y
UO2S6AL5gRIvVUVY+kwyArG9rhzRW2QkPB1BJhM=
X-Google-Smtp-Source: AGHT+IHkDQz6pEMZKGrFEuh3m3pTf7iq6PA68X6INTknU9ETrv463VEAjosYU+y+U6xCerj+HkWuqtH6A4KAQsASla8=
X-Received: by 2002:a25:ce12:0:b0:c6f:b843:c502 with SMTP id
x18-20020a25ce12000000b00c6fb843c502mr1528750ybe.25.1691757453324; Fri, 11
Aug 2023 05:37:33 -0700 (PDT)
MIME-Version: 1.0
References: <CALDr01SrSDZXeuVyBC3b0RTvPhN-c7TNJsis1Qq9rMxC+jV3cg@mail.gmail.com>
<MN2PR12MB3085D6D1454F1CD24C9A7B428203A@MN2PR12MB3085.namprd12.prod.outlook.com>
<CALDr01QRBvwMeC0atQZq7K0MtYy6bY80X=u95guei=dQjjqFcg@mail.gmail.com>
<2290932.ElGaqSPkdT@thomas>
In-Reply-To: <2290932.ElGaqSPkdT@thomas>
From: Igor de Paula <igordptx@gmail.com>
Date: Fri, 11 Aug 2023 15:37:22 +0300
Message-ID: <CALDr01Tv7VQTczidUSjQkp94VS8cG=yLKTC9ayX_b8iSjGjfJg@mail.gmail.com>
Subject: Re: help
To: Thomas Monjalon <thomas@monjalon.net>
Cc: "Varghese, Vipin" <Vipin.Varghese@amd.com>, "Yigit,
Ferruh" <Ferruh.Yigit@amd.com>,
Jochen Behrens <jbehrens@vmware.com>, "users@dpdk.org" <users@dpdk.org>,
"Gupta, Nipun" <Nipun.Gupta@amd.com>, "Agarwal,
Nikhil" <nikhil.agarwal@amd.com>,
Ronak Doshi <doshir@vmware.com>, "Immanni, Venkat" <Venkat.Immanni@amd.com>,
Chenbo Xia <chenbo.xia@intel.com>
Content-Type: multipart/alternative; boundary="0000000000000d36d10602a4fa9c"
X-BeenThere: users@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK usage discussions <users.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/users>,
<mailto:users-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/users/>
List-Post: <mailto:users@dpdk.org>
List-Help: <mailto:users-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/users>,
<mailto:users-request@dpdk.org?subject=subscribe>
Errors-To: users-bounces@dpdk.org
--0000000000000d36d10602a4fa9c
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Sure, I will submit a patch (it would be a first).
Thanks
On Fri, Aug 11, 2023 at 3:30=E2=80=AFPM Thomas Monjalon <thomas@monjalon.ne=
t> wrote:
> Thanks for the info.
> Do you think it should be written in the vmxnet3 page of the DPDK
> documentation?
> If yes, would you like to initiate a patch for review?
>
>
> 11/08/2023 10:43, Igor de Paula:
> > Hi again,
> > I got this resolved with VMWARE support so I thought to share it here.
> > What I originally wanted was to use IOVA-VA on an AMD host. Which didn'=
t
> > work. I have learned that the ESXI version that supports
> > virtual IOMMU in AMD hosts (which is a prerequisite to IOVA-VA) is ESXI
> 7.0
> > U1. After updating it worked. On Intel hosts ESXI 6.7 supports it alrea=
dy
> > as far as I know.
> >
> >
> > On Tue, Jul 25, 2023 at 6:19=E2=80=AFPM Varghese, Vipin <Vipin.Varghese=
@amd.com>
> > wrote:
> >
> > > [AMD Official Use Only - General]
> > >
> > > Like I said earlier, trying with the Intel host I have on VMWARE,
> > > specifically Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> > > With IOMMU enabled, VMXNET3 works with VA as well as PA.
> > >
> > > [VV] since ` enable_unsafe_iommu: not enabled` on Intel platform, cou=
ld
> > > it be possible the specific version EXSI hypervisor supports the HW
> IOMMU
> > > specific to the platform. My suspicion will be in case on AMD platfor=
m
> > > changes for required to enable HW iommu might not be available to
> specific
> > > EXSI (hypervior OS) used.
> > >
> > >
> > >
> > > I am not an expert on virtio_user PMD, but I can check if it will wor=
k
> > > with PA with deferred setting for vmx_net3 PMD are ok?
> > >
> > >
> > >
> > > *From:* Igor de Paula <igordptx@gmail.com>
> > > *Sent:* Tuesday, July 25, 2023 8:42 PM
> > > *To:* Varghese, Vipin <Vipin.Varghese@amd.com>
> > > *Cc:* Yigit, Ferruh <Ferruh.Yigit@amd.com>; Jochen Behrens <
> > > jbehrens@vmware.com>; Thomas Monjalon <thomas@monjalon.net>;
> > > users@dpdk.org; Gupta, Nipun <Nipun.Gupta@amd.com>; Agarwal, Nikhil <
> > > nikhil.agarwal@amd.com>; Ronak Doshi <doshir@vmware.com>; Immanni,
> Venkat
> > > <Venkat.Immanni@amd.com>; Chenbo Xia <chenbo.xia@intel.com>
> > > *Subject:* Re: help
> > >
> > >
> > >
> > > *Caution:* This message originated from an External Source. Use prope=
r
> > > caution when opening attachments, clicking links, or responding.
> > >
> > >
> > >
> > > Well,
> > > Like I said earlier, trying with the Intel host I have on VMWARE,
> > > specifically Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> > > With IOMMU enabled, VMXNET3 works with VA as well as PA.
> > > Meaning, PA works regardless if IOMMU is enabled or not. From my
> > > experience anyway.
> > > That's why I thought that:
> > > virtio_user needs VA to work.
> > > For some reason VMXNET3 does not work with VA (only on AMD host).
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Jul 25, 2023 at 4:04=E2=80=AFPM Varghese, Vipin <
> Vipin.Varghese@amd.com>
> > > wrote:
> > >
> > > [AMD Official Use Only - General]
> > >
> > >
> > >
> > > Thanks Igor,
> > >
> > >
> > >
> > > As suspected the vmx_net3 works with
> > >
> > >
> > >
> > > 1. Iommu: disabled
> > > 2. enable_unsafe_iommu: enabled
> > > 3. dpdk eal iova mode: PA
> > >
> > >
> > >
> > > as pointed by you in logs, the virtio_user fails as it expects VA too=
.
> > >
> > >
> > >
> > > Will check and get back.
> > >
> > >
> > >
> > > *From:* Igor de Paula <igordptx@gmail.com>
> > > *Sent:* Tuesday, July 25, 2023 8:16 PM
> > > *To:* Yigit, Ferruh <Ferruh.Yigit@amd.com>
> > > *Cc:* Jochen Behrens <jbehrens@vmware.com>; Thomas Monjalon <
> > > thomas@monjalon.net>; users@dpdk.org; Gupta, Nipun <
> Nipun.Gupta@amd.com>;
> > > Agarwal, Nikhil <nikhil.agarwal@amd.com>; Ronak Doshi <
> doshir@vmware.com>;
> > > Immanni, Venkat <Venkat.Immanni@amd.com>; Varghese, Vipin <
> > > Vipin.Varghese@amd.com>; Chenbo Xia <chenbo.xia@intel.com>
> > > *Subject:* Re: help
> > >
> > >
> > >
> > > *Caution:* This message originated from an External Source. Use prope=
r
> > > caution when opening attachments, clicking links, or responding.
> > >
> > >
> > >
> > > Hi,
> > > Attaching the logs of EAL when trying to run a configuration with
> > > virtio_user port when IOMMU is
> > > disabled and enable_unsafe_iommu is enabled. As you can see it forces
> IOVA
> > > as PA but the viritui_user needs IOVA as VA.
> > > I am also attaching the output of dmesg. I am not sure which kernel
> logs
> > > you wanted... if there is anything else please let me know..
> > > Regarding the ESXI logs, they are HUGE so I will send to you on a
> separate
> > > email.
> > >
> > >
> > >
> > > On Fri, Jul 21, 2023 at 1:14=E2=80=AFPM Ferruh Yigit <ferruh.yigit@am=
d.com>
> wrote:
> > >
> > > On 7/21/2023 12:39 PM, Igor de Paula wrote:
> > > > I am trying to use virtio_user for an interface with the
> > > > kernel:
> > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html =
<
> > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html>
> > > > I think this requires IOVA as va.
> > > >
> > >
> > > I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo=
,
> > > he may know better.
> > >
> > > Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and
> > > '--iova-mode=3Dpa'?
> > >
> > >
> > > > It does work with Intel host and IOMMU
> > > > enabled. Part of the negotiation when setting it up is getting the
> IOMMU
> > > > number so I thought it has to have IOMMU.
> > > >
> > >
> > > Yes, issue looks like related to the IOMMU, and it may be either
> related
> > > to HW support, or ESXi iommu driver support, we will check using belo=
w
> > > information you provided.
> > >
> > > > I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but
> > > > again, that didn't work.
> > > > ESXI version - VMware ESXi, 7.0.0, 16324942
> > > > AMD: AMD EPYC 7452 32-Core Processo
> > > >
> > > > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @
> 2.60GHz
> > > > Regarding the logs I will try and attach it soon.
> > > >
> > >
> > > Thanks for info, waiting for logs.
> > >
> > > >
> > > > On Fri, Jul 21, 2023 at 12:21=E2=80=AFPM Ferruh Yigit <ferruh.yigit=
@amd.com
> > > > <mailto:ferruh.yigit@amd.com>> wrote:
> > > >
> > > > Hi Igor,
> > > >
> > > > VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this
> can you
> > > > please try with:
> > > > - enable 'enable_unsafe_noiommu_mode' flag
> > > > - Force PA mode via '--iova-mode=3Dpa' eal parameter
> > > >
> > > >
> > > > Also to be able to figure out AMD IOMMU support level, can you
> please
> > > > provide:
> > > > - AMD part number
> > > > - ESXi/hypervisor version
> > > > - ESXi & VM kernel logs
> > > >
> > > >
> > > > Thanks,
> > > > Ferruh
> > > >
> > > > On 7/20/2023 5:21 PM, Jochen Behrens wrote:
> > > > > +Ronak from the ESX team
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > In our usage, we do set amd_iommu=3Doff in the boot command l=
ine
> from
> > > > > grub. (Or intel_iommu=3Doff for Intel processors.)
> > > > >
> > > > >
> > > > >
> > > > > Jochen
> > > > >
> > > > >
> > > > >
> > > > > *From: *Thomas Monjalon <thomas@monjalon.net
> > > > <mailto:thomas@monjalon.net>>
> > > > > *Date: *Thursday, July 20, 2023 at 6:00 AM
> > > > > *To: *Igor de Paula <igordptx@gmail.com <mailto:
> igordptx@gmail.com
> > > >>
> > > > > *Cc: *users@dpdk.org <mailto:users@dpdk.org> <users@dpdk.org
> > > > <mailto:users@dpdk.org>>, Jochen Behrens
> > > > > <jbehrens@vmware.com <mailto:jbehrens@vmware.com>>, Nipun
> Gupta
> > > > <nipun.gupta@amd.com <mailto:nipun.gupta@amd.com>>, Nikhil
> Agarwal
> > > > > <nikhil.agarwal@amd.com <mailto:nikhil.agarwal@amd.com>>,
> Ferruh
> > > > Yigit <ferruh.yigit@amd.com <mailto:ferruh.yigit@amd.com>>
> > > > > *Subject: *Re: help
> > > > >
> > > > > !! External Email
> > > > >
> > > > > +Cc some AMD maintainers, they can have an idea about IOMMU
> > > settings.
> > > > >
> > > > >
> > > > > 20/07/2023 14:44, Igor de Paula:
> > > > >> I have enabled it in the host and in the BIOS for AMD...
> > > > >> In the Bios I changed to amd_iommu=3Don and in the host it's=
the
> > > > same for
> > > > >> either.
> > > > >>
> > > > >> On Thu, Jul 20, 2023 at 1:31=E2=80=AFPM Thomas Monjalon
> > > > <thomas@monjalon.net <mailto:thomas@monjalon.net>> wrote:
> > > > >>
> > > > >> > 20/07/2023 11:35, Igor de Paula:
> > > > >> > > The weird thing is that it only happens when I am using =
a
> > > > host with an
> > > > >> > AMD
> > > > >> > > processor. It doesn't happen when I use a host with an
> Intel
> > > > processor.
> > > > >> >
> > > > >> > So it's probably a matter of BIOS settings for the IOMMU?
> > > > >> >
> > > > >> >
> > > > >> > > On Thu, Jul 20, 2023 at 10:32=E2=80=AFAM Thomas Monjalon
> > > > <thomas@monjalon.net <mailto:thomas@monjalon.net>>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > +Cc the vmxnet3 maintainer.
> > > > >> > > >
> > > > >> > > > Please Jochen, do you have an idea what's wrong below?
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > 20/07/2023 11:25, Igor de Paula:
> > > > >> > > > > This is because it can't negotiate the IOMMU type wi=
th
> > > > any port.
> > > > >> > > > >
> > > > >> > > > > On Thu, Jul 20, 2023 at 5:08=E2=80=AFAM Thomas Monja=
lon
> > > > <thomas@monjalon.net <mailto:thomas@monjalon.net>
> > > > >> > >
> > > > >> > > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Hello,
> > > > >> > > > > >
> > > > >> > > > > > The first error is "Cause: Error: number of ports
> must
> > > > be even"
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > 03/05/2023 18:13, Igor de Paula:
> > > > >> > > > > > > I am running a VM inside a VMWARE server
> (vSphere).
> > > > >> > > > > > > My goal it to set up DPDK with two HW ports, and
> set
> > > up a
> > > > >> > > > virtio_user to
> > > > >> > > > > > > interact with the kernel stack.
> > > > >> > > > > > > In another app I have it working but instead of
> > > > virtio_user I am
> > > > >> > > > running
> > > > >> > > > > > > KNI, it works in IOVA-PA mode.
> > > > >> > > > > > > I am looking to replace the KNI.
> > > > >> > > > > > >
> > > > >> > > > > > > When I try to set up virtio_user port as in the
> doc:
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > >
> > > > >> >
> > > >
> > >
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virt=
io-user-as-exception-path
> > > <
> > >
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virt=
io-user-as-exception-path
> >
> > > <
> > >
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virt=
io-user-as-exception-path
> > > <
> > >
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virt=
io-user-as-exception-path
> > > >>
> > > > >> > > > > > > I get a error it can't run in PA mode.
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > When I try to run as VA mode from a parameter, I
> get
> > > the
> > > > >> > following
> > > > >> > > > > > errors:
> > > > >> > > > > > > EAL: lib.eal log level changed from info to debu=
g
> > > > >> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0
> > > > >> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0
> > > > >> > > > > > > EAL: Support maximum 128 logical core(s) by
> > > > configuration.
> > > > >> > > > > > > EAL: Detected 2 lcore(s)
> > > > >> > > > > > > EAL: Detected 1 NUMA nodes
> > > > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.=
3'
> > > > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21'
> > > > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so'
> > > > >> > > > > > > EAL: Detected static linkage of DPDK
> > > > >> > > > > > > EAL: Ask a virtual area of 0x7000 bytes
> > > > >> > > > > > > EAL: Virtual area found at 0x100000000 (size =3D
> 0x7000)
> > > > >> > > > > > > EAL: Multi-process socket
> /var/run/dpdk/rte/mp_socket
> > > > >> > > > > > > EAL: DPAA Bus not present. Skipping.
> > > > >> > > > > > > EAL: VFIO PCI modules not loaded
> > > > >> > > > > > > EAL: Selected IOVA mode 'VA'
> > > > >> > > > > > > EAL: Probing VFIO support...
> > > > >> > > > > > > EAL: IOMMU type 1 (Type 1) is supported
> > > > >> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported
> > > > >> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported
> > > > >> > > > > > > EAL: VFIO support initialized
> > > > >> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes
> > > > >> > > > > > > EAL: Virtual area found at 0x100007000 (size =3D
> > > 0x5b000)
> > > > >> > > > > > > EAL: Setting up physically contiguous memory...
> > > > >> > > > > > > EAL: Setting maximum number of open files to
> 1048576
> > > > >> > > > > > > EAL: Detected memory type: socket_id:0
> > > > hugepage_sz:1073741824
> > > > >> > > > > > > EAL: Creating 2 segment lists: n_segs:128
> socket_id:0
> > > > >> > > > > > hugepage_sz:1073741824
> > > > >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> > > > >> > > > > > > EAL: Virtual area found at 0x100062000 (size =3D
> 0x2000)
> > > > >> > > > > > > EAL: Memseg list allocated at socket 0, page siz=
e
> > > > 0x100000kB
> > > > >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > > >> > > > > > > EAL: Virtual area found at 0x140000000 (size =3D
> > > > 0x2000000000)
> > > > >> > > > > > > EAL: VA reserved for memseg list at 0x140000000,
> size
> > > > 2000000000
> > > > >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> > > > >> > > > > > > EAL: Virtual area found at 0x2140000000 (size =
=3D
> > > 0x2000)
> > > > >> > > > > > > EAL: Memseg list allocated at socket 0, page siz=
e
> > > > 0x100000kB
> > > > >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > > >> > > > > > > EAL: Virtual area found at 0x2180000000 (size =
=3D
> > > > 0x2000000000)
> > > > >> > > > > > > EAL: VA reserved for memseg list at 0x2180000000=
,
> > > > size 2000000000
> > > > >> > > > > > > EAL: TSC frequency is ~2350000 KHz
> > > > >> > > > > > > EAL: Main lcore 0 is ready
> > > (tid=3D7f8ad790ec00;cpuset=3D[0])
> > > > >> > > > > > > EAL: lcore 1 is ready
> (tid=3D7f8ad6907400;cpuset=3D[1])
> > > > >> > > > > > > EAL: Trying to obtain current memory policy.
> > > > >> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0
> > > > >> > > > > > > EAL: Restoring previous memory policy: 0
> > > > >> > > > > > > EAL: request: mp_malloc_sync
> > > > >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> > > > >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3
> > > > >> > > > > > > EAL: Expecting 'PA' IOVA mode but current mode
> is
> > > > 'VA', not
> > > > >> > > > > > initializing
> > > > >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be use=
d
> > > > >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3
> > > > >> > > > > > > EAL: Expecting 'PA' IOVA mode but current mode
> is
> > > > 'VA', not
> > > > >> > > > > > initializing
> > > > >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be use=
d
> > > > >> > > > > > > EAL: Bus (pci) probe failed.
> > > > >> > > > > > > EAL: lib.telemetry log level changed from
> disabled to
> > > > warning
> > > > >> > > > > > > EAL: Error - exiting with code: 1
> > > > >> > > > > > > Cause: Error: number of ports must be even
> > > > >> > > > > > > EAL: request: mp_malloc_sync
> > > > >> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > For some reason the HW ports won't setup. From
> what I
> > > > understand
> > > > >> > > > > > > net_vmxnet3 should work with VA mode.
> > > > >> > > > > > > I enabled I/OMUU for the VM.
> > > > >> > > > > > > The weird thing even when enabled, I still have
> the
> > > > >> > > > > > > enable_unsafe_noiommu_mode flag on.
> > > > >> > > > > > > And because it's on the this:
> > > > >> > > > > > >
> > > > >> > > > > > > dev_iova_mode =3D pci_device_iova_mode(dr, dev);
> > > > >> > > > > > >
> > > > >> > > > > > > return PA mode, and it fails.
> > > > >> > > > > > >
> > > > >> > > > > > > When I disable it by modifying
> > > > >> > > > > > >
> > > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get
> > > > >> > another
> > > > >> > > > > > error.
> > > > >> > > > > > > The error is that it doesn't find a suitable IOM=
MU
> > > type:
> > > > >> > > > > > > Just putting the relevant message:
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> > > > >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3
> > > > >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19
> (No
> > > > such device)
> > > > >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (=
No
> > > > such device)
> > > > >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 1=
9
> (No
> > > > such
> > > > >> > device)
> > > > >> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type
> > > > >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be use=
d
> > > > >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3
> > > > >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19
> (No
> > > > such device)
> > > > >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (=
No
> > > > such device)
> > > > >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 1=
9
> (No
> > > > such
> > > > >> > device)
> > > > >> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type
> > > > >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be use=
d
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > I can see where it fails:
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > const struct vfio_iommu_type *
> > > > >> > > > > > > vfio_set_iommu_type(int vfio_container_fd)
> > > > >> > > > > > > {
> > > > >> > > > > > > unsigned idx;
> > > > >> > > > > > > for (idx =3D 0; idx < RTE_DIM(iommu_types); idx+=
+) {
> > > > >> > > > > > > const struct vfio_iommu_type *t =3D
> &iommu_types[idx];
> > > > >> > > > > > >
> > > > >> > > > > > > int ret =3D ioctl(vfio_container_fd, VFIO_SET_IO=
MMU,
> > > > >> > > > > > > t->type_id);
> > > > >> > > > > > > if (!ret) {
> > > > >> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> > > > >> > > > > > > t->type_id, t->name);
> > > > >> > > > > > > return t;
> > > > >> > > > > > > }
> > > > >> > > > > > > /* not an error, there may be more supported IOM=
MU
> > > > types */
> > > > >> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s)
> failed,
> > > > error "
> > > > >> > > > > > > "%i (%s)\n", t->type_id, t->name, errno,
> > > > >> > > > > > > strerror(errno));
> > > > >> > > > > > > }
> > > > >> > > > > > > /* if we didn't find a suitable IOMMU type, fail
> */
> > > > >> > > > > > > return NULL;
> > > > >> > > > > > > }
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > The ioctl returns -1.
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > I tried many things.
> > > > >> > > > > > > Enabling/disabling IOMMU.
> > > > >> > > > > > > Modifying the enable_unsafe_noiommu_mode flag.
> > > > >> > > > > > > Running in PA mode.
> > > > >> > > > > > > Any help will be much appreciated.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > !! External Email: This email originated from outside of the
> > > > > organization. Do not click links or open attachments unless y=
ou
> > > > > recognize the sender.
> > > > >
> > > >
> > >
> > >
> >
>
>
>
>
>
>
--0000000000000d36d10602a4fa9c
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Sure, I will submit a patch (it would be a first).=C2=A0<b=
r>Thanks</div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmai=
l_attr">On Fri, Aug 11, 2023 at 3:30=E2=80=AFPM Thomas Monjalon <<a href=
=3D"mailto:thomas@monjalon.net">thomas@monjalon.net</a>> wrote:<br></div=
><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border=
-left:1px solid rgb(204,204,204);padding-left:1ex">Thanks for the info.<br>
Do you think it should be written in the vmxnet3 page of the DPDK documenta=
tion?<br>
If yes, would you like to initiate a patch for review?<br>
<br>
<br>
11/08/2023 10:43, Igor de Paula:<br>
> Hi again,<br>
> I got this resolved with VMWARE support so I thought to share it here.=
<br>
> What I originally wanted was to use IOVA-VA on an AMD host. Which didn=
't<br>
> work. I have learned that the ESXI version that supports<br>
> virtual IOMMU in AMD hosts (which is a prerequisite to IOVA-VA) is ESX=
I 7.0<br>
> U1. After updating it worked. On Intel hosts ESXI 6.7 supports it alre=
ady<br>
> as far as I know.<br>
> <br>
> <br>
> On Tue, Jul 25, 2023 at 6:19=E2=80=AFPM Varghese, Vipin <<a href=3D=
"mailto:Vipin.Varghese@amd.com" target=3D"_blank">Vipin.Varghese@amd.com</a=
>><br>
> wrote:<br>
> <br>
> > [AMD Official Use Only - General]<br>
> ><br>
> > Like I said earlier, trying with the Intel host I have on VMWARE,=
<br>
> > specifically=C2=A0 Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz<br>
> > With IOMMU enabled, VMXNET3 works with VA as well as PA.<br>
> ><br>
> > [VV] since ` enable_unsafe_iommu: not enabled` on Intel platform,=
could<br>
> > it be possible the specific version EXSI hypervisor supports the =
HW IOMMU<br>
> > specific to the platform. My suspicion will be in case on AMD pla=
tform<br>
> >=C2=A0 changes for required to enable HW iommu might not be availa=
ble to specific<br>
> > EXSI (hypervior OS) used.<br>
> ><br>
> ><br>
> ><br>
> > I am not an expert on virtio_user PMD, but I can check if it will=
work<br>
> > with PA with deferred setting for vmx_net3 PMD are ok?<br>
> ><br>
> ><br>
> ><br>
> > *From:* Igor de Paula <<a href=3D"mailto:igordptx@gmail.com" t=
arget=3D"_blank">igordptx@gmail.com</a>><br>
> > *Sent:* Tuesday, July 25, 2023 8:42 PM<br>
> > *To:* Varghese, Vipin <<a href=3D"mailto:Vipin.Varghese@amd.co=
m" target=3D"_blank">Vipin.Varghese@amd.com</a>><br>
> > *Cc:* Yigit, Ferruh <<a href=3D"mailto:Ferruh.Yigit@amd.com" t=
arget=3D"_blank">Ferruh.Yigit@amd.com</a>>; Jochen Behrens <<br>
> > <a href=3D"mailto:jbehrens@vmware.com" target=3D"_blank">jbehrens=
@vmware.com</a>>; Thomas Monjalon <<a href=3D"mailto:thomas@monjalon.=
net" target=3D"_blank">thomas@monjalon.net</a>>;<br>
> > <a href=3D"mailto:users@dpdk.org" target=3D"_blank">users@dpdk.or=
g</a>; Gupta, Nipun <<a href=3D"mailto:Nipun.Gupta@amd.com" target=3D"_b=
lank">Nipun.Gupta@amd.com</a>>; Agarwal, Nikhil <<br>
> > <a href=3D"mailto:nikhil.agarwal@amd.com" target=3D"_blank">nikhi=
l.agarwal@amd.com</a>>; Ronak Doshi <<a href=3D"mailto:doshir@vmware.=
com" target=3D"_blank">doshir@vmware.com</a>>; Immanni, Venkat<br>
> > <<a href=3D"mailto:Venkat.Immanni@amd.com" target=3D"_blank">V=
enkat.Immanni@amd.com</a>>; Chenbo Xia <<a href=3D"mailto:chenbo.xia@=
intel.com" target=3D"_blank">chenbo.xia@intel.com</a>><br>
> > *Subject:* Re: help<br>
> ><br>
> ><br>
> ><br>
> > *Caution:* This message originated from an External Source. Use p=
roper<br>
> > caution when opening attachments, clicking links, or responding.<=
br>
> ><br>
> ><br>
> ><br>
> > Well,<br>
> > Like I said earlier, trying with the Intel host I have on VMWARE,=
<br>
> > specifically=C2=A0 Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz<br>
> > With IOMMU enabled, VMXNET3 works with VA as well as PA.<br>
> > Meaning, PA works regardless if IOMMU is enabled or not. From my<=
br>
> > experience anyway.<br>
> > That's why I thought that:<br>
> > virtio_user needs VA to work.<br>
> > For some reason VMXNET3 does not work with VA (only on AMD host).=
<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Tue, Jul 25, 2023 at 4:04=E2=80=AFPM Varghese, Vipin <<a hr=
ef=3D"mailto:Vipin.Varghese@amd.com" target=3D"_blank">Vipin.Varghese@amd.c=
om</a>><br>
> > wrote:<br>
> ><br>
> > [AMD Official Use Only - General]<br>
> ><br>
> ><br>
> ><br>
> > Thanks Igor,<br>
> ><br>
> ><br>
> ><br>
> > As suspected the vmx_net3 works with<br>
> ><br>
> ><br>
> ><br>
> >=C2=A0 =C2=A0 1. Iommu: disabled<br>
> >=C2=A0 =C2=A0 2. enable_unsafe_iommu: enabled<br>
> >=C2=A0 =C2=A0 3. dpdk eal iova mode: PA<br>
> ><br>
> ><br>
> ><br>
> > as pointed by you in logs, the virtio_user fails as it expects VA=
too.<br>
> ><br>
> ><br>
> ><br>
> > Will check and get back.<br>
> ><br>
> ><br>
> ><br>
> > *From:* Igor de Paula <<a href=3D"mailto:igordptx@gmail.com" t=
arget=3D"_blank">igordptx@gmail.com</a>><br>
> > *Sent:* Tuesday, July 25, 2023 8:16 PM<br>
> > *To:* Yigit, Ferruh <<a href=3D"mailto:Ferruh.Yigit@amd.com" t=
arget=3D"_blank">Ferruh.Yigit@amd.com</a>><br>
> > *Cc:* Jochen Behrens <<a href=3D"mailto:jbehrens@vmware.com" t=
arget=3D"_blank">jbehrens@vmware.com</a>>; Thomas Monjalon <<br>
> > <a href=3D"mailto:thomas@monjalon.net" target=3D"_blank">thomas@m=
onjalon.net</a>>; <a href=3D"mailto:users@dpdk.org" target=3D"_blank">us=
ers@dpdk.org</a>; Gupta, Nipun <<a href=3D"mailto:Nipun.Gupta@amd.com" t=
arget=3D"_blank">Nipun.Gupta@amd.com</a>>;<br>
> > Agarwal, Nikhil <<a href=3D"mailto:nikhil.agarwal@amd.com" tar=
get=3D"_blank">nikhil.agarwal@amd.com</a>>; Ronak Doshi <<a href=3D"m=
ailto:doshir@vmware.com" target=3D"_blank">doshir@vmware.com</a>>;<br>
> > Immanni, Venkat <<a href=3D"mailto:Venkat.Immanni@amd.com" tar=
get=3D"_blank">Venkat.Immanni@amd.com</a>>; Varghese, Vipin <<br>
> > <a href=3D"mailto:Vipin.Varghese@amd.com" target=3D"_blank">Vipin=
.Varghese@amd.com</a>>; Chenbo Xia <<a href=3D"mailto:chenbo.xia@inte=
l.com" target=3D"_blank">chenbo.xia@intel.com</a>><br>
> > *Subject:* Re: help<br>
> ><br>
> ><br>
> ><br>
> > *Caution:* This message originated from an External Source. Use p=
roper<br>
> > caution when opening attachments, clicking links, or responding.<=
br>
> ><br>
> ><br>
> ><br>
> > Hi,<br>
> > Attaching the logs of EAL when trying to run a configuration with=
<br>
> > virtio_user port when IOMMU is<br>
> > disabled and enable_unsafe_iommu is enabled. As you can see it fo=
rces IOVA<br>
> > as PA but the viritui_user needs IOVA as VA.<br>
> > I am also attaching the output of dmesg. I am not sure which kern=
el logs<br>
> > you wanted... if there is anything else please let me know..<br>
> > Regarding the ESXI logs, they are HUGE so I will send to you on a=
separate<br>
> > email.<br>
> ><br>
> ><br>
> ><br>
> > On Fri, Jul 21, 2023 at 1:14=E2=80=AFPM Ferruh Yigit <<a href=
=3D"mailto:ferruh.yigit@amd.com" target=3D"_blank">ferruh.yigit@amd.com</a>=
> wrote:<br>
> ><br>
> > On 7/21/2023 12:39 PM, Igor de Paula wrote:<br>
> > > I am trying to use virtio_user for an interface with the<br>
> > > kernel:<br>
> > <a href=3D"https://doc.dpdk.org/guides/howto/virtio_user_as_excep=
tion_path.html" rel=3D"noreferrer" target=3D"_blank">https://doc.dpdk.org/g=
uides/howto/virtio_user_as_exception_path.html</a> <<br>
> > <a href=3D"https://doc.dpdk.org/guides/howto/virtio_user_as_excep=
tion_path.html" rel=3D"noreferrer" target=3D"_blank">https://doc.dpdk.org/g=
uides/howto/virtio_user_as_exception_path.html</a>><br>
> > > I think this requires IOVA as va.<br>
> > ><br>
> ><br>
> > I am not sure if virtio-user has IOVA as VA requirement, cc'e=
d Chenbo,<br>
> > he may know better.<br>
> ><br>
> > Meanwhile can you give a try to 'enable_unsafe_noiommu_mode&#=
39; and<br>
> > '--iova-mode=3Dpa'?<br>
> ><br>
> ><br>
> > > It does work with Intel host and IOMMU<br>
> > > enabled. Part of the negotiation when setting it up is getti=
ng the IOMMU<br>
> > > number so I thought it has to have IOMMU.<br>
> > ><br>
> ><br>
> > Yes, issue looks like related to the IOMMU, and it may be either =
related<br>
> > to HW support, or ESXi iommu driver support, we will check using =
below<br>
> > information you provided.<br>
> ><br>
> > > I tried disabling IOMMU and enabling enable_unsafe_noiommu f=
lag but<br>
> > > again, that didn't work.<br>
> > > ESXI version - VMware ESXi, 7.0.0, 16324942<br>
> > > AMD:=C2=A0 AMD EPYC 7452 32-Core Processo<br>
> > ><br>
> > > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A=
v4 @ 2.60GHz<br>
> > > Regarding the logs I will try and attach it soon.<br>
> > ><br>
> ><br>
> > Thanks for info, waiting for logs.<br>
> ><br>
> > ><br>
> > > On Fri, Jul 21, 2023 at 12:21=E2=80=AFPM Ferruh Yigit <<a=
href=3D"mailto:ferruh.yigit@amd.com" target=3D"_blank">ferruh.yigit@amd.co=
m</a><br>
> > > <mailto:<a href=3D"mailto:ferruh.yigit@amd.com" target=3D=
"_blank">ferruh.yigit@amd.com</a>>> wrote:<br>
> > ><br>
> > >=C2=A0 =C2=A0 =C2=A0Hi Igor,<br>
> > ><br>
> > >=C2=A0 =C2=A0 =C2=A0VM doesn't have IOMMU, and vmxnet3 re=
quires PA mode, for this can you<br>
> > >=C2=A0 =C2=A0 =C2=A0please try with:<br>
> > >=C2=A0 =C2=A0 =C2=A0- enable 'enable_unsafe_noiommu_mode&=
#39; flag<br>
> > >=C2=A0 =C2=A0 =C2=A0- Force PA mode via '--iova-mode=3Dpa=
' eal parameter<br>
> > ><br>
> > ><br>
> > >=C2=A0 =C2=A0 =C2=A0Also to be able to figure out AMD IOMMU s=
upport level, can you please<br>
> > >=C2=A0 =C2=A0 =C2=A0provide:<br>
> > >=C2=A0 =C2=A0 =C2=A0- AMD part number<br>
> > >=C2=A0 =C2=A0 =C2=A0- ESXi/hypervisor version<br>
> > >=C2=A0 =C2=A0 =C2=A0- ESXi & VM kernel logs<br>
> > ><br>
> > ><br>
> > >=C2=A0 =C2=A0 =C2=A0Thanks,<br>
> > >=C2=A0 =C2=A0 =C2=A0Ferruh<br>
> > ><br>
> > >=C2=A0 =C2=A0 =C2=A0On 7/20/2023 5:21 PM, Jochen Behrens wrot=
e:<br>
> > >=C2=A0 =C2=A0 =C2=A0> +Ronak from the ESX team<br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0> In our usage, we do set amd_iommu=3D=
off in the boot command line from<br>
> > >=C2=A0 =C2=A0 =C2=A0> grub. (Or intel_iommu=3Doff for Inte=
l processors.)<br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0Jochen<br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0> *From: *Thomas Monjalon <<a href=
=3D"mailto:thomas@monjalon.net" target=3D"_blank">thomas@monjalon.net</a><b=
r>
> > >=C2=A0 =C2=A0 =C2=A0<mailto:<a href=3D"mailto:thomas@monja=
lon.net" target=3D"_blank">thomas@monjalon.net</a>>><br>
> > >=C2=A0 =C2=A0 =C2=A0> *Date: *Thursday, July 20, 2023 at 6=
:00 AM<br>
> > >=C2=A0 =C2=A0 =C2=A0> *To: *Igor de Paula <<a href=3D"m=
ailto:igordptx@gmail.com" target=3D"_blank">igordptx@gmail.com</a> <mail=
to:<a href=3D"mailto:igordptx@gmail.com" target=3D"_blank">igordptx@gmail.c=
om</a><br>
> > >><br>
> > >=C2=A0 =C2=A0 =C2=A0> *Cc: *<a href=3D"mailto:users@dpdk.o=
rg" target=3D"_blank">users@dpdk.org</a> <mailto:<a href=3D"mailto:users=
@dpdk.org" target=3D"_blank">users@dpdk.org</a>> <<a href=3D"mailto:u=
sers@dpdk.org" target=3D"_blank">users@dpdk.org</a><br>
> > >=C2=A0 =C2=A0 =C2=A0<mailto:<a href=3D"mailto:users@dpdk.o=
rg" target=3D"_blank">users@dpdk.org</a>>>, Jochen Behrens<br>
> > >=C2=A0 =C2=A0 =C2=A0> <<a href=3D"mailto:jbehrens@vmwar=
e.com" target=3D"_blank">jbehrens@vmware.com</a> <mailto:<a href=3D"mail=
to:jbehrens@vmware.com" target=3D"_blank">jbehrens@vmware.com</a>>>, =
Nipun Gupta<br>
> > >=C2=A0 =C2=A0 =C2=A0<<a href=3D"mailto:nipun.gupta@amd.com=
" target=3D"_blank">nipun.gupta@amd.com</a> <mailto:<a href=3D"mailto:ni=
pun.gupta@amd.com" target=3D"_blank">nipun.gupta@amd.com</a>>>, Nikhi=
l Agarwal<br>
> > >=C2=A0 =C2=A0 =C2=A0> <<a href=3D"mailto:nikhil.agarwal=
@amd.com" target=3D"_blank">nikhil.agarwal@amd.com</a> <mailto:<a href=
=3D"mailto:nikhil.agarwal@amd.com" target=3D"_blank">nikhil.agarwal@amd.com=
</a>>>, Ferruh<br>
> > >=C2=A0 =C2=A0 =C2=A0Yigit <<a href=3D"mailto:ferruh.yigit@=
amd.com" target=3D"_blank">ferruh.yigit@amd.com</a> <mailto:<a href=3D"m=
ailto:ferruh.yigit@amd.com" target=3D"_blank">ferruh.yigit@amd.com</a>>&=
gt;<br>
> > >=C2=A0 =C2=A0 =C2=A0> *Subject: *Re: help<br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0> !! External Email<br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0> +Cc some AMD maintainers, they can h=
ave an idea about IOMMU<br>
> > settings.<br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0> 20/07/2023 14:44, Igor de Paula:<br>
> > >=C2=A0 =C2=A0 =C2=A0>> I have enabled it in the host an=
d in the BIOS for AMD...<br>
> > >=C2=A0 =C2=A0 =C2=A0>> In the Bios I changed to amd_iom=
mu=3Don and in the host it's the<br>
> > >=C2=A0 =C2=A0 =C2=A0same for<br>
> > >=C2=A0 =C2=A0 =C2=A0>> either.<br>
> > >=C2=A0 =C2=A0 =C2=A0>><br>
> > >=C2=A0 =C2=A0 =C2=A0>> On Thu, Jul 20, 2023 at 1:31=E2=
=80=AFPM Thomas Monjalon<br>
> > >=C2=A0 =C2=A0 =C2=A0<<a href=3D"mailto:thomas@monjalon.net=
" target=3D"_blank">thomas@monjalon.net</a> <mailto:<a href=3D"mailto:th=
omas@monjalon.net" target=3D"_blank">thomas@monjalon.net</a>>> wrote:=
<br>
> > >=C2=A0 =C2=A0 =C2=A0>><br>
> > >=C2=A0 =C2=A0 =C2=A0>> > 20/07/2023 11:35, Igor de P=
aula:<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > The weird thing is tha=
t it only happens when I am using a<br>
> > >=C2=A0 =C2=A0 =C2=A0host with an<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > AMD<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > processor. It doesn=
9;t happen when I use a host with an Intel<br>
> > >=C2=A0 =C2=A0 =C2=A0processor.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> > So it's probably a matt=
er of BIOS settings for the IOMMU?<br>
> > >=C2=A0 =C2=A0 =C2=A0>> ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > On Thu, Jul 20, 2023 a=
t 10:32=E2=80=AFAM Thomas Monjalon<br>
> > >=C2=A0 =C2=A0 =C2=A0<<a href=3D"mailto:thomas@monjalon.net=
" target=3D"_blank">thomas@monjalon.net</a> <mailto:<a href=3D"mailto:th=
omas@monjalon.net" target=3D"_blank">thomas@monjalon.net</a>>><br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > wrote:<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > +Cc the vmxnet3 m=
aintainer.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > Please Jochen, do=
you have an idea what's wrong below?<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > 20/07/2023 11:25,=
Igor de Paula:<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > This is beca=
use it can't negotiate the IOMMU type with<br>
> > >=C2=A0 =C2=A0 =C2=A0any port.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > On Thu, Jul =
20, 2023 at 5:08=E2=80=AFAM Thomas Monjalon<br>
> > >=C2=A0 =C2=A0 =C2=A0<<a href=3D"mailto:thomas@monjalon.net=
" target=3D"_blank">thomas@monjalon.net</a> <mailto:<a href=3D"mailto:th=
omas@monjalon.net" target=3D"_blank">thomas@monjalon.net</a>><br>
> > >=C2=A0 =C2=A0 =C2=A0>> > ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > wrote:<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > Hello,<=
br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > The fir=
st error is "Cause: Error: number of ports must<br>
> > >=C2=A0 =C2=A0 =C2=A0be even"<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > 03/05/2=
023 18:13, Igor de Paula:<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > I =
am running a VM inside a VMWARE server (vSphere).<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > My=
goal it to set up DPDK with two HW ports, and set<br>
> > up a<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > virtio_user to<br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > in=
teract with the kernel stack.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > In=
another app I have it working but instead of<br>
> > >=C2=A0 =C2=A0 =C2=A0virtio_user I am<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > running<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > KN=
I, it works in IOVA-PA mode.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > I =
am looking to replace the KNI.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Wh=
en I try to set up virtio_user port as in the doc:<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> ><br>
> > ><br>
> > <a href=3D"https://doc.dpdk.org/guides/howto/virtio_user_as_excep=
tion_path.html#virtio-user-as-exception-path" rel=3D"noreferrer" target=3D"=
_blank">https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.htm=
l#virtio-user-as-exception-path</a><br>
> > <<br>
> > <a href=3D"https://doc.dpdk.org/guides/howto/virtio_user_as_excep=
tion_path.html#virtio-user-as-exception-path" rel=3D"noreferrer" target=3D"=
_blank">https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.htm=
l#virtio-user-as-exception-path</a>><br>
> > <<br>
> > <a href=3D"https://doc.dpdk.org/guides/howto/virtio_user_as_excep=
tion_path.html#virtio-user-as-exception-path" rel=3D"noreferrer" target=3D"=
_blank">https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.htm=
l#virtio-user-as-exception-path</a><br>
> > <<br>
> > <a href=3D"https://doc.dpdk.org/guides/howto/virtio_user_as_excep=
tion_path.html#virtio-user-as-exception-path" rel=3D"noreferrer" target=3D"=
_blank">https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.htm=
l#virtio-user-as-exception-path</a><br>
> > >><br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > I =
get a error it can't run in PA mode.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Wh=
en I try to run as VA mode from a parameter, I get<br>
> > the<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > following<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > errors:=
<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: lib.eal log level changed from info to debug<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Detected lcore 0 as core 0 on socket 0<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Detected lcore 1 as core 0 on socket 0<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Support maximum 128 logical core(s) by<br>
> > >=C2=A0 =C2=A0 =C2=A0configuration.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Detected 2 lcore(s)<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Detected 1 NUMA nodes<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Checking presence of .so 'librte_eal.so.21.3'<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Checking presence of .so 'librte_eal.so.21'<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Checking presence of .so 'librte_eal.so'<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Detected static linkage of DPDK<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Ask a virtual area of 0x7000 bytes<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Virtual area found at 0x100000000 (size =3D 0x7000)<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Multi-process socket /var/run/dpdk/rte/mp_socket<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: DPAA Bus not present. Skipping.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: VFIO PCI modules not loaded<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Selected IOVA mode 'VA'<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Probing VFIO support...<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: IOMMU type 1 (Type 1) is supported<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: IOMMU type 7 (sPAPR) is not supported<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: IOMMU type 8 (No-IOMMU) is supported<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: VFIO support initialized<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Ask a virtual area of 0x5b000 bytes<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Virtual area found at 0x100007000 (size =3D<br>
> > 0x5b000)<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Setting up physically contiguous memory...<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Setting maximum number of open files to 1048576<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Detected memory type: socket_id:0<br>
> > >=C2=A0 =C2=A0 =C2=A0hugepage_sz:1073741824<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Creating 2 segment lists: n_segs:128 socket_id:0<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > hugepag=
e_sz:1073741824<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Ask a virtual area of 0x2000 bytes<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Virtual area found at 0x100062000 (size =3D 0x2000)<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Memseg list allocated at socket 0, page size<br>
> > >=C2=A0 =C2=A0 =C2=A00x100000kB<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Ask a virtual area of 0x2000000000 bytes<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Virtual area found at 0x140000000 (size =3D<br>
> > >=C2=A0 =C2=A0 =C2=A00x2000000000)<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: VA reserved for memseg list at 0x140000000, size<br>
> > >=C2=A0 =C2=A0 =C2=A02000000000<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Ask a virtual area of 0x2000 bytes<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Virtual area found at 0x2140000000 (size =3D<br>
> > 0x2000)<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Memseg list allocated at socket 0, page size<br>
> > >=C2=A0 =C2=A0 =C2=A00x100000kB<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Ask a virtual area of 0x2000000000 bytes<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Virtual area found at 0x2180000000 (size =3D<br>
> > >=C2=A0 =C2=A0 =C2=A00x2000000000)<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: VA reserved for memseg list at 0x2180000000,<br>
> > >=C2=A0 =C2=A0 =C2=A0size 2000000000<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: TSC frequency is ~2350000 KHz<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Main lcore 0 is ready<br>
> > (tid=3D7f8ad790ec00;cpuset=3D[0])<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: lcore 1 is ready (tid=3D7f8ad6907400;cpuset=3D[1])<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Trying to obtain current memory policy.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Setting policy MPOL_PREFERRED for socket 0<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Restoring previous memory policy: 0<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: request: mp_malloc_sync<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Heap on socket 0 was expanded by 1024MB<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: PCI device 0000:0b:00.0 on NUMA socket -1<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L:=C2=A0 =C2=A0probe driver: 15ad:7b0 net_vmxnet3<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L:=C2=A0 =C2=A0Expecting 'PA' IOVA mode but current mode is<br>
> > >=C2=A0 =C2=A0 =C2=A0'VA', not<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > initial=
izing<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Requested device 0000:0b:00.0 cannot be used<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: PCI device 0000:13:00.0 on NUMA socket -1<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L:=C2=A0 =C2=A0probe driver: 15ad:7b0 net_vmxnet3<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L:=C2=A0 =C2=A0Expecting 'PA' IOVA mode but current mode is<br>
> > >=C2=A0 =C2=A0 =C2=A0'VA', not<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > initial=
izing<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Requested device 0000:13:00.0 cannot be used<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Bus (pci) probe failed.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: lib.telemetry log level changed from disabled to<br>
> > >=C2=A0 =C2=A0 =C2=A0warning<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Error - exiting with code: 1<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > >=C2=
=A0 =C2=A0Cause: Error: number of ports must be even<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: request: mp_malloc_sync<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Heap on socket 0 was shrunk by 1024MB<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Fo=
r some reason the HW ports won't setup. From what I<br>
> > >=C2=A0 =C2=A0 =C2=A0understand<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > ne=
t_vmxnet3 should work with VA mode.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > I =
enabled I/OMUU for the VM.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Th=
e weird thing even when enabled, I still have the<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > en=
able_unsafe_noiommu_mode flag on.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > An=
d because it's on the this:<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > de=
v_iova_mode =3D pci_device_iova_mode(dr, dev);<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > re=
turn PA mode, and it fails.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Wh=
en I disable it by modifying<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0/sys/module/vfio/parameters/enable_unsafe=
_noiommu_mode, I get<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > another<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > error.<=
br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Th=
e error is that it doesn't find a suitable IOMMU<br>
> > type:<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Ju=
st putting the relevant message:<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Heap on socket 0 was expanded by 1024MB<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: PCI device 0000:0b:00.0 on NUMA socket -1<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L:=C2=A0 =C2=A0probe driver: 15ad:7b0 net_vmxnet3<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Set IOMMU type 1 (Type 1) failed, error 19 (No<br>
> > >=C2=A0 =C2=A0 =C2=A0such device)<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Set IOMMU type 7 (sPAPR) failed, error 19 (No<br>
> > >=C2=A0 =C2=A0 =C2=A0such device)<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No<br>
> > >=C2=A0 =C2=A0 =C2=A0such<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > device)<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: 0000:0b:00.0 failed to select IOMMU type<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Requested device 0000:0b:00.0 cannot be used<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: PCI device 0000:13:00.0 on NUMA socket -1<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L:=C2=A0 =C2=A0probe driver: 15ad:7b0 net_vmxnet3<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Set IOMMU type 1 (Type 1) failed, error 19 (No<br>
> > >=C2=A0 =C2=A0 =C2=A0such device)<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Set IOMMU type 7 (sPAPR) failed, error 19 (No<br>
> > >=C2=A0 =C2=A0 =C2=A0such device)<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No<br>
> > >=C2=A0 =C2=A0 =C2=A0such<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > device)<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: 0000:13:00.0 failed to select IOMMU type<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA=
L: Requested device 0000:13:00.0 cannot be used<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > I =
can see where it fails:<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > co=
nst struct vfio_iommu_type *<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > vf=
io_set_iommu_type(int vfio_container_fd)<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > {<=
br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > un=
signed idx;<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > fo=
r (idx =3D 0; idx < RTE_DIM(iommu_types); idx++) {<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > co=
nst struct vfio_iommu_type *t =3D &iommu_types[idx];<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > in=
t ret =3D ioctl(vfio_container_fd, VFIO_SET_IOMMU,<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > t-=
>type_id);<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > if=
(!ret) {<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > RT=
E_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > t-=
>type_id, t->name);<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > re=
turn t;<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > }<=
br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > /*=
not an error, there may be more supported IOMMU<br>
> > >=C2=A0 =C2=A0 =C2=A0types */<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > RT=
E_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed,<br>
> > >=C2=A0 =C2=A0 =C2=A0error "<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > &q=
uot;%i (%s)\n", t->type_id, t->name, errno,<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > st=
rerror(errno));<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > }<=
br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > /*=
if we didn't find a suitable IOMMU type, fail */<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > re=
turn NULL;<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > }<=
br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Th=
e ioctl returns -1.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > ><br=
>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > I =
tried many things.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > En=
abling/disabling IOMMU.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Mo=
difying the enable_unsafe_noiommu_mode flag.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Ru=
nning in PA mode.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > An=
y help will be much appreciated.<br>
> > >=C2=A0 =C2=A0 =C2=A0>> ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> ><br>
> > >=C2=A0 =C2=A0 =C2=A0>> ><br>
> > >=C2=A0 =C2=A0 =C2=A0>><br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > >=C2=A0 =C2=A0 =C2=A0> !! External Email: This email origin=
ated from outside of the<br>
> > >=C2=A0 =C2=A0 =C2=A0> organization. Do not click links or =
open attachments unless you<br>
> > >=C2=A0 =C2=A0 =C2=A0> recognize the sender.<br>
> > >=C2=A0 =C2=A0 =C2=A0><br>
> > ><br>
> ><br>
> ><br>
> <br>
<br>
<br>
<br>
<br>
<br>
</blockquote></div>
--0000000000000d36d10602a4fa9c--