From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <users-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id 3621643033
for <public@inbox.dpdk.org>; Fri, 11 Aug 2023 14:30:07 +0200 (CEST)
Received: from mails.dpdk.org (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id A2F8B40F16;
Fri, 11 Aug 2023 14:30:06 +0200 (CEST)
Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com
[66.111.4.29]) by mails.dpdk.org (Postfix) with ESMTP id A3F1940E03
for <users@dpdk.org>; Fri, 11 Aug 2023 14:30:05 +0200 (CEST)
Received: from compute6.internal (compute6.nyi.internal [10.202.2.47])
by mailout.nyi.internal (Postfix) with ESMTP id 16C925C008F;
Fri, 11 Aug 2023 08:30:02 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
by compute6.internal (MEProxy); Fri, 11 Aug 2023 08:30:02 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h=
cc:cc:content-transfer-encoding:content-type:content-type:date
:date:from:from:in-reply-to:in-reply-to:message-id:mime-version
:references:reply-to:sender:subject:subject:to:to; s=fm3; t=
1691757002; x=1691843402; bh=fnmNTg0GLMey25zPWEdp5plHrcr03Rd8+R7
kRzv+U6g=; b=w3bW5ssVi/udD2aTb3K6PV1FbTR1Ad22ZbRyJTNdmkVt3Mt7Cty
zaTwBqRzpyyJiAOBn2/w1at2ka+9q7OHJiZH8jGUNtLSgBr+xSzM+VcvmYLyNFlu
PZR/9W+KqOwj4brNhsjLwcDiyQY8dY2TV3StIWiMlJ7IBPF3YRpJ//iH9mOAWJwq
EvGvApOiL0QbiasvhYCB/4VfgPrgmeAiOyjRVF9EkOdKm55S3TgxjeeRvYtXIRTh
j63Q2mHCZT0PbTenx7eS6RWBIheHMz+koED/8HQ3Iaizp3Ebe8I+3YbWcC+cpL+n
gX1nnc5rXfFpeUPDw6NreOtqnfbDkZtzr7A==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
messagingengine.com; h=cc:cc:content-transfer-encoding
:content-type:content-type:date:date:feedback-id:feedback-id
:from:from:in-reply-to:in-reply-to:message-id:mime-version
:references:reply-to:sender:subject:subject:to:to:x-me-proxy
:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
1691757002; x=1691843402; bh=fnmNTg0GLMey25zPWEdp5plHrcr03Rd8+R7
kRzv+U6g=; b=pnD/LujyaN/P/TSZMfN8Ae8OX0yfIfuo4kRjQXqs4gUKNF3Oy+P
sS27lqdvTjFKIJ4jtOhvfNMoVEpYN7+qzqZGK0he5/55NGgBt0FpRWNTOJmtuS7K
+4WXENJwt1m3G9egZVnsebbZwQ6Gl0J+2V1YTWbsTUiYo/RJaiabd/pzQss7JFSM
KXWkiZBmy0zOprpv4STxFhn1mTGWFHQKThDYwMzHqbZAfla+VhdBAwtdiKP2iwRE
bzENVu1wEPwkjIn4yKlDmtR9xz3keRUrcICj6IqQry9nVmIq1yDLdiZNHzZbh/4v
OaLOqYAny1Y+guX2952f9v247KqZng8NThg==
X-ME-Sender: <xms:ySnWZKKBmAyvNb7C4J9eYsqYcVOh--4-NbUcIQ-I1OQIxo1_9WRS9g>
<xme:ySnWZCLatKYmhchqZfSC6rOLEBoFs9-XqHAHtBt3fBMVLYglj-I9UDwpKg9NyYNSp
6-gisTIhXfdu38iPw>
X-ME-Received: <xmr:ySnWZKvEswFHs7fpzc4rcIY3pY1H4UzHeleNpf32BFlvPcyhlxcHBjV1tVYGdO9del8_k0SRyCPNdwMfsFYuIf1gCg>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrleekgdehudcutefuodetggdotefrodftvf
curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
fjughrpefhvfevufffkfgjfhgggfgtsehtqhertddttdejnecuhfhrohhmpefvhhhomhgr
shcuofhonhhjrghlohhnuceothhhohhmrghssehmohhnjhgrlhhonhdrnhgvtheqnecugg
ftrfgrthhtvghrnhepffdtuefhhedvkeelleevffdvlefhleehvdegtddvvdduueeivedt
gfejvddugeefnecuffhomhgrihhnpeguphgukhdrohhrghenucevlhhushhtvghrufhiii
gvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehthhhomhgrshesmhhonhhjrghlohhn
rdhnvght
X-ME-Proxy: <xmx:ySnWZPbKwxhmkaBWxoCPzmBW_-MVFHVSrJQA8sf1ttWaQdBaBBdSDg>
<xmx:ySnWZBYzwh49KjwLmr1KxNaDPxOI8IaOcUKwdYCN9gjmS7QO311DZA>
<xmx:ySnWZLAgxD990tIu946wYiZvIX9s3H7bGg_Y1BSMiGrDqSwO4R0UDw>
<xmx:yinWZPMtWUtue-SiDOEjlj4y7dbWACsigyLqLIZvKDf-ohYRr3_Ikw>
Feedback-ID: i47234305:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri,
11 Aug 2023 08:29:59 -0400 (EDT)
From: Thomas Monjalon <thomas@monjalon.net>
To: Igor de Paula <igordptx@gmail.com>
Cc: "Varghese, Vipin" <Vipin.Varghese@amd.com>,
"Yigit, Ferruh" <Ferruh.Yigit@amd.com>, Jochen Behrens <jbehrens@vmware.com>,
"users@dpdk.org" <users@dpdk.org>, "Gupta, Nipun" <Nipun.Gupta@amd.com>,
"Agarwal, Nikhil" <nikhil.agarwal@amd.com>, Ronak Doshi <doshir@vmware.com>,
"Immanni, Venkat" <Venkat.Immanni@amd.com>, Chenbo Xia <chenbo.xia@intel.com>
Subject: Re: help
Date: Fri, 11 Aug 2023 14:29:57 +0200
Message-ID: <2290932.ElGaqSPkdT@thomas>
In-Reply-To: <CALDr01QRBvwMeC0atQZq7K0MtYy6bY80X=u95guei=dQjjqFcg@mail.gmail.com>
References: <CALDr01SrSDZXeuVyBC3b0RTvPhN-c7TNJsis1Qq9rMxC+jV3cg@mail.gmail.com>
<MN2PR12MB3085D6D1454F1CD24C9A7B428203A@MN2PR12MB3085.namprd12.prod.outlook.com>
<CALDr01QRBvwMeC0atQZq7K0MtYy6bY80X=u95guei=dQjjqFcg@mail.gmail.com>
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="UTF-8"
X-BeenThere: users@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK usage discussions <users.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/users>,
<mailto:users-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/users/>
List-Post: <mailto:users@dpdk.org>
List-Help: <mailto:users-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/users>,
<mailto:users-request@dpdk.org?subject=subscribe>
Errors-To: users-bounces@dpdk.org
Thanks for the info.
Do you think it should be written in the vmxnet3 page of the DPDK documenta=
tion?
If yes, would you like to initiate a patch for review?
11/08/2023 10:43, Igor de Paula:
> Hi again,
> I got this resolved with VMWARE support so I thought to share it here.
> What I originally wanted was to use IOVA-VA on an AMD host. Which didn't
> work. I have learned that the ESXI version that supports
> virtual IOMMU in AMD hosts (which is a prerequisite to IOVA-VA) is ESXI 7=
=2E0
> U1. After updating it worked. On Intel hosts ESXI 6.7 supports it already
> as far as I know.
>=20
>=20
> On Tue, Jul 25, 2023 at 6:19=E2=80=AFPM Varghese, Vipin <Vipin.Varghese@a=
md.com>
> wrote:
>=20
> > [AMD Official Use Only - General]
> >
> > Like I said earlier, trying with the Intel host I have on VMWARE,
> > specifically Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> > With IOMMU enabled, VMXNET3 works with VA as well as PA.
> >
> > [VV] since ` enable_unsafe_iommu: not enabled` on Intel platform, could
> > it be possible the specific version EXSI hypervisor supports the HW IOM=
MU
> > specific to the platform. My suspicion will be in case on AMD platform
> > changes for required to enable HW iommu might not be available to spec=
ific
> > EXSI (hypervior OS) used.
> >
> >
> >
> > I am not an expert on virtio_user PMD, but I can check if it will work
> > with PA with deferred setting for vmx_net3 PMD are ok?
> >
> >
> >
> > *From:* Igor de Paula <igordptx@gmail.com>
> > *Sent:* Tuesday, July 25, 2023 8:42 PM
> > *To:* Varghese, Vipin <Vipin.Varghese@amd.com>
> > *Cc:* Yigit, Ferruh <Ferruh.Yigit@amd.com>; Jochen Behrens <
> > jbehrens@vmware.com>; Thomas Monjalon <thomas@monjalon.net>;
> > users@dpdk.org; Gupta, Nipun <Nipun.Gupta@amd.com>; Agarwal, Nikhil <
> > nikhil.agarwal@amd.com>; Ronak Doshi <doshir@vmware.com>; Immanni, Venk=
at
> > <Venkat.Immanni@amd.com>; Chenbo Xia <chenbo.xia@intel.com>
> > *Subject:* Re: help
> >
> >
> >
> > *Caution:* This message originated from an External Source. Use proper
> > caution when opening attachments, clicking links, or responding.
> >
> >
> >
> > Well,
> > Like I said earlier, trying with the Intel host I have on VMWARE,
> > specifically Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> > With IOMMU enabled, VMXNET3 works with VA as well as PA.
> > Meaning, PA works regardless if IOMMU is enabled or not. From my
> > experience anyway.
> > That's why I thought that:
> > virtio_user needs VA to work.
> > For some reason VMXNET3 does not work with VA (only on AMD host).
> >
> >
> >
> >
> >
> > On Tue, Jul 25, 2023 at 4:04=E2=80=AFPM Varghese, Vipin <Vipin.Varghese=
@amd.com>
> > wrote:
> >
> > [AMD Official Use Only - General]
> >
> >
> >
> > Thanks Igor,
> >
> >
> >
> > As suspected the vmx_net3 works with
> >
> >
> >
> > 1. Iommu: disabled
> > 2. enable_unsafe_iommu: enabled
> > 3. dpdk eal iova mode: PA
> >
> >
> >
> > as pointed by you in logs, the virtio_user fails as it expects VA too.
> >
> >
> >
> > Will check and get back.
> >
> >
> >
> > *From:* Igor de Paula <igordptx@gmail.com>
> > *Sent:* Tuesday, July 25, 2023 8:16 PM
> > *To:* Yigit, Ferruh <Ferruh.Yigit@amd.com>
> > *Cc:* Jochen Behrens <jbehrens@vmware.com>; Thomas Monjalon <
> > thomas@monjalon.net>; users@dpdk.org; Gupta, Nipun <Nipun.Gupta@amd.com=
>;
> > Agarwal, Nikhil <nikhil.agarwal@amd.com>; Ronak Doshi <doshir@vmware.co=
m>;
> > Immanni, Venkat <Venkat.Immanni@amd.com>; Varghese, Vipin <
> > Vipin.Varghese@amd.com>; Chenbo Xia <chenbo.xia@intel.com>
> > *Subject:* Re: help
> >
> >
> >
> > *Caution:* This message originated from an External Source. Use proper
> > caution when opening attachments, clicking links, or responding.
> >
> >
> >
> > Hi,
> > Attaching the logs of EAL when trying to run a configuration with
> > virtio_user port when IOMMU is
> > disabled and enable_unsafe_iommu is enabled. As you can see it forces I=
OVA
> > as PA but the viritui_user needs IOVA as VA.
> > I am also attaching the output of dmesg. I am not sure which kernel logs
> > you wanted... if there is anything else please let me know..
> > Regarding the ESXI logs, they are HUGE so I will send to you on a separ=
ate
> > email.
> >
> >
> >
> > On Fri, Jul 21, 2023 at 1:14=E2=80=AFPM Ferruh Yigit <ferruh.yigit@amd.=
com> wrote:
> >
> > On 7/21/2023 12:39 PM, Igor de Paula wrote:
> > > I am trying to use virtio_user for an interface with the
> > > kernel:
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html <
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html>
> > > I think this requires IOVA as va.
> > >
> >
> > I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo,
> > he may know better.
> >
> > Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and
> > '--iova-mode=3Dpa'?
> >
> >
> > > It does work with Intel host and IOMMU
> > > enabled. Part of the negotiation when setting it up is getting the IO=
MMU
> > > number so I thought it has to have IOMMU.
> > >
> >
> > Yes, issue looks like related to the IOMMU, and it may be either related
> > to HW support, or ESXi iommu driver support, we will check using below
> > information you provided.
> >
> > > I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but
> > > again, that didn't work.
> > > ESXI version - VMware ESXi, 7.0.0, 16324942
> > > AMD: AMD EPYC 7452 32-Core Processo
> > >
> > > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.6=
0GHz
> > > Regarding the logs I will try and attach it soon.
> > >
> >
> > Thanks for info, waiting for logs.
> >
> > >
> > > On Fri, Jul 21, 2023 at 12:21=E2=80=AFPM Ferruh Yigit <ferruh.yigit@a=
md.com
> > > <mailto:ferruh.yigit@amd.com>> wrote:
> > >
> > > Hi Igor,
> > >
> > > VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can=
you
> > > please try with:
> > > - enable 'enable_unsafe_noiommu_mode' flag
> > > - Force PA mode via '--iova-mode=3Dpa' eal parameter
> > >
> > >
> > > Also to be able to figure out AMD IOMMU support level, can you pl=
ease
> > > provide:
> > > - AMD part number
> > > - ESXi/hypervisor version
> > > - ESXi & VM kernel logs
> > >
> > >
> > > Thanks,
> > > Ferruh
> > >
> > > On 7/20/2023 5:21 PM, Jochen Behrens wrote:
> > > > +Ronak from the ESX team
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > In our usage, we do set amd_iommu=3Doff in the boot command lin=
e from
> > > > grub. (Or intel_iommu=3Doff for Intel processors.)
> > > >
> > > >
> > > >
> > > > Jochen
> > > >
> > > >
> > > >
> > > > *From: *Thomas Monjalon <thomas@monjalon.net
> > > <mailto:thomas@monjalon.net>>
> > > > *Date: *Thursday, July 20, 2023 at 6:00 AM
> > > > *To: *Igor de Paula <igordptx@gmail.com <mailto:igordptx@gmail.=
com
> > >>
> > > > *Cc: *users@dpdk.org <mailto:users@dpdk.org> <users@dpdk.org
> > > <mailto:users@dpdk.org>>, Jochen Behrens
> > > > <jbehrens@vmware.com <mailto:jbehrens@vmware.com>>, Nipun Gupta
> > > <nipun.gupta@amd.com <mailto:nipun.gupta@amd.com>>, Nikhil Agarwal
> > > > <nikhil.agarwal@amd.com <mailto:nikhil.agarwal@amd.com>>, Ferruh
> > > Yigit <ferruh.yigit@amd.com <mailto:ferruh.yigit@amd.com>>
> > > > *Subject: *Re: help
> > > >
> > > > !! External Email
> > > >
> > > > +Cc some AMD maintainers, they can have an idea about IOMMU
> > settings.
> > > >
> > > >
> > > > 20/07/2023 14:44, Igor de Paula:
> > > >> I have enabled it in the host and in the BIOS for AMD...
> > > >> In the Bios I changed to amd_iommu=3Don and in the host it's t=
he
> > > same for
> > > >> either.
> > > >>
> > > >> On Thu, Jul 20, 2023 at 1:31=E2=80=AFPM Thomas Monjalon
> > > <thomas@monjalon.net <mailto:thomas@monjalon.net>> wrote:
> > > >>
> > > >> > 20/07/2023 11:35, Igor de Paula:
> > > >> > > The weird thing is that it only happens when I am using a
> > > host with an
> > > >> > AMD
> > > >> > > processor. It doesn't happen when I use a host with an Int=
el
> > > processor.
> > > >> >
> > > >> > So it's probably a matter of BIOS settings for the IOMMU?
> > > >> >
> > > >> >
> > > >> > > On Thu, Jul 20, 2023 at 10:32=E2=80=AFAM Thomas Monjalon
> > > <thomas@monjalon.net <mailto:thomas@monjalon.net>>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > +Cc the vmxnet3 maintainer.
> > > >> > > >
> > > >> > > > Please Jochen, do you have an idea what's wrong below?
> > > >> > > >
> > > >> > > >
> > > >> > > > 20/07/2023 11:25, Igor de Paula:
> > > >> > > > > This is because it can't negotiate the IOMMU type with
> > > any port.
> > > >> > > > >
> > > >> > > > > On Thu, Jul 20, 2023 at 5:08=E2=80=AFAM Thomas Monjalon
> > > <thomas@monjalon.net <mailto:thomas@monjalon.net>
> > > >> > >
> > > >> > > > wrote:
> > > >> > > > >
> > > >> > > > > > Hello,
> > > >> > > > > >
> > > >> > > > > > The first error is "Cause: Error: number of ports mu=
st
> > > be even"
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > 03/05/2023 18:13, Igor de Paula:
> > > >> > > > > > > I am running a VM inside a VMWARE server (vSphere).
> > > >> > > > > > > My goal it to set up DPDK with two HW ports, and s=
et
> > up a
> > > >> > > > virtio_user to
> > > >> > > > > > > interact with the kernel stack.
> > > >> > > > > > > In another app I have it working but instead of
> > > virtio_user I am
> > > >> > > > running
> > > >> > > > > > > KNI, it works in IOVA-PA mode.
> > > >> > > > > > > I am looking to replace the KNI.
> > > >> > > > > > >
> > > >> > > > > > > When I try to set up virtio_user port as in the do=
c:
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> >
> > >
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#vi=
rtio-user-as-exception-path
> > <
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#vi=
rtio-user-as-exception-path>
> > <
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#vi=
rtio-user-as-exception-path
> > <
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#vi=
rtio-user-as-exception-path
> > >>
> > > >> > > > > > > I get a error it can't run in PA mode.
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > When I try to run as VA mode from a parameter, I g=
et
> > the
> > > >> > following
> > > >> > > > > > errors:
> > > >> > > > > > > EAL: lib.eal log level changed from info to debug
> > > >> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0
> > > >> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0
> > > >> > > > > > > EAL: Support maximum 128 logical core(s) by
> > > configuration.
> > > >> > > > > > > EAL: Detected 2 lcore(s)
> > > >> > > > > > > EAL: Detected 1 NUMA nodes
> > > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3'
> > > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21'
> > > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so'
> > > >> > > > > > > EAL: Detected static linkage of DPDK
> > > >> > > > > > > EAL: Ask a virtual area of 0x7000 bytes
> > > >> > > > > > > EAL: Virtual area found at 0x100000000 (size =3D 0=
x7000)
> > > >> > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_soc=
ket
> > > >> > > > > > > EAL: DPAA Bus not present. Skipping.
> > > >> > > > > > > EAL: VFIO PCI modules not loaded
> > > >> > > > > > > EAL: Selected IOVA mode 'VA'
> > > >> > > > > > > EAL: Probing VFIO support...
> > > >> > > > > > > EAL: IOMMU type 1 (Type 1) is supported
> > > >> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported
> > > >> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported
> > > >> > > > > > > EAL: VFIO support initialized
> > > >> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes
> > > >> > > > > > > EAL: Virtual area found at 0x100007000 (size =3D
> > 0x5b000)
> > > >> > > > > > > EAL: Setting up physically contiguous memory...
> > > >> > > > > > > EAL: Setting maximum number of open files to 10485=
76
> > > >> > > > > > > EAL: Detected memory type: socket_id:0
> > > hugepage_sz:1073741824
> > > >> > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_i=
d:0
> > > >> > > > > > hugepage_sz:1073741824
> > > >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> > > >> > > > > > > EAL: Virtual area found at 0x100062000 (size =3D 0=
x2000)
> > > >> > > > > > > EAL: Memseg list allocated at socket 0, page size
> > > 0x100000kB
> > > >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > >> > > > > > > EAL: Virtual area found at 0x140000000 (size =3D
> > > 0x2000000000)
> > > >> > > > > > > EAL: VA reserved for memseg list at 0x140000000, s=
ize
> > > 2000000000
> > > >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> > > >> > > > > > > EAL: Virtual area found at 0x2140000000 (size =3D
> > 0x2000)
> > > >> > > > > > > EAL: Memseg list allocated at socket 0, page size
> > > 0x100000kB
> > > >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > >> > > > > > > EAL: Virtual area found at 0x2180000000 (size =3D
> > > 0x2000000000)
> > > >> > > > > > > EAL: VA reserved for memseg list at 0x2180000000,
> > > size 2000000000
> > > >> > > > > > > EAL: TSC frequency is ~2350000 KHz
> > > >> > > > > > > EAL: Main lcore 0 is ready
> > (tid=3D7f8ad790ec00;cpuset=3D[0])
> > > >> > > > > > > EAL: lcore 1 is ready (tid=3D7f8ad6907400;cpuset=
=3D[1])
> > > >> > > > > > > EAL: Trying to obtain current memory policy.
> > > >> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0
> > > >> > > > > > > EAL: Restoring previous memory policy: 0
> > > >> > > > > > > EAL: request: mp_malloc_sync
> > > >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> > > >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3
> > > >> > > > > > > EAL: Expecting 'PA' IOVA mode but current mode is
> > > 'VA', not
> > > >> > > > > > initializing
> > > >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> > > >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3
> > > >> > > > > > > EAL: Expecting 'PA' IOVA mode but current mode is
> > > 'VA', not
> > > >> > > > > > initializing
> > > >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> > > >> > > > > > > EAL: Bus (pci) probe failed.
> > > >> > > > > > > EAL: lib.telemetry log level changed from disabled=
to
> > > warning
> > > >> > > > > > > EAL: Error - exiting with code: 1
> > > >> > > > > > > Cause: Error: number of ports must be even
> > > >> > > > > > > EAL: request: mp_malloc_sync
> > > >> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > For some reason the HW ports won't setup. From wha=
t I
> > > understand
> > > >> > > > > > > net_vmxnet3 should work with VA mode.
> > > >> > > > > > > I enabled I/OMUU for the VM.
> > > >> > > > > > > The weird thing even when enabled, I still have the
> > > >> > > > > > > enable_unsafe_noiommu_mode flag on.
> > > >> > > > > > > And because it's on the this:
> > > >> > > > > > >
> > > >> > > > > > > dev_iova_mode =3D pci_device_iova_mode(dr, dev);
> > > >> > > > > > >
> > > >> > > > > > > return PA mode, and it fails.
> > > >> > > > > > >
> > > >> > > > > > > When I disable it by modifying
> > > >> > > > > > >
> > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get
> > > >> > another
> > > >> > > > > > error.
> > > >> > > > > > > The error is that it doesn't find a suitable IOMMU
> > type:
> > > >> > > > > > > Just putting the relevant message:
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> > > >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3
> > > >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No
> > > such device)
> > > >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
> > > such device)
> > > >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 =
(No
> > > such
> > > >> > device)
> > > >> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type
> > > >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> > > >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3
> > > >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No
> > > such device)
> > > >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
> > > such device)
> > > >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 =
(No
> > > such
> > > >> > device)
> > > >> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type
> > > >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > I can see where it fails:
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > const struct vfio_iommu_type *
> > > >> > > > > > > vfio_set_iommu_type(int vfio_container_fd)
> > > >> > > > > > > {
> > > >> > > > > > > unsigned idx;
> > > >> > > > > > > for (idx =3D 0; idx < RTE_DIM(iommu_types); idx++)=
{
> > > >> > > > > > > const struct vfio_iommu_type *t =3D &iommu_types[i=
dx];
> > > >> > > > > > >
> > > >> > > > > > > int ret =3D ioctl(vfio_container_fd, VFIO_SET_IOMM=
U,
> > > >> > > > > > > t->type_id);
> > > >> > > > > > > if (!ret) {
> > > >> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> > > >> > > > > > > t->type_id, t->name);
> > > >> > > > > > > return t;
> > > >> > > > > > > }
> > > >> > > > > > > /* not an error, there may be more supported IOMMU
> > > types */
> > > >> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed,
> > > error "
> > > >> > > > > > > "%i (%s)\n", t->type_id, t->name, errno,
> > > >> > > > > > > strerror(errno));
> > > >> > > > > > > }
> > > >> > > > > > > /* if we didn't find a suitable IOMMU type, fail */
> > > >> > > > > > > return NULL;
> > > >> > > > > > > }
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > The ioctl returns -1.
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > I tried many things.
> > > >> > > > > > > Enabling/disabling IOMMU.
> > > >> > > > > > > Modifying the enable_unsafe_noiommu_mode flag.
> > > >> > > > > > > Running in PA mode.
> > > >> > > > > > > Any help will be much appreciated.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > !! External Email: This email originated from outside of the
> > > > organization. Do not click links or open attachments unless you
> > > > recognize the sender.
> > > >
> > >
> >
> >
>=20