From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7B6CE42ED9 for ; Fri, 21 Jul 2023 13:40:13 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E6D2340DDC; Fri, 21 Jul 2023 13:40:12 +0200 (CEST) Received: from mail-yb1-f169.google.com (mail-yb1-f169.google.com [209.85.219.169]) by mails.dpdk.org (Postfix) with ESMTP id 5150D40DD8 for ; Fri, 21 Jul 2023 13:40:11 +0200 (CEST) Received: by mail-yb1-f169.google.com with SMTP id 3f1490d57ef6-c2cf4e61bc6so1738037276.3 for ; Fri, 21 Jul 2023 04:40:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689939610; x=1690544410; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=kQGjBO0LujblVY50y8qzYnH4P/OhuDKeHlXJqLahRUc=; b=MX89dywkKDNDkF2A1b9/thHuxNTYR/hGxBrd6jdevlffNWhr3PybjPf0SDwscd8TcS I+Q9MSC1VvcmuAMaEwcx02Bf2H/Se2k4llOM7NPGFIoM/aVuD9CoxH4PiC8llC04s3pj 3Nyw9dvcZ1ymjxOqcphk/2YPAE+yluXJa3EA8HEfBi5lwD0/Fyf4AMxAtFG/ma0/inBE tK19oACRzAS3ZIZr74lZf4F6n1vL3QlMJiJMh59iSlCpa0rBN7YqFqCb9rlaxZ6Rbacl B28fKxyi5oerYfz15cizVA4qQ+pACi6wzscDLkFYoV8x4x6BRot4A4CbUzProS8X4mln hvYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689939610; x=1690544410; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=kQGjBO0LujblVY50y8qzYnH4P/OhuDKeHlXJqLahRUc=; b=IUnyLzV5q9GEXn3X5WCms+mkqzrHL8R+c3AjMSp800HSCNEvP+NXOLJBT1t2Cej1Gh npO0j7gDFzd7Xmu9czb5wgWBMBvlFMB2lIpBU/Xw1ZvsKXZWgi6QEO9vvEmCQJ4rFG8B oAfShuBd9JK5SkwJx/5B3turUwqrZ189eL74J3jsRxfdzHaEduXR8Afxd1QY2mRDVcEH BlvZig3yMZLYzS7m4wk/ca4l71DhHhtEcOL9ARyS1tlsTs2MW/IpEZNKqROJKUwH3dsY LvOHlhDt6Zu6BxImaeYv/Py083dLTNBEKitpdWu8dURoM5q4EtUoltqW3/D15RgdWwaX Prew== X-Gm-Message-State: ABy/qLZR/7eIHw33Kaz6INCOj1TO/XEaxcNbjEP2DljNREFJuZul3Igi Q6RcxuKStJ+2UwpbzVMp6HU8MoWxCfZ+RU+aPisgCLltuHHkfJ13 X-Google-Smtp-Source: APBJJlFufNO2V3vq+sAqUCmjCg3nlyuxuGFGrS2MlNfi4F1QP0cCOzXwklQaZtP4sFQ96DQ4QYYECw/w4IAgRT/W8is= X-Received: by 2002:a25:abf0:0:b0:d00:afc8:bdf4 with SMTP id v103-20020a25abf0000000b00d00afc8bdf4mr1547181ybi.62.1689939610444; Fri, 21 Jul 2023 04:40:10 -0700 (PDT) MIME-Version: 1.0 References: <2663502.2IRrRt1zHL@thomas> <7070795.j6PcuT4dK6@thomas> <32a6b0b0-c76d-b535-e727-e62e796ca429@amd.com> In-Reply-To: <32a6b0b0-c76d-b535-e727-e62e796ca429@amd.com> From: Igor de Paula Date: Fri, 21 Jul 2023 12:39:59 +0100 Message-ID: Subject: Re: help To: Ferruh Yigit Cc: Jochen Behrens , Thomas Monjalon , "users@dpdk.org" , Nipun Gupta , Nikhil Agarwal , Ronak Doshi , venkat.immanni@amd.com, "Varghese, Vipin" Content-Type: multipart/alternative; boundary="0000000000002c29250600fdba46" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --0000000000002c29250600fdba46 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I am trying to use virtio_user for an interface with the kernel: https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html I think this requires IOVA as va. It does work with Intel host and IOMMU enabled. Part of the negotiation when setting it up is getting the IOMMU number so I thought it has to have IOMMU. I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but again, that didn't work. ESXI version - VMware ESXi, 7.0.0, 16324942 AMD: AMD EPYC 7452 32-Core Processo On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz Regarding the logs I will try and attach it soon. Thanks On Fri, Jul 21, 2023 at 12:21=E2=80=AFPM Ferruh Yigit wrote: > Hi Igor, > > VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you > please try with: > - enable 'enable_unsafe_noiommu_mode' flag > - Force PA mode via '--iova-mode=3Dpa' eal parameter > > > Also to be able to figure out AMD IOMMU support level, can you please > provide: > - AMD part number > - ESXi/hypervisor version > - ESXi & VM kernel logs > > > Thanks, > Ferruh > > On 7/20/2023 5:21 PM, Jochen Behrens wrote: > > +Ronak from the ESX team > > > > > > > > > > > > In our usage, we do set amd_iommu=3Doff in the boot command line from > > grub. (Or intel_iommu=3Doff for Intel processors.) > > > > > > > > Jochen > > > > > > > > *From: *Thomas Monjalon > > *Date: *Thursday, July 20, 2023 at 6:00 AM > > *To: *Igor de Paula > > *Cc: *users@dpdk.org , Jochen Behrens > > , Nipun Gupta , Nikhil Agarwa= l > > , Ferruh Yigit > > *Subject: *Re: help > > > > !! External Email > > > > +Cc some AMD maintainers, they can have an idea about IOMMU settings. > > > > > > 20/07/2023 14:44, Igor de Paula: > >> I have enabled it in the host and in the BIOS for AMD... > >> In the Bios I changed to amd_iommu=3Don and in the host it's the same = for > >> either. > >> > >> On Thu, Jul 20, 2023 at 1:31=E2=80=AFPM Thomas Monjalon > wrote: > >> > >> > 20/07/2023 11:35, Igor de Paula: > >> > > The weird thing is that it only happens when I am using a host wit= h > an > >> > AMD > >> > > processor. It doesn't happen when I use a host with an Intel > processor. > >> > > >> > So it's probably a matter of BIOS settings for the IOMMU? > >> > > >> > > >> > > On Thu, Jul 20, 2023 at 10:32=E2=80=AFAM Thomas Monjalon < > thomas@monjalon.net> > >> > > wrote: > >> > > > >> > > > +Cc the vmxnet3 maintainer. > >> > > > > >> > > > Please Jochen, do you have an idea what's wrong below? > >> > > > > >> > > > > >> > > > 20/07/2023 11:25, Igor de Paula: > >> > > > > This is because it can't negotiate the IOMMU type with any por= t. > >> > > > > > >> > > > > On Thu, Jul 20, 2023 at 5:08=E2=80=AFAM Thomas Monjalon < > thomas@monjalon.net > >> > > > >> > > > wrote: > >> > > > > > >> > > > > > Hello, > >> > > > > > > >> > > > > > The first error is "Cause: Error: number of ports must be > even" > >> > > > > > > >> > > > > > > >> > > > > > 03/05/2023 18:13, Igor de Paula: > >> > > > > > > I am running a VM inside a VMWARE server (vSphere). > >> > > > > > > My goal it to set up DPDK with two HW ports, and set up a > >> > > > virtio_user to > >> > > > > > > interact with the kernel stack. > >> > > > > > > In another app I have it working but instead of virtio_use= r > I am > >> > > > running > >> > > > > > > KNI, it works in IOVA-PA mode. > >> > > > > > > I am looking to replace the KNI. > >> > > > > > > > >> > > > > > > When I try to set up virtio_user port as in the doc: > >> > > > > > > > >> > > > > > > >> > > > > >> > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virt= io-user-as-exception-path > < > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virt= io-user-as-exception-path > > > >> > > > > > > I get a error it can't run in PA mode. > >> > > > > > > > >> > > > > > > > >> > > > > > > When I try to run as VA mode from a parameter, I get the > >> > following > >> > > > > > errors: > >> > > > > > > EAL: lib.eal log level changed from info to debug > >> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0 > >> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0 > >> > > > > > > EAL: Support maximum 128 logical core(s) by configuration. > >> > > > > > > EAL: Detected 2 lcore(s) > >> > > > > > > EAL: Detected 1 NUMA nodes > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3' > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21' > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so' > >> > > > > > > EAL: Detected static linkage of DPDK > >> > > > > > > EAL: Ask a virtual area of 0x7000 bytes > >> > > > > > > EAL: Virtual area found at 0x100000000 (size =3D 0x7000) > >> > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > >> > > > > > > EAL: DPAA Bus not present. Skipping. > >> > > > > > > EAL: VFIO PCI modules not loaded > >> > > > > > > EAL: Selected IOVA mode 'VA' > >> > > > > > > EAL: Probing VFIO support... > >> > > > > > > EAL: IOMMU type 1 (Type 1) is supported > >> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported > >> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported > >> > > > > > > EAL: VFIO support initialized > >> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes > >> > > > > > > EAL: Virtual area found at 0x100007000 (size =3D 0x5b000) > >> > > > > > > EAL: Setting up physically contiguous memory... > >> > > > > > > EAL: Setting maximum number of open files to 1048576 > >> > > > > > > EAL: Detected memory type: socket_id:0 > hugepage_sz:1073741824 > >> > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0 > >> > > > > > hugepage_sz:1073741824 > >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes > >> > > > > > > EAL: Virtual area found at 0x100062000 (size =3D 0x2000) > >> > > > > > > EAL: Memseg list allocated at socket 0, page size 0x100000= kB > >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes > >> > > > > > > EAL: Virtual area found at 0x140000000 (size =3D 0x2000000= 000) > >> > > > > > > EAL: VA reserved for memseg list at 0x140000000, size > 2000000000 > >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes > >> > > > > > > EAL: Virtual area found at 0x2140000000 (size =3D 0x2000) > >> > > > > > > EAL: Memseg list allocated at socket 0, page size 0x100000= kB > >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes > >> > > > > > > EAL: Virtual area found at 0x2180000000 (size =3D > 0x2000000000) > >> > > > > > > EAL: VA reserved for memseg list at 0x2180000000, size > 2000000000 > >> > > > > > > EAL: TSC frequency is ~2350000 KHz > >> > > > > > > EAL: Main lcore 0 is ready (tid=3D7f8ad790ec00;cpuset=3D[0= ]) > >> > > > > > > EAL: lcore 1 is ready (tid=3D7f8ad6907400;cpuset=3D[1]) > >> > > > > > > EAL: Trying to obtain current memory policy. > >> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0 > >> > > > > > > EAL: Restoring previous memory policy: 0 > >> > > > > > > EAL: request: mp_malloc_sync > >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB > >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > >> > > > > > > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', > not > >> > > > > > initializing > >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used > >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1 > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > >> > > > > > > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', > not > >> > > > > > initializing > >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used > >> > > > > > > EAL: Bus (pci) probe failed. > >> > > > > > > EAL: lib.telemetry log level changed from disabled to > warning > >> > > > > > > EAL: Error - exiting with code: 1 > >> > > > > > > Cause: Error: number of ports must be even > >> > > > > > > EAL: request: mp_malloc_sync > >> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > For some reason the HW ports won't setup. From what I > understand > >> > > > > > > net_vmxnet3 should work with VA mode. > >> > > > > > > I enabled I/OMUU for the VM. > >> > > > > > > The weird thing even when enabled, I still have the > >> > > > > > > enable_unsafe_noiommu_mode flag on. > >> > > > > > > And because it's on the this: > >> > > > > > > > >> > > > > > > dev_iova_mode =3D pci_device_iova_mode(dr, dev); > >> > > > > > > > >> > > > > > > return PA mode, and it fails. > >> > > > > > > > >> > > > > > > When I disable it by modifying > >> > > > > > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I > get > >> > another > >> > > > > > error. > >> > > > > > > The error is that it doesn't find a suitable IOMMU type: > >> > > > > > > Just putting the relevant message: > >> > > > > > > > >> > > > > > > > >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB > >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such > device) > >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such > device) > >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such > >> > device) > >> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type > >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used > >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1 > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such > device) > >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such > device) > >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such > >> > device) > >> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type > >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used > >> > > > > > > > >> > > > > > > > >> > > > > > > I can see where it fails: > >> > > > > > > > >> > > > > > > > >> > > > > > > const struct vfio_iommu_type * > >> > > > > > > vfio_set_iommu_type(int vfio_container_fd) > >> > > > > > > { > >> > > > > > > unsigned idx; > >> > > > > > > for (idx =3D 0; idx < RTE_DIM(iommu_types); idx++) { > >> > > > > > > const struct vfio_iommu_type *t =3D &iommu_types[idx]; > >> > > > > > > > >> > > > > > > int ret =3D ioctl(vfio_container_fd, VFIO_SET_IOMMU, > >> > > > > > > t->type_id); > >> > > > > > > if (!ret) { > >> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n", > >> > > > > > > t->type_id, t->name); > >> > > > > > > return t; > >> > > > > > > } > >> > > > > > > /* not an error, there may be more supported IOMMU types *= / > >> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error = " > >> > > > > > > "%i (%s)\n", t->type_id, t->name, errno, > >> > > > > > > strerror(errno)); > >> > > > > > > } > >> > > > > > > /* if we didn't find a suitable IOMMU type, fail */ > >> > > > > > > return NULL; > >> > > > > > > } > >> > > > > > > > >> > > > > > > > >> > > > > > > The ioctl returns -1. > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > I tried many things. > >> > > > > > > Enabling/disabling IOMMU. > >> > > > > > > Modifying the enable_unsafe_noiommu_mode flag. > >> > > > > > > Running in PA mode. > >> > > > > > > Any help will be much appreciated. > >> > > >> > > >> > > >> > > >> > > >> > > >> > > > > > > > > > > > > > > !! External Email: This email originated from outside of the > > organization. Do not click links or open attachments unless you > > recognize the sender. > > > > --0000000000002c29250600fdba46 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I am trying to use virtio_user for an interface with the k= ernel:=C2=A0https://doc.dpdk.org/guides/howto/virtio_user_as_exceptio= n_path.html
I think this requires IOVA as va. It does work with Inte= l host and IOMMU enabled. Part of the negotiation when setting it up is get= ting the IOMMU number so I thought it has to have IOMMU.
I tried disabli= ng IOMMU and enabling enable_unsafe_noiommu flag but again, that didn't= work.
ESXI version - VMware ESXi, 7.0.0, 1632= 4942
AMD:=C2=A0 AMD EPYC 7452 32-Core Processo

On an Intel= host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
Reg= arding the logs I will try and attach it soon.
Thanks

On Fri, Jul 21, 2023 at 12:21=E2=80=AFPM Ferr= uh Yigit <ferruh.yigit@amd.com> wrote:
Hi= Igor,

VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you please try with:
- enable 'enable_unsafe_noiommu_mode' flag
- Force PA mode via '--iova-mode=3Dpa' eal parameter


Also to be able to figure out AMD IOMMU support level, can you please
provide:
- AMD part number
- ESXi/hypervisor version
- ESXi & VM kernel logs


Thanks,
Ferruh

On 7/20/2023 5:21 PM, Jochen Behrens wrote:
> +Ronak from the ESX team
>
> =C2=A0
>
> =C2=A0
>
> In our usage, we do set amd_iommu=3Doff in the boot command line from<= br> > grub. (Or intel_iommu=3Doff for Intel processors.)
>
> =C2=A0
>
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 Jochen
>
> =C2=A0
>
> *From: *Thomas Monjalon <thomas@monjalon.net>
> *Date: *Thursday, July 20, 2023 at 6:00 AM
> *To: *Igor de Paula <igordptx@gmail.com>
> *Cc: *users@dpdk.o= rg <users@dpdk.o= rg>, Jochen Behrens
> <jbehrens@= vmware.com>, Nipun Gupta <nipun.gupta@amd.com>, Nikhil Agarwal
> <nikhil= .agarwal@amd.com>, Ferruh Yigit <ferruh.yigit@amd.com>
> *Subject: *Re: help
>
> !! External Email
>
> +Cc some AMD maintainers, they can have an idea about IOMMU settings.<= br> >
>
> 20/07/2023 14:44, Igor de Paula:
>> I have enabled it in the host and in the BIOS for AMD...
>> In the Bios I changed to amd_iommu=3Don and in the host it's t= he same for
>> either.
>>
>> On Thu, Jul 20, 2023 at 1:31=E2=80=AFPM Thomas Monjalon <thomas@monjalon.net&= gt; wrote:
>>
>> > 20/07/2023 11:35, Igor de Paula:
>> > > The weird thing is that it only happens when I am using = a host with an
>> > AMD
>> > > processor. It doesn't happen when I use a host with = an Intel processor.
>> >
>> > So it's probably a matter of BIOS settings for the IOMMU?=
>> >
>> >
>> > > On Thu, Jul 20, 2023 at 10:32=E2=80=AFAM Thomas Monjalon= <thomas@monjal= on.net>
>> > > wrote:
>> > >
>> > > > +Cc the vmxnet3 maintainer.
>> > > >
>> > > > Please Jochen, do you have an idea what's wrong= below?
>> > > >
>> > > >
>> > > > 20/07/2023 11:25, Igor de Paula:
>> > > > > This is because it can't negotiate the IOM= MU type with any port.
>> > > > >
>> > > > > On Thu, Jul 20, 2023 at 5:08=E2=80=AFAM Thomas= Monjalon <thom= as@monjalon.net
>> > >
>> > > > wrote:
>> > > > >
>> > > > > > Hello,
>> > > > > >
>> > > > > > The first error is "Cause: Error: nu= mber of ports must be even"
>> > > > > >
>> > > > > >
>> > > > > > 03/05/2023 18:13, Igor de Paula:
>> > > > > > > I am running a VM inside a VMWARE se= rver (vSphere).
>> > > > > > > My goal it to set up DPDK with two H= W ports, and set up a
>> > > > virtio_user to
>> > > > > > > interact with the kernel stack.
>> > > > > > > In another app I have it working but= instead of virtio_user I am
>> > > > running
>> > > > > > > KNI, it works in IOVA-PA mode.
>> > > > > > > I am looking to replace the KNI.
>> > > > > > >
>> > > > > > > When I try to set up virtio_user por= t as in the doc:
>> > > > > > >
>> > > > > >
>> > > >
>> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path= .html#virtio-user-as-exception-path <https://doc.dpdk.org/guides/howto= /virtio_user_as_exception_path.html#virtio-user-as-exception-path> >> > > > > > > I get a error it can't run in PA= mode.
>> > > > > > >
>> > > > > > >
>> > > > > > > When I try to run as VA mode from a = parameter, I get the
>> > following
>> > > > > > errors:
>> > > > > > > EAL: lib.eal log level changed from = info to debug
>> > > > > > > EAL: Detected lcore 0 as core 0 on s= ocket 0
>> > > > > > > EAL: Detected lcore 1 as core 0 on s= ocket 0
>> > > > > > > EAL: Support maximum 128 logical cor= e(s) by configuration.
>> > > > > > > EAL: Detected 2 lcore(s)
>> > > > > > > EAL: Detected 1 NUMA nodes
>> > > > > > > EAL: Checking presence of .so 'l= ibrte_eal.so.21.3'
>> > > > > > > EAL: Checking presence of .so 'l= ibrte_eal.so.21'
>> > > > > > > EAL: Checking presence of .so 'l= ibrte_eal.so'
>> > > > > > > EAL: Detected static linkage of DPDK=
>> > > > > > > EAL: Ask a virtual area of 0x7000 by= tes
>> > > > > > > EAL: Virtual area found at 0x1000000= 00 (size =3D 0x7000)
>> > > > > > > EAL: Multi-process socket /var/run/d= pdk/rte/mp_socket
>> > > > > > > EAL: DPAA Bus not present. Skipping.=
>> > > > > > > EAL: VFIO PCI modules not loaded
>> > > > > > > EAL: Selected IOVA mode 'VA'=
>> > > > > > > EAL: Probing VFIO support...
>> > > > > > > EAL: IOMMU type 1 (Type 1) is suppor= ted
>> > > > > > > EAL: IOMMU type 7 (sPAPR) is not sup= ported
>> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supp= orted
>> > > > > > > EAL: VFIO support initialized
>> > > > > > > EAL: Ask a virtual area of 0x5b000 b= ytes
>> > > > > > > EAL: Virtual area found at 0x1000070= 00 (size =3D 0x5b000)
>> > > > > > > EAL: Setting up physically contiguou= s memory...
>> > > > > > > EAL: Setting maximum number of open = files to 1048576
>> > > > > > > EAL: Detected memory type: socket_id= :0 hugepage_sz:1073741824
>> > > > > > > EAL: Creating 2 segment lists: n_seg= s:128 socket_id:0
>> > > > > > hugepage_sz:1073741824
>> > > > > > > EAL: Ask a virtual area of 0x2000 by= tes
>> > > > > > > EAL: Virtual area found at 0x1000620= 00 (size =3D 0x2000)
>> > > > > > > EAL: Memseg list allocated at socket= 0, page size 0x100000kB
>> > > > > > > EAL: Ask a virtual area of 0x2000000= 000 bytes
>> > > > > > > EAL: Virtual area found at 0x1400000= 00 (size =3D 0x2000000000)
>> > > > > > > EAL: VA reserved for memseg list at = 0x140000000, size 2000000000
>> > > > > > > EAL: Ask a virtual area of 0x2000 by= tes
>> > > > > > > EAL: Virtual area found at 0x2140000= 000 (size =3D 0x2000)
>> > > > > > > EAL: Memseg list allocated at socket= 0, page size 0x100000kB
>> > > > > > > EAL: Ask a virtual area of 0x2000000= 000 bytes
>> > > > > > > EAL: Virtual area found at 0x2180000= 000 (size =3D 0x2000000000)
>> > > > > > > EAL: VA reserved for memseg list at = 0x2180000000, size 2000000000
>> > > > > > > EAL: TSC frequency is ~2350000 KHz >> > > > > > > EAL: Main lcore 0 is ready (tid=3D7f= 8ad790ec00;cpuset=3D[0])
>> > > > > > > EAL: lcore 1 is ready (tid=3D7f8ad69= 07400;cpuset=3D[1])
>> > > > > > > EAL: Trying to obtain current memory= policy.
>> > > > > > > EAL: Setting policy MPOL_PREFERRED f= or socket 0
>> > > > > > > EAL: Restoring previous memory polic= y: 0
>> > > > > > > EAL: request: mp_malloc_sync
>> > > > > > > EAL: Heap on socket 0 was expanded b= y 1024MB
>> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA= socket -1
>> > > > > > > EAL:=C2=A0=C2=A0 probe driver: 15ad:= 7b0 net_vmxnet3
>> > > > > > > EAL:=C2=A0=C2=A0 Expecting 'PA&#= 39; IOVA mode but current mode is 'VA', not
>> > > > > > initializing
>> > > > > > > EAL: Requested device 0000:0b:00.0 c= annot be used
>> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA= socket -1
>> > > > > > > EAL:=C2=A0=C2=A0 probe driver: 15ad:= 7b0 net_vmxnet3
>> > > > > > > EAL:=C2=A0=C2=A0 Expecting 'PA&#= 39; IOVA mode but current mode is 'VA', not
>> > > > > > initializing
>> > > > > > > EAL: Requested device 0000:13:00.0 c= annot be used
>> > > > > > > EAL: Bus (pci) probe failed.
>> > > > > > > EAL: lib.telemetry log level changed= from disabled to warning
>> > > > > > > EAL: Error - exiting with code: 1 >> > > > > > >=C2=A0=C2=A0 Cause: Error: number of = ports must be even
>> > > > > > > EAL: request: mp_malloc_sync
>> > > > > > > EAL: Heap on socket 0 was shrunk by = 1024MB
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > For some reason the HW ports won'= ;t setup. From what I understand
>> > > > > > > net_vmxnet3 should work with VA mode= .
>> > > > > > > I enabled I/OMUU for the VM.
>> > > > > > > The weird thing even when enabled, I= still have the
>> > > > > > > enable_unsafe_noiommu_mode flag on.<= br> >> > > > > > > And because it's on the this: >> > > > > > >
>> > > > > > > dev_iova_mode =3D pci_device_iova_mo= de(dr, dev);
>> > > > > > >
>> > > > > > > return PA mode, and it fails.
>> > > > > > >
>> > > > > > > When I disable it by modifying
>> > > > > > > /sys/module/vfio/parameters/enable_u= nsafe_noiommu_mode, I get
>> > another
>> > > > > > error.
>> > > > > > > The error is that it doesn't fin= d a suitable IOMMU type:
>> > > > > > > Just putting the relevant message: >> > > > > > >
>> > > > > > >
>> > > > > > > EAL: Heap on socket 0 was expanded b= y 1024MB
>> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA= socket -1
>> > > > > > > EAL:=C2=A0=C2=A0 probe driver: 15ad:= 7b0 net_vmxnet3
>> > > > > > > EAL: Set IOMMU type 1 (Type 1) faile= d, error 19 (No such device)
>> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed= , error 19 (No such device)
>> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) fai= led, error 19 (No such
>> > device)
>> > > > > > > EAL: 0000:0b:00.0 failed to select I= OMMU type
>> > > > > > > EAL: Requested device 0000:0b:00.0 c= annot be used
>> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA= socket -1
>> > > > > > > EAL:=C2=A0=C2=A0 probe driver: 15ad:= 7b0 net_vmxnet3
>> > > > > > > EAL: Set IOMMU type 1 (Type 1) faile= d, error 19 (No such device)
>> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed= , error 19 (No such device)
>> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) fai= led, error 19 (No such
>> > device)
>> > > > > > > EAL: 0000:13:00.0 failed to select I= OMMU type
>> > > > > > > EAL: Requested device 0000:13:00.0 c= annot be used
>> > > > > > >
>> > > > > > >
>> > > > > > > I can see where it fails:
>> > > > > > >
>> > > > > > >
>> > > > > > > const struct vfio_iommu_type *
>> > > > > > > vfio_set_iommu_type(int vfio_contain= er_fd)
>> > > > > > > {
>> > > > > > > unsigned idx;
>> > > > > > > for (idx =3D 0; idx < RTE_DIM(iom= mu_types); idx++) {
>> > > > > > > const struct vfio_iommu_type *t =3D = &iommu_types[idx];
>> > > > > > >
>> > > > > > > int ret =3D ioctl(vfio_container_fd,= VFIO_SET_IOMMU,
>> > > > > > > t->type_id);
>> > > > > > > if (!ret) {
>> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU= type %d (%s)\n",
>> > > > > > > t->type_id, t->name);
>> > > > > > > return t;
>> > > > > > > }
>> > > > > > > /* not an error, there may be more s= upported IOMMU types */
>> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU = type %d (%s) failed, error "
>> > > > > > > "%i (%s)\n", t->type_id= , t->name, errno,
>> > > > > > > strerror(errno));
>> > > > > > > }
>> > > > > > > /* if we didn't find a suitable = IOMMU type, fail */
>> > > > > > > return NULL;
>> > > > > > > }
>> > > > > > >
>> > > > > > >
>> > > > > > > The ioctl returns -1.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > I tried many things.
>> > > > > > > Enabling/disabling IOMMU.
>> > > > > > > Modifying the enable_unsafe_noiommu_= mode flag.
>> > > > > > > Running in PA mode.
>> > > > > > > Any help will be much appreciated. >> >
>> >
>> >
>> >
>> >
>> >
>>
>
>
>
>
>
>
> !! External Email: This email originated from outside of the
> organization. Do not click links or open attachments unless you
> recognize the sender.
>

--0000000000002c29250600fdba46--