Re: help

[Igor de Paula <igordptx@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 17103 bytes --]

Well,
Like I said earlier, trying with the Intel host I have on VMWARE,
specifically  Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
With IOMMU enabled, VMXNET3 works with VA as well as PA.
Meaning, PA works regardless if IOMMU is enabled or not. From my experience
anyway.
That's why I thought that:
virtio_user needs VA to work.
For some reason VMXNET3 does not work with VA (only on AMD host).


On Tue, Jul 25, 2023 at 4:04 PM Varghese, Vipin <Vipin.Varghese@amd.com>
wrote:

> [AMD Official Use Only - General]
>
> Thanks Igor,
>
>
>
> As suspected the vmx_net3 works with
>
>
>
>    1. Iommu: disabled
>    2. enable_unsafe_iommu: enabled
>    3. dpdk eal iova mode: PA
>
>
>
> as pointed by you in logs, the virtio_user fails as it expects VA too.
>
>
>
> Will check and get back.
>
>
>
> *From:* Igor de Paula <igordptx@gmail.com>
> *Sent:* Tuesday, July 25, 2023 8:16 PM
> *To:* Yigit, Ferruh <Ferruh.Yigit@amd.com>
> *Cc:* Jochen Behrens <jbehrens@vmware.com>; Thomas Monjalon <
> thomas@monjalon.net>; users@dpdk.org; Gupta, Nipun <Nipun.Gupta@amd.com>;
> Agarwal, Nikhil <nikhil.agarwal@amd.com>; Ronak Doshi <doshir@vmware.com>;
> Immanni, Venkat <Venkat.Immanni@amd.com>; Varghese, Vipin <
> Vipin.Varghese@amd.com>; Chenbo Xia <chenbo.xia@intel.com>
> *Subject:* Re: help
>
>
>
> *Caution:* This message originated from an External Source. Use proper
> caution when opening attachments, clicking links, or responding.
>
>
>
> Hi,
> Attaching the logs of EAL when trying to run a configuration with
> virtio_user port when IOMMU is
> disabled and enable_unsafe_iommu is enabled. As you can see it forces IOVA
> as PA but the viritui_user needs IOVA as VA.
> I am also attaching the output of dmesg. I am not sure which kernel logs
> you wanted... if there is anything else please let me know..
> Regarding the ESXI logs, they are HUGE so I will send to you on a separate
> email.
>
>
>
> On Fri, Jul 21, 2023 at 1:14 PM Ferruh Yigit <ferruh.yigit@amd.com> wrote:
>
> On 7/21/2023 12:39 PM, Igor de Paula wrote:
> > I am trying to use virtio_user for an interface with the
> > kernel:
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html <
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html>
> > I think this requires IOVA as va.
> >
>
> I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo,
> he may know better.
>
> Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and
> '--iova-mode=pa'?
>
>
> > It does work with Intel host and IOMMU
> > enabled. Part of the negotiation when setting it up is getting the IOMMU
> > number so I thought it has to have IOMMU.
> >
>
> Yes, issue looks like related to the IOMMU, and it may be either related
> to HW support, or ESXi iommu driver support, we will check using below
> information you provided.
>
> > I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but
> > again, that didn't work.
> > ESXI version - VMware ESXi, 7.0.0, 16324942
> > AMD:  AMD EPYC 7452 32-Core Processo
> >
> > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> > Regarding the logs I will try and attach it soon.
> >
>
> Thanks for info, waiting for logs.
>
> >
> > On Fri, Jul 21, 2023 at 12:21 PM Ferruh Yigit <ferruh.yigit@amd.com
> > <mailto:ferruh.yigit@amd.com>> wrote:
> >
> >     Hi Igor,
> >
> >     VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you
> >     please try with:
> >     - enable 'enable_unsafe_noiommu_mode' flag
> >     - Force PA mode via '--iova-mode=pa' eal parameter
> >
> >
> >     Also to be able to figure out AMD IOMMU support level, can you please
> >     provide:
> >     - AMD part number
> >     - ESXi/hypervisor version
> >     - ESXi & VM kernel logs
> >
> >
> >     Thanks,
> >     Ferruh
> >
> >     On 7/20/2023 5:21 PM, Jochen Behrens wrote:
> >     > +Ronak from the ESX team
> >     >
> >     >
> >     >
> >     >
> >     >
> >     > In our usage, we do set amd_iommu=off in the boot command line from
> >     > grub. (Or intel_iommu=off for Intel processors.)
> >     >
> >     >
> >     >
> >     >                 Jochen
> >     >
> >     >
> >     >
> >     > *From: *Thomas Monjalon <thomas@monjalon.net
> >     <mailto:thomas@monjalon.net>>
> >     > *Date: *Thursday, July 20, 2023 at 6:00 AM
> >     > *To: *Igor de Paula <igordptx@gmail.com <mailto:igordptx@gmail.com
> >>
> >     > *Cc: *users@dpdk.org <mailto:users@dpdk.org> <users@dpdk.org
> >     <mailto:users@dpdk.org>>, Jochen Behrens
> >     > <jbehrens@vmware.com <mailto:jbehrens@vmware.com>>, Nipun Gupta
> >     <nipun.gupta@amd.com <mailto:nipun.gupta@amd.com>>, Nikhil Agarwal
> >     > <nikhil.agarwal@amd.com <mailto:nikhil.agarwal@amd.com>>, Ferruh
> >     Yigit <ferruh.yigit@amd.com <mailto:ferruh.yigit@amd.com>>
> >     > *Subject: *Re: help
> >     >
> >     > !! External Email
> >     >
> >     > +Cc some AMD maintainers, they can have an idea about IOMMU
> settings.
> >     >
> >     >
> >     > 20/07/2023 14:44, Igor de Paula:
> >     >> I have enabled it in the host and in the BIOS for AMD...
> >     >> In the Bios I changed to amd_iommu=on and in the host it's the
> >     same for
> >     >> either.
> >     >>
> >     >> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon
> >     <thomas@monjalon.net <mailto:thomas@monjalon.net>> wrote:
> >     >>
> >     >> > 20/07/2023 11:35, Igor de Paula:
> >     >> > > The weird thing is that it only happens when I am using a
> >     host with an
> >     >> > AMD
> >     >> > > processor. It doesn't happen when I use a host with an Intel
> >     processor.
> >     >> >
> >     >> > So it's probably a matter of BIOS settings for the IOMMU?
> >     >> >
> >     >> >
> >     >> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon
> >     <thomas@monjalon.net <mailto:thomas@monjalon.net>>
> >     >> > > wrote:
> >     >> > >
> >     >> > > > +Cc the vmxnet3 maintainer.
> >     >> > > >
> >     >> > > > Please Jochen, do you have an idea what's wrong below?
> >     >> > > >
> >     >> > > >
> >     >> > > > 20/07/2023 11:25, Igor de Paula:
> >     >> > > > > This is because it can't negotiate the IOMMU type with
> >     any port.
> >     >> > > > >
> >     >> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon
> >     <thomas@monjalon.net <mailto:thomas@monjalon.net>
> >     >> > >
> >     >> > > > wrote:
> >     >> > > > >
> >     >> > > > > > Hello,
> >     >> > > > > >
> >     >> > > > > > The first error is "Cause: Error: number of ports must
> >     be even"
> >     >> > > > > >
> >     >> > > > > >
> >     >> > > > > > 03/05/2023 18:13, Igor de Paula:
> >     >> > > > > > > I am running a VM inside a VMWARE server (vSphere).
> >     >> > > > > > > My goal it to set up DPDK with two HW ports, and set
> up a
> >     >> > > > virtio_user to
> >     >> > > > > > > interact with the kernel stack.
> >     >> > > > > > > In another app I have it working but instead of
> >     virtio_user I am
> >     >> > > > running
> >     >> > > > > > > KNI, it works in IOVA-PA mode.
> >     >> > > > > > > I am looking to replace the KNI.
> >     >> > > > > > >
> >     >> > > > > > > When I try to set up virtio_user port as in the doc:
> >     >> > > > > > >
> >     >> > > > > >
> >     >> > > >
> >     >> >
> >
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> <
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path>
> <
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> <
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> >>
> >     >> > > > > > > I get a error it can't run in PA mode.
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > When I try to run as VA mode from a parameter, I get
> the
> >     >> > following
> >     >> > > > > > errors:
> >     >> > > > > > > EAL: lib.eal log level changed from info to debug
> >     >> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0
> >     >> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0
> >     >> > > > > > > EAL: Support maximum 128 logical core(s) by
> >     configuration.
> >     >> > > > > > > EAL: Detected 2 lcore(s)
> >     >> > > > > > > EAL: Detected 1 NUMA nodes
> >     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3'
> >     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21'
> >     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so'
> >     >> > > > > > > EAL: Detected static linkage of DPDK
> >     >> > > > > > > EAL: Ask a virtual area of 0x7000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x100000000 (size = 0x7000)
> >     >> > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> >     >> > > > > > > EAL: DPAA Bus not present. Skipping.
> >     >> > > > > > > EAL: VFIO PCI modules not loaded
> >     >> > > > > > > EAL: Selected IOVA mode 'VA'
> >     >> > > > > > > EAL: Probing VFIO support...
> >     >> > > > > > > EAL: IOMMU type 1 (Type 1) is supported
> >     >> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported
> >     >> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported
> >     >> > > > > > > EAL: VFIO support initialized
> >     >> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x100007000 (size =
> 0x5b000)
> >     >> > > > > > > EAL: Setting up physically contiguous memory...
> >     >> > > > > > > EAL: Setting maximum number of open files to 1048576
> >     >> > > > > > > EAL: Detected memory type: socket_id:0
> >     hugepage_sz:1073741824
> >     >> > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
> >     >> > > > > > hugepage_sz:1073741824
> >     >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000)
> >     >> > > > > > > EAL: Memseg list allocated at socket 0, page size
> >     0x100000kB
> >     >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x140000000 (size =
> >     0x2000000000)
> >     >> > > > > > > EAL: VA reserved for memseg list at 0x140000000, size
> >     2000000000
> >     >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x2140000000 (size =
> 0x2000)
> >     >> > > > > > > EAL: Memseg list allocated at socket 0, page size
> >     0x100000kB
> >     >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x2180000000 (size =
> >     0x2000000000)
> >     >> > > > > > > EAL: VA reserved for memseg list at 0x2180000000,
> >     size 2000000000
> >     >> > > > > > > EAL: TSC frequency is ~2350000 KHz
> >     >> > > > > > > EAL: Main lcore 0 is ready
> (tid=7f8ad790ec00;cpuset=[0])
> >     >> > > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
> >     >> > > > > > > EAL: Trying to obtain current memory policy.
> >     >> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0
> >     >> > > > > > > EAL: Restoring previous memory policy: 0
> >     >> > > > > > > EAL: request: mp_malloc_sync
> >     >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> >     >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >     >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is
> >     'VA', not
> >     >> > > > > > initializing
> >     >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> >     >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >     >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is
> >     'VA', not
> >     >> > > > > > initializing
> >     >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> >     >> > > > > > > EAL: Bus (pci) probe failed.
> >     >> > > > > > > EAL: lib.telemetry log level changed from disabled to
> >     warning
> >     >> > > > > > > EAL: Error - exiting with code: 1
> >     >> > > > > > >   Cause: Error: number of ports must be even
> >     >> > > > > > > EAL: request: mp_malloc_sync
> >     >> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > For some reason the HW ports won't setup. From what I
> >     understand
> >     >> > > > > > > net_vmxnet3 should work with VA mode.
> >     >> > > > > > > I enabled I/OMUU for the VM.
> >     >> > > > > > > The weird thing even when enabled, I still have the
> >     >> > > > > > > enable_unsafe_noiommu_mode flag on.
> >     >> > > > > > > And because it's on the this:
> >     >> > > > > > >
> >     >> > > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev);
> >     >> > > > > > >
> >     >> > > > > > > return PA mode, and it fails.
> >     >> > > > > > >
> >     >> > > > > > > When I disable it by modifying
> >     >> > > > > > >
> >     /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get
> >     >> > another
> >     >> > > > > > error.
> >     >> > > > > > > The error is that it doesn't find a suitable IOMMU
> type:
> >     >> > > > > > > Just putting the relevant message:
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> >     >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >     >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No
> >     such device)
> >     >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
> >     such device)
> >     >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
> >     such
> >     >> > device)
> >     >> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type
> >     >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> >     >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >     >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No
> >     such device)
> >     >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
> >     such device)
> >     >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
> >     such
> >     >> > device)
> >     >> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type
> >     >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > I can see where it fails:
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > const struct vfio_iommu_type *
> >     >> > > > > > > vfio_set_iommu_type(int vfio_container_fd)
> >     >> > > > > > > {
> >     >> > > > > > > unsigned idx;
> >     >> > > > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
> >     >> > > > > > > const struct vfio_iommu_type *t = &iommu_types[idx];
> >     >> > > > > > >
> >     >> > > > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
> >     >> > > > > > > t->type_id);
> >     >> > > > > > > if (!ret) {
> >     >> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> >     >> > > > > > > t->type_id, t->name);
> >     >> > > > > > > return t;
> >     >> > > > > > > }
> >     >> > > > > > > /* not an error, there may be more supported IOMMU
> >     types */
> >     >> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed,
> >     error "
> >     >> > > > > > > "%i (%s)\n", t->type_id, t->name, errno,
> >     >> > > > > > > strerror(errno));
> >     >> > > > > > > }
> >     >> > > > > > > /* if we didn't find a suitable IOMMU type, fail */
> >     >> > > > > > > return NULL;
> >     >> > > > > > > }
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > The ioctl returns -1.
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > I tried many things.
> >     >> > > > > > > Enabling/disabling IOMMU.
> >     >> > > > > > > Modifying the enable_unsafe_noiommu_mode flag.
> >     >> > > > > > > Running in PA mode.
> >     >> > > > > > > Any help will be much appreciated.
> >     >> >
> >     >> >
> >     >> >
> >     >> >
> >     >> >
> >     >> >
> >     >>
> >     >
> >     >
> >     >
> >     >
> >     >
> >     >
> >     > !! External Email: This email originated from outside of the
> >     > organization. Do not click links or open attachments unless you
> >     > recognize the sender.
> >     >
> >
>
>

[-- Attachment #2: Type: text/html, Size: 30079 bytes --]