help

help

[Igor de Paula]

[-- Attachment #1: Type: text/plain, Size: 5595 bytes --]

I am running a VM inside a VMWARE server (vSphere).
My goal it to set up DPDK with two HW ports, and set up a virtio_user to
interact with the kernel stack.
In another app I have it working but instead of virtio_user I am running
KNI, it works in IOVA-PA mode.
I am looking to replace the KNI.

When I try to set up virtio_user port as in the doc:
https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
I get a error it can't run in PA mode.


When I try to run as VA mode from a parameter, I get the following errors:
EAL: lib.eal log level changed from info to debug
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 2 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Checking presence of .so 'librte_eal.so.21.3'
EAL: Checking presence of .so 'librte_eal.so.21'
EAL: Checking presence of .so 'librte_eal.so'
EAL: Detected static linkage of DPDK
EAL: Ask a virtual area of 0x7000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x7000)
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: DPAA Bus not present. Skipping.
EAL: VFIO PCI modules not loaded
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
EAL: IOMMU type 1 (Type 1) is supported
EAL: IOMMU type 7 (sPAPR) is not supported
EAL: IOMMU type 8 (No-IOMMU) is supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x5b000 bytes
EAL: Virtual area found at 0x100007000 (size = 0x5b000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 1048576
EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
EAL: Creating 2 segment lists: n_segs:128 socket_id:0 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x2000 bytes
EAL: Virtual area found at 0x100062000 (size = 0x2000)
EAL: Memseg list allocated at socket 0, page size 0x100000kB
EAL: Ask a virtual area of 0x2000000000 bytes
EAL: Virtual area found at 0x140000000 (size = 0x2000000000)
EAL: VA reserved for memseg list at 0x140000000, size 2000000000
EAL: Ask a virtual area of 0x2000 bytes
EAL: Virtual area found at 0x2140000000 (size = 0x2000)
EAL: Memseg list allocated at socket 0, page size 0x100000kB
EAL: Ask a virtual area of 0x2000000000 bytes
EAL: Virtual area found at 0x2180000000 (size = 0x2000000000)
EAL: VA reserved for memseg list at 0x2180000000, size 2000000000
EAL: TSC frequency is ~2350000 KHz
EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0])
EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 1024MB
EAL: PCI device 0000:0b:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 net_vmxnet3
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0000:0b:00.0 cannot be used
EAL: PCI device 0000:13:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 net_vmxnet3
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0000:13:00.0 cannot be used
EAL: Bus (pci) probe failed.
EAL: lib.telemetry log level changed from disabled to warning
EAL: Error - exiting with code: 1
  Cause: Error: number of ports must be even
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 1024MB



For some reason the HW ports won't setup. From what I understand
net_vmxnet3 should work with VA mode.
I enabled I/OMUU for the VM.
The weird thing even when enabled, I still have the
enable_unsafe_noiommu_mode flag on.
And because it's on the this:

dev_iova_mode = pci_device_iova_mode(dr, dev);

return PA mode, and it fails.

When I disable it by modifying
/sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get another error.
The error is that it doesn't find a suitable IOMMU type:
Just putting the relevant message:


EAL: Heap on socket 0 was expanded by 1024MB
EAL: PCI device 0000:0b:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 net_vmxnet3
EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device)
EAL: 0000:0b:00.0 failed to select IOMMU type
EAL: Requested device 0000:0b:00.0 cannot be used
EAL: PCI device 0000:13:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 net_vmxnet3
EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device)
EAL: 0000:13:00.0 failed to select IOMMU type
EAL: Requested device 0000:13:00.0 cannot be used


I can see where it fails:


const struct vfio_iommu_type *
vfio_set_iommu_type(int vfio_container_fd)
{
unsigned idx;
for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
const struct vfio_iommu_type *t = &iommu_types[idx];

int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
t->type_id);
if (!ret) {
RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
t->type_id, t->name);
return t;
}
/* not an error, there may be more supported IOMMU types */
RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error "
"%i (%s)\n", t->type_id, t->name, errno,
strerror(errno));
}
/* if we didn't find a suitable IOMMU type, fail */
return NULL;
}


The ioctl returns -1.



I tried many things.
Enabling/disabling IOMMU.
Modifying the enable_unsafe_noiommu_mode flag.
Running in PA mode.
Any help will be much appreciated.

[-- Attachment #2: Type: text/html, Size: 6342 bytes --]

Re: help

[Thomas Monjalon]

Hello,

The first error is "Cause: Error: number of ports must be even"


03/05/2023 18:13, Igor de Paula:
> I am running a VM inside a VMWARE server (vSphere).
> My goal it to set up DPDK with two HW ports, and set up a virtio_user to
> interact with the kernel stack.
> In another app I have it working but instead of virtio_user I am running
> KNI, it works in IOVA-PA mode.
> I am looking to replace the KNI.
> 
> When I try to set up virtio_user port as in the doc:
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> I get a error it can't run in PA mode.
> 
> 
> When I try to run as VA mode from a parameter, I get the following errors:
> EAL: lib.eal log level changed from info to debug
> EAL: Detected lcore 0 as core 0 on socket 0
> EAL: Detected lcore 1 as core 0 on socket 0
> EAL: Support maximum 128 logical core(s) by configuration.
> EAL: Detected 2 lcore(s)
> EAL: Detected 1 NUMA nodes
> EAL: Checking presence of .so 'librte_eal.so.21.3'
> EAL: Checking presence of .so 'librte_eal.so.21'
> EAL: Checking presence of .so 'librte_eal.so'
> EAL: Detected static linkage of DPDK
> EAL: Ask a virtual area of 0x7000 bytes
> EAL: Virtual area found at 0x100000000 (size = 0x7000)
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: DPAA Bus not present. Skipping.
> EAL: VFIO PCI modules not loaded
> EAL: Selected IOVA mode 'VA'
> EAL: Probing VFIO support...
> EAL: IOMMU type 1 (Type 1) is supported
> EAL: IOMMU type 7 (sPAPR) is not supported
> EAL: IOMMU type 8 (No-IOMMU) is supported
> EAL: VFIO support initialized
> EAL: Ask a virtual area of 0x5b000 bytes
> EAL: Virtual area found at 0x100007000 (size = 0x5b000)
> EAL: Setting up physically contiguous memory...
> EAL: Setting maximum number of open files to 1048576
> EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
> EAL: Creating 2 segment lists: n_segs:128 socket_id:0 hugepage_sz:1073741824
> EAL: Ask a virtual area of 0x2000 bytes
> EAL: Virtual area found at 0x100062000 (size = 0x2000)
> EAL: Memseg list allocated at socket 0, page size 0x100000kB
> EAL: Ask a virtual area of 0x2000000000 bytes
> EAL: Virtual area found at 0x140000000 (size = 0x2000000000)
> EAL: VA reserved for memseg list at 0x140000000, size 2000000000
> EAL: Ask a virtual area of 0x2000 bytes
> EAL: Virtual area found at 0x2140000000 (size = 0x2000)
> EAL: Memseg list allocated at socket 0, page size 0x100000kB
> EAL: Ask a virtual area of 0x2000000000 bytes
> EAL: Virtual area found at 0x2180000000 (size = 0x2000000000)
> EAL: VA reserved for memseg list at 0x2180000000, size 2000000000
> EAL: TSC frequency is ~2350000 KHz
> EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0])
> EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
> EAL: Trying to obtain current memory policy.
> EAL: Setting policy MPOL_PREFERRED for socket 0
> EAL: Restoring previous memory policy: 0
> EAL: request: mp_malloc_sync
> EAL: Heap on socket 0 was expanded by 1024MB
> EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> EAL:   probe driver: 15ad:7b0 net_vmxnet3
> EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
> EAL: Requested device 0000:0b:00.0 cannot be used
> EAL: PCI device 0000:13:00.0 on NUMA socket -1
> EAL:   probe driver: 15ad:7b0 net_vmxnet3
> EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
> EAL: Requested device 0000:13:00.0 cannot be used
> EAL: Bus (pci) probe failed.
> EAL: lib.telemetry log level changed from disabled to warning
> EAL: Error - exiting with code: 1
>   Cause: Error: number of ports must be even
> EAL: request: mp_malloc_sync
> EAL: Heap on socket 0 was shrunk by 1024MB
> 
> 
> 
> For some reason the HW ports won't setup. From what I understand
> net_vmxnet3 should work with VA mode.
> I enabled I/OMUU for the VM.
> The weird thing even when enabled, I still have the
> enable_unsafe_noiommu_mode flag on.
> And because it's on the this:
> 
> dev_iova_mode = pci_device_iova_mode(dr, dev);
> 
> return PA mode, and it fails.
> 
> When I disable it by modifying
> /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get another error.
> The error is that it doesn't find a suitable IOMMU type:
> Just putting the relevant message:
> 
> 
> EAL: Heap on socket 0 was expanded by 1024MB
> EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> EAL:   probe driver: 15ad:7b0 net_vmxnet3
> EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
> EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
> EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device)
> EAL: 0000:0b:00.0 failed to select IOMMU type
> EAL: Requested device 0000:0b:00.0 cannot be used
> EAL: PCI device 0000:13:00.0 on NUMA socket -1
> EAL:   probe driver: 15ad:7b0 net_vmxnet3
> EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
> EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
> EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device)
> EAL: 0000:13:00.0 failed to select IOMMU type
> EAL: Requested device 0000:13:00.0 cannot be used
> 
> 
> I can see where it fails:
> 
> 
> const struct vfio_iommu_type *
> vfio_set_iommu_type(int vfio_container_fd)
> {
> unsigned idx;
> for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
> const struct vfio_iommu_type *t = &iommu_types[idx];
> 
> int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
> t->type_id);
> if (!ret) {
> RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> t->type_id, t->name);
> return t;
> }
> /* not an error, there may be more supported IOMMU types */
> RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error "
> "%i (%s)\n", t->type_id, t->name, errno,
> strerror(errno));
> }
> /* if we didn't find a suitable IOMMU type, fail */
> return NULL;
> }
> 
> 
> The ioctl returns -1.
> 
> 
> 
> I tried many things.
> Enabling/disabling IOMMU.
> Modifying the enable_unsafe_noiommu_mode flag.
> Running in PA mode.
> Any help will be much appreciated.
> 

Re: help

[Igor de Paula]

[-- Attachment #1: Type: text/plain, Size: 6622 bytes --]

This is because it can't negotiate the IOMMU type with any port.

On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon <thomas@monjalon.net> wrote:

> Hello,
>
> The first error is "Cause: Error: number of ports must be even"
>
>
> 03/05/2023 18:13, Igor de Paula:
> > I am running a VM inside a VMWARE server (vSphere).
> > My goal it to set up DPDK with two HW ports, and set up a virtio_user to
> > interact with the kernel stack.
> > In another app I have it working but instead of virtio_user I am running
> > KNI, it works in IOVA-PA mode.
> > I am looking to replace the KNI.
> >
> > When I try to set up virtio_user port as in the doc:
> >
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> > I get a error it can't run in PA mode.
> >
> >
> > When I try to run as VA mode from a parameter, I get the following
> errors:
> > EAL: lib.eal log level changed from info to debug
> > EAL: Detected lcore 0 as core 0 on socket 0
> > EAL: Detected lcore 1 as core 0 on socket 0
> > EAL: Support maximum 128 logical core(s) by configuration.
> > EAL: Detected 2 lcore(s)
> > EAL: Detected 1 NUMA nodes
> > EAL: Checking presence of .so 'librte_eal.so.21.3'
> > EAL: Checking presence of .so 'librte_eal.so.21'
> > EAL: Checking presence of .so 'librte_eal.so'
> > EAL: Detected static linkage of DPDK
> > EAL: Ask a virtual area of 0x7000 bytes
> > EAL: Virtual area found at 0x100000000 (size = 0x7000)
> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > EAL: DPAA Bus not present. Skipping.
> > EAL: VFIO PCI modules not loaded
> > EAL: Selected IOVA mode 'VA'
> > EAL: Probing VFIO support...
> > EAL: IOMMU type 1 (Type 1) is supported
> > EAL: IOMMU type 7 (sPAPR) is not supported
> > EAL: IOMMU type 8 (No-IOMMU) is supported
> > EAL: VFIO support initialized
> > EAL: Ask a virtual area of 0x5b000 bytes
> > EAL: Virtual area found at 0x100007000 (size = 0x5b000)
> > EAL: Setting up physically contiguous memory...
> > EAL: Setting maximum number of open files to 1048576
> > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
> > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
> hugepage_sz:1073741824
> > EAL: Ask a virtual area of 0x2000 bytes
> > EAL: Virtual area found at 0x100062000 (size = 0x2000)
> > EAL: Memseg list allocated at socket 0, page size 0x100000kB
> > EAL: Ask a virtual area of 0x2000000000 bytes
> > EAL: Virtual area found at 0x140000000 (size = 0x2000000000)
> > EAL: VA reserved for memseg list at 0x140000000, size 2000000000
> > EAL: Ask a virtual area of 0x2000 bytes
> > EAL: Virtual area found at 0x2140000000 (size = 0x2000)
> > EAL: Memseg list allocated at socket 0, page size 0x100000kB
> > EAL: Ask a virtual area of 0x2000000000 bytes
> > EAL: Virtual area found at 0x2180000000 (size = 0x2000000000)
> > EAL: VA reserved for memseg list at 0x2180000000, size 2000000000
> > EAL: TSC frequency is ~2350000 KHz
> > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0])
> > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
> > EAL: Trying to obtain current memory policy.
> > EAL: Setting policy MPOL_PREFERRED for socket 0
> > EAL: Restoring previous memory policy: 0
> > EAL: request: mp_malloc_sync
> > EAL: Heap on socket 0 was expanded by 1024MB
> > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
> initializing
> > EAL: Requested device 0000:0b:00.0 cannot be used
> > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
> initializing
> > EAL: Requested device 0000:13:00.0 cannot be used
> > EAL: Bus (pci) probe failed.
> > EAL: lib.telemetry log level changed from disabled to warning
> > EAL: Error - exiting with code: 1
> >   Cause: Error: number of ports must be even
> > EAL: request: mp_malloc_sync
> > EAL: Heap on socket 0 was shrunk by 1024MB
> >
> >
> >
> > For some reason the HW ports won't setup. From what I understand
> > net_vmxnet3 should work with VA mode.
> > I enabled I/OMUU for the VM.
> > The weird thing even when enabled, I still have the
> > enable_unsafe_noiommu_mode flag on.
> > And because it's on the this:
> >
> > dev_iova_mode = pci_device_iova_mode(dr, dev);
> >
> > return PA mode, and it fails.
> >
> > When I disable it by modifying
> > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get another
> error.
> > The error is that it doesn't find a suitable IOMMU type:
> > Just putting the relevant message:
> >
> >
> > EAL: Heap on socket 0 was expanded by 1024MB
> > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
> > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
> > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device)
> > EAL: 0000:0b:00.0 failed to select IOMMU type
> > EAL: Requested device 0000:0b:00.0 cannot be used
> > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
> > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
> > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device)
> > EAL: 0000:13:00.0 failed to select IOMMU type
> > EAL: Requested device 0000:13:00.0 cannot be used
> >
> >
> > I can see where it fails:
> >
> >
> > const struct vfio_iommu_type *
> > vfio_set_iommu_type(int vfio_container_fd)
> > {
> > unsigned idx;
> > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
> > const struct vfio_iommu_type *t = &iommu_types[idx];
> >
> > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
> > t->type_id);
> > if (!ret) {
> > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> > t->type_id, t->name);
> > return t;
> > }
> > /* not an error, there may be more supported IOMMU types */
> > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error "
> > "%i (%s)\n", t->type_id, t->name, errno,
> > strerror(errno));
> > }
> > /* if we didn't find a suitable IOMMU type, fail */
> > return NULL;
> > }
> >
> >
> > The ioctl returns -1.
> >
> >
> >
> > I tried many things.
> > Enabling/disabling IOMMU.
> > Modifying the enable_unsafe_noiommu_mode flag.
> > Running in PA mode.
> > Any help will be much appreciated.
> >
>
>
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 7978 bytes --]

Re: help

[Thomas Monjalon]

+Cc the vmxnet3 maintainer.

Please Jochen, do you have an idea what's wrong below?


20/07/2023 11:25, Igor de Paula:
> This is because it can't negotiate the IOMMU type with any port.
> 
> On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> 
> > Hello,
> >
> > The first error is "Cause: Error: number of ports must be even"
> >
> >
> > 03/05/2023 18:13, Igor de Paula:
> > > I am running a VM inside a VMWARE server (vSphere).
> > > My goal it to set up DPDK with two HW ports, and set up a virtio_user to
> > > interact with the kernel stack.
> > > In another app I have it working but instead of virtio_user I am running
> > > KNI, it works in IOVA-PA mode.
> > > I am looking to replace the KNI.
> > >
> > > When I try to set up virtio_user port as in the doc:
> > >
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> > > I get a error it can't run in PA mode.
> > >
> > >
> > > When I try to run as VA mode from a parameter, I get the following
> > errors:
> > > EAL: lib.eal log level changed from info to debug
> > > EAL: Detected lcore 0 as core 0 on socket 0
> > > EAL: Detected lcore 1 as core 0 on socket 0
> > > EAL: Support maximum 128 logical core(s) by configuration.
> > > EAL: Detected 2 lcore(s)
> > > EAL: Detected 1 NUMA nodes
> > > EAL: Checking presence of .so 'librte_eal.so.21.3'
> > > EAL: Checking presence of .so 'librte_eal.so.21'
> > > EAL: Checking presence of .so 'librte_eal.so'
> > > EAL: Detected static linkage of DPDK
> > > EAL: Ask a virtual area of 0x7000 bytes
> > > EAL: Virtual area found at 0x100000000 (size = 0x7000)
> > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > > EAL: DPAA Bus not present. Skipping.
> > > EAL: VFIO PCI modules not loaded
> > > EAL: Selected IOVA mode 'VA'
> > > EAL: Probing VFIO support...
> > > EAL: IOMMU type 1 (Type 1) is supported
> > > EAL: IOMMU type 7 (sPAPR) is not supported
> > > EAL: IOMMU type 8 (No-IOMMU) is supported
> > > EAL: VFIO support initialized
> > > EAL: Ask a virtual area of 0x5b000 bytes
> > > EAL: Virtual area found at 0x100007000 (size = 0x5b000)
> > > EAL: Setting up physically contiguous memory...
> > > EAL: Setting maximum number of open files to 1048576
> > > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
> > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
> > hugepage_sz:1073741824
> > > EAL: Ask a virtual area of 0x2000 bytes
> > > EAL: Virtual area found at 0x100062000 (size = 0x2000)
> > > EAL: Memseg list allocated at socket 0, page size 0x100000kB
> > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > EAL: Virtual area found at 0x140000000 (size = 0x2000000000)
> > > EAL: VA reserved for memseg list at 0x140000000, size 2000000000
> > > EAL: Ask a virtual area of 0x2000 bytes
> > > EAL: Virtual area found at 0x2140000000 (size = 0x2000)
> > > EAL: Memseg list allocated at socket 0, page size 0x100000kB
> > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > EAL: Virtual area found at 0x2180000000 (size = 0x2000000000)
> > > EAL: VA reserved for memseg list at 0x2180000000, size 2000000000
> > > EAL: TSC frequency is ~2350000 KHz
> > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0])
> > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
> > > EAL: Trying to obtain current memory policy.
> > > EAL: Setting policy MPOL_PREFERRED for socket 0
> > > EAL: Restoring previous memory policy: 0
> > > EAL: request: mp_malloc_sync
> > > EAL: Heap on socket 0 was expanded by 1024MB
> > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
> > initializing
> > > EAL: Requested device 0000:0b:00.0 cannot be used
> > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
> > initializing
> > > EAL: Requested device 0000:13:00.0 cannot be used
> > > EAL: Bus (pci) probe failed.
> > > EAL: lib.telemetry log level changed from disabled to warning
> > > EAL: Error - exiting with code: 1
> > >   Cause: Error: number of ports must be even
> > > EAL: request: mp_malloc_sync
> > > EAL: Heap on socket 0 was shrunk by 1024MB
> > >
> > >
> > >
> > > For some reason the HW ports won't setup. From what I understand
> > > net_vmxnet3 should work with VA mode.
> > > I enabled I/OMUU for the VM.
> > > The weird thing even when enabled, I still have the
> > > enable_unsafe_noiommu_mode flag on.
> > > And because it's on the this:
> > >
> > > dev_iova_mode = pci_device_iova_mode(dr, dev);
> > >
> > > return PA mode, and it fails.
> > >
> > > When I disable it by modifying
> > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get another
> > error.
> > > The error is that it doesn't find a suitable IOMMU type:
> > > Just putting the relevant message:
> > >
> > >
> > > EAL: Heap on socket 0 was expanded by 1024MB
> > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
> > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
> > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device)
> > > EAL: 0000:0b:00.0 failed to select IOMMU type
> > > EAL: Requested device 0000:0b:00.0 cannot be used
> > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
> > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
> > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device)
> > > EAL: 0000:13:00.0 failed to select IOMMU type
> > > EAL: Requested device 0000:13:00.0 cannot be used
> > >
> > >
> > > I can see where it fails:
> > >
> > >
> > > const struct vfio_iommu_type *
> > > vfio_set_iommu_type(int vfio_container_fd)
> > > {
> > > unsigned idx;
> > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
> > > const struct vfio_iommu_type *t = &iommu_types[idx];
> > >
> > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
> > > t->type_id);
> > > if (!ret) {
> > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> > > t->type_id, t->name);
> > > return t;
> > > }
> > > /* not an error, there may be more supported IOMMU types */
> > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error "
> > > "%i (%s)\n", t->type_id, t->name, errno,
> > > strerror(errno));
> > > }
> > > /* if we didn't find a suitable IOMMU type, fail */
> > > return NULL;
> > > }
> > >
> > >
> > > The ioctl returns -1.
> > >
> > >
> > >
> > > I tried many things.
> > > Enabling/disabling IOMMU.
> > > Modifying the enable_unsafe_noiommu_mode flag.
> > > Running in PA mode.
> > > Any help will be much appreciated.
> > >
> >
> >
> >
> >
> >
> >
> 

Re: help

[Igor de Paula]

[-- Attachment #1: Type: text/plain, Size: 7704 bytes --]

The weird thing is that it only happens when I am using a host with an AMD
processor. It doesn't happen when I use a host with an Intel processor.

On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon <thomas@monjalon.net>
wrote:

> +Cc the vmxnet3 maintainer.
>
> Please Jochen, do you have an idea what's wrong below?
>
>
> 20/07/2023 11:25, Igor de Paula:
> > This is because it can't negotiate the IOMMU type with any port.
> >
> > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon <thomas@monjalon.net>
> wrote:
> >
> > > Hello,
> > >
> > > The first error is "Cause: Error: number of ports must be even"
> > >
> > >
> > > 03/05/2023 18:13, Igor de Paula:
> > > > I am running a VM inside a VMWARE server (vSphere).
> > > > My goal it to set up DPDK with two HW ports, and set up a
> virtio_user to
> > > > interact with the kernel stack.
> > > > In another app I have it working but instead of virtio_user I am
> running
> > > > KNI, it works in IOVA-PA mode.
> > > > I am looking to replace the KNI.
> > > >
> > > > When I try to set up virtio_user port as in the doc:
> > > >
> > >
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> > > > I get a error it can't run in PA mode.
> > > >
> > > >
> > > > When I try to run as VA mode from a parameter, I get the following
> > > errors:
> > > > EAL: lib.eal log level changed from info to debug
> > > > EAL: Detected lcore 0 as core 0 on socket 0
> > > > EAL: Detected lcore 1 as core 0 on socket 0
> > > > EAL: Support maximum 128 logical core(s) by configuration.
> > > > EAL: Detected 2 lcore(s)
> > > > EAL: Detected 1 NUMA nodes
> > > > EAL: Checking presence of .so 'librte_eal.so.21.3'
> > > > EAL: Checking presence of .so 'librte_eal.so.21'
> > > > EAL: Checking presence of .so 'librte_eal.so'
> > > > EAL: Detected static linkage of DPDK
> > > > EAL: Ask a virtual area of 0x7000 bytes
> > > > EAL: Virtual area found at 0x100000000 (size = 0x7000)
> > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > > > EAL: DPAA Bus not present. Skipping.
> > > > EAL: VFIO PCI modules not loaded
> > > > EAL: Selected IOVA mode 'VA'
> > > > EAL: Probing VFIO support...
> > > > EAL: IOMMU type 1 (Type 1) is supported
> > > > EAL: IOMMU type 7 (sPAPR) is not supported
> > > > EAL: IOMMU type 8 (No-IOMMU) is supported
> > > > EAL: VFIO support initialized
> > > > EAL: Ask a virtual area of 0x5b000 bytes
> > > > EAL: Virtual area found at 0x100007000 (size = 0x5b000)
> > > > EAL: Setting up physically contiguous memory...
> > > > EAL: Setting maximum number of open files to 1048576
> > > > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
> > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
> > > hugepage_sz:1073741824
> > > > EAL: Ask a virtual area of 0x2000 bytes
> > > > EAL: Virtual area found at 0x100062000 (size = 0x2000)
> > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB
> > > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > > EAL: Virtual area found at 0x140000000 (size = 0x2000000000)
> > > > EAL: VA reserved for memseg list at 0x140000000, size 2000000000
> > > > EAL: Ask a virtual area of 0x2000 bytes
> > > > EAL: Virtual area found at 0x2140000000 (size = 0x2000)
> > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB
> > > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > > EAL: Virtual area found at 0x2180000000 (size = 0x2000000000)
> > > > EAL: VA reserved for memseg list at 0x2180000000, size 2000000000
> > > > EAL: TSC frequency is ~2350000 KHz
> > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0])
> > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
> > > > EAL: Trying to obtain current memory policy.
> > > > EAL: Setting policy MPOL_PREFERRED for socket 0
> > > > EAL: Restoring previous memory policy: 0
> > > > EAL: request: mp_malloc_sync
> > > > EAL: Heap on socket 0 was expanded by 1024MB
> > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
> > > initializing
> > > > EAL: Requested device 0000:0b:00.0 cannot be used
> > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
> > > initializing
> > > > EAL: Requested device 0000:13:00.0 cannot be used
> > > > EAL: Bus (pci) probe failed.
> > > > EAL: lib.telemetry log level changed from disabled to warning
> > > > EAL: Error - exiting with code: 1
> > > >   Cause: Error: number of ports must be even
> > > > EAL: request: mp_malloc_sync
> > > > EAL: Heap on socket 0 was shrunk by 1024MB
> > > >
> > > >
> > > >
> > > > For some reason the HW ports won't setup. From what I understand
> > > > net_vmxnet3 should work with VA mode.
> > > > I enabled I/OMUU for the VM.
> > > > The weird thing even when enabled, I still have the
> > > > enable_unsafe_noiommu_mode flag on.
> > > > And because it's on the this:
> > > >
> > > > dev_iova_mode = pci_device_iova_mode(dr, dev);
> > > >
> > > > return PA mode, and it fails.
> > > >
> > > > When I disable it by modifying
> > > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get another
> > > error.
> > > > The error is that it doesn't find a suitable IOMMU type:
> > > > Just putting the relevant message:
> > > >
> > > >
> > > > EAL: Heap on socket 0 was expanded by 1024MB
> > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
> > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
> > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device)
> > > > EAL: 0000:0b:00.0 failed to select IOMMU type
> > > > EAL: Requested device 0000:0b:00.0 cannot be used
> > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
> > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
> > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device)
> > > > EAL: 0000:13:00.0 failed to select IOMMU type
> > > > EAL: Requested device 0000:13:00.0 cannot be used
> > > >
> > > >
> > > > I can see where it fails:
> > > >
> > > >
> > > > const struct vfio_iommu_type *
> > > > vfio_set_iommu_type(int vfio_container_fd)
> > > > {
> > > > unsigned idx;
> > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
> > > > const struct vfio_iommu_type *t = &iommu_types[idx];
> > > >
> > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
> > > > t->type_id);
> > > > if (!ret) {
> > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> > > > t->type_id, t->name);
> > > > return t;
> > > > }
> > > > /* not an error, there may be more supported IOMMU types */
> > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error "
> > > > "%i (%s)\n", t->type_id, t->name, errno,
> > > > strerror(errno));
> > > > }
> > > > /* if we didn't find a suitable IOMMU type, fail */
> > > > return NULL;
> > > > }
> > > >
> > > >
> > > > The ioctl returns -1.
> > > >
> > > >
> > > >
> > > > I tried many things.
> > > > Enabling/disabling IOMMU.
> > > > Modifying the enable_unsafe_noiommu_mode flag.
> > > > Running in PA mode.
> > > > Any help will be much appreciated.
> > > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
>
>
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 10177 bytes --]

Re: help

[Thomas Monjalon]

20/07/2023 11:35, Igor de Paula:
> The weird thing is that it only happens when I am using a host with an AMD
> processor. It doesn't happen when I use a host with an Intel processor.

So it's probably a matter of BIOS settings for the IOMMU?


> On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon <thomas@monjalon.net>
> wrote:
> 
> > +Cc the vmxnet3 maintainer.
> >
> > Please Jochen, do you have an idea what's wrong below?
> >
> >
> > 20/07/2023 11:25, Igor de Paula:
> > > This is because it can't negotiate the IOMMU type with any port.
> > >
> > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon <thomas@monjalon.net>
> > wrote:
> > >
> > > > Hello,
> > > >
> > > > The first error is "Cause: Error: number of ports must be even"
> > > >
> > > >
> > > > 03/05/2023 18:13, Igor de Paula:
> > > > > I am running a VM inside a VMWARE server (vSphere).
> > > > > My goal it to set up DPDK with two HW ports, and set up a
> > virtio_user to
> > > > > interact with the kernel stack.
> > > > > In another app I have it working but instead of virtio_user I am
> > running
> > > > > KNI, it works in IOVA-PA mode.
> > > > > I am looking to replace the KNI.
> > > > >
> > > > > When I try to set up virtio_user port as in the doc:
> > > > >
> > > >
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> > > > > I get a error it can't run in PA mode.
> > > > >
> > > > >
> > > > > When I try to run as VA mode from a parameter, I get the following
> > > > errors:
> > > > > EAL: lib.eal log level changed from info to debug
> > > > > EAL: Detected lcore 0 as core 0 on socket 0
> > > > > EAL: Detected lcore 1 as core 0 on socket 0
> > > > > EAL: Support maximum 128 logical core(s) by configuration.
> > > > > EAL: Detected 2 lcore(s)
> > > > > EAL: Detected 1 NUMA nodes
> > > > > EAL: Checking presence of .so 'librte_eal.so.21.3'
> > > > > EAL: Checking presence of .so 'librte_eal.so.21'
> > > > > EAL: Checking presence of .so 'librte_eal.so'
> > > > > EAL: Detected static linkage of DPDK
> > > > > EAL: Ask a virtual area of 0x7000 bytes
> > > > > EAL: Virtual area found at 0x100000000 (size = 0x7000)
> > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > > > > EAL: DPAA Bus not present. Skipping.
> > > > > EAL: VFIO PCI modules not loaded
> > > > > EAL: Selected IOVA mode 'VA'
> > > > > EAL: Probing VFIO support...
> > > > > EAL: IOMMU type 1 (Type 1) is supported
> > > > > EAL: IOMMU type 7 (sPAPR) is not supported
> > > > > EAL: IOMMU type 8 (No-IOMMU) is supported
> > > > > EAL: VFIO support initialized
> > > > > EAL: Ask a virtual area of 0x5b000 bytes
> > > > > EAL: Virtual area found at 0x100007000 (size = 0x5b000)
> > > > > EAL: Setting up physically contiguous memory...
> > > > > EAL: Setting maximum number of open files to 1048576
> > > > > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
> > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
> > > > hugepage_sz:1073741824
> > > > > EAL: Ask a virtual area of 0x2000 bytes
> > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000)
> > > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB
> > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > > > EAL: Virtual area found at 0x140000000 (size = 0x2000000000)
> > > > > EAL: VA reserved for memseg list at 0x140000000, size 2000000000
> > > > > EAL: Ask a virtual area of 0x2000 bytes
> > > > > EAL: Virtual area found at 0x2140000000 (size = 0x2000)
> > > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB
> > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > > > EAL: Virtual area found at 0x2180000000 (size = 0x2000000000)
> > > > > EAL: VA reserved for memseg list at 0x2180000000, size 2000000000
> > > > > EAL: TSC frequency is ~2350000 KHz
> > > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0])
> > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
> > > > > EAL: Trying to obtain current memory policy.
> > > > > EAL: Setting policy MPOL_PREFERRED for socket 0
> > > > > EAL: Restoring previous memory policy: 0
> > > > > EAL: request: mp_malloc_sync
> > > > > EAL: Heap on socket 0 was expanded by 1024MB
> > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
> > > > initializing
> > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
> > > > initializing
> > > > > EAL: Requested device 0000:13:00.0 cannot be used
> > > > > EAL: Bus (pci) probe failed.
> > > > > EAL: lib.telemetry log level changed from disabled to warning
> > > > > EAL: Error - exiting with code: 1
> > > > >   Cause: Error: number of ports must be even
> > > > > EAL: request: mp_malloc_sync
> > > > > EAL: Heap on socket 0 was shrunk by 1024MB
> > > > >
> > > > >
> > > > >
> > > > > For some reason the HW ports won't setup. From what I understand
> > > > > net_vmxnet3 should work with VA mode.
> > > > > I enabled I/OMUU for the VM.
> > > > > The weird thing even when enabled, I still have the
> > > > > enable_unsafe_noiommu_mode flag on.
> > > > > And because it's on the this:
> > > > >
> > > > > dev_iova_mode = pci_device_iova_mode(dr, dev);
> > > > >
> > > > > return PA mode, and it fails.
> > > > >
> > > > > When I disable it by modifying
> > > > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get another
> > > > error.
> > > > > The error is that it doesn't find a suitable IOMMU type:
> > > > > Just putting the relevant message:
> > > > >
> > > > >
> > > > > EAL: Heap on socket 0 was expanded by 1024MB
> > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
> > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
> > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device)
> > > > > EAL: 0000:0b:00.0 failed to select IOMMU type
> > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
> > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
> > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device)
> > > > > EAL: 0000:13:00.0 failed to select IOMMU type
> > > > > EAL: Requested device 0000:13:00.0 cannot be used
> > > > >
> > > > >
> > > > > I can see where it fails:
> > > > >
> > > > >
> > > > > const struct vfio_iommu_type *
> > > > > vfio_set_iommu_type(int vfio_container_fd)
> > > > > {
> > > > > unsigned idx;
> > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
> > > > > const struct vfio_iommu_type *t = &iommu_types[idx];
> > > > >
> > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
> > > > > t->type_id);
> > > > > if (!ret) {
> > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> > > > > t->type_id, t->name);
> > > > > return t;
> > > > > }
> > > > > /* not an error, there may be more supported IOMMU types */
> > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error "
> > > > > "%i (%s)\n", t->type_id, t->name, errno,
> > > > > strerror(errno));
> > > > > }
> > > > > /* if we didn't find a suitable IOMMU type, fail */
> > > > > return NULL;
> > > > > }
> > > > >
> > > > >
> > > > > The ioctl returns -1.
> > > > >
> > > > >
> > > > >
> > > > > I tried many things.
> > > > > Enabling/disabling IOMMU.
> > > > > Modifying the enable_unsafe_noiommu_mode flag.
> > > > > Running in PA mode.
> > > > > Any help will be much appreciated.

Re: help

[Igor de Paula]

[-- Attachment #1: Type: text/plain, Size: 8709 bytes --]

I have enabled it in the host and in the BIOS for AMD...
In the Bios I changed to amd_iommu=on and in the host it's the same for
either.

On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon <thomas@monjalon.net> wrote:

> 20/07/2023 11:35, Igor de Paula:
> > The weird thing is that it only happens when I am using a host with an
> AMD
> > processor. It doesn't happen when I use a host with an Intel processor.
>
> So it's probably a matter of BIOS settings for the IOMMU?
>
>
> > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon <thomas@monjalon.net>
> > wrote:
> >
> > > +Cc the vmxnet3 maintainer.
> > >
> > > Please Jochen, do you have an idea what's wrong below?
> > >
> > >
> > > 20/07/2023 11:25, Igor de Paula:
> > > > This is because it can't negotiate the IOMMU type with any port.
> > > >
> > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon <thomas@monjalon.net
> >
> > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > The first error is "Cause: Error: number of ports must be even"
> > > > >
> > > > >
> > > > > 03/05/2023 18:13, Igor de Paula:
> > > > > > I am running a VM inside a VMWARE server (vSphere).
> > > > > > My goal it to set up DPDK with two HW ports, and set up a
> > > virtio_user to
> > > > > > interact with the kernel stack.
> > > > > > In another app I have it working but instead of virtio_user I am
> > > running
> > > > > > KNI, it works in IOVA-PA mode.
> > > > > > I am looking to replace the KNI.
> > > > > >
> > > > > > When I try to set up virtio_user port as in the doc:
> > > > > >
> > > > >
> > >
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> > > > > > I get a error it can't run in PA mode.
> > > > > >
> > > > > >
> > > > > > When I try to run as VA mode from a parameter, I get the
> following
> > > > > errors:
> > > > > > EAL: lib.eal log level changed from info to debug
> > > > > > EAL: Detected lcore 0 as core 0 on socket 0
> > > > > > EAL: Detected lcore 1 as core 0 on socket 0
> > > > > > EAL: Support maximum 128 logical core(s) by configuration.
> > > > > > EAL: Detected 2 lcore(s)
> > > > > > EAL: Detected 1 NUMA nodes
> > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3'
> > > > > > EAL: Checking presence of .so 'librte_eal.so.21'
> > > > > > EAL: Checking presence of .so 'librte_eal.so'
> > > > > > EAL: Detected static linkage of DPDK
> > > > > > EAL: Ask a virtual area of 0x7000 bytes
> > > > > > EAL: Virtual area found at 0x100000000 (size = 0x7000)
> > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > > > > > EAL: DPAA Bus not present. Skipping.
> > > > > > EAL: VFIO PCI modules not loaded
> > > > > > EAL: Selected IOVA mode 'VA'
> > > > > > EAL: Probing VFIO support...
> > > > > > EAL: IOMMU type 1 (Type 1) is supported
> > > > > > EAL: IOMMU type 7 (sPAPR) is not supported
> > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported
> > > > > > EAL: VFIO support initialized
> > > > > > EAL: Ask a virtual area of 0x5b000 bytes
> > > > > > EAL: Virtual area found at 0x100007000 (size = 0x5b000)
> > > > > > EAL: Setting up physically contiguous memory...
> > > > > > EAL: Setting maximum number of open files to 1048576
> > > > > > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
> > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
> > > > > hugepage_sz:1073741824
> > > > > > EAL: Ask a virtual area of 0x2000 bytes
> > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000)
> > > > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB
> > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > > > > EAL: Virtual area found at 0x140000000 (size = 0x2000000000)
> > > > > > EAL: VA reserved for memseg list at 0x140000000, size 2000000000
> > > > > > EAL: Ask a virtual area of 0x2000 bytes
> > > > > > EAL: Virtual area found at 0x2140000000 (size = 0x2000)
> > > > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB
> > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > > > > EAL: Virtual area found at 0x2180000000 (size = 0x2000000000)
> > > > > > EAL: VA reserved for memseg list at 0x2180000000, size 2000000000
> > > > > > EAL: TSC frequency is ~2350000 KHz
> > > > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0])
> > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
> > > > > > EAL: Trying to obtain current memory policy.
> > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0
> > > > > > EAL: Restoring previous memory policy: 0
> > > > > > EAL: request: mp_malloc_sync
> > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
> > > > > initializing
> > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
> > > > > initializing
> > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> > > > > > EAL: Bus (pci) probe failed.
> > > > > > EAL: lib.telemetry log level changed from disabled to warning
> > > > > > EAL: Error - exiting with code: 1
> > > > > >   Cause: Error: number of ports must be even
> > > > > > EAL: request: mp_malloc_sync
> > > > > > EAL: Heap on socket 0 was shrunk by 1024MB
> > > > > >
> > > > > >
> > > > > >
> > > > > > For some reason the HW ports won't setup. From what I understand
> > > > > > net_vmxnet3 should work with VA mode.
> > > > > > I enabled I/OMUU for the VM.
> > > > > > The weird thing even when enabled, I still have the
> > > > > > enable_unsafe_noiommu_mode flag on.
> > > > > > And because it's on the this:
> > > > > >
> > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev);
> > > > > >
> > > > > > return PA mode, and it fails.
> > > > > >
> > > > > > When I disable it by modifying
> > > > > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get
> another
> > > > > error.
> > > > > > The error is that it doesn't find a suitable IOMMU type:
> > > > > > Just putting the relevant message:
> > > > > >
> > > > > >
> > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
> > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
> > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such
> device)
> > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type
> > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
> > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
> > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such
> device)
> > > > > > EAL: 0000:13:00.0 failed to select IOMMU type
> > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> > > > > >
> > > > > >
> > > > > > I can see where it fails:
> > > > > >
> > > > > >
> > > > > > const struct vfio_iommu_type *
> > > > > > vfio_set_iommu_type(int vfio_container_fd)
> > > > > > {
> > > > > > unsigned idx;
> > > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
> > > > > > const struct vfio_iommu_type *t = &iommu_types[idx];
> > > > > >
> > > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
> > > > > > t->type_id);
> > > > > > if (!ret) {
> > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> > > > > > t->type_id, t->name);
> > > > > > return t;
> > > > > > }
> > > > > > /* not an error, there may be more supported IOMMU types */
> > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error "
> > > > > > "%i (%s)\n", t->type_id, t->name, errno,
> > > > > > strerror(errno));
> > > > > > }
> > > > > > /* if we didn't find a suitable IOMMU type, fail */
> > > > > > return NULL;
> > > > > > }
> > > > > >
> > > > > >
> > > > > > The ioctl returns -1.
> > > > > >
> > > > > >
> > > > > >
> > > > > > I tried many things.
> > > > > > Enabling/disabling IOMMU.
> > > > > > Modifying the enable_unsafe_noiommu_mode flag.
> > > > > > Running in PA mode.
> > > > > > Any help will be much appreciated.
>
>
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 12257 bytes --]

Re: help

[Thomas Monjalon]

+Cc some AMD maintainers, they can have an idea about IOMMU settings.


20/07/2023 14:44, Igor de Paula:
> I have enabled it in the host and in the BIOS for AMD...
> In the Bios I changed to amd_iommu=on and in the host it's the same for
> either.
> 
> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> 
> > 20/07/2023 11:35, Igor de Paula:
> > > The weird thing is that it only happens when I am using a host with an
> > AMD
> > > processor. It doesn't happen when I use a host with an Intel processor.
> >
> > So it's probably a matter of BIOS settings for the IOMMU?
> >
> >
> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon <thomas@monjalon.net>
> > > wrote:
> > >
> > > > +Cc the vmxnet3 maintainer.
> > > >
> > > > Please Jochen, do you have an idea what's wrong below?
> > > >
> > > >
> > > > 20/07/2023 11:25, Igor de Paula:
> > > > > This is because it can't negotiate the IOMMU type with any port.
> > > > >
> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon <thomas@monjalon.net
> > >
> > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > The first error is "Cause: Error: number of ports must be even"
> > > > > >
> > > > > >
> > > > > > 03/05/2023 18:13, Igor de Paula:
> > > > > > > I am running a VM inside a VMWARE server (vSphere).
> > > > > > > My goal it to set up DPDK with two HW ports, and set up a
> > > > virtio_user to
> > > > > > > interact with the kernel stack.
> > > > > > > In another app I have it working but instead of virtio_user I am
> > > > running
> > > > > > > KNI, it works in IOVA-PA mode.
> > > > > > > I am looking to replace the KNI.
> > > > > > >
> > > > > > > When I try to set up virtio_user port as in the doc:
> > > > > > >
> > > > > >
> > > >
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> > > > > > > I get a error it can't run in PA mode.
> > > > > > >
> > > > > > >
> > > > > > > When I try to run as VA mode from a parameter, I get the
> > following
> > > > > > errors:
> > > > > > > EAL: lib.eal log level changed from info to debug
> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0
> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0
> > > > > > > EAL: Support maximum 128 logical core(s) by configuration.
> > > > > > > EAL: Detected 2 lcore(s)
> > > > > > > EAL: Detected 1 NUMA nodes
> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3'
> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21'
> > > > > > > EAL: Checking presence of .so 'librte_eal.so'
> > > > > > > EAL: Detected static linkage of DPDK
> > > > > > > EAL: Ask a virtual area of 0x7000 bytes
> > > > > > > EAL: Virtual area found at 0x100000000 (size = 0x7000)
> > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > > > > > > EAL: DPAA Bus not present. Skipping.
> > > > > > > EAL: VFIO PCI modules not loaded
> > > > > > > EAL: Selected IOVA mode 'VA'
> > > > > > > EAL: Probing VFIO support...
> > > > > > > EAL: IOMMU type 1 (Type 1) is supported
> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported
> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported
> > > > > > > EAL: VFIO support initialized
> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes
> > > > > > > EAL: Virtual area found at 0x100007000 (size = 0x5b000)
> > > > > > > EAL: Setting up physically contiguous memory...
> > > > > > > EAL: Setting maximum number of open files to 1048576
> > > > > > > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
> > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
> > > > > > hugepage_sz:1073741824
> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> > > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000)
> > > > > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB
> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > > > > > EAL: Virtual area found at 0x140000000 (size = 0x2000000000)
> > > > > > > EAL: VA reserved for memseg list at 0x140000000, size 2000000000
> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> > > > > > > EAL: Virtual area found at 0x2140000000 (size = 0x2000)
> > > > > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB
> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > > > > > EAL: Virtual area found at 0x2180000000 (size = 0x2000000000)
> > > > > > > EAL: VA reserved for memseg list at 0x2180000000, size 2000000000
> > > > > > > EAL: TSC frequency is ~2350000 KHz
> > > > > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0])
> > > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
> > > > > > > EAL: Trying to obtain current memory policy.
> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0
> > > > > > > EAL: Restoring previous memory policy: 0
> > > > > > > EAL: request: mp_malloc_sync
> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
> > > > > > initializing
> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
> > > > > > initializing
> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> > > > > > > EAL: Bus (pci) probe failed.
> > > > > > > EAL: lib.telemetry log level changed from disabled to warning
> > > > > > > EAL: Error - exiting with code: 1
> > > > > > >   Cause: Error: number of ports must be even
> > > > > > > EAL: request: mp_malloc_sync
> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > For some reason the HW ports won't setup. From what I understand
> > > > > > > net_vmxnet3 should work with VA mode.
> > > > > > > I enabled I/OMUU for the VM.
> > > > > > > The weird thing even when enabled, I still have the
> > > > > > > enable_unsafe_noiommu_mode flag on.
> > > > > > > And because it's on the this:
> > > > > > >
> > > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev);
> > > > > > >
> > > > > > > return PA mode, and it fails.
> > > > > > >
> > > > > > > When I disable it by modifying
> > > > > > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get
> > another
> > > > > > error.
> > > > > > > The error is that it doesn't find a suitable IOMMU type:
> > > > > > > Just putting the relevant message:
> > > > > > >
> > > > > > >
> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such
> > device)
> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type
> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such
> > device)
> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type
> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> > > > > > >
> > > > > > >
> > > > > > > I can see where it fails:
> > > > > > >
> > > > > > >
> > > > > > > const struct vfio_iommu_type *
> > > > > > > vfio_set_iommu_type(int vfio_container_fd)
> > > > > > > {
> > > > > > > unsigned idx;
> > > > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
> > > > > > > const struct vfio_iommu_type *t = &iommu_types[idx];
> > > > > > >
> > > > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
> > > > > > > t->type_id);
> > > > > > > if (!ret) {
> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> > > > > > > t->type_id, t->name);
> > > > > > > return t;
> > > > > > > }
> > > > > > > /* not an error, there may be more supported IOMMU types */
> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error "
> > > > > > > "%i (%s)\n", t->type_id, t->name, errno,
> > > > > > > strerror(errno));
> > > > > > > }
> > > > > > > /* if we didn't find a suitable IOMMU type, fail */
> > > > > > > return NULL;
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > > The ioctl returns -1.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I tried many things.
> > > > > > > Enabling/disabling IOMMU.
> > > > > > > Modifying the enable_unsafe_noiommu_mode flag.
> > > > > > > Running in PA mode.
> > > > > > > Any help will be much appreciated.
> >
> >
> >
> >
> >
> >
> 

Re: help

[Jochen Behrens]

[-- Attachment #1: Type: text/plain, Size: 10391 bytes --]

+Ronak from the ESX team


In our usage, we do set amd_iommu=off in the boot command line from grub. (Or intel_iommu=off for Intel processors.)

                Jochen

From: Thomas Monjalon <thomas@monjalon.net>
Date: Thursday, July 20, 2023 at 6:00 AM
To: Igor de Paula <igordptx@gmail.com>
Cc: users@dpdk.org <users@dpdk.org>, Jochen Behrens <jbehrens@vmware.com>, Nipun Gupta <nipun.gupta@amd.com>, Nikhil Agarwal <nikhil.agarwal@amd.com>, Ferruh Yigit <ferruh.yigit@amd.com>
Subject: Re: help
!! External Email

+Cc some AMD maintainers, they can have an idea about IOMMU settings.


20/07/2023 14:44, Igor de Paula:
> I have enabled it in the host and in the BIOS for AMD...
> In the Bios I changed to amd_iommu=on and in the host it's the same for
> either.
>
> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> > 20/07/2023 11:35, Igor de Paula:
> > > The weird thing is that it only happens when I am using a host with an
> > AMD
> > > processor. It doesn't happen when I use a host with an Intel processor.
> >
> > So it's probably a matter of BIOS settings for the IOMMU?
> >
> >
> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon <thomas@monjalon.net>
> > > wrote:
> > >
> > > > +Cc the vmxnet3 maintainer.
> > > >
> > > > Please Jochen, do you have an idea what's wrong below?
> > > >
> > > >
> > > > 20/07/2023 11:25, Igor de Paula:
> > > > > This is because it can't negotiate the IOMMU type with any port.
> > > > >
> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon <thomas@monjalon.net
> > >
> > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > The first error is "Cause: Error: number of ports must be even"
> > > > > >
> > > > > >
> > > > > > 03/05/2023 18:13, Igor de Paula:
> > > > > > > I am running a VM inside a VMWARE server (vSphere).
> > > > > > > My goal it to set up DPDK with two HW ports, and set up a
> > > > virtio_user to
> > > > > > > interact with the kernel stack.
> > > > > > > In another app I have it working but instead of virtio_user I am
> > > > running
> > > > > > > KNI, it works in IOVA-PA mode.
> > > > > > > I am looking to replace the KNI.
> > > > > > >
> > > > > > > When I try to set up virtio_user port as in the doc:
> > > > > > >
> > > > > >
> > > >
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Fhowto%2Fvirtio_user_as_exception_path.html%23virtio-user-as-exception-path&data=05%7C01%7Cjbehrens%40vmware.com%7Cfbb1f6df362e4537d62608db8921432f%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638254548229277606%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3NuWkbtQx2q8NJ2CzArNfuDjP%2FYIVv1kMc%2Bo4IiHlmM%3D&reserved=0<https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path>
> > > > > > > I get a error it can't run in PA mode.
> > > > > > >
> > > > > > >
> > > > > > > When I try to run as VA mode from a parameter, I get the
> > following
> > > > > > errors:
> > > > > > > EAL: lib.eal log level changed from info to debug
> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0
> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0
> > > > > > > EAL: Support maximum 128 logical core(s) by configuration.
> > > > > > > EAL: Detected 2 lcore(s)
> > > > > > > EAL: Detected 1 NUMA nodes
> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3'
> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21'
> > > > > > > EAL: Checking presence of .so 'librte_eal.so'
> > > > > > > EAL: Detected static linkage of DPDK
> > > > > > > EAL: Ask a virtual area of 0x7000 bytes
> > > > > > > EAL: Virtual area found at 0x100000000 (size = 0x7000)
> > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > > > > > > EAL: DPAA Bus not present. Skipping.
> > > > > > > EAL: VFIO PCI modules not loaded
> > > > > > > EAL: Selected IOVA mode 'VA'
> > > > > > > EAL: Probing VFIO support...
> > > > > > > EAL: IOMMU type 1 (Type 1) is supported
> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported
> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported
> > > > > > > EAL: VFIO support initialized
> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes
> > > > > > > EAL: Virtual area found at 0x100007000 (size = 0x5b000)
> > > > > > > EAL: Setting up physically contiguous memory...
> > > > > > > EAL: Setting maximum number of open files to 1048576
> > > > > > > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
> > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
> > > > > > hugepage_sz:1073741824
> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> > > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000)
> > > > > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB
> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > > > > > EAL: Virtual area found at 0x140000000 (size = 0x2000000000)
> > > > > > > EAL: VA reserved for memseg list at 0x140000000, size 2000000000
> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> > > > > > > EAL: Virtual area found at 0x2140000000 (size = 0x2000)
> > > > > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB
> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > > > > > EAL: Virtual area found at 0x2180000000 (size = 0x2000000000)
> > > > > > > EAL: VA reserved for memseg list at 0x2180000000, size 2000000000
> > > > > > > EAL: TSC frequency is ~2350000 KHz
> > > > > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0])
> > > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
> > > > > > > EAL: Trying to obtain current memory policy.
> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0
> > > > > > > EAL: Restoring previous memory policy: 0
> > > > > > > EAL: request: mp_malloc_sync
> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
> > > > > > initializing
> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
> > > > > > initializing
> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> > > > > > > EAL: Bus (pci) probe failed.
> > > > > > > EAL: lib.telemetry log level changed from disabled to warning
> > > > > > > EAL: Error - exiting with code: 1
> > > > > > >   Cause: Error: number of ports must be even
> > > > > > > EAL: request: mp_malloc_sync
> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > For some reason the HW ports won't setup. From what I understand
> > > > > > > net_vmxnet3 should work with VA mode.
> > > > > > > I enabled I/OMUU for the VM.
> > > > > > > The weird thing even when enabled, I still have the
> > > > > > > enable_unsafe_noiommu_mode flag on.
> > > > > > > And because it's on the this:
> > > > > > >
> > > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev);
> > > > > > >
> > > > > > > return PA mode, and it fails.
> > > > > > >
> > > > > > > When I disable it by modifying
> > > > > > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get
> > another
> > > > > > error.
> > > > > > > The error is that it doesn't find a suitable IOMMU type:
> > > > > > > Just putting the relevant message:
> > > > > > >
> > > > > > >
> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such
> > device)
> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type
> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such
> > device)
> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type
> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> > > > > > >
> > > > > > >
> > > > > > > I can see where it fails:
> > > > > > >
> > > > > > >
> > > > > > > const struct vfio_iommu_type *
> > > > > > > vfio_set_iommu_type(int vfio_container_fd)
> > > > > > > {
> > > > > > > unsigned idx;
> > > > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
> > > > > > > const struct vfio_iommu_type *t = &iommu_types[idx];
> > > > > > >
> > > > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
> > > > > > > t->type_id);
> > > > > > > if (!ret) {
> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> > > > > > > t->type_id, t->name);
> > > > > > > return t;
> > > > > > > }
> > > > > > > /* not an error, there may be more supported IOMMU types */
> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error "
> > > > > > > "%i (%s)\n", t->type_id, t->name, errno,
> > > > > > > strerror(errno));
> > > > > > > }
> > > > > > > /* if we didn't find a suitable IOMMU type, fail */
> > > > > > > return NULL;
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > > The ioctl returns -1.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I tried many things.
> > > > > > > Enabling/disabling IOMMU.
> > > > > > > Modifying the enable_unsafe_noiommu_mode flag.
> > > > > > > Running in PA mode.
> > > > > > > Any help will be much appreciated.
> >
> >
> >
> >
> >
> >
>






!! External Email: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender.

[-- Attachment #2: Type: text/html, Size: 17424 bytes --]

Re: help

[Ferruh Yigit]

Hi Igor,

VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you
please try with:
- enable 'enable_unsafe_noiommu_mode' flag
- Force PA mode via '--iova-mode=pa' eal parameter


Also to be able to figure out AMD IOMMU support level, can you please
provide:
- AMD part number
- ESXi/hypervisor version
- ESXi & VM kernel logs


Thanks,
Ferruh

On 7/20/2023 5:21 PM, Jochen Behrens wrote:
> +Ronak from the ESX team
> 
>  
> 
>  
> 
> In our usage, we do set amd_iommu=off in the boot command line from
> grub. (Or intel_iommu=off for Intel processors.)
> 
>  
> 
>                 Jochen
> 
>  
> 
> *From: *Thomas Monjalon <thomas@monjalon.net>
> *Date: *Thursday, July 20, 2023 at 6:00 AM
> *To: *Igor de Paula <igordptx@gmail.com>
> *Cc: *users@dpdk.org <users@dpdk.org>, Jochen Behrens
> <jbehrens@vmware.com>, Nipun Gupta <nipun.gupta@amd.com>, Nikhil Agarwal
> <nikhil.agarwal@amd.com>, Ferruh Yigit <ferruh.yigit@amd.com>
> *Subject: *Re: help
> 
> !! External Email
> 
> +Cc some AMD maintainers, they can have an idea about IOMMU settings.
> 
> 
> 20/07/2023 14:44, Igor de Paula:
>> I have enabled it in the host and in the BIOS for AMD...
>> In the Bios I changed to amd_iommu=on and in the host it's the same for
>> either.
>>
>> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>>
>> > 20/07/2023 11:35, Igor de Paula:
>> > > The weird thing is that it only happens when I am using a host with an
>> > AMD
>> > > processor. It doesn't happen when I use a host with an Intel processor.
>> >
>> > So it's probably a matter of BIOS settings for the IOMMU?
>> >
>> >
>> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon <thomas@monjalon.net>
>> > > wrote:
>> > >
>> > > > +Cc the vmxnet3 maintainer.
>> > > >
>> > > > Please Jochen, do you have an idea what's wrong below?
>> > > >
>> > > >
>> > > > 20/07/2023 11:25, Igor de Paula:
>> > > > > This is because it can't negotiate the IOMMU type with any port.
>> > > > >
>> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon <thomas@monjalon.net
>> > >
>> > > > wrote:
>> > > > >
>> > > > > > Hello,
>> > > > > >
>> > > > > > The first error is "Cause: Error: number of ports must be even"
>> > > > > >
>> > > > > >
>> > > > > > 03/05/2023 18:13, Igor de Paula:
>> > > > > > > I am running a VM inside a VMWARE server (vSphere).
>> > > > > > > My goal it to set up DPDK with two HW ports, and set up a
>> > > > virtio_user to
>> > > > > > > interact with the kernel stack.
>> > > > > > > In another app I have it working but instead of virtio_user I am
>> > > > running
>> > > > > > > KNI, it works in IOVA-PA mode.
>> > > > > > > I am looking to replace the KNI.
>> > > > > > >
>> > > > > > > When I try to set up virtio_user port as in the doc:
>> > > > > > >
>> > > > > >
>> > > >
>> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path>
>> > > > > > > I get a error it can't run in PA mode.
>> > > > > > >
>> > > > > > >
>> > > > > > > When I try to run as VA mode from a parameter, I get the
>> > following
>> > > > > > errors:
>> > > > > > > EAL: lib.eal log level changed from info to debug
>> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0
>> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0
>> > > > > > > EAL: Support maximum 128 logical core(s) by configuration.
>> > > > > > > EAL: Detected 2 lcore(s)
>> > > > > > > EAL: Detected 1 NUMA nodes
>> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3'
>> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21'
>> > > > > > > EAL: Checking presence of .so 'librte_eal.so'
>> > > > > > > EAL: Detected static linkage of DPDK
>> > > > > > > EAL: Ask a virtual area of 0x7000 bytes
>> > > > > > > EAL: Virtual area found at 0x100000000 (size = 0x7000)
>> > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> > > > > > > EAL: DPAA Bus not present. Skipping.
>> > > > > > > EAL: VFIO PCI modules not loaded
>> > > > > > > EAL: Selected IOVA mode 'VA'
>> > > > > > > EAL: Probing VFIO support...
>> > > > > > > EAL: IOMMU type 1 (Type 1) is supported
>> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported
>> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported
>> > > > > > > EAL: VFIO support initialized
>> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes
>> > > > > > > EAL: Virtual area found at 0x100007000 (size = 0x5b000)
>> > > > > > > EAL: Setting up physically contiguous memory...
>> > > > > > > EAL: Setting maximum number of open files to 1048576
>> > > > > > > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
>> > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
>> > > > > > hugepage_sz:1073741824
>> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
>> > > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000)
>> > > > > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB
>> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
>> > > > > > > EAL: Virtual area found at 0x140000000 (size = 0x2000000000)
>> > > > > > > EAL: VA reserved for memseg list at 0x140000000, size 2000000000
>> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
>> > > > > > > EAL: Virtual area found at 0x2140000000 (size = 0x2000)
>> > > > > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB
>> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
>> > > > > > > EAL: Virtual area found at 0x2180000000 (size = 0x2000000000)
>> > > > > > > EAL: VA reserved for memseg list at 0x2180000000, size 2000000000
>> > > > > > > EAL: TSC frequency is ~2350000 KHz
>> > > > > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0])
>> > > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
>> > > > > > > EAL: Trying to obtain current memory policy.
>> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0
>> > > > > > > EAL: Restoring previous memory policy: 0
>> > > > > > > EAL: request: mp_malloc_sync
>> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
>> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
>> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
>> > > > > > initializing
>> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
>> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
>> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
>> > > > > > initializing
>> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
>> > > > > > > EAL: Bus (pci) probe failed.
>> > > > > > > EAL: lib.telemetry log level changed from disabled to warning
>> > > > > > > EAL: Error - exiting with code: 1
>> > > > > > >   Cause: Error: number of ports must be even
>> > > > > > > EAL: request: mp_malloc_sync
>> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > For some reason the HW ports won't setup. From what I understand
>> > > > > > > net_vmxnet3 should work with VA mode.
>> > > > > > > I enabled I/OMUU for the VM.
>> > > > > > > The weird thing even when enabled, I still have the
>> > > > > > > enable_unsafe_noiommu_mode flag on.
>> > > > > > > And because it's on the this:
>> > > > > > >
>> > > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev);
>> > > > > > >
>> > > > > > > return PA mode, and it fails.
>> > > > > > >
>> > > > > > > When I disable it by modifying
>> > > > > > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get
>> > another
>> > > > > > error.
>> > > > > > > The error is that it doesn't find a suitable IOMMU type:
>> > > > > > > Just putting the relevant message:
>> > > > > > >
>> > > > > > >
>> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
>> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
>> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
>> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
>> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such
>> > device)
>> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type
>> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
>> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
>> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
>> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
>> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such
>> > device)
>> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type
>> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
>> > > > > > >
>> > > > > > >
>> > > > > > > I can see where it fails:
>> > > > > > >
>> > > > > > >
>> > > > > > > const struct vfio_iommu_type *
>> > > > > > > vfio_set_iommu_type(int vfio_container_fd)
>> > > > > > > {
>> > > > > > > unsigned idx;
>> > > > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
>> > > > > > > const struct vfio_iommu_type *t = &iommu_types[idx];
>> > > > > > >
>> > > > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
>> > > > > > > t->type_id);
>> > > > > > > if (!ret) {
>> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
>> > > > > > > t->type_id, t->name);
>> > > > > > > return t;
>> > > > > > > }
>> > > > > > > /* not an error, there may be more supported IOMMU types */
>> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error "
>> > > > > > > "%i (%s)\n", t->type_id, t->name, errno,
>> > > > > > > strerror(errno));
>> > > > > > > }
>> > > > > > > /* if we didn't find a suitable IOMMU type, fail */
>> > > > > > > return NULL;
>> > > > > > > }
>> > > > > > >
>> > > > > > >
>> > > > > > > The ioctl returns -1.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > I tried many things.
>> > > > > > > Enabling/disabling IOMMU.
>> > > > > > > Modifying the enable_unsafe_noiommu_mode flag.
>> > > > > > > Running in PA mode.
>> > > > > > > Any help will be much appreciated.
>> >
>> >
>> >
>> >
>> >
>> >
>>
> 
> 
> 
> 
> 
> 
> !! External Email: This email originated from outside of the
> organization. Do not click links or open attachments unless you
> recognize the sender.
> 

Re: help

[Igor de Paula]

[-- Attachment #1: Type: text/plain, Size: 12050 bytes --]

I am trying to use virtio_user for an interface with the kernel:
https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html
I think this requires IOVA as va. It does work with Intel host and IOMMU
enabled. Part of the negotiation when setting it up is getting the IOMMU
number so I thought it has to have IOMMU.
I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but again,
that didn't work.
ESXI version - VMware ESXi, 7.0.0, 16324942
AMD:  AMD EPYC 7452 32-Core Processo

On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
Regarding the logs I will try and attach it soon.
Thanks

On Fri, Jul 21, 2023 at 12:21 PM Ferruh Yigit <ferruh.yigit@amd.com> wrote:

> Hi Igor,
>
> VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you
> please try with:
> - enable 'enable_unsafe_noiommu_mode' flag
> - Force PA mode via '--iova-mode=pa' eal parameter
>
>
> Also to be able to figure out AMD IOMMU support level, can you please
> provide:
> - AMD part number
> - ESXi/hypervisor version
> - ESXi & VM kernel logs
>
>
> Thanks,
> Ferruh
>
> On 7/20/2023 5:21 PM, Jochen Behrens wrote:
> > +Ronak from the ESX team
> >
> >
> >
> >
> >
> > In our usage, we do set amd_iommu=off in the boot command line from
> > grub. (Or intel_iommu=off for Intel processors.)
> >
> >
> >
> >                 Jochen
> >
> >
> >
> > *From: *Thomas Monjalon <thomas@monjalon.net>
> > *Date: *Thursday, July 20, 2023 at 6:00 AM
> > *To: *Igor de Paula <igordptx@gmail.com>
> > *Cc: *users@dpdk.org <users@dpdk.org>, Jochen Behrens
> > <jbehrens@vmware.com>, Nipun Gupta <nipun.gupta@amd.com>, Nikhil Agarwal
> > <nikhil.agarwal@amd.com>, Ferruh Yigit <ferruh.yigit@amd.com>
> > *Subject: *Re: help
> >
> > !! External Email
> >
> > +Cc some AMD maintainers, they can have an idea about IOMMU settings.
> >
> >
> > 20/07/2023 14:44, Igor de Paula:
> >> I have enabled it in the host and in the BIOS for AMD...
> >> In the Bios I changed to amd_iommu=on and in the host it's the same for
> >> either.
> >>
> >> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon <thomas@monjalon.net>
> wrote:
> >>
> >> > 20/07/2023 11:35, Igor de Paula:
> >> > > The weird thing is that it only happens when I am using a host with
> an
> >> > AMD
> >> > > processor. It doesn't happen when I use a host with an Intel
> processor.
> >> >
> >> > So it's probably a matter of BIOS settings for the IOMMU?
> >> >
> >> >
> >> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon <
> thomas@monjalon.net>
> >> > > wrote:
> >> > >
> >> > > > +Cc the vmxnet3 maintainer.
> >> > > >
> >> > > > Please Jochen, do you have an idea what's wrong below?
> >> > > >
> >> > > >
> >> > > > 20/07/2023 11:25, Igor de Paula:
> >> > > > > This is because it can't negotiate the IOMMU type with any port.
> >> > > > >
> >> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon <
> thomas@monjalon.net
> >> > >
> >> > > > wrote:
> >> > > > >
> >> > > > > > Hello,
> >> > > > > >
> >> > > > > > The first error is "Cause: Error: number of ports must be
> even"
> >> > > > > >
> >> > > > > >
> >> > > > > > 03/05/2023 18:13, Igor de Paula:
> >> > > > > > > I am running a VM inside a VMWARE server (vSphere).
> >> > > > > > > My goal it to set up DPDK with two HW ports, and set up a
> >> > > > virtio_user to
> >> > > > > > > interact with the kernel stack.
> >> > > > > > > In another app I have it working but instead of virtio_user
> I am
> >> > > > running
> >> > > > > > > KNI, it works in IOVA-PA mode.
> >> > > > > > > I am looking to replace the KNI.
> >> > > > > > >
> >> > > > > > > When I try to set up virtio_user port as in the doc:
> >> > > > > > >
> >> > > > > >
> >> > > >
> >> >
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> <
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> >
> >> > > > > > > I get a error it can't run in PA mode.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > When I try to run as VA mode from a parameter, I get the
> >> > following
> >> > > > > > errors:
> >> > > > > > > EAL: lib.eal log level changed from info to debug
> >> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0
> >> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0
> >> > > > > > > EAL: Support maximum 128 logical core(s) by configuration.
> >> > > > > > > EAL: Detected 2 lcore(s)
> >> > > > > > > EAL: Detected 1 NUMA nodes
> >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3'
> >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21'
> >> > > > > > > EAL: Checking presence of .so 'librte_eal.so'
> >> > > > > > > EAL: Detected static linkage of DPDK
> >> > > > > > > EAL: Ask a virtual area of 0x7000 bytes
> >> > > > > > > EAL: Virtual area found at 0x100000000 (size = 0x7000)
> >> > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> >> > > > > > > EAL: DPAA Bus not present. Skipping.
> >> > > > > > > EAL: VFIO PCI modules not loaded
> >> > > > > > > EAL: Selected IOVA mode 'VA'
> >> > > > > > > EAL: Probing VFIO support...
> >> > > > > > > EAL: IOMMU type 1 (Type 1) is supported
> >> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported
> >> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported
> >> > > > > > > EAL: VFIO support initialized
> >> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes
> >> > > > > > > EAL: Virtual area found at 0x100007000 (size = 0x5b000)
> >> > > > > > > EAL: Setting up physically contiguous memory...
> >> > > > > > > EAL: Setting maximum number of open files to 1048576
> >> > > > > > > EAL: Detected memory type: socket_id:0
> hugepage_sz:1073741824
> >> > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
> >> > > > > > hugepage_sz:1073741824
> >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> >> > > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000)
> >> > > > > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB
> >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> >> > > > > > > EAL: Virtual area found at 0x140000000 (size = 0x2000000000)
> >> > > > > > > EAL: VA reserved for memseg list at 0x140000000, size
> 2000000000
> >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> >> > > > > > > EAL: Virtual area found at 0x2140000000 (size = 0x2000)
> >> > > > > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB
> >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> >> > > > > > > EAL: Virtual area found at 0x2180000000 (size =
> 0x2000000000)
> >> > > > > > > EAL: VA reserved for memseg list at 0x2180000000, size
> 2000000000
> >> > > > > > > EAL: TSC frequency is ~2350000 KHz
> >> > > > > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0])
> >> > > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
> >> > > > > > > EAL: Trying to obtain current memory policy.
> >> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0
> >> > > > > > > EAL: Restoring previous memory policy: 0
> >> > > > > > > EAL: request: mp_malloc_sync
> >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA',
> not
> >> > > > > > initializing
> >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA',
> not
> >> > > > > > initializing
> >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> >> > > > > > > EAL: Bus (pci) probe failed.
> >> > > > > > > EAL: lib.telemetry log level changed from disabled to
> warning
> >> > > > > > > EAL: Error - exiting with code: 1
> >> > > > > > >   Cause: Error: number of ports must be even
> >> > > > > > > EAL: request: mp_malloc_sync
> >> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > For some reason the HW ports won't setup. From what I
> understand
> >> > > > > > > net_vmxnet3 should work with VA mode.
> >> > > > > > > I enabled I/OMUU for the VM.
> >> > > > > > > The weird thing even when enabled, I still have the
> >> > > > > > > enable_unsafe_noiommu_mode flag on.
> >> > > > > > > And because it's on the this:
> >> > > > > > >
> >> > > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev);
> >> > > > > > >
> >> > > > > > > return PA mode, and it fails.
> >> > > > > > >
> >> > > > > > > When I disable it by modifying
> >> > > > > > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I
> get
> >> > another
> >> > > > > > error.
> >> > > > > > > The error is that it doesn't find a suitable IOMMU type:
> >> > > > > > > Just putting the relevant message:
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such
> device)
> >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such
> device)
> >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such
> >> > device)
> >> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type
> >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such
> device)
> >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such
> device)
> >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such
> >> > device)
> >> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type
> >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > I can see where it fails:
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > const struct vfio_iommu_type *
> >> > > > > > > vfio_set_iommu_type(int vfio_container_fd)
> >> > > > > > > {
> >> > > > > > > unsigned idx;
> >> > > > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
> >> > > > > > > const struct vfio_iommu_type *t = &iommu_types[idx];
> >> > > > > > >
> >> > > > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
> >> > > > > > > t->type_id);
> >> > > > > > > if (!ret) {
> >> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> >> > > > > > > t->type_id, t->name);
> >> > > > > > > return t;
> >> > > > > > > }
> >> > > > > > > /* not an error, there may be more supported IOMMU types */
> >> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error "
> >> > > > > > > "%i (%s)\n", t->type_id, t->name, errno,
> >> > > > > > > strerror(errno));
> >> > > > > > > }
> >> > > > > > > /* if we didn't find a suitable IOMMU type, fail */
> >> > > > > > > return NULL;
> >> > > > > > > }
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > The ioctl returns -1.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > I tried many things.
> >> > > > > > > Enabling/disabling IOMMU.
> >> > > > > > > Modifying the enable_unsafe_noiommu_mode flag.
> >> > > > > > > Running in PA mode.
> >> > > > > > > Any help will be much appreciated.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >>
> >
> >
> >
> >
> >
> >
> > !! External Email: This email originated from outside of the
> > organization. Do not click links or open attachments unless you
> > recognize the sender.
> >
>
>

[-- Attachment #2: Type: text/html, Size: 19226 bytes --]

Re: help

[Ferruh Yigit]

On 7/21/2023 12:39 PM, Igor de Paula wrote:
> I am trying to use virtio_user for an interface with the
> kernel: https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html>
> I think this requires IOVA as va.
>

I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo,
he may know better.

Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and
'--iova-mode=pa'?


> It does work with Intel host and IOMMU
> enabled. Part of the negotiation when setting it up is getting the IOMMU
> number so I thought it has to have IOMMU.
>

Yes, issue looks like related to the IOMMU, and it may be either related
to HW support, or ESXi iommu driver support, we will check using below
information you provided.

> I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but
> again, that didn't work.
> ESXI version - VMware ESXi, 7.0.0, 16324942
> AMD:  AMD EPYC 7452 32-Core Processo
> 
> On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> Regarding the logs I will try and attach it soon.
> 

Thanks for info, waiting for logs.

> 
> On Fri, Jul 21, 2023 at 12:21 PM Ferruh Yigit <ferruh.yigit@amd.com
> <mailto:ferruh.yigit@amd.com>> wrote:
> 
>     Hi Igor,
> 
>     VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you
>     please try with:
>     - enable 'enable_unsafe_noiommu_mode' flag
>     - Force PA mode via '--iova-mode=pa' eal parameter
> 
> 
>     Also to be able to figure out AMD IOMMU support level, can you please
>     provide:
>     - AMD part number
>     - ESXi/hypervisor version
>     - ESXi & VM kernel logs
> 
> 
>     Thanks,
>     Ferruh
> 
>     On 7/20/2023 5:21 PM, Jochen Behrens wrote:
>     > +Ronak from the ESX team
>     >
>     >  
>     >
>     >  
>     >
>     > In our usage, we do set amd_iommu=off in the boot command line from
>     > grub. (Or intel_iommu=off for Intel processors.)
>     >
>     >  
>     >
>     >                 Jochen
>     >
>     >  
>     >
>     > *From: *Thomas Monjalon <thomas@monjalon.net
>     <mailto:thomas@monjalon.net>>
>     > *Date: *Thursday, July 20, 2023 at 6:00 AM
>     > *To: *Igor de Paula <igordptx@gmail.com <mailto:igordptx@gmail.com>>
>     > *Cc: *users@dpdk.org <mailto:users@dpdk.org> <users@dpdk.org
>     <mailto:users@dpdk.org>>, Jochen Behrens
>     > <jbehrens@vmware.com <mailto:jbehrens@vmware.com>>, Nipun Gupta
>     <nipun.gupta@amd.com <mailto:nipun.gupta@amd.com>>, Nikhil Agarwal
>     > <nikhil.agarwal@amd.com <mailto:nikhil.agarwal@amd.com>>, Ferruh
>     Yigit <ferruh.yigit@amd.com <mailto:ferruh.yigit@amd.com>>
>     > *Subject: *Re: help
>     >
>     > !! External Email
>     >
>     > +Cc some AMD maintainers, they can have an idea about IOMMU settings.
>     >
>     >
>     > 20/07/2023 14:44, Igor de Paula:
>     >> I have enabled it in the host and in the BIOS for AMD...
>     >> In the Bios I changed to amd_iommu=on and in the host it's the
>     same for
>     >> either.
>     >>
>     >> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon
>     <thomas@monjalon.net <mailto:thomas@monjalon.net>> wrote:
>     >>
>     >> > 20/07/2023 11:35, Igor de Paula:
>     >> > > The weird thing is that it only happens when I am using a
>     host with an
>     >> > AMD
>     >> > > processor. It doesn't happen when I use a host with an Intel
>     processor.
>     >> >
>     >> > So it's probably a matter of BIOS settings for the IOMMU?
>     >> >
>     >> >
>     >> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon
>     <thomas@monjalon.net <mailto:thomas@monjalon.net>>
>     >> > > wrote:
>     >> > >
>     >> > > > +Cc the vmxnet3 maintainer.
>     >> > > >
>     >> > > > Please Jochen, do you have an idea what's wrong below?
>     >> > > >
>     >> > > >
>     >> > > > 20/07/2023 11:25, Igor de Paula:
>     >> > > > > This is because it can't negotiate the IOMMU type with
>     any port.
>     >> > > > >
>     >> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon
>     <thomas@monjalon.net <mailto:thomas@monjalon.net>
>     >> > >
>     >> > > > wrote:
>     >> > > > >
>     >> > > > > > Hello,
>     >> > > > > >
>     >> > > > > > The first error is "Cause: Error: number of ports must
>     be even"
>     >> > > > > >
>     >> > > > > >
>     >> > > > > > 03/05/2023 18:13, Igor de Paula:
>     >> > > > > > > I am running a VM inside a VMWARE server (vSphere).
>     >> > > > > > > My goal it to set up DPDK with two HW ports, and set up a
>     >> > > > virtio_user to
>     >> > > > > > > interact with the kernel stack.
>     >> > > > > > > In another app I have it working but instead of
>     virtio_user I am
>     >> > > > running
>     >> > > > > > > KNI, it works in IOVA-PA mode.
>     >> > > > > > > I am looking to replace the KNI.
>     >> > > > > > >
>     >> > > > > > > When I try to set up virtio_user port as in the doc:
>     >> > > > > > >
>     >> > > > > >
>     >> > > >
>     >> >
>     https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path> <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path>>
>     >> > > > > > > I get a error it can't run in PA mode.
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > When I try to run as VA mode from a parameter, I get the
>     >> > following
>     >> > > > > > errors:
>     >> > > > > > > EAL: lib.eal log level changed from info to debug
>     >> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0
>     >> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0
>     >> > > > > > > EAL: Support maximum 128 logical core(s) by
>     configuration.
>     >> > > > > > > EAL: Detected 2 lcore(s)
>     >> > > > > > > EAL: Detected 1 NUMA nodes
>     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3'
>     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21'
>     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so'
>     >> > > > > > > EAL: Detected static linkage of DPDK
>     >> > > > > > > EAL: Ask a virtual area of 0x7000 bytes
>     >> > > > > > > EAL: Virtual area found at 0x100000000 (size = 0x7000)
>     >> > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>     >> > > > > > > EAL: DPAA Bus not present. Skipping.
>     >> > > > > > > EAL: VFIO PCI modules not loaded
>     >> > > > > > > EAL: Selected IOVA mode 'VA'
>     >> > > > > > > EAL: Probing VFIO support...
>     >> > > > > > > EAL: IOMMU type 1 (Type 1) is supported
>     >> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported
>     >> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported
>     >> > > > > > > EAL: VFIO support initialized
>     >> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes
>     >> > > > > > > EAL: Virtual area found at 0x100007000 (size = 0x5b000)
>     >> > > > > > > EAL: Setting up physically contiguous memory...
>     >> > > > > > > EAL: Setting maximum number of open files to 1048576
>     >> > > > > > > EAL: Detected memory type: socket_id:0
>     hugepage_sz:1073741824
>     >> > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
>     >> > > > > > hugepage_sz:1073741824
>     >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
>     >> > > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000)
>     >> > > > > > > EAL: Memseg list allocated at socket 0, page size
>     0x100000kB
>     >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
>     >> > > > > > > EAL: Virtual area found at 0x140000000 (size =
>     0x2000000000)
>     >> > > > > > > EAL: VA reserved for memseg list at 0x140000000, size
>     2000000000
>     >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
>     >> > > > > > > EAL: Virtual area found at 0x2140000000 (size = 0x2000)
>     >> > > > > > > EAL: Memseg list allocated at socket 0, page size
>     0x100000kB
>     >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
>     >> > > > > > > EAL: Virtual area found at 0x2180000000 (size =
>     0x2000000000)
>     >> > > > > > > EAL: VA reserved for memseg list at 0x2180000000,
>     size 2000000000
>     >> > > > > > > EAL: TSC frequency is ~2350000 KHz
>     >> > > > > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0])
>     >> > > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
>     >> > > > > > > EAL: Trying to obtain current memory policy.
>     >> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0
>     >> > > > > > > EAL: Restoring previous memory policy: 0
>     >> > > > > > > EAL: request: mp_malloc_sync
>     >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
>     >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
>     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>     >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is
>     'VA', not
>     >> > > > > > initializing
>     >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
>     >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
>     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>     >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is
>     'VA', not
>     >> > > > > > initializing
>     >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
>     >> > > > > > > EAL: Bus (pci) probe failed.
>     >> > > > > > > EAL: lib.telemetry log level changed from disabled to
>     warning
>     >> > > > > > > EAL: Error - exiting with code: 1
>     >> > > > > > >   Cause: Error: number of ports must be even
>     >> > > > > > > EAL: request: mp_malloc_sync
>     >> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > For some reason the HW ports won't setup. From what I
>     understand
>     >> > > > > > > net_vmxnet3 should work with VA mode.
>     >> > > > > > > I enabled I/OMUU for the VM.
>     >> > > > > > > The weird thing even when enabled, I still have the
>     >> > > > > > > enable_unsafe_noiommu_mode flag on.
>     >> > > > > > > And because it's on the this:
>     >> > > > > > >
>     >> > > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev);
>     >> > > > > > >
>     >> > > > > > > return PA mode, and it fails.
>     >> > > > > > >
>     >> > > > > > > When I disable it by modifying
>     >> > > > > > >
>     /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get
>     >> > another
>     >> > > > > > error.
>     >> > > > > > > The error is that it doesn't find a suitable IOMMU type:
>     >> > > > > > > Just putting the relevant message:
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
>     >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
>     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>     >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No
>     such device)
>     >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
>     such device)
>     >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
>     such
>     >> > device)
>     >> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type
>     >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
>     >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
>     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>     >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No
>     such device)
>     >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
>     such device)
>     >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
>     such
>     >> > device)
>     >> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type
>     >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > I can see where it fails:
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > const struct vfio_iommu_type *
>     >> > > > > > > vfio_set_iommu_type(int vfio_container_fd)
>     >> > > > > > > {
>     >> > > > > > > unsigned idx;
>     >> > > > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
>     >> > > > > > > const struct vfio_iommu_type *t = &iommu_types[idx];
>     >> > > > > > >
>     >> > > > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
>     >> > > > > > > t->type_id);
>     >> > > > > > > if (!ret) {
>     >> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
>     >> > > > > > > t->type_id, t->name);
>     >> > > > > > > return t;
>     >> > > > > > > }
>     >> > > > > > > /* not an error, there may be more supported IOMMU
>     types */
>     >> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed,
>     error "
>     >> > > > > > > "%i (%s)\n", t->type_id, t->name, errno,
>     >> > > > > > > strerror(errno));
>     >> > > > > > > }
>     >> > > > > > > /* if we didn't find a suitable IOMMU type, fail */
>     >> > > > > > > return NULL;
>     >> > > > > > > }
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > The ioctl returns -1.
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > I tried many things.
>     >> > > > > > > Enabling/disabling IOMMU.
>     >> > > > > > > Modifying the enable_unsafe_noiommu_mode flag.
>     >> > > > > > > Running in PA mode.
>     >> > > > > > > Any help will be much appreciated.
>     >> >
>     >> >
>     >> >
>     >> >
>     >> >
>     >> >
>     >>
>     >
>     >
>     >
>     >
>     >
>     >
>     > !! External Email: This email originated from outside of the
>     > organization. Do not click links or open attachments unless you
>     > recognize the sender.
>     >
> 

Re: help

[Igor de Paula]

[-- Attachment #1.1: Type: text/plain, Size: 15583 bytes --]

Hi,
Attaching the logs of EAL when trying to run a configuration with
virtio_user port when IOMMU is
disabled and enable_unsafe_iommu is enabled. As you can see it forces IOVA
as PA but the viritui_user needs IOVA as VA.
I am also attaching the output of dmesg. I am not sure which kernel logs
you wanted... if there is anything else please let me know..
Regarding the ESXI logs, they are HUGE so I will send to you on a separate
email.

On Fri, Jul 21, 2023 at 1:14 PM Ferruh Yigit <ferruh.yigit@amd.com> wrote:

> On 7/21/2023 12:39 PM, Igor de Paula wrote:
> > I am trying to use virtio_user for an interface with the
> > kernel:
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html <
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html>
> > I think this requires IOVA as va.
> >
>
> I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo,
> he may know better.
>
> Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and
> '--iova-mode=pa'?
>
>
> > It does work with Intel host and IOMMU
> > enabled. Part of the negotiation when setting it up is getting the IOMMU
> > number so I thought it has to have IOMMU.
> >
>
> Yes, issue looks like related to the IOMMU, and it may be either related
> to HW support, or ESXi iommu driver support, we will check using below
> information you provided.
>
> > I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but
> > again, that didn't work.
> > ESXI version - VMware ESXi, 7.0.0, 16324942
> > AMD:  AMD EPYC 7452 32-Core Processo
> >
> > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> > Regarding the logs I will try and attach it soon.
> >
>
> Thanks for info, waiting for logs.
>
> >
> > On Fri, Jul 21, 2023 at 12:21 PM Ferruh Yigit <ferruh.yigit@amd.com
> > <mailto:ferruh.yigit@amd.com>> wrote:
> >
> >     Hi Igor,
> >
> >     VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you
> >     please try with:
> >     - enable 'enable_unsafe_noiommu_mode' flag
> >     - Force PA mode via '--iova-mode=pa' eal parameter
> >
> >
> >     Also to be able to figure out AMD IOMMU support level, can you please
> >     provide:
> >     - AMD part number
> >     - ESXi/hypervisor version
> >     - ESXi & VM kernel logs
> >
> >
> >     Thanks,
> >     Ferruh
> >
> >     On 7/20/2023 5:21 PM, Jochen Behrens wrote:
> >     > +Ronak from the ESX team
> >     >
> >     >
> >     >
> >     >
> >     >
> >     > In our usage, we do set amd_iommu=off in the boot command line from
> >     > grub. (Or intel_iommu=off for Intel processors.)
> >     >
> >     >
> >     >
> >     >                 Jochen
> >     >
> >     >
> >     >
> >     > *From: *Thomas Monjalon <thomas@monjalon.net
> >     <mailto:thomas@monjalon.net>>
> >     > *Date: *Thursday, July 20, 2023 at 6:00 AM
> >     > *To: *Igor de Paula <igordptx@gmail.com <mailto:igordptx@gmail.com
> >>
> >     > *Cc: *users@dpdk.org <mailto:users@dpdk.org> <users@dpdk.org
> >     <mailto:users@dpdk.org>>, Jochen Behrens
> >     > <jbehrens@vmware.com <mailto:jbehrens@vmware.com>>, Nipun Gupta
> >     <nipun.gupta@amd.com <mailto:nipun.gupta@amd.com>>, Nikhil Agarwal
> >     > <nikhil.agarwal@amd.com <mailto:nikhil.agarwal@amd.com>>, Ferruh
> >     Yigit <ferruh.yigit@amd.com <mailto:ferruh.yigit@amd.com>>
> >     > *Subject: *Re: help
> >     >
> >     > !! External Email
> >     >
> >     > +Cc some AMD maintainers, they can have an idea about IOMMU
> settings.
> >     >
> >     >
> >     > 20/07/2023 14:44, Igor de Paula:
> >     >> I have enabled it in the host and in the BIOS for AMD...
> >     >> In the Bios I changed to amd_iommu=on and in the host it's the
> >     same for
> >     >> either.
> >     >>
> >     >> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon
> >     <thomas@monjalon.net <mailto:thomas@monjalon.net>> wrote:
> >     >>
> >     >> > 20/07/2023 11:35, Igor de Paula:
> >     >> > > The weird thing is that it only happens when I am using a
> >     host with an
> >     >> > AMD
> >     >> > > processor. It doesn't happen when I use a host with an Intel
> >     processor.
> >     >> >
> >     >> > So it's probably a matter of BIOS settings for the IOMMU?
> >     >> >
> >     >> >
> >     >> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon
> >     <thomas@monjalon.net <mailto:thomas@monjalon.net>>
> >     >> > > wrote:
> >     >> > >
> >     >> > > > +Cc the vmxnet3 maintainer.
> >     >> > > >
> >     >> > > > Please Jochen, do you have an idea what's wrong below?
> >     >> > > >
> >     >> > > >
> >     >> > > > 20/07/2023 11:25, Igor de Paula:
> >     >> > > > > This is because it can't negotiate the IOMMU type with
> >     any port.
> >     >> > > > >
> >     >> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon
> >     <thomas@monjalon.net <mailto:thomas@monjalon.net>
> >     >> > >
> >     >> > > > wrote:
> >     >> > > > >
> >     >> > > > > > Hello,
> >     >> > > > > >
> >     >> > > > > > The first error is "Cause: Error: number of ports must
> >     be even"
> >     >> > > > > >
> >     >> > > > > >
> >     >> > > > > > 03/05/2023 18:13, Igor de Paula:
> >     >> > > > > > > I am running a VM inside a VMWARE server (vSphere).
> >     >> > > > > > > My goal it to set up DPDK with two HW ports, and set
> up a
> >     >> > > > virtio_user to
> >     >> > > > > > > interact with the kernel stack.
> >     >> > > > > > > In another app I have it working but instead of
> >     virtio_user I am
> >     >> > > > running
> >     >> > > > > > > KNI, it works in IOVA-PA mode.
> >     >> > > > > > > I am looking to replace the KNI.
> >     >> > > > > > >
> >     >> > > > > > > When I try to set up virtio_user port as in the doc:
> >     >> > > > > > >
> >     >> > > > > >
> >     >> > > >
> >     >> >
> >
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> <
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path>
> <
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> <
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> >>
> >     >> > > > > > > I get a error it can't run in PA mode.
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > When I try to run as VA mode from a parameter, I get
> the
> >     >> > following
> >     >> > > > > > errors:
> >     >> > > > > > > EAL: lib.eal log level changed from info to debug
> >     >> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0
> >     >> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0
> >     >> > > > > > > EAL: Support maximum 128 logical core(s) by
> >     configuration.
> >     >> > > > > > > EAL: Detected 2 lcore(s)
> >     >> > > > > > > EAL: Detected 1 NUMA nodes
> >     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3'
> >     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21'
> >     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so'
> >     >> > > > > > > EAL: Detected static linkage of DPDK
> >     >> > > > > > > EAL: Ask a virtual area of 0x7000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x100000000 (size = 0x7000)
> >     >> > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> >     >> > > > > > > EAL: DPAA Bus not present. Skipping.
> >     >> > > > > > > EAL: VFIO PCI modules not loaded
> >     >> > > > > > > EAL: Selected IOVA mode 'VA'
> >     >> > > > > > > EAL: Probing VFIO support...
> >     >> > > > > > > EAL: IOMMU type 1 (Type 1) is supported
> >     >> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported
> >     >> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported
> >     >> > > > > > > EAL: VFIO support initialized
> >     >> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x100007000 (size =
> 0x5b000)
> >     >> > > > > > > EAL: Setting up physically contiguous memory...
> >     >> > > > > > > EAL: Setting maximum number of open files to 1048576
> >     >> > > > > > > EAL: Detected memory type: socket_id:0
> >     hugepage_sz:1073741824
> >     >> > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
> >     >> > > > > > hugepage_sz:1073741824
> >     >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000)
> >     >> > > > > > > EAL: Memseg list allocated at socket 0, page size
> >     0x100000kB
> >     >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x140000000 (size =
> >     0x2000000000)
> >     >> > > > > > > EAL: VA reserved for memseg list at 0x140000000, size
> >     2000000000
> >     >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x2140000000 (size =
> 0x2000)
> >     >> > > > > > > EAL: Memseg list allocated at socket 0, page size
> >     0x100000kB
> >     >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x2180000000 (size =
> >     0x2000000000)
> >     >> > > > > > > EAL: VA reserved for memseg list at 0x2180000000,
> >     size 2000000000
> >     >> > > > > > > EAL: TSC frequency is ~2350000 KHz
> >     >> > > > > > > EAL: Main lcore 0 is ready
> (tid=7f8ad790ec00;cpuset=[0])
> >     >> > > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
> >     >> > > > > > > EAL: Trying to obtain current memory policy.
> >     >> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0
> >     >> > > > > > > EAL: Restoring previous memory policy: 0
> >     >> > > > > > > EAL: request: mp_malloc_sync
> >     >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> >     >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >     >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is
> >     'VA', not
> >     >> > > > > > initializing
> >     >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> >     >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >     >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is
> >     'VA', not
> >     >> > > > > > initializing
> >     >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> >     >> > > > > > > EAL: Bus (pci) probe failed.
> >     >> > > > > > > EAL: lib.telemetry log level changed from disabled to
> >     warning
> >     >> > > > > > > EAL: Error - exiting with code: 1
> >     >> > > > > > >   Cause: Error: number of ports must be even
> >     >> > > > > > > EAL: request: mp_malloc_sync
> >     >> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > For some reason the HW ports won't setup. From what I
> >     understand
> >     >> > > > > > > net_vmxnet3 should work with VA mode.
> >     >> > > > > > > I enabled I/OMUU for the VM.
> >     >> > > > > > > The weird thing even when enabled, I still have the
> >     >> > > > > > > enable_unsafe_noiommu_mode flag on.
> >     >> > > > > > > And because it's on the this:
> >     >> > > > > > >
> >     >> > > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev);
> >     >> > > > > > >
> >     >> > > > > > > return PA mode, and it fails.
> >     >> > > > > > >
> >     >> > > > > > > When I disable it by modifying
> >     >> > > > > > >
> >     /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get
> >     >> > another
> >     >> > > > > > error.
> >     >> > > > > > > The error is that it doesn't find a suitable IOMMU
> type:
> >     >> > > > > > > Just putting the relevant message:
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> >     >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >     >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No
> >     such device)
> >     >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
> >     such device)
> >     >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
> >     such
> >     >> > device)
> >     >> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type
> >     >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> >     >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >     >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No
> >     such device)
> >     >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
> >     such device)
> >     >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
> >     such
> >     >> > device)
> >     >> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type
> >     >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > I can see where it fails:
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > const struct vfio_iommu_type *
> >     >> > > > > > > vfio_set_iommu_type(int vfio_container_fd)
> >     >> > > > > > > {
> >     >> > > > > > > unsigned idx;
> >     >> > > > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
> >     >> > > > > > > const struct vfio_iommu_type *t = &iommu_types[idx];
> >     >> > > > > > >
> >     >> > > > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
> >     >> > > > > > > t->type_id);
> >     >> > > > > > > if (!ret) {
> >     >> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> >     >> > > > > > > t->type_id, t->name);
> >     >> > > > > > > return t;
> >     >> > > > > > > }
> >     >> > > > > > > /* not an error, there may be more supported IOMMU
> >     types */
> >     >> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed,
> >     error "
> >     >> > > > > > > "%i (%s)\n", t->type_id, t->name, errno,
> >     >> > > > > > > strerror(errno));
> >     >> > > > > > > }
> >     >> > > > > > > /* if we didn't find a suitable IOMMU type, fail */
> >     >> > > > > > > return NULL;
> >     >> > > > > > > }
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > The ioctl returns -1.
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > I tried many things.
> >     >> > > > > > > Enabling/disabling IOMMU.
> >     >> > > > > > > Modifying the enable_unsafe_noiommu_mode flag.
> >     >> > > > > > > Running in PA mode.
> >     >> > > > > > > Any help will be much appreciated.
> >     >> >
> >     >> >
> >     >> >
> >     >> >
> >     >> >
> >     >> >
> >     >>
> >     >
> >     >
> >     >
> >     >
> >     >
> >     >
> >     > !! External Email: This email originated from outside of the
> >     > organization. Do not click links or open attachments unless you
> >     > recognize the sender.
> >     >
> >
>
>

[-- Attachment #1.2: Type: text/html, Size: 25293 bytes --]

[-- Attachment #2: kernel2.log --]
[-- Type: application/octet-stream, Size: 104400 bytes --]

[    0.000000] Linux version 5.4.0-139-generic (buildd@lcy02-amd64-112) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)) #156-Ubuntu SMP Fri Jan 20 17:27:18 UTC 2023 (Ubuntu 5.4.0-139.156-generic 5.4.224)
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-5.4.0-139-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro splash quiet isolcpus=1-1 nohz_full=1-1 rcu_nocbs=1-1 default_hugepagesz=1G hugepagesz=1G hugepages=12 nmi_watchdog=0 net.ifnames=0 biosdevname=0
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Hygon HygonGenuine
[    0.000000]   Centaur CentaurHauls
[    0.000000]   zhaoxin   Shanghai  
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'compacted' format.
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009f3ff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009f400-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000dc000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bfedffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bfee0000-0x00000000bfefefff] ACPI data
[    0.000000] BIOS-e820: [mem 0x00000000bfeff000-0x00000000bfefffff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x00000000bff00000-0x00000000bfffffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000f0000000-0x00000000f7ffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec0ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffe0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000043fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.7 present.
[    0.000000] DMI: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/09/2019
[    0.000000] vmware: hypercall mode: 0x00
[    0.000000] Hypervisor detected: VMware
[    0.000000] vmware: TSC freq read from hypervisor : 2350.000 MHz
[    0.000000] vmware: Host bus clock speed read from hypervisor : 66000000 Hz
[    0.000000] vmware: using sched offset of 2848245640 ns
[    0.000001] tsc: Detected 2350.000 MHz processor
[    0.002636] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.002637] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.002641] last_pfn = 0x440000 max_arch_pfn = 0x400000000
[    0.002659] MTRR default type: uncachable
[    0.002660] MTRR fixed ranges enabled:
[    0.002661]   00000-9FFFF write-back
[    0.002662]   A0000-BFFFF uncachable
[    0.002662]   C0000-CFFFF write-protect
[    0.002663]   D0000-EFFFF uncachable
[    0.002663]   F0000-FFFFF write-protect
[    0.002664] MTRR variable ranges enabled:
[    0.002665]   0 base 00000000000 mask 7F800000000 write-back
[    0.002666]   1 base 000C0000000 mask 7FFC0000000 uncachable
[    0.002666]   2 disabled
[    0.002666]   3 disabled
[    0.002667]   4 disabled
[    0.002667]   5 disabled
[    0.002667]   6 disabled
[    0.002668]   7 disabled
[    0.002687] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
[    0.002699] total RAM covered: 31744M
[    0.003112] Found optimal setting for mtrr clean up
[    0.003113]  gran_size: 64K 	chunk_size: 64K 	num_reg: 5  	lose cover RAM: 0G
[    0.003150] e820: update [mem 0xc0000000-0xffffffff] usable ==> reserved
[    0.003155] last_pfn = 0xc0000 max_arch_pfn = 0x400000000
[    0.005763] found SMP MP-table at [mem 0x000f6a70-0x000f6a7f]
[    0.005859] check: Scanning 1 areas for low memory corruption
[    0.005878] Using GB pages for direct mapping
[    0.006099] RAMDISK: [mem 0x2db1b000-0x32d84fff]
[    0.006105] ACPI: Early table checksum verification disabled
[    0.006108] ACPI: RSDP 0x00000000000F6A00 000024 (v02 PTLTD )
[    0.006111] ACPI: XSDT 0x00000000BFEEEA0F 00005C (v01 INTEL  440BX    06040000 VMW  01324272)
[    0.006116] ACPI: FACP 0x00000000BFEFEE73 0000F4 (v04 INTEL  440BX    06040000 PTL  000F4240)
[    0.006119] ACPI: DSDT 0x00000000BFEEEC95 0101DE (v01 PTLTD  Custom   06040000 MSFT 03000001)
[    0.006125] ACPI: FACS 0x00000000BFEFFFC0 000040
[    0.006126] ACPI: FACS 0x00000000BFEFFFC0 000040
[    0.006128] ACPI: BOOT 0x00000000BFEEEC6D 000028 (v01 PTLTD  $SBFTBL$ 06040000  LTP 00000001)
[    0.006130] ACPI: APIC 0x00000000BFEEEC0F 00005E (v01 PTLTD  ? APIC   06040000  LTP 00000000)
[    0.006132] ACPI: MCFG 0x00000000BFEEEBD3 00003C (v01 PTLTD  $PCITBL$ 06040000  LTP 00000001)
[    0.006135] ACPI: SRAT 0x00000000BFEEEB0B 0000C8 (v02 VMWARE MEMPLUG  06040000 VMW  00000001)
[    0.006137] ACPI: HPET 0x00000000BFEEEAD3 000038 (v01 VMWARE VMW HPET 06040000 VMW  00000001)
[    0.006139] ACPI: WAET 0x00000000BFEEEAAB 000028 (v01 VMWARE VMW WAET 06040000 VMW  00000001)
[    0.006141] ACPI: Reserving FACP table memory at [mem 0xbfefee73-0xbfefef66]
[    0.006141] ACPI: Reserving DSDT table memory at [mem 0xbfeeec95-0xbfefee72]
[    0.006142] ACPI: Reserving FACS table memory at [mem 0xbfefffc0-0xbfefffff]
[    0.006143] ACPI: Reserving FACS table memory at [mem 0xbfefffc0-0xbfefffff]
[    0.006143] ACPI: Reserving BOOT table memory at [mem 0xbfeeec6d-0xbfeeec94]
[    0.006143] ACPI: Reserving APIC table memory at [mem 0xbfeeec0f-0xbfeeec6c]
[    0.006144] ACPI: Reserving MCFG table memory at [mem 0xbfeeebd3-0xbfeeec0e]
[    0.006144] ACPI: Reserving SRAT table memory at [mem 0xbfeeeb0b-0xbfeeebd2]
[    0.006145] ACPI: Reserving HPET table memory at [mem 0xbfeeead3-0xbfeeeb0a]
[    0.006145] ACPI: Reserving WAET table memory at [mem 0xbfeeeaab-0xbfeeead2]
[    0.006162] ACPI: Local APIC address 0xfee00000
[    0.006202] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[    0.006202] SRAT: PXM 0 -> APIC 0x02 -> Node 0
[    0.006204] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff]
[    0.006205] ACPI: SRAT: Node 0 PXM 0 [mem 0x00100000-0xbfffffff]
[    0.006205] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x43fffffff]
[    0.006207] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0xbfffffff] -> [mem 0x00000000-0xbfffffff]
[    0.006208] NUMA: Node 0 [mem 0x00000000-0xbfffffff] + [mem 0x100000000-0x43fffffff] -> [mem 0x00000000-0x43fffffff]
[    0.006215] NODE_DATA(0) allocated [mem 0x43ffd5000-0x43fffffff]
[    0.006585] Zone ranges:
[    0.006586]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.006587]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.006588]   Normal   [mem 0x0000000100000000-0x000000043fffffff]
[    0.006589]   Device   empty
[    0.006589] Movable zone start for each node
[    0.006591] Early memory node ranges
[    0.006592]   node   0: [mem 0x0000000000001000-0x000000000009efff]
[    0.006593]   node   0: [mem 0x0000000000100000-0x00000000bfedffff]
[    0.006593]   node   0: [mem 0x00000000bff00000-0x00000000bfffffff]
[    0.006594]   node   0: [mem 0x0000000100000000-0x000000043fffffff]
[    0.006606] Zeroed struct page in unavailable ranges: 130 pages
[    0.006607] Initmem setup node 0 [mem 0x0000000000001000-0x000000043fffffff]
[    0.006608] On node 0 totalpages: 4194174
[    0.006609]   DMA zone: 64 pages used for memmap
[    0.006609]   DMA zone: 21 pages reserved
[    0.006610]   DMA zone: 3998 pages, LIFO batch:0
[    0.006732]   DMA32 zone: 12224 pages used for memmap
[    0.006733]   DMA32 zone: 782304 pages, LIFO batch:63
[    0.038739]   Normal zone: 53248 pages used for memmap
[    0.038741]   Normal zone: 3407872 pages, LIFO batch:63
[    0.155962] ACPI: PM-Timer IO Port: 0x1008
[    0.155965] ACPI: Local APIC address 0xfee00000
[    0.155978] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
[    0.155979] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[    0.156010] IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23
[    0.156012] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
[    0.156014] ACPI: IRQ0 used by override.
[    0.156014] ACPI: IRQ9 used by override.
[    0.156016] Using ACPI (MADT) for SMP configuration information
[    0.156018] ACPI: HPET id: 0x8086af01 base: 0xfed00000
[    0.156029] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[    0.156042] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.156043] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
[    0.156043] PM: Registered nosave memory: [mem 0x000a0000-0x000dbfff]
[    0.156044] PM: Registered nosave memory: [mem 0x000dc000-0x000fffff]
[    0.156045] PM: Registered nosave memory: [mem 0xbfee0000-0xbfefefff]
[    0.156045] PM: Registered nosave memory: [mem 0xbfeff000-0xbfefffff]
[    0.156046] PM: Registered nosave memory: [mem 0xc0000000-0xefffffff]
[    0.156046] PM: Registered nosave memory: [mem 0xf0000000-0xf7ffffff]
[    0.156047] PM: Registered nosave memory: [mem 0xf8000000-0xfebfffff]
[    0.156047] PM: Registered nosave memory: [mem 0xfec00000-0xfec0ffff]
[    0.156048] PM: Registered nosave memory: [mem 0xfec10000-0xfedfffff]
[    0.156048] PM: Registered nosave memory: [mem 0xfee00000-0xfee00fff]
[    0.156048] PM: Registered nosave memory: [mem 0xfee01000-0xfffdffff]
[    0.156049] PM: Registered nosave memory: [mem 0xfffe0000-0xffffffff]
[    0.156050] [mem 0xc0000000-0xefffffff] available for PCI devices
[    0.156051] Booting paravirtualized kernel on VMware hypervisor
[    0.156055] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.156063] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1
[    0.157280] percpu: Embedded 60 pages/cpu s208896 r8192 d28672 u1048576
[    0.157284] pcpu-alloc: s208896 r8192 d28672 u1048576 alloc=1*2097152
[    0.157285] pcpu-alloc: [0] 0 1 
[    0.157307] Built 1 zonelists, mobility grouping on.  Total pages: 4128617
[    0.157308] Policy zone: Normal
[    0.157309] Kernel command line: BOOT_IMAGE=/vmlinuz-5.4.0-139-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro splash quiet isolcpus=1-1 nohz_full=1-1 rcu_nocbs=1-1 default_hugepagesz=1G hugepagesz=1G hugepages=12 nmi_watchdog=0 net.ifnames=0 biosdevname=0
[    0.157385] Housekeeping: nohz unsupported. Build with CONFIG_NO_HZ_FULL
[    0.163667] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes, linear)
[    0.166735] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear)
[    0.166792] mem auto-init: stack:off, heap alloc:on, heap free:off
[    0.193190] Calgary: detecting Calgary via BIOS EBDA area
[    0.193192] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
[    0.217249] Memory: 3719036K/16776696K available (14339K kernel code, 2394K rwdata, 9500K rodata, 2764K init, 4944K bss, 13057660K reserved, 0K cma-reserved)
[    0.217859] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.217895] ftrace: allocating 44631 entries in 175 pages
[    0.231906] rcu: Hierarchical RCU implementation.
[    0.231907] rcu: 	RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=2.
[    0.231908] 	Tasks RCU enabled.
[    0.231909] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.231910] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.234288] NR_IRQS: 524544, nr_irqs: 440, preallocated irqs: 16
[    0.234533] random: crng init done
[    0.234549] Console: colour dummy device 80x25
[    0.234552] printk: console [tty0] enabled
[    0.234567] ACPI: Core revision 20190816
[    0.234785] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484882848 ns
[    0.234828] APIC: Switch to symmetric I/O mode setup
[    0.234937] x2apic enabled
[    0.235128] Switched APIC routing to physical x2apic.
[    0.235942] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.235963] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x21dfb64b751, max_idle_ns: 440795265875 ns
[    0.235965] Calibrating delay loop (skipped) preset value.. 4700.00 BogoMIPS (lpj=9400000)
[    0.235967] pid_max: default: 32768 minimum: 301
[    0.235998] LSM: Security Framework initializing
[    0.236007] Yama: becoming mindful.
[    0.236033] AppArmor: AppArmor initialized
[    0.236152] Mount-cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.236248] Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.236267] *** VALIDATE tmpfs ***
[    0.236387] *** VALIDATE proc ***
[    0.236426] *** VALIDATE cgroup1 ***
[    0.236428] *** VALIDATE cgroup2 ***
[    0.236518] LVT offset 2 assigned for vector 0xf4
[    0.236529] Last level iTLB entries: 4KB 1024, 2MB 1024, 4MB 512
[    0.236529] Last level dTLB entries: 4KB 2048, 2MB 2048, 4MB 1024, 1GB 0
[    0.236532] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[    0.236533] Spectre V2 : Mitigation: Retpolines
[    0.236534] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[    0.236534] Spectre V2 : Spectre v2 / SpectreRSB : Filling RSB on VMEXIT
[    0.236535] RETBleed: Vulnerable
[    0.236536] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
[    0.236537] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp
[    0.236732] Freeing SMP alternatives memory: 40K
[    0.240115] smpboot: CPU0: AMD EPYC 7452 32-Core Processor (family: 0x17, model: 0x31, stepping: 0x0)
[    0.240232] Performance Events: AMD PMU driver.
[    0.240238] ... version:                0
[    0.240238] ... bit width:              48
[    0.240239] ... generic registers:      4
[    0.240239] ... value mask:             0000ffffffffffff
[    0.240239] ... max period:             00007fffffffffff
[    0.240240] ... fixed-purpose events:   0
[    0.240240] ... event mask:             000000000000000f
[    0.240281] rcu: Hierarchical SRCU implementation.
[    0.244266] smp: Bringing up secondary CPUs ...
[    0.244423] x86: Booting SMP configuration:
[    0.244424] .... node  #0, CPUs:      #1
[    0.005449] smpboot: CPU 1 Converting physical 2 to logical package 1
[    0.005449] smpboot: CPU 1 Converting physical 2 to logical die 1
[    0.245035] smp: Brought up 1 node, 2 CPUs
[    0.245035] smpboot: Max logical packages: 2
[    0.245035] smpboot: Total of 2 processors activated (9400.00 BogoMIPS)
[    0.245035] devtmpfs: initialized
[    0.245035] x86/mm: Memory block size: 128MB
[    0.246035] PM: Registering ACPI NVS region [mem 0xbfeff000-0xbfefffff] (4096 bytes)
[    0.246083] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.246098] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[    0.246132] pinctrl core: initialized pinctrl subsystem
[    0.246275] PM: RTC time: 13:58:59, date: 2023-07-25
[    0.246411] NET: Registered protocol family 16
[    0.246512] audit: initializing netlink subsys (disabled)
[    0.246632] EISA bus registered
[    0.246638] cpuidle: using governor ladder
[    0.246641] cpuidle: using governor menu
[    0.246767] audit: type=2000 audit(1690293538.008:1): state=initialized audit_enabled=0 res=1
[    0.246827] Simple Boot Flag at 0x36 set to 0x80
[    0.246839] ACPI: bus type PCI registered
[    0.246841] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.248084] PCI: MMCONFIG for domain 0000 [bus 00-7f] at [mem 0xf0000000-0xf7ffffff] (base 0xf0000000)
[    0.248086] PCI: MMCONFIG at [mem 0xf0000000-0xf7ffffff] reserved in E820
[    0.248095] PCI: Using configuration type 1 for base access
[    0.267329] HugeTLB registered 1.00 GiB page size, pre-allocated 12 pages
[    0.268343] ACPI: Added _OSI(Module Device)
[    0.268344] ACPI: Added _OSI(Processor Device)
[    0.268345] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.268345] ACPI: Added _OSI(Processor Aggregator Device)
[    0.268346] ACPI: Added _OSI(Linux-Dell-Video)
[    0.268346] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    0.268347] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[    0.278423] ACPI: 1 ACPI AML tables successfully acquired and loaded
[    0.279830] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[    0.282398] ACPI: Interpreter enabled
[    0.282410] ACPI: (supports S0 S1 S4 S5)
[    0.282411] ACPI: Using IOAPIC for interrupt routing
[    0.282430] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.282869] ACPI: Enabled 4 GPEs in block 00 to 0F
[    0.318736] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f])
[    0.318742] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    0.318935] acpi PNP0A03:00: _OSC: platform does not support [AER LTR]
[    0.319138] acpi PNP0A03:00: _OSC: OS now controls [PCIeHotplug SHPCHotplug PME PCIeCapability]
[    0.320669] PCI host bridge to bus 0000:00
[    0.320671] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.320672] pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000d3fff window]
[    0.320672] pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff window]
[    0.320673] pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff window]
[    0.320674] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]
[    0.320674] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.320675] pci_bus 0000:00: root bus resource [io  0x0d00-0xfeff window]
[    0.320676] pci_bus 0000:00: root bus resource [bus 00-7f]
[    0.320712] pci 0000:00:00.0: [8086:7190] type 00 class 0x060000
[    0.321288] pci 0000:00:01.0: [8086:7191] type 01 class 0x060400
[    0.322192] pci 0000:00:07.0: [8086:7110] type 00 class 0x060100
[    0.322729] pci 0000:00:07.1: [8086:7111] type 00 class 0x01018a
[    0.323269] pci 0000:00:07.1: reg 0x20: [io  0x1060-0x106f]
[    0.323479] pci 0000:00:07.1: legacy IDE quirk: reg 0x10: [io  0x01f0-0x01f7]
[    0.323480] pci 0000:00:07.1: legacy IDE quirk: reg 0x14: [io  0x03f6]
[    0.323480] pci 0000:00:07.1: legacy IDE quirk: reg 0x18: [io  0x0170-0x0177]
[    0.323481] pci 0000:00:07.1: legacy IDE quirk: reg 0x1c: [io  0x0376]
[    0.323677] pci 0000:00:07.3: [8086:7113] type 00 class 0x068000
[    0.324320] pci 0000:00:07.3: quirk: [io  0x1000-0x103f] claimed by PIIX4 ACPI
[    0.324327] pci 0000:00:07.3: quirk: [io  0x1040-0x104f] claimed by PIIX4 SMB
[    0.324580] pci 0000:00:07.7: [15ad:0740] type 00 class 0x088000
[    0.324879] pci 0000:00:07.7: reg 0x10: [io  0x1080-0x10bf]
[    0.325111] pci 0000:00:07.7: reg 0x14: [mem 0xfebfe000-0xfebfffff 64bit]
[    0.326960] pci 0000:00:0f.0: [15ad:0405] type 00 class 0x030000
[    0.327696] pci 0000:00:0f.0: reg 0x10: [io  0x1070-0x107f]
[    0.328317] pci 0000:00:0f.0: reg 0x14: [mem 0xe8000000-0xefffffff pref]
[    0.328926] pci 0000:00:0f.0: reg 0x18: [mem 0xfe000000-0xfe7fffff]
[    0.331370] pci 0000:00:0f.0: reg 0x30: [mem 0x00000000-0x00007fff pref]
[    0.331817] pci 0000:00:10.0: [1000:0030] type 00 class 0x010000
[    0.332151] pci 0000:00:10.0: reg 0x10: [io  0x1400-0x14ff]
[    0.332489] pci 0000:00:10.0: reg 0x14: [mem 0xfeba0000-0xfebbffff 64bit]
[    0.332822] pci 0000:00:10.0: reg 0x1c: [mem 0xfebc0000-0xfebdffff 64bit]
[    0.333440] pci 0000:00:10.0: reg 0x30: [mem 0x00000000-0x00003fff pref]
[    0.333833] pci 0000:00:11.0: [15ad:0790] type 01 class 0x060401
[    0.334741] pci 0000:00:15.0: [15ad:07a0] type 01 class 0x060400
[    0.335364] pci 0000:00:15.0: PME# supported from D0 D3hot D3cold
[    0.335600] pci 0000:00:15.1: [15ad:07a0] type 01 class 0x060400
[    0.336319] pci 0000:00:15.1: PME# supported from D0 D3hot D3cold
[    0.336538] pci 0000:00:15.2: [15ad:07a0] type 01 class 0x060400
[    0.337157] pci 0000:00:15.2: PME# supported from D0 D3hot D3cold
[    0.337374] pci 0000:00:15.3: [15ad:07a0] type 01 class 0x060400
[    0.337994] pci 0000:00:15.3: PME# supported from D0 D3hot D3cold
[    0.338218] pci 0000:00:15.4: [15ad:07a0] type 01 class 0x060400
[    0.338837] pci 0000:00:15.4: PME# supported from D0 D3hot D3cold
[    0.339066] pci 0000:00:15.5: [15ad:07a0] type 01 class 0x060400
[    0.339686] pci 0000:00:15.5: PME# supported from D0 D3hot D3cold
[    0.339907] pci 0000:00:15.6: [15ad:07a0] type 01 class 0x060400
[    0.340530] pci 0000:00:15.6: PME# supported from D0 D3hot D3cold
[    0.340756] pci 0000:00:15.7: [15ad:07a0] type 01 class 0x060400
[    0.341377] pci 0000:00:15.7: PME# supported from D0 D3hot D3cold
[    0.341672] pci 0000:00:16.0: [15ad:07a0] type 01 class 0x060400
[    0.342292] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
[    0.342524] pci 0000:00:16.1: [15ad:07a0] type 01 class 0x060400
[    0.343146] pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
[    0.343367] pci 0000:00:16.2: [15ad:07a0] type 01 class 0x060400
[    0.343997] pci 0000:00:16.2: PME# supported from D0 D3hot D3cold
[    0.344221] pci 0000:00:16.3: [15ad:07a0] type 01 class 0x060400
[    0.344842] pci 0000:00:16.3: PME# supported from D0 D3hot D3cold
[    0.345062] pci 0000:00:16.4: [15ad:07a0] type 01 class 0x060400
[    0.345682] pci 0000:00:16.4: PME# supported from D0 D3hot D3cold
[    0.345902] pci 0000:00:16.5: [15ad:07a0] type 01 class 0x060400
[    0.346530] pci 0000:00:16.5: PME# supported from D0 D3hot D3cold
[    0.346750] pci 0000:00:16.6: [15ad:07a0] type 01 class 0x060400
[    0.347371] pci 0000:00:16.6: PME# supported from D0 D3hot D3cold
[    0.347589] pci 0000:00:16.7: [15ad:07a0] type 01 class 0x060400
[    0.348213] pci 0000:00:16.7: PME# supported from D0 D3hot D3cold
[    0.348512] pci 0000:00:17.0: [15ad:07a0] type 01 class 0x060400
[    0.349132] pci 0000:00:17.0: PME# supported from D0 D3hot D3cold
[    0.349363] pci 0000:00:17.1: [15ad:07a0] type 01 class 0x060400
[    0.349984] pci 0000:00:17.1: PME# supported from D0 D3hot D3cold
[    0.350203] pci 0000:00:17.2: [15ad:07a0] type 01 class 0x060400
[    0.350825] pci 0000:00:17.2: PME# supported from D0 D3hot D3cold
[    0.351046] pci 0000:00:17.3: [15ad:07a0] type 01 class 0x060400
[    0.351666] pci 0000:00:17.3: PME# supported from D0 D3hot D3cold
[    0.351889] pci 0000:00:17.4: [15ad:07a0] type 01 class 0x060400
[    0.352513] pci 0000:00:17.4: PME# supported from D0 D3hot D3cold
[    0.352748] pci 0000:00:17.5: [15ad:07a0] type 01 class 0x060400
[    0.353375] pci 0000:00:17.5: PME# supported from D0 D3hot D3cold
[    0.353601] pci 0000:00:17.6: [15ad:07a0] type 01 class 0x060400
[    0.354223] pci 0000:00:17.6: PME# supported from D0 D3hot D3cold
[    0.354441] pci 0000:00:17.7: [15ad:07a0] type 01 class 0x060400
[    0.355064] pci 0000:00:17.7: PME# supported from D0 D3hot D3cold
[    0.355359] pci 0000:00:18.0: [15ad:07a0] type 01 class 0x060400
[    0.355990] pci 0000:00:18.0: PME# supported from D0 D3hot D3cold
[    0.356226] pci 0000:00:18.1: [15ad:07a0] type 01 class 0x060400
[    0.356846] pci 0000:00:18.1: PME# supported from D0 D3hot D3cold
[    0.357064] pci 0000:00:18.2: [15ad:07a0] type 01 class 0x060400
[    0.357685] pci 0000:00:18.2: PME# supported from D0 D3hot D3cold
[    0.357902] pci 0000:00:18.3: [15ad:07a0] type 01 class 0x060400
[    0.358527] pci 0000:00:18.3: PME# supported from D0 D3hot D3cold
[    0.358747] pci 0000:00:18.4: [15ad:07a0] type 01 class 0x060400
[    0.359369] pci 0000:00:18.4: PME# supported from D0 D3hot D3cold
[    0.359587] pci 0000:00:18.5: [15ad:07a0] type 01 class 0x060400
[    0.360213] pci 0000:00:18.5: PME# supported from D0 D3hot D3cold
[    0.360434] pci 0000:00:18.6: [15ad:07a0] type 01 class 0x060400
[    0.361054] pci 0000:00:18.6: PME# supported from D0 D3hot D3cold
[    0.361274] pci 0000:00:18.7: [15ad:07a0] type 01 class 0x060400
[    0.361894] pci 0000:00:18.7: PME# supported from D0 D3hot D3cold
[    0.362761] pci_bus 0000:01: extended config space not accessible
[    0.365289] pci 0000:00:01.0: PCI bridge to [bus 01]
[    0.365424] pci_bus 0000:02: extended config space not accessible
[    0.365616] acpiphp: Slot [32] registered
[    0.365627] acpiphp: Slot [33] registered
[    0.365641] acpiphp: Slot [34] registered
[    0.365655] acpiphp: Slot [35] registered
[    0.365668] acpiphp: Slot [36] registered
[    0.365684] acpiphp: Slot [37] registered
[    0.365698] acpiphp: Slot [38] registered
[    0.365711] acpiphp: Slot [39] registered
[    0.365724] acpiphp: Slot [40] registered
[    0.365741] acpiphp: Slot [41] registered
[    0.365754] acpiphp: Slot [42] registered
[    0.365770] acpiphp: Slot [43] registered
[    0.365783] acpiphp: Slot [44] registered
[    0.365797] acpiphp: Slot [45] registered
[    0.365810] acpiphp: Slot [46] registered
[    0.365823] acpiphp: Slot [47] registered
[    0.365836] acpiphp: Slot [48] registered
[    0.365853] acpiphp: Slot [49] registered
[    0.365866] acpiphp: Slot [50] registered
[    0.365879] acpiphp: Slot [51] registered
[    0.365893] acpiphp: Slot [52] registered
[    0.365906] acpiphp: Slot [53] registered
[    0.365920] acpiphp: Slot [54] registered
[    0.365936] acpiphp: Slot [55] registered
[    0.365949] acpiphp: Slot [56] registered
[    0.365966] acpiphp: Slot [57] registered
[    0.365980] acpiphp: Slot [58] registered
[    0.365993] acpiphp: Slot [59] registered
[    0.366007] acpiphp: Slot [60] registered
[    0.366024] acpiphp: Slot [61] registered
[    0.366038] acpiphp: Slot [62] registered
[    0.366051] acpiphp: Slot [63] registered
[    0.366194] pci 0000:02:01.0: [15ad:07e0] type 00 class 0x010601
[    0.367210] pci 0000:02:01.0: reg 0x24: [mem 0xfd5ff000-0xfd5fffff]
[    0.367369] pci 0000:02:01.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[    0.367626] pci 0000:02:01.0: PME# supported from D3hot
[    0.370454] pci 0000:00:11.0: PCI bridge to [bus 02] (subtractive decode)
[    0.370467] pci 0000:00:11.0:   bridge window [io  0x2000-0x3fff]
[    0.370481] pci 0000:00:11.0:   bridge window [mem 0xfd500000-0xfdffffff]
[    0.370506] pci 0000:00:11.0:   bridge window [mem 0xe7b00000-0xe7ffffff 64bit pref]
[    0.370507] pci 0000:00:11.0:   bridge window [mem 0x000a0000-0x000bffff window] (subtractive decode)
[    0.370508] pci 0000:00:11.0:   bridge window [mem 0x000d0000-0x000d3fff window] (subtractive decode)
[    0.370508] pci 0000:00:11.0:   bridge window [mem 0x000d4000-0x000d7fff window] (subtractive decode)
[    0.370509] pci 0000:00:11.0:   bridge window [mem 0x000d8000-0x000dbfff window] (subtractive decode)
[    0.370510] pci 0000:00:11.0:   bridge window [mem 0xc0000000-0xfebfffff window] (subtractive decode)
[    0.370510] pci 0000:00:11.0:   bridge window [io  0x0000-0x0cf7 window] (subtractive decode)
[    0.370511] pci 0000:00:11.0:   bridge window [io  0x0d00-0xfeff window] (subtractive decode)
[    0.370765] pci 0000:03:00.0: [15ad:07b0] type 00 class 0x020000
[    0.371266] pci 0000:03:00.0: reg 0x10: [mem 0xfd4fc000-0xfd4fcfff]
[    0.371682] pci 0000:03:00.0: reg 0x14: [mem 0xfd4fd000-0xfd4fdfff]
[    0.371966] pci 0000:03:00.0: reg 0x18: [mem 0xfd4fe000-0xfd4fffff]
[    0.372378] pci 0000:03:00.0: reg 0x1c: [io  0x4000-0x400f]
[    0.373622] pci 0000:03:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[    0.373950] pci 0000:03:00.0: supports D1 D2
[    0.373951] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.374204] pci 0000:03:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    0.376864] pci 0000:00:15.0: PCI bridge to [bus 03]
[    0.376879] pci 0000:00:15.0:   bridge window [io  0x4000-0x4fff]
[    0.376892] pci 0000:00:15.0:   bridge window [mem 0xfd400000-0xfd4fffff]
[    0.379599] pci 0000:00:15.1: PCI bridge to [bus 04]
[    0.379613] pci 0000:00:15.1:   bridge window [io  0x8000-0x8fff]
[    0.379626] pci 0000:00:15.1:   bridge window [mem 0xfd000000-0xfd0fffff]
[    0.379651] pci 0000:00:15.1:   bridge window [mem 0xe7900000-0xe79fffff 64bit pref]
[    0.382337] pci 0000:00:15.2: PCI bridge to [bus 05]
[    0.382351] pci 0000:00:15.2:   bridge window [io  0xc000-0xcfff]
[    0.382363] pci 0000:00:15.2:   bridge window [mem 0xfcc00000-0xfccfffff]
[    0.382388] pci 0000:00:15.2:   bridge window [mem 0xe7500000-0xe75fffff 64bit pref]
[    0.385057] pci 0000:00:15.3: PCI bridge to [bus 06]
[    0.385082] pci 0000:00:15.3:   bridge window [mem 0xfc800000-0xfc8fffff]
[    0.385107] pci 0000:00:15.3:   bridge window [mem 0xe7100000-0xe71fffff 64bit pref]
[    0.387775] pci 0000:00:15.4: PCI bridge to [bus 07]
[    0.387801] pci 0000:00:15.4:   bridge window [mem 0xfc400000-0xfc4fffff]
[    0.387825] pci 0000:00:15.4:   bridge window [mem 0xe6d00000-0xe6dfffff 64bit pref]
[    0.390501] pci 0000:00:15.5: PCI bridge to [bus 08]
[    0.390527] pci 0000:00:15.5:   bridge window [mem 0xfc000000-0xfc0fffff]
[    0.390552] pci 0000:00:15.5:   bridge window [mem 0xe6900000-0xe69fffff 64bit pref]
[    0.393233] pci 0000:00:15.6: PCI bridge to [bus 09]
[    0.393258] pci 0000:00:15.6:   bridge window [mem 0xfbc00000-0xfbcfffff]
[    0.393283] pci 0000:00:15.6:   bridge window [mem 0xe6500000-0xe65fffff 64bit pref]
[    0.395956] pci 0000:00:15.7: PCI bridge to [bus 0a]
[    0.395984] pci 0000:00:15.7:   bridge window [mem 0xfb800000-0xfb8fffff]
[    0.396009] pci 0000:00:15.7:   bridge window [mem 0xe6100000-0xe61fffff 64bit pref]
[    0.396259] pci 0000:0b:00.0: [15ad:07b0] type 00 class 0x020000
[    0.396759] pci 0000:0b:00.0: reg 0x10: [mem 0xfd3fc000-0xfd3fcfff]
[    0.397177] pci 0000:0b:00.0: reg 0x14: [mem 0xfd3fd000-0xfd3fdfff]
[    0.397594] pci 0000:0b:00.0: reg 0x18: [mem 0xfd3fe000-0xfd3fffff]
[    0.398009] pci 0000:0b:00.0: reg 0x1c: [io  0x5000-0x500f]
[    0.399247] pci 0000:0b:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[    0.399565] pci 0000:0b:00.0: supports D1 D2
[    0.399566] pci 0000:0b:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.399818] pci 0000:0b:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    0.402354] pci 0000:00:16.0: PCI bridge to [bus 0b]
[    0.402369] pci 0000:00:16.0:   bridge window [io  0x5000-0x5fff]
[    0.402382] pci 0000:00:16.0:   bridge window [mem 0xfd300000-0xfd3fffff]
[    0.405094] pci 0000:00:16.1: PCI bridge to [bus 0c]
[    0.405107] pci 0000:00:16.1:   bridge window [io  0x9000-0x9fff]
[    0.405120] pci 0000:00:16.1:   bridge window [mem 0xfcf00000-0xfcffffff]
[    0.405145] pci 0000:00:16.1:   bridge window [mem 0xe7800000-0xe78fffff 64bit pref]
[    0.407827] pci 0000:00:16.2: PCI bridge to [bus 0d]
[    0.407843] pci 0000:00:16.2:   bridge window [io  0xd000-0xdfff]
[    0.407856] pci 0000:00:16.2:   bridge window [mem 0xfcb00000-0xfcbfffff]
[    0.407880] pci 0000:00:16.2:   bridge window [mem 0xe7400000-0xe74fffff 64bit pref]
[    0.410553] pci 0000:00:16.3: PCI bridge to [bus 0e]
[    0.410580] pci 0000:00:16.3:   bridge window [mem 0xfc700000-0xfc7fffff]
[    0.410605] pci 0000:00:16.3:   bridge window [mem 0xe7000000-0xe70fffff 64bit pref]
[    0.413282] pci 0000:00:16.4: PCI bridge to [bus 0f]
[    0.413308] pci 0000:00:16.4:   bridge window [mem 0xfc300000-0xfc3fffff]
[    0.413333] pci 0000:00:16.4:   bridge window [mem 0xe6c00000-0xe6cfffff 64bit pref]
[    0.416023] pci 0000:00:16.5: PCI bridge to [bus 10]
[    0.416048] pci 0000:00:16.5:   bridge window [mem 0xfbf00000-0xfbffffff]
[    0.416073] pci 0000:00:16.5:   bridge window [mem 0xe6800000-0xe68fffff 64bit pref]
[    0.418749] pci 0000:00:16.6: PCI bridge to [bus 11]
[    0.418777] pci 0000:00:16.6:   bridge window [mem 0xfbb00000-0xfbbfffff]
[    0.418801] pci 0000:00:16.6:   bridge window [mem 0xe6400000-0xe64fffff 64bit pref]
[    0.421488] pci 0000:00:16.7: PCI bridge to [bus 12]
[    0.421515] pci 0000:00:16.7:   bridge window [mem 0xfb700000-0xfb7fffff]
[    0.421540] pci 0000:00:16.7:   bridge window [mem 0xe6000000-0xe60fffff 64bit pref]
[    0.421794] pci 0000:13:00.0: [15ad:07b0] type 00 class 0x020000
[    0.422287] pci 0000:13:00.0: reg 0x10: [mem 0xfd2fc000-0xfd2fcfff]
[    0.422701] pci 0000:13:00.0: reg 0x14: [mem 0xfd2fd000-0xfd2fdfff]
[    0.423116] pci 0000:13:00.0: reg 0x18: [mem 0xfd2fe000-0xfd2fffff]
[    0.423528] pci 0000:13:00.0: reg 0x1c: [io  0x6000-0x600f]
[    0.424801] pci 0000:13:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[    0.425124] pci 0000:13:00.0: supports D1 D2
[    0.425125] pci 0000:13:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.425391] pci 0000:13:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    0.427922] pci 0000:00:17.0: PCI bridge to [bus 13]
[    0.427937] pci 0000:00:17.0:   bridge window [io  0x6000-0x6fff]
[    0.427966] pci 0000:00:17.0:   bridge window [mem 0xfd200000-0xfd2fffff]
[    0.430674] pci 0000:00:17.1: PCI bridge to [bus 14]
[    0.430689] pci 0000:00:17.1:   bridge window [io  0xa000-0xafff]
[    0.430702] pci 0000:00:17.1:   bridge window [mem 0xfce00000-0xfcefffff]
[    0.430727] pci 0000:00:17.1:   bridge window [mem 0xe7700000-0xe77fffff 64bit pref]
[    0.433413] pci 0000:00:17.2: PCI bridge to [bus 15]
[    0.433426] pci 0000:00:17.2:   bridge window [io  0xe000-0xefff]
[    0.433439] pci 0000:00:17.2:   bridge window [mem 0xfca00000-0xfcafffff]
[    0.433464] pci 0000:00:17.2:   bridge window [mem 0xe7300000-0xe73fffff 64bit pref]
[    0.436142] pci 0000:00:17.3: PCI bridge to [bus 16]
[    0.436169] pci 0000:00:17.3:   bridge window [mem 0xfc600000-0xfc6fffff]
[    0.436193] pci 0000:00:17.3:   bridge window [mem 0xe6f00000-0xe6ffffff 64bit pref]
[    0.438865] pci 0000:00:17.4: PCI bridge to [bus 17]
[    0.438892] pci 0000:00:17.4:   bridge window [mem 0xfc200000-0xfc2fffff]
[    0.438917] pci 0000:00:17.4:   bridge window [mem 0xe6b00000-0xe6bfffff 64bit pref]
[    0.441601] pci 0000:00:17.5: PCI bridge to [bus 18]
[    0.441628] pci 0000:00:17.5:   bridge window [mem 0xfbe00000-0xfbefffff]
[    0.441652] pci 0000:00:17.5:   bridge window [mem 0xe6700000-0xe67fffff 64bit pref]
[    0.444328] pci 0000:00:17.6: PCI bridge to [bus 19]
[    0.444354] pci 0000:00:17.6:   bridge window [mem 0xfba00000-0xfbafffff]
[    0.444379] pci 0000:00:17.6:   bridge window [mem 0xe6300000-0xe63fffff 64bit pref]
[    0.447064] pci 0000:00:17.7: PCI bridge to [bus 1a]
[    0.447090] pci 0000:00:17.7:   bridge window [mem 0xfb600000-0xfb6fffff]
[    0.447115] pci 0000:00:17.7:   bridge window [mem 0xe5f00000-0xe5ffffff 64bit pref]
[    0.449794] pci 0000:00:18.0: PCI bridge to [bus 1b]
[    0.449808] pci 0000:00:18.0:   bridge window [io  0x7000-0x7fff]
[    0.449820] pci 0000:00:18.0:   bridge window [mem 0xfd100000-0xfd1fffff]
[    0.449845] pci 0000:00:18.0:   bridge window [mem 0xe7a00000-0xe7afffff 64bit pref]
[    0.452530] pci 0000:00:18.1: PCI bridge to [bus 1c]
[    0.452544] pci 0000:00:18.1:   bridge window [io  0xb000-0xbfff]
[    0.452557] pci 0000:00:18.1:   bridge window [mem 0xfcd00000-0xfcdfffff]
[    0.452581] pci 0000:00:18.1:   bridge window [mem 0xe7600000-0xe76fffff 64bit pref]
[    0.455257] pci 0000:00:18.2: PCI bridge to [bus 1d]
[    0.455282] pci 0000:00:18.2:   bridge window [mem 0xfc900000-0xfc9fffff]
[    0.455307] pci 0000:00:18.2:   bridge window [mem 0xe7200000-0xe72fffff 64bit pref]
[    0.457988] pci 0000:00:18.3: PCI bridge to [bus 1e]
[    0.458014] pci 0000:00:18.3:   bridge window [mem 0xfc500000-0xfc5fffff]
[    0.458039] pci 0000:00:18.3:   bridge window [mem 0xe6e00000-0xe6efffff 64bit pref]
[    0.460718] pci 0000:00:18.4: PCI bridge to [bus 1f]
[    0.460745] pci 0000:00:18.4:   bridge window [mem 0xfc100000-0xfc1fffff]
[    0.460769] pci 0000:00:18.4:   bridge window [mem 0xe6a00000-0xe6afffff 64bit pref]
[    0.463454] pci 0000:00:18.5: PCI bridge to [bus 20]
[    0.463481] pci 0000:00:18.5:   bridge window [mem 0xfbd00000-0xfbdfffff]
[    0.463505] pci 0000:00:18.5:   bridge window [mem 0xe6600000-0xe66fffff 64bit pref]
[    0.466190] pci 0000:00:18.6: PCI bridge to [bus 21]
[    0.466215] pci 0000:00:18.6:   bridge window [mem 0xfb900000-0xfb9fffff]
[    0.466240] pci 0000:00:18.6:   bridge window [mem 0xe6200000-0xe62fffff 64bit pref]
[    0.468913] pci 0000:00:18.7: PCI bridge to [bus 22]
[    0.468939] pci 0000:00:18.7:   bridge window [mem 0xfb500000-0xfb5fffff]
[    0.468964] pci 0000:00:18.7:   bridge window [mem 0xe5e00000-0xe5efffff 64bit pref]
[    0.470419] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 *9 10 11 14 15)
[    0.470488] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 *11 14 15)
[    0.470555] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 *7 9 10 11 14 15)
[    0.470622] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 *10 11 14 15)
[    0.476634] iommu: Default domain type: Translated 
[    0.476787] SCSI subsystem initialized
[    0.476814] libata version 3.00 loaded.
[    0.476850] pci 0000:00:0f.0: vgaarb: setting as boot VGA device
[    0.476852] pci 0000:00:0f.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    0.476855] pci 0000:00:0f.0: vgaarb: bridge control possible
[    0.476856] vgaarb: loaded
[    0.476878] ACPI: bus type USB registered
[    0.476890] usbcore: registered new interface driver usbfs
[    0.476896] usbcore: registered new interface driver hub
[    0.476900] usbcore: registered new device driver usb
[    0.476916] pps_core: LinuxPPS API ver. 1 registered
[    0.476916] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.476918] PTP clock support registered
[    0.476947] EDAC MC: Ver: 3.0.0
[    0.477800] PCI: Using ACPI for IRQ routing
[    0.490007] PCI: pci_cache_line_size set to 64 bytes
[    0.490571] e820: reserve RAM buffer [mem 0x0009f400-0x0009ffff]
[    0.490572] e820: reserve RAM buffer [mem 0xbfee0000-0xbfffffff]
[    0.490660] NetLabel: Initializing
[    0.490660] NetLabel:  domain hash size = 128
[    0.490661] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    0.490671] NetLabel:  unlabeled traffic allowed by default
[    0.490817] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
[    0.490821] hpet0: 16 comparators, 64-bit 14.318180 MHz counter
[    0.491990] clocksource: Switched to clocksource tsc-early
[    0.504589] *** VALIDATE bpf ***
[    0.504629] VFS: Disk quotas dquot_6.6.0
[    0.504643] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.504665] *** VALIDATE ramfs ***
[    0.504667] *** VALIDATE hugetlbfs ***
[    0.504760] AppArmor: AppArmor Filesystem Enabled
[    0.504777] pnp: PnP ACPI init
[    0.504954] system 00:00: [io  0x1000-0x103f] has been reserved
[    0.504955] system 00:00: [io  0x1040-0x104f] has been reserved
[    0.504956] system 00:00: [io  0x0cf0-0x0cf1] has been reserved
[    0.504959] system 00:00: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.504991] pnp 00:01: Plug and Play ACPI device, IDs PNP0b00 (active)
[    0.505007] pnp 00:02: Plug and Play ACPI device, IDs PNP0303 (active)
[    0.505020] pnp 00:03: Plug and Play ACPI device, IDs VMW0003 PNP0f13 (active)
[    0.505184] system 00:04: [mem 0xfed00000-0xfed003ff] has been reserved
[    0.505187] system 00:04: Plug and Play ACPI device, IDs PNP0103 PNP0c01 (active)
[    0.506851] system 00:05: [io  0xfce0-0xfcff] has been reserved
[    0.506853] system 00:05: [mem 0xf0000000-0xf7ffffff] has been reserved
[    0.506854] system 00:05: [mem 0xfe800000-0xfe9fffff] has been reserved
[    0.506856] system 00:05: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.510808] pnp: PnP ACPI: found 6 devices
[    0.511746] thermal_sys: Registered thermal governor 'fair_share'
[    0.511747] thermal_sys: Registered thermal governor 'bang_bang'
[    0.511747] thermal_sys: Registered thermal governor 'step_wise'
[    0.511748] thermal_sys: Registered thermal governor 'user_space'
[    0.511748] thermal_sys: Registered thermal governor 'power_allocator'
[    0.516245] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    0.516282] pci 0000:00:15.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 03] add_size 200000 add_align 100000
[    0.516284] pci 0000:00:15.3: bridge window [io  0x1000-0x0fff] to [bus 06] add_size 1000
[    0.516285] pci 0000:00:15.4: bridge window [io  0x1000-0x0fff] to [bus 07] add_size 1000
[    0.516286] pci 0000:00:15.5: bridge window [io  0x1000-0x0fff] to [bus 08] add_size 1000
[    0.516287] pci 0000:00:15.6: bridge window [io  0x1000-0x0fff] to [bus 09] add_size 1000
[    0.516288] pci 0000:00:15.7: bridge window [io  0x1000-0x0fff] to [bus 0a] add_size 1000
[    0.516289] pci 0000:00:16.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 0b] add_size 200000 add_align 100000
[    0.516290] pci 0000:00:16.3: bridge window [io  0x1000-0x0fff] to [bus 0e] add_size 1000
[    0.516291] pci 0000:00:16.4: bridge window [io  0x1000-0x0fff] to [bus 0f] add_size 1000
[    0.516292] pci 0000:00:16.5: bridge window [io  0x1000-0x0fff] to [bus 10] add_size 1000
[    0.516293] pci 0000:00:16.6: bridge window [io  0x1000-0x0fff] to [bus 11] add_size 1000
[    0.516294] pci 0000:00:16.7: bridge window [io  0x1000-0x0fff] to [bus 12] add_size 1000
[    0.516295] pci 0000:00:17.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 13] add_size 200000 add_align 100000
[    0.516296] pci 0000:00:17.3: bridge window [io  0x1000-0x0fff] to [bus 16] add_size 1000
[    0.516297] pci 0000:00:17.4: bridge window [io  0x1000-0x0fff] to [bus 17] add_size 1000
[    0.516298] pci 0000:00:17.5: bridge window [io  0x1000-0x0fff] to [bus 18] add_size 1000
[    0.516299] pci 0000:00:17.6: bridge window [io  0x1000-0x0fff] to [bus 19] add_size 1000
[    0.516300] pci 0000:00:17.7: bridge window [io  0x1000-0x0fff] to [bus 1a] add_size 1000
[    0.516301] pci 0000:00:18.2: bridge window [io  0x1000-0x0fff] to [bus 1d] add_size 1000
[    0.516302] pci 0000:00:18.3: bridge window [io  0x1000-0x0fff] to [bus 1e] add_size 1000
[    0.516303] pci 0000:00:18.4: bridge window [io  0x1000-0x0fff] to [bus 1f] add_size 1000
[    0.516304] pci 0000:00:18.5: bridge window [io  0x1000-0x0fff] to [bus 20] add_size 1000
[    0.516307] pci 0000:00:18.6: bridge window [io  0x1000-0x0fff] to [bus 21] add_size 1000
[    0.516308] pci 0000:00:18.7: bridge window [io  0x1000-0x0fff] to [bus 22] add_size 1000
[    0.516322] pci 0000:00:15.0: BAR 15: assigned [mem 0xc0000000-0xc01fffff 64bit pref]
[    0.516325] pci 0000:00:16.0: BAR 15: assigned [mem 0xc0200000-0xc03fffff 64bit pref]
[    0.516327] pci 0000:00:17.0: BAR 15: assigned [mem 0xc0400000-0xc05fffff 64bit pref]
[    0.516329] pci 0000:00:0f.0: BAR 6: assigned [mem 0xc0600000-0xc0607fff pref]
[    0.516330] pci 0000:00:10.0: BAR 6: assigned [mem 0xc0608000-0xc060bfff pref]
[    0.516332] pci 0000:00:15.3: BAR 13: no space for [io  size 0x1000]
[    0.516333] pci 0000:00:15.3: BAR 13: failed to assign [io  size 0x1000]
[    0.516335] pci 0000:00:15.4: BAR 13: no space for [io  size 0x1000]
[    0.516335] pci 0000:00:15.4: BAR 13: failed to assign [io  size 0x1000]
[    0.516337] pci 0000:00:15.5: BAR 13: no space for [io  size 0x1000]
[    0.516337] pci 0000:00:15.5: BAR 13: failed to assign [io  size 0x1000]
[    0.516338] pci 0000:00:15.6: BAR 13: no space for [io  size 0x1000]
[    0.516339] pci 0000:00:15.6: BAR 13: failed to assign [io  size 0x1000]
[    0.516340] pci 0000:00:15.7: BAR 13: no space for [io  size 0x1000]
[    0.516341] pci 0000:00:15.7: BAR 13: failed to assign [io  size 0x1000]
[    0.516342] pci 0000:00:16.3: BAR 13: no space for [io  size 0x1000]
[    0.516343] pci 0000:00:16.3: BAR 13: failed to assign [io  size 0x1000]
[    0.516344] pci 0000:00:16.4: BAR 13: no space for [io  size 0x1000]
[    0.516345] pci 0000:00:16.4: BAR 13: failed to assign [io  size 0x1000]
[    0.516346] pci 0000:00:16.5: BAR 13: no space for [io  size 0x1000]
[    0.516347] pci 0000:00:16.5: BAR 13: failed to assign [io  size 0x1000]
[    0.516348] pci 0000:00:16.6: BAR 13: no space for [io  size 0x1000]
[    0.516349] pci 0000:00:16.6: BAR 13: failed to assign [io  size 0x1000]
[    0.516350] pci 0000:00:16.7: BAR 13: no space for [io  size 0x1000]
[    0.516350] pci 0000:00:16.7: BAR 13: failed to assign [io  size 0x1000]
[    0.516352] pci 0000:00:17.3: BAR 13: no space for [io  size 0x1000]
[    0.516352] pci 0000:00:17.3: BAR 13: failed to assign [io  size 0x1000]
[    0.516354] pci 0000:00:17.4: BAR 13: no space for [io  size 0x1000]
[    0.516354] pci 0000:00:17.4: BAR 13: failed to assign [io  size 0x1000]
[    0.516356] pci 0000:00:17.5: BAR 13: no space for [io  size 0x1000]
[    0.516356] pci 0000:00:17.5: BAR 13: failed to assign [io  size 0x1000]
[    0.516357] pci 0000:00:17.6: BAR 13: no space for [io  size 0x1000]
[    0.516358] pci 0000:00:17.6: BAR 13: failed to assign [io  size 0x1000]
[    0.516359] pci 0000:00:17.7: BAR 13: no space for [io  size 0x1000]
[    0.516360] pci 0000:00:17.7: BAR 13: failed to assign [io  size 0x1000]
[    0.516361] pci 0000:00:18.2: BAR 13: no space for [io  size 0x1000]
[    0.516362] pci 0000:00:18.2: BAR 13: failed to assign [io  size 0x1000]
[    0.516363] pci 0000:00:18.3: BAR 13: no space for [io  size 0x1000]
[    0.516364] pci 0000:00:18.3: BAR 13: failed to assign [io  size 0x1000]
[    0.516365] pci 0000:00:18.4: BAR 13: no space for [io  size 0x1000]
[    0.516366] pci 0000:00:18.4: BAR 13: failed to assign [io  size 0x1000]
[    0.516367] pci 0000:00:18.5: BAR 13: no space for [io  size 0x1000]
[    0.516368] pci 0000:00:18.5: BAR 13: failed to assign [io  size 0x1000]
[    0.516369] pci 0000:00:18.6: BAR 13: no space for [io  size 0x1000]
[    0.516369] pci 0000:00:18.6: BAR 13: failed to assign [io  size 0x1000]
[    0.516371] pci 0000:00:18.7: BAR 13: no space for [io  size 0x1000]
[    0.516371] pci 0000:00:18.7: BAR 13: failed to assign [io  size 0x1000]
[    0.516375] pci 0000:00:18.7: BAR 13: no space for [io  size 0x1000]
[    0.516375] pci 0000:00:18.7: BAR 13: failed to assign [io  size 0x1000]
[    0.516377] pci 0000:00:18.6: BAR 13: no space for [io  size 0x1000]
[    0.516377] pci 0000:00:18.6: BAR 13: failed to assign [io  size 0x1000]
[    0.516378] pci 0000:00:18.5: BAR 13: no space for [io  size 0x1000]
[    0.516379] pci 0000:00:18.5: BAR 13: failed to assign [io  size 0x1000]
[    0.516380] pci 0000:00:18.4: BAR 13: no space for [io  size 0x1000]
[    0.516381] pci 0000:00:18.4: BAR 13: failed to assign [io  size 0x1000]
[    0.516382] pci 0000:00:18.3: BAR 13: no space for [io  size 0x1000]
[    0.516383] pci 0000:00:18.3: BAR 13: failed to assign [io  size 0x1000]
[    0.516384] pci 0000:00:18.2: BAR 13: no space for [io  size 0x1000]
[    0.516385] pci 0000:00:18.2: BAR 13: failed to assign [io  size 0x1000]
[    0.516386] pci 0000:00:17.7: BAR 13: no space for [io  size 0x1000]
[    0.516387] pci 0000:00:17.7: BAR 13: failed to assign [io  size 0x1000]
[    0.516388] pci 0000:00:17.6: BAR 13: no space for [io  size 0x1000]
[    0.516389] pci 0000:00:17.6: BAR 13: failed to assign [io  size 0x1000]
[    0.516390] pci 0000:00:17.5: BAR 13: no space for [io  size 0x1000]
[    0.516391] pci 0000:00:17.5: BAR 13: failed to assign [io  size 0x1000]
[    0.516392] pci 0000:00:17.4: BAR 13: no space for [io  size 0x1000]
[    0.516392] pci 0000:00:17.4: BAR 13: failed to assign [io  size 0x1000]
[    0.516394] pci 0000:00:17.3: BAR 13: no space for [io  size 0x1000]
[    0.516394] pci 0000:00:17.3: BAR 13: failed to assign [io  size 0x1000]
[    0.516396] pci 0000:00:16.7: BAR 13: no space for [io  size 0x1000]
[    0.516396] pci 0000:00:16.7: BAR 13: failed to assign [io  size 0x1000]
[    0.516398] pci 0000:00:16.6: BAR 13: no space for [io  size 0x1000]
[    0.516398] pci 0000:00:16.6: BAR 13: failed to assign [io  size 0x1000]
[    0.516399] pci 0000:00:16.5: BAR 13: no space for [io  size 0x1000]
[    0.516400] pci 0000:00:16.5: BAR 13: failed to assign [io  size 0x1000]
[    0.516401] pci 0000:00:16.4: BAR 13: no space for [io  size 0x1000]
[    0.516402] pci 0000:00:16.4: BAR 13: failed to assign [io  size 0x1000]
[    0.516403] pci 0000:00:16.3: BAR 13: no space for [io  size 0x1000]
[    0.516404] pci 0000:00:16.3: BAR 13: failed to assign [io  size 0x1000]
[    0.516405] pci 0000:00:15.7: BAR 13: no space for [io  size 0x1000]
[    0.516406] pci 0000:00:15.7: BAR 13: failed to assign [io  size 0x1000]
[    0.516407] pci 0000:00:15.6: BAR 13: no space for [io  size 0x1000]
[    0.516408] pci 0000:00:15.6: BAR 13: failed to assign [io  size 0x1000]
[    0.516409] pci 0000:00:15.5: BAR 13: no space for [io  size 0x1000]
[    0.516410] pci 0000:00:15.5: BAR 13: failed to assign [io  size 0x1000]
[    0.516411] pci 0000:00:15.4: BAR 13: no space for [io  size 0x1000]
[    0.516411] pci 0000:00:15.4: BAR 13: failed to assign [io  size 0x1000]
[    0.516413] pci 0000:00:15.3: BAR 13: no space for [io  size 0x1000]
[    0.516413] pci 0000:00:15.3: BAR 13: failed to assign [io  size 0x1000]
[    0.516415] pci 0000:00:01.0: PCI bridge to [bus 01]
[    0.516476] pci 0000:02:01.0: BAR 6: assigned [mem 0xfd500000-0xfd50ffff pref]
[    0.516477] pci 0000:00:11.0: PCI bridge to [bus 02]
[    0.516485] pci 0000:00:11.0:   bridge window [io  0x2000-0x3fff]
[    0.516504] pci 0000:00:11.0:   bridge window [mem 0xfd500000-0xfdffffff]
[    0.516517] pci 0000:00:11.0:   bridge window [mem 0xe7b00000-0xe7ffffff 64bit pref]
[    0.516543] pci 0000:03:00.0: BAR 6: assigned [mem 0xfd400000-0xfd40ffff pref]
[    0.516543] pci 0000:00:15.0: PCI bridge to [bus 03]
[    0.516550] pci 0000:00:15.0:   bridge window [io  0x4000-0x4fff]
[    0.516569] pci 0000:00:15.0:   bridge window [mem 0xfd400000-0xfd4fffff]
[    0.516582] pci 0000:00:15.0:   bridge window [mem 0xc0000000-0xc01fffff 64bit pref]
[    0.516625] pci 0000:00:15.1: PCI bridge to [bus 04]
[    0.516632] pci 0000:00:15.1:   bridge window [io  0x8000-0x8fff]
[    0.516651] pci 0000:00:15.1:   bridge window [mem 0xfd000000-0xfd0fffff]
[    0.516664] pci 0000:00:15.1:   bridge window [mem 0xe7900000-0xe79fffff 64bit pref]
[    0.516692] pci 0000:00:15.2: PCI bridge to [bus 05]
[    0.516699] pci 0000:00:15.2:   bridge window [io  0xc000-0xcfff]
[    0.516718] pci 0000:00:15.2:   bridge window [mem 0xfcc00000-0xfccfffff]
[    0.516731] pci 0000:00:15.2:   bridge window [mem 0xe7500000-0xe75fffff 64bit pref]
[    0.516759] pci 0000:00:15.3: PCI bridge to [bus 06]
[    0.516778] pci 0000:00:15.3:   bridge window [mem 0xfc800000-0xfc8fffff]
[    0.516791] pci 0000:00:15.3:   bridge window [mem 0xe7100000-0xe71fffff 64bit pref]
[    0.516819] pci 0000:00:15.4: PCI bridge to [bus 07]
[    0.516838] pci 0000:00:15.4:   bridge window [mem 0xfc400000-0xfc4fffff]
[    0.516851] pci 0000:00:15.4:   bridge window [mem 0xe6d00000-0xe6dfffff 64bit pref]
[    0.516879] pci 0000:00:15.5: PCI bridge to [bus 08]
[    0.516898] pci 0000:00:15.5:   bridge window [mem 0xfc000000-0xfc0fffff]
[    0.516911] pci 0000:00:15.5:   bridge window [mem 0xe6900000-0xe69fffff 64bit pref]
[    0.516940] pci 0000:00:15.6: PCI bridge to [bus 09]
[    0.516958] pci 0000:00:15.6:   bridge window [mem 0xfbc00000-0xfbcfffff]
[    0.516971] pci 0000:00:15.6:   bridge window [mem 0xe6500000-0xe65fffff 64bit pref]
[    0.516999] pci 0000:00:15.7: PCI bridge to [bus 0a]
[    0.517018] pci 0000:00:15.7:   bridge window [mem 0xfb800000-0xfb8fffff]
[    0.517031] pci 0000:00:15.7:   bridge window [mem 0xe6100000-0xe61fffff 64bit pref]
[    0.517060] pci 0000:0b:00.0: BAR 6: assigned [mem 0xfd300000-0xfd30ffff pref]
[    0.517060] pci 0000:00:16.0: PCI bridge to [bus 0b]
[    0.517068] pci 0000:00:16.0:   bridge window [io  0x5000-0x5fff]
[    0.517088] pci 0000:00:16.0:   bridge window [mem 0xfd300000-0xfd3fffff]
[    0.517101] pci 0000:00:16.0:   bridge window [mem 0xc0200000-0xc03fffff 64bit pref]
[    0.517129] pci 0000:00:16.1: PCI bridge to [bus 0c]
[    0.517136] pci 0000:00:16.1:   bridge window [io  0x9000-0x9fff]
[    0.517155] pci 0000:00:16.1:   bridge window [mem 0xfcf00000-0xfcffffff]
[    0.517168] pci 0000:00:16.1:   bridge window [mem 0xe7800000-0xe78fffff 64bit pref]
[    0.517196] pci 0000:00:16.2: PCI bridge to [bus 0d]
[    0.517203] pci 0000:00:16.2:   bridge window [io  0xd000-0xdfff]
[    0.517222] pci 0000:00:16.2:   bridge window [mem 0xfcb00000-0xfcbfffff]
[    0.517235] pci 0000:00:16.2:   bridge window [mem 0xe7400000-0xe74fffff 64bit pref]
[    0.517263] pci 0000:00:16.3: PCI bridge to [bus 0e]
[    0.517282] pci 0000:00:16.3:   bridge window [mem 0xfc700000-0xfc7fffff]
[    0.517295] pci 0000:00:16.3:   bridge window [mem 0xe7000000-0xe70fffff 64bit pref]
[    0.517323] pci 0000:00:16.4: PCI bridge to [bus 0f]
[    0.517342] pci 0000:00:16.4:   bridge window [mem 0xfc300000-0xfc3fffff]
[    0.517355] pci 0000:00:16.4:   bridge window [mem 0xe6c00000-0xe6cfffff 64bit pref]
[    0.517383] pci 0000:00:16.5: PCI bridge to [bus 10]
[    0.517402] pci 0000:00:16.5:   bridge window [mem 0xfbf00000-0xfbffffff]
[    0.517415] pci 0000:00:16.5:   bridge window [mem 0xe6800000-0xe68fffff 64bit pref]
[    0.517443] pci 0000:00:16.6: PCI bridge to [bus 11]
[    0.517462] pci 0000:00:16.6:   bridge window [mem 0xfbb00000-0xfbbfffff]
[    0.517475] pci 0000:00:16.6:   bridge window [mem 0xe6400000-0xe64fffff 64bit pref]
[    0.517503] pci 0000:00:16.7: PCI bridge to [bus 12]
[    0.517522] pci 0000:00:16.7:   bridge window [mem 0xfb700000-0xfb7fffff]
[    0.517534] pci 0000:00:16.7:   bridge window [mem 0xe6000000-0xe60fffff 64bit pref]
[    0.517563] pci 0000:13:00.0: BAR 6: assigned [mem 0xfd200000-0xfd20ffff pref]
[    0.517564] pci 0000:00:17.0: PCI bridge to [bus 13]
[    0.517571] pci 0000:00:17.0:   bridge window [io  0x6000-0x6fff]
[    0.517590] pci 0000:00:17.0:   bridge window [mem 0xfd200000-0xfd2fffff]
[    0.517602] pci 0000:00:17.0:   bridge window [mem 0xc0400000-0xc05fffff 64bit pref]
[    0.517631] pci 0000:00:17.1: PCI bridge to [bus 14]
[    0.517638] pci 0000:00:17.1:   bridge window [io  0xa000-0xafff]
[    0.517657] pci 0000:00:17.1:   bridge window [mem 0xfce00000-0xfcefffff]
[    0.517669] pci 0000:00:17.1:   bridge window [mem 0xe7700000-0xe77fffff 64bit pref]
[    0.517698] pci 0000:00:17.2: PCI bridge to [bus 15]
[    0.517705] pci 0000:00:17.2:   bridge window [io  0xe000-0xefff]
[    0.517724] pci 0000:00:17.2:   bridge window [mem 0xfca00000-0xfcafffff]
[    0.517737] pci 0000:00:17.2:   bridge window [mem 0xe7300000-0xe73fffff 64bit pref]
[    0.517765] pci 0000:00:17.3: PCI bridge to [bus 16]
[    0.517784] pci 0000:00:17.3:   bridge window [mem 0xfc600000-0xfc6fffff]
[    0.517796] pci 0000:00:17.3:   bridge window [mem 0xe6f00000-0xe6ffffff 64bit pref]
[    0.517825] pci 0000:00:17.4: PCI bridge to [bus 17]
[    0.517843] pci 0000:00:17.4:   bridge window [mem 0xfc200000-0xfc2fffff]
[    0.517856] pci 0000:00:17.4:   bridge window [mem 0xe6b00000-0xe6bfffff 64bit pref]
[    0.517885] pci 0000:00:17.5: PCI bridge to [bus 18]
[    0.517903] pci 0000:00:17.5:   bridge window [mem 0xfbe00000-0xfbefffff]
[    0.517916] pci 0000:00:17.5:   bridge window [mem 0xe6700000-0xe67fffff 64bit pref]
[    0.517944] pci 0000:00:17.6: PCI bridge to [bus 19]
[    0.517963] pci 0000:00:17.6:   bridge window [mem 0xfba00000-0xfbafffff]
[    0.517976] pci 0000:00:17.6:   bridge window [mem 0xe6300000-0xe63fffff 64bit pref]
[    0.518012] pci 0000:00:17.7: PCI bridge to [bus 1a]
[    0.518031] pci 0000:00:17.7:   bridge window [mem 0xfb600000-0xfb6fffff]
[    0.518043] pci 0000:00:17.7:   bridge window [mem 0xe5f00000-0xe5ffffff 64bit pref]
[    0.518072] pci 0000:00:18.0: PCI bridge to [bus 1b]
[    0.518080] pci 0000:00:18.0:   bridge window [io  0x7000-0x7fff]
[    0.518099] pci 0000:00:18.0:   bridge window [mem 0xfd100000-0xfd1fffff]
[    0.518112] pci 0000:00:18.0:   bridge window [mem 0xe7a00000-0xe7afffff 64bit pref]
[    0.518140] pci 0000:00:18.1: PCI bridge to [bus 1c]
[    0.518147] pci 0000:00:18.1:   bridge window [io  0xb000-0xbfff]
[    0.518166] pci 0000:00:18.1:   bridge window [mem 0xfcd00000-0xfcdfffff]
[    0.518179] pci 0000:00:18.1:   bridge window [mem 0xe7600000-0xe76fffff 64bit pref]
[    0.518207] pci 0000:00:18.2: PCI bridge to [bus 1d]
[    0.518226] pci 0000:00:18.2:   bridge window [mem 0xfc900000-0xfc9fffff]
[    0.518239] pci 0000:00:18.2:   bridge window [mem 0xe7200000-0xe72fffff 64bit pref]
[    0.518267] pci 0000:00:18.3: PCI bridge to [bus 1e]
[    0.518286] pci 0000:00:18.3:   bridge window [mem 0xfc500000-0xfc5fffff]
[    0.518298] pci 0000:00:18.3:   bridge window [mem 0xe6e00000-0xe6efffff 64bit pref]
[    0.518327] pci 0000:00:18.4: PCI bridge to [bus 1f]
[    0.518346] pci 0000:00:18.4:   bridge window [mem 0xfc100000-0xfc1fffff]
[    0.518358] pci 0000:00:18.4:   bridge window [mem 0xe6a00000-0xe6afffff 64bit pref]
[    0.518387] pci 0000:00:18.5: PCI bridge to [bus 20]
[    0.518405] pci 0000:00:18.5:   bridge window [mem 0xfbd00000-0xfbdfffff]
[    0.518418] pci 0000:00:18.5:   bridge window [mem 0xe6600000-0xe66fffff 64bit pref]
[    0.518446] pci 0000:00:18.6: PCI bridge to [bus 21]
[    0.518465] pci 0000:00:18.6:   bridge window [mem 0xfb900000-0xfb9fffff]
[    0.518478] pci 0000:00:18.6:   bridge window [mem 0xe6200000-0xe62fffff 64bit pref]
[    0.518506] pci 0000:00:18.7: PCI bridge to [bus 22]
[    0.518525] pci 0000:00:18.7:   bridge window [mem 0xfb500000-0xfb5fffff]
[    0.518538] pci 0000:00:18.7:   bridge window [mem 0xe5e00000-0xe5efffff 64bit pref]
[    0.518567] pci_bus 0000:00: resource 4 [mem 0x000a0000-0x000bffff window]
[    0.518568] pci_bus 0000:00: resource 5 [mem 0x000d0000-0x000d3fff window]
[    0.518568] pci_bus 0000:00: resource 6 [mem 0x000d4000-0x000d7fff window]
[    0.518569] pci_bus 0000:00: resource 7 [mem 0x000d8000-0x000dbfff window]
[    0.518570] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]
[    0.518571] pci_bus 0000:00: resource 9 [io  0x0000-0x0cf7 window]
[    0.518571] pci_bus 0000:00: resource 10 [io  0x0d00-0xfeff window]
[    0.518572] pci_bus 0000:02: resource 0 [io  0x2000-0x3fff]
[    0.518573] pci_bus 0000:02: resource 1 [mem 0xfd500000-0xfdffffff]
[    0.518574] pci_bus 0000:02: resource 2 [mem 0xe7b00000-0xe7ffffff 64bit pref]
[    0.518575] pci_bus 0000:02: resource 4 [mem 0x000a0000-0x000bffff window]
[    0.518575] pci_bus 0000:02: resource 5 [mem 0x000d0000-0x000d3fff window]
[    0.518576] pci_bus 0000:02: resource 6 [mem 0x000d4000-0x000d7fff window]
[    0.518577] pci_bus 0000:02: resource 7 [mem 0x000d8000-0x000dbfff window]
[    0.518577] pci_bus 0000:02: resource 8 [mem 0xc0000000-0xfebfffff window]
[    0.518578] pci_bus 0000:02: resource 9 [io  0x0000-0x0cf7 window]
[    0.518579] pci_bus 0000:02: resource 10 [io  0x0d00-0xfeff window]
[    0.518580] pci_bus 0000:03: resource 0 [io  0x4000-0x4fff]
[    0.518580] pci_bus 0000:03: resource 1 [mem 0xfd400000-0xfd4fffff]
[    0.518581] pci_bus 0000:03: resource 2 [mem 0xc0000000-0xc01fffff 64bit pref]
[    0.518582] pci_bus 0000:04: resource 0 [io  0x8000-0x8fff]
[    0.518582] pci_bus 0000:04: resource 1 [mem 0xfd000000-0xfd0fffff]
[    0.518583] pci_bus 0000:04: resource 2 [mem 0xe7900000-0xe79fffff 64bit pref]
[    0.518584] pci_bus 0000:05: resource 0 [io  0xc000-0xcfff]
[    0.518585] pci_bus 0000:05: resource 1 [mem 0xfcc00000-0xfccfffff]
[    0.518585] pci_bus 0000:05: resource 2 [mem 0xe7500000-0xe75fffff 64bit pref]
[    0.518586] pci_bus 0000:06: resource 1 [mem 0xfc800000-0xfc8fffff]
[    0.518587] pci_bus 0000:06: resource 2 [mem 0xe7100000-0xe71fffff 64bit pref]
[    0.518587] pci_bus 0000:07: resource 1 [mem 0xfc400000-0xfc4fffff]
[    0.518588] pci_bus 0000:07: resource 2 [mem 0xe6d00000-0xe6dfffff 64bit pref]
[    0.518589] pci_bus 0000:08: resource 1 [mem 0xfc000000-0xfc0fffff]
[    0.518590] pci_bus 0000:08: resource 2 [mem 0xe6900000-0xe69fffff 64bit pref]
[    0.518590] pci_bus 0000:09: resource 1 [mem 0xfbc00000-0xfbcfffff]
[    0.518591] pci_bus 0000:09: resource 2 [mem 0xe6500000-0xe65fffff 64bit pref]
[    0.518592] pci_bus 0000:0a: resource 1 [mem 0xfb800000-0xfb8fffff]
[    0.518592] pci_bus 0000:0a: resource 2 [mem 0xe6100000-0xe61fffff 64bit pref]
[    0.518593] pci_bus 0000:0b: resource 0 [io  0x5000-0x5fff]
[    0.518594] pci_bus 0000:0b: resource 1 [mem 0xfd300000-0xfd3fffff]
[    0.518595] pci_bus 0000:0b: resource 2 [mem 0xc0200000-0xc03fffff 64bit pref]
[    0.518595] pci_bus 0000:0c: resource 0 [io  0x9000-0x9fff]
[    0.518596] pci_bus 0000:0c: resource 1 [mem 0xfcf00000-0xfcffffff]
[    0.518597] pci_bus 0000:0c: resource 2 [mem 0xe7800000-0xe78fffff 64bit pref]
[    0.518598] pci_bus 0000:0d: resource 0 [io  0xd000-0xdfff]
[    0.518598] pci_bus 0000:0d: resource 1 [mem 0xfcb00000-0xfcbfffff]
[    0.518599] pci_bus 0000:0d: resource 2 [mem 0xe7400000-0xe74fffff 64bit pref]
[    0.518600] pci_bus 0000:0e: resource 1 [mem 0xfc700000-0xfc7fffff]
[    0.518600] pci_bus 0000:0e: resource 2 [mem 0xe7000000-0xe70fffff 64bit pref]
[    0.518601] pci_bus 0000:0f: resource 1 [mem 0xfc300000-0xfc3fffff]
[    0.518602] pci_bus 0000:0f: resource 2 [mem 0xe6c00000-0xe6cfffff 64bit pref]
[    0.518602] pci_bus 0000:10: resource 1 [mem 0xfbf00000-0xfbffffff]
[    0.518603] pci_bus 0000:10: resource 2 [mem 0xe6800000-0xe68fffff 64bit pref]
[    0.518604] pci_bus 0000:11: resource 1 [mem 0xfbb00000-0xfbbfffff]
[    0.518605] pci_bus 0000:11: resource 2 [mem 0xe6400000-0xe64fffff 64bit pref]
[    0.518605] pci_bus 0000:12: resource 1 [mem 0xfb700000-0xfb7fffff]
[    0.518606] pci_bus 0000:12: resource 2 [mem 0xe6000000-0xe60fffff 64bit pref]
[    0.518607] pci_bus 0000:13: resource 0 [io  0x6000-0x6fff]
[    0.518608] pci_bus 0000:13: resource 1 [mem 0xfd200000-0xfd2fffff]
[    0.518608] pci_bus 0000:13: resource 2 [mem 0xc0400000-0xc05fffff 64bit pref]
[    0.518609] pci_bus 0000:14: resource 0 [io  0xa000-0xafff]
[    0.518610] pci_bus 0000:14: resource 1 [mem 0xfce00000-0xfcefffff]
[    0.518610] pci_bus 0000:14: resource 2 [mem 0xe7700000-0xe77fffff 64bit pref]
[    0.518611] pci_bus 0000:15: resource 0 [io  0xe000-0xefff]
[    0.518612] pci_bus 0000:15: resource 1 [mem 0xfca00000-0xfcafffff]
[    0.518613] pci_bus 0000:15: resource 2 [mem 0xe7300000-0xe73fffff 64bit pref]
[    0.518613] pci_bus 0000:16: resource 1 [mem 0xfc600000-0xfc6fffff]
[    0.518614] pci_bus 0000:16: resource 2 [mem 0xe6f00000-0xe6ffffff 64bit pref]
[    0.518615] pci_bus 0000:17: resource 1 [mem 0xfc200000-0xfc2fffff]
[    0.518615] pci_bus 0000:17: resource 2 [mem 0xe6b00000-0xe6bfffff 64bit pref]
[    0.518616] pci_bus 0000:18: resource 1 [mem 0xfbe00000-0xfbefffff]
[    0.518617] pci_bus 0000:18: resource 2 [mem 0xe6700000-0xe67fffff 64bit pref]
[    0.518618] pci_bus 0000:19: resource 1 [mem 0xfba00000-0xfbafffff]
[    0.518618] pci_bus 0000:19: resource 2 [mem 0xe6300000-0xe63fffff 64bit pref]
[    0.518619] pci_bus 0000:1a: resource 1 [mem 0xfb600000-0xfb6fffff]
[    0.518620] pci_bus 0000:1a: resource 2 [mem 0xe5f00000-0xe5ffffff 64bit pref]
[    0.518620] pci_bus 0000:1b: resource 0 [io  0x7000-0x7fff]
[    0.518621] pci_bus 0000:1b: resource 1 [mem 0xfd100000-0xfd1fffff]
[    0.518622] pci_bus 0000:1b: resource 2 [mem 0xe7a00000-0xe7afffff 64bit pref]
[    0.518623] pci_bus 0000:1c: resource 0 [io  0xb000-0xbfff]
[    0.518623] pci_bus 0000:1c: resource 1 [mem 0xfcd00000-0xfcdfffff]
[    0.518624] pci_bus 0000:1c: resource 2 [mem 0xe7600000-0xe76fffff 64bit pref]
[    0.518625] pci_bus 0000:1d: resource 1 [mem 0xfc900000-0xfc9fffff]
[    0.518625] pci_bus 0000:1d: resource 2 [mem 0xe7200000-0xe72fffff 64bit pref]
[    0.518626] pci_bus 0000:1e: resource 1 [mem 0xfc500000-0xfc5fffff]
[    0.518627] pci_bus 0000:1e: resource 2 [mem 0xe6e00000-0xe6efffff 64bit pref]
[    0.518628] pci_bus 0000:1f: resource 1 [mem 0xfc100000-0xfc1fffff]
[    0.518628] pci_bus 0000:1f: resource 2 [mem 0xe6a00000-0xe6afffff 64bit pref]
[    0.518629] pci_bus 0000:20: resource 1 [mem 0xfbd00000-0xfbdfffff]
[    0.518630] pci_bus 0000:20: resource 2 [mem 0xe6600000-0xe66fffff 64bit pref]
[    0.518631] pci_bus 0000:21: resource 1 [mem 0xfb900000-0xfb9fffff]
[    0.518631] pci_bus 0000:21: resource 2 [mem 0xe6200000-0xe62fffff 64bit pref]
[    0.518632] pci_bus 0000:22: resource 1 [mem 0xfb500000-0xfb5fffff]
[    0.518633] pci_bus 0000:22: resource 2 [mem 0xe5e00000-0xe5efffff 64bit pref]
[    0.518835] NET: Registered protocol family 2
[    0.519637] IP idents hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.520827] tcp_listen_portaddr_hash hash table entries: 8192 (order: 5, 131072 bytes, linear)
[    0.521331] TCP established hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.521838] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    0.521918] TCP: Hash tables configured (established 131072 bind 65536)
[    0.522082] UDP hash table entries: 8192 (order: 6, 262144 bytes, linear)
[    0.522201] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes, linear)
[    0.522246] NET: Registered protocol family 1
[    0.522249] NET: Registered protocol family 44
[    0.522252] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[    0.522323] pci 0000:00:0f.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    0.522587] PCI: CLS 32 bytes, default 64
[    0.522622] Trying to unpack rootfs image as initramfs...
[    0.772736] Freeing initrd memory: 84392K
[    0.772761] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    0.772763] software IO TLB: mapped [mem 0xbbee0000-0xbfee0000] (64MB)
[    0.772955] check: Scanning for low memory corruption every 60 seconds
[    0.773307] Initialise system trusted keyrings
[    0.773314] Key type blacklist registered
[    0.773349] workingset: timestamp_bits=36 max_order=22 bucket_order=0
[    0.774242] zbud: loaded
[    0.774495] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.774600] fuse: init (API version 7.31)
[    0.774615] *** VALIDATE fuse ***
[    0.774616] *** VALIDATE fuse ***
[    0.774683] Platform Keyring initialized
[    0.775755] Key type asymmetric registered
[    0.775756] Asymmetric key parser 'x509' registered
[    0.775762] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 244)
[    0.775786] io scheduler mq-deadline registered
[    0.776318] pcieport 0000:00:15.0: PME: Signaling with IRQ 24
[    0.776385] pcieport 0000:00:15.0: pciehp: Slot #160 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.776986] pcieport 0000:00:15.1: PME: Signaling with IRQ 25
[    0.777042] pcieport 0000:00:15.1: pciehp: Slot #161 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.777657] pcieport 0000:00:15.2: PME: Signaling with IRQ 26
[    0.777725] pcieport 0000:00:15.2: pciehp: Slot #162 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.778304] pcieport 0000:00:15.3: PME: Signaling with IRQ 27
[    0.778358] pcieport 0000:00:15.3: pciehp: Slot #163 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.778933] pcieport 0000:00:15.4: PME: Signaling with IRQ 28
[    0.778988] pcieport 0000:00:15.4: pciehp: Slot #164 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.779557] pcieport 0000:00:15.5: PME: Signaling with IRQ 29
[    0.779611] pcieport 0000:00:15.5: pciehp: Slot #165 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.780197] pcieport 0000:00:15.6: PME: Signaling with IRQ 30
[    0.780251] pcieport 0000:00:15.6: pciehp: Slot #166 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.780860] pcieport 0000:00:15.7: PME: Signaling with IRQ 31
[    0.780915] pcieport 0000:00:15.7: pciehp: Slot #167 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.781486] pcieport 0000:00:16.0: PME: Signaling with IRQ 32
[    0.781543] pcieport 0000:00:16.0: pciehp: Slot #192 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.782110] pcieport 0000:00:16.1: PME: Signaling with IRQ 33
[    0.782170] pcieport 0000:00:16.1: pciehp: Slot #193 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.782754] pcieport 0000:00:16.2: PME: Signaling with IRQ 34
[    0.782809] pcieport 0000:00:16.2: pciehp: Slot #194 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.783376] pcieport 0000:00:16.3: PME: Signaling with IRQ 35
[    0.783432] pcieport 0000:00:16.3: pciehp: Slot #195 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.784046] pcieport 0000:00:16.4: PME: Signaling with IRQ 36
[    0.784101] pcieport 0000:00:16.4: pciehp: Slot #196 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.784671] pcieport 0000:00:16.5: PME: Signaling with IRQ 37
[    0.784725] pcieport 0000:00:16.5: pciehp: Slot #197 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.785298] pcieport 0000:00:16.6: PME: Signaling with IRQ 38
[    0.785353] pcieport 0000:00:16.6: pciehp: Slot #198 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.785930] pcieport 0000:00:16.7: PME: Signaling with IRQ 39
[    0.785985] pcieport 0000:00:16.7: pciehp: Slot #199 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.786565] pcieport 0000:00:17.0: PME: Signaling with IRQ 40
[    0.786622] pcieport 0000:00:17.0: pciehp: Slot #224 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.787189] pcieport 0000:00:17.1: PME: Signaling with IRQ 41
[    0.787245] pcieport 0000:00:17.1: pciehp: Slot #225 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.787819] pcieport 0000:00:17.2: PME: Signaling with IRQ 42
[    0.787874] pcieport 0000:00:17.2: pciehp: Slot #226 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.788452] pcieport 0000:00:17.3: PME: Signaling with IRQ 43
[    0.788506] pcieport 0000:00:17.3: pciehp: Slot #227 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.789082] pcieport 0000:00:17.4: PME: Signaling with IRQ 44
[    0.789137] pcieport 0000:00:17.4: pciehp: Slot #228 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.789711] pcieport 0000:00:17.5: PME: Signaling with IRQ 45
[    0.789766] pcieport 0000:00:17.5: pciehp: Slot #229 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.790347] pcieport 0000:00:17.6: PME: Signaling with IRQ 46
[    0.790402] pcieport 0000:00:17.6: pciehp: Slot #230 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.790972] pcieport 0000:00:17.7: PME: Signaling with IRQ 47
[    0.791026] pcieport 0000:00:17.7: pciehp: Slot #231 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.791604] pcieport 0000:00:18.0: PME: Signaling with IRQ 48
[    0.791661] pcieport 0000:00:18.0: pciehp: Slot #256 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.792239] pcieport 0000:00:18.1: PME: Signaling with IRQ 49
[    0.792293] pcieport 0000:00:18.1: pciehp: Slot #257 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.792870] pcieport 0000:00:18.2: PME: Signaling with IRQ 50
[    0.792925] pcieport 0000:00:18.2: pciehp: Slot #258 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.793494] pcieport 0000:00:18.3: PME: Signaling with IRQ 51
[    0.793550] pcieport 0000:00:18.3: pciehp: Slot #259 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.794132] pcieport 0000:00:18.4: PME: Signaling with IRQ 52
[    0.794187] pcieport 0000:00:18.4: pciehp: Slot #260 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.794799] pcieport 0000:00:18.5: PME: Signaling with IRQ 53
[    0.794855] pcieport 0000:00:18.5: pciehp: Slot #261 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.795430] pcieport 0000:00:18.6: PME: Signaling with IRQ 54
[    0.795488] pcieport 0000:00:18.6: pciehp: Slot #262 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.796098] pcieport 0000:00:18.7: PME: Signaling with IRQ 55
[    0.796155] pcieport 0000:00:18.7: pciehp: Slot #263 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl+ LLActRep+
[    0.796453] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    0.796499] vesafb: mode is 640x480x32, linelength=2560, pages=0
[    0.796499] vesafb: scrolling: redraw
[    0.796501] vesafb: Truecolor: size=8:8:8:8, shift=24:16:8:0
[    0.796511] vesafb: framebuffer at 0xe8000000, mapped to 0x0000000089548c03, using 1216k, total 1216k
[    0.796541] fbcon: Deferring console take-over
[    0.796541] fb0: VESA VGA frame buffer device
[    0.796817] ACPI: AC Adapter [ACAD] (on-line)
[    0.796867] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[    0.796896] ACPI: Power Button [PWRF]
[    0.797304] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[    0.798488] Linux agpgart interface v0.103
[    0.798596] agpgart-intel 0000:00:00.0: Intel 440BX Chipset
[    0.798889] agpgart-intel 0000:00:00.0: AGP aperture is 256M @ 0x0
[    0.927992] loop: module loaded
[    0.928123] ata_piix 0000:00:07.1: version 2.13
[    0.928406] scsi host0: ata_piix
[    0.928471] scsi host1: ata_piix
[    0.928495] ata1: PATA max UDMA/33 cmd 0x1f0 ctl 0x3f6 bmdma 0x1060 irq 14
[    0.928496] ata2: PATA max UDMA/33 cmd 0x170 ctl 0x376 bmdma 0x1068 irq 15
[    0.928572] tun: Universal TUN/TAP device driver, 1.6
[    0.928757] PPP generic driver version 2.4.2
[    0.928787] VFIO - User Level meta-driver version: 0.3
[    0.928949] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.928951] ehci-pci: EHCI PCI platform driver
[    0.928958] ehci-platform: EHCI generic platform driver
[    0.928962] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.928964] ohci-pci: OHCI PCI platform driver
[    0.928970] ohci-platform: OHCI generic platform driver
[    0.928973] uhci_hcd: USB Universal Host Controller Interface driver
[    0.929022] i8042: PNP: PS/2 Controller [PNP0303:KBC,PNP0f13:MOUS] at 0x60,0x64 irq 1,12
[    0.929522] serio: i8042 KBD port at 0x60,0x64 irq 1
[    0.929526] serio: i8042 AUX port at 0x60,0x64 irq 12
[    0.929599] mousedev: PS/2 mouse device common for all mice
[    0.929949] rtc_cmos 00:01: registered as rtc0
[    0.929969] rtc_cmos 00:01: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
[    0.929973] i2c /dev entries driver
[    0.929997] device-mapper: uevent: version 1.0.3
[    0.930030] device-mapper: ioctl: 4.41.0-ioctl (2019-09-16) initialised: dm-devel@redhat.com
[    0.930042] platform eisa.0: Probing EISA bus 0
[    0.930043] platform eisa.0: EISA: Cannot allocate resource for mainboard
[    0.930044] platform eisa.0: Cannot allocate resource for EISA slot 1
[    0.930045] platform eisa.0: Cannot allocate resource for EISA slot 2
[    0.930045] platform eisa.0: Cannot allocate resource for EISA slot 3
[    0.930046] platform eisa.0: Cannot allocate resource for EISA slot 4
[    0.930047] platform eisa.0: Cannot allocate resource for EISA slot 5
[    0.930047] platform eisa.0: Cannot allocate resource for EISA slot 6
[    0.930048] platform eisa.0: Cannot allocate resource for EISA slot 7
[    0.930048] platform eisa.0: Cannot allocate resource for EISA slot 8
[    0.930049] platform eisa.0: EISA: Detected 0 cards
[    0.930168] ledtrig-cpu: registered to indicate activity on CPUs
[    0.930250] drop_monitor: Initializing network drop monitor service
[    0.930360] NET: Registered protocol family 10
[    0.932034] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
[    0.934754] Segment Routing with IPv6
[    0.934769] NET: Registered protocol family 17
[    0.934795] Key type dns_resolver registered
[    0.935014] RAS: Correctable Errors collector initialized.
[    0.935019] IPI shorthand broadcast: enabled
[    0.935030] sched_clock: Marking stable (933482469, 1449730)->(931497100, 3435099)
[    0.935122] registered taskstats version 1
[    0.935134] Loading compiled-in X.509 certificates
[    0.935593] Loaded X.509 cert 'Build time autogenerated kernel key: 166555f2d1679971571dbaea8993cd87f322a585'
[    0.936021] Loaded X.509 cert 'Canonical Ltd. Live Patch Signing: 14df34d1a87cf37625abec039ef2bf521249b969'
[    0.936436] Loaded X.509 cert 'Canonical Ltd. Kernel Module Signing: 88f752e560a1e0737e31163a466ad7b70a850c19'
[    0.936437] blacklist: Loading compiled-in revocation X.509 certificates
[    0.936451] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing: 61482aa2830d0ab2ad5af10b7250da9033ddcef0'
[    0.936461] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (2017): 242ade75ac4a15e50d50c84b0d45ff3eae707a03'
[    0.936471] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (ESM 2018): 365188c1d374d6b07c3c8f240f8ef722433d6a8b'
[    0.936484] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (2019): c0746fd6c5da3ae827864651ad66ae47fe24b3e8'
[    0.936494] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (2021 v1): a8d54bbb3825cfb94fa13c9f8a594a195c107b8d'
[    0.936504] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (2021 v2): 4cf046892d6fd3c9a5b03f98d845f90851dc6a8c'
[    0.936515] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (2021 v3): 100437bb6de6e469b581e61cd66bce3ef4ed53af'
[    0.936526] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (Ubuntu Core 2019): c1d57b8f6b743f23ee41f4f7ee292f06eecadfb9'
[    0.936541] zswap: loaded using pool lzo/zbud
[    0.936625] Key type ._fscrypt registered
[    0.936626] Key type .fscrypt registered
[    0.939714] Key type big_key registered
[    0.941255] Key type encrypted registered
[    0.941257] AppArmor: AppArmor sha1 policy hashing enabled
[    0.941262] ima: No TPM chip found, activating TPM-bypass!
[    0.941265] ima: Allocated hash algorithm: sha1
[    0.941270] ima: No architecture policies found
[    0.941282] evm: Initialising EVM extended attributes:
[    0.941282] evm: security.selinux
[    0.941282] evm: security.SMACK64
[    0.941283] evm: security.SMACK64EXEC
[    0.941283] evm: security.SMACK64TRANSMUTE
[    0.941283] evm: security.SMACK64MMAP
[    0.941283] evm: security.apparmor
[    0.941284] evm: security.ima
[    0.941284] evm: security.capability
[    0.941284] evm: HMAC attrs: 0x1
[    0.942563] PM:   Magic number: 15:705:989
[    0.942682] memory memory32: hash matches
[    0.942774] rtc_cmos 00:01: setting system clock to 2023-07-25T13:58:59 UTC (1690293539)
[    1.109705] Freeing unused decrypted memory: 2040K
[    1.110052] Freeing unused kernel image memory: 2764K
[    1.110054] Write protecting the kernel read-only data: 26624k
[    1.110452] Freeing unused kernel image memory: 2036K
[    1.110575] Freeing unused kernel image memory: 740K
[    1.119167] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    1.119168] Run /init as init process
[    1.294354] piix4_smbus 0000:00:07.3: SMBus Host Controller not enabled!
[    1.294366] fbcon: Taking over console
[    1.294418] Console: switching to colour frame buffer device 80x30
[    1.306491] Fusion MPT base driver 3.04.20
[    1.306492] Copyright (c) 1999-2008 LSI Corporation
[    1.310148] ahci 0000:02:01.0: version 3.0
[    1.311159] ahci 0000:02:01.0: AHCI 0001.0300 32 slots 30 ports 6 Gbps 0x3fffffff impl SATA mode
[    1.311161] ahci 0000:02:01.0: flags: 64bit ncq clo only 
[    1.316271] VMware vmxnet3 virtual NIC driver - version 1.4.17.0-k-NAPI
[    1.316299] vmxnet3 0000:03:00.0: # of Tx queues : 2, # of Rx queues : 2
[    1.317430] vmxnet3 0000:03:00.0 eth0: NIC Link is Up 10000 Mbps
[    1.317462] vmxnet3 0000:0b:00.0: # of Tx queues : 2, # of Rx queues : 2
[    1.318370] vmxnet3 0000:0b:00.0 eth1: NIC Link is Up 10000 Mbps
[    1.318397] vmxnet3 0000:13:00.0: # of Tx queues : 2, # of Rx queues : 2
[    1.319365] vmxnet3 0000:13:00.0 eth2: NIC Link is Up 10000 Mbps
[    1.333116] scsi host2: ahci
[    1.336905] Fusion MPT SPI Host driver 3.04.20
[    1.340126] scsi host3: ahci
[    1.344875] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4
[    1.345031] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3
[    1.345094] mptbase: ioc0: Initiating bringup
[    1.348026] scsi host4: ahci
[    1.355710] scsi host5: ahci
[    1.359041] cryptd: max_cpu_qlen set to 1000
[    1.362785] AVX2 version of gcm_enc/dec engaged.
[    1.362786] AES CTR mode by8 optimization enabled
[    1.362806] scsi host6: ahci
[    1.373252] scsi host7: ahci
[    1.381118] scsi host8: ahci
[    1.392011] scsi host9: ahci
[    1.398929] scsi host10: ahci
[    1.405644] scsi host11: ahci
[    1.405823] [drm] DMA map mode: Caching DMA mappings.
[    1.405950] [drm] Capabilities:
[    1.405950] [drm]   Rect copy.
[    1.405950] [drm]   Cursor.
[    1.405950] [drm]   Cursor bypass.
[    1.405950] [drm]   Cursor bypass 2.
[    1.405951] [drm]   8bit emulation.
[    1.405951] [drm]   Alpha cursor.
[    1.405951] [drm]   Extended Fifo.
[    1.405951] [drm]   Multimon.
[    1.405951] [drm]   Pitchlock.
[    1.405952] [drm]   Irq mask.
[    1.405952] [drm]   Display Topology.
[    1.405952] [drm]   GMR.
[    1.405952] [drm]   Traces.
[    1.405952] [drm]   GMR2.
[    1.405953] [drm]   Screen Object 2.
[    1.405953] [drm]   Command Buffers.
[    1.405953] [drm]   Command Buffers 2.
[    1.405953] [drm]   Guest Backed Resources.
[    1.405953] [drm]   DX Features.
[    1.405954] [drm]   HP Command Queue.
[    1.405954] [drm] Capabilities2:
[    1.405954] [drm]   Grow oTable.
[    1.405954] [drm]   IntraSurface copy.
[    1.405955] [drm] Max GMR ids is 64
[    1.405955] [drm] Max number of GMR pages is 65536
[    1.405956] [drm] Max dedicated hypervisor surface memory is 0 kiB
[    1.405956] [drm] Maximum display memory size is 4096 kiB
[    1.405956] [drm] VRAM at 0xe8000000 size is 4096 kiB
[    1.405957] [drm] MMIO at 0xfe000000 size is 256 kiB
[    1.405981] vmwgfx 0000:00:0f.0: BAR 1: can't reserve [mem 0xe8000000-0xefffffff pref]
[    1.405982] [drm] It appears like vesafb is loaded. Ignore above error if any.
[    1.416010] scsi host12: ahci
[    1.416094] [TTM] Zone  kernel: Available graphics memory: 8196980 KiB
[    1.416095] [TTM] Zone   dma32: Available graphics memory: 2097152 KiB
[    1.416095] [TTM] Initializing pool allocator
[    1.416098] [TTM] Initializing DMA pool allocator
[    1.416152] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[    1.416153] [drm] No driver support for vblank timestamp query.
[    1.416403] [drm] Screen Target Display device initialized
[    1.416666] [drm] width 640
[    1.416670] [drm] height 480
[    1.416673] [drm] bpp 32
[    1.417524] [drm] Fifo max 0x00040000 min 0x00001000 cap 0x0000077f
[    1.420257] [drm] Using command buffers with DMA pool.
[    1.422429] [drm] DX: no.
[    1.422430] [drm] Atomic: yes.
[    1.422430] [drm] SM4_1: no.
[    1.426826] scsi host13: ahci
[    1.439083] scsi host14: ahci
[    1.439867] checking generic (e8000000 130000) vs hw (e8000000 400000)
[    1.439868] fb0: switching to svgadrmfb from VESA VGA
[    1.439918] Console: switching to colour dummy device 80x25
[    1.439994] fbcon: svgadrmfb (fb0) is primary device
[    1.441756] Console: switching to colour frame buffer device 100x37
[    1.445168] [drm] Initialized vmwgfx 2.15.0 20180704 for 0000:00:0f.0 on minor 0
[    1.447986] scsi host15: ahci
[    1.450257] scsi host16: ahci
[    1.450442] scsi host17: ahci
[    1.450615] scsi host18: ahci
[    1.450850] scsi host19: ahci
[    1.451063] scsi host20: ahci
[    1.451273] scsi host21: ahci
[    1.451515] scsi host22: ahci
[    1.451756] scsi host23: ahci
[    1.452213] ioc0: LSI53C1030 B0: Capabilities={Initiator}
[    1.452570] scsi host24: ahci
[    1.452800] scsi host25: ahci
[    1.453041] scsi host26: ahci
[    1.453260] scsi host27: ahci
[    1.453464] scsi host28: ahci
[    1.453667] scsi host29: ahci
[    1.453935] scsi host30: ahci
[    1.454159] scsi host31: ahci
[    1.454230] ata3: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ff100 irq 56
[    1.454231] ata4: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ff180 irq 56
[    1.454232] ata5: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ff200 irq 56
[    1.454233] ata6: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ff280 irq 56
[    1.454233] ata7: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ff300 irq 56
[    1.454234] ata8: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ff380 irq 56
[    1.454235] ata9: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ff400 irq 56
[    1.454236] ata10: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ff480 irq 56
[    1.454236] ata11: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ff500 irq 56
[    1.454237] ata12: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ff580 irq 56
[    1.454238] ata13: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ff600 irq 56
[    1.454239] ata14: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ff680 irq 56
[    1.454240] ata15: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ff700 irq 56
[    1.454240] ata16: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ff780 irq 56
[    1.454241] ata17: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ff800 irq 56
[    1.454242] ata18: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ff880 irq 56
[    1.454242] ata19: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ff900 irq 56
[    1.454243] ata20: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ff980 irq 56
[    1.454244] ata21: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ffa00 irq 56
[    1.454244] ata22: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ffa80 irq 56
[    1.454245] ata23: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ffb00 irq 56
[    1.454246] ata24: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ffb80 irq 56
[    1.454246] ata25: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ffc00 irq 56
[    1.454247] ata26: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ffc80 irq 56
[    1.454248] ata27: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ffd00 irq 56
[    1.454249] ata28: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ffd80 irq 56
[    1.454249] ata29: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ffe00 irq 56
[    1.454250] ata30: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5ffe80 irq 56
[    1.454251] ata31: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5fff00 irq 56
[    1.454251] ata32: SATA max UDMA/133 abar m4096@0xfd5ff000 port 0xfd5fff80 irq 56
[    1.692652] scsi host32: ioc0: LSI53C1030 B0, FwRev=01032920h, Ports=1, MaxQ=128, IRQ=17
[    1.766345] ata3: SATA link down (SStatus 0 SControl 300)
[    1.770591] ata14: SATA link down (SStatus 0 SControl 300)
[    1.770618] ata17: SATA link down (SStatus 0 SControl 300)
[    1.770627] ata18: SATA link down (SStatus 0 SControl 300)
[    1.770635] ata23: SATA link down (SStatus 0 SControl 300)
[    1.770643] ata6: SATA link down (SStatus 0 SControl 300)
[    1.770652] ata21: SATA link down (SStatus 0 SControl 300)
[    1.770660] ata11: SATA link down (SStatus 0 SControl 300)
[    1.770668] ata16: SATA link down (SStatus 0 SControl 300)
[    1.770677] ata31: SATA link down (SStatus 0 SControl 300)
[    1.770685] ata20: SATA link down (SStatus 0 SControl 300)
[    1.770693] ata30: SATA link down (SStatus 0 SControl 300)
[    1.770702] ata26: SATA link down (SStatus 0 SControl 300)
[    1.770710] ata10: SATA link down (SStatus 0 SControl 300)
[    1.770718] ata13: SATA link down (SStatus 0 SControl 300)
[    1.770726] ata29: SATA link down (SStatus 0 SControl 300)
[    1.770734] ata25: SATA link down (SStatus 0 SControl 300)
[    1.770742] ata7: SATA link down (SStatus 0 SControl 300)
[    1.770750] ata19: SATA link down (SStatus 0 SControl 300)
[    1.770759] ata24: SATA link down (SStatus 0 SControl 300)
[    1.770767] ata15: SATA link down (SStatus 0 SControl 300)
[    1.770775] ata4: SATA link down (SStatus 0 SControl 300)
[    1.770801] ata9: SATA link down (SStatus 0 SControl 300)
[    1.770809] ata5: SATA link down (SStatus 0 SControl 300)
[    1.770853] ata12: SATA link down (SStatus 0 SControl 300)
[    1.770862] ata28: SATA link down (SStatus 0 SControl 300)
[    1.770870] ata32: SATA link down (SStatus 0 SControl 300)
[    1.770878] ata27: SATA link down (SStatus 0 SControl 300)
[    1.770886] ata22: SATA link down (SStatus 0 SControl 300)
[    1.770894] ata8: SATA link down (SStatus 0 SControl 300)
[    1.803996] tsc: Refined TSC clocksource calibration: 2350.000 MHz
[    1.804002] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x21dfb64b751, max_idle_ns: 440795265875 ns
[    1.804055] clocksource: Switched to clocksource tsc
[    1.860219] scsi 32:0:0:0: Direct-Access     VMware   Virtual disk     2.0  PQ: 0 ANSI: 6
[    1.875977] scsi target32:0:0: Beginning Domain Validation
[    1.876529] scsi target32:0:0: Domain Validation skipping write tests
[    1.876530] scsi target32:0:0: Ending Domain Validation
[    1.876556] scsi target32:0:0: FAST-80 WIDE SCSI 160.0 MB/s DT (12.5 ns, offset 127)
[    1.877450] sd 32:0:0:0: Attached scsi generic sg0 type 0
[    1.877550] sd 32:0:0:0: [sda] 524288000 512-byte logical blocks: (268 GB/250 GiB)
[    1.877567] sd 32:0:0:0: [sda] Write Protect is off
[    1.877568] sd 32:0:0:0: [sda] Mode Sense: 31 00 00 00
[    1.877583] sd 32:0:0:0: [sda] Cache data unavailable
[    1.877584] sd 32:0:0:0: [sda] Assuming drive cache: write through
[    1.894060]  sda: sda1 sda2 sda3
[    1.908284] sd 32:0:0:0: [sda] Attached SCSI disk
[    2.083969] raid6: avx2x4   gen() 29040 MB/s
[    2.131969] raid6: avx2x4   xor() 15373 MB/s
[    2.179980] raid6: avx2x2   gen() 29363 MB/s
[    2.227970] raid6: avx2x2   xor() 18245 MB/s
[    2.275971] raid6: avx2x1   gen() 16268 MB/s
[    2.323971] raid6: avx2x1   xor() 15233 MB/s
[    2.371971] raid6: sse2x4   gen() 14875 MB/s
[    2.419970] raid6: sse2x4   xor()  8635 MB/s
[    2.467974] raid6: sse2x2   gen() 14475 MB/s
[    2.515971] raid6: sse2x2   xor()  9335 MB/s
[    2.563976] raid6: sse2x1   gen()  6869 MB/s
[    2.611972] raid6: sse2x1   xor()  7631 MB/s
[    2.611973] raid6: using algorithm avx2x2 gen() 29363 MB/s
[    2.611973] raid6: .... xor() 18245 MB/s, rmw enabled
[    2.611974] raid6: using avx2x2 recovery algorithm
[    2.613131] xor: automatically using best checksumming function   avx       
[    2.613919] async_tx: api initialized (async)
[    2.697469] Btrfs loaded, crc32c=crc32c-intel
[    2.734459] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
[    2.849761] systemd[1]: Inserted module 'autofs4'
[    2.861586] systemd[1]: systemd 245.4-4ubuntu3.15 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
[    2.861605] systemd[1]: Detected virtualization vmware.
[    2.861610] systemd[1]: Detected architecture x86-64.
[    2.880981] systemd[1]: Set hostname to <itrinegy>.
[    3.036053] systemd[1]: multi-user.target: Found ordering cycle on getty.target/start
[    3.036055] systemd[1]: multi-user.target: Found dependency on getty@tty1.service/start
[    3.036057] systemd[1]: multi-user.target: Found dependency on cloud-init.target/start
[    3.036058] systemd[1]: multi-user.target: Found dependency on cloud-final.service/start
[    3.036059] systemd[1]: multi-user.target: Found dependency on multi-user.target/start
[    3.036061] systemd[1]: multi-user.target: Job getty.target/start deleted to break ordering cycle starting with multi-user.target/start
[    3.037393] systemd[1]: Created slice system-modprobe.slice.
[    3.037592] systemd[1]: Created slice system-systemd\x2dfsck.slice.
[    3.037785] systemd[1]: Created slice User and Session Slice.
[    3.037824] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    3.038018] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[    3.038050] systemd[1]: Reached target User and Group Name Lookups.
[    3.038065] systemd[1]: Reached target Slices.
[    3.038112] systemd[1]: Listening on Device-mapper event daemon FIFOs.
[    3.038167] systemd[1]: Listening on LVM2 poll daemon socket.
[    3.038198] systemd[1]: Listening on multipathd control socket.
[    3.038243] systemd[1]: Listening on Syslog Socket.
[    3.038288] systemd[1]: Listening on fsck to fsckd communication Socket.
[    3.038320] systemd[1]: Listening on initctl Compatibility Named Pipe.
[    3.038418] systemd[1]: Listening on Journal Audit Socket.
[    3.038465] systemd[1]: Listening on Journal Socket (/dev/log).
[    3.038518] systemd[1]: Listening on Journal Socket.
[    3.038601] systemd[1]: Listening on Network Service Netlink Socket.
[    3.038665] systemd[1]: Listening on udev Control Socket.
[    3.038708] systemd[1]: Listening on udev Kernel Socket.
[    3.039340] systemd[1]: Mounting Huge Pages File System...
[    3.040061] systemd[1]: Mounting POSIX Message Queue File System...
[    3.040757] systemd[1]: Mounting Kernel Debug File System...
[    3.041355] systemd[1]: Mounting Kernel Trace File System...
[    3.042428] systemd[1]: Starting Journal Service...
[    3.043126] systemd[1]: Starting Set the console keyboard layout...
[    3.044073] systemd[1]: Starting Create list of static device nodes for the current kernel...
[    3.056221] systemd[1]: Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[    3.056298] systemd[1]: Condition check resulted in Load Kernel Module drm being skipped.
[    3.056370] systemd[1]: Condition check resulted in OpenVSwitch configuration for cleanup being skipped.
[    3.056982] systemd[1]: Condition check resulted in Set Up Additional Binary Formats being skipped.
[    3.057019] systemd[1]: Condition check resulted in File System Check on Root Device being skipped.
[    3.058415] systemd[1]: Starting Load Kernel Modules...
[    3.059153] systemd[1]: Starting Remount Root and Kernel File Systems...
[    3.059912] systemd[1]: Starting udev Coldplug all Devices...
[    3.064152] systemd[1]: Starting Uncomplicated firewall...
[    3.065591] systemd[1]: Mounted Kernel Debug File System.
[    3.065687] systemd[1]: Mounted Kernel Trace File System.
[    3.078976] systemd[1]: Finished Uncomplicated firewall.
[    3.080383] systemd[1]: Mounted POSIX Message Queue File System.
[    3.081197] systemd[1]: Finished Create list of static device nodes for the current kernel.
[    3.081930] systemd[1]: Mounted Huge Pages File System.
[    3.093915] EXT4-fs (dm-0): re-mounted. Opts: (null)
[    3.097856] systemd[1]: Finished Remount Root and Kernel File Systems.
[    3.098492] systemd[1]: Activating swap /swap.img...
[    3.098946] systemd[1]: Condition check resulted in Rebuild Hardware Database being skipped.
[    3.098988] systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped.
[    3.099543] systemd[1]: Starting Load/Save Random Seed...
[    3.100135] systemd[1]: Starting Create System Users...
[    3.113004] systemd[1]: Finished Load Kernel Modules.
[    3.113664] systemd[1]: Mounting FUSE Control File System...
[    3.114290] systemd[1]: Mounting Kernel Configuration File System...
[    3.114862] systemd[1]: Starting Apply Kernel Variables...
[    3.120837] systemd[1]: Mounted Kernel Configuration File System.
[    3.121000] systemd[1]: Mounted FUSE Control File System.
[    3.124216] systemd[1]: Finished Load/Save Random Seed.
[    3.130047] systemd[1]: Finished Create System Users.
[    3.130659] systemd[1]: Starting Create Static Device Nodes in /dev...
[    3.138864] systemd[1]: Finished Apply Kernel Variables.
[    3.141882] systemd[1]: Finished Create Static Device Nodes in /dev.
[    3.142573] systemd[1]: Starting udev Kernel Device Manager...
[    3.209066] systemd[1]: Finished Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
[    3.215638] systemd[1]: Started udev Kernel Device Manager.
[    3.332650] systemd[1]: Finished Set the console keyboard layout.
[    3.447869] systemd[1]: Finished udev Coldplug all Devices.
[    3.448660] systemd[1]: Starting Show Plymouth Boot Screen...
[    3.449250] systemd[1]: Starting udev Wait for Complete Device Initialization...
[    3.455512] systemd[1]: Received SIGRTMIN+20 from PID 539 (plymouthd).
[    3.506937] systemd[1]: Started Journal Service.
[    3.512427] systemd-journald[489]: Received client request to flush runtime journal.
[    3.536022] Adding 4194300k swap on /swap.img.  Priority:-2 extents:6 across:4612092k SSFS
[    4.043263] vmw_vmci 0000:00:07.7: Found VMCI PCI device at 0x11080, irq 16
[    4.043315] vmw_vmci 0000:00:07.7: Using capabilities 0xc
[    4.043786] Guest personality initialized and is active
[    4.043819] VMCI host device registered (name=vmci, major=10, minor=59)
[    4.043819] Initialized host personality
[    4.251099] Decoding supported only on Scalable MCA processors.
[    4.280418] Decoding supported only on Scalable MCA processors.
[    4.679676] alua: device handler registered
[    4.680625] emc: device handler registered
[    4.681578] rdac: device handler registered
[    4.774243] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null)
[    5.260326] audit: type=1400 audit(1690293543.813:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=728 comm="apparmor_parser"
[    5.260328] audit: type=1400 audit(1690293543.813:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=728 comm="apparmor_parser"
[    5.262678] audit: type=1400 audit(1690293543.813:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/man" pid=730 comm="apparmor_parser"
[    5.262680] audit: type=1400 audit(1690293543.813:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_filter" pid=730 comm="apparmor_parser"
[    5.262681] audit: type=1400 audit(1690293543.813:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_groff" pid=730 comm="apparmor_parser"
[    5.268559] audit: type=1400 audit(1690293543.821:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=731 comm="apparmor_parser"
[    5.271454] audit: type=1400 audit(1690293543.821:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine" pid=733 comm="apparmor_parser"
[    5.271457] audit: type=1400 audit(1690293543.821:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=733 comm="apparmor_parser"
[    5.284486] audit: type=1400 audit(1690293543.837:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/tcpdump" pid=734 comm="apparmor_parser"
[    5.286333] audit: type=1400 audit(1690293543.837:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=735 comm="apparmor_parser"
[    5.327223] NET: Registered protocol family 40
[    6.322613] vmxnet3 0000:03:00.0 eth0: intr type 3, mode 0, 3 vectors allocated
[    6.323802] vmxnet3 0000:03:00.0 eth0: NIC Link is Up 10000 Mbps
[    9.291810] aufs 5.4.3-20200302
[   11.385945] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   11.388894] Bridge firewalling registered
[   11.411452] bpfilter: Loaded bpfilter_umh pid 999
[   11.411615] Started bpfilter
[   11.541765] Initializing XFRM netlink socket
[   12.286258] cgroup: cgroup: disabling cgroup2 socket matching due to net_prio or net_cls activation
[   12.960351] docker0: port 1(veth05c9925) entered blocking state
[   12.960352] docker0: port 1(veth05c9925) entered disabled state
[   12.960408] device veth05c9925 entered promiscuous mode
[   13.364138] eth0: renamed from vethfcbc662
[   13.384341] IPv6: ADDRCONF(NETDEV_CHANGE): veth05c9925: link becomes ready
[   13.384372] docker0: port 1(veth05c9925) entered blocking state
[   13.384372] docker0: port 1(veth05c9925) entered forwarding state
[   13.384399] IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready
[   13.495881] docker0: port 1(veth05c9925) entered disabled state
[   13.497703] vethfcbc662: renamed from eth0
[   13.534009] docker0: port 1(veth05c9925) entered disabled state
[   13.537472] device veth05c9925 left promiscuous mode
[   13.537474] docker0: port 1(veth05c9925) entered disabled state
[  220.467885] vfio-pci: probe of 0000:13:00.0 failed with error -22
[  220.467898] vfio-pci: probe of 0000:13:00.0 failed with error -22
[  241.885079] vfio-pci: probe of 0000:13:00.0 failed with error -22
[  241.885095] vfio-pci: probe of 0000:13:00.0 failed with error -22
[  336.787422] vfio-pci: probe of 0000:13:00.0 failed with error -22
[  336.787437] vfio-pci: probe of 0000:13:00.0 failed with error -22
[  339.812164] vfio-pci: probe of 0000:13:00.0 failed with error -22
[  339.812178] vfio-pci: probe of 0000:13:00.0 failed with error -22
[  526.002419] vfio-pci: probe of 0000:13:00.0 failed with error -22
[  526.002433] vfio-pci: probe of 0000:13:00.0 failed with error -22
[  640.948189] vfio-pci: probe of 0000:13:00.0 failed with error -22
[  640.948203] vfio-pci: probe of 0000:13:00.0 failed with error -22
[  672.447306] vfio-pci: probe of 0000:0b:00.0 failed with error -22
[  672.447320] vfio-pci: probe of 0000:0b:00.0 failed with error -22
[  675.784756] vfio-pci: probe of 0000:0b:00.0 failed with error -22
[  675.784770] vfio-pci: probe of 0000:0b:00.0 failed with error -22
[  992.882236] vfio-pci: probe of 0000:13:00.0 failed with error -22
[  992.882249] vfio-pci: probe of 0000:13:00.0 failed with error -22
[ 1066.479352] vfio-pci: probe of 0000:13:00.0 failed with error -22
[ 1066.479365] vfio-pci: probe of 0000:13:00.0 failed with error -22
[ 1068.793441] vfio-pci: probe of 0000:13:00.0 failed with error -22
[ 1068.793455] vfio-pci: probe of 0000:13:00.0 failed with error -22
[ 1149.133518] vfio-pci: probe of 0000:13:00.0 failed with error -22
[ 1149.133532] vfio-pci: probe of 0000:13:00.0 failed with error -22
[ 1555.759278] rte_kni: loading out-of-tree module taints kernel.
[ 1555.759325] rte_kni: module verification failed: signature and/or required key missing - tainting kernel
[ 1557.358872] vfio-pci 0000:0b:00.0: Adding to iommu group 0
[ 1557.358873] vfio-pci 0000:0b:00.0: Adding kernel taint for vfio-noiommu group on device
[ 1557.532479] vfio-pci 0000:13:00.0: Adding to iommu group 1
[ 1557.532481] vfio-pci 0000:13:00.0: Adding kernel taint for vfio-noiommu group on device
[ 1564.753686] vfio-pci 0000:0b:00.0: vfio-noiommu device opened by user (ipperte:4433)
[ 1564.853139] vfio-pci 0000:13:00.0: vfio-noiommu device opened by user (ipperte:4433)
[ 1567.003856] rte_kni: Creating kni...
[ 1567.033334] rte_kni: Creating kni...
[ 1567.053129] rte_kni: Creating kni...
[ 1567.075882] rte_kni: Creating kni...
[ 1567.100853] rte_kni: Creating kni...
[ 1719.873710] vfio-pci 0000:0b:00.0: vfio-noiommu device opened by user (dpdk-testpmd:4753)
[ 1719.974169] vfio-pci 0000:13:00.0: vfio-noiommu device opened by user (dpdk-testpmd:4753)

[-- Attachment #3: logs2.eal --]
[-- Type: application/octet-stream, Size: 3499 bytes --]

EAL: Detected 2 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Some buses want 'VA' but forcing 'DC' because other buses want 'PA'.
EAL: Depending on the final decision by the EAL, not all buses may be able to initialize.
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Ignore mapping IO port bar(3)
EAL: Probe PCI driver: net_vmxnet3 (15ad:7b0) device: 0000:0b:00.0 (socket 0)
EAL: Ignore mapping IO port bar(3)
EAL: Probe PCI driver: net_vmxnet3 (15ad:7b0) device: 0000:13:00.0 (socket 0)
vdev_probe_all_drivers(): virtio_user0 requires VA IOVA mode but current mode is PA, not initializing
vdev_probe(): failed to initialize virtio_user0 device
EAL: Bus (vdev) probe failed.
testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Number of ports detected: 2
Configuring Port 0 (socket 0)
Port 0: 00:50:56:90:E8:FA
Configuring Port 1 (socket 0)
Port 1: 00:50:56:90:B4:3A
Checking link statuses...

Signal 2 received, preparing to exit...
Done
No commandline core given, start packet forwarding
io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP allocation mode: native
Logical Core 1 (socket 0) forwards packets on 2 streams:
  RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01
  RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

  io packet forwarding packets/burst=32
  nb forwarding cores=1 - nb forwarding ports=2
  port 0: RX queue number: 1 Tx queue number: 1
    Rx offloads=0x0 Tx offloads=0x0
    RX queue: 0
      RX desc=0 - RX free threshold=0
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=0 - TX free threshold=0
      TX threshold registers: pthresh=0 hthresh=0  wthresh=0
      TX offloads=0x0 - TX RS bit threshold=0
  port 1: RX queue number: 1 Tx queue number: 1
    Rx offloads=0x0 Tx offloads=0x0
    RX queue: 0
      RX desc=0 - RX free threshold=0
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=0 - TX free threshold=0
      TX threshold registers: pthresh=0 hthresh=0  wthresh=0
      TX offloads=0x0 - TX RS bit threshold=0
Press enter to exit
Telling cores to stop...
Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 2              RX-dropped: 0             RX-total: 2
  TX-packets: 2              TX-dropped: 0             TX-total: 2
  ----------------------------------------------------------------------------

  ---------------------- Forward statistics for port 1  ----------------------
  RX-packets: 2              RX-dropped: 0             RX-total: 2
  TX-packets: 2              TX-dropped: 0             TX-total: 2
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 4              RX-dropped: 0             RX-total: 4
  TX-packets: 4              TX-dropped: 0             TX-total: 4
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.

Stopping port 0...
Stopping ports...
Done

Stopping port 1...
Stopping ports...
Done

Shutting down port 0...
Closing ports...
Port 0 is closed
Done

Shutting down port 1...

RE: help

[Varghese, Vipin]

[-- Attachment #1: Type: text/plain, Size: 16439 bytes --]

[AMD Official Use Only - General]

Thanks Igor,

As suspected the vmx_net3 works with


  1.  Iommu: disabled
  2.  enable_unsafe_iommu: enabled
  3.  dpdk eal iova mode: PA

as pointed by you in logs, the virtio_user fails as it expects VA too.

Will check and get back.

From: Igor de Paula <igordptx@gmail.com>
Sent: Tuesday, July 25, 2023 8:16 PM
To: Yigit, Ferruh <Ferruh.Yigit@amd.com>
Cc: Jochen Behrens <jbehrens@vmware.com>; Thomas Monjalon <thomas@monjalon.net>; users@dpdk.org; Gupta, Nipun <Nipun.Gupta@amd.com>; Agarwal, Nikhil <nikhil.agarwal@amd.com>; Ronak Doshi <doshir@vmware.com>; Immanni, Venkat <Venkat.Immanni@amd.com>; Varghese, Vipin <Vipin.Varghese@amd.com>; Chenbo Xia <chenbo.xia@intel.com>
Subject: Re: help

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.

Hi,
Attaching the logs of EAL when trying to run a configuration with virtio_user port when IOMMU is
disabled and enable_unsafe_iommu is enabled. As you can see it forces IOVA as PA but the viritui_user needs IOVA as VA.
I am also attaching the output of dmesg. I am not sure which kernel logs you wanted... if there is anything else please let me know..
Regarding the ESXI logs, they are HUGE so I will send to you on a separate email.

On Fri, Jul 21, 2023 at 1:14 PM Ferruh Yigit <ferruh.yigit@amd.com<mailto:ferruh.yigit@amd.com>> wrote:
On 7/21/2023 12:39 PM, Igor de Paula wrote:
> I am trying to use virtio_user for an interface with the
> kernel: https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html>
> I think this requires IOVA as va.
>

I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo,
he may know better.

Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and
'--iova-mode=pa'?


> It does work with Intel host and IOMMU
> enabled. Part of the negotiation when setting it up is getting the IOMMU
> number so I thought it has to have IOMMU.
>

Yes, issue looks like related to the IOMMU, and it may be either related
to HW support, or ESXi iommu driver support, we will check using below
information you provided.

> I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but
> again, that didn't work.
> ESXI version - VMware ESXi, 7.0.0, 16324942
> AMD:  AMD EPYC 7452 32-Core Processo
>
> On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> Regarding the logs I will try and attach it soon.
>

Thanks for info, waiting for logs.

>
> On Fri, Jul 21, 2023 at 12:21 PM Ferruh Yigit <ferruh.yigit@amd.com<mailto:ferruh.yigit@amd.com>
> <mailto:ferruh.yigit@amd.com<mailto:ferruh.yigit@amd.com>>> wrote:
>
>     Hi Igor,
>
>     VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you
>     please try with:
>     - enable 'enable_unsafe_noiommu_mode' flag
>     - Force PA mode via '--iova-mode=pa' eal parameter
>
>
>     Also to be able to figure out AMD IOMMU support level, can you please
>     provide:
>     - AMD part number
>     - ESXi/hypervisor version
>     - ESXi & VM kernel logs
>
>
>     Thanks,
>     Ferruh
>
>     On 7/20/2023 5:21 PM, Jochen Behrens wrote:
>     > +Ronak from the ESX team
>     >
>     >
>     >
>     >
>     >
>     > In our usage, we do set amd_iommu=off in the boot command line from
>     > grub. (Or intel_iommu=off for Intel processors.)
>     >
>     >
>     >
>     >                 Jochen
>     >
>     >
>     >
>     > *From: *Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>
>     <mailto:thomas@monjalon.net<mailto:thomas@monjalon.net>>>
>     > *Date: *Thursday, July 20, 2023 at 6:00 AM
>     > *To: *Igor de Paula <igordptx@gmail.com<mailto:igordptx@gmail.com> <mailto:igordptx@gmail.com<mailto:igordptx@gmail.com>>>
>     > *Cc: *users@dpdk.org<mailto:users@dpdk.org> <mailto:users@dpdk.org<mailto:users@dpdk.org>> <users@dpdk.org<mailto:users@dpdk.org>
>     <mailto:users@dpdk.org<mailto:users@dpdk.org>>>, Jochen Behrens
>     > <jbehrens@vmware.com<mailto:jbehrens@vmware.com> <mailto:jbehrens@vmware.com<mailto:jbehrens@vmware.com>>>, Nipun Gupta
>     <nipun.gupta@amd.com<mailto:nipun.gupta@amd.com> <mailto:nipun.gupta@amd.com<mailto:nipun.gupta@amd.com>>>, Nikhil Agarwal
>     > <nikhil.agarwal@amd.com<mailto:nikhil.agarwal@amd.com> <mailto:nikhil.agarwal@amd.com<mailto:nikhil.agarwal@amd.com>>>, Ferruh
>     Yigit <ferruh.yigit@amd.com<mailto:ferruh.yigit@amd.com> <mailto:ferruh.yigit@amd.com<mailto:ferruh.yigit@amd.com>>>
>     > *Subject: *Re: help
>     >
>     > !! External Email
>     >
>     > +Cc some AMD maintainers, they can have an idea about IOMMU settings.
>     >
>     >
>     > 20/07/2023 14:44, Igor de Paula:
>     >> I have enabled it in the host and in the BIOS for AMD...
>     >> In the Bios I changed to amd_iommu=on and in the host it's the
>     same for
>     >> either.
>     >>
>     >> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon
>     <thomas@monjalon.net<mailto:thomas@monjalon.net> <mailto:thomas@monjalon.net<mailto:thomas@monjalon.net>>> wrote:
>     >>
>     >> > 20/07/2023 11:35, Igor de Paula:
>     >> > > The weird thing is that it only happens when I am using a
>     host with an
>     >> > AMD
>     >> > > processor. It doesn't happen when I use a host with an Intel
>     processor.
>     >> >
>     >> > So it's probably a matter of BIOS settings for the IOMMU?
>     >> >
>     >> >
>     >> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon
>     <thomas@monjalon.net<mailto:thomas@monjalon.net> <mailto:thomas@monjalon.net<mailto:thomas@monjalon.net>>>
>     >> > > wrote:
>     >> > >
>     >> > > > +Cc the vmxnet3 maintainer.
>     >> > > >
>     >> > > > Please Jochen, do you have an idea what's wrong below?
>     >> > > >
>     >> > > >
>     >> > > > 20/07/2023 11:25, Igor de Paula:
>     >> > > > > This is because it can't negotiate the IOMMU type with
>     any port.
>     >> > > > >
>     >> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon
>     <thomas@monjalon.net<mailto:thomas@monjalon.net> <mailto:thomas@monjalon.net<mailto:thomas@monjalon.net>>
>     >> > >
>     >> > > > wrote:
>     >> > > > >
>     >> > > > > > Hello,
>     >> > > > > >
>     >> > > > > > The first error is "Cause: Error: number of ports must
>     be even"
>     >> > > > > >
>     >> > > > > >
>     >> > > > > > 03/05/2023 18:13, Igor de Paula:
>     >> > > > > > > I am running a VM inside a VMWARE server (vSphere).
>     >> > > > > > > My goal it to set up DPDK with two HW ports, and set up a
>     >> > > > virtio_user to
>     >> > > > > > > interact with the kernel stack.
>     >> > > > > > > In another app I have it working but instead of
>     virtio_user I am
>     >> > > > running
>     >> > > > > > > KNI, it works in IOVA-PA mode.
>     >> > > > > > > I am looking to replace the KNI.
>     >> > > > > > >
>     >> > > > > > > When I try to set up virtio_user port as in the doc:
>     >> > > > > > >
>     >> > > > > >
>     >> > > >
>     >> >
>     https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path> <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path>>
>     >> > > > > > > I get a error it can't run in PA mode.
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > When I try to run as VA mode from a parameter, I get the
>     >> > following
>     >> > > > > > errors:
>     >> > > > > > > EAL: lib.eal log level changed from info to debug
>     >> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0
>     >> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0
>     >> > > > > > > EAL: Support maximum 128 logical core(s) by
>     configuration.
>     >> > > > > > > EAL: Detected 2 lcore(s)
>     >> > > > > > > EAL: Detected 1 NUMA nodes
>     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3'
>     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21'
>     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so'
>     >> > > > > > > EAL: Detected static linkage of DPDK
>     >> > > > > > > EAL: Ask a virtual area of 0x7000 bytes
>     >> > > > > > > EAL: Virtual area found at 0x100000000 (size = 0x7000)
>     >> > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>     >> > > > > > > EAL: DPAA Bus not present. Skipping.
>     >> > > > > > > EAL: VFIO PCI modules not loaded
>     >> > > > > > > EAL: Selected IOVA mode 'VA'
>     >> > > > > > > EAL: Probing VFIO support...
>     >> > > > > > > EAL: IOMMU type 1 (Type 1) is supported
>     >> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported
>     >> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported
>     >> > > > > > > EAL: VFIO support initialized
>     >> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes
>     >> > > > > > > EAL: Virtual area found at 0x100007000 (size = 0x5b000)
>     >> > > > > > > EAL: Setting up physically contiguous memory...
>     >> > > > > > > EAL: Setting maximum number of open files to 1048576
>     >> > > > > > > EAL: Detected memory type: socket_id:0
>     hugepage_sz:1073741824
>     >> > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
>     >> > > > > > hugepage_sz:1073741824
>     >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
>     >> > > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000)
>     >> > > > > > > EAL: Memseg list allocated at socket 0, page size
>     0x100000kB
>     >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
>     >> > > > > > > EAL: Virtual area found at 0x140000000 (size =
>     0x2000000000)
>     >> > > > > > > EAL: VA reserved for memseg list at 0x140000000, size
>     2000000000
>     >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
>     >> > > > > > > EAL: Virtual area found at 0x2140000000 (size = 0x2000)
>     >> > > > > > > EAL: Memseg list allocated at socket 0, page size
>     0x100000kB
>     >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
>     >> > > > > > > EAL: Virtual area found at 0x2180000000 (size =
>     0x2000000000)
>     >> > > > > > > EAL: VA reserved for memseg list at 0x2180000000,
>     size 2000000000
>     >> > > > > > > EAL: TSC frequency is ~2350000 KHz
>     >> > > > > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0])
>     >> > > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
>     >> > > > > > > EAL: Trying to obtain current memory policy.
>     >> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0
>     >> > > > > > > EAL: Restoring previous memory policy: 0
>     >> > > > > > > EAL: request: mp_malloc_sync
>     >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
>     >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
>     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>     >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is
>     'VA', not
>     >> > > > > > initializing
>     >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
>     >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
>     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>     >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is
>     'VA', not
>     >> > > > > > initializing
>     >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
>     >> > > > > > > EAL: Bus (pci) probe failed.
>     >> > > > > > > EAL: lib.telemetry log level changed from disabled to
>     warning
>     >> > > > > > > EAL: Error - exiting with code: 1
>     >> > > > > > >   Cause: Error: number of ports must be even
>     >> > > > > > > EAL: request: mp_malloc_sync
>     >> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > For some reason the HW ports won't setup. From what I
>     understand
>     >> > > > > > > net_vmxnet3 should work with VA mode.
>     >> > > > > > > I enabled I/OMUU for the VM.
>     >> > > > > > > The weird thing even when enabled, I still have the
>     >> > > > > > > enable_unsafe_noiommu_mode flag on.
>     >> > > > > > > And because it's on the this:
>     >> > > > > > >
>     >> > > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev);
>     >> > > > > > >
>     >> > > > > > > return PA mode, and it fails.
>     >> > > > > > >
>     >> > > > > > > When I disable it by modifying
>     >> > > > > > >
>     /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get
>     >> > another
>     >> > > > > > error.
>     >> > > > > > > The error is that it doesn't find a suitable IOMMU type:
>     >> > > > > > > Just putting the relevant message:
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
>     >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
>     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>     >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No
>     such device)
>     >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
>     such device)
>     >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
>     such
>     >> > device)
>     >> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type
>     >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
>     >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
>     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>     >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No
>     such device)
>     >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
>     such device)
>     >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
>     such
>     >> > device)
>     >> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type
>     >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > I can see where it fails:
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > const struct vfio_iommu_type *
>     >> > > > > > > vfio_set_iommu_type(int vfio_container_fd)
>     >> > > > > > > {
>     >> > > > > > > unsigned idx;
>     >> > > > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
>     >> > > > > > > const struct vfio_iommu_type *t = &iommu_types[idx];
>     >> > > > > > >
>     >> > > > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
>     >> > > > > > > t->type_id);
>     >> > > > > > > if (!ret) {
>     >> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
>     >> > > > > > > t->type_id, t->name);
>     >> > > > > > > return t;
>     >> > > > > > > }
>     >> > > > > > > /* not an error, there may be more supported IOMMU
>     types */
>     >> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed,
>     error "
>     >> > > > > > > "%i (%s)\n", t->type_id, t->name, errno,
>     >> > > > > > > strerror(errno));
>     >> > > > > > > }
>     >> > > > > > > /* if we didn't find a suitable IOMMU type, fail */
>     >> > > > > > > return NULL;
>     >> > > > > > > }
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > The ioctl returns -1.
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > I tried many things.
>     >> > > > > > > Enabling/disabling IOMMU.
>     >> > > > > > > Modifying the enable_unsafe_noiommu_mode flag.
>     >> > > > > > > Running in PA mode.
>     >> > > > > > > Any help will be much appreciated.
>     >> >
>     >> >
>     >> >
>     >> >
>     >> >
>     >> >
>     >>
>     >
>     >
>     >
>     >
>     >
>     >
>     > !! External Email: This email originated from outside of the
>     > organization. Do not click links or open attachments unless you
>     > recognize the sender.
>     >
>

[-- Attachment #2: Type: text/html, Size: 34820 bytes --]

Re: help

[Igor de Paula]

[-- Attachment #1: Type: text/plain, Size: 17103 bytes --]

Well,
Like I said earlier, trying with the Intel host I have on VMWARE,
specifically  Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
With IOMMU enabled, VMXNET3 works with VA as well as PA.
Meaning, PA works regardless if IOMMU is enabled or not. From my experience
anyway.
That's why I thought that:
virtio_user needs VA to work.
For some reason VMXNET3 does not work with VA (only on AMD host).


On Tue, Jul 25, 2023 at 4:04 PM Varghese, Vipin <Vipin.Varghese@amd.com>
wrote:

> [AMD Official Use Only - General]
>
> Thanks Igor,
>
>
>
> As suspected the vmx_net3 works with
>
>
>
>    1. Iommu: disabled
>    2. enable_unsafe_iommu: enabled
>    3. dpdk eal iova mode: PA
>
>
>
> as pointed by you in logs, the virtio_user fails as it expects VA too.
>
>
>
> Will check and get back.
>
>
>
> *From:* Igor de Paula <igordptx@gmail.com>
> *Sent:* Tuesday, July 25, 2023 8:16 PM
> *To:* Yigit, Ferruh <Ferruh.Yigit@amd.com>
> *Cc:* Jochen Behrens <jbehrens@vmware.com>; Thomas Monjalon <
> thomas@monjalon.net>; users@dpdk.org; Gupta, Nipun <Nipun.Gupta@amd.com>;
> Agarwal, Nikhil <nikhil.agarwal@amd.com>; Ronak Doshi <doshir@vmware.com>;
> Immanni, Venkat <Venkat.Immanni@amd.com>; Varghese, Vipin <
> Vipin.Varghese@amd.com>; Chenbo Xia <chenbo.xia@intel.com>
> *Subject:* Re: help
>
>
>
> *Caution:* This message originated from an External Source. Use proper
> caution when opening attachments, clicking links, or responding.
>
>
>
> Hi,
> Attaching the logs of EAL when trying to run a configuration with
> virtio_user port when IOMMU is
> disabled and enable_unsafe_iommu is enabled. As you can see it forces IOVA
> as PA but the viritui_user needs IOVA as VA.
> I am also attaching the output of dmesg. I am not sure which kernel logs
> you wanted... if there is anything else please let me know..
> Regarding the ESXI logs, they are HUGE so I will send to you on a separate
> email.
>
>
>
> On Fri, Jul 21, 2023 at 1:14 PM Ferruh Yigit <ferruh.yigit@amd.com> wrote:
>
> On 7/21/2023 12:39 PM, Igor de Paula wrote:
> > I am trying to use virtio_user for an interface with the
> > kernel:
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html <
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html>
> > I think this requires IOVA as va.
> >
>
> I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo,
> he may know better.
>
> Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and
> '--iova-mode=pa'?
>
>
> > It does work with Intel host and IOMMU
> > enabled. Part of the negotiation when setting it up is getting the IOMMU
> > number so I thought it has to have IOMMU.
> >
>
> Yes, issue looks like related to the IOMMU, and it may be either related
> to HW support, or ESXi iommu driver support, we will check using below
> information you provided.
>
> > I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but
> > again, that didn't work.
> > ESXI version - VMware ESXi, 7.0.0, 16324942
> > AMD:  AMD EPYC 7452 32-Core Processo
> >
> > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> > Regarding the logs I will try and attach it soon.
> >
>
> Thanks for info, waiting for logs.
>
> >
> > On Fri, Jul 21, 2023 at 12:21 PM Ferruh Yigit <ferruh.yigit@amd.com
> > <mailto:ferruh.yigit@amd.com>> wrote:
> >
> >     Hi Igor,
> >
> >     VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you
> >     please try with:
> >     - enable 'enable_unsafe_noiommu_mode' flag
> >     - Force PA mode via '--iova-mode=pa' eal parameter
> >
> >
> >     Also to be able to figure out AMD IOMMU support level, can you please
> >     provide:
> >     - AMD part number
> >     - ESXi/hypervisor version
> >     - ESXi & VM kernel logs
> >
> >
> >     Thanks,
> >     Ferruh
> >
> >     On 7/20/2023 5:21 PM, Jochen Behrens wrote:
> >     > +Ronak from the ESX team
> >     >
> >     >
> >     >
> >     >
> >     >
> >     > In our usage, we do set amd_iommu=off in the boot command line from
> >     > grub. (Or intel_iommu=off for Intel processors.)
> >     >
> >     >
> >     >
> >     >                 Jochen
> >     >
> >     >
> >     >
> >     > *From: *Thomas Monjalon <thomas@monjalon.net
> >     <mailto:thomas@monjalon.net>>
> >     > *Date: *Thursday, July 20, 2023 at 6:00 AM
> >     > *To: *Igor de Paula <igordptx@gmail.com <mailto:igordptx@gmail.com
> >>
> >     > *Cc: *users@dpdk.org <mailto:users@dpdk.org> <users@dpdk.org
> >     <mailto:users@dpdk.org>>, Jochen Behrens
> >     > <jbehrens@vmware.com <mailto:jbehrens@vmware.com>>, Nipun Gupta
> >     <nipun.gupta@amd.com <mailto:nipun.gupta@amd.com>>, Nikhil Agarwal
> >     > <nikhil.agarwal@amd.com <mailto:nikhil.agarwal@amd.com>>, Ferruh
> >     Yigit <ferruh.yigit@amd.com <mailto:ferruh.yigit@amd.com>>
> >     > *Subject: *Re: help
> >     >
> >     > !! External Email
> >     >
> >     > +Cc some AMD maintainers, they can have an idea about IOMMU
> settings.
> >     >
> >     >
> >     > 20/07/2023 14:44, Igor de Paula:
> >     >> I have enabled it in the host and in the BIOS for AMD...
> >     >> In the Bios I changed to amd_iommu=on and in the host it's the
> >     same for
> >     >> either.
> >     >>
> >     >> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon
> >     <thomas@monjalon.net <mailto:thomas@monjalon.net>> wrote:
> >     >>
> >     >> > 20/07/2023 11:35, Igor de Paula:
> >     >> > > The weird thing is that it only happens when I am using a
> >     host with an
> >     >> > AMD
> >     >> > > processor. It doesn't happen when I use a host with an Intel
> >     processor.
> >     >> >
> >     >> > So it's probably a matter of BIOS settings for the IOMMU?
> >     >> >
> >     >> >
> >     >> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon
> >     <thomas@monjalon.net <mailto:thomas@monjalon.net>>
> >     >> > > wrote:
> >     >> > >
> >     >> > > > +Cc the vmxnet3 maintainer.
> >     >> > > >
> >     >> > > > Please Jochen, do you have an idea what's wrong below?
> >     >> > > >
> >     >> > > >
> >     >> > > > 20/07/2023 11:25, Igor de Paula:
> >     >> > > > > This is because it can't negotiate the IOMMU type with
> >     any port.
> >     >> > > > >
> >     >> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon
> >     <thomas@monjalon.net <mailto:thomas@monjalon.net>
> >     >> > >
> >     >> > > > wrote:
> >     >> > > > >
> >     >> > > > > > Hello,
> >     >> > > > > >
> >     >> > > > > > The first error is "Cause: Error: number of ports must
> >     be even"
> >     >> > > > > >
> >     >> > > > > >
> >     >> > > > > > 03/05/2023 18:13, Igor de Paula:
> >     >> > > > > > > I am running a VM inside a VMWARE server (vSphere).
> >     >> > > > > > > My goal it to set up DPDK with two HW ports, and set
> up a
> >     >> > > > virtio_user to
> >     >> > > > > > > interact with the kernel stack.
> >     >> > > > > > > In another app I have it working but instead of
> >     virtio_user I am
> >     >> > > > running
> >     >> > > > > > > KNI, it works in IOVA-PA mode.
> >     >> > > > > > > I am looking to replace the KNI.
> >     >> > > > > > >
> >     >> > > > > > > When I try to set up virtio_user port as in the doc:
> >     >> > > > > > >
> >     >> > > > > >
> >     >> > > >
> >     >> >
> >
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> <
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path>
> <
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> <
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> >>
> >     >> > > > > > > I get a error it can't run in PA mode.
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > When I try to run as VA mode from a parameter, I get
> the
> >     >> > following
> >     >> > > > > > errors:
> >     >> > > > > > > EAL: lib.eal log level changed from info to debug
> >     >> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0
> >     >> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0
> >     >> > > > > > > EAL: Support maximum 128 logical core(s) by
> >     configuration.
> >     >> > > > > > > EAL: Detected 2 lcore(s)
> >     >> > > > > > > EAL: Detected 1 NUMA nodes
> >     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3'
> >     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21'
> >     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so'
> >     >> > > > > > > EAL: Detected static linkage of DPDK
> >     >> > > > > > > EAL: Ask a virtual area of 0x7000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x100000000 (size = 0x7000)
> >     >> > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> >     >> > > > > > > EAL: DPAA Bus not present. Skipping.
> >     >> > > > > > > EAL: VFIO PCI modules not loaded
> >     >> > > > > > > EAL: Selected IOVA mode 'VA'
> >     >> > > > > > > EAL: Probing VFIO support...
> >     >> > > > > > > EAL: IOMMU type 1 (Type 1) is supported
> >     >> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported
> >     >> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported
> >     >> > > > > > > EAL: VFIO support initialized
> >     >> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x100007000 (size =
> 0x5b000)
> >     >> > > > > > > EAL: Setting up physically contiguous memory...
> >     >> > > > > > > EAL: Setting maximum number of open files to 1048576
> >     >> > > > > > > EAL: Detected memory type: socket_id:0
> >     hugepage_sz:1073741824
> >     >> > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
> >     >> > > > > > hugepage_sz:1073741824
> >     >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000)
> >     >> > > > > > > EAL: Memseg list allocated at socket 0, page size
> >     0x100000kB
> >     >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x140000000 (size =
> >     0x2000000000)
> >     >> > > > > > > EAL: VA reserved for memseg list at 0x140000000, size
> >     2000000000
> >     >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x2140000000 (size =
> 0x2000)
> >     >> > > > > > > EAL: Memseg list allocated at socket 0, page size
> >     0x100000kB
> >     >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x2180000000 (size =
> >     0x2000000000)
> >     >> > > > > > > EAL: VA reserved for memseg list at 0x2180000000,
> >     size 2000000000
> >     >> > > > > > > EAL: TSC frequency is ~2350000 KHz
> >     >> > > > > > > EAL: Main lcore 0 is ready
> (tid=7f8ad790ec00;cpuset=[0])
> >     >> > > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
> >     >> > > > > > > EAL: Trying to obtain current memory policy.
> >     >> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0
> >     >> > > > > > > EAL: Restoring previous memory policy: 0
> >     >> > > > > > > EAL: request: mp_malloc_sync
> >     >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> >     >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >     >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is
> >     'VA', not
> >     >> > > > > > initializing
> >     >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> >     >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >     >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is
> >     'VA', not
> >     >> > > > > > initializing
> >     >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> >     >> > > > > > > EAL: Bus (pci) probe failed.
> >     >> > > > > > > EAL: lib.telemetry log level changed from disabled to
> >     warning
> >     >> > > > > > > EAL: Error - exiting with code: 1
> >     >> > > > > > >   Cause: Error: number of ports must be even
> >     >> > > > > > > EAL: request: mp_malloc_sync
> >     >> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > For some reason the HW ports won't setup. From what I
> >     understand
> >     >> > > > > > > net_vmxnet3 should work with VA mode.
> >     >> > > > > > > I enabled I/OMUU for the VM.
> >     >> > > > > > > The weird thing even when enabled, I still have the
> >     >> > > > > > > enable_unsafe_noiommu_mode flag on.
> >     >> > > > > > > And because it's on the this:
> >     >> > > > > > >
> >     >> > > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev);
> >     >> > > > > > >
> >     >> > > > > > > return PA mode, and it fails.
> >     >> > > > > > >
> >     >> > > > > > > When I disable it by modifying
> >     >> > > > > > >
> >     /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get
> >     >> > another
> >     >> > > > > > error.
> >     >> > > > > > > The error is that it doesn't find a suitable IOMMU
> type:
> >     >> > > > > > > Just putting the relevant message:
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> >     >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >     >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No
> >     such device)
> >     >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
> >     such device)
> >     >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
> >     such
> >     >> > device)
> >     >> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type
> >     >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> >     >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >     >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No
> >     such device)
> >     >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
> >     such device)
> >     >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
> >     such
> >     >> > device)
> >     >> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type
> >     >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > I can see where it fails:
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > const struct vfio_iommu_type *
> >     >> > > > > > > vfio_set_iommu_type(int vfio_container_fd)
> >     >> > > > > > > {
> >     >> > > > > > > unsigned idx;
> >     >> > > > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
> >     >> > > > > > > const struct vfio_iommu_type *t = &iommu_types[idx];
> >     >> > > > > > >
> >     >> > > > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
> >     >> > > > > > > t->type_id);
> >     >> > > > > > > if (!ret) {
> >     >> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> >     >> > > > > > > t->type_id, t->name);
> >     >> > > > > > > return t;
> >     >> > > > > > > }
> >     >> > > > > > > /* not an error, there may be more supported IOMMU
> >     types */
> >     >> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed,
> >     error "
> >     >> > > > > > > "%i (%s)\n", t->type_id, t->name, errno,
> >     >> > > > > > > strerror(errno));
> >     >> > > > > > > }
> >     >> > > > > > > /* if we didn't find a suitable IOMMU type, fail */
> >     >> > > > > > > return NULL;
> >     >> > > > > > > }
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > The ioctl returns -1.
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > I tried many things.
> >     >> > > > > > > Enabling/disabling IOMMU.
> >     >> > > > > > > Modifying the enable_unsafe_noiommu_mode flag.
> >     >> > > > > > > Running in PA mode.
> >     >> > > > > > > Any help will be much appreciated.
> >     >> >
> >     >> >
> >     >> >
> >     >> >
> >     >> >
> >     >> >
> >     >>
> >     >
> >     >
> >     >
> >     >
> >     >
> >     >
> >     > !! External Email: This email originated from outside of the
> >     > organization. Do not click links or open attachments unless you
> >     > recognize the sender.
> >     >
> >
>
>

[-- Attachment #2: Type: text/html, Size: 30079 bytes --]

RE: help

[Varghese, Vipin]

[-- Attachment #1: Type: text/plain, Size: 18547 bytes --]

[AMD Official Use Only - General]

Like I said earlier, trying with the Intel host I have on VMWARE, specifically  Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
With IOMMU enabled, VMXNET3 works with VA as well as PA.
[VV] since ` enable_unsafe_iommu: not enabled` on Intel platform, could it be possible the specific version EXSI hypervisor supports the HW IOMMU specific to the platform. My suspicion will be in case on AMD platform  changes for required to enable HW iommu might not be available to specific EXSI (hypervior OS) used.

I am not an expert on virtio_user PMD, but I can check if it will work with PA with deferred setting for vmx_net3 PMD are ok?

From: Igor de Paula <igordptx@gmail.com>
Sent: Tuesday, July 25, 2023 8:42 PM
To: Varghese, Vipin <Vipin.Varghese@amd.com>
Cc: Yigit, Ferruh <Ferruh.Yigit@amd.com>; Jochen Behrens <jbehrens@vmware.com>; Thomas Monjalon <thomas@monjalon.net>; users@dpdk.org; Gupta, Nipun <Nipun.Gupta@amd.com>; Agarwal, Nikhil <nikhil.agarwal@amd.com>; Ronak Doshi <doshir@vmware.com>; Immanni, Venkat <Venkat.Immanni@amd.com>; Chenbo Xia <chenbo.xia@intel.com>
Subject: Re: help

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.

Well,
Like I said earlier, trying with the Intel host I have on VMWARE, specifically  Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
With IOMMU enabled, VMXNET3 works with VA as well as PA.
Meaning, PA works regardless if IOMMU is enabled or not. From my experience anyway.
That's why I thought that:
virtio_user needs VA to work.
For some reason VMXNET3 does not work with VA (only on AMD host).


On Tue, Jul 25, 2023 at 4:04 PM Varghese, Vipin <Vipin.Varghese@amd.com<mailto:Vipin.Varghese@amd.com>> wrote:

[AMD Official Use Only - General]

Thanks Igor,

As suspected the vmx_net3 works with


  1.  Iommu: disabled
  2.  enable_unsafe_iommu: enabled
  3.  dpdk eal iova mode: PA

as pointed by you in logs, the virtio_user fails as it expects VA too.

Will check and get back.

From: Igor de Paula <igordptx@gmail.com<mailto:igordptx@gmail.com>>
Sent: Tuesday, July 25, 2023 8:16 PM
To: Yigit, Ferruh <Ferruh.Yigit@amd.com<mailto:Ferruh.Yigit@amd.com>>
Cc: Jochen Behrens <jbehrens@vmware.com<mailto:jbehrens@vmware.com>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; users@dpdk.org<mailto:users@dpdk.org>; Gupta, Nipun <Nipun.Gupta@amd.com<mailto:Nipun.Gupta@amd.com>>; Agarwal, Nikhil <nikhil.agarwal@amd.com<mailto:nikhil.agarwal@amd.com>>; Ronak Doshi <doshir@vmware.com<mailto:doshir@vmware.com>>; Immanni, Venkat <Venkat.Immanni@amd.com<mailto:Venkat.Immanni@amd.com>>; Varghese, Vipin <Vipin.Varghese@amd.com<mailto:Vipin.Varghese@amd.com>>; Chenbo Xia <chenbo.xia@intel.com<mailto:chenbo.xia@intel.com>>
Subject: Re: help

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.

Hi,
Attaching the logs of EAL when trying to run a configuration with virtio_user port when IOMMU is
disabled and enable_unsafe_iommu is enabled. As you can see it forces IOVA as PA but the viritui_user needs IOVA as VA.
I am also attaching the output of dmesg. I am not sure which kernel logs you wanted... if there is anything else please let me know..
Regarding the ESXI logs, they are HUGE so I will send to you on a separate email.

On Fri, Jul 21, 2023 at 1:14 PM Ferruh Yigit <ferruh.yigit@amd.com<mailto:ferruh.yigit@amd.com>> wrote:
On 7/21/2023 12:39 PM, Igor de Paula wrote:
> I am trying to use virtio_user for an interface with the
> kernel: https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html>
> I think this requires IOVA as va.
>

I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo,
he may know better.

Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and
'--iova-mode=pa'?


> It does work with Intel host and IOMMU
> enabled. Part of the negotiation when setting it up is getting the IOMMU
> number so I thought it has to have IOMMU.
>

Yes, issue looks like related to the IOMMU, and it may be either related
to HW support, or ESXi iommu driver support, we will check using below
information you provided.

> I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but
> again, that didn't work.
> ESXI version - VMware ESXi, 7.0.0, 16324942
> AMD:  AMD EPYC 7452 32-Core Processo
>
> On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> Regarding the logs I will try and attach it soon.
>

Thanks for info, waiting for logs.

>
> On Fri, Jul 21, 2023 at 12:21 PM Ferruh Yigit <ferruh.yigit@amd.com<mailto:ferruh.yigit@amd.com>
> <mailto:ferruh.yigit@amd.com<mailto:ferruh.yigit@amd.com>>> wrote:
>
>     Hi Igor,
>
>     VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you
>     please try with:
>     - enable 'enable_unsafe_noiommu_mode' flag
>     - Force PA mode via '--iova-mode=pa' eal parameter
>
>
>     Also to be able to figure out AMD IOMMU support level, can you please
>     provide:
>     - AMD part number
>     - ESXi/hypervisor version
>     - ESXi & VM kernel logs
>
>
>     Thanks,
>     Ferruh
>
>     On 7/20/2023 5:21 PM, Jochen Behrens wrote:
>     > +Ronak from the ESX team
>     >
>     >
>     >
>     >
>     >
>     > In our usage, we do set amd_iommu=off in the boot command line from
>     > grub. (Or intel_iommu=off for Intel processors.)
>     >
>     >
>     >
>     >                 Jochen
>     >
>     >
>     >
>     > *From: *Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>
>     <mailto:thomas@monjalon.net<mailto:thomas@monjalon.net>>>
>     > *Date: *Thursday, July 20, 2023 at 6:00 AM
>     > *To: *Igor de Paula <igordptx@gmail.com<mailto:igordptx@gmail.com> <mailto:igordptx@gmail.com<mailto:igordptx@gmail.com>>>
>     > *Cc: *users@dpdk.org<mailto:users@dpdk.org> <mailto:users@dpdk.org<mailto:users@dpdk.org>> <users@dpdk.org<mailto:users@dpdk.org>
>     <mailto:users@dpdk.org<mailto:users@dpdk.org>>>, Jochen Behrens
>     > <jbehrens@vmware.com<mailto:jbehrens@vmware.com> <mailto:jbehrens@vmware.com<mailto:jbehrens@vmware.com>>>, Nipun Gupta
>     <nipun.gupta@amd.com<mailto:nipun.gupta@amd.com> <mailto:nipun.gupta@amd.com<mailto:nipun.gupta@amd.com>>>, Nikhil Agarwal
>     > <nikhil.agarwal@amd.com<mailto:nikhil.agarwal@amd.com> <mailto:nikhil.agarwal@amd.com<mailto:nikhil.agarwal@amd.com>>>, Ferruh
>     Yigit <ferruh.yigit@amd.com<mailto:ferruh.yigit@amd.com> <mailto:ferruh.yigit@amd.com<mailto:ferruh.yigit@amd.com>>>
>     > *Subject: *Re: help
>     >
>     > !! External Email
>     >
>     > +Cc some AMD maintainers, they can have an idea about IOMMU settings.
>     >
>     >
>     > 20/07/2023 14:44, Igor de Paula:
>     >> I have enabled it in the host and in the BIOS for AMD...
>     >> In the Bios I changed to amd_iommu=on and in the host it's the
>     same for
>     >> either.
>     >>
>     >> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon
>     <thomas@monjalon.net<mailto:thomas@monjalon.net> <mailto:thomas@monjalon.net<mailto:thomas@monjalon.net>>> wrote:
>     >>
>     >> > 20/07/2023 11:35, Igor de Paula:
>     >> > > The weird thing is that it only happens when I am using a
>     host with an
>     >> > AMD
>     >> > > processor. It doesn't happen when I use a host with an Intel
>     processor.
>     >> >
>     >> > So it's probably a matter of BIOS settings for the IOMMU?
>     >> >
>     >> >
>     >> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon
>     <thomas@monjalon.net<mailto:thomas@monjalon.net> <mailto:thomas@monjalon.net<mailto:thomas@monjalon.net>>>
>     >> > > wrote:
>     >> > >
>     >> > > > +Cc the vmxnet3 maintainer.
>     >> > > >
>     >> > > > Please Jochen, do you have an idea what's wrong below?
>     >> > > >
>     >> > > >
>     >> > > > 20/07/2023 11:25, Igor de Paula:
>     >> > > > > This is because it can't negotiate the IOMMU type with
>     any port.
>     >> > > > >
>     >> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon
>     <thomas@monjalon.net<mailto:thomas@monjalon.net> <mailto:thomas@monjalon.net<mailto:thomas@monjalon.net>>
>     >> > >
>     >> > > > wrote:
>     >> > > > >
>     >> > > > > > Hello,
>     >> > > > > >
>     >> > > > > > The first error is "Cause: Error: number of ports must
>     be even"
>     >> > > > > >
>     >> > > > > >
>     >> > > > > > 03/05/2023 18:13, Igor de Paula:
>     >> > > > > > > I am running a VM inside a VMWARE server (vSphere).
>     >> > > > > > > My goal it to set up DPDK with two HW ports, and set up a
>     >> > > > virtio_user to
>     >> > > > > > > interact with the kernel stack.
>     >> > > > > > > In another app I have it working but instead of
>     virtio_user I am
>     >> > > > running
>     >> > > > > > > KNI, it works in IOVA-PA mode.
>     >> > > > > > > I am looking to replace the KNI.
>     >> > > > > > >
>     >> > > > > > > When I try to set up virtio_user port as in the doc:
>     >> > > > > > >
>     >> > > > > >
>     >> > > >
>     >> >
>     https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path> <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path>>
>     >> > > > > > > I get a error it can't run in PA mode.
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > When I try to run as VA mode from a parameter, I get the
>     >> > following
>     >> > > > > > errors:
>     >> > > > > > > EAL: lib.eal log level changed from info to debug
>     >> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0
>     >> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0
>     >> > > > > > > EAL: Support maximum 128 logical core(s) by
>     configuration.
>     >> > > > > > > EAL: Detected 2 lcore(s)
>     >> > > > > > > EAL: Detected 1 NUMA nodes
>     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3'
>     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21'
>     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so'
>     >> > > > > > > EAL: Detected static linkage of DPDK
>     >> > > > > > > EAL: Ask a virtual area of 0x7000 bytes
>     >> > > > > > > EAL: Virtual area found at 0x100000000 (size = 0x7000)
>     >> > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>     >> > > > > > > EAL: DPAA Bus not present. Skipping.
>     >> > > > > > > EAL: VFIO PCI modules not loaded
>     >> > > > > > > EAL: Selected IOVA mode 'VA'
>     >> > > > > > > EAL: Probing VFIO support...
>     >> > > > > > > EAL: IOMMU type 1 (Type 1) is supported
>     >> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported
>     >> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported
>     >> > > > > > > EAL: VFIO support initialized
>     >> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes
>     >> > > > > > > EAL: Virtual area found at 0x100007000 (size = 0x5b000)
>     >> > > > > > > EAL: Setting up physically contiguous memory...
>     >> > > > > > > EAL: Setting maximum number of open files to 1048576
>     >> > > > > > > EAL: Detected memory type: socket_id:0
>     hugepage_sz:1073741824
>     >> > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
>     >> > > > > > hugepage_sz:1073741824
>     >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
>     >> > > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000)
>     >> > > > > > > EAL: Memseg list allocated at socket 0, page size
>     0x100000kB
>     >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
>     >> > > > > > > EAL: Virtual area found at 0x140000000 (size =
>     0x2000000000)
>     >> > > > > > > EAL: VA reserved for memseg list at 0x140000000, size
>     2000000000
>     >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
>     >> > > > > > > EAL: Virtual area found at 0x2140000000 (size = 0x2000)
>     >> > > > > > > EAL: Memseg list allocated at socket 0, page size
>     0x100000kB
>     >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
>     >> > > > > > > EAL: Virtual area found at 0x2180000000 (size =
>     0x2000000000)
>     >> > > > > > > EAL: VA reserved for memseg list at 0x2180000000,
>     size 2000000000
>     >> > > > > > > EAL: TSC frequency is ~2350000 KHz
>     >> > > > > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0])
>     >> > > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
>     >> > > > > > > EAL: Trying to obtain current memory policy.
>     >> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0
>     >> > > > > > > EAL: Restoring previous memory policy: 0
>     >> > > > > > > EAL: request: mp_malloc_sync
>     >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
>     >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
>     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>     >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is
>     'VA', not
>     >> > > > > > initializing
>     >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
>     >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
>     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>     >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is
>     'VA', not
>     >> > > > > > initializing
>     >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
>     >> > > > > > > EAL: Bus (pci) probe failed.
>     >> > > > > > > EAL: lib.telemetry log level changed from disabled to
>     warning
>     >> > > > > > > EAL: Error - exiting with code: 1
>     >> > > > > > >   Cause: Error: number of ports must be even
>     >> > > > > > > EAL: request: mp_malloc_sync
>     >> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > For some reason the HW ports won't setup. From what I
>     understand
>     >> > > > > > > net_vmxnet3 should work with VA mode.
>     >> > > > > > > I enabled I/OMUU for the VM.
>     >> > > > > > > The weird thing even when enabled, I still have the
>     >> > > > > > > enable_unsafe_noiommu_mode flag on.
>     >> > > > > > > And because it's on the this:
>     >> > > > > > >
>     >> > > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev);
>     >> > > > > > >
>     >> > > > > > > return PA mode, and it fails.
>     >> > > > > > >
>     >> > > > > > > When I disable it by modifying
>     >> > > > > > >
>     /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get
>     >> > another
>     >> > > > > > error.
>     >> > > > > > > The error is that it doesn't find a suitable IOMMU type:
>     >> > > > > > > Just putting the relevant message:
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
>     >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
>     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>     >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No
>     such device)
>     >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
>     such device)
>     >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
>     such
>     >> > device)
>     >> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type
>     >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
>     >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
>     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>     >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No
>     such device)
>     >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
>     such device)
>     >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
>     such
>     >> > device)
>     >> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type
>     >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > I can see where it fails:
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > const struct vfio_iommu_type *
>     >> > > > > > > vfio_set_iommu_type(int vfio_container_fd)
>     >> > > > > > > {
>     >> > > > > > > unsigned idx;
>     >> > > > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
>     >> > > > > > > const struct vfio_iommu_type *t = &iommu_types[idx];
>     >> > > > > > >
>     >> > > > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
>     >> > > > > > > t->type_id);
>     >> > > > > > > if (!ret) {
>     >> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
>     >> > > > > > > t->type_id, t->name);
>     >> > > > > > > return t;
>     >> > > > > > > }
>     >> > > > > > > /* not an error, there may be more supported IOMMU
>     types */
>     >> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed,
>     error "
>     >> > > > > > > "%i (%s)\n", t->type_id, t->name, errno,
>     >> > > > > > > strerror(errno));
>     >> > > > > > > }
>     >> > > > > > > /* if we didn't find a suitable IOMMU type, fail */
>     >> > > > > > > return NULL;
>     >> > > > > > > }
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > The ioctl returns -1.
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > >
>     >> > > > > > > I tried many things.
>     >> > > > > > > Enabling/disabling IOMMU.
>     >> > > > > > > Modifying the enable_unsafe_noiommu_mode flag.
>     >> > > > > > > Running in PA mode.
>     >> > > > > > > Any help will be much appreciated.
>     >> >
>     >> >
>     >> >
>     >> >
>     >> >
>     >> >
>     >>
>     >
>     >
>     >
>     >
>     >
>     >
>     > !! External Email: This email originated from outside of the
>     > organization. Do not click links or open attachments unless you
>     > recognize the sender.
>     >
>

[-- Attachment #2: Type: text/html, Size: 39465 bytes --]

Re: help

[Igor de Paula]

[-- Attachment #1: Type: text/plain, Size: 18997 bytes --]

Hi again,
I got this resolved with VMWARE support so I thought to share it here.
What I originally wanted was to use IOVA-VA on an AMD host. Which didn't
work. I have learned that the ESXI version that supports
virtual IOMMU in AMD hosts (which is a prerequisite to IOVA-VA) is ESXI 7.0
U1. After updating it worked. On Intel hosts ESXI 6.7 supports it already
as far as I know.


On Tue, Jul 25, 2023 at 6:19 PM Varghese, Vipin <Vipin.Varghese@amd.com>
wrote:

> [AMD Official Use Only - General]
>
> Like I said earlier, trying with the Intel host I have on VMWARE,
> specifically  Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> With IOMMU enabled, VMXNET3 works with VA as well as PA.
>
> [VV] since ` enable_unsafe_iommu: not enabled` on Intel platform, could
> it be possible the specific version EXSI hypervisor supports the HW IOMMU
> specific to the platform. My suspicion will be in case on AMD platform
>  changes for required to enable HW iommu might not be available to specific
> EXSI (hypervior OS) used.
>
>
>
> I am not an expert on virtio_user PMD, but I can check if it will work
> with PA with deferred setting for vmx_net3 PMD are ok?
>
>
>
> *From:* Igor de Paula <igordptx@gmail.com>
> *Sent:* Tuesday, July 25, 2023 8:42 PM
> *To:* Varghese, Vipin <Vipin.Varghese@amd.com>
> *Cc:* Yigit, Ferruh <Ferruh.Yigit@amd.com>; Jochen Behrens <
> jbehrens@vmware.com>; Thomas Monjalon <thomas@monjalon.net>;
> users@dpdk.org; Gupta, Nipun <Nipun.Gupta@amd.com>; Agarwal, Nikhil <
> nikhil.agarwal@amd.com>; Ronak Doshi <doshir@vmware.com>; Immanni, Venkat
> <Venkat.Immanni@amd.com>; Chenbo Xia <chenbo.xia@intel.com>
> *Subject:* Re: help
>
>
>
> *Caution:* This message originated from an External Source. Use proper
> caution when opening attachments, clicking links, or responding.
>
>
>
> Well,
> Like I said earlier, trying with the Intel host I have on VMWARE,
> specifically  Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> With IOMMU enabled, VMXNET3 works with VA as well as PA.
> Meaning, PA works regardless if IOMMU is enabled or not. From my
> experience anyway.
> That's why I thought that:
> virtio_user needs VA to work.
> For some reason VMXNET3 does not work with VA (only on AMD host).
>
>
>
>
>
> On Tue, Jul 25, 2023 at 4:04 PM Varghese, Vipin <Vipin.Varghese@amd.com>
> wrote:
>
> [AMD Official Use Only - General]
>
>
>
> Thanks Igor,
>
>
>
> As suspected the vmx_net3 works with
>
>
>
>    1. Iommu: disabled
>    2. enable_unsafe_iommu: enabled
>    3. dpdk eal iova mode: PA
>
>
>
> as pointed by you in logs, the virtio_user fails as it expects VA too.
>
>
>
> Will check and get back.
>
>
>
> *From:* Igor de Paula <igordptx@gmail.com>
> *Sent:* Tuesday, July 25, 2023 8:16 PM
> *To:* Yigit, Ferruh <Ferruh.Yigit@amd.com>
> *Cc:* Jochen Behrens <jbehrens@vmware.com>; Thomas Monjalon <
> thomas@monjalon.net>; users@dpdk.org; Gupta, Nipun <Nipun.Gupta@amd.com>;
> Agarwal, Nikhil <nikhil.agarwal@amd.com>; Ronak Doshi <doshir@vmware.com>;
> Immanni, Venkat <Venkat.Immanni@amd.com>; Varghese, Vipin <
> Vipin.Varghese@amd.com>; Chenbo Xia <chenbo.xia@intel.com>
> *Subject:* Re: help
>
>
>
> *Caution:* This message originated from an External Source. Use proper
> caution when opening attachments, clicking links, or responding.
>
>
>
> Hi,
> Attaching the logs of EAL when trying to run a configuration with
> virtio_user port when IOMMU is
> disabled and enable_unsafe_iommu is enabled. As you can see it forces IOVA
> as PA but the viritui_user needs IOVA as VA.
> I am also attaching the output of dmesg. I am not sure which kernel logs
> you wanted... if there is anything else please let me know..
> Regarding the ESXI logs, they are HUGE so I will send to you on a separate
> email.
>
>
>
> On Fri, Jul 21, 2023 at 1:14 PM Ferruh Yigit <ferruh.yigit@amd.com> wrote:
>
> On 7/21/2023 12:39 PM, Igor de Paula wrote:
> > I am trying to use virtio_user for an interface with the
> > kernel:
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html <
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html>
> > I think this requires IOVA as va.
> >
>
> I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo,
> he may know better.
>
> Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and
> '--iova-mode=pa'?
>
>
> > It does work with Intel host and IOMMU
> > enabled. Part of the negotiation when setting it up is getting the IOMMU
> > number so I thought it has to have IOMMU.
> >
>
> Yes, issue looks like related to the IOMMU, and it may be either related
> to HW support, or ESXi iommu driver support, we will check using below
> information you provided.
>
> > I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but
> > again, that didn't work.
> > ESXI version - VMware ESXi, 7.0.0, 16324942
> > AMD:  AMD EPYC 7452 32-Core Processo
> >
> > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> > Regarding the logs I will try and attach it soon.
> >
>
> Thanks for info, waiting for logs.
>
> >
> > On Fri, Jul 21, 2023 at 12:21 PM Ferruh Yigit <ferruh.yigit@amd.com
> > <mailto:ferruh.yigit@amd.com>> wrote:
> >
> >     Hi Igor,
> >
> >     VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you
> >     please try with:
> >     - enable 'enable_unsafe_noiommu_mode' flag
> >     - Force PA mode via '--iova-mode=pa' eal parameter
> >
> >
> >     Also to be able to figure out AMD IOMMU support level, can you please
> >     provide:
> >     - AMD part number
> >     - ESXi/hypervisor version
> >     - ESXi & VM kernel logs
> >
> >
> >     Thanks,
> >     Ferruh
> >
> >     On 7/20/2023 5:21 PM, Jochen Behrens wrote:
> >     > +Ronak from the ESX team
> >     >
> >     >
> >     >
> >     >
> >     >
> >     > In our usage, we do set amd_iommu=off in the boot command line from
> >     > grub. (Or intel_iommu=off for Intel processors.)
> >     >
> >     >
> >     >
> >     >                 Jochen
> >     >
> >     >
> >     >
> >     > *From: *Thomas Monjalon <thomas@monjalon.net
> >     <mailto:thomas@monjalon.net>>
> >     > *Date: *Thursday, July 20, 2023 at 6:00 AM
> >     > *To: *Igor de Paula <igordptx@gmail.com <mailto:igordptx@gmail.com
> >>
> >     > *Cc: *users@dpdk.org <mailto:users@dpdk.org> <users@dpdk.org
> >     <mailto:users@dpdk.org>>, Jochen Behrens
> >     > <jbehrens@vmware.com <mailto:jbehrens@vmware.com>>, Nipun Gupta
> >     <nipun.gupta@amd.com <mailto:nipun.gupta@amd.com>>, Nikhil Agarwal
> >     > <nikhil.agarwal@amd.com <mailto:nikhil.agarwal@amd.com>>, Ferruh
> >     Yigit <ferruh.yigit@amd.com <mailto:ferruh.yigit@amd.com>>
> >     > *Subject: *Re: help
> >     >
> >     > !! External Email
> >     >
> >     > +Cc some AMD maintainers, they can have an idea about IOMMU
> settings.
> >     >
> >     >
> >     > 20/07/2023 14:44, Igor de Paula:
> >     >> I have enabled it in the host and in the BIOS for AMD...
> >     >> In the Bios I changed to amd_iommu=on and in the host it's the
> >     same for
> >     >> either.
> >     >>
> >     >> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon
> >     <thomas@monjalon.net <mailto:thomas@monjalon.net>> wrote:
> >     >>
> >     >> > 20/07/2023 11:35, Igor de Paula:
> >     >> > > The weird thing is that it only happens when I am using a
> >     host with an
> >     >> > AMD
> >     >> > > processor. It doesn't happen when I use a host with an Intel
> >     processor.
> >     >> >
> >     >> > So it's probably a matter of BIOS settings for the IOMMU?
> >     >> >
> >     >> >
> >     >> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon
> >     <thomas@monjalon.net <mailto:thomas@monjalon.net>>
> >     >> > > wrote:
> >     >> > >
> >     >> > > > +Cc the vmxnet3 maintainer.
> >     >> > > >
> >     >> > > > Please Jochen, do you have an idea what's wrong below?
> >     >> > > >
> >     >> > > >
> >     >> > > > 20/07/2023 11:25, Igor de Paula:
> >     >> > > > > This is because it can't negotiate the IOMMU type with
> >     any port.
> >     >> > > > >
> >     >> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon
> >     <thomas@monjalon.net <mailto:thomas@monjalon.net>
> >     >> > >
> >     >> > > > wrote:
> >     >> > > > >
> >     >> > > > > > Hello,
> >     >> > > > > >
> >     >> > > > > > The first error is "Cause: Error: number of ports must
> >     be even"
> >     >> > > > > >
> >     >> > > > > >
> >     >> > > > > > 03/05/2023 18:13, Igor de Paula:
> >     >> > > > > > > I am running a VM inside a VMWARE server (vSphere).
> >     >> > > > > > > My goal it to set up DPDK with two HW ports, and set
> up a
> >     >> > > > virtio_user to
> >     >> > > > > > > interact with the kernel stack.
> >     >> > > > > > > In another app I have it working but instead of
> >     virtio_user I am
> >     >> > > > running
> >     >> > > > > > > KNI, it works in IOVA-PA mode.
> >     >> > > > > > > I am looking to replace the KNI.
> >     >> > > > > > >
> >     >> > > > > > > When I try to set up virtio_user port as in the doc:
> >     >> > > > > > >
> >     >> > > > > >
> >     >> > > >
> >     >> >
> >
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> <
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path>
> <
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> <
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> >>
> >     >> > > > > > > I get a error it can't run in PA mode.
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > When I try to run as VA mode from a parameter, I get
> the
> >     >> > following
> >     >> > > > > > errors:
> >     >> > > > > > > EAL: lib.eal log level changed from info to debug
> >     >> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0
> >     >> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0
> >     >> > > > > > > EAL: Support maximum 128 logical core(s) by
> >     configuration.
> >     >> > > > > > > EAL: Detected 2 lcore(s)
> >     >> > > > > > > EAL: Detected 1 NUMA nodes
> >     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3'
> >     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21'
> >     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so'
> >     >> > > > > > > EAL: Detected static linkage of DPDK
> >     >> > > > > > > EAL: Ask a virtual area of 0x7000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x100000000 (size = 0x7000)
> >     >> > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> >     >> > > > > > > EAL: DPAA Bus not present. Skipping.
> >     >> > > > > > > EAL: VFIO PCI modules not loaded
> >     >> > > > > > > EAL: Selected IOVA mode 'VA'
> >     >> > > > > > > EAL: Probing VFIO support...
> >     >> > > > > > > EAL: IOMMU type 1 (Type 1) is supported
> >     >> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported
> >     >> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported
> >     >> > > > > > > EAL: VFIO support initialized
> >     >> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x100007000 (size =
> 0x5b000)
> >     >> > > > > > > EAL: Setting up physically contiguous memory...
> >     >> > > > > > > EAL: Setting maximum number of open files to 1048576
> >     >> > > > > > > EAL: Detected memory type: socket_id:0
> >     hugepage_sz:1073741824
> >     >> > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
> >     >> > > > > > hugepage_sz:1073741824
> >     >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000)
> >     >> > > > > > > EAL: Memseg list allocated at socket 0, page size
> >     0x100000kB
> >     >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x140000000 (size =
> >     0x2000000000)
> >     >> > > > > > > EAL: VA reserved for memseg list at 0x140000000, size
> >     2000000000
> >     >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x2140000000 (size =
> 0x2000)
> >     >> > > > > > > EAL: Memseg list allocated at socket 0, page size
> >     0x100000kB
> >     >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> >     >> > > > > > > EAL: Virtual area found at 0x2180000000 (size =
> >     0x2000000000)
> >     >> > > > > > > EAL: VA reserved for memseg list at 0x2180000000,
> >     size 2000000000
> >     >> > > > > > > EAL: TSC frequency is ~2350000 KHz
> >     >> > > > > > > EAL: Main lcore 0 is ready
> (tid=7f8ad790ec00;cpuset=[0])
> >     >> > > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
> >     >> > > > > > > EAL: Trying to obtain current memory policy.
> >     >> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0
> >     >> > > > > > > EAL: Restoring previous memory policy: 0
> >     >> > > > > > > EAL: request: mp_malloc_sync
> >     >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> >     >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >     >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is
> >     'VA', not
> >     >> > > > > > initializing
> >     >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> >     >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >     >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is
> >     'VA', not
> >     >> > > > > > initializing
> >     >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> >     >> > > > > > > EAL: Bus (pci) probe failed.
> >     >> > > > > > > EAL: lib.telemetry log level changed from disabled to
> >     warning
> >     >> > > > > > > EAL: Error - exiting with code: 1
> >     >> > > > > > >   Cause: Error: number of ports must be even
> >     >> > > > > > > EAL: request: mp_malloc_sync
> >     >> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > For some reason the HW ports won't setup. From what I
> >     understand
> >     >> > > > > > > net_vmxnet3 should work with VA mode.
> >     >> > > > > > > I enabled I/OMUU for the VM.
> >     >> > > > > > > The weird thing even when enabled, I still have the
> >     >> > > > > > > enable_unsafe_noiommu_mode flag on.
> >     >> > > > > > > And because it's on the this:
> >     >> > > > > > >
> >     >> > > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev);
> >     >> > > > > > >
> >     >> > > > > > > return PA mode, and it fails.
> >     >> > > > > > >
> >     >> > > > > > > When I disable it by modifying
> >     >> > > > > > >
> >     /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get
> >     >> > another
> >     >> > > > > > error.
> >     >> > > > > > > The error is that it doesn't find a suitable IOMMU
> type:
> >     >> > > > > > > Just putting the relevant message:
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> >     >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >     >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No
> >     such device)
> >     >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
> >     such device)
> >     >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
> >     such
> >     >> > device)
> >     >> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type
> >     >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> >     >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> >     >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No
> >     such device)
> >     >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
> >     such device)
> >     >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
> >     such
> >     >> > device)
> >     >> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type
> >     >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > I can see where it fails:
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > const struct vfio_iommu_type *
> >     >> > > > > > > vfio_set_iommu_type(int vfio_container_fd)
> >     >> > > > > > > {
> >     >> > > > > > > unsigned idx;
> >     >> > > > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
> >     >> > > > > > > const struct vfio_iommu_type *t = &iommu_types[idx];
> >     >> > > > > > >
> >     >> > > > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
> >     >> > > > > > > t->type_id);
> >     >> > > > > > > if (!ret) {
> >     >> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> >     >> > > > > > > t->type_id, t->name);
> >     >> > > > > > > return t;
> >     >> > > > > > > }
> >     >> > > > > > > /* not an error, there may be more supported IOMMU
> >     types */
> >     >> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed,
> >     error "
> >     >> > > > > > > "%i (%s)\n", t->type_id, t->name, errno,
> >     >> > > > > > > strerror(errno));
> >     >> > > > > > > }
> >     >> > > > > > > /* if we didn't find a suitable IOMMU type, fail */
> >     >> > > > > > > return NULL;
> >     >> > > > > > > }
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > The ioctl returns -1.
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > >
> >     >> > > > > > > I tried many things.
> >     >> > > > > > > Enabling/disabling IOMMU.
> >     >> > > > > > > Modifying the enable_unsafe_noiommu_mode flag.
> >     >> > > > > > > Running in PA mode.
> >     >> > > > > > > Any help will be much appreciated.
> >     >> >
> >     >> >
> >     >> >
> >     >> >
> >     >> >
> >     >> >
> >     >>
> >     >
> >     >
> >     >
> >     >
> >     >
> >     >
> >     > !! External Email: This email originated from outside of the
> >     > organization. Do not click links or open attachments unless you
> >     > recognize the sender.
> >     >
> >
>
>

[-- Attachment #2: Type: text/html, Size: 34302 bytes --]

Re: help

[Thomas Monjalon]

Thanks for the info.
Do you think it should be written in the vmxnet3 page of the DPDK documentation?
If yes, would you like to initiate a patch for review?


11/08/2023 10:43, Igor de Paula:
> Hi again,
> I got this resolved with VMWARE support so I thought to share it here.
> What I originally wanted was to use IOVA-VA on an AMD host. Which didn't
> work. I have learned that the ESXI version that supports
> virtual IOMMU in AMD hosts (which is a prerequisite to IOVA-VA) is ESXI 7.0
> U1. After updating it worked. On Intel hosts ESXI 6.7 supports it already
> as far as I know.
> 
> 
> On Tue, Jul 25, 2023 at 6:19 PM Varghese, Vipin <Vipin.Varghese@amd.com>
> wrote:
> 
> > [AMD Official Use Only - General]
> >
> > Like I said earlier, trying with the Intel host I have on VMWARE,
> > specifically  Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> > With IOMMU enabled, VMXNET3 works with VA as well as PA.
> >
> > [VV] since ` enable_unsafe_iommu: not enabled` on Intel platform, could
> > it be possible the specific version EXSI hypervisor supports the HW IOMMU
> > specific to the platform. My suspicion will be in case on AMD platform
> >  changes for required to enable HW iommu might not be available to specific
> > EXSI (hypervior OS) used.
> >
> >
> >
> > I am not an expert on virtio_user PMD, but I can check if it will work
> > with PA with deferred setting for vmx_net3 PMD are ok?
> >
> >
> >
> > *From:* Igor de Paula <igordptx@gmail.com>
> > *Sent:* Tuesday, July 25, 2023 8:42 PM
> > *To:* Varghese, Vipin <Vipin.Varghese@amd.com>
> > *Cc:* Yigit, Ferruh <Ferruh.Yigit@amd.com>; Jochen Behrens <
> > jbehrens@vmware.com>; Thomas Monjalon <thomas@monjalon.net>;
> > users@dpdk.org; Gupta, Nipun <Nipun.Gupta@amd.com>; Agarwal, Nikhil <
> > nikhil.agarwal@amd.com>; Ronak Doshi <doshir@vmware.com>; Immanni, Venkat
> > <Venkat.Immanni@amd.com>; Chenbo Xia <chenbo.xia@intel.com>
> > *Subject:* Re: help
> >
> >
> >
> > *Caution:* This message originated from an External Source. Use proper
> > caution when opening attachments, clicking links, or responding.
> >
> >
> >
> > Well,
> > Like I said earlier, trying with the Intel host I have on VMWARE,
> > specifically  Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> > With IOMMU enabled, VMXNET3 works with VA as well as PA.
> > Meaning, PA works regardless if IOMMU is enabled or not. From my
> > experience anyway.
> > That's why I thought that:
> > virtio_user needs VA to work.
> > For some reason VMXNET3 does not work with VA (only on AMD host).
> >
> >
> >
> >
> >
> > On Tue, Jul 25, 2023 at 4:04 PM Varghese, Vipin <Vipin.Varghese@amd.com>
> > wrote:
> >
> > [AMD Official Use Only - General]
> >
> >
> >
> > Thanks Igor,
> >
> >
> >
> > As suspected the vmx_net3 works with
> >
> >
> >
> >    1. Iommu: disabled
> >    2. enable_unsafe_iommu: enabled
> >    3. dpdk eal iova mode: PA
> >
> >
> >
> > as pointed by you in logs, the virtio_user fails as it expects VA too.
> >
> >
> >
> > Will check and get back.
> >
> >
> >
> > *From:* Igor de Paula <igordptx@gmail.com>
> > *Sent:* Tuesday, July 25, 2023 8:16 PM
> > *To:* Yigit, Ferruh <Ferruh.Yigit@amd.com>
> > *Cc:* Jochen Behrens <jbehrens@vmware.com>; Thomas Monjalon <
> > thomas@monjalon.net>; users@dpdk.org; Gupta, Nipun <Nipun.Gupta@amd.com>;
> > Agarwal, Nikhil <nikhil.agarwal@amd.com>; Ronak Doshi <doshir@vmware.com>;
> > Immanni, Venkat <Venkat.Immanni@amd.com>; Varghese, Vipin <
> > Vipin.Varghese@amd.com>; Chenbo Xia <chenbo.xia@intel.com>
> > *Subject:* Re: help
> >
> >
> >
> > *Caution:* This message originated from an External Source. Use proper
> > caution when opening attachments, clicking links, or responding.
> >
> >
> >
> > Hi,
> > Attaching the logs of EAL when trying to run a configuration with
> > virtio_user port when IOMMU is
> > disabled and enable_unsafe_iommu is enabled. As you can see it forces IOVA
> > as PA but the viritui_user needs IOVA as VA.
> > I am also attaching the output of dmesg. I am not sure which kernel logs
> > you wanted... if there is anything else please let me know..
> > Regarding the ESXI logs, they are HUGE so I will send to you on a separate
> > email.
> >
> >
> >
> > On Fri, Jul 21, 2023 at 1:14 PM Ferruh Yigit <ferruh.yigit@amd.com> wrote:
> >
> > On 7/21/2023 12:39 PM, Igor de Paula wrote:
> > > I am trying to use virtio_user for an interface with the
> > > kernel:
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html <
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html>
> > > I think this requires IOVA as va.
> > >
> >
> > I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo,
> > he may know better.
> >
> > Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and
> > '--iova-mode=pa'?
> >
> >
> > > It does work with Intel host and IOMMU
> > > enabled. Part of the negotiation when setting it up is getting the IOMMU
> > > number so I thought it has to have IOMMU.
> > >
> >
> > Yes, issue looks like related to the IOMMU, and it may be either related
> > to HW support, or ESXi iommu driver support, we will check using below
> > information you provided.
> >
> > > I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but
> > > again, that didn't work.
> > > ESXI version - VMware ESXi, 7.0.0, 16324942
> > > AMD:  AMD EPYC 7452 32-Core Processo
> > >
> > > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> > > Regarding the logs I will try and attach it soon.
> > >
> >
> > Thanks for info, waiting for logs.
> >
> > >
> > > On Fri, Jul 21, 2023 at 12:21 PM Ferruh Yigit <ferruh.yigit@amd.com
> > > <mailto:ferruh.yigit@amd.com>> wrote:
> > >
> > >     Hi Igor,
> > >
> > >     VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you
> > >     please try with:
> > >     - enable 'enable_unsafe_noiommu_mode' flag
> > >     - Force PA mode via '--iova-mode=pa' eal parameter
> > >
> > >
> > >     Also to be able to figure out AMD IOMMU support level, can you please
> > >     provide:
> > >     - AMD part number
> > >     - ESXi/hypervisor version
> > >     - ESXi & VM kernel logs
> > >
> > >
> > >     Thanks,
> > >     Ferruh
> > >
> > >     On 7/20/2023 5:21 PM, Jochen Behrens wrote:
> > >     > +Ronak from the ESX team
> > >     >
> > >     >
> > >     >
> > >     >
> > >     >
> > >     > In our usage, we do set amd_iommu=off in the boot command line from
> > >     > grub. (Or intel_iommu=off for Intel processors.)
> > >     >
> > >     >
> > >     >
> > >     >                 Jochen
> > >     >
> > >     >
> > >     >
> > >     > *From: *Thomas Monjalon <thomas@monjalon.net
> > >     <mailto:thomas@monjalon.net>>
> > >     > *Date: *Thursday, July 20, 2023 at 6:00 AM
> > >     > *To: *Igor de Paula <igordptx@gmail.com <mailto:igordptx@gmail.com
> > >>
> > >     > *Cc: *users@dpdk.org <mailto:users@dpdk.org> <users@dpdk.org
> > >     <mailto:users@dpdk.org>>, Jochen Behrens
> > >     > <jbehrens@vmware.com <mailto:jbehrens@vmware.com>>, Nipun Gupta
> > >     <nipun.gupta@amd.com <mailto:nipun.gupta@amd.com>>, Nikhil Agarwal
> > >     > <nikhil.agarwal@amd.com <mailto:nikhil.agarwal@amd.com>>, Ferruh
> > >     Yigit <ferruh.yigit@amd.com <mailto:ferruh.yigit@amd.com>>
> > >     > *Subject: *Re: help
> > >     >
> > >     > !! External Email
> > >     >
> > >     > +Cc some AMD maintainers, they can have an idea about IOMMU
> > settings.
> > >     >
> > >     >
> > >     > 20/07/2023 14:44, Igor de Paula:
> > >     >> I have enabled it in the host and in the BIOS for AMD...
> > >     >> In the Bios I changed to amd_iommu=on and in the host it's the
> > >     same for
> > >     >> either.
> > >     >>
> > >     >> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon
> > >     <thomas@monjalon.net <mailto:thomas@monjalon.net>> wrote:
> > >     >>
> > >     >> > 20/07/2023 11:35, Igor de Paula:
> > >     >> > > The weird thing is that it only happens when I am using a
> > >     host with an
> > >     >> > AMD
> > >     >> > > processor. It doesn't happen when I use a host with an Intel
> > >     processor.
> > >     >> >
> > >     >> > So it's probably a matter of BIOS settings for the IOMMU?
> > >     >> >
> > >     >> >
> > >     >> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon
> > >     <thomas@monjalon.net <mailto:thomas@monjalon.net>>
> > >     >> > > wrote:
> > >     >> > >
> > >     >> > > > +Cc the vmxnet3 maintainer.
> > >     >> > > >
> > >     >> > > > Please Jochen, do you have an idea what's wrong below?
> > >     >> > > >
> > >     >> > > >
> > >     >> > > > 20/07/2023 11:25, Igor de Paula:
> > >     >> > > > > This is because it can't negotiate the IOMMU type with
> > >     any port.
> > >     >> > > > >
> > >     >> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon
> > >     <thomas@monjalon.net <mailto:thomas@monjalon.net>
> > >     >> > >
> > >     >> > > > wrote:
> > >     >> > > > >
> > >     >> > > > > > Hello,
> > >     >> > > > > >
> > >     >> > > > > > The first error is "Cause: Error: number of ports must
> > >     be even"
> > >     >> > > > > >
> > >     >> > > > > >
> > >     >> > > > > > 03/05/2023 18:13, Igor de Paula:
> > >     >> > > > > > > I am running a VM inside a VMWARE server (vSphere).
> > >     >> > > > > > > My goal it to set up DPDK with two HW ports, and set
> > up a
> > >     >> > > > virtio_user to
> > >     >> > > > > > > interact with the kernel stack.
> > >     >> > > > > > > In another app I have it working but instead of
> > >     virtio_user I am
> > >     >> > > > running
> > >     >> > > > > > > KNI, it works in IOVA-PA mode.
> > >     >> > > > > > > I am looking to replace the KNI.
> > >     >> > > > > > >
> > >     >> > > > > > > When I try to set up virtio_user port as in the doc:
> > >     >> > > > > > >
> > >     >> > > > > >
> > >     >> > > >
> > >     >> >
> > >
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> > <
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path>
> > <
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> > <
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> > >>
> > >     >> > > > > > > I get a error it can't run in PA mode.
> > >     >> > > > > > >
> > >     >> > > > > > >
> > >     >> > > > > > > When I try to run as VA mode from a parameter, I get
> > the
> > >     >> > following
> > >     >> > > > > > errors:
> > >     >> > > > > > > EAL: lib.eal log level changed from info to debug
> > >     >> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0
> > >     >> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0
> > >     >> > > > > > > EAL: Support maximum 128 logical core(s) by
> > >     configuration.
> > >     >> > > > > > > EAL: Detected 2 lcore(s)
> > >     >> > > > > > > EAL: Detected 1 NUMA nodes
> > >     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3'
> > >     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21'
> > >     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so'
> > >     >> > > > > > > EAL: Detected static linkage of DPDK
> > >     >> > > > > > > EAL: Ask a virtual area of 0x7000 bytes
> > >     >> > > > > > > EAL: Virtual area found at 0x100000000 (size = 0x7000)
> > >     >> > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > >     >> > > > > > > EAL: DPAA Bus not present. Skipping.
> > >     >> > > > > > > EAL: VFIO PCI modules not loaded
> > >     >> > > > > > > EAL: Selected IOVA mode 'VA'
> > >     >> > > > > > > EAL: Probing VFIO support...
> > >     >> > > > > > > EAL: IOMMU type 1 (Type 1) is supported
> > >     >> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported
> > >     >> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported
> > >     >> > > > > > > EAL: VFIO support initialized
> > >     >> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes
> > >     >> > > > > > > EAL: Virtual area found at 0x100007000 (size =
> > 0x5b000)
> > >     >> > > > > > > EAL: Setting up physically contiguous memory...
> > >     >> > > > > > > EAL: Setting maximum number of open files to 1048576
> > >     >> > > > > > > EAL: Detected memory type: socket_id:0
> > >     hugepage_sz:1073741824
> > >     >> > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
> > >     >> > > > > > hugepage_sz:1073741824
> > >     >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> > >     >> > > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000)
> > >     >> > > > > > > EAL: Memseg list allocated at socket 0, page size
> > >     0x100000kB
> > >     >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> > >     >> > > > > > > EAL: Virtual area found at 0x140000000 (size =
> > >     0x2000000000)
> > >     >> > > > > > > EAL: VA reserved for memseg list at 0x140000000, size
> > >     2000000000
> > >     >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> > >     >> > > > > > > EAL: Virtual area found at 0x2140000000 (size =
> > 0x2000)
> > >     >> > > > > > > EAL: Memseg list allocated at socket 0, page size
> > >     0x100000kB
> > >     >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> > >     >> > > > > > > EAL: Virtual area found at 0x2180000000 (size =
> > >     0x2000000000)
> > >     >> > > > > > > EAL: VA reserved for memseg list at 0x2180000000,
> > >     size 2000000000
> > >     >> > > > > > > EAL: TSC frequency is ~2350000 KHz
> > >     >> > > > > > > EAL: Main lcore 0 is ready
> > (tid=7f8ad790ec00;cpuset=[0])
> > >     >> > > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
> > >     >> > > > > > > EAL: Trying to obtain current memory policy.
> > >     >> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0
> > >     >> > > > > > > EAL: Restoring previous memory policy: 0
> > >     >> > > > > > > EAL: request: mp_malloc_sync
> > >     >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> > >     >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > >     >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is
> > >     'VA', not
> > >     >> > > > > > initializing
> > >     >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> > >     >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > >     >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is
> > >     'VA', not
> > >     >> > > > > > initializing
> > >     >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> > >     >> > > > > > > EAL: Bus (pci) probe failed.
> > >     >> > > > > > > EAL: lib.telemetry log level changed from disabled to
> > >     warning
> > >     >> > > > > > > EAL: Error - exiting with code: 1
> > >     >> > > > > > >   Cause: Error: number of ports must be even
> > >     >> > > > > > > EAL: request: mp_malloc_sync
> > >     >> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB
> > >     >> > > > > > >
> > >     >> > > > > > >
> > >     >> > > > > > >
> > >     >> > > > > > > For some reason the HW ports won't setup. From what I
> > >     understand
> > >     >> > > > > > > net_vmxnet3 should work with VA mode.
> > >     >> > > > > > > I enabled I/OMUU for the VM.
> > >     >> > > > > > > The weird thing even when enabled, I still have the
> > >     >> > > > > > > enable_unsafe_noiommu_mode flag on.
> > >     >> > > > > > > And because it's on the this:
> > >     >> > > > > > >
> > >     >> > > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev);
> > >     >> > > > > > >
> > >     >> > > > > > > return PA mode, and it fails.
> > >     >> > > > > > >
> > >     >> > > > > > > When I disable it by modifying
> > >     >> > > > > > >
> > >     /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get
> > >     >> > another
> > >     >> > > > > > error.
> > >     >> > > > > > > The error is that it doesn't find a suitable IOMMU
> > type:
> > >     >> > > > > > > Just putting the relevant message:
> > >     >> > > > > > >
> > >     >> > > > > > >
> > >     >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> > >     >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > >     >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No
> > >     such device)
> > >     >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
> > >     such device)
> > >     >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
> > >     such
> > >     >> > device)
> > >     >> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type
> > >     >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> > >     >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > >     >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No
> > >     such device)
> > >     >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
> > >     such device)
> > >     >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
> > >     such
> > >     >> > device)
> > >     >> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type
> > >     >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> > >     >> > > > > > >
> > >     >> > > > > > >
> > >     >> > > > > > > I can see where it fails:
> > >     >> > > > > > >
> > >     >> > > > > > >
> > >     >> > > > > > > const struct vfio_iommu_type *
> > >     >> > > > > > > vfio_set_iommu_type(int vfio_container_fd)
> > >     >> > > > > > > {
> > >     >> > > > > > > unsigned idx;
> > >     >> > > > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
> > >     >> > > > > > > const struct vfio_iommu_type *t = &iommu_types[idx];
> > >     >> > > > > > >
> > >     >> > > > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
> > >     >> > > > > > > t->type_id);
> > >     >> > > > > > > if (!ret) {
> > >     >> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> > >     >> > > > > > > t->type_id, t->name);
> > >     >> > > > > > > return t;
> > >     >> > > > > > > }
> > >     >> > > > > > > /* not an error, there may be more supported IOMMU
> > >     types */
> > >     >> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed,
> > >     error "
> > >     >> > > > > > > "%i (%s)\n", t->type_id, t->name, errno,
> > >     >> > > > > > > strerror(errno));
> > >     >> > > > > > > }
> > >     >> > > > > > > /* if we didn't find a suitable IOMMU type, fail */
> > >     >> > > > > > > return NULL;
> > >     >> > > > > > > }
> > >     >> > > > > > >
> > >     >> > > > > > >
> > >     >> > > > > > > The ioctl returns -1.
> > >     >> > > > > > >
> > >     >> > > > > > >
> > >     >> > > > > > >
> > >     >> > > > > > > I tried many things.
> > >     >> > > > > > > Enabling/disabling IOMMU.
> > >     >> > > > > > > Modifying the enable_unsafe_noiommu_mode flag.
> > >     >> > > > > > > Running in PA mode.
> > >     >> > > > > > > Any help will be much appreciated.
> > >     >> >
> > >     >> >
> > >     >> >
> > >     >> >
> > >     >> >
> > >     >> >
> > >     >>
> > >     >
> > >     >
> > >     >
> > >     >
> > >     >
> > >     >
> > >     > !! External Email: This email originated from outside of the
> > >     > organization. Do not click links or open attachments unless you
> > >     > recognize the sender.
> > >     >
> > >
> >
> >
> 

Re: help

[Igor de Paula]

[-- Attachment #1: Type: text/plain, Size: 21434 bytes --]

Sure, I will submit a patch (it would be a first).
Thanks

On Fri, Aug 11, 2023 at 3:30 PM Thomas Monjalon <thomas@monjalon.net> wrote:

> Thanks for the info.
> Do you think it should be written in the vmxnet3 page of the DPDK
> documentation?
> If yes, would you like to initiate a patch for review?
>
>
> 11/08/2023 10:43, Igor de Paula:
> > Hi again,
> > I got this resolved with VMWARE support so I thought to share it here.
> > What I originally wanted was to use IOVA-VA on an AMD host. Which didn't
> > work. I have learned that the ESXI version that supports
> > virtual IOMMU in AMD hosts (which is a prerequisite to IOVA-VA) is ESXI
> 7.0
> > U1. After updating it worked. On Intel hosts ESXI 6.7 supports it already
> > as far as I know.
> >
> >
> > On Tue, Jul 25, 2023 at 6:19 PM Varghese, Vipin <Vipin.Varghese@amd.com>
> > wrote:
> >
> > > [AMD Official Use Only - General]
> > >
> > > Like I said earlier, trying with the Intel host I have on VMWARE,
> > > specifically  Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> > > With IOMMU enabled, VMXNET3 works with VA as well as PA.
> > >
> > > [VV] since ` enable_unsafe_iommu: not enabled` on Intel platform, could
> > > it be possible the specific version EXSI hypervisor supports the HW
> IOMMU
> > > specific to the platform. My suspicion will be in case on AMD platform
> > >  changes for required to enable HW iommu might not be available to
> specific
> > > EXSI (hypervior OS) used.
> > >
> > >
> > >
> > > I am not an expert on virtio_user PMD, but I can check if it will work
> > > with PA with deferred setting for vmx_net3 PMD are ok?
> > >
> > >
> > >
> > > *From:* Igor de Paula <igordptx@gmail.com>
> > > *Sent:* Tuesday, July 25, 2023 8:42 PM
> > > *To:* Varghese, Vipin <Vipin.Varghese@amd.com>
> > > *Cc:* Yigit, Ferruh <Ferruh.Yigit@amd.com>; Jochen Behrens <
> > > jbehrens@vmware.com>; Thomas Monjalon <thomas@monjalon.net>;
> > > users@dpdk.org; Gupta, Nipun <Nipun.Gupta@amd.com>; Agarwal, Nikhil <
> > > nikhil.agarwal@amd.com>; Ronak Doshi <doshir@vmware.com>; Immanni,
> Venkat
> > > <Venkat.Immanni@amd.com>; Chenbo Xia <chenbo.xia@intel.com>
> > > *Subject:* Re: help
> > >
> > >
> > >
> > > *Caution:* This message originated from an External Source. Use proper
> > > caution when opening attachments, clicking links, or responding.
> > >
> > >
> > >
> > > Well,
> > > Like I said earlier, trying with the Intel host I have on VMWARE,
> > > specifically  Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> > > With IOMMU enabled, VMXNET3 works with VA as well as PA.
> > > Meaning, PA works regardless if IOMMU is enabled or not. From my
> > > experience anyway.
> > > That's why I thought that:
> > > virtio_user needs VA to work.
> > > For some reason VMXNET3 does not work with VA (only on AMD host).
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Jul 25, 2023 at 4:04 PM Varghese, Vipin <
> Vipin.Varghese@amd.com>
> > > wrote:
> > >
> > > [AMD Official Use Only - General]
> > >
> > >
> > >
> > > Thanks Igor,
> > >
> > >
> > >
> > > As suspected the vmx_net3 works with
> > >
> > >
> > >
> > >    1. Iommu: disabled
> > >    2. enable_unsafe_iommu: enabled
> > >    3. dpdk eal iova mode: PA
> > >
> > >
> > >
> > > as pointed by you in logs, the virtio_user fails as it expects VA too.
> > >
> > >
> > >
> > > Will check and get back.
> > >
> > >
> > >
> > > *From:* Igor de Paula <igordptx@gmail.com>
> > > *Sent:* Tuesday, July 25, 2023 8:16 PM
> > > *To:* Yigit, Ferruh <Ferruh.Yigit@amd.com>
> > > *Cc:* Jochen Behrens <jbehrens@vmware.com>; Thomas Monjalon <
> > > thomas@monjalon.net>; users@dpdk.org; Gupta, Nipun <
> Nipun.Gupta@amd.com>;
> > > Agarwal, Nikhil <nikhil.agarwal@amd.com>; Ronak Doshi <
> doshir@vmware.com>;
> > > Immanni, Venkat <Venkat.Immanni@amd.com>; Varghese, Vipin <
> > > Vipin.Varghese@amd.com>; Chenbo Xia <chenbo.xia@intel.com>
> > > *Subject:* Re: help
> > >
> > >
> > >
> > > *Caution:* This message originated from an External Source. Use proper
> > > caution when opening attachments, clicking links, or responding.
> > >
> > >
> > >
> > > Hi,
> > > Attaching the logs of EAL when trying to run a configuration with
> > > virtio_user port when IOMMU is
> > > disabled and enable_unsafe_iommu is enabled. As you can see it forces
> IOVA
> > > as PA but the viritui_user needs IOVA as VA.
> > > I am also attaching the output of dmesg. I am not sure which kernel
> logs
> > > you wanted... if there is anything else please let me know..
> > > Regarding the ESXI logs, they are HUGE so I will send to you on a
> separate
> > > email.
> > >
> > >
> > >
> > > On Fri, Jul 21, 2023 at 1:14 PM Ferruh Yigit <ferruh.yigit@amd.com>
> wrote:
> > >
> > > On 7/21/2023 12:39 PM, Igor de Paula wrote:
> > > > I am trying to use virtio_user for an interface with the
> > > > kernel:
> > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html <
> > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html>
> > > > I think this requires IOVA as va.
> > > >
> > >
> > > I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo,
> > > he may know better.
> > >
> > > Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and
> > > '--iova-mode=pa'?
> > >
> > >
> > > > It does work with Intel host and IOMMU
> > > > enabled. Part of the negotiation when setting it up is getting the
> IOMMU
> > > > number so I thought it has to have IOMMU.
> > > >
> > >
> > > Yes, issue looks like related to the IOMMU, and it may be either
> related
> > > to HW support, or ESXi iommu driver support, we will check using below
> > > information you provided.
> > >
> > > > I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but
> > > > again, that didn't work.
> > > > ESXI version - VMware ESXi, 7.0.0, 16324942
> > > > AMD:  AMD EPYC 7452 32-Core Processo
> > > >
> > > > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @
> 2.60GHz
> > > > Regarding the logs I will try and attach it soon.
> > > >
> > >
> > > Thanks for info, waiting for logs.
> > >
> > > >
> > > > On Fri, Jul 21, 2023 at 12:21 PM Ferruh Yigit <ferruh.yigit@amd.com
> > > > <mailto:ferruh.yigit@amd.com>> wrote:
> > > >
> > > >     Hi Igor,
> > > >
> > > >     VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this
> can you
> > > >     please try with:
> > > >     - enable 'enable_unsafe_noiommu_mode' flag
> > > >     - Force PA mode via '--iova-mode=pa' eal parameter
> > > >
> > > >
> > > >     Also to be able to figure out AMD IOMMU support level, can you
> please
> > > >     provide:
> > > >     - AMD part number
> > > >     - ESXi/hypervisor version
> > > >     - ESXi & VM kernel logs
> > > >
> > > >
> > > >     Thanks,
> > > >     Ferruh
> > > >
> > > >     On 7/20/2023 5:21 PM, Jochen Behrens wrote:
> > > >     > +Ronak from the ESX team
> > > >     >
> > > >     >
> > > >     >
> > > >     >
> > > >     >
> > > >     > In our usage, we do set amd_iommu=off in the boot command line
> from
> > > >     > grub. (Or intel_iommu=off for Intel processors.)
> > > >     >
> > > >     >
> > > >     >
> > > >     >                 Jochen
> > > >     >
> > > >     >
> > > >     >
> > > >     > *From: *Thomas Monjalon <thomas@monjalon.net
> > > >     <mailto:thomas@monjalon.net>>
> > > >     > *Date: *Thursday, July 20, 2023 at 6:00 AM
> > > >     > *To: *Igor de Paula <igordptx@gmail.com <mailto:
> igordptx@gmail.com
> > > >>
> > > >     > *Cc: *users@dpdk.org <mailto:users@dpdk.org> <users@dpdk.org
> > > >     <mailto:users@dpdk.org>>, Jochen Behrens
> > > >     > <jbehrens@vmware.com <mailto:jbehrens@vmware.com>>, Nipun
> Gupta
> > > >     <nipun.gupta@amd.com <mailto:nipun.gupta@amd.com>>, Nikhil
> Agarwal
> > > >     > <nikhil.agarwal@amd.com <mailto:nikhil.agarwal@amd.com>>,
> Ferruh
> > > >     Yigit <ferruh.yigit@amd.com <mailto:ferruh.yigit@amd.com>>
> > > >     > *Subject: *Re: help
> > > >     >
> > > >     > !! External Email
> > > >     >
> > > >     > +Cc some AMD maintainers, they can have an idea about IOMMU
> > > settings.
> > > >     >
> > > >     >
> > > >     > 20/07/2023 14:44, Igor de Paula:
> > > >     >> I have enabled it in the host and in the BIOS for AMD...
> > > >     >> In the Bios I changed to amd_iommu=on and in the host it's the
> > > >     same for
> > > >     >> either.
> > > >     >>
> > > >     >> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon
> > > >     <thomas@monjalon.net <mailto:thomas@monjalon.net>> wrote:
> > > >     >>
> > > >     >> > 20/07/2023 11:35, Igor de Paula:
> > > >     >> > > The weird thing is that it only happens when I am using a
> > > >     host with an
> > > >     >> > AMD
> > > >     >> > > processor. It doesn't happen when I use a host with an
> Intel
> > > >     processor.
> > > >     >> >
> > > >     >> > So it's probably a matter of BIOS settings for the IOMMU?
> > > >     >> >
> > > >     >> >
> > > >     >> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon
> > > >     <thomas@monjalon.net <mailto:thomas@monjalon.net>>
> > > >     >> > > wrote:
> > > >     >> > >
> > > >     >> > > > +Cc the vmxnet3 maintainer.
> > > >     >> > > >
> > > >     >> > > > Please Jochen, do you have an idea what's wrong below?
> > > >     >> > > >
> > > >     >> > > >
> > > >     >> > > > 20/07/2023 11:25, Igor de Paula:
> > > >     >> > > > > This is because it can't negotiate the IOMMU type with
> > > >     any port.
> > > >     >> > > > >
> > > >     >> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon
> > > >     <thomas@monjalon.net <mailto:thomas@monjalon.net>
> > > >     >> > >
> > > >     >> > > > wrote:
> > > >     >> > > > >
> > > >     >> > > > > > Hello,
> > > >     >> > > > > >
> > > >     >> > > > > > The first error is "Cause: Error: number of ports
> must
> > > >     be even"
> > > >     >> > > > > >
> > > >     >> > > > > >
> > > >     >> > > > > > 03/05/2023 18:13, Igor de Paula:
> > > >     >> > > > > > > I am running a VM inside a VMWARE server
> (vSphere).
> > > >     >> > > > > > > My goal it to set up DPDK with two HW ports, and
> set
> > > up a
> > > >     >> > > > virtio_user to
> > > >     >> > > > > > > interact with the kernel stack.
> > > >     >> > > > > > > In another app I have it working but instead of
> > > >     virtio_user I am
> > > >     >> > > > running
> > > >     >> > > > > > > KNI, it works in IOVA-PA mode.
> > > >     >> > > > > > > I am looking to replace the KNI.
> > > >     >> > > > > > >
> > > >     >> > > > > > > When I try to set up virtio_user port as in the
> doc:
> > > >     >> > > > > > >
> > > >     >> > > > > >
> > > >     >> > > >
> > > >     >> >
> > > >
> > >
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> > > <
> > >
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> >
> > > <
> > >
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> > > <
> > >
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path
> > > >>
> > > >     >> > > > > > > I get a error it can't run in PA mode.
> > > >     >> > > > > > >
> > > >     >> > > > > > >
> > > >     >> > > > > > > When I try to run as VA mode from a parameter, I
> get
> > > the
> > > >     >> > following
> > > >     >> > > > > > errors:
> > > >     >> > > > > > > EAL: lib.eal log level changed from info to debug
> > > >     >> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0
> > > >     >> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0
> > > >     >> > > > > > > EAL: Support maximum 128 logical core(s) by
> > > >     configuration.
> > > >     >> > > > > > > EAL: Detected 2 lcore(s)
> > > >     >> > > > > > > EAL: Detected 1 NUMA nodes
> > > >     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3'
> > > >     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21'
> > > >     >> > > > > > > EAL: Checking presence of .so 'librte_eal.so'
> > > >     >> > > > > > > EAL: Detected static linkage of DPDK
> > > >     >> > > > > > > EAL: Ask a virtual area of 0x7000 bytes
> > > >     >> > > > > > > EAL: Virtual area found at 0x100000000 (size =
> 0x7000)
> > > >     >> > > > > > > EAL: Multi-process socket
> /var/run/dpdk/rte/mp_socket
> > > >     >> > > > > > > EAL: DPAA Bus not present. Skipping.
> > > >     >> > > > > > > EAL: VFIO PCI modules not loaded
> > > >     >> > > > > > > EAL: Selected IOVA mode 'VA'
> > > >     >> > > > > > > EAL: Probing VFIO support...
> > > >     >> > > > > > > EAL: IOMMU type 1 (Type 1) is supported
> > > >     >> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported
> > > >     >> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported
> > > >     >> > > > > > > EAL: VFIO support initialized
> > > >     >> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes
> > > >     >> > > > > > > EAL: Virtual area found at 0x100007000 (size =
> > > 0x5b000)
> > > >     >> > > > > > > EAL: Setting up physically contiguous memory...
> > > >     >> > > > > > > EAL: Setting maximum number of open files to
> 1048576
> > > >     >> > > > > > > EAL: Detected memory type: socket_id:0
> > > >     hugepage_sz:1073741824
> > > >     >> > > > > > > EAL: Creating 2 segment lists: n_segs:128
> socket_id:0
> > > >     >> > > > > > hugepage_sz:1073741824
> > > >     >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> > > >     >> > > > > > > EAL: Virtual area found at 0x100062000 (size =
> 0x2000)
> > > >     >> > > > > > > EAL: Memseg list allocated at socket 0, page size
> > > >     0x100000kB
> > > >     >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > >     >> > > > > > > EAL: Virtual area found at 0x140000000 (size =
> > > >     0x2000000000)
> > > >     >> > > > > > > EAL: VA reserved for memseg list at 0x140000000,
> size
> > > >     2000000000
> > > >     >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
> > > >     >> > > > > > > EAL: Virtual area found at 0x2140000000 (size =
> > > 0x2000)
> > > >     >> > > > > > > EAL: Memseg list allocated at socket 0, page size
> > > >     0x100000kB
> > > >     >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > >     >> > > > > > > EAL: Virtual area found at 0x2180000000 (size =
> > > >     0x2000000000)
> > > >     >> > > > > > > EAL: VA reserved for memseg list at 0x2180000000,
> > > >     size 2000000000
> > > >     >> > > > > > > EAL: TSC frequency is ~2350000 KHz
> > > >     >> > > > > > > EAL: Main lcore 0 is ready
> > > (tid=7f8ad790ec00;cpuset=[0])
> > > >     >> > > > > > > EAL: lcore 1 is ready
> (tid=7f8ad6907400;cpuset=[1])
> > > >     >> > > > > > > EAL: Trying to obtain current memory policy.
> > > >     >> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0
> > > >     >> > > > > > > EAL: Restoring previous memory policy: 0
> > > >     >> > > > > > > EAL: request: mp_malloc_sync
> > > >     >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> > > >     >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > >     >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode
> is
> > > >     'VA', not
> > > >     >> > > > > > initializing
> > > >     >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> > > >     >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > >     >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode
> is
> > > >     'VA', not
> > > >     >> > > > > > initializing
> > > >     >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> > > >     >> > > > > > > EAL: Bus (pci) probe failed.
> > > >     >> > > > > > > EAL: lib.telemetry log level changed from
> disabled to
> > > >     warning
> > > >     >> > > > > > > EAL: Error - exiting with code: 1
> > > >     >> > > > > > >   Cause: Error: number of ports must be even
> > > >     >> > > > > > > EAL: request: mp_malloc_sync
> > > >     >> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB
> > > >     >> > > > > > >
> > > >     >> > > > > > >
> > > >     >> > > > > > >
> > > >     >> > > > > > > For some reason the HW ports won't setup. From
> what I
> > > >     understand
> > > >     >> > > > > > > net_vmxnet3 should work with VA mode.
> > > >     >> > > > > > > I enabled I/OMUU for the VM.
> > > >     >> > > > > > > The weird thing even when enabled, I still have
> the
> > > >     >> > > > > > > enable_unsafe_noiommu_mode flag on.
> > > >     >> > > > > > > And because it's on the this:
> > > >     >> > > > > > >
> > > >     >> > > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev);
> > > >     >> > > > > > >
> > > >     >> > > > > > > return PA mode, and it fails.
> > > >     >> > > > > > >
> > > >     >> > > > > > > When I disable it by modifying
> > > >     >> > > > > > >
> > > >     /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get
> > > >     >> > another
> > > >     >> > > > > > error.
> > > >     >> > > > > > > The error is that it doesn't find a suitable IOMMU
> > > type:
> > > >     >> > > > > > > Just putting the relevant message:
> > > >     >> > > > > > >
> > > >     >> > > > > > >
> > > >     >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
> > > >     >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > >     >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19
> (No
> > > >     such device)
> > > >     >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
> > > >     such device)
> > > >     >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19
> (No
> > > >     such
> > > >     >> > device)
> > > >     >> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type
> > > >     >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
> > > >     >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > >     >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
> > > >     >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19
> (No
> > > >     such device)
> > > >     >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No
> > > >     such device)
> > > >     >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19
> (No
> > > >     such
> > > >     >> > device)
> > > >     >> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type
> > > >     >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
> > > >     >> > > > > > >
> > > >     >> > > > > > >
> > > >     >> > > > > > > I can see where it fails:
> > > >     >> > > > > > >
> > > >     >> > > > > > >
> > > >     >> > > > > > > const struct vfio_iommu_type *
> > > >     >> > > > > > > vfio_set_iommu_type(int vfio_container_fd)
> > > >     >> > > > > > > {
> > > >     >> > > > > > > unsigned idx;
> > > >     >> > > > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
> > > >     >> > > > > > > const struct vfio_iommu_type *t =
> &iommu_types[idx];
> > > >     >> > > > > > >
> > > >     >> > > > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
> > > >     >> > > > > > > t->type_id);
> > > >     >> > > > > > > if (!ret) {
> > > >     >> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> > > >     >> > > > > > > t->type_id, t->name);
> > > >     >> > > > > > > return t;
> > > >     >> > > > > > > }
> > > >     >> > > > > > > /* not an error, there may be more supported IOMMU
> > > >     types */
> > > >     >> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s)
> failed,
> > > >     error "
> > > >     >> > > > > > > "%i (%s)\n", t->type_id, t->name, errno,
> > > >     >> > > > > > > strerror(errno));
> > > >     >> > > > > > > }
> > > >     >> > > > > > > /* if we didn't find a suitable IOMMU type, fail
> */
> > > >     >> > > > > > > return NULL;
> > > >     >> > > > > > > }
> > > >     >> > > > > > >
> > > >     >> > > > > > >
> > > >     >> > > > > > > The ioctl returns -1.
> > > >     >> > > > > > >
> > > >     >> > > > > > >
> > > >     >> > > > > > >
> > > >     >> > > > > > > I tried many things.
> > > >     >> > > > > > > Enabling/disabling IOMMU.
> > > >     >> > > > > > > Modifying the enable_unsafe_noiommu_mode flag.
> > > >     >> > > > > > > Running in PA mode.
> > > >     >> > > > > > > Any help will be much appreciated.
> > > >     >> >
> > > >     >> >
> > > >     >> >
> > > >     >> >
> > > >     >> >
> > > >     >> >
> > > >     >>
> > > >     >
> > > >     >
> > > >     >
> > > >     >
> > > >     >
> > > >     >
> > > >     > !! External Email: This email originated from outside of the
> > > >     > organization. Do not click links or open attachments unless you
> > > >     > recognize the sender.
> > > >     >
> > > >
> > >
> > >
> >
>
>
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 35725 bytes --]