From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 68E6D42A51 for ; Wed, 3 May 2023 18:11:15 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2A8AE41144; Wed, 3 May 2023 18:11:15 +0200 (CEST) Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by mails.dpdk.org (Postfix) with ESMTP id 4316E410F9 for ; Wed, 3 May 2023 18:11:13 +0200 (CEST) Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-1aae46e62e9so32582275ad.2 for ; Wed, 03 May 2023 09:11:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683130272; x=1685722272; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=ElheTEdWTgZS/p4kyHYRpzR0UdwxoWMUuhM0vVAes7c=; b=bx+1WZmxGHOLUmvKFtPE/tfCk4nxh0xQLKdd/b7W/cgfW661pbUr6OY5gKrFOUlMJi IbFMR6msoAsdiKQkYaKukpwEPAKkQpgsNdSCNIiSPsT/1o5EspXJDcFsg5k81BDLg08B I3vItRHJur2Oh5+MmKPIB3uIEUMRYFbItyvCOAO/ysQKFleFSGDQ+B5t2nt4OUwlvxcT KPiDctinCrApYOK+gm1a0vi6zlrSW+xyIWepFbct+vqgbOp6UFgyw68C8PZMT4/uMbg2 Xpd7+PCIfht4CZKMzQElW6vRDmnCrsC2hdxE/ozF6zvDHBuKdPCY/do94Wdae9bhjfqB kDbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683130272; x=1685722272; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ElheTEdWTgZS/p4kyHYRpzR0UdwxoWMUuhM0vVAes7c=; b=FKbVyFjzTSRCVvAIoyOD/Z5cWfj3lKk5h5IeNpScFE347G9YLsF2RIu7RddlpsfL/+ yKJ37cXXmNuRTBnM4FfTZ53xPhcrme/pdDEhid8vXXoAgU8hj0ikJBqnBxFZU4OZNtVy v8mThy+Hm4MAMENfLgkX2w/058ZN6yxl/Q4Gp6H7yHHXbJL/DSpnSW8BhozaLgdN1e5k xKS7FDJDDrl2kexFZnm+40Vz7/YX0NMtb/MewMQ3mT+lDZfrjU22IroS/8F97Hme3xVi DnjA380xAaoQeSH/Hj+fA5m3HXKjDRPBPiKVS3l3entkAujnFjhUzvK38yxOizlNr8RV VdOw== X-Gm-Message-State: AC+VfDxFeTcqwzDKpa8FnD/wNhtUmbIn78S3Msn/qdX32ysEAqVX2R77 ZDLtX0l9zORmO/95lAK+3YMmqH6z4UVJwmqMDa8qQQZ6qjE= X-Google-Smtp-Source: ACHHUZ7A7/uGMD2kcd3iA+HYNOvt1pOvc3R7Qdt89lnxEvoGBgPSugZOEJ4HDZbSGGMZbfFZRM+TXBwpIiVzhPUDI8s= X-Received: by 2002:a17:902:e806:b0:1ab:1dff:9540 with SMTP id u6-20020a170902e80600b001ab1dff9540mr689945plg.14.1683130271881; Wed, 03 May 2023 09:11:11 -0700 (PDT) MIME-Version: 1.0 From: Igor de Paula Date: Wed, 3 May 2023 17:13:29 +0100 Message-ID: Subject: help To: users@dpdk.org Content-Type: multipart/alternative; boundary="000000000000f75f8f05facc4d4c" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --000000000000f75f8f05facc4d4c Content-Type: text/plain; charset="UTF-8" I am running a VM inside a VMWARE server (vSphere). My goal it to set up DPDK with two HW ports, and set up a virtio_user to interact with the kernel stack. In another app I have it working but instead of virtio_user I am running KNI, it works in IOVA-PA mode. I am looking to replace the KNI. When I try to set up virtio_user port as in the doc: https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path I get a error it can't run in PA mode. When I try to run as VA mode from a parameter, I get the following errors: EAL: lib.eal log level changed from info to debug EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 0 on socket 0 EAL: Support maximum 128 logical core(s) by configuration. EAL: Detected 2 lcore(s) EAL: Detected 1 NUMA nodes EAL: Checking presence of .so 'librte_eal.so.21.3' EAL: Checking presence of .so 'librte_eal.so.21' EAL: Checking presence of .so 'librte_eal.so' EAL: Detected static linkage of DPDK EAL: Ask a virtual area of 0x7000 bytes EAL: Virtual area found at 0x100000000 (size = 0x7000) EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: DPAA Bus not present. Skipping. EAL: VFIO PCI modules not loaded EAL: Selected IOVA mode 'VA' EAL: Probing VFIO support... EAL: IOMMU type 1 (Type 1) is supported EAL: IOMMU type 7 (sPAPR) is not supported EAL: IOMMU type 8 (No-IOMMU) is supported EAL: VFIO support initialized EAL: Ask a virtual area of 0x5b000 bytes EAL: Virtual area found at 0x100007000 (size = 0x5b000) EAL: Setting up physically contiguous memory... EAL: Setting maximum number of open files to 1048576 EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824 EAL: Creating 2 segment lists: n_segs:128 socket_id:0 hugepage_sz:1073741824 EAL: Ask a virtual area of 0x2000 bytes EAL: Virtual area found at 0x100062000 (size = 0x2000) EAL: Memseg list allocated at socket 0, page size 0x100000kB EAL: Ask a virtual area of 0x2000000000 bytes EAL: Virtual area found at 0x140000000 (size = 0x2000000000) EAL: VA reserved for memseg list at 0x140000000, size 2000000000 EAL: Ask a virtual area of 0x2000 bytes EAL: Virtual area found at 0x2140000000 (size = 0x2000) EAL: Memseg list allocated at socket 0, page size 0x100000kB EAL: Ask a virtual area of 0x2000000000 bytes EAL: Virtual area found at 0x2180000000 (size = 0x2000000000) EAL: VA reserved for memseg list at 0x2180000000, size 2000000000 EAL: TSC frequency is ~2350000 KHz EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0]) EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1]) EAL: Trying to obtain current memory policy. EAL: Setting policy MPOL_PREFERRED for socket 0 EAL: Restoring previous memory policy: 0 EAL: request: mp_malloc_sync EAL: Heap on socket 0 was expanded by 1024MB EAL: PCI device 0000:0b:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 net_vmxnet3 EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0000:0b:00.0 cannot be used EAL: PCI device 0000:13:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 net_vmxnet3 EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0000:13:00.0 cannot be used EAL: Bus (pci) probe failed. EAL: lib.telemetry log level changed from disabled to warning EAL: Error - exiting with code: 1 Cause: Error: number of ports must be even EAL: request: mp_malloc_sync EAL: Heap on socket 0 was shrunk by 1024MB For some reason the HW ports won't setup. From what I understand net_vmxnet3 should work with VA mode. I enabled I/OMUU for the VM. The weird thing even when enabled, I still have the enable_unsafe_noiommu_mode flag on. And because it's on the this: dev_iova_mode = pci_device_iova_mode(dr, dev); return PA mode, and it fails. When I disable it by modifying /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get another error. The error is that it doesn't find a suitable IOMMU type: Just putting the relevant message: EAL: Heap on socket 0 was expanded by 1024MB EAL: PCI device 0000:0b:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 net_vmxnet3 EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device) EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device) EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device) EAL: 0000:0b:00.0 failed to select IOMMU type EAL: Requested device 0000:0b:00.0 cannot be used EAL: PCI device 0000:13:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 net_vmxnet3 EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device) EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device) EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device) EAL: 0000:13:00.0 failed to select IOMMU type EAL: Requested device 0000:13:00.0 cannot be used I can see where it fails: const struct vfio_iommu_type * vfio_set_iommu_type(int vfio_container_fd) { unsigned idx; for (idx = 0; idx < RTE_DIM(iommu_types); idx++) { const struct vfio_iommu_type *t = &iommu_types[idx]; int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU, t->type_id); if (!ret) { RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n", t->type_id, t->name); return t; } /* not an error, there may be more supported IOMMU types */ RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error " "%i (%s)\n", t->type_id, t->name, errno, strerror(errno)); } /* if we didn't find a suitable IOMMU type, fail */ return NULL; } The ioctl returns -1. I tried many things. Enabling/disabling IOMMU. Modifying the enable_unsafe_noiommu_mode flag. Running in PA mode. Any help will be much appreciated. --000000000000f75f8f05facc4d4c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I am running a VM inside a VMWARE server (vSphere).
My = goal it to set up DPDK with two HW ports, and set up a virtio_user to inter= act with the kernel stack.
In another app I have it working but instead = of virtio_user I am running KNI, it works in IOVA-PA mode.
I am looking = to replace the KNI.

When I try to set up virtio_user port as in the = doc:
https://doc.dpdk.org/guides/how= to/virtio_user_as_exception_path.html#virtio-user-as-exception-path
= I get a error it can't run in PA mode.


When I try to run as = VA mode from a parameter, I get the following errors:
EAL: lib.eal log l= evel changed from info to debug
EAL: Detected lcore 0 as core 0 on socke= t 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Support maximum = 128 logical core(s) by configuration.
EAL: Detected 2 lcore(s)
EAL: D= etected 1 NUMA nodes
EAL: Checking presence of .so 'librte_eal.so.21= .3'
EAL: Checking presence of .so 'librte_eal.so.21'
EAL:= Checking presence of .so 'librte_eal.so'
EAL: Detected static l= inkage of DPDK
EAL: Ask a virtual area of 0x7000 bytes
EAL: Virtual a= rea found at 0x100000000 (size =3D 0x7000)
EAL: Multi-process socket /va= r/run/dpdk/rte/mp_socket
EAL: DPAA Bus not present. Skipping.
EAL: VF= IO PCI modules not loaded
EAL: Selected IOVA mode 'VA'
EAL: P= robing VFIO support...
EAL: IOMMU type 1 (Type 1) is supported
EAL: I= OMMU type 7 (sPAPR) is not supported
EAL: IOMMU type 8 (No-IOMMU) is sup= ported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x5b0= 00 bytes
EAL: Virtual area found at 0x100007000 (size =3D 0x5b000)
EA= L: Setting up physically contiguous memory...
EAL: Setting maximum numbe= r of open files to 1048576
EAL: Detected memory type: socket_id:0 hugepa= ge_sz:1073741824
EAL: Creating 2 segment lists: n_segs:128 socket_id:0 h= ugepage_sz:1073741824
EAL: Ask a virtual area of 0x2000 bytes
EAL: Vi= rtual area found at 0x100062000 (size =3D 0x2000)
EAL: Memseg list alloc= ated at socket 0, page size 0x100000kB
EAL: Ask a virtual area of 0x2000= 000000 bytes
EAL: Virtual area found at 0x140000000 (size =3D 0x20000000= 00)
EAL: VA reserved for memseg list at 0x140000000, size 2000000000
= EAL: Ask a virtual area of 0x2000 bytes
EAL: Virtual area found at 0x214= 0000000 (size =3D 0x2000)
EAL: Memseg list allocated at socket 0, page s= ize 0x100000kB
EAL: Ask a virtual area of 0x2000000000 bytes
EAL: Vir= tual area found at 0x2180000000 (size =3D 0x2000000000)
EAL: VA reserved= for memseg list at 0x2180000000, size 2000000000
EAL: TSC frequency is = ~2350000 KHz
EAL: Main lcore 0 is ready (tid=3D7f8ad790ec00;cpuset=3D[0]= )
EAL: lcore 1 is ready (tid=3D7f8ad6907400;cpuset=3D[1])
EAL: Trying= to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for= socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_m= alloc_sync
EAL: Heap on socket 0 was expanded by 1024MB
EAL: PCI devi= ce 0000:0b:00.0 on NUMA socket -1
EAL: =C2=A0 probe driver: 15ad:7b0 net= _vmxnet3
EAL: =C2=A0 Expecting 'PA' IOVA mode but current mode i= s 'VA', not initializing
EAL: Requested device 0000:0b:00.0 cann= ot be used
EAL: PCI device 0000:13:00.0 on NUMA socket -1
EAL: =C2=A0= probe driver: 15ad:7b0 net_vmxnet3
EAL: =C2=A0 Expecting 'PA' I= OVA mode but current mode is 'VA', not initializing
EAL: Request= ed device 0000:13:00.0 cannot be used
EAL: Bus (pci) probe failed.
EA= L: lib.telemetry log level changed from disabled to warning
EAL: Error -= exiting with code: 1
=C2=A0 Cause: Error: number of ports must be even<= br>EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 1024= MB



For some reason the HW ports won't setup. From what I= understand net_vmxnet3 should work with VA mode.
I enabled I/OMUU for t= he VM.
The weird thing even when enabled, I still have the enable_unsafe= _noiommu_mode flag on.
And because it's on the this:

dev_iova= _mode =3D pci_device_iova_mode(dr, dev);

return PA mode, and it fail= s.

When I disable it by modifying /sys/module/vfio/parameters/enable= _unsafe_noiommu_mode, I get another error.
The error is that it doesn= 9;t find a suitable IOMMU type:
Just putting the relevant message:

EAL: Heap on socket 0 was expanded by 1024MB
EAL: PCI device 0000:= 0b:00.0 on NUMA socket -1
EAL: =C2=A0 probe driver: 15ad:7b0 net_vmxnet3=
EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
EAL= : Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
EAL: Set IO= MMU type 8 (No-IOMMU) failed, error 19 (No such device)
EAL: 0000:0b:00.= 0 failed to select IOMMU type
EAL: Requested device 0000:0b:00.0 cannot = be used
EAL: PCI device 0000:13:00.0 on NUMA socket -1
EAL: =C2=A0 pr= obe driver: 15ad:7b0 net_vmxnet3
EAL: Set IOMMU type 1 (Type 1) failed, = error 19 (No such device)
EAL: Set IOMMU type 7 (sPAPR) failed, error 19= (No such device)
EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No = such device)
EAL: 0000:13:00.0 failed to select IOMMU type
EAL: Reque= sted device 0000:13:00.0 cannot be used


I can see where it fails= :


const struct vfio_iommu_type *
vfio_set_iommu_type(int vfio= _container_fd)
{
unsigned idx;
for (idx =3D 0; idx < RTE_DIM(= iommu_types); idx++) {
const struct vfio_iommu_type *t =3D &iommu_= types[idx];

int ret =3D ioctl(vfio_container_fd, VFIO_SET_IOMMU, t->type_id);
if (!ret) {
RTE_LOG(INFO, EAL, "Using= IOMMU type %d (%s)\n",
t->type_id, t->name);
retu= rn t;
}
/* not an error, there may be more supported IOMMU types = */
RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error &quo= t;
"%i (%s)\n", t->type_id, t->name, errno,
s= trerror(errno));
}
/* if we didn't find a suitable IOMMU type, = fail */
return NULL;
}


The ioctl returns -1.


<= br>I tried many things.
Enabling/disabling IOMMU.
Modifying the enabl= e_unsafe_noiommu_mode flag.
Running in PA mode.
Any help will be much= appreciated.=C2=A0
--000000000000f75f8f05facc4d4c--