From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 27B7143032 for ; Fri, 11 Aug 2023 10:44:01 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DEC8840F16; Fri, 11 Aug 2023 10:44:00 +0200 (CEST) Received: from mail-yb1-f171.google.com (mail-yb1-f171.google.com [209.85.219.171]) by mails.dpdk.org (Postfix) with ESMTP id 345DB40E03 for ; Fri, 11 Aug 2023 10:43:59 +0200 (CEST) Received: by mail-yb1-f171.google.com with SMTP id 3f1490d57ef6-d64f0c2bc95so1048313276.1 for ; Fri, 11 Aug 2023 01:43:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691743438; x=1692348238; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=2kx4/upL+QBsuHoQMPFoBiOSpJYH341hcMJXhrC2mCM=; b=l4znz7y2dZv14WsFW0hoapKQLnEv6uOSe2JziNsjsgD50zJ7GQ6FWcQW0slBuP6dBA g+iz33kRUm4nYaRAfqVgMhmmJ6i0ra2/14QPfLH+dVbTHe3PnCFyl8K45ZVB6C0qRoAl n/yFaQbRQpRid7QK+o30qB+6fcT+VfmnXkhtUyerVrN2XbK4GWvRqyjCxgK2l6505oAz QFUMsVIbw8bALPDyER63RiYzwfKfcaVzFU2KgMXNNBNYrpJax4H6OOi20J5DgFi30Lzq sqX55Lnebn2os85gQPehfZCC3PTbtbJVtV8QMhWXzkxPdoJEwSYC8vfGN1ha7weL5u9k 7iXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691743438; x=1692348238; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=2kx4/upL+QBsuHoQMPFoBiOSpJYH341hcMJXhrC2mCM=; b=PVGXpStDpDaE2g+xQKS28essWElWran/LLIJoJhkpTBQBdc7mz++he8OQRFafeUroH 2i8+mlNLkC0IVTmys03D5DzSAQ6GDH8cCiVEmv6RO3B32LVpj0nu3wni3M71KSWAcvVw RhidCorbhLWVebMq6X0FS7QrzONdgiY3d1s0T3SQqRTm53/gb8F8dOh+0FqY8msck5e4 eo6bEjjVTdhLQdj3ROYCjt4+AfuGZwBZIZ6QIL6Xf8ZordrW9wi4Iz8W6iuXTRahdUHx aZyfa0Xy8LFgZknUI4gBR9B6wsbg4TPI/DD/zjPiSrL7b/H1AAc8/YTmNekY/AhJZ+KZ b/gg== X-Gm-Message-State: AOJu0YwOS6jofxbnvXrK8pOFif13rYG97TEZXxqSxBYHYcq7lnQ6p5DP tRzkJv6ESTPgUSQme3KwakWu4yHTCgeyCRnc6do= X-Google-Smtp-Source: AGHT+IFbKgutFfQxKViAX705sfNldMsbJaxFqhR/gztpJ5t5f1OOtnZWNXPA3Kz3fQbbNI3KCZNUG/7/W4woflEGqb0= X-Received: by 2002:a25:ab27:0:b0:d49:4869:1bd1 with SMTP id u36-20020a25ab27000000b00d4948691bd1mr1073265ybi.6.1691743438307; Fri, 11 Aug 2023 01:43:58 -0700 (PDT) MIME-Version: 1.0 References: <2663502.2IRrRt1zHL@thomas> <7070795.j6PcuT4dK6@thomas> <32a6b0b0-c76d-b535-e727-e62e796ca429@amd.com> <69c5475c-fe50-552d-7299-47d2781749aa@amd.com> In-Reply-To: From: Igor de Paula Date: Fri, 11 Aug 2023 11:43:47 +0300 Message-ID: Subject: Re: help To: "Varghese, Vipin" Cc: "Yigit, Ferruh" , Jochen Behrens , Thomas Monjalon , "users@dpdk.org" , "Gupta, Nipun" , "Agarwal, Nikhil" , Ronak Doshi , "Immanni, Venkat" , Chenbo Xia Content-Type: multipart/alternative; boundary="000000000000b12f850602a1b6d6" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --000000000000b12f850602a1b6d6 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi again, I got this resolved with VMWARE support so I thought to share it here. What I originally wanted was to use IOVA-VA on an AMD host. Which didn't work. I have learned that the ESXI version that supports virtual IOMMU in AMD hosts (which is a prerequisite to IOVA-VA) is ESXI 7.0 U1. After updating it worked. On Intel hosts ESXI 6.7 supports it already as far as I know. On Tue, Jul 25, 2023 at 6:19=E2=80=AFPM Varghese, Vipin wrote: > [AMD Official Use Only - General] > > Like I said earlier, trying with the Intel host I have on VMWARE, > specifically Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz > With IOMMU enabled, VMXNET3 works with VA as well as PA. > > [VV] since ` enable_unsafe_iommu: not enabled` on Intel platform, could > it be possible the specific version EXSI hypervisor supports the HW IOMMU > specific to the platform. My suspicion will be in case on AMD platform > changes for required to enable HW iommu might not be available to specif= ic > EXSI (hypervior OS) used. > > > > I am not an expert on virtio_user PMD, but I can check if it will work > with PA with deferred setting for vmx_net3 PMD are ok? > > > > *From:* Igor de Paula > *Sent:* Tuesday, July 25, 2023 8:42 PM > *To:* Varghese, Vipin > *Cc:* Yigit, Ferruh ; Jochen Behrens < > jbehrens@vmware.com>; Thomas Monjalon ; > users@dpdk.org; Gupta, Nipun ; Agarwal, Nikhil < > nikhil.agarwal@amd.com>; Ronak Doshi ; Immanni, Venkat > ; Chenbo Xia > *Subject:* Re: help > > > > *Caution:* This message originated from an External Source. Use proper > caution when opening attachments, clicking links, or responding. > > > > Well, > Like I said earlier, trying with the Intel host I have on VMWARE, > specifically Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz > With IOMMU enabled, VMXNET3 works with VA as well as PA. > Meaning, PA works regardless if IOMMU is enabled or not. From my > experience anyway. > That's why I thought that: > virtio_user needs VA to work. > For some reason VMXNET3 does not work with VA (only on AMD host). > > > > > > On Tue, Jul 25, 2023 at 4:04=E2=80=AFPM Varghese, Vipin > wrote: > > [AMD Official Use Only - General] > > > > Thanks Igor, > > > > As suspected the vmx_net3 works with > > > > 1. Iommu: disabled > 2. enable_unsafe_iommu: enabled > 3. dpdk eal iova mode: PA > > > > as pointed by you in logs, the virtio_user fails as it expects VA too. > > > > Will check and get back. > > > > *From:* Igor de Paula > *Sent:* Tuesday, July 25, 2023 8:16 PM > *To:* Yigit, Ferruh > *Cc:* Jochen Behrens ; Thomas Monjalon < > thomas@monjalon.net>; users@dpdk.org; Gupta, Nipun ; > Agarwal, Nikhil ; Ronak Doshi = ; > Immanni, Venkat ; Varghese, Vipin < > Vipin.Varghese@amd.com>; Chenbo Xia > *Subject:* Re: help > > > > *Caution:* This message originated from an External Source. Use proper > caution when opening attachments, clicking links, or responding. > > > > Hi, > Attaching the logs of EAL when trying to run a configuration with > virtio_user port when IOMMU is > disabled and enable_unsafe_iommu is enabled. As you can see it forces IOV= A > as PA but the viritui_user needs IOVA as VA. > I am also attaching the output of dmesg. I am not sure which kernel logs > you wanted... if there is anything else please let me know.. > Regarding the ESXI logs, they are HUGE so I will send to you on a separat= e > email. > > > > On Fri, Jul 21, 2023 at 1:14=E2=80=AFPM Ferruh Yigit wrote: > > On 7/21/2023 12:39 PM, Igor de Paula wrote: > > I am trying to use virtio_user for an interface with the > > kernel: > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html < > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html> > > I think this requires IOVA as va. > > > > I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo, > he may know better. > > Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and > '--iova-mode=3Dpa'? > > > > It does work with Intel host and IOMMU > > enabled. Part of the negotiation when setting it up is getting the IOMM= U > > number so I thought it has to have IOMMU. > > > > Yes, issue looks like related to the IOMMU, and it may be either related > to HW support, or ESXi iommu driver support, we will check using below > information you provided. > > > I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but > > again, that didn't work. > > ESXI version - VMware ESXi, 7.0.0, 16324942 > > AMD: AMD EPYC 7452 32-Core Processo > > > > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60G= Hz > > Regarding the logs I will try and attach it soon. > > > > Thanks for info, waiting for logs. > > > > > On Fri, Jul 21, 2023 at 12:21=E2=80=AFPM Ferruh Yigit > > wrote: > > > > Hi Igor, > > > > VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can y= ou > > please try with: > > - enable 'enable_unsafe_noiommu_mode' flag > > - Force PA mode via '--iova-mode=3Dpa' eal parameter > > > > > > Also to be able to figure out AMD IOMMU support level, can you plea= se > > provide: > > - AMD part number > > - ESXi/hypervisor version > > - ESXi & VM kernel logs > > > > > > Thanks, > > Ferruh > > > > On 7/20/2023 5:21 PM, Jochen Behrens wrote: > > > +Ronak from the ESX team > > > > > > > > > > > > > > > > > > In our usage, we do set amd_iommu=3Doff in the boot command line = from > > > grub. (Or intel_iommu=3Doff for Intel processors.) > > > > > > > > > > > > Jochen > > > > > > > > > > > > *From: *Thomas Monjalon > > > > > *Date: *Thursday, July 20, 2023 at 6:00 AM > > > *To: *Igor de Paula >> > > > *Cc: *users@dpdk.org > >, Jochen Behrens > > > >, Nipun Gupta > > >, Nikhil Agarwal > > > >, Ferruh > > Yigit > > > > *Subject: *Re: help > > > > > > !! External Email > > > > > > +Cc some AMD maintainers, they can have an idea about IOMMU > settings. > > > > > > > > > 20/07/2023 14:44, Igor de Paula: > > >> I have enabled it in the host and in the BIOS for AMD... > > >> In the Bios I changed to amd_iommu=3Don and in the host it's the > > same for > > >> either. > > >> > > >> On Thu, Jul 20, 2023 at 1:31=E2=80=AFPM Thomas Monjalon > > > wrote: > > >> > > >> > 20/07/2023 11:35, Igor de Paula: > > >> > > The weird thing is that it only happens when I am using a > > host with an > > >> > AMD > > >> > > processor. It doesn't happen when I use a host with an Intel > > processor. > > >> > > > >> > So it's probably a matter of BIOS settings for the IOMMU? > > >> > > > >> > > > >> > > On Thu, Jul 20, 2023 at 10:32=E2=80=AFAM Thomas Monjalon > > > > > >> > > wrote: > > >> > > > > >> > > > +Cc the vmxnet3 maintainer. > > >> > > > > > >> > > > Please Jochen, do you have an idea what's wrong below? > > >> > > > > > >> > > > > > >> > > > 20/07/2023 11:25, Igor de Paula: > > >> > > > > This is because it can't negotiate the IOMMU type with > > any port. > > >> > > > > > > >> > > > > On Thu, Jul 20, 2023 at 5:08=E2=80=AFAM Thomas Monjalon > > > > >> > > > > >> > > > wrote: > > >> > > > > > > >> > > > > > Hello, > > >> > > > > > > > >> > > > > > The first error is "Cause: Error: number of ports must > > be even" > > >> > > > > > > > >> > > > > > > > >> > > > > > 03/05/2023 18:13, Igor de Paula: > > >> > > > > > > I am running a VM inside a VMWARE server (vSphere). > > >> > > > > > > My goal it to set up DPDK with two HW ports, and set > up a > > >> > > > virtio_user to > > >> > > > > > > interact with the kernel stack. > > >> > > > > > > In another app I have it working but instead of > > virtio_user I am > > >> > > > running > > >> > > > > > > KNI, it works in IOVA-PA mode. > > >> > > > > > > I am looking to replace the KNI. > > >> > > > > > > > > >> > > > > > > When I try to set up virtio_user port as in the doc: > > >> > > > > > > > > >> > > > > > > > >> > > > > > >> > > > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virt= io-user-as-exception-path > < > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virt= io-user-as-exception-path> > < > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virt= io-user-as-exception-path > < > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virt= io-user-as-exception-path > >> > > >> > > > > > > I get a error it can't run in PA mode. > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > When I try to run as VA mode from a parameter, I get > the > > >> > following > > >> > > > > > errors: > > >> > > > > > > EAL: lib.eal log level changed from info to debug > > >> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0 > > >> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0 > > >> > > > > > > EAL: Support maximum 128 logical core(s) by > > configuration. > > >> > > > > > > EAL: Detected 2 lcore(s) > > >> > > > > > > EAL: Detected 1 NUMA nodes > > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3' > > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21' > > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so' > > >> > > > > > > EAL: Detected static linkage of DPDK > > >> > > > > > > EAL: Ask a virtual area of 0x7000 bytes > > >> > > > > > > EAL: Virtual area found at 0x100000000 (size =3D 0x7= 000) > > >> > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socke= t > > >> > > > > > > EAL: DPAA Bus not present. Skipping. > > >> > > > > > > EAL: VFIO PCI modules not loaded > > >> > > > > > > EAL: Selected IOVA mode 'VA' > > >> > > > > > > EAL: Probing VFIO support... > > >> > > > > > > EAL: IOMMU type 1 (Type 1) is supported > > >> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported > > >> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported > > >> > > > > > > EAL: VFIO support initialized > > >> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes > > >> > > > > > > EAL: Virtual area found at 0x100007000 (size =3D > 0x5b000) > > >> > > > > > > EAL: Setting up physically contiguous memory... > > >> > > > > > > EAL: Setting maximum number of open files to 1048576 > > >> > > > > > > EAL: Detected memory type: socket_id:0 > > hugepage_sz:1073741824 > > >> > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:= 0 > > >> > > > > > hugepage_sz:1073741824 > > >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes > > >> > > > > > > EAL: Virtual area found at 0x100062000 (size =3D 0x2= 000) > > >> > > > > > > EAL: Memseg list allocated at socket 0, page size > > 0x100000kB > > >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes > > >> > > > > > > EAL: Virtual area found at 0x140000000 (size =3D > > 0x2000000000) > > >> > > > > > > EAL: VA reserved for memseg list at 0x140000000, siz= e > > 2000000000 > > >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes > > >> > > > > > > EAL: Virtual area found at 0x2140000000 (size =3D > 0x2000) > > >> > > > > > > EAL: Memseg list allocated at socket 0, page size > > 0x100000kB > > >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes > > >> > > > > > > EAL: Virtual area found at 0x2180000000 (size =3D > > 0x2000000000) > > >> > > > > > > EAL: VA reserved for memseg list at 0x2180000000, > > size 2000000000 > > >> > > > > > > EAL: TSC frequency is ~2350000 KHz > > >> > > > > > > EAL: Main lcore 0 is ready > (tid=3D7f8ad790ec00;cpuset=3D[0]) > > >> > > > > > > EAL: lcore 1 is ready (tid=3D7f8ad6907400;cpuset=3D[= 1]) > > >> > > > > > > EAL: Trying to obtain current memory policy. > > >> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0 > > >> > > > > > > EAL: Restoring previous memory policy: 0 > > >> > > > > > > EAL: request: mp_malloc_sync > > >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB > > >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > >> > > > > > > EAL: Expecting 'PA' IOVA mode but current mode is > > 'VA', not > > >> > > > > > initializing > > >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used > > >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1 > > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > >> > > > > > > EAL: Expecting 'PA' IOVA mode but current mode is > > 'VA', not > > >> > > > > > initializing > > >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used > > >> > > > > > > EAL: Bus (pci) probe failed. > > >> > > > > > > EAL: lib.telemetry log level changed from disabled t= o > > warning > > >> > > > > > > EAL: Error - exiting with code: 1 > > >> > > > > > > Cause: Error: number of ports must be even > > >> > > > > > > EAL: request: mp_malloc_sync > > >> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > For some reason the HW ports won't setup. From what = I > > understand > > >> > > > > > > net_vmxnet3 should work with VA mode. > > >> > > > > > > I enabled I/OMUU for the VM. > > >> > > > > > > The weird thing even when enabled, I still have the > > >> > > > > > > enable_unsafe_noiommu_mode flag on. > > >> > > > > > > And because it's on the this: > > >> > > > > > > > > >> > > > > > > dev_iova_mode =3D pci_device_iova_mode(dr, dev); > > >> > > > > > > > > >> > > > > > > return PA mode, and it fails. > > >> > > > > > > > > >> > > > > > > When I disable it by modifying > > >> > > > > > > > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get > > >> > another > > >> > > > > > error. > > >> > > > > > > The error is that it doesn't find a suitable IOMMU > type: > > >> > > > > > > Just putting the relevant message: > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB > > >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No > > such device) > > >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No > > such device) > > >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (N= o > > such > > >> > device) > > >> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type > > >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used > > >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1 > > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No > > such device) > > >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No > > such device) > > >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (N= o > > such > > >> > device) > > >> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type > > >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > I can see where it fails: > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > const struct vfio_iommu_type * > > >> > > > > > > vfio_set_iommu_type(int vfio_container_fd) > > >> > > > > > > { > > >> > > > > > > unsigned idx; > > >> > > > > > > for (idx =3D 0; idx < RTE_DIM(iommu_types); idx++) { > > >> > > > > > > const struct vfio_iommu_type *t =3D &iommu_types[idx= ]; > > >> > > > > > > > > >> > > > > > > int ret =3D ioctl(vfio_container_fd, VFIO_SET_IOMMU, > > >> > > > > > > t->type_id); > > >> > > > > > > if (!ret) { > > >> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n", > > >> > > > > > > t->type_id, t->name); > > >> > > > > > > return t; > > >> > > > > > > } > > >> > > > > > > /* not an error, there may be more supported IOMMU > > types */ > > >> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, > > error " > > >> > > > > > > "%i (%s)\n", t->type_id, t->name, errno, > > >> > > > > > > strerror(errno)); > > >> > > > > > > } > > >> > > > > > > /* if we didn't find a suitable IOMMU type, fail */ > > >> > > > > > > return NULL; > > >> > > > > > > } > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > The ioctl returns -1. > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > I tried many things. > > >> > > > > > > Enabling/disabling IOMMU. > > >> > > > > > > Modifying the enable_unsafe_noiommu_mode flag. > > >> > > > > > > Running in PA mode. > > >> > > > > > > Any help will be much appreciated. > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > !! External Email: This email originated from outside of the > > > organization. Do not click links or open attachments unless you > > > recognize the sender. > > > > > > > --000000000000b12f850602a1b6d6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi again,
I got this resolved with VMWARE support so I = thought to share it here.
What I originally wanted was to use IOVA-VA on= an AMD host. Which=C2=A0didn't work. I have learned that the ESXI vers= ion that supports
virtual IOMMU in AMD hosts (which is a prerequisi= te=C2=A0to IOVA-VA) is ESXI 7.0 U1. After updating it worked. On Intel host= s ESXI 6.7 supports it already as far as I know.


On Tue, Jul 2= 5, 2023 at 6:19=E2=80=AFPM Varghese, Vipin <Vipin.Varghese@amd.com> wrote:

[AMD Official Use Only - General]


Like I said earlier, trying with the Intel host I ha= ve on VMWARE, specifically=C2=A0 Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
With IOMMU enabled, VMXNET3 works with VA as well as PA.

[VV] since `= enable_unsafe_iommu: not enabled` on Intel platform, could it be possible = the specific version EXSI hypervisor supports the HW IOMMU specific to the platform. My suspicion will be in case on AMD pla= tform =C2=A0changes for required to enable HW iommu might not be available = to specific EXSI (hypervior OS) used.

=C2=A0

I am not an expert on virtio_user PMD, but I can che= ck if it will work with PA with deferred setting for vmx_net3 PMD are ok?

=C2=A0

From: Igor de Paula <igordptx@gmail.com>
Sent: Tuesday, July 25, 2023 8:42 PM
To: Varghese, Vipin <Vipin.Varghese@amd.com>
Cc: Yigit, Ferruh <Ferruh.Yigit@amd.com>; Jochen Behrens <jbehrens@vmware.com>; Tho= mas Monjalon <t= homas@monjalon.net>; users@dpdk.org; Gupta, Nipun <Nipun.Gupta@amd.com>; Agarwal, Nikhil <nikhil.agarwal@am= d.com>; Ronak Doshi <doshir@vmware.com>; Immanni, Venkat <Ven= kat.Immanni@amd.com>; Chenbo Xia <chenbo.xia@intel.com>
Subject: Re: help

=C2=A0

Caution: This message originated from an External Source. Use pr= oper caution when opening attachments, clicking links, or responding.

=C2=A0

Well,
Like I said earlier, trying with the Intel host I have on VMWARE, specifica= lly=C2=A0 Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
With IOMMU enabled, VMXNET3 works with VA as well as PA.
Meaning, PA works regardless if IOMMU is enabled or not. From my experience= anyway.
That's why I thought=C2=A0that:
virtio_user needs VA to work.
For some reason VMXNET3 does not work with VA (only on AMD host).

=C2=A0

=C2=A0

On Tue, Jul 25, 2023 at 4:04=E2=80=AFPM Varghese, Vi= pin <Vipin.V= arghese@amd.com> wrote:

[AMD Official Use Only - General]<= /p>

=C2=A0

Thanks Igor,

=C2=A0

As suspected the vmx_net3 works with

=C2=A0

  1. Iommu: disabled
  2. enable_unsafe_iommu: enabled
  3. dpdk eal iova mode: PA

=C2=A0

as pointed by you in logs, the virtio_user fails as = it expects VA too.

=C2=A0

Will check and get back.

=C2=A0

From: Igor de Paula <igordptx@gmail.com>
Sent: Tuesday, July 25, 2023 8:16 PM
To: Yigit, Ferruh <Ferruh.Yigit@amd.com>
Cc: Jochen Behrens <jbehrens@vmware.com>; Thomas Monjalon <thomas@monjalon.net>; users@dpdk.org; Gup= ta, Nipun <Nipu= n.Gupta@amd.com>; Agarwal, Nikhil <nikhil.agarwal@amd.com>; Ronak Doshi <dos= hir@vmware.com>; Immanni, Venkat <Venkat.Immanni@amd.com>; Varghese, Vipi= n <Vipin.Var= ghese@amd.com>; Chenbo Xia <c= henbo.xia@intel.com>
Subject: Re: help

=C2=A0

Caution: This message originated from an External Source. Use pr= oper caution when opening attachments, clicking links, or responding.

=C2=A0

Hi,
Attaching the logs of EAL when trying to run a configuration with virtio_us= er port when IOMMU is=C2=A0
disabled and enable_unsafe_iommu is enabled. As you can see it forces IOVA = as PA but the viritui_user needs IOVA as VA.
I am also attaching=C2=A0the output of dmesg. I am not sure which kernel lo= gs you wanted... if there is anything else please=C2=A0let me know..
Regarding the ESXI logs, they are HUGE so I will send to you on a separate = email.

=C2=A0

On Fri, Jul 21, 2023 at 1:14=E2=80=AFPM Ferruh Yigit= <ferruh.yigit= @amd.com> wrote:

On 7/21/2023 12:39 PM, = Igor de Paula wrote:
> I am trying to use virtio_user for an interface with the
> kernel:=C2=A0https://doc.dpdk.org/guides/howto= /virtio_user_as_exception_path.html <https:/= /doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html>
> I think this requires IOVA as va.
>

I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo,<= br> he may know better.

Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and '--iova-mode=3Dpa'?


> It does work with Intel host and IOMMU
> enabled. Part of the negotiation when setting it up is getting the IOM= MU
> number so I thought it has to have IOMMU.
>

Yes, issue looks like related to the IOMMU, and it may be either related to HW support, or ESXi iommu driver support, we will check using below
information you provided.

> I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but > again, that didn't work.
> ESXI version - VMware ESXi, 7.0.0, 16324942
> AMD:=C2=A0 AMD EPYC 7452 32-Core Processo
>
> On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60= GHz
> Regarding the logs I will try and attach it soon.
>

Thanks for info, waiting for logs.

>
> On Fri, Jul 21, 2023 at 12:21=E2=80=AFPM Ferruh Yigit <ferruh.yigit@amd.com
> <mailto:f= erruh.yigit@amd.com>> wrote:
>
>=C2=A0 =C2=A0 =C2=A0Hi Igor,
>
>=C2=A0 =C2=A0 =C2=A0VM doesn't have IOMMU, and vmxnet3 requires PA = mode, for this can you
>=C2=A0 =C2=A0 =C2=A0please try with:
>=C2=A0 =C2=A0 =C2=A0- enable 'enable_unsafe_noiommu_mode' flag<= br> >=C2=A0 =C2=A0 =C2=A0- Force PA mode via '--iova-mode=3Dpa' eal = parameter
>
>
>=C2=A0 =C2=A0 =C2=A0Also to be able to figure out AMD IOMMU support lev= el, can you please
>=C2=A0 =C2=A0 =C2=A0provide:
>=C2=A0 =C2=A0 =C2=A0- AMD part number
>=C2=A0 =C2=A0 =C2=A0- ESXi/hypervisor version
>=C2=A0 =C2=A0 =C2=A0- ESXi & VM kernel logs
>
>
>=C2=A0 =C2=A0 =C2=A0Thanks,
>=C2=A0 =C2=A0 =C2=A0Ferruh
>
>=C2=A0 =C2=A0 =C2=A0On 7/20/2023 5:21 PM, Jochen Behrens wrote:
>=C2=A0 =C2=A0 =C2=A0> +Ronak from the ESX team
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> =C2=A0
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> =C2=A0
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> In our usage, we do set amd_iommu=3Doff in the= boot command line from
>=C2=A0 =C2=A0 =C2=A0> grub. (Or intel_iommu=3Doff for Intel processo= rs.)
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> =C2=A0
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Jochen
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> =C2=A0
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> *From: *Thomas Monjalon <thomas@monjalon.net
>=C2=A0 =C2=A0 =C2=A0<mailto:thomas@monjalon.net>>
>=C2=A0 =C2=A0 =C2=A0> *Date: *Thursday, July 20, 2023 at 6:00 AM
>=C2=A0 =C2=A0 =C2=A0> *To: *Igor de Paula <igordptx@gmail.com <mailto:igordptx@gmail.com>= >
>=C2=A0 =C2=A0 =C2=A0> *Cc: *users@dpdk.org <mailto:users@dpdk.org> <users@dpdk.org
>=C2=A0 =C2=A0 =C2=A0<mailto:users@dpdk.org>>, Jochen Behrens
>=C2=A0 =C2=A0 =C2=A0> <jbehrens@vmware.com <mailto:jbehrens@vmware.com>>, Nipun Gupt= a
>=C2=A0 =C2=A0 =C2=A0<nipun.gupta@amd.com <mailto:nipun.gupta@amd.com>>, Nikhil Agarwa= l
>=C2=A0 =C2=A0 =C2=A0> <nikhil.agarwal@amd.com <mailto:nikhil.agarwal@amd.com>>= , Ferruh
>=C2=A0 =C2=A0 =C2=A0Yigit <ferruh.yigit@amd.com <mailto:ferruh.yigit@amd.com>>
>=C2=A0 =C2=A0 =C2=A0> *Subject: *Re: help
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> !! External Email
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> +Cc some AMD maintainers, they can have an ide= a about IOMMU settings.
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> 20/07/2023 14:44, Igor de Paula:
>=C2=A0 =C2=A0 =C2=A0>> I have enabled it in the host and in the B= IOS for AMD...
>=C2=A0 =C2=A0 =C2=A0>> In the Bios I changed to amd_iommu=3Don an= d in the host it's the
>=C2=A0 =C2=A0 =C2=A0same for
>=C2=A0 =C2=A0 =C2=A0>> either.
>=C2=A0 =C2=A0 =C2=A0>>
>=C2=A0 =C2=A0 =C2=A0>> On Thu, Jul 20, 2023 at 1:31=E2=80=AFPM Th= omas Monjalon
>=C2=A0 =C2=A0 =C2=A0<thomas@monjalon.net <mailto:thomas@monjalon.net>> wrote:
>=C2=A0 =C2=A0 =C2=A0>>
>=C2=A0 =C2=A0 =C2=A0>> > 20/07/2023 11:35, Igor de Paula:
>=C2=A0 =C2=A0 =C2=A0>> > > The weird thing is that it only = happens when I am using a
>=C2=A0 =C2=A0 =C2=A0host with an
>=C2=A0 =C2=A0 =C2=A0>> > AMD
>=C2=A0 =C2=A0 =C2=A0>> > > processor. It doesn't happen= when I use a host with an Intel
>=C2=A0 =C2=A0 =C2=A0processor.
>=C2=A0 =C2=A0 =C2=A0>> >
>=C2=A0 =C2=A0 =C2=A0>> > So it's probably a matter of BIOS= settings for the IOMMU?
>=C2=A0 =C2=A0 =C2=A0>> >
>=C2=A0 =C2=A0 =C2=A0>> >
>=C2=A0 =C2=A0 =C2=A0>> > > On Thu, Jul 20, 2023 at 10:32=E2= =80=AFAM Thomas Monjalon
>=C2=A0 =C2=A0 =C2=A0<thomas@monjalon.net <mailto:thomas@monjalon.net>>
>=C2=A0 =C2=A0 =C2=A0>> > > wrote:
>=C2=A0 =C2=A0 =C2=A0>> > >
>=C2=A0 =C2=A0 =C2=A0>> > > > +Cc the vmxnet3 maintainer.=
>=C2=A0 =C2=A0 =C2=A0>> > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > Please Jochen, do you have = an idea what's wrong below?
>=C2=A0 =C2=A0 =C2=A0>> > > >
>=C2=A0 =C2=A0 =C2=A0>> > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > 20/07/2023 11:25, Igor de P= aula:
>=C2=A0 =C2=A0 =C2=A0>> > > > > This is because it can= 't negotiate the IOMMU type with
>=C2=A0 =C2=A0 =C2=A0any port.
>=C2=A0 =C2=A0 =C2=A0>> > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > On Thu, Jul 20, 2023 a= t 5:08=E2=80=AFAM Thomas Monjalon
>=C2=A0 =C2=A0 =C2=A0<thomas@monjalon.net <mailto:thomas@monjalon.net>
>=C2=A0 =C2=A0 =C2=A0>> > >
>=C2=A0 =C2=A0 =C2=A0>> > > > wrote:
>=C2=A0 =C2=A0 =C2=A0>> > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > Hello,
>=C2=A0 =C2=A0 =C2=A0>> > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > The first error i= s "Cause: Error: number of ports must
>=C2=A0 =C2=A0 =C2=A0be even"
>=C2=A0 =C2=A0 =C2=A0>> > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > 03/05/2023 18:13,= Igor de Paula:
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > I am running= a VM inside a VMWARE server (vSphere).
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > My goal it t= o set up DPDK with two HW ports, and set up a
>=C2=A0 =C2=A0 =C2=A0>> > > > virtio_user to
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > interact wit= h the kernel stack.
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > In another a= pp I have it working but instead of
>=C2=A0 =C2=A0 =C2=A0virtio_user I am
>=C2=A0 =C2=A0 =C2=A0>> > > > running
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > KNI, it work= s in IOVA-PA mode.
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > I am looking= to replace the KNI.
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > When I try t= o set up virtio_user port as in the doc:
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > >
>=C2=A0 =C2=A0 =C2=A0>> >
>=C2=A0 =C2=A0 =C2=A0https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#vir= tio-user-as-exception-path <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_pa= th.html#virtio-user-as-exception-path> <https://doc.dpdk= .org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-excepti= on-path <https= ://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user= -as-exception-path>>
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > I get a erro= r it can't run in PA mode.
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > When I try t= o run as VA mode from a parameter, I get the
>=C2=A0 =C2=A0 =C2=A0>> > following
>=C2=A0 =C2=A0 =C2=A0>> > > > > > errors:
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: lib.eal= log level changed from info to debug
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Detecte= d lcore 0 as core 0 on socket 0
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Detecte= d lcore 1 as core 0 on socket 0
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Support= maximum 128 logical core(s) by
>=C2=A0 =C2=A0 =C2=A0configuration.
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Detecte= d 2 lcore(s)
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Detecte= d 1 NUMA nodes
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Checkin= g presence of .so 'librte_eal.so.21.3'
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Checkin= g presence of .so 'librte_eal.so.21'
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Checkin= g presence of .so 'librte_eal.so'
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Detecte= d static linkage of DPDK
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Ask a v= irtual area of 0x7000 bytes
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Virtual= area found at 0x100000000 (size =3D 0x7000)
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Multi-p= rocess socket /var/run/dpdk/rte/mp_socket
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: DPAA Bu= s not present. Skipping.
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: VFIO PC= I modules not loaded
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Selecte= d IOVA mode 'VA'
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Probing= VFIO support...
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: IOMMU t= ype 1 (Type 1) is supported
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: IOMMU t= ype 7 (sPAPR) is not supported
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: IOMMU t= ype 8 (No-IOMMU) is supported
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: VFIO su= pport initialized
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Ask a v= irtual area of 0x5b000 bytes
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Virtual= area found at 0x100007000 (size =3D 0x5b000)
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Setting= up physically contiguous memory...
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Setting= maximum number of open files to 1048576
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Detecte= d memory type: socket_id:0
>=C2=A0 =C2=A0 =C2=A0hugepage_sz:1073741824
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Creatin= g 2 segment lists: n_segs:128 socket_id:0
>=C2=A0 =C2=A0 =C2=A0>> > > > > > hugepage_sz:10737= 41824
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Ask a v= irtual area of 0x2000 bytes
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Virtual= area found at 0x100062000 (size =3D 0x2000)
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Memseg = list allocated at socket 0, page size
>=C2=A0 =C2=A0 =C2=A00x100000kB
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Ask a v= irtual area of 0x2000000000 bytes
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Virtual= area found at 0x140000000 (size =3D
>=C2=A0 =C2=A0 =C2=A00x2000000000)
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: VA rese= rved for memseg list at 0x140000000, size
>=C2=A0 =C2=A0 =C2=A02000000000
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Ask a v= irtual area of 0x2000 bytes
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Virtual= area found at 0x2140000000 (size =3D 0x2000)
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Memseg = list allocated at socket 0, page size
>=C2=A0 =C2=A0 =C2=A00x100000kB
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Ask a v= irtual area of 0x2000000000 bytes
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Virtual= area found at 0x2180000000 (size =3D
>=C2=A0 =C2=A0 =C2=A00x2000000000)
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: VA rese= rved for memseg list at 0x2180000000,
>=C2=A0 =C2=A0 =C2=A0size 2000000000
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: TSC fre= quency is ~2350000 KHz
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Main lc= ore 0 is ready (tid=3D7f8ad790ec00;cpuset=3D[0])
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: lcore 1= is ready (tid=3D7f8ad6907400;cpuset=3D[1])
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Trying = to obtain current memory policy.
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Setting= policy MPOL_PREFERRED for socket 0
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Restori= ng previous memory policy: 0
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: request= : mp_malloc_sync
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Heap on= socket 0 was expanded by 1024MB
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: PCI dev= ice 0000:0b:00.0 on NUMA socket -1
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL:=C2=A0= =C2=A0 probe driver: 15ad:7b0 net_vmxnet3
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL:=C2=A0= =C2=A0 Expecting 'PA' IOVA mode but current mode is
>=C2=A0 =C2=A0 =C2=A0'VA', not
>=C2=A0 =C2=A0 =C2=A0>> > > > > > initializing
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Request= ed device 0000:0b:00.0 cannot be used
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: PCI dev= ice 0000:13:00.0 on NUMA socket -1
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL:=C2=A0= =C2=A0 probe driver: 15ad:7b0 net_vmxnet3
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL:=C2=A0= =C2=A0 Expecting 'PA' IOVA mode but current mode is
>=C2=A0 =C2=A0 =C2=A0'VA', not
>=C2=A0 =C2=A0 =C2=A0>> > > > > > initializing
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Request= ed device 0000:13:00.0 cannot be used
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Bus (pc= i) probe failed.
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: lib.tel= emetry log level changed from disabled to
>=C2=A0 =C2=A0 =C2=A0warning
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Error -= exiting with code: 1
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >=C2=A0=C2=A0 = Cause: Error: number of ports must be even
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: request= : mp_malloc_sync
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Heap on= socket 0 was shrunk by 1024MB
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > For some rea= son the HW ports won't setup. From what I
>=C2=A0 =C2=A0 =C2=A0understand
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > net_vmxnet3 = should work with VA mode.
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > I enabled I/= OMUU for the VM.
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > The weird th= ing even when enabled, I still have the
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > enable_unsaf= e_noiommu_mode flag on.
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > And because = it's on the this:
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > dev_iova_mod= e =3D pci_device_iova_mode(dr, dev);
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > return PA mo= de, and it fails.
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > When I disab= le it by modifying
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0/sys/module/vfio/parameters/enable_unsafe_noiommu_m= ode, I get
>=C2=A0 =C2=A0 =C2=A0>> > another
>=C2=A0 =C2=A0 =C2=A0>> > > > > > error.
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > The error is= that it doesn't find a suitable IOMMU type:
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > Just putting= the relevant message:
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Heap on= socket 0 was expanded by 1024MB
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: PCI dev= ice 0000:0b:00.0 on NUMA socket -1
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL:=C2=A0= =C2=A0 probe driver: 15ad:7b0 net_vmxnet3
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Set IOM= MU type 1 (Type 1) failed, error 19 (No
>=C2=A0 =C2=A0 =C2=A0such device)
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Set IOM= MU type 7 (sPAPR) failed, error 19 (No
>=C2=A0 =C2=A0 =C2=A0such device)
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Set IOM= MU type 8 (No-IOMMU) failed, error 19 (No
>=C2=A0 =C2=A0 =C2=A0such
>=C2=A0 =C2=A0 =C2=A0>> > device)
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: 0000:0b= :00.0 failed to select IOMMU type
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Request= ed device 0000:0b:00.0 cannot be used
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: PCI dev= ice 0000:13:00.0 on NUMA socket -1
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL:=C2=A0= =C2=A0 probe driver: 15ad:7b0 net_vmxnet3
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Set IOM= MU type 1 (Type 1) failed, error 19 (No
>=C2=A0 =C2=A0 =C2=A0such device)
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Set IOM= MU type 7 (sPAPR) failed, error 19 (No
>=C2=A0 =C2=A0 =C2=A0such device)
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Set IOM= MU type 8 (No-IOMMU) failed, error 19 (No
>=C2=A0 =C2=A0 =C2=A0such
>=C2=A0 =C2=A0 =C2=A0>> > device)
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: 0000:13= :00.0 failed to select IOMMU type
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > EAL: Request= ed device 0000:13:00.0 cannot be used
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > I can see wh= ere it fails:
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > const struct= vfio_iommu_type *
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > vfio_set_iom= mu_type(int vfio_container_fd)
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > {
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > unsigned idx= ;
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > for (idx =3D= 0; idx < RTE_DIM(iommu_types); idx++) {
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > const struct= vfio_iommu_type *t =3D &iommu_types[idx];
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > int ret =3D = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > t->type_i= d);
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > if (!ret) {<= br> >=C2=A0 =C2=A0 =C2=A0>> > > > > > > RTE_LOG(INFO= , EAL, "Using IOMMU type %d (%s)\n",
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > t->type_i= d, t->name);
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > return t; >=C2=A0 =C2=A0 =C2=A0>> > > > > > > }
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > /* not an er= ror, there may be more supported IOMMU
>=C2=A0 =C2=A0 =C2=A0types */
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > RTE_LOG(DEBU= G, EAL, "Set IOMMU type %d (%s) failed,
>=C2=A0 =C2=A0 =C2=A0error "
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > "%i (%s= )\n", t->type_id, t->name, errno,
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > strerror(err= no));
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > }
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > /* if we did= n't find a suitable IOMMU type, fail */
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > return NULL;=
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > }
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > The ioctl re= turns -1.
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > >
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > I tried many= things.
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > Enabling/dis= abling IOMMU.
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > Modifying th= e enable_unsafe_noiommu_mode flag.
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > Running in P= A mode.
>=C2=A0 =C2=A0 =C2=A0>> > > > > > > Any help wil= l be much appreciated.
>=C2=A0 =C2=A0 =C2=A0>> >
>=C2=A0 =C2=A0 =C2=A0>> >
>=C2=A0 =C2=A0 =C2=A0>> >
>=C2=A0 =C2=A0 =C2=A0>> >
>=C2=A0 =C2=A0 =C2=A0>> >
>=C2=A0 =C2=A0 =C2=A0>> >
>=C2=A0 =C2=A0 =C2=A0>>
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> !! External Email: This email originated from = outside of the
>=C2=A0 =C2=A0 =C2=A0> organization. Do not click links or open attac= hments unless you
>=C2=A0 =C2=A0 =C2=A0> recognize the sender.
>=C2=A0 =C2=A0 =C2=A0>
>

--000000000000b12f850602a1b6d6--