From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8A10943033 for ; Fri, 11 Aug 2023 14:37:36 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CD7EA42D55; Fri, 11 Aug 2023 14:37:35 +0200 (CEST) Received: from mail-yb1-f171.google.com (mail-yb1-f171.google.com [209.85.219.171]) by mails.dpdk.org (Postfix) with ESMTP id 36E5F40E03 for ; Fri, 11 Aug 2023 14:37:34 +0200 (CEST) Received: by mail-yb1-f171.google.com with SMTP id 3f1490d57ef6-d63c0a6568fso1581201276.0 for ; Fri, 11 Aug 2023 05:37:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691757453; x=1692362253; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=CP++u263TJAGU5YjriEn8bfDQmP/PYdxHhv6DDD4qbA=; b=aZpHPeBtlFOdK7mUTu1NWtWgvAzxufXixlIa0ZoWAumw5l8QBpLhf+EYxGCE0fJx6m uTFNWi5iCo/gAJ3IAnLZjN8Y9UzyDzoFOUKZrixad1L0AVIb04z+jHFCI7IeCBz1o8QJ jQ5Rsqpra/+A/DQQgzsraAt9m2ETF7lAazWK5cxIZEqEcstMg1iEca5SY0hvRjHO7nZW 0ywCsWm5oGgbXR0EvhenKe3BKijnFa+KJWFy5LYJjmBq5F3IWiDCFXRBCTq0KNB3SJRG xwv+zYEytLMigoQAaSSFOX0RbFvQOrjhJPUoSjDgwk3yn1Pkca32lgRofthEoSJpzD19 Ntig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691757453; x=1692362253; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=CP++u263TJAGU5YjriEn8bfDQmP/PYdxHhv6DDD4qbA=; b=dmBzU/Xd6PibITrbdjaRZeHAkCBOmbDxtyeiSbHDjPP7bf223XW4tlayT5vCuDNmN6 NBRnoIEC4iDCB/2N+6r2iAIP75u82GTK0MfDLRWu7PDX0MlnUSbwdJDhIX6f/Wa6Iqov MKOMAtNXzBq17CRWbm6GXSO3KL5yvHaCr597bx0qL/qIqWgwNxUYQZ4hHJjHAFJg3Nx3 ZcLaQ1rrOw4tUY1O0Kp7QcjleNP2l8K/UXFn4OdOW94vhEdd0wNq5XxY7cZCwc+NJAh4 +ZE34Fq4G75m5EftgtqgXIGPbplkp5jqCe2k6lNFA8Mfh55B/oU/FtdfPUtdt+55lJGC wqxA== X-Gm-Message-State: AOJu0Yx4TCX0sTogNGxbZQcjvR26KdiHTeqZ6yaP2XAoDmyAnvTFqh9y UO2S6AL5gRIvVUVY+kwyArG9rhzRW2QkPB1BJhM= X-Google-Smtp-Source: AGHT+IHkDQz6pEMZKGrFEuh3m3pTf7iq6PA68X6INTknU9ETrv463VEAjosYU+y+U6xCerj+HkWuqtH6A4KAQsASla8= X-Received: by 2002:a25:ce12:0:b0:c6f:b843:c502 with SMTP id x18-20020a25ce12000000b00c6fb843c502mr1528750ybe.25.1691757453324; Fri, 11 Aug 2023 05:37:33 -0700 (PDT) MIME-Version: 1.0 References: <2290932.ElGaqSPkdT@thomas> In-Reply-To: <2290932.ElGaqSPkdT@thomas> From: Igor de Paula Date: Fri, 11 Aug 2023 15:37:22 +0300 Message-ID: Subject: Re: help To: Thomas Monjalon Cc: "Varghese, Vipin" , "Yigit, Ferruh" , Jochen Behrens , "users@dpdk.org" , "Gupta, Nipun" , "Agarwal, Nikhil" , Ronak Doshi , "Immanni, Venkat" , Chenbo Xia Content-Type: multipart/alternative; boundary="0000000000000d36d10602a4fa9c" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --0000000000000d36d10602a4fa9c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sure, I will submit a patch (it would be a first). Thanks On Fri, Aug 11, 2023 at 3:30=E2=80=AFPM Thomas Monjalon wrote: > Thanks for the info. > Do you think it should be written in the vmxnet3 page of the DPDK > documentation? > If yes, would you like to initiate a patch for review? > > > 11/08/2023 10:43, Igor de Paula: > > Hi again, > > I got this resolved with VMWARE support so I thought to share it here. > > What I originally wanted was to use IOVA-VA on an AMD host. Which didn'= t > > work. I have learned that the ESXI version that supports > > virtual IOMMU in AMD hosts (which is a prerequisite to IOVA-VA) is ESXI > 7.0 > > U1. After updating it worked. On Intel hosts ESXI 6.7 supports it alrea= dy > > as far as I know. > > > > > > On Tue, Jul 25, 2023 at 6:19=E2=80=AFPM Varghese, Vipin > > wrote: > > > > > [AMD Official Use Only - General] > > > > > > Like I said earlier, trying with the Intel host I have on VMWARE, > > > specifically Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz > > > With IOMMU enabled, VMXNET3 works with VA as well as PA. > > > > > > [VV] since ` enable_unsafe_iommu: not enabled` on Intel platform, cou= ld > > > it be possible the specific version EXSI hypervisor supports the HW > IOMMU > > > specific to the platform. My suspicion will be in case on AMD platfor= m > > > changes for required to enable HW iommu might not be available to > specific > > > EXSI (hypervior OS) used. > > > > > > > > > > > > I am not an expert on virtio_user PMD, but I can check if it will wor= k > > > with PA with deferred setting for vmx_net3 PMD are ok? > > > > > > > > > > > > *From:* Igor de Paula > > > *Sent:* Tuesday, July 25, 2023 8:42 PM > > > *To:* Varghese, Vipin > > > *Cc:* Yigit, Ferruh ; Jochen Behrens < > > > jbehrens@vmware.com>; Thomas Monjalon ; > > > users@dpdk.org; Gupta, Nipun ; Agarwal, Nikhil < > > > nikhil.agarwal@amd.com>; Ronak Doshi ; Immanni, > Venkat > > > ; Chenbo Xia > > > *Subject:* Re: help > > > > > > > > > > > > *Caution:* This message originated from an External Source. Use prope= r > > > caution when opening attachments, clicking links, or responding. > > > > > > > > > > > > Well, > > > Like I said earlier, trying with the Intel host I have on VMWARE, > > > specifically Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz > > > With IOMMU enabled, VMXNET3 works with VA as well as PA. > > > Meaning, PA works regardless if IOMMU is enabled or not. From my > > > experience anyway. > > > That's why I thought that: > > > virtio_user needs VA to work. > > > For some reason VMXNET3 does not work with VA (only on AMD host). > > > > > > > > > > > > > > > > > > On Tue, Jul 25, 2023 at 4:04=E2=80=AFPM Varghese, Vipin < > Vipin.Varghese@amd.com> > > > wrote: > > > > > > [AMD Official Use Only - General] > > > > > > > > > > > > Thanks Igor, > > > > > > > > > > > > As suspected the vmx_net3 works with > > > > > > > > > > > > 1. Iommu: disabled > > > 2. enable_unsafe_iommu: enabled > > > 3. dpdk eal iova mode: PA > > > > > > > > > > > > as pointed by you in logs, the virtio_user fails as it expects VA too= . > > > > > > > > > > > > Will check and get back. > > > > > > > > > > > > *From:* Igor de Paula > > > *Sent:* Tuesday, July 25, 2023 8:16 PM > > > *To:* Yigit, Ferruh > > > *Cc:* Jochen Behrens ; Thomas Monjalon < > > > thomas@monjalon.net>; users@dpdk.org; Gupta, Nipun < > Nipun.Gupta@amd.com>; > > > Agarwal, Nikhil ; Ronak Doshi < > doshir@vmware.com>; > > > Immanni, Venkat ; Varghese, Vipin < > > > Vipin.Varghese@amd.com>; Chenbo Xia > > > *Subject:* Re: help > > > > > > > > > > > > *Caution:* This message originated from an External Source. Use prope= r > > > caution when opening attachments, clicking links, or responding. > > > > > > > > > > > > Hi, > > > Attaching the logs of EAL when trying to run a configuration with > > > virtio_user port when IOMMU is > > > disabled and enable_unsafe_iommu is enabled. As you can see it forces > IOVA > > > as PA but the viritui_user needs IOVA as VA. > > > I am also attaching the output of dmesg. I am not sure which kernel > logs > > > you wanted... if there is anything else please let me know.. > > > Regarding the ESXI logs, they are HUGE so I will send to you on a > separate > > > email. > > > > > > > > > > > > On Fri, Jul 21, 2023 at 1:14=E2=80=AFPM Ferruh Yigit > wrote: > > > > > > On 7/21/2023 12:39 PM, Igor de Paula wrote: > > > > I am trying to use virtio_user for an interface with the > > > > kernel: > > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html = < > > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html> > > > > I think this requires IOVA as va. > > > > > > > > > > I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo= , > > > he may know better. > > > > > > Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and > > > '--iova-mode=3Dpa'? > > > > > > > > > > It does work with Intel host and IOMMU > > > > enabled. Part of the negotiation when setting it up is getting the > IOMMU > > > > number so I thought it has to have IOMMU. > > > > > > > > > > Yes, issue looks like related to the IOMMU, and it may be either > related > > > to HW support, or ESXi iommu driver support, we will check using belo= w > > > information you provided. > > > > > > > I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but > > > > again, that didn't work. > > > > ESXI version - VMware ESXi, 7.0.0, 16324942 > > > > AMD: AMD EPYC 7452 32-Core Processo > > > > > > > > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ > 2.60GHz > > > > Regarding the logs I will try and attach it soon. > > > > > > > > > > Thanks for info, waiting for logs. > > > > > > > > > > > On Fri, Jul 21, 2023 at 12:21=E2=80=AFPM Ferruh Yigit > > > > wrote: > > > > > > > > Hi Igor, > > > > > > > > VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this > can you > > > > please try with: > > > > - enable 'enable_unsafe_noiommu_mode' flag > > > > - Force PA mode via '--iova-mode=3Dpa' eal parameter > > > > > > > > > > > > Also to be able to figure out AMD IOMMU support level, can you > please > > > > provide: > > > > - AMD part number > > > > - ESXi/hypervisor version > > > > - ESXi & VM kernel logs > > > > > > > > > > > > Thanks, > > > > Ferruh > > > > > > > > On 7/20/2023 5:21 PM, Jochen Behrens wrote: > > > > > +Ronak from the ESX team > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In our usage, we do set amd_iommu=3Doff in the boot command l= ine > from > > > > > grub. (Or intel_iommu=3Doff for Intel processors.) > > > > > > > > > > > > > > > > > > > > Jochen > > > > > > > > > > > > > > > > > > > > *From: *Thomas Monjalon > > > > > > > > > *Date: *Thursday, July 20, 2023 at 6:00 AM > > > > > *To: *Igor de Paula igordptx@gmail.com > > > >> > > > > > *Cc: *users@dpdk.org > > > >, Jochen Behrens > > > > > >, Nipun > Gupta > > > > >, Nikhil > Agarwal > > > > > >, > Ferruh > > > > Yigit > > > > > > *Subject: *Re: help > > > > > > > > > > !! External Email > > > > > > > > > > +Cc some AMD maintainers, they can have an idea about IOMMU > > > settings. > > > > > > > > > > > > > > > 20/07/2023 14:44, Igor de Paula: > > > > >> I have enabled it in the host and in the BIOS for AMD... > > > > >> In the Bios I changed to amd_iommu=3Don and in the host it's= the > > > > same for > > > > >> either. > > > > >> > > > > >> On Thu, Jul 20, 2023 at 1:31=E2=80=AFPM Thomas Monjalon > > > > > wrote: > > > > >> > > > > >> > 20/07/2023 11:35, Igor de Paula: > > > > >> > > The weird thing is that it only happens when I am using = a > > > > host with an > > > > >> > AMD > > > > >> > > processor. It doesn't happen when I use a host with an > Intel > > > > processor. > > > > >> > > > > > >> > So it's probably a matter of BIOS settings for the IOMMU? > > > > >> > > > > > >> > > > > > >> > > On Thu, Jul 20, 2023 at 10:32=E2=80=AFAM Thomas Monjalon > > > > > > > > > >> > > wrote: > > > > >> > > > > > > >> > > > +Cc the vmxnet3 maintainer. > > > > >> > > > > > > > >> > > > Please Jochen, do you have an idea what's wrong below? > > > > >> > > > > > > > >> > > > > > > > >> > > > 20/07/2023 11:25, Igor de Paula: > > > > >> > > > > This is because it can't negotiate the IOMMU type wi= th > > > > any port. > > > > >> > > > > > > > > >> > > > > On Thu, Jul 20, 2023 at 5:08=E2=80=AFAM Thomas Monja= lon > > > > > > > > >> > > > > > > >> > > > wrote: > > > > >> > > > > > > > > >> > > > > > Hello, > > > > >> > > > > > > > > > >> > > > > > The first error is "Cause: Error: number of ports > must > > > > be even" > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > 03/05/2023 18:13, Igor de Paula: > > > > >> > > > > > > I am running a VM inside a VMWARE server > (vSphere). > > > > >> > > > > > > My goal it to set up DPDK with two HW ports, and > set > > > up a > > > > >> > > > virtio_user to > > > > >> > > > > > > interact with the kernel stack. > > > > >> > > > > > > In another app I have it working but instead of > > > > virtio_user I am > > > > >> > > > running > > > > >> > > > > > > KNI, it works in IOVA-PA mode. > > > > >> > > > > > > I am looking to replace the KNI. > > > > >> > > > > > > > > > > >> > > > > > > When I try to set up virtio_user port as in the > doc: > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virt= io-user-as-exception-path > > > < > > > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virt= io-user-as-exception-path > > > > > < > > > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virt= io-user-as-exception-path > > > < > > > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virt= io-user-as-exception-path > > > >> > > > > >> > > > > > > I get a error it can't run in PA mode. > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > When I try to run as VA mode from a parameter, I > get > > > the > > > > >> > following > > > > >> > > > > > errors: > > > > >> > > > > > > EAL: lib.eal log level changed from info to debu= g > > > > >> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0 > > > > >> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0 > > > > >> > > > > > > EAL: Support maximum 128 logical core(s) by > > > > configuration. > > > > >> > > > > > > EAL: Detected 2 lcore(s) > > > > >> > > > > > > EAL: Detected 1 NUMA nodes > > > > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.= 3' > > > > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21' > > > > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so' > > > > >> > > > > > > EAL: Detected static linkage of DPDK > > > > >> > > > > > > EAL: Ask a virtual area of 0x7000 bytes > > > > >> > > > > > > EAL: Virtual area found at 0x100000000 (size =3D > 0x7000) > > > > >> > > > > > > EAL: Multi-process socket > /var/run/dpdk/rte/mp_socket > > > > >> > > > > > > EAL: DPAA Bus not present. Skipping. > > > > >> > > > > > > EAL: VFIO PCI modules not loaded > > > > >> > > > > > > EAL: Selected IOVA mode 'VA' > > > > >> > > > > > > EAL: Probing VFIO support... > > > > >> > > > > > > EAL: IOMMU type 1 (Type 1) is supported > > > > >> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported > > > > >> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported > > > > >> > > > > > > EAL: VFIO support initialized > > > > >> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes > > > > >> > > > > > > EAL: Virtual area found at 0x100007000 (size =3D > > > 0x5b000) > > > > >> > > > > > > EAL: Setting up physically contiguous memory... > > > > >> > > > > > > EAL: Setting maximum number of open files to > 1048576 > > > > >> > > > > > > EAL: Detected memory type: socket_id:0 > > > > hugepage_sz:1073741824 > > > > >> > > > > > > EAL: Creating 2 segment lists: n_segs:128 > socket_id:0 > > > > >> > > > > > hugepage_sz:1073741824 > > > > >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes > > > > >> > > > > > > EAL: Virtual area found at 0x100062000 (size =3D > 0x2000) > > > > >> > > > > > > EAL: Memseg list allocated at socket 0, page siz= e > > > > 0x100000kB > > > > >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes > > > > >> > > > > > > EAL: Virtual area found at 0x140000000 (size =3D > > > > 0x2000000000) > > > > >> > > > > > > EAL: VA reserved for memseg list at 0x140000000, > size > > > > 2000000000 > > > > >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes > > > > >> > > > > > > EAL: Virtual area found at 0x2140000000 (size = =3D > > > 0x2000) > > > > >> > > > > > > EAL: Memseg list allocated at socket 0, page siz= e > > > > 0x100000kB > > > > >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes > > > > >> > > > > > > EAL: Virtual area found at 0x2180000000 (size = =3D > > > > 0x2000000000) > > > > >> > > > > > > EAL: VA reserved for memseg list at 0x2180000000= , > > > > size 2000000000 > > > > >> > > > > > > EAL: TSC frequency is ~2350000 KHz > > > > >> > > > > > > EAL: Main lcore 0 is ready > > > (tid=3D7f8ad790ec00;cpuset=3D[0]) > > > > >> > > > > > > EAL: lcore 1 is ready > (tid=3D7f8ad6907400;cpuset=3D[1]) > > > > >> > > > > > > EAL: Trying to obtain current memory policy. > > > > >> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0 > > > > >> > > > > > > EAL: Restoring previous memory policy: 0 > > > > >> > > > > > > EAL: request: mp_malloc_sync > > > > >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB > > > > >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > > > > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > >> > > > > > > EAL: Expecting 'PA' IOVA mode but current mode > is > > > > 'VA', not > > > > >> > > > > > initializing > > > > >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be use= d > > > > >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1 > > > > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > >> > > > > > > EAL: Expecting 'PA' IOVA mode but current mode > is > > > > 'VA', not > > > > >> > > > > > initializing > > > > >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be use= d > > > > >> > > > > > > EAL: Bus (pci) probe failed. > > > > >> > > > > > > EAL: lib.telemetry log level changed from > disabled to > > > > warning > > > > >> > > > > > > EAL: Error - exiting with code: 1 > > > > >> > > > > > > Cause: Error: number of ports must be even > > > > >> > > > > > > EAL: request: mp_malloc_sync > > > > >> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > For some reason the HW ports won't setup. From > what I > > > > understand > > > > >> > > > > > > net_vmxnet3 should work with VA mode. > > > > >> > > > > > > I enabled I/OMUU for the VM. > > > > >> > > > > > > The weird thing even when enabled, I still have > the > > > > >> > > > > > > enable_unsafe_noiommu_mode flag on. > > > > >> > > > > > > And because it's on the this: > > > > >> > > > > > > > > > > >> > > > > > > dev_iova_mode =3D pci_device_iova_mode(dr, dev); > > > > >> > > > > > > > > > > >> > > > > > > return PA mode, and it fails. > > > > >> > > > > > > > > > > >> > > > > > > When I disable it by modifying > > > > >> > > > > > > > > > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get > > > > >> > another > > > > >> > > > > > error. > > > > >> > > > > > > The error is that it doesn't find a suitable IOM= MU > > > type: > > > > >> > > > > > > Just putting the relevant message: > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB > > > > >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > > > > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 > (No > > > > such device) > > > > >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (= No > > > > such device) > > > > >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 1= 9 > (No > > > > such > > > > >> > device) > > > > >> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type > > > > >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be use= d > > > > >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1 > > > > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 > (No > > > > such device) > > > > >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (= No > > > > such device) > > > > >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 1= 9 > (No > > > > such > > > > >> > device) > > > > >> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type > > > > >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be use= d > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > I can see where it fails: > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > const struct vfio_iommu_type * > > > > >> > > > > > > vfio_set_iommu_type(int vfio_container_fd) > > > > >> > > > > > > { > > > > >> > > > > > > unsigned idx; > > > > >> > > > > > > for (idx =3D 0; idx < RTE_DIM(iommu_types); idx+= +) { > > > > >> > > > > > > const struct vfio_iommu_type *t =3D > &iommu_types[idx]; > > > > >> > > > > > > > > > > >> > > > > > > int ret =3D ioctl(vfio_container_fd, VFIO_SET_IO= MMU, > > > > >> > > > > > > t->type_id); > > > > >> > > > > > > if (!ret) { > > > > >> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n", > > > > >> > > > > > > t->type_id, t->name); > > > > >> > > > > > > return t; > > > > >> > > > > > > } > > > > >> > > > > > > /* not an error, there may be more supported IOM= MU > > > > types */ > > > > >> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) > failed, > > > > error " > > > > >> > > > > > > "%i (%s)\n", t->type_id, t->name, errno, > > > > >> > > > > > > strerror(errno)); > > > > >> > > > > > > } > > > > >> > > > > > > /* if we didn't find a suitable IOMMU type, fail > */ > > > > >> > > > > > > return NULL; > > > > >> > > > > > > } > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > The ioctl returns -1. > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > I tried many things. > > > > >> > > > > > > Enabling/disabling IOMMU. > > > > >> > > > > > > Modifying the enable_unsafe_noiommu_mode flag. > > > > >> > > > > > > Running in PA mode. > > > > >> > > > > > > Any help will be much appreciated. > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > !! External Email: This email originated from outside of the > > > > > organization. Do not click links or open attachments unless y= ou > > > > > recognize the sender. > > > > > > > > > > > > > > > > > > > > > > > --0000000000000d36d10602a4fa9c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Sure, I will submit a patch (it would be a first).=C2=A0Thanks

On Fri, Aug 11, 2023 at 3:30=E2=80=AFPM Thomas Monjalon <thomas@monjalon.net> wrote:
Thanks for the info.
Do you think it should be written in the vmxnet3 page of the DPDK documenta= tion?
If yes, would you like to initiate a patch for review?


11/08/2023 10:43, Igor de Paula:
> Hi again,
> I got this resolved with VMWARE support so I thought to share it here.=
> What I originally wanted was to use IOVA-VA on an AMD host. Which didn= 't
> work. I have learned that the ESXI version that supports
> virtual IOMMU in AMD hosts (which is a prerequisite to IOVA-VA) is ESX= I 7.0
> U1. After updating it worked. On Intel hosts ESXI 6.7 supports it alre= ady
> as far as I know.
>
>
> On Tue, Jul 25, 2023 at 6:19=E2=80=AFPM Varghese, Vipin <Vipin.Varghese@amd.com>
> wrote:
>
> > [AMD Official Use Only - General]
> >
> > Like I said earlier, trying with the Intel host I have on VMWARE,=
> > specifically=C2=A0 Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> > With IOMMU enabled, VMXNET3 works with VA as well as PA.
> >
> > [VV] since ` enable_unsafe_iommu: not enabled` on Intel platform,= could
> > it be possible the specific version EXSI hypervisor supports the = HW IOMMU
> > specific to the platform. My suspicion will be in case on AMD pla= tform
> >=C2=A0 changes for required to enable HW iommu might not be availa= ble to specific
> > EXSI (hypervior OS) used.
> >
> >
> >
> > I am not an expert on virtio_user PMD, but I can check if it will= work
> > with PA with deferred setting for vmx_net3 PMD are ok?
> >
> >
> >
> > *From:* Igor de Paula <igordptx@gmail.com>
> > *Sent:* Tuesday, July 25, 2023 8:42 PM
> > *To:* Varghese, Vipin <Vipin.Varghese@amd.com>
> > *Cc:* Yigit, Ferruh <Ferruh.Yigit@amd.com>; Jochen Behrens <
> > jbehrens= @vmware.com>; Thomas Monjalon <thomas@monjalon.net>;
> > users@dpdk.or= g; Gupta, Nipun <Nipun.Gupta@amd.com>; Agarwal, Nikhil <
> > nikhi= l.agarwal@amd.com>; Ronak Doshi <doshir@vmware.com>; Immanni, Venkat
> > <V= enkat.Immanni@amd.com>; Chenbo Xia <chenbo.xia@intel.com>
> > *Subject:* Re: help
> >
> >
> >
> > *Caution:* This message originated from an External Source. Use p= roper
> > caution when opening attachments, clicking links, or responding.<= br> > >
> >
> >
> > Well,
> > Like I said earlier, trying with the Intel host I have on VMWARE,=
> > specifically=C2=A0 Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
> > With IOMMU enabled, VMXNET3 works with VA as well as PA.
> > Meaning, PA works regardless if IOMMU is enabled or not. From my<= br> > > experience anyway.
> > That's why I thought that:
> > virtio_user needs VA to work.
> > For some reason VMXNET3 does not work with VA (only on AMD host).=
> >
> >
> >
> >
> >
> > On Tue, Jul 25, 2023 at 4:04=E2=80=AFPM Varghese, Vipin <Vipin.Varghese@amd.c= om>
> > wrote:
> >
> > [AMD Official Use Only - General]
> >
> >
> >
> > Thanks Igor,
> >
> >
> >
> > As suspected the vmx_net3 works with
> >
> >
> >
> >=C2=A0 =C2=A0 1. Iommu: disabled
> >=C2=A0 =C2=A0 2. enable_unsafe_iommu: enabled
> >=C2=A0 =C2=A0 3. dpdk eal iova mode: PA
> >
> >
> >
> > as pointed by you in logs, the virtio_user fails as it expects VA= too.
> >
> >
> >
> > Will check and get back.
> >
> >
> >
> > *From:* Igor de Paula <igordptx@gmail.com>
> > *Sent:* Tuesday, July 25, 2023 8:16 PM
> > *To:* Yigit, Ferruh <Ferruh.Yigit@amd.com>
> > *Cc:* Jochen Behrens <jbehrens@vmware.com>; Thomas Monjalon <
> > thomas@m= onjalon.net>; us= ers@dpdk.org; Gupta, Nipun <Nipun.Gupta@amd.com>;
> > Agarwal, Nikhil <nikhil.agarwal@amd.com>; Ronak Doshi <doshir@vmware.com>;
> > Immanni, Venkat <Venkat.Immanni@amd.com>; Varghese, Vipin <
> > Vipin= .Varghese@amd.com>; Chenbo Xia <chenbo.xia@intel.com>
> > *Subject:* Re: help
> >
> >
> >
> > *Caution:* This message originated from an External Source. Use p= roper
> > caution when opening attachments, clicking links, or responding.<= br> > >
> >
> >
> > Hi,
> > Attaching the logs of EAL when trying to run a configuration with=
> > virtio_user port when IOMMU is
> > disabled and enable_unsafe_iommu is enabled. As you can see it fo= rces IOVA
> > as PA but the viritui_user needs IOVA as VA.
> > I am also attaching the output of dmesg. I am not sure which kern= el logs
> > you wanted... if there is anything else please let me know..
> > Regarding the ESXI logs, they are HUGE so I will send to you on a= separate
> > email.
> >
> >
> >
> > On Fri, Jul 21, 2023 at 1:14=E2=80=AFPM Ferruh Yigit <ferruh.yigit@amd.com= > wrote:
> >
> > On 7/21/2023 12:39 PM, Igor de Paula wrote:
> > > I am trying to use virtio_user for an interface with the
> > > kernel:
> > https://doc.dpdk.org/g= uides/howto/virtio_user_as_exception_path.html <
> > https://doc.dpdk.org/g= uides/howto/virtio_user_as_exception_path.html>
> > > I think this requires IOVA as va.
> > >
> >
> > I am not sure if virtio-user has IOVA as VA requirement, cc'e= d Chenbo,
> > he may know better.
> >
> > Meanwhile can you give a try to 'enable_unsafe_noiommu_mode&#= 39; and
> > '--iova-mode=3Dpa'?
> >
> >
> > > It does work with Intel host and IOMMU
> > > enabled. Part of the negotiation when setting it up is getti= ng the IOMMU
> > > number so I thought it has to have IOMMU.
> > >
> >
> > Yes, issue looks like related to the IOMMU, and it may be either = related
> > to HW support, or ESXi iommu driver support, we will check using = below
> > information you provided.
> >
> > > I tried disabling IOMMU and enabling enable_unsafe_noiommu f= lag but
> > > again, that didn't work.
> > > ESXI version - VMware ESXi, 7.0.0, 16324942
> > > AMD:=C2=A0 AMD EPYC 7452 32-Core Processo
> > >
> > > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A= v4 @ 2.60GHz
> > > Regarding the logs I will try and attach it soon.
> > >
> >
> > Thanks for info, waiting for logs.
> >
> > >
> > > On Fri, Jul 21, 2023 at 12:21=E2=80=AFPM Ferruh Yigit <ferruh.yigit@amd.co= m
> > > <mailto:ferruh.yigit@amd.com>> wrote:
> > >
> > >=C2=A0 =C2=A0 =C2=A0Hi Igor,
> > >
> > >=C2=A0 =C2=A0 =C2=A0VM doesn't have IOMMU, and vmxnet3 re= quires PA mode, for this can you
> > >=C2=A0 =C2=A0 =C2=A0please try with:
> > >=C2=A0 =C2=A0 =C2=A0- enable 'enable_unsafe_noiommu_mode&= #39; flag
> > >=C2=A0 =C2=A0 =C2=A0- Force PA mode via '--iova-mode=3Dpa= ' eal parameter
> > >
> > >
> > >=C2=A0 =C2=A0 =C2=A0Also to be able to figure out AMD IOMMU s= upport level, can you please
> > >=C2=A0 =C2=A0 =C2=A0provide:
> > >=C2=A0 =C2=A0 =C2=A0- AMD part number
> > >=C2=A0 =C2=A0 =C2=A0- ESXi/hypervisor version
> > >=C2=A0 =C2=A0 =C2=A0- ESXi & VM kernel logs
> > >
> > >
> > >=C2=A0 =C2=A0 =C2=A0Thanks,
> > >=C2=A0 =C2=A0 =C2=A0Ferruh
> > >
> > >=C2=A0 =C2=A0 =C2=A0On 7/20/2023 5:21 PM, Jochen Behrens wrot= e:
> > >=C2=A0 =C2=A0 =C2=A0> +Ronak from the ESX team
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0> In our usage, we do set amd_iommu=3D= off in the boot command line from
> > >=C2=A0 =C2=A0 =C2=A0> grub. (Or intel_iommu=3Doff for Inte= l processors.)
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0Jochen
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0> *From: *Thomas Monjalon <thomas@monjalon.net > > >=C2=A0 =C2=A0 =C2=A0<mailto:thomas@monjalon.net>>
> > >=C2=A0 =C2=A0 =C2=A0> *Date: *Thursday, July 20, 2023 at 6= :00 AM
> > >=C2=A0 =C2=A0 =C2=A0> *To: *Igor de Paula <igordptx@gmail.com <mail= to:igordptx@gmail.c= om
> > >>
> > >=C2=A0 =C2=A0 =C2=A0> *Cc: *users@dpdk.org <mailto:users@dpdk.org> <users@dpdk.org
> > >=C2=A0 =C2=A0 =C2=A0<mailto:users@dpdk.org>>, Jochen Behrens
> > >=C2=A0 =C2=A0 =C2=A0> <jbehrens@vmware.com <mailto:jbehrens@vmware.com>>, = Nipun Gupta
> > >=C2=A0 =C2=A0 =C2=A0<nipun.gupta@amd.com <mailto:nipun.gupta@amd.com>>, Nikhi= l Agarwal
> > >=C2=A0 =C2=A0 =C2=A0> <nikhil.agarwal@amd.com <mailto:nikhil.agarwal@amd.com= >>, Ferruh
> > >=C2=A0 =C2=A0 =C2=A0Yigit <ferruh.yigit@amd.com <mailto:ferruh.yigit@amd.com>&= gt;
> > >=C2=A0 =C2=A0 =C2=A0> *Subject: *Re: help
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0> !! External Email
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0> +Cc some AMD maintainers, they can h= ave an idea about IOMMU
> > settings.
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0> 20/07/2023 14:44, Igor de Paula:
> > >=C2=A0 =C2=A0 =C2=A0>> I have enabled it in the host an= d in the BIOS for AMD...
> > >=C2=A0 =C2=A0 =C2=A0>> In the Bios I changed to amd_iom= mu=3Don and in the host it's the
> > >=C2=A0 =C2=A0 =C2=A0same for
> > >=C2=A0 =C2=A0 =C2=A0>> either.
> > >=C2=A0 =C2=A0 =C2=A0>>
> > >=C2=A0 =C2=A0 =C2=A0>> On Thu, Jul 20, 2023 at 1:31=E2= =80=AFPM Thomas Monjalon
> > >=C2=A0 =C2=A0 =C2=A0<thomas@monjalon.net <mailto:thomas@monjalon.net>> wrote:=
> > >=C2=A0 =C2=A0 =C2=A0>>
> > >=C2=A0 =C2=A0 =C2=A0>> > 20/07/2023 11:35, Igor de P= aula:
> > >=C2=A0 =C2=A0 =C2=A0>> > > The weird thing is tha= t it only happens when I am using a
> > >=C2=A0 =C2=A0 =C2=A0host with an
> > >=C2=A0 =C2=A0 =C2=A0>> > AMD
> > >=C2=A0 =C2=A0 =C2=A0>> > > processor. It doesn= 9;t happen when I use a host with an Intel
> > >=C2=A0 =C2=A0 =C2=A0processor.
> > >=C2=A0 =C2=A0 =C2=A0>> >
> > >=C2=A0 =C2=A0 =C2=A0>> > So it's probably a matt= er of BIOS settings for the IOMMU?
> > >=C2=A0 =C2=A0 =C2=A0>> >
> > >=C2=A0 =C2=A0 =C2=A0>> >
> > >=C2=A0 =C2=A0 =C2=A0>> > > On Thu, Jul 20, 2023 a= t 10:32=E2=80=AFAM Thomas Monjalon
> > >=C2=A0 =C2=A0 =C2=A0<thomas@monjalon.net <mailto:thomas@monjalon.net>>
> > >=C2=A0 =C2=A0 =C2=A0>> > > wrote:
> > >=C2=A0 =C2=A0 =C2=A0>> > >
> > >=C2=A0 =C2=A0 =C2=A0>> > > > +Cc the vmxnet3 m= aintainer.
> > >=C2=A0 =C2=A0 =C2=A0>> > > >
> > >=C2=A0 =C2=A0 =C2=A0>> > > > Please Jochen, do= you have an idea what's wrong below?
> > >=C2=A0 =C2=A0 =C2=A0>> > > >
> > >=C2=A0 =C2=A0 =C2=A0>> > > >
> > >=C2=A0 =C2=A0 =C2=A0>> > > > 20/07/2023 11:25,= Igor de Paula:
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > This is beca= use it can't negotiate the IOMMU type with
> > >=C2=A0 =C2=A0 =C2=A0any port.
> > >=C2=A0 =C2=A0 =C2=A0>> > > > >
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > On Thu, Jul = 20, 2023 at 5:08=E2=80=AFAM Thomas Monjalon
> > >=C2=A0 =C2=A0 =C2=A0<thomas@monjalon.net <mailto:thomas@monjalon.net>
> > >=C2=A0 =C2=A0 =C2=A0>> > >
> > >=C2=A0 =C2=A0 =C2=A0>> > > > wrote:
> > >=C2=A0 =C2=A0 =C2=A0>> > > > >
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > Hello,<= br> > > >=C2=A0 =C2=A0 =C2=A0>> > > > > >
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > The fir= st error is "Cause: Error: number of ports must
> > >=C2=A0 =C2=A0 =C2=A0be even"
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > >
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > >
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > 03/05/2= 023 18:13, Igor de Paula:
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > I = am running a VM inside a VMWARE server (vSphere).
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > My= goal it to set up DPDK with two HW ports, and set
> > up a
> > >=C2=A0 =C2=A0 =C2=A0>> > > > virtio_user to > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > in= teract with the kernel stack.
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > In= another app I have it working but instead of
> > >=C2=A0 =C2=A0 =C2=A0virtio_user I am
> > >=C2=A0 =C2=A0 =C2=A0>> > > > running
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > KN= I, it works in IOVA-PA mode.
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > I = am looking to replace the KNI.
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Wh= en I try to set up virtio_user port as in the doc:
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > >
> > >=C2=A0 =C2=A0 =C2=A0>> > > >
> > >=C2=A0 =C2=A0 =C2=A0>> >
> > >
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.htm= l#virtio-user-as-exception-path
> > <
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.htm= l#virtio-user-as-exception-path>
> > <
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.htm= l#virtio-user-as-exception-path
> > <
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.htm= l#virtio-user-as-exception-path
> > >>
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > I = get a error it can't run in PA mode.
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Wh= en I try to run as VA mode from a parameter, I get
> > the
> > >=C2=A0 =C2=A0 =C2=A0>> > following
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > errors:=
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: lib.eal log level changed from info to debug
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Detected lcore 0 as core 0 on socket 0
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Detected lcore 1 as core 0 on socket 0
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Support maximum 128 logical core(s) by
> > >=C2=A0 =C2=A0 =C2=A0configuration.
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Detected 2 lcore(s)
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Detected 1 NUMA nodes
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Checking presence of .so 'librte_eal.so.21.3'
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Checking presence of .so 'librte_eal.so.21'
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Checking presence of .so 'librte_eal.so'
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Detected static linkage of DPDK
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Ask a virtual area of 0x7000 bytes
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Virtual area found at 0x100000000 (size =3D 0x7000)
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Multi-process socket /var/run/dpdk/rte/mp_socket
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: DPAA Bus not present. Skipping.
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: VFIO PCI modules not loaded
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Selected IOVA mode 'VA'
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Probing VFIO support...
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: IOMMU type 1 (Type 1) is supported
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: IOMMU type 7 (sPAPR) is not supported
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: IOMMU type 8 (No-IOMMU) is supported
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: VFIO support initialized
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Ask a virtual area of 0x5b000 bytes
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Virtual area found at 0x100007000 (size =3D
> > 0x5b000)
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Setting up physically contiguous memory...
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Setting maximum number of open files to 1048576
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Detected memory type: socket_id:0
> > >=C2=A0 =C2=A0 =C2=A0hugepage_sz:1073741824
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Creating 2 segment lists: n_segs:128 socket_id:0
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > hugepag= e_sz:1073741824
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Ask a virtual area of 0x2000 bytes
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Virtual area found at 0x100062000 (size =3D 0x2000)
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Memseg list allocated at socket 0, page size
> > >=C2=A0 =C2=A0 =C2=A00x100000kB
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Ask a virtual area of 0x2000000000 bytes
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Virtual area found at 0x140000000 (size =3D
> > >=C2=A0 =C2=A0 =C2=A00x2000000000)
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: VA reserved for memseg list at 0x140000000, size
> > >=C2=A0 =C2=A0 =C2=A02000000000
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Ask a virtual area of 0x2000 bytes
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Virtual area found at 0x2140000000 (size =3D
> > 0x2000)
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Memseg list allocated at socket 0, page size
> > >=C2=A0 =C2=A0 =C2=A00x100000kB
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Ask a virtual area of 0x2000000000 bytes
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Virtual area found at 0x2180000000 (size =3D
> > >=C2=A0 =C2=A0 =C2=A00x2000000000)
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: VA reserved for memseg list at 0x2180000000,
> > >=C2=A0 =C2=A0 =C2=A0size 2000000000
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: TSC frequency is ~2350000 KHz
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Main lcore 0 is ready
> > (tid=3D7f8ad790ec00;cpuset=3D[0])
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: lcore 1 is ready (tid=3D7f8ad6907400;cpuset=3D[1])
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Trying to obtain current memory policy.
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Setting policy MPOL_PREFERRED for socket 0
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Restoring previous memory policy: 0
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: request: mp_malloc_sync
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Heap on socket 0 was expanded by 1024MB
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: PCI device 0000:0b:00.0 on NUMA socket -1
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L:=C2=A0 =C2=A0probe driver: 15ad:7b0 net_vmxnet3
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L:=C2=A0 =C2=A0Expecting 'PA' IOVA mode but current mode is
> > >=C2=A0 =C2=A0 =C2=A0'VA', not
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > initial= izing
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Requested device 0000:0b:00.0 cannot be used
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: PCI device 0000:13:00.0 on NUMA socket -1
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L:=C2=A0 =C2=A0probe driver: 15ad:7b0 net_vmxnet3
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L:=C2=A0 =C2=A0Expecting 'PA' IOVA mode but current mode is
> > >=C2=A0 =C2=A0 =C2=A0'VA', not
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > initial= izing
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Requested device 0000:13:00.0 cannot be used
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Bus (pci) probe failed.
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: lib.telemetry log level changed from disabled to
> > >=C2=A0 =C2=A0 =C2=A0warning
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Error - exiting with code: 1
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > >=C2= =A0 =C2=A0Cause: Error: number of ports must be even
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: request: mp_malloc_sync
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Heap on socket 0 was shrunk by 1024MB
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Fo= r some reason the HW ports won't setup. From what I
> > >=C2=A0 =C2=A0 =C2=A0understand
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > ne= t_vmxnet3 should work with VA mode.
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > I = enabled I/OMUU for the VM.
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Th= e weird thing even when enabled, I still have the
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > en= able_unsafe_noiommu_mode flag on.
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > An= d because it's on the this:
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > de= v_iova_mode =3D pci_device_iova_mode(dr, dev);
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > re= turn PA mode, and it fails.
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Wh= en I disable it by modifying
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0/sys/module/vfio/parameters/enable_unsafe= _noiommu_mode, I get
> > >=C2=A0 =C2=A0 =C2=A0>> > another
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > error.<= br> > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Th= e error is that it doesn't find a suitable IOMMU
> > type:
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Ju= st putting the relevant message:
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Heap on socket 0 was expanded by 1024MB
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: PCI device 0000:0b:00.0 on NUMA socket -1
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L:=C2=A0 =C2=A0probe driver: 15ad:7b0 net_vmxnet3
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Set IOMMU type 1 (Type 1) failed, error 19 (No
> > >=C2=A0 =C2=A0 =C2=A0such device)
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Set IOMMU type 7 (sPAPR) failed, error 19 (No
> > >=C2=A0 =C2=A0 =C2=A0such device)
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
> > >=C2=A0 =C2=A0 =C2=A0such
> > >=C2=A0 =C2=A0 =C2=A0>> > device)
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: 0000:0b:00.0 failed to select IOMMU type
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Requested device 0000:0b:00.0 cannot be used
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: PCI device 0000:13:00.0 on NUMA socket -1
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L:=C2=A0 =C2=A0probe driver: 15ad:7b0 net_vmxnet3
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Set IOMMU type 1 (Type 1) failed, error 19 (No
> > >=C2=A0 =C2=A0 =C2=A0such device)
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Set IOMMU type 7 (sPAPR) failed, error 19 (No
> > >=C2=A0 =C2=A0 =C2=A0such device)
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
> > >=C2=A0 =C2=A0 =C2=A0such
> > >=C2=A0 =C2=A0 =C2=A0>> > device)
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: 0000:13:00.0 failed to select IOMMU type
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > EA= L: Requested device 0000:13:00.0 cannot be used
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > I = can see where it fails:
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > co= nst struct vfio_iommu_type *
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > vf= io_set_iommu_type(int vfio_container_fd)
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > {<= br> > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > un= signed idx;
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > fo= r (idx =3D 0; idx < RTE_DIM(iommu_types); idx++) {
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > co= nst struct vfio_iommu_type *t =3D &iommu_types[idx];
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > in= t ret =3D ioctl(vfio_container_fd, VFIO_SET_IOMMU,
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > t-= >type_id);
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > if= (!ret) {
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > RT= E_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > t-= >type_id, t->name);
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > re= turn t;
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > }<= br> > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > /*= not an error, there may be more supported IOMMU
> > >=C2=A0 =C2=A0 =C2=A0types */
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > RT= E_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed,
> > >=C2=A0 =C2=A0 =C2=A0error "
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > &q= uot;%i (%s)\n", t->type_id, t->name, errno,
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > st= rerror(errno));
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > }<= br> > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > /*= if we didn't find a suitable IOMMU type, fail */
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > re= turn NULL;
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > }<= br> > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Th= e ioctl returns -1.
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > > > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > I = tried many things.
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > En= abling/disabling IOMMU.
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Mo= difying the enable_unsafe_noiommu_mode flag.
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > Ru= nning in PA mode.
> > >=C2=A0 =C2=A0 =C2=A0>> > > > > > > An= y help will be much appreciated.
> > >=C2=A0 =C2=A0 =C2=A0>> >
> > >=C2=A0 =C2=A0 =C2=A0>> >
> > >=C2=A0 =C2=A0 =C2=A0>> >
> > >=C2=A0 =C2=A0 =C2=A0>> >
> > >=C2=A0 =C2=A0 =C2=A0>> >
> > >=C2=A0 =C2=A0 =C2=A0>> >
> > >=C2=A0 =C2=A0 =C2=A0>>
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0>
> > >=C2=A0 =C2=A0 =C2=A0> !! External Email: This email origin= ated from outside of the
> > >=C2=A0 =C2=A0 =C2=A0> organization. Do not click links or = open attachments unless you
> > >=C2=A0 =C2=A0 =C2=A0> recognize the sender.
> > >=C2=A0 =C2=A0 =C2=A0>
> > >
> >
> >
>





--0000000000000d36d10602a4fa9c--