From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5CA7D42C04 for ; Thu, 1 Jun 2023 16:03:35 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C644B40DDC; Thu, 1 Jun 2023 16:03:34 +0200 (CEST) Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by mails.dpdk.org (Postfix) with ESMTP id 6037B406BA for ; Thu, 1 Jun 2023 16:03:33 +0200 (CEST) Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-256c8bed212so291040a91.3 for ; Thu, 01 Jun 2023 07:03:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685628212; x=1688220212; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=kdcdxNmyvGIReW8a10rU+RWAQTSTbQDoyg0j+/DfB2A=; b=ef+oGF8UCY0iSJ5gkaS2Jme14ifC9Hd5kAn9yzREipsyewc+ztx9vlQal2Aa/s99lb cWqy8jG/5jpQK4H0Wx20BYuYPJ/Fj4JNQ/sc7CSYNkta2/iF/w8xb7HDO/rAKLb+TfRn wo56OyGGYp40UOPZZLz51xdOuGqIldJzeqCJ8lBIkdtrlh+aFucTFjH5soKOycIaVnnW Ug76rZRRLum3IieHDyxrB2QFFpgEe4oGyRCn0762wBuY1PcenLJW7r/2S86mEhRdxtn+ geePp9TwX7u0m4I59K/sjYaJ+V8XO7F+0OLnHNTOmWei7NlkkXNvxmml93YgyWkcutYh uS7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685628212; x=1688220212; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=kdcdxNmyvGIReW8a10rU+RWAQTSTbQDoyg0j+/DfB2A=; b=OnHLr5yEhDh/ZypYmaSBakn7ZxpdKm0vIuvq37tt9nCy6RFx5WZft0A/V9pG2562YO ok1LoJtzK74OFTqCjMBOK+0zwvyV34cBYQoFs/heaC72miCPM8su64M3Ip3iUeG6UbhL hG/rvAoBgd6GhdW3E3JMuO2M6p4m2CIZUbVfs+aq6+LhCDAGx0JUJZN/4vMyBKddprn5 AfNCjOML8f1moqfWmHifH0KoszCEeN0sAvbaqMlyXQAyBwnQYGlrGzQfKwbyl9uV7bFr Rm381FjUx2LD5GUhaWx73yQWbF3SYr9wcLPO2Iw5XSVpXwN3NW2+eKwRi23/cYZ0uqh3 kG0Q== X-Gm-Message-State: AC+VfDz84mRjlzgs4s3YJX6LJAvJ09Jj/9y0W1P+gFSqZiPcOQXEDArj 7QabyHze8yv+JS8Rwkix+SPNY8rJq7bHzbYJFHlTRUah9lI= X-Google-Smtp-Source: ACHHUZ6y+q2CsRzYjIyvWYBE0UPbkYMWwSRJ9xKMEa4oggVhgwYkCn6Alk70kS9PYo8SU40Aa4OXvCo3xTeSXocu4Tg= X-Received: by 2002:a17:90a:8b92:b0:258:817a:814e with SMTP id z18-20020a17090a8b9200b00258817a814emr2424091pjn.28.1685628211866; Thu, 01 Jun 2023 07:03:31 -0700 (PDT) MIME-Version: 1.0 From: Igor de Paula Date: Thu, 1 Jun 2023 15:03:21 +0100 Message-ID: Subject: VMXNET3 driver IOVA support To: users@dpdk.org Content-Type: multipart/alternative; boundary="000000000000caaf5505fd11e6f6" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --000000000000caaf5505fd11e6f6 Content-Type: text/plain; charset="UTF-8" Hi, I am using a Vmware host (ESXI 7.0) where I have IOMMU enabled. I am using virtio_user ports as an exception path so I am trying to use IOVA as VA. Just as an example, I am trying to run the l2fw example application with --iova-mode=va. Although I shows there is support for IOMMU, it fails to negotiate the IOMMU type, here is the output of: ./dpdk-l2fwd --iova-mode=va --log-level=eal,8 EAL: lib.eal log level changed from info to debug EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 0 on socket 0 EAL: Support maximum 128 logical core(s) by configuration. EAL: Detected 2 lcore(s) EAL: Detected 1 NUMA nodes EAL: Checking presence of .so 'librte_eal.so.21.3' EAL: Checking presence of .so 'librte_eal.so.21' EAL: Checking presence of .so 'librte_eal.so' EAL: Detected static linkage of DPDK EAL: Ask a virtual area of 0x7000 bytes EAL: Virtual area found at 0x100000000 (size = 0x7000) EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: DPAA Bus not present. Skipping. EAL: VFIO PCI modules not loaded EAL: Selected IOVA mode 'VA' EAL: Probing VFIO support... *EAL: IOMMU type 1 (Type 1) is supported* EAL: IOMMU type 7 (sPAPR) is not supported EAL: IOMMU type 8 (No-IOMMU) is not supported EAL: VFIO support initialized EAL: Ask a virtual area of 0x5b000 bytes EAL: Virtual area found at 0x100007000 (size = 0x5b000) EAL: Setting up physically contiguous memory... EAL: Setting maximum number of open files to 1048576 EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824 EAL: Creating 2 segment lists: n_segs:128 socket_id:0 hugepage_sz:1073741824 EAL: Ask a virtual area of 0x2000 bytes EAL: Virtual area found at 0x100062000 (size = 0x2000) EAL: Memseg list allocated at socket 0, page size 0x100000kB EAL: Ask a virtual area of 0x2000000000 bytes EAL: Virtual area found at 0x140000000 (size = 0x2000000000) EAL: VA reserved for memseg list at 0x140000000, size 2000000000 EAL: Ask a virtual area of 0x2000 bytes EAL: Virtual area found at 0x2140000000 (size = 0x2000) EAL: Memseg list allocated at socket 0, page size 0x100000kB EAL: Ask a virtual area of 0x2000000000 bytes EAL: Virtual area found at 0x2180000000 (size = 0x2000000000) EAL: VA reserved for memseg list at 0x2180000000, size 2000000000 EAL: TSC frequency is ~2350000 KHz EAL: Main lcore 0 is ready (tid=7f3f3a1e5c00;cpuset=[0]) EAL: lcore 1 is ready (tid=7f3f391de400;cpuset=[1]) EAL: Trying to obtain current memory policy. EAL: Setting policy MPOL_PREFERRED for socket 0 EAL: Restoring previous memory policy: 0 EAL: request: mp_malloc_sync EAL: Heap on socket 0 was expanded by 1024MB EAL: PCI device 0000:03:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 net_vmxnet3 EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:0b:00.0 on NUMA socket -1 EAL: probe driver: 15ad:7b0 net_vmxnet3 EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device) EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device) EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device) EAL: 0000:0b:00.0 failed to select IOMMU type EAL: Requested device 0000:0b:00.0 cannot be used EAL: PCI device 0000:13:00.0 on NUMA socket -1 *EAL: probe driver: 15ad:7b0 net_vmxnet3* *EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device)* EAL: 0000:13:00.0 failed to select IOMMU type EAL: Requested device 0000:13:00.0 cannot be used EAL: lib.telemetry log level changed from disabled to warning MAC updating enabled EAL: Error - exiting with code: 1 Cause: No Ethernet ports - bye EAL: request: mp_malloc_sync EAL: Heap on socket 0 was shrunk by 1024MB The negotiation for IOMMU type happens in rte_vfio_setup_device(), it calls the function: set_iommu_type() in the following snippet: /* * pick an IOMMU type and set up DMA mappings for container * * needs to be done only once, only when first group is * assigned to a container and only in primary process. * Note this can happen several times with the hotplug * functionality. */ if (internal_conf->process_type == RTE_PROC_PRIMARY && vfio_cfg->vfio_active_groups == 1 && vfio_group_device_count(vfio_group_fd) == 0) { const struct vfio_iommu_type *t; /* select an IOMMU type which we will be using */ t = vfio_set_iommu_type(vfio_container_fd); if (!t) { RTE_LOG(ERR, EAL, "%s failed to select IOMMU type\n", dev_addr); close(vfio_group_fd); rte_vfio_clear_group(vfio_group_fd); return -1; } In there there is a ioctl call: unsigned idx; for (idx = 0; idx < RTE_DIM(iommu_types); idx++) { const struct vfio_iommu_type *t = &iommu_types[idx]; int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU, t->type_id); which fails. Its also weird for me the when using ./usertool/dpdk-devbind.py --status I gett the following output: Network devices using DPDK-compatible driver ============================================ *0000:0b:00.0 'VMXNET3 Ethernet Controller 07b0' drv=vfio-pci unused=vmxnet30000:13:00.0 'VMXNET3 Ethernet Controller 07b0' drv=vfio-pci unused=vmxnet3* Network devices using kernel driver =================================== 0000:03:00.0 'VMXNET3 Ethernet Controller 07b0' if=eth0 drv=vmxnet3 unused=vfio-pci *Active* Meaning the driver that is connected is vfio-pci. So why is it probing VMXNET3? Does anyone have any experience in running IOVA as VA in VMWARE host? --000000000000caaf5505fd11e6f6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,
I am using a Vmware host (ESXI 7.0) where I have IO= MMU enabled. I am using virtio_user ports as an exception path so I am tryi= ng to use IOVA as VA. Just as an example, I am trying to run the l2fw examp= le application with --iova-mode=3Dva.
Although I shows there is support = for IOMMU, it fails to negotiate the IOMMU type, here is the output of: ./d= pdk-l2fwd --iova-mode=3Dva --log-level=3Deal,8

EAL: lib.eal log leve= l changed from info to debug
EAL: Detected lcore 0 as core 0 on socket 0=
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Support maximum 128= logical core(s) by configuration.
EAL: Detected 2 lcore(s)
EAL: Dete= cted 1 NUMA nodes
EAL: Checking presence of .so 'librte_eal.so.21.3&= #39;
EAL: Checking presence of .so 'librte_eal.so.21'
EAL: Ch= ecking presence of .so 'librte_eal.so'
EAL: Detected static link= age of DPDK
EAL: Ask a virtual area of 0x7000 bytes
EAL: Virtual area= found at 0x100000000 (size =3D 0x7000)
EAL: Multi-process socket /var/r= un/dpdk/rte/mp_socket
EAL: DPAA Bus not present. Skipping.
EAL: VFIO = PCI modules not loaded
EAL: Selected IOVA mode 'VA'
EAL: Prob= ing VFIO support...
EAL: IOMMU type 1 (Type 1) is supported
EA= L: IOMMU type 7 (sPAPR) is not supported
EAL: IOMMU type 8 (No-IOMMU) is= not supported
EAL: VFIO support initialized
EAL: Ask a virtual area = of 0x5b000 bytes
EAL: Virtual area found at 0x100007000 (size =3D 0x5b00= 0)
EAL: Setting up physically contiguous memory...
EAL: Setting maxim= um number of open files to 1048576
EAL: Detected memory type: socket_id:= 0 hugepage_sz:1073741824
EAL: Creating 2 segment lists: n_segs:128 socke= t_id:0 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x2000 bytesEAL: Virtual area found at 0x100062000 (size =3D 0x2000)
EAL: Memseg li= st allocated at socket 0, page size 0x100000kB
EAL: Ask a virtual area o= f 0x2000000000 bytes
EAL: Virtual area found at 0x140000000 (size =3D 0x= 2000000000)
EAL: VA reserved for memseg list at 0x140000000, size 200000= 0000
EAL: Ask a virtual area of 0x2000 bytes
EAL: Virtual area found = at 0x2140000000 (size =3D 0x2000)
EAL: Memseg list allocated at socket 0= , page size 0x100000kB
EAL: Ask a virtual area of 0x2000000000 bytes
= EAL: Virtual area found at 0x2180000000 (size =3D 0x2000000000)
EAL: VA = reserved for memseg list at 0x2180000000, size 2000000000
EAL: TSC frequ= ency is ~2350000 KHz
EAL: Main lcore 0 is ready (tid=3D7f3f3a1e5c00;cpus= et=3D[0])
EAL: lcore 1 is ready (tid=3D7f3f391de400;cpuset=3D[1])
EAL= : Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFE= RRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: reque= st: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 1024MB
EAL: = PCI device 0000:03:00.0 on NUMA socket -1
EAL: =C2=A0 probe driver: 15ad= :7b0 net_vmxnet3
EAL: =C2=A0 Not managed by a supported kernel driver, s= kipped
EAL: PCI device 0000:0b:00.0 on NUMA socket -1
EAL: =C2=A0 pro= be driver: 15ad:7b0 net_vmxnet3
EAL: Set IOMMU type 1 (Type 1) failed, e= rror 19 (No such device)
EAL: Set IOMMU type 7 (sPAPR) failed, error 19 = (No such device)
EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No s= uch device)
EAL: 0000:0b:00.0 failed to select IOMMU type
EAL: Reques= ted device 0000:0b:00.0 cannot be used
EAL: PCI device 0000:13:00.0 on N= UMA socket -1
EAL: =C2=A0 probe driver: 15ad:7b0 net_vmxnet3
<= b>EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
EAL: = Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
EAL: Set IOMM= U type 8 (No-IOMMU) failed, error 19 (No such device)
EAL: 0000:13:0= 0.0 failed to select IOMMU type
EAL: Requested device 0000:13:00.0 canno= t be used
EAL: lib.telemetry log level changed from disabled to warning<= br>MAC updating enabled
EAL: Error - exiting with code: 1
=C2=A0 Caus= e: No Ethernet ports - bye
EAL: request: mp_malloc_sync
EAL: Heap on = socket 0 was shrunk by 1024MB

The negotiation for IOMMU type happens= in=C2=A0rte_vfio_setup_device(), it calls the function:
set_iommu_type(= ) in the following=C2=A0snippet:

/*
* pick an IOMMU type and s= et up DMA mappings for container
*
* needs to be done only once= , only when first group is
* assigned to a container and only in prim= ary process.
* Note this can happen several times with the hotplug * functionality.
*/
if (internal_conf->process_type =3D= =3D RTE_PROC_PRIMARY &&
vfio_cfg->vfio_active_groups =3D= =3D 1 &&
vfio_group_device_count(vfio_group_fd) =3D=3D 0) {<= br> const struct vfio_iommu_type *t;

/* select an IOMMU type wh= ich we will be using */
t =3D vfio_set_iommu_type(vfio_container_fd);=
if (!t) {
RTE_LOG(ERR, EAL,
"%s failed to select= IOMMU type\n",
dev_addr);
close(vfio_group_fd);
= rte_vfio_clear_group(vfio_group_fd);
return -1;
}
In there there is a ioctl call:
unsigned idx;
for (idx = =3D 0; idx < RTE_DIM(iommu_types); idx++) {
const struct vfio_iommu= _type *t =3D &iommu_types[idx];

int ret =3D ioctl(vfio_contain= er_fd, VFIO_SET_IOMMU,
t->type_id);

which fails.

Its also weird for me the when using ./usertool/dpdk-devbin= d.py --status
I gett the following output:
Network devices using DPDK= -compatible driver
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D
0000:0b:00.0 'VMXNET3 Ethernet Controller 07b0' drv=3D= vfio-pci unused=3Dvmxnet3
0000:13:00.0 'VMXNET3 Ethernet Controller = 07b0' drv=3Dvfio-pci unused=3Dvmxnet3

Network devices using = kernel driver
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
0000:03:00.0 'VMXNE= T3 Ethernet Controller 07b0' if=3Deth0 drv=3Dvmxnet3 unused=3Dvfio-pci = *Active*

Meaning the driver that is connected is vfio-pci. So w= hy is it probing VMXNET3?

Does anyone have any experience in running= IOVA as VA in VMWARE host?


--000000000000caaf5505fd11e6f6--