From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 23F8242EBF for ; Thu, 20 Jul 2023 11:35:22 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 10A25427F5; Thu, 20 Jul 2023 11:35:22 +0200 (CEST) Received: from mail-yb1-f181.google.com (mail-yb1-f181.google.com [209.85.219.181]) by mails.dpdk.org (Postfix) with ESMTP id 9B6BC41144 for ; Thu, 20 Jul 2023 11:35:20 +0200 (CEST) Received: by mail-yb1-f181.google.com with SMTP id 3f1490d57ef6-c11e2b31b95so531646276.3 for ; Thu, 20 Jul 2023 02:35:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689845720; x=1690450520; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=/PV4LZOYFV0Y3u5KWyy03wn4mZ3AyRp8z1GdyIFKhlU=; b=rahLWsGLnAf7pL1KmU+hZo+yustsx9fN14hYOe4GnIvzfn5fKtC2CFO6G6yyErBqpV ggMX4UEWR1SfkhpRSpf+G2LtMJYLpYMpqgHcGFxjZgeL964StN0fW3cG29U5KDCMtTGu poGiKEmlUjyrMHglb1U0HGVpnD10nKdDVTpPJCW7XrJgXhA57t1ewZD5mC0VaiD2LZ9R rLt+i63XMkL7kG4I7jbqgCqhPfYpsTbPrjQ2wh/qJth+5737yeBRU1TCmjEYgZffVySf GIE19iGJEzw4doLSh9vFM8PGyTe9HVGvNOoDRqvktFb0exEYF8wRNlhRsoD9a9mqSCET BKgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689845720; x=1690450520; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/PV4LZOYFV0Y3u5KWyy03wn4mZ3AyRp8z1GdyIFKhlU=; b=FTA8ChW1n6I9b2+RQFPkOw83FfggqDiROygup1rd18BvQTeR+xhRw/S4qh4+1jq5/n 8PG4kct9GAlN9ovNpt20quUA1j/HfRDLO0sg2qGoj1fqP+Vc56I6Bpl4Ammtjap7UQt0 aIjfDlqVgqzU8kteWzxsK06DjNqvlkhEhAPz11z+lcipwLgBfj1G1TadgV+YzMPTioME xZisGEDvsADntZuQ6N4Mnxsu5KcTZexXUXg/qjkxyHZodwg55yhDrNpoE/fTZJf+r/sQ oFA8OYDtCl60tI6F/LU5s6nln9YqFJs+duInFUFG6NPkiCmZyL/jMypEry0bP+b6k8BK j10g== X-Gm-Message-State: ABy/qLb1aZ3enJyqbbeyoV9bR/kozvM2xCh2uK4ca7FgmEXocbxyPbjS iumX/bXKY9zzxoHYILT6DFZykMYwLhBm5c4PfmIPeowV X-Google-Smtp-Source: APBJJlGhQkwqxW7ZZD7hhS3ZZwcuv9Sw1cPSH6l1lY5vJY88/3cxPT8qjkT2AuEYVxKJNp4ImHc2qVLgBq9jchLlp0g= X-Received: by 2002:a25:e807:0:b0:c68:c71e:cdb1 with SMTP id k7-20020a25e807000000b00c68c71ecdb1mr3968512ybd.26.1689845719873; Thu, 20 Jul 2023 02:35:19 -0700 (PDT) MIME-Version: 1.0 References: <10081895.2WqB4rESCP@thomas> <8740856.JRmrKFJ9eK@thomas> In-Reply-To: <8740856.JRmrKFJ9eK@thomas> From: Igor de Paula Date: Thu, 20 Jul 2023 10:35:09 +0100 Message-ID: Subject: Re: help To: Thomas Monjalon Cc: users@dpdk.org, Jochen Behrens Content-Type: multipart/alternative; boundary="000000000000dbbd180600e7ddfa" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --000000000000dbbd180600e7ddfa Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable The weird thing is that it only happens when I am using a host with an AMD processor. It doesn't happen when I use a host with an Intel processor. On Thu, Jul 20, 2023 at 10:32=E2=80=AFAM Thomas Monjalon wrote: > +Cc the vmxnet3 maintainer. > > Please Jochen, do you have an idea what's wrong below? > > > 20/07/2023 11:25, Igor de Paula: > > This is because it can't negotiate the IOMMU type with any port. > > > > On Thu, Jul 20, 2023 at 5:08=E2=80=AFAM Thomas Monjalon > wrote: > > > > > Hello, > > > > > > The first error is "Cause: Error: number of ports must be even" > > > > > > > > > 03/05/2023 18:13, Igor de Paula: > > > > I am running a VM inside a VMWARE server (vSphere). > > > > My goal it to set up DPDK with two HW ports, and set up a > virtio_user to > > > > interact with the kernel stack. > > > > In another app I have it working but instead of virtio_user I am > running > > > > KNI, it works in IOVA-PA mode. > > > > I am looking to replace the KNI. > > > > > > > > When I try to set up virtio_user port as in the doc: > > > > > > > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virt= io-user-as-exception-path > > > > I get a error it can't run in PA mode. > > > > > > > > > > > > When I try to run as VA mode from a parameter, I get the following > > > errors: > > > > EAL: lib.eal log level changed from info to debug > > > > EAL: Detected lcore 0 as core 0 on socket 0 > > > > EAL: Detected lcore 1 as core 0 on socket 0 > > > > EAL: Support maximum 128 logical core(s) by configuration. > > > > EAL: Detected 2 lcore(s) > > > > EAL: Detected 1 NUMA nodes > > > > EAL: Checking presence of .so 'librte_eal.so.21.3' > > > > EAL: Checking presence of .so 'librte_eal.so.21' > > > > EAL: Checking presence of .so 'librte_eal.so' > > > > EAL: Detected static linkage of DPDK > > > > EAL: Ask a virtual area of 0x7000 bytes > > > > EAL: Virtual area found at 0x100000000 (size =3D 0x7000) > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > > > > EAL: DPAA Bus not present. Skipping. > > > > EAL: VFIO PCI modules not loaded > > > > EAL: Selected IOVA mode 'VA' > > > > EAL: Probing VFIO support... > > > > EAL: IOMMU type 1 (Type 1) is supported > > > > EAL: IOMMU type 7 (sPAPR) is not supported > > > > EAL: IOMMU type 8 (No-IOMMU) is supported > > > > EAL: VFIO support initialized > > > > EAL: Ask a virtual area of 0x5b000 bytes > > > > EAL: Virtual area found at 0x100007000 (size =3D 0x5b000) > > > > EAL: Setting up physically contiguous memory... > > > > EAL: Setting maximum number of open files to 1048576 > > > > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824 > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0 > > > hugepage_sz:1073741824 > > > > EAL: Ask a virtual area of 0x2000 bytes > > > > EAL: Virtual area found at 0x100062000 (size =3D 0x2000) > > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB > > > > EAL: Ask a virtual area of 0x2000000000 bytes > > > > EAL: Virtual area found at 0x140000000 (size =3D 0x2000000000) > > > > EAL: VA reserved for memseg list at 0x140000000, size 2000000000 > > > > EAL: Ask a virtual area of 0x2000 bytes > > > > EAL: Virtual area found at 0x2140000000 (size =3D 0x2000) > > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB > > > > EAL: Ask a virtual area of 0x2000000000 bytes > > > > EAL: Virtual area found at 0x2180000000 (size =3D 0x2000000000) > > > > EAL: VA reserved for memseg list at 0x2180000000, size 2000000000 > > > > EAL: TSC frequency is ~2350000 KHz > > > > EAL: Main lcore 0 is ready (tid=3D7f8ad790ec00;cpuset=3D[0]) > > > > EAL: lcore 1 is ready (tid=3D7f8ad6907400;cpuset=3D[1]) > > > > EAL: Trying to obtain current memory policy. > > > > EAL: Setting policy MPOL_PREFERRED for socket 0 > > > > EAL: Restoring previous memory policy: 0 > > > > EAL: request: mp_malloc_sync > > > > EAL: Heap on socket 0 was expanded by 1024MB > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not > > > initializing > > > > EAL: Requested device 0000:0b:00.0 cannot be used > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1 > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not > > > initializing > > > > EAL: Requested device 0000:13:00.0 cannot be used > > > > EAL: Bus (pci) probe failed. > > > > EAL: lib.telemetry log level changed from disabled to warning > > > > EAL: Error - exiting with code: 1 > > > > Cause: Error: number of ports must be even > > > > EAL: request: mp_malloc_sync > > > > EAL: Heap on socket 0 was shrunk by 1024MB > > > > > > > > > > > > > > > > For some reason the HW ports won't setup. From what I understand > > > > net_vmxnet3 should work with VA mode. > > > > I enabled I/OMUU for the VM. > > > > The weird thing even when enabled, I still have the > > > > enable_unsafe_noiommu_mode flag on. > > > > And because it's on the this: > > > > > > > > dev_iova_mode =3D pci_device_iova_mode(dr, dev); > > > > > > > > return PA mode, and it fails. > > > > > > > > When I disable it by modifying > > > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get anoth= er > > > error. > > > > The error is that it doesn't find a suitable IOMMU type: > > > > Just putting the relevant message: > > > > > > > > > > > > EAL: Heap on socket 0 was expanded by 1024MB > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device) > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device) > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device) > > > > EAL: 0000:0b:00.0 failed to select IOMMU type > > > > EAL: Requested device 0000:0b:00.0 cannot be used > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1 > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device) > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device) > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device) > > > > EAL: 0000:13:00.0 failed to select IOMMU type > > > > EAL: Requested device 0000:13:00.0 cannot be used > > > > > > > > > > > > I can see where it fails: > > > > > > > > > > > > const struct vfio_iommu_type * > > > > vfio_set_iommu_type(int vfio_container_fd) > > > > { > > > > unsigned idx; > > > > for (idx =3D 0; idx < RTE_DIM(iommu_types); idx++) { > > > > const struct vfio_iommu_type *t =3D &iommu_types[idx]; > > > > > > > > int ret =3D ioctl(vfio_container_fd, VFIO_SET_IOMMU, > > > > t->type_id); > > > > if (!ret) { > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n", > > > > t->type_id, t->name); > > > > return t; > > > > } > > > > /* not an error, there may be more supported IOMMU types */ > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error " > > > > "%i (%s)\n", t->type_id, t->name, errno, > > > > strerror(errno)); > > > > } > > > > /* if we didn't find a suitable IOMMU type, fail */ > > > > return NULL; > > > > } > > > > > > > > > > > > The ioctl returns -1. > > > > > > > > > > > > > > > > I tried many things. > > > > Enabling/disabling IOMMU. > > > > Modifying the enable_unsafe_noiommu_mode flag. > > > > Running in PA mode. > > > > Any help will be much appreciated. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --000000000000dbbd180600e7ddfa Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
The weird thing is that it only happens when I am using a = host with an AMD processor. It doesn't happen when I use a host with an= Intel processor.

On Thu, Jul 20, 2023 at 10:32=E2=80=AFAM Thomas Monjal= on <thomas@monjalon.net> w= rote:
+Cc the vm= xnet3 maintainer.

Please Jochen, do you have an idea what's wrong below?


20/07/2023 11:25, Igor de Paula:
> This is because it can't negotiate the IOMMU type with any port. >
> On Thu, Jul 20, 2023 at 5:08=E2=80=AFAM Thomas Monjalon <thomas@monjalon.net> = wrote:
>
> > Hello,
> >
> > The first error is "Cause: Error: number of ports must be ev= en"
> >
> >
> > 03/05/2023 18:13, Igor de Paula:
> > > I am running a VM inside a VMWARE server (vSphere).
> > > My goal it to set up DPDK with two HW ports, and set up a vi= rtio_user to
> > > interact with the kernel stack.
> > > In another app I have it working but instead of virtio_user = I am running
> > > KNI, it works in IOVA-PA mode.
> > > I am looking to replace the KNI.
> > >
> > > When I try to set up virtio_user port as in the doc:
> > >
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.htm= l#virtio-user-as-exception-path
> > > I get a error it can't run in PA mode.
> > >
> > >
> > > When I try to run as VA mode from a parameter, I get the fol= lowing
> > errors:
> > > EAL: lib.eal log level changed from info to debug
> > > EAL: Detected lcore 0 as core 0 on socket 0
> > > EAL: Detected lcore 1 as core 0 on socket 0
> > > EAL: Support maximum 128 logical core(s) by configuration. > > > EAL: Detected 2 lcore(s)
> > > EAL: Detected 1 NUMA nodes
> > > EAL: Checking presence of .so 'librte_eal.so.21.3' > > > EAL: Checking presence of .so 'librte_eal.so.21'
> > > EAL: Checking presence of .so 'librte_eal.so'
> > > EAL: Detected static linkage of DPDK
> > > EAL: Ask a virtual area of 0x7000 bytes
> > > EAL: Virtual area found at 0x100000000 (size =3D 0x7000)
> > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > > EAL: DPAA Bus not present. Skipping.
> > > EAL: VFIO PCI modules not loaded
> > > EAL: Selected IOVA mode 'VA'
> > > EAL: Probing VFIO support...
> > > EAL: IOMMU type 1 (Type 1) is supported
> > > EAL: IOMMU type 7 (sPAPR) is not supported
> > > EAL: IOMMU type 8 (No-IOMMU) is supported
> > > EAL: VFIO support initialized
> > > EAL: Ask a virtual area of 0x5b000 bytes
> > > EAL: Virtual area found at 0x100007000 (size =3D 0x5b000) > > > EAL: Setting up physically contiguous memory...
> > > EAL: Setting maximum number of open files to 1048576
> > > EAL: Detected memory type: socket_id:0 hugepage_sz:107374182= 4
> > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
> > hugepage_sz:1073741824
> > > EAL: Ask a virtual area of 0x2000 bytes
> > > EAL: Virtual area found at 0x100062000 (size =3D 0x2000)
> > > EAL: Memseg list allocated at socket 0, page size 0x100000kB=
> > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > EAL: Virtual area found at 0x140000000 (size =3D 0x200000000= 0)
> > > EAL: VA reserved for memseg list at 0x140000000, size 200000= 0000
> > > EAL: Ask a virtual area of 0x2000 bytes
> > > EAL: Virtual area found at 0x2140000000 (size =3D 0x2000) > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB=
> > > EAL: Ask a virtual area of 0x2000000000 bytes
> > > EAL: Virtual area found at 0x2180000000 (size =3D 0x20000000= 00)
> > > EAL: VA reserved for memseg list at 0x2180000000, size 20000= 00000
> > > EAL: TSC frequency is ~2350000 KHz
> > > EAL: Main lcore 0 is ready (tid=3D7f8ad790ec00;cpuset=3D[0])=
> > > EAL: lcore 1 is ready (tid=3D7f8ad6907400;cpuset=3D[1])
> > > EAL: Trying to obtain current memory policy.
> > > EAL: Setting policy MPOL_PREFERRED for socket 0
> > > EAL: Restoring previous memory policy: 0
> > > EAL: request: mp_malloc_sync
> > > EAL: Heap on socket 0 was expanded by 1024MB
> > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > EAL:=C2=A0 =C2=A0probe driver: 15ad:7b0 net_vmxnet3
> > > EAL:=C2=A0 =C2=A0Expecting 'PA' IOVA mode but curren= t mode is 'VA', not
> > initializing
> > > EAL: Requested device 0000:0b:00.0 cannot be used
> > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > EAL:=C2=A0 =C2=A0probe driver: 15ad:7b0 net_vmxnet3
> > > EAL:=C2=A0 =C2=A0Expecting 'PA' IOVA mode but curren= t mode is 'VA', not
> > initializing
> > > EAL: Requested device 0000:13:00.0 cannot be used
> > > EAL: Bus (pci) probe failed.
> > > EAL: lib.telemetry log level changed from disabled to warnin= g
> > > EAL: Error - exiting with code: 1
> > >=C2=A0 =C2=A0Cause: Error: number of ports must be even
> > > EAL: request: mp_malloc_sync
> > > EAL: Heap on socket 0 was shrunk by 1024MB
> > >
> > >
> > >
> > > For some reason the HW ports won't setup. From what I un= derstand
> > > net_vmxnet3 should work with VA mode.
> > > I enabled I/OMUU for the VM.
> > > The weird thing even when enabled, I still have the
> > > enable_unsafe_noiommu_mode flag on.
> > > And because it's on the this:
> > >
> > > dev_iova_mode =3D pci_device_iova_mode(dr, dev);
> > >
> > > return PA mode, and it fails.
> > >
> > > When I disable it by modifying
> > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I ge= t another
> > error.
> > > The error is that it doesn't find a suitable IOMMU type:=
> > > Just putting the relevant message:
> > >
> > >
> > > EAL: Heap on socket 0 was expanded by 1024MB
> > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> > > EAL:=C2=A0 =C2=A0probe driver: 15ad:7b0 net_vmxnet3
> > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such dev= ice)
> > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such devi= ce)
> > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such d= evice)
> > > EAL: 0000:0b:00.0 failed to select IOMMU type
> > > EAL: Requested device 0000:0b:00.0 cannot be used
> > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
> > > EAL:=C2=A0 =C2=A0probe driver: 15ad:7b0 net_vmxnet3
> > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such dev= ice)
> > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such devi= ce)
> > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such d= evice)
> > > EAL: 0000:13:00.0 failed to select IOMMU type
> > > EAL: Requested device 0000:13:00.0 cannot be used
> > >
> > >
> > > I can see where it fails:
> > >
> > >
> > > const struct vfio_iommu_type *
> > > vfio_set_iommu_type(int vfio_container_fd)
> > > {
> > > unsigned idx;
> > > for (idx =3D 0; idx < RTE_DIM(iommu_types); idx++) {
> > > const struct vfio_iommu_type *t =3D &iommu_types[idx]; > > >
> > > int ret =3D ioctl(vfio_container_fd, VFIO_SET_IOMMU,
> > > t->type_id);
> > > if (!ret) {
> > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n", > > > t->type_id, t->name);
> > > return t;
> > > }
> > > /* not an error, there may be more supported IOMMU types */<= br> > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, err= or "
> > > "%i (%s)\n", t->type_id, t->name, errno,
> > > strerror(errno));
> > > }
> > > /* if we didn't find a suitable IOMMU type, fail */
> > > return NULL;
> > > }
> > >
> > >
> > > The ioctl returns -1.
> > >
> > >
> > >
> > > I tried many things.
> > > Enabling/disabling IOMMU.
> > > Modifying the enable_unsafe_noiommu_mode flag.
> > > Running in PA mode.
> > > Any help will be much appreciated.
> > >
> >
> >
> >
> >
> >
> >
>





--000000000000dbbd180600e7ddfa--