From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 136A242EBF for ; Thu, 20 Jul 2023 11:25:25 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C3ADD40E2D; Thu, 20 Jul 2023 11:25:24 +0200 (CEST) Received: from mail-yb1-f172.google.com (mail-yb1-f172.google.com [209.85.219.172]) by mails.dpdk.org (Postfix) with ESMTP id A3D2440DF5 for ; Thu, 20 Jul 2023 11:25:23 +0200 (CEST) Received: by mail-yb1-f172.google.com with SMTP id 3f1490d57ef6-cb4de3bd997so2576832276.1 for ; Thu, 20 Jul 2023 02:25:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689845123; x=1690449923; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=MfVESHOQiQWoLLSSGiRiw5wXRYmRpVvlIBwUx76Kydo=; b=Gw+iDrz9Ccp7BDuH1ZbjKJHU47gQZ7Wot6c6Rv84fbZAxoSsJ94MZn1yzZ6VKFx1k1 pHnpKILy9Rv2PRqqh7VXIoPKexHBNkDQXSg+06SsyqXO5nwE+L7u0lhSi1K+2xLt/So5 TRrz7HfM0mE2Qr0IoMMZmxb/HcoVx1FX9w4t+2jZjuu2/hbblG3IbvsyBd7FBw0gNMRB 4cHQLw9NkHvQijOtCQBga3CMG0fiKj2mOrKsI2VVBsQnnSTayhEj0WzbGVzwlDMBUeJq sdQjvsb3dS80xeoK6DuvWT7vAUc1ugQbtLaahtD8TTLeY+in77arvVdT6JxSH8uYwUWC 5b1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689845123; x=1690449923; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=MfVESHOQiQWoLLSSGiRiw5wXRYmRpVvlIBwUx76Kydo=; b=R8+1ajJ9LkPycwiM7Ql57w6aGLA+YERZj1EAer/A4q6gIz9VCAmdQE5WbfdGlLa1jx cUZeTV0gAHrIMMrqQcBHm+FJfF2nRwZXdcrWckYw7VZa7UYInMwr0qrnS3O8hzIiSFyu H6sHPL2Aak2VT4YLKFaJZ5G+TOu/LsFBKXy7hyotr2ItIv5IAjtk78mE98RJbY7xjc7O esypa8EDIP6hxqudeEJoZBYOXZLCam7DpEXpcoCbOCJDWmMI5AvBw7IH8ywTqQW5qmPP seHcqnSqiFb48Yf9gZR0Bo8jswQFndqiSh13WbiP3Espn6LBlKb1cfYV4xAl/wMZp9uK lrTQ== X-Gm-Message-State: ABy/qLbQ0/gqR+cP6WjAZUMtD7RTnQ0E7EPtQy3OfWKMG4W5k4b4W2Jy /PtifnvuAbE92Awy1XwTJdrs0UZz4tnPiYyLa7rv0Kog X-Google-Smtp-Source: APBJJlFTNgZpjoAT35bq9sAuXU7z1JkzSyMDUd9nVNwkvS4lgKVncrFgkzVvYaqPW6FxWNxiexktFsBicW8gpCwua3c= X-Received: by 2002:a25:e702:0:b0:ce9:f5f6:b04a with SMTP id e2-20020a25e702000000b00ce9f5f6b04amr5190169ybh.4.1689845122805; Thu, 20 Jul 2023 02:25:22 -0700 (PDT) MIME-Version: 1.0 References: <10081895.2WqB4rESCP@thomas> In-Reply-To: <10081895.2WqB4rESCP@thomas> From: Igor de Paula Date: Thu, 20 Jul 2023 10:25:11 +0100 Message-ID: Subject: Re: help To: Thomas Monjalon Cc: users@dpdk.org Content-Type: multipart/alternative; boundary="0000000000004535aa0600e7ba3d" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --0000000000004535aa0600e7ba3d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable This is because it can't negotiate the IOMMU type with any port. On Thu, Jul 20, 2023 at 5:08=E2=80=AFAM Thomas Monjalon wrote: > Hello, > > The first error is "Cause: Error: number of ports must be even" > > > 03/05/2023 18:13, Igor de Paula: > > I am running a VM inside a VMWARE server (vSphere). > > My goal it to set up DPDK with two HW ports, and set up a virtio_user t= o > > interact with the kernel stack. > > In another app I have it working but instead of virtio_user I am runnin= g > > KNI, it works in IOVA-PA mode. > > I am looking to replace the KNI. > > > > When I try to set up virtio_user port as in the doc: > > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virt= io-user-as-exception-path > > I get a error it can't run in PA mode. > > > > > > When I try to run as VA mode from a parameter, I get the following > errors: > > EAL: lib.eal log level changed from info to debug > > EAL: Detected lcore 0 as core 0 on socket 0 > > EAL: Detected lcore 1 as core 0 on socket 0 > > EAL: Support maximum 128 logical core(s) by configuration. > > EAL: Detected 2 lcore(s) > > EAL: Detected 1 NUMA nodes > > EAL: Checking presence of .so 'librte_eal.so.21.3' > > EAL: Checking presence of .so 'librte_eal.so.21' > > EAL: Checking presence of .so 'librte_eal.so' > > EAL: Detected static linkage of DPDK > > EAL: Ask a virtual area of 0x7000 bytes > > EAL: Virtual area found at 0x100000000 (size =3D 0x7000) > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > > EAL: DPAA Bus not present. Skipping. > > EAL: VFIO PCI modules not loaded > > EAL: Selected IOVA mode 'VA' > > EAL: Probing VFIO support... > > EAL: IOMMU type 1 (Type 1) is supported > > EAL: IOMMU type 7 (sPAPR) is not supported > > EAL: IOMMU type 8 (No-IOMMU) is supported > > EAL: VFIO support initialized > > EAL: Ask a virtual area of 0x5b000 bytes > > EAL: Virtual area found at 0x100007000 (size =3D 0x5b000) > > EAL: Setting up physically contiguous memory... > > EAL: Setting maximum number of open files to 1048576 > > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824 > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0 > hugepage_sz:1073741824 > > EAL: Ask a virtual area of 0x2000 bytes > > EAL: Virtual area found at 0x100062000 (size =3D 0x2000) > > EAL: Memseg list allocated at socket 0, page size 0x100000kB > > EAL: Ask a virtual area of 0x2000000000 bytes > > EAL: Virtual area found at 0x140000000 (size =3D 0x2000000000) > > EAL: VA reserved for memseg list at 0x140000000, size 2000000000 > > EAL: Ask a virtual area of 0x2000 bytes > > EAL: Virtual area found at 0x2140000000 (size =3D 0x2000) > > EAL: Memseg list allocated at socket 0, page size 0x100000kB > > EAL: Ask a virtual area of 0x2000000000 bytes > > EAL: Virtual area found at 0x2180000000 (size =3D 0x2000000000) > > EAL: VA reserved for memseg list at 0x2180000000, size 2000000000 > > EAL: TSC frequency is ~2350000 KHz > > EAL: Main lcore 0 is ready (tid=3D7f8ad790ec00;cpuset=3D[0]) > > EAL: lcore 1 is ready (tid=3D7f8ad6907400;cpuset=3D[1]) > > EAL: Trying to obtain current memory policy. > > EAL: Setting policy MPOL_PREFERRED for socket 0 > > EAL: Restoring previous memory policy: 0 > > EAL: request: mp_malloc_sync > > EAL: Heap on socket 0 was expanded by 1024MB > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not > initializing > > EAL: Requested device 0000:0b:00.0 cannot be used > > EAL: PCI device 0000:13:00.0 on NUMA socket -1 > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not > initializing > > EAL: Requested device 0000:13:00.0 cannot be used > > EAL: Bus (pci) probe failed. > > EAL: lib.telemetry log level changed from disabled to warning > > EAL: Error - exiting with code: 1 > > Cause: Error: number of ports must be even > > EAL: request: mp_malloc_sync > > EAL: Heap on socket 0 was shrunk by 1024MB > > > > > > > > For some reason the HW ports won't setup. From what I understand > > net_vmxnet3 should work with VA mode. > > I enabled I/OMUU for the VM. > > The weird thing even when enabled, I still have the > > enable_unsafe_noiommu_mode flag on. > > And because it's on the this: > > > > dev_iova_mode =3D pci_device_iova_mode(dr, dev); > > > > return PA mode, and it fails. > > > > When I disable it by modifying > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get another > error. > > The error is that it doesn't find a suitable IOMMU type: > > Just putting the relevant message: > > > > > > EAL: Heap on socket 0 was expanded by 1024MB > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device) > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device) > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device) > > EAL: 0000:0b:00.0 failed to select IOMMU type > > EAL: Requested device 0000:0b:00.0 cannot be used > > EAL: PCI device 0000:13:00.0 on NUMA socket -1 > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device) > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device) > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device) > > EAL: 0000:13:00.0 failed to select IOMMU type > > EAL: Requested device 0000:13:00.0 cannot be used > > > > > > I can see where it fails: > > > > > > const struct vfio_iommu_type * > > vfio_set_iommu_type(int vfio_container_fd) > > { > > unsigned idx; > > for (idx =3D 0; idx < RTE_DIM(iommu_types); idx++) { > > const struct vfio_iommu_type *t =3D &iommu_types[idx]; > > > > int ret =3D ioctl(vfio_container_fd, VFIO_SET_IOMMU, > > t->type_id); > > if (!ret) { > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n", > > t->type_id, t->name); > > return t; > > } > > /* not an error, there may be more supported IOMMU types */ > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error " > > "%i (%s)\n", t->type_id, t->name, errno, > > strerror(errno)); > > } > > /* if we didn't find a suitable IOMMU type, fail */ > > return NULL; > > } > > > > > > The ioctl returns -1. > > > > > > > > I tried many things. > > Enabling/disabling IOMMU. > > Modifying the enable_unsafe_noiommu_mode flag. > > Running in PA mode. > > Any help will be much appreciated. > > > > > > > > --0000000000004535aa0600e7ba3d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
This is because it can't negotiate the IOMMU type with= any port.

On Thu, Jul 20, 2023 at 5:08=E2=80=AFAM Thomas Monjalon <thomas@monjalon.net> wrote:
Hello,

The first error is "Cause: Error: number of ports must be even"

03/05/2023 18:13, Igor de Paula:
> I am running a VM inside a VMWARE server (vSphere).
> My goal it to set up DPDK with two HW ports, and set up a virtio_user = to
> interact with the kernel stack.
> In another app I have it working but instead of virtio_user I am runni= ng
> KNI, it works in IOVA-PA mode.
> I am looking to replace the KNI.
>
> When I try to set up virtio_user port as in the doc:
> https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#vir= tio-user-as-exception-path
> I get a error it can't run in PA mode.
>
>
> When I try to run as VA mode from a parameter, I get the following err= ors:
> EAL: lib.eal log level changed from info to debug
> EAL: Detected lcore 0 as core 0 on socket 0
> EAL: Detected lcore 1 as core 0 on socket 0
> EAL: Support maximum 128 logical core(s) by configuration.
> EAL: Detected 2 lcore(s)
> EAL: Detected 1 NUMA nodes
> EAL: Checking presence of .so 'librte_eal.so.21.3'
> EAL: Checking presence of .so 'librte_eal.so.21'
> EAL: Checking presence of .so 'librte_eal.so'
> EAL: Detected static linkage of DPDK
> EAL: Ask a virtual area of 0x7000 bytes
> EAL: Virtual area found at 0x100000000 (size =3D 0x7000)
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: DPAA Bus not present. Skipping.
> EAL: VFIO PCI modules not loaded
> EAL: Selected IOVA mode 'VA'
> EAL: Probing VFIO support...
> EAL: IOMMU type 1 (Type 1) is supported
> EAL: IOMMU type 7 (sPAPR) is not supported
> EAL: IOMMU type 8 (No-IOMMU) is supported
> EAL: VFIO support initialized
> EAL: Ask a virtual area of 0x5b000 bytes
> EAL: Virtual area found at 0x100007000 (size =3D 0x5b000)
> EAL: Setting up physically contiguous memory...
> EAL: Setting maximum number of open files to 1048576
> EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
> EAL: Creating 2 segment lists: n_segs:128 socket_id:0 hugepage_sz:1073= 741824
> EAL: Ask a virtual area of 0x2000 bytes
> EAL: Virtual area found at 0x100062000 (size =3D 0x2000)
> EAL: Memseg list allocated at socket 0, page size 0x100000kB
> EAL: Ask a virtual area of 0x2000000000 bytes
> EAL: Virtual area found at 0x140000000 (size =3D 0x2000000000)
> EAL: VA reserved for memseg list at 0x140000000, size 2000000000
> EAL: Ask a virtual area of 0x2000 bytes
> EAL: Virtual area found at 0x2140000000 (size =3D 0x2000)
> EAL: Memseg list allocated at socket 0, page size 0x100000kB
> EAL: Ask a virtual area of 0x2000000000 bytes
> EAL: Virtual area found at 0x2180000000 (size =3D 0x2000000000)
> EAL: VA reserved for memseg list at 0x2180000000, size 2000000000
> EAL: TSC frequency is ~2350000 KHz
> EAL: Main lcore 0 is ready (tid=3D7f8ad790ec00;cpuset=3D[0])
> EAL: lcore 1 is ready (tid=3D7f8ad6907400;cpuset=3D[1])
> EAL: Trying to obtain current memory policy.
> EAL: Setting policy MPOL_PREFERRED for socket 0
> EAL: Restoring previous memory policy: 0
> EAL: request: mp_malloc_sync
> EAL: Heap on socket 0 was expanded by 1024MB
> EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> EAL:=C2=A0 =C2=A0probe driver: 15ad:7b0 net_vmxnet3
> EAL:=C2=A0 =C2=A0Expecting 'PA' IOVA mode but current mode is = 'VA', not initializing
> EAL: Requested device 0000:0b:00.0 cannot be used
> EAL: PCI device 0000:13:00.0 on NUMA socket -1
> EAL:=C2=A0 =C2=A0probe driver: 15ad:7b0 net_vmxnet3
> EAL:=C2=A0 =C2=A0Expecting 'PA' IOVA mode but current mode is = 'VA', not initializing
> EAL: Requested device 0000:13:00.0 cannot be used
> EAL: Bus (pci) probe failed.
> EAL: lib.telemetry log level changed from disabled to warning
> EAL: Error - exiting with code: 1
>=C2=A0 =C2=A0Cause: Error: number of ports must be even
> EAL: request: mp_malloc_sync
> EAL: Heap on socket 0 was shrunk by 1024MB
>
>
>
> For some reason the HW ports won't setup. From what I understand > net_vmxnet3 should work with VA mode.
> I enabled I/OMUU for the VM.
> The weird thing even when enabled, I still have the
> enable_unsafe_noiommu_mode flag on.
> And because it's on the this:
>
> dev_iova_mode =3D pci_device_iova_mode(dr, dev);
>
> return PA mode, and it fails.
>
> When I disable it by modifying
> /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get another = error.
> The error is that it doesn't find a suitable IOMMU type:
> Just putting the relevant message:
>
>
> EAL: Heap on socket 0 was expanded by 1024MB
> EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> EAL:=C2=A0 =C2=A0probe driver: 15ad:7b0 net_vmxnet3
> EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
> EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
> EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device)
> EAL: 0000:0b:00.0 failed to select IOMMU type
> EAL: Requested device 0000:0b:00.0 cannot be used
> EAL: PCI device 0000:13:00.0 on NUMA socket -1
> EAL:=C2=A0 =C2=A0probe driver: 15ad:7b0 net_vmxnet3
> EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
> EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
> EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device)
> EAL: 0000:13:00.0 failed to select IOMMU type
> EAL: Requested device 0000:13:00.0 cannot be used
>
>
> I can see where it fails:
>
>
> const struct vfio_iommu_type *
> vfio_set_iommu_type(int vfio_container_fd)
> {
> unsigned idx;
> for (idx =3D 0; idx < RTE_DIM(iommu_types); idx++) {
> const struct vfio_iommu_type *t =3D &iommu_types[idx];
>
> int ret =3D ioctl(vfio_container_fd, VFIO_SET_IOMMU,
> t->type_id);
> if (!ret) {
> RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
> t->type_id, t->name);
> return t;
> }
> /* not an error, there may be more supported IOMMU types */
> RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error "<= br> > "%i (%s)\n", t->type_id, t->name, errno,
> strerror(errno));
> }
> /* if we didn't find a suitable IOMMU type, fail */
> return NULL;
> }
>
>
> The ioctl returns -1.
>
>
>
> I tried many things.
> Enabling/disabling IOMMU.
> Modifying the enable_unsafe_noiommu_mode flag.
> Running in PA mode.
> Any help will be much appreciated.
>





--0000000000004535aa0600e7ba3d--