[dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[Takeshi Yoshimura]

Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI
drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC)
because of the inconsistency of user's request and the result of
device capability for IOVA mode. I updated the code to force IOVA as
PA in ppc as before because current ppc driver does not support VA
mode.

Theoretically, ppc can support VA mode, but I suspect that ppc with
VA mode may have performance issues to create a big DMA window
(VA often uses higher addresses than PA). So, I didn't change the
code to check device capability in ppc.

Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>
---
 lib/librte_eal/linux/eal/eal.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c
index 946222ccd..db2dec922 100644
--- a/lib/librte_eal/linux/eal/eal.c
+++ b/lib/librte_eal/linux/eal/eal.c
@@ -1121,6 +1121,12 @@ rte_eal_init(int argc, char **argv)
 				RTE_LOG(DEBUG, EAL, "KNI can not work since physical addresses are unavailable\n");
 			}
 		}
+#endif
+#ifdef RTE_ARCH_PPC_64
+		if (iova_mode == RTE_IOVA_VA) {
+			iova_mode = RTE_IOVA_PA;
+			RTE_LOG(WARNING, EAL, "Forcing IOVA as 'PA' because PPC uses PA mode.\n");
+		}
 #endif
 		rte_eal_get_configuration()->iova_mode = iova_mode;
 	} else {
-- 
2.17.1

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[Burakov, Anatoly]

On 31-Jul-19 4:35 AM, Takeshi Yoshimura wrote:
> Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI
> drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC)
> because of the inconsistency of user's request and the result of
> device capability for IOVA mode. I updated the code to force IOVA as
> PA in ppc as before because current ppc driver does not support VA
> mode.
> 
> Theoretically, ppc can support VA mode, but I suspect that ppc with
> VA mode may have performance issues to create a big DMA window
> (VA often uses higher addresses than PA). So, I didn't change the
> code to check device capability in ppc.
> 
> Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>
> ---
>   lib/librte_eal/linux/eal/eal.c | 6 ++++++
>   1 file changed, 6 insertions(+)
> 
> diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c
> index 946222ccd..db2dec922 100644
> --- a/lib/librte_eal/linux/eal/eal.c
> +++ b/lib/librte_eal/linux/eal/eal.c
> @@ -1121,6 +1121,12 @@ rte_eal_init(int argc, char **argv)
>   				RTE_LOG(DEBUG, EAL, "KNI can not work since physical addresses are unavailable\n");
>   			}
>   		}
> +#endif
> +#ifdef RTE_ARCH_PPC_64
> +		if (iova_mode == RTE_IOVA_VA) {
> +			iova_mode = RTE_IOVA_PA;
> +			RTE_LOG(WARNING, EAL, "Forcing IOVA as 'PA' because PPC uses PA mode.\n");
> +		}
>   #endif

I would have preferred a space between #endif and a new #ifdef, but 
otherwise

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[David Marchand]

On Wed, Jul 31, 2019 at 5:36 AM Takeshi Yoshimura <tyos@jp.ibm.com> wrote:
>
> Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI
> drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC)
> because of the inconsistency of user's request and the result of
> device capability for IOVA mode. I updated the code to force IOVA as
> PA in ppc as before because current ppc driver does not support VA
> mode.

What is the "ppc driver" you are referring to?
Do you mean the vfio implementation for ppc?

>
> Theoretically, ppc can support VA mode, but I suspect that ppc with
> VA mode may have performance issues to create a big DMA window
> (VA often uses higher addresses than PA). So, I didn't change the
> code to check device capability in ppc.

Sorry, I am confused.
Is VA mode not working at all? or is it a performance issue?


-- 
David Marchand

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[Takeshi T Yoshimura]

-----David Marchand <david.marchand@redhat.com> wrote: -----

>To: Takeshi Yoshimura <tyos@jp.ibm.com>
>From: David Marchand <david.marchand@redhat.com>
>Date: 07/31/2019 06:29PM
>Cc: dev <dev@dpdk.org>
>Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in
>ppc
>
>On Wed, Jul 31, 2019 at 5:36 AM Takeshi Yoshimura <tyos@jp.ibm.com>
>wrote:
>>
>> Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI
>> drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC)
>> because of the inconsistency of user's request and the result of
>> device capability for IOVA mode. I updated the code to force IOVA
>as
>> PA in ppc as before because current ppc driver does not support VA
>> mode.
>
>What is the "ppc driver" you are referring to?
>Do you mean the vfio implementation for ppc?
>
>>
>> Theoretically, ppc can support VA mode, but I suspect that ppc with
>> VA mode may have performance issues to create a big DMA window
>> (VA often uses higher addresses than PA). So, I didn't change the
>> code to check device capability in ppc.
>
>Sorry, I am confused.
>Is VA mode not working at all? or is it a performance issue?
>
>
>-- 
>David Marchand
>
>

Current code does not implement VA mode for ppc. So, yes. VA mode
is not working under current dpdk on ppc.

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[Takeshi T Yoshimura]

-----David Marchand <david.marchand@redhat.com> wrote: -----

>To: Takeshi Yoshimura <tyos@jp.ibm.com>
>From: David Marchand <david.marchand@redhat.com>
>Date: 07/31/2019 06:29PM
>Cc: dev <dev@dpdk.org>
>Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in
>ppc
>
>On Wed, Jul 31, 2019 at 5:36 AM Takeshi Yoshimura <tyos@jp.ibm.com>
>wrote:
>>
>> Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI
>> drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC)
>> because of the inconsistency of user's request and the result of
>> device capability for IOVA mode. I updated the code to force IOVA
>as
>> PA in ppc as before because current ppc driver does not support VA
>> mode.
>
>What is the "ppc driver" you are referring to?
>Do you mean the vfio implementation for ppc?
>
>>
>> Theoretically, ppc can support VA mode, but I suspect that ppc with
>> VA mode may have performance issues to create a big DMA window
>> (VA often uses higher addresses than PA). So, I didn't change the
>> code to check device capability in ppc.
>
>Sorry, I am confused.
>Is VA mode not working at all? or is it a performance issue?
>
>
>-- 
>David Marchand
>
>

Ah, I overlooked your first question. The "ppc driver" is in
drivers/bus/pci/linux/pci.c. In pci.c, pci_one_device_iommu_support_va()
always returns false in ppc. so, the driver does not allow VA mode in ppc.

This return value is passed to rte_pci_probe_one_driver()
in drivers/bus/pci/pci_common.c. And the error log appeared like:

EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[David Marchand]

On Wed, Jul 31, 2019 at 12:44 PM Takeshi T Yoshimura <TYOS@jp.ibm.com> wrote:
> Ah, I overlooked your first question. The "ppc driver" is in
> drivers/bus/pci/linux/pci.c. In pci.c, pci_one_device_iommu_support_va()
> always returns false in ppc. so, the driver does not allow VA mode in ppc.
>
> This return value is passed to rte_pci_probe_one_driver()
> in drivers/bus/pci/pci_common.c. And the error log appeared like:
>
> EAL: PCI device 0000:01:00.0 on NUMA socket 0
> EAL:   probe driver: 144d:a822 spdk_nvme
> EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
>

In which situation does this error happen?
When probing the device at eal init time? or later when hotplugging?
Can you give the full eal logs with --log-level lib.*:debug?


-- 
David Marchand

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[David Christensen]

On 7/31/19 3:58 AM, David Marchand wrote:
> --log-level lib.*:debug

Here's the output from my Power 9 system running 19.08-RC2 (I'm not 
using VFIO since I have an MLX5 adapter).

sudo -E LD_LIBRARY_PATH=/home/davec/src/dpdk/build/lib 
/home/davec/src/dpdk/build/app/testpmd --log-level lib.\*:debug -w 
0000:01:00.0 -w 0000:01:00.1 -l 4,8-79 -n 4  -- --rxq=32 --rxd=1024 
--txq=32 --txd=4096 --nb-cores=64 -i -a --numa --forward-mode=rxonly
EAL: Detected lcore 0 as core 0 on socket 0
<snip> ...
EAL: Detected lcore 159 as core 2140 on socket 8
EAL: Support maximum 1536 logical core(s) by configuration.
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or 
directory)
EAL: VFIO PCI modules not loaded
EAL: DPAA Bus not present. Skipping.
EAL: Bus pci wants IOVA as 'DC'
EAL: Bus dpaa_bus wants IOVA as 'DC'
EAL: Bus fslmc wants IOVA as 'DC'
EAL: Buses did not request a specific IOVA mode.
EAL: IOMMU is available, selecting IOVA as VA mode.
EAL: Module /sys/module/rte_kni not found! error 2 (No such file or 
directory)
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
EAL: VFIO modules not loaded, skipping VFIO support...
EAL: Ask a virtual area of 0x30000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x30000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 4096
EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
EAL: Detected memory type: socket_id:8 hugepage_sz:1073741824
EAL: Creating 4 segment lists: n_segs:32 socket_id:0 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x100030000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x140000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x940000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x980000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x1180000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x11c0000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x19c0000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x1a00000000 (size = 0x800000000)
EAL: Creating 4 segment lists: n_segs:32 socket_id:8 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x2200000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x2240000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x2a40000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x2a80000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x3280000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x32c0000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x3ac0000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x3b00000000 (size = 0x800000000)
EAL: TSC frequency is ~510000 KHz
EAL: Master lcore 4 is ready (tid=7fffab066c90;cpuset=[4])
EAL: lcore 9 is ready (tid=7fffa917dc90;cpuset=[9])
<snip> ...
EAL: lcore 74 is ready (tid=7ffea9fbdc90;cpuset=[74])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 1024MB
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 15b3:1019 net_mlx5
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0000:01:00.0 cannot be used
EAL: PCI device 0000:01:00.1 on NUMA socket 0
EAL:   probe driver: 15b3:1019 net_mlx5
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0000:01:00.1 cannot be used
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
testpmd: No probed ethernet devices
Interactive-mode selected
Auto-start selected
Set rxonly packet forwarding mode
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=723456, 
size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 2048MB
Done
Start automatic packet forwarding
rxonly packet forwarding - ports=0 - cores=0 - streams=0 - NUMA support 
enabled, MP allocation mode: native

   rxonly packet forwarding packets/burst=32
   nb forwarding cores=64 - nb forwarding ports=0
testpmd>

Dave

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[David Marchand]

On Wed, Jul 31, 2019 at 9:23 PM David Christensen
<drc@linux.vnet.ibm.com> wrote:
>
> On 7/31/19 3:58 AM, David Marchand wrote:
> > --log-level lib.*:debug
>
> Here's the output from my Power 9 system running 19.08-RC2 (I'm not
> using VFIO since I have an MLX5 adapter).
>
> sudo -E LD_LIBRARY_PATH=/home/davec/src/dpdk/build/lib
> /home/davec/src/dpdk/build/app/testpmd --log-level lib.\*:debug -w
> 0000:01:00.0 -w 0000:01:00.1 -l 4,8-79 -n 4  -- --rxq=32 --rxd=1024
> --txq=32 --txd=4096 --nb-cores=64 -i -a --numa --forward-mode=rxonly
> EAL: Detected lcore 0 as core 0 on socket 0
> <snip> ...
> EAL: Detected lcore 159 as core 2140 on socket 8
> EAL: Support maximum 1536 logical core(s) by configuration.
> EAL: Detected 160 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or
> directory)
> EAL: VFIO PCI modules not loaded
> EAL: DPAA Bus not present. Skipping.
> EAL: Bus pci wants IOVA as 'DC'

Ok, I think I understand now.

Can you try this hunk ?

diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index 9794552..6d3b553 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -620,9 +620,6 @@ rte_pci_get_iommu_class(void)
        FOREACH_DEVICE_ON_PCIBUS(dev) {
                if (pci_ignore_device(dev))
                        continue;
-               if (dev->kdrv == RTE_KDRV_UNKNOWN ||
-                   dev->kdrv == RTE_KDRV_NONE)
-                       continue;
                FOREACH_DRIVER_ON_PCIBUS(drv) {
                        enum rte_iova_mode dev_iova_mode;




-- 
David Marchand

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[David Christensen]

>> Here's the output from my Power 9 system running 19.08-RC2 (I'm not
>> using VFIO since I have an MLX5 adapter).
>>
>> sudo -E LD_LIBRARY_PATH=/home/davec/src/dpdk/build/lib
>> /home/davec/src/dpdk/build/app/testpmd --log-level lib.\*:debug -w
>> 0000:01:00.0 -w 0000:01:00.1 -l 4,8-79 -n 4  -- --rxq=32 --rxd=1024
>> --txq=32 --txd=4096 --nb-cores=64 -i -a --numa --forward-mode=rxonly
>> EAL: Detected lcore 0 as core 0 on socket 0
>> <snip> ...
>> EAL: Detected lcore 159 as core 2140 on socket 8
>> EAL: Support maximum 1536 logical core(s) by configuration.
>> EAL: Detected 160 lcore(s)
>> EAL: Detected 2 NUMA nodes
>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or
>> directory)
>> EAL: VFIO PCI modules not loaded
>> EAL: DPAA Bus not present. Skipping.
>> EAL: Bus pci wants IOVA as 'DC'
> 
> Ok, I think I understand now.
> 
> Can you try this hunk ?
> 

IOVA mode is now reported as PA.

MLX5 PMD fails to load correctly, but that also occurs without the patch 
and when the EAL command-line option "--iova-mode=pa" is used.  This 
worked previously in RC1 and may be a separate issue.

sudo -E LD_LIBRARY_PATH=/home/davec/src/dpdk/build/lib 
/home/davec/src/dpdk/build/app/testpmd --log-level="lib.*:debug" -w 
0000:01:00.0 -w 0000:01:00.1 -l 4,8-79 -n 4  -- --rxq=32 --rxd=1024 
--txq=32 --txd=4096 --nb-cores=64 -i -a --numa --forward-mode=rxonly
EAL: Detected lcore 0 as core 0 on socket 0
<snip> ...
EAL: Detected lcore 159 as core 2140 on socket 8
EAL: Support maximum 1536 logical core(s) by configuration.
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or 
directory)
EAL: VFIO PCI modules not loaded
EAL: DPAA Bus not present. Skipping.
EAL: PCI driver net_mlx5 for device 0000:01:00.0 wants IOVA as 'PA'
EAL: PCI driver net_mlx5 for device 0000:01:00.1 wants IOVA as 'PA'
EAL: Bus pci wants IOVA as 'PA'
EAL: Bus dpaa_bus wants IOVA as 'DC'
EAL: Bus fslmc wants IOVA as 'DC'
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
EAL: VFIO modules not loaded, skipping VFIO support...
EAL: Ask a virtual area of 0x30000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x30000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 4096
EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
EAL: Detected memory type: socket_id:8 hugepage_sz:1073741824
EAL: Creating 4 segment lists: n_segs:32 socket_id:0 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x100030000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x140000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x940000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x980000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x1180000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x11c0000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x19c0000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x1a00000000 (size = 0x800000000)
EAL: Creating 4 segment lists: n_segs:32 socket_id:8 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x2200000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x2240000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x2a40000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x2a80000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x3280000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x32c0000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x3ac0000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x3b00000000 (size = 0x800000000)
EAL: TSC frequency is ~510000 KHz
EAL: Master lcore 4 is ready (tid=7fffabb46c90;cpuset=[4])
EAL: lcore 9 is ready (tid=7fffa9c5dc90;cpuset=[9])
<snip> ...
EAL: lcore 73 is ready (tid=7ffea77edc90;cpuset=[73])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 1024MB
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 15b3:1019 net_mlx5
EAL: Mem event callback 'MLX5_MEM_EVENT_CB:(nil)' registered
net_mlx5: mlx5.c:1683: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel 
offloading disabled due to old OFED/rdma-core version or firmware 
configuration
EAL: PCI device 0000:01:00.1 on NUMA socket 0
EAL:   probe driver: 15b3:1019 net_mlx5
net_mlx5: mlx5.c:1683: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel 
offloading disabled due to old OFED/rdma-core version or firmware 
configuration
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
testpmd: /home/davec/src/dpdk/drivers/net/mlx5/mlx5_ethdev.c:520: 
mlx5_set_txlimit_params: Assertion `config->txq_inline_min >= 0' failed.
Aborted


Dave

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[David Marchand]

On Wed, Jul 31, 2019 at 11:20 PM David Christensen
<drc@linux.vnet.ibm.com> wrote:
>
> >> Here's the output from my Power 9 system running 19.08-RC2 (I'm not
> >> using VFIO since I have an MLX5 adapter).
> >>
> >> sudo -E LD_LIBRARY_PATH=/home/davec/src/dpdk/build/lib
> >> /home/davec/src/dpdk/build/app/testpmd --log-level lib.\*:debug -w
> >> 0000:01:00.0 -w 0000:01:00.1 -l 4,8-79 -n 4  -- --rxq=32 --rxd=1024
> >> --txq=32 --txd=4096 --nb-cores=64 -i -a --numa --forward-mode=rxonly
> >> EAL: Detected lcore 0 as core 0 on socket 0
> >> <snip> ...
> >> EAL: Detected lcore 159 as core 2140 on socket 8
> >> EAL: Support maximum 1536 logical core(s) by configuration.
> >> EAL: Detected 160 lcore(s)
> >> EAL: Detected 2 NUMA nodes
> >> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> >> EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or
> >> directory)
> >> EAL: VFIO PCI modules not loaded
> >> EAL: DPAA Bus not present. Skipping.
> >> EAL: Bus pci wants IOVA as 'DC'
> >
> > Ok, I think I understand now.
> >
> > Can you try this hunk ?
> >
>
> IOVA mode is now reported as PA.

Ok thanks.
Still trying to understand Takeshi Yoshimura issue which looks different.


> MLX5 PMD fails to load correctly, but that also occurs without the patch
> and when the EAL command-line option "--iova-mode=pa" is used.  This
> worked previously in RC1 and may be a separate issue.

Saw your patch on mlx5, yes separate issue.


-- 
David Marchand

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[David Marchand]

On Wed, Jul 31, 2019 at 12:44 PM Takeshi T Yoshimura <TYOS@jp.ibm.com> wrote:
>
> -----David Marchand <david.marchand@redhat.com> wrote: -----
>
> >To: Takeshi Yoshimura <tyos@jp.ibm.com>
> >From: David Marchand <david.marchand@redhat.com>
> >Date: 07/31/2019 06:29PM
> >Cc: dev <dev@dpdk.org>
> >Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in
> >ppc
> >
> >On Wed, Jul 31, 2019 at 5:36 AM Takeshi Yoshimura <tyos@jp.ibm.com>
> >wrote:
> >>
> >> Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI
> >> drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC)
> >> because of the inconsistency of user's request and the result of
> >> device capability for IOVA mode. I updated the code to force IOVA
> >as
> >> PA in ppc as before because current ppc driver does not support VA
> >> mode.
> >
> >What is the "ppc driver" you are referring to?
> >Do you mean the vfio implementation for ppc?
> >
> >>
> >> Theoretically, ppc can support VA mode, but I suspect that ppc with
> >> VA mode may have performance issues to create a big DMA window
> >> (VA often uses higher addresses than PA). So, I didn't change the
> >> code to check device capability in ppc.
> >
> >Sorry, I am confused.
> >Is VA mode not working at all? or is it a performance issue?
> >
> >
> >--
> >David Marchand
> >
> >
>
> Ah, I overlooked your first question. The "ppc driver" is in
> drivers/bus/pci/linux/pci.c. In pci.c, pci_one_device_iommu_support_va()
> always returns false in ppc. so, the driver does not allow VA mode in ppc.
>
> This return value is passed to rte_pci_probe_one_driver()
> in drivers/bus/pci/pci_common.c. And the error log appeared like:
>
> EAL: PCI device 0000:01:00.0 on NUMA socket 0
> EAL:   probe driver: 144d:a822 spdk_nvme
> EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing

Can you confirm on which kernel driver this device was bound ?
I'd like to be sure this is the same issue than David.


-- 
David Marchand

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[Takeshi T Yoshimura]

-----David Marchand <david.marchand@redhat.com> wrote: -----

>To: Takeshi T Yoshimura <TYOS@jp.ibm.com>
>From: David Marchand <david.marchand@redhat.com>
>Date: 08/01/2019 04:45AM
>Cc: dev <dev@dpdk.org>, David Christensen <drc@linux.vnet.ibm.com>
>Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in
>ppc
>
>On Wed, Jul 31, 2019 at 12:44 PM Takeshi T Yoshimura
><TYOS@jp.ibm.com> wrote:
>>
>> -----David Marchand <david.marchand@redhat.com> wrote: -----
>>
>> >To: Takeshi Yoshimura <tyos@jp.ibm.com>
>> >From: David Marchand <david.marchand@redhat.com>
>> >Date: 07/31/2019 06:29PM
>> >Cc: dev <dev@dpdk.org>
>> >Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA
>in
>> >ppc
>> >
>> >On Wed, Jul 31, 2019 at 5:36 AM Takeshi Yoshimura
><tyos@jp.ibm.com>
>> >wrote:
>> >>
>> >> Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI
>> >> drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC)
>> >> because of the inconsistency of user's request and the result of
>> >> device capability for IOVA mode. I updated the code to force
>IOVA
>> >as
>> >> PA in ppc as before because current ppc driver does not support
>VA
>> >> mode.
>> >
>> >What is the "ppc driver" you are referring to?
>> >Do you mean the vfio implementation for ppc?
>> >
>> >>
>> >> Theoretically, ppc can support VA mode, but I suspect that ppc
>with
>> >> VA mode may have performance issues to create a big DMA window
>> >> (VA often uses higher addresses than PA). So, I didn't change
>the
>> >> code to check device capability in ppc.
>> >
>> >Sorry, I am confused.
>> >Is VA mode not working at all? or is it a performance issue?
>> >
>> >
>> >--
>> >David Marchand
>> >
>> >
>>
>> Ah, I overlooked your first question. The "ppc driver" is in
>> drivers/bus/pci/linux/pci.c. In pci.c,
>pci_one_device_iommu_support_va()
>> always returns false in ppc. so, the driver does not allow VA mode
>in ppc.
>>
>> This return value is passed to rte_pci_probe_one_driver()
>> in drivers/bus/pci/pci_common.c. And the error log appeared like:
>>
>> EAL: PCI device 0000:01:00.0 on NUMA socket 0
>> EAL:   probe driver: 144d:a822 spdk_nvme
>> EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
>initializing
>
>Can you confirm on which kernel driver this device was bound ?
>I'd like to be sure this is the same issue than David.
>
>
>-- 
>David Marchand
>
>

Here's the full debug log I got. I ran upstream SPDK on NVMe on IBM Power9 AC922.
The operating system is Ubuntu 18.04 and Linux 4.17.

Starting SPDK v19.07-pre / DPDK 19.08.0-rc3 initialization...
[ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --log-level=lib.eal:8 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid139059 ]
EAL: Probing VFIO support...
EAL:   IOMMU type 1 (Type 1) is not supported
EAL:   IOMMU type 7 (sPAPR) is supported
EAL:   IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x30000 bytes
EAL: Virtual area found at 0x200000000000 (size = 0x30000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 1048576
EAL: Detected memory type: socket_id:0 hugepage_sz:2097152
EAL: Detected memory type: socket_id:8 hugepage_sz:2097152
EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200000030000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200000200000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200400200000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200400400000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200800400000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200800600000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200c00600000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200c00800000 (size = 0x400000000)
EAL: Creating 4 segment lists: n_segs:8192 socket_id:8 hugepage_sz:2097152
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201000800000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201000a00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201400a00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201400c00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201800c00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201800e00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201c00e00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201c01000000 (size = 0x400000000)
EAL: Hugepages will be freed exactly as allocated.
EAL: No shared files mode enabled, IPC is disabled
EAL: TSC frequency is ~510000 KHz
EAL: Master lcore 0 is ready (tid=77e7345a44a0;cpuset=[0])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: No shared files mode enabled, IPC is disabled
EAL: Heap on socket 0 was expanded by 2MB
EAL: No shared files mode enabled, IPC is disabled
EAL: Mem event callback 'spdk:(nil)' registered
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0000:01:00.0 cannot be used
EAL: PCI device 0003:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0003:01:00.0 cannot be used
EAL: PCI device 0030:01:00.0 on NUMA socket 8
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0030:01:00.0 cannot be used

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[David Marchand]

Hello,

On Thu, Aug 1, 2019 at 6:10 AM Takeshi T Yoshimura <TYOS@jp.ibm.com> wrote:
> Here's the full debug log I got. I ran upstream SPDK on NVMe on IBM Power9 AC922.
> The operating system is Ubuntu 18.04 and Linux 4.17.
>
> Starting SPDK v19.07-pre / DPDK 19.08.0-rc3 initialization...
> [ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --log-level=lib.eal:8 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid139059 ]

We are missing the initial logs here (eal log level is set twice, but
this should not be a problem).
Is it truncated by spdk ? Is there a way to get them ?

Thanks.

-- 
David Marchand

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[Takeshi T Yoshimura]

-----David Marchand <david.marchand@redhat.com> wrote: -----

>To: Takeshi T Yoshimura <TYOS@jp.ibm.com>
>From: David Marchand <david.marchand@redhat.com>
>Date: 08/01/2019 02:01PM
>Cc: dev <dev@dpdk.org>, David Christensen <drc@linux.vnet.ibm.com>
>Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in
>ppc
>
>Hello,
>
>On Thu, Aug 1, 2019 at 6:10 AM Takeshi T Yoshimura <TYOS@jp.ibm.com>
>wrote:
>> Here's the full debug log I got. I ran upstream SPDK on NVMe on IBM
>Power9 AC922.
>> The operating system is Ubuntu 18.04 and Linux 4.17.
>>
>> Starting SPDK v19.07-pre / DPDK 19.08.0-rc3 initialization...
>> [ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0
>--log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6
>--log-level=lib.eal:8 --base-virtaddr=0x200000000000
>--match-allocations --file-prefix=spdk_pid139059 ]
>
>We are missing the initial logs here (eal log level is set twice, but
>this should not be a problem).
>Is it truncated by spdk ? Is there a way to get them ?
>
>Thanks.
>
>-- 
>David Marchand
>
>

Okay. I should have set the --log-level=lib.*:debug. Here is the log.

Starting SPDK v19.10-pre / DPDK 19.08.0-rc3 initialization...
[ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --log-level=lib.*:debug --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid159692 ]
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Detected lcore 2 as core 0 on socket 0
EAL: Detected lcore 3 as core 0 on socket 0
EAL: Detected lcore 4 as core 4 on socket 0
EAL: Detected lcore 5 as core 4 on socket 0
EAL: Detected lcore 6 as core 4 on socket 0
EAL: Detected lcore 7 as core 4 on socket 0
EAL: Detected lcore 8 as core 8 on socket 0
EAL: Detected lcore 9 as core 8 on socket 0
EAL: Detected lcore 10 as core 8 on socket 0
EAL: Detected lcore 11 as core 8 on socket 0
EAL: Detected lcore 12 as core 12 on socket 0
EAL: Detected lcore 13 as core 12 on socket 0
EAL: Detected lcore 14 as core 12 on socket 0
EAL: Detected lcore 15 as core 12 on socket 0
EAL: Detected lcore 16 as core 24 on socket 0
EAL: Detected lcore 17 as core 24 on socket 0
EAL: Detected lcore 18 as core 24 on socket 0
EAL: Detected lcore 19 as core 24 on socket 0
EAL: Detected lcore 20 as core 28 on socket 0
EAL: Detected lcore 21 as core 28 on socket 0
EAL: Detected lcore 22 as core 28 on socket 0
EAL: Detected lcore 23 as core 28 on socket 0
EAL: Detected lcore 24 as core 32 on socket 0
EAL: Detected lcore 25 as core 32 on socket 0
EAL: Detected lcore 26 as core 32 on socket 0
EAL: Detected lcore 27 as core 32 on socket 0
EAL: Detected lcore 28 as core 36 on socket 0
EAL: Detected lcore 29 as core 36 on socket 0
EAL: Detected lcore 30 as core 36 on socket 0
EAL: Detected lcore 31 as core 36 on socket 0
EAL: Detected lcore 32 as core 48 on socket 0
EAL: Detected lcore 33 as core 48 on socket 0
EAL: Detected lcore 34 as core 48 on socket 0
EAL: Detected lcore 35 as core 48 on socket 0
EAL: Detected lcore 36 as core 52 on socket 0
EAL: Detected lcore 37 as core 52 on socket 0
EAL: Detected lcore 38 as core 52 on socket 0
EAL: Detected lcore 39 as core 52 on socket 0
EAL: Detected lcore 40 as core 56 on socket 0
EAL: Detected lcore 41 as core 56 on socket 0
EAL: Detected lcore 42 as core 56 on socket 0
EAL: Detected lcore 43 as core 56 on socket 0
EAL: Detected lcore 44 as core 60 on socket 0
EAL: Detected lcore 45 as core 60 on socket 0
EAL: Detected lcore 46 as core 60 on socket 0
EAL: Detected lcore 47 as core 60 on socket 0
EAL: Detected lcore 48 as core 64 on socket 0
EAL: Detected lcore 49 as core 64 on socket 0
EAL: Detected lcore 50 as core 64 on socket 0
EAL: Detected lcore 51 as core 64 on socket 0
EAL: Detected lcore 52 as core 68 on socket 0
EAL: Detected lcore 53 as core 68 on socket 0
EAL: Detected lcore 54 as core 68 on socket 0
EAL: Detected lcore 55 as core 68 on socket 0
EAL: Detected lcore 56 as core 72 on socket 0
EAL: Detected lcore 57 as core 72 on socket 0
EAL: Detected lcore 58 as core 72 on socket 0
EAL: Detected lcore 59 as core 72 on socket 0
EAL: Detected lcore 60 as core 76 on socket 0
EAL: Detected lcore 61 as core 76 on socket 0
EAL: Detected lcore 62 as core 76 on socket 0
EAL: Detected lcore 63 as core 76 on socket 0
EAL: Detected lcore 64 as core 80 on socket 0
EAL: Detected lcore 65 as core 80 on socket 0
EAL: Detected lcore 66 as core 80 on socket 0
EAL: Detected lcore 67 as core 80 on socket 0
EAL: Detected lcore 68 as core 84 on socket 0
EAL: Detected lcore 69 as core 84 on socket 0
EAL: Detected lcore 70 as core 84 on socket 0
EAL: Detected lcore 71 as core 84 on socket 0
EAL: Detected lcore 72 as core 88 on socket 0
EAL: Detected lcore 73 as core 88 on socket 0
EAL: Detected lcore 74 as core 88 on socket 0
EAL: Detected lcore 75 as core 88 on socket 0
EAL: Detected lcore 76 as core 92 on socket 0
EAL: Detected lcore 77 as core 92 on socket 0
EAL: Detected lcore 78 as core 92 on socket 0
EAL: Detected lcore 79 as core 92 on socket 0
EAL: Detected lcore 80 as core 2048 on socket 8
EAL: Detected lcore 81 as core 2048 on socket 8
EAL: Detected lcore 82 as core 2048 on socket 8
EAL: Detected lcore 83 as core 2048 on socket 8
EAL: Detected lcore 84 as core 2052 on socket 8
EAL: Detected lcore 85 as core 2052 on socket 8
EAL: Detected lcore 86 as core 2052 on socket 8
EAL: Detected lcore 87 as core 2052 on socket 8
EAL: Detected lcore 88 as core 2056 on socket 8
EAL: Detected lcore 89 as core 2056 on socket 8
EAL: Detected lcore 90 as core 2056 on socket 8
EAL: Detected lcore 91 as core 2056 on socket 8
EAL: Detected lcore 92 as core 2060 on socket 8
EAL: Detected lcore 93 as core 2060 on socket 8
EAL: Detected lcore 94 as core 2060 on socket 8
EAL: Detected lcore 95 as core 2060 on socket 8
EAL: Detected lcore 96 as core 2064 on socket 8
EAL: Detected lcore 97 as core 2064 on socket 8
EAL: Detected lcore 98 as core 2064 on socket 8
EAL: Detected lcore 99 as core 2064 on socket 8
EAL: Detected lcore 100 as core 2068 on socket 8
EAL: Detected lcore 101 as core 2068 on socket 8
EAL: Detected lcore 102 as core 2068 on socket 8
EAL: Detected lcore 103 as core 2068 on socket 8
EAL: Detected lcore 104 as core 2072 on socket 8
EAL: Detected lcore 105 as core 2072 on socket 8
EAL: Detected lcore 106 as core 2072 on socket 8
EAL: Detected lcore 107 as core 2072 on socket 8
EAL: Detected lcore 108 as core 2076 on socket 8
EAL: Detected lcore 109 as core 2076 on socket 8
EAL: Detected lcore 110 as core 2076 on socket 8
EAL: Detected lcore 111 as core 2076 on socket 8
EAL: Detected lcore 112 as core 2080 on socket 8
EAL: Detected lcore 113 as core 2080 on socket 8
EAL: Detected lcore 114 as core 2080 on socket 8
EAL: Detected lcore 115 as core 2080 on socket 8
EAL: Detected lcore 116 as core 2084 on socket 8
EAL: Detected lcore 117 as core 2084 on socket 8
EAL: Detected lcore 118 as core 2084 on socket 8
EAL: Detected lcore 119 as core 2084 on socket 8
EAL: Detected lcore 120 as core 2088 on socket 8
EAL: Detected lcore 121 as core 2088 on socket 8
EAL: Detected lcore 122 as core 2088 on socket 8
EAL: Detected lcore 123 as core 2088 on socket 8
EAL: Detected lcore 124 as core 2092 on socket 8
EAL: Detected lcore 125 as core 2092 on socket 8
EAL: Detected lcore 126 as core 2092 on socket 8
EAL: Detected lcore 127 as core 2092 on socket 8
EAL: Detected lcore 128 as core 2096 on socket 8
EAL: Detected lcore 129 as core 2096 on socket 8
EAL: Detected lcore 130 as core 2096 on socket 8
EAL: Detected lcore 131 as core 2096 on socket 8
EAL: Detected lcore 132 as core 2100 on socket 8
EAL: Detected lcore 133 as core 2100 on socket 8
EAL: Detected lcore 134 as core 2100 on socket 8
EAL: Detected lcore 135 as core 2100 on socket 8
EAL: Detected lcore 136 as core 2104 on socket 8
EAL: Detected lcore 137 as core 2104 on socket 8
EAL: Detected lcore 138 as core 2104 on socket 8
EAL: Detected lcore 139 as core 2104 on socket 8
EAL: Detected lcore 140 as core 2108 on socket 8
EAL: Detected lcore 141 as core 2108 on socket 8
EAL: Detected lcore 142 as core 2108 on socket 8
EAL: Detected lcore 143 as core 2108 on socket 8
EAL: Detected lcore 144 as core 2112 on socket 8
EAL: Detected lcore 145 as core 2112 on socket 8
EAL: Detected lcore 146 as core 2112 on socket 8
EAL: Detected lcore 147 as core 2112 on socket 8
EAL: Detected lcore 148 as core 2116 on socket 8
EAL: Detected lcore 149 as core 2116 on socket 8
EAL: Detected lcore 150 as core 2116 on socket 8
EAL: Detected lcore 151 as core 2116 on socket 8
EAL: Detected lcore 152 as core 2136 on socket 8
EAL: Detected lcore 153 as core 2136 on socket 8
EAL: Detected lcore 154 as core 2136 on socket 8
EAL: Detected lcore 155 as core 2136 on socket 8
EAL: Detected lcore 156 as core 2140 on socket 8
EAL: Detected lcore 157 as core 2140 on socket 8
EAL: Detected lcore 158 as core 2140 on socket 8
EAL: Detected lcore 159 as core 2140 on socket 8
EAL: Support maximum 1536 logical core(s) by configuration.
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No shared files mode enabled, IPC will be disabled
EAL: No shared files mode enabled, IPC is disabled
EAL: VFIO PCI modules not loaded
EAL: Bus pci wants IOVA as 'DC'
EAL: Buses did not request a specific IOVA mode.
EAL: IOMMU is available, selecting IOVA as VA mode.
EAL: Module /sys/module/rte_kni not found! error 2 (No such file or directory)
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL:   IOMMU type 1 (Type 1) is not supported
EAL:   IOMMU type 7 (sPAPR) is supported
EAL:   IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x30000 bytes
EAL: Virtual area found at 0x200000000000 (size = 0x30000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 1048576
EAL: Detected memory type: socket_id:0 hugepage_sz:2097152
EAL: Detected memory type: socket_id:8 hugepage_sz:2097152
EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200000030000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200000200000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200400200000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200400400000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200800400000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200800600000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200c00600000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200c00800000 (size = 0x400000000)
EAL: Creating 4 segment lists: n_segs:8192 socket_id:8 hugepage_sz:2097152
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201000800000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201000a00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201400a00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201400c00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201800c00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201800e00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201c00e00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201c01000000 (size = 0x400000000)
EAL: Hugepages will be freed exactly as allocated.
EAL: No shared files mode enabled, IPC is disabled
EAL: TSC frequency is ~510000 KHz
EAL: Master lcore 0 is ready (tid=7430194644a0;cpuset=[0])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: No shared files mode enabled, IPC is disabled
EAL: Heap on socket 0 was expanded by 2MB
EAL: No shared files mode enabled, IPC is disabled
EAL: Mem event callback 'spdk:(nil)' registered
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0000:01:00.0 cannot be used
EAL: PCI device 0003:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0003:01:00.0 cannot be used
EAL: PCI device 0030:01:00.0 on NUMA socket 8
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0030:01:00.0 cannot be used
No NVMe controllers found.

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[David Marchand]

On Thu, Aug 1, 2019 at 7:25 AM Takeshi T Yoshimura <TYOS@jp.ibm.com> wrote:
> Okay. I should have set the --log-level=lib.*:debug. Here is the log.
>
> Starting SPDK v19.10-pre / DPDK 19.08.0-rc3 initialization...
> [ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --log-level=lib.*:debug --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid159692 ]

[snip]

> EAL: No shared files mode enabled, IPC will be disabled
> EAL: No shared files mode enabled, IPC is disabled
> EAL: VFIO PCI modules not loaded

EAL thinks that vfio kmod is not loaded at this point.

> EAL: Bus pci wants IOVA as 'DC'

We see no log here for the devices that are used later.

- Are the (below) devices bound to vfio-pci or any other kmod at this
point of the init?
- Is the spdk_nvme driver loaded at this point?
- I suppose forcing --iova-mode=pa resolves the issue, but can you confirm this?


> EAL: Buses did not request a specific IOVA mode.
> EAL: IOMMU is available, selecting IOVA as VA mode.
> EAL: Module /sys/module/rte_kni not found! error 2 (No such file or directory)
> EAL: Selected IOVA mode 'VA'

[snip]

> EAL: PCI device 0000:01:00.0 on NUMA socket 0
> EAL:   probe driver: 144d:a822 spdk_nvme
> EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
> EAL: Requested device 0000:01:00.0 cannot be used
> EAL: PCI device 0003:01:00.0 on NUMA socket 0
> EAL:   probe driver: 144d:a822 spdk_nvme
> EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
> EAL: Requested device 0003:01:00.0 cannot be used
> EAL: PCI device 0030:01:00.0 on NUMA socket 8
> EAL:   probe driver: 144d:a822 spdk_nvme
> EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
> EAL: Requested device 0030:01:00.0 cannot be used
> No NVMe controllers found.



-- 
David Marchand

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[Takeshi T Yoshimura]

> - Are the (below) devices bound to vfio-pci or any other kmod at this point of the init?

Yes, vfio-pci is bound to 0030:01:00.0 in the log.

> - Is the spdk_nvme driver loaded at this point?

No, spdk just attempts to initialize dpdk at this point.
After the initialization finishes, spdk probes the nvme driver.

> - I suppose forcing --iova-mode=pa resolves the issue, but can you confirm this?

Yes, you are right. I needed to modify the spdk core to do this, anyway..
Here is the log when I specify --iova-mode=pa for an example app of spdk (identify)



Starting SPDK v19.10-pre / DPDK 19.08.0-rc3 initialization...
[ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --log-level=lib.*:debug --log-level=lib.cryptodev:5 --log-level=user1:6 --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid140641 ]
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Detected lcore 2 as core 0 on socket 0
EAL: Detected lcore 3 as core 0 on socket 0
EAL: Detected lcore 4 as core 4 on socket 0
EAL: Detected lcore 5 as core 4 on socket 0
EAL: Detected lcore 6 as core 4 on socket 0
EAL: Detected lcore 7 as core 4 on socket 0
EAL: Detected lcore 8 as core 8 on socket 0
EAL: Detected lcore 9 as core 8 on socket 0
EAL: Detected lcore 10 as core 8 on socket 0
EAL: Detected lcore 11 as core 8 on socket 0
EAL: Detected lcore 12 as core 12 on socket 0
EAL: Detected lcore 13 as core 12 on socket 0
EAL: Detected lcore 14 as core 12 on socket 0
EAL: Detected lcore 15 as core 12 on socket 0
EAL: Detected lcore 16 as core 24 on socket 0
EAL: Detected lcore 17 as core 24 on socket 0
EAL: Detected lcore 18 as core 24 on socket 0
EAL: Detected lcore 19 as core 24 on socket 0
EAL: Detected lcore 20 as core 28 on socket 0
EAL: Detected lcore 21 as core 28 on socket 0
EAL: Detected lcore 22 as core 28 on socket 0
EAL: Detected lcore 23 as core 28 on socket 0
EAL: Detected lcore 24 as core 32 on socket 0
EAL: Detected lcore 25 as core 32 on socket 0
EAL: Detected lcore 26 as core 32 on socket 0
EAL: Detected lcore 27 as core 32 on socket 0
EAL: Detected lcore 28 as core 36 on socket 0
EAL: Detected lcore 29 as core 36 on socket 0
EAL: Detected lcore 30 as core 36 on socket 0
EAL: Detected lcore 31 as core 36 on socket 0
EAL: Detected lcore 32 as core 48 on socket 0
EAL: Detected lcore 33 as core 48 on socket 0
EAL: Detected lcore 34 as core 48 on socket 0
EAL: Detected lcore 35 as core 48 on socket 0
EAL: Detected lcore 36 as core 52 on socket 0
EAL: Detected lcore 37 as core 52 on socket 0
EAL: Detected lcore 38 as core 52 on socket 0
EAL: Detected lcore 39 as core 52 on socket 0
EAL: Detected lcore 40 as core 56 on socket 0
EAL: Detected lcore 41 as core 56 on socket 0
EAL: Detected lcore 42 as core 56 on socket 0
EAL: Detected lcore 43 as core 56 on socket 0
EAL: Detected lcore 44 as core 60 on socket 0
EAL: Detected lcore 45 as core 60 on socket 0
EAL: Detected lcore 46 as core 60 on socket 0
EAL: Detected lcore 47 as core 60 on socket 0
EAL: Detected lcore 48 as core 64 on socket 0
EAL: Detected lcore 49 as core 64 on socket 0
EAL: Detected lcore 50 as core 64 on socket 0
EAL: Detected lcore 51 as core 64 on socket 0
EAL: Detected lcore 52 as core 68 on socket 0
EAL: Detected lcore 53 as core 68 on socket 0
EAL: Detected lcore 54 as core 68 on socket 0
EAL: Detected lcore 55 as core 68 on socket 0
EAL: Detected lcore 56 as core 72 on socket 0
EAL: Detected lcore 57 as core 72 on socket 0
EAL: Detected lcore 58 as core 72 on socket 0
EAL: Detected lcore 59 as core 72 on socket 0
EAL: Detected lcore 60 as core 76 on socket 0
EAL: Detected lcore 61 as core 76 on socket 0
EAL: Detected lcore 62 as core 76 on socket 0
EAL: Detected lcore 63 as core 76 on socket 0
EAL: Detected lcore 64 as core 80 on socket 0
EAL: Detected lcore 65 as core 80 on socket 0
EAL: Detected lcore 66 as core 80 on socket 0
EAL: Detected lcore 67 as core 80 on socket 0
EAL: Detected lcore 68 as core 84 on socket 0
EAL: Detected lcore 69 as core 84 on socket 0
EAL: Detected lcore 70 as core 84 on socket 0
EAL: Detected lcore 71 as core 84 on socket 0
EAL: Detected lcore 72 as core 88 on socket 0
EAL: Detected lcore 73 as core 88 on socket 0
EAL: Detected lcore 74 as core 88 on socket 0
EAL: Detected lcore 75 as core 88 on socket 0
EAL: Detected lcore 76 as core 92 on socket 0
EAL: Detected lcore 77 as core 92 on socket 0
EAL: Detected lcore 78 as core 92 on socket 0
EAL: Detected lcore 79 as core 92 on socket 0
EAL: Detected lcore 80 as core 2048 on socket 8
EAL: Detected lcore 81 as core 2048 on socket 8
EAL: Detected lcore 82 as core 2048 on socket 8
EAL: Detected lcore 83 as core 2048 on socket 8
EAL: Detected lcore 84 as core 2052 on socket 8
EAL: Detected lcore 85 as core 2052 on socket 8
EAL: Detected lcore 86 as core 2052 on socket 8
EAL: Detected lcore 87 as core 2052 on socket 8
EAL: Detected lcore 88 as core 2056 on socket 8
EAL: Detected lcore 89 as core 2056 on socket 8
EAL: Detected lcore 90 as core 2056 on socket 8
EAL: Detected lcore 91 as core 2056 on socket 8
EAL: Detected lcore 92 as core 2060 on socket 8
EAL: Detected lcore 93 as core 2060 on socket 8
EAL: Detected lcore 94 as core 2060 on socket 8
EAL: Detected lcore 95 as core 2060 on socket 8
EAL: Detected lcore 96 as core 2064 on socket 8
EAL: Detected lcore 97 as core 2064 on socket 8
EAL: Detected lcore 98 as core 2064 on socket 8
EAL: Detected lcore 99 as core 2064 on socket 8
EAL: Detected lcore 100 as core 2068 on socket 8
EAL: Detected lcore 101 as core 2068 on socket 8
EAL: Detected lcore 102 as core 2068 on socket 8
EAL: Detected lcore 103 as core 2068 on socket 8
EAL: Detected lcore 104 as core 2072 on socket 8
EAL: Detected lcore 105 as core 2072 on socket 8
EAL: Detected lcore 106 as core 2072 on socket 8
EAL: Detected lcore 107 as core 2072 on socket 8
EAL: Detected lcore 108 as core 2076 on socket 8
EAL: Detected lcore 109 as core 2076 on socket 8
EAL: Detected lcore 110 as core 2076 on socket 8
EAL: Detected lcore 111 as core 2076 on socket 8
EAL: Detected lcore 112 as core 2080 on socket 8
EAL: Detected lcore 113 as core 2080 on socket 8
EAL: Detected lcore 114 as core 2080 on socket 8
EAL: Detected lcore 115 as core 2080 on socket 8
EAL: Detected lcore 116 as core 2084 on socket 8
EAL: Detected lcore 117 as core 2084 on socket 8
EAL: Detected lcore 118 as core 2084 on socket 8
EAL: Detected lcore 119 as core 2084 on socket 8
EAL: Detected lcore 120 as core 2088 on socket 8
EAL: Detected lcore 121 as core 2088 on socket 8
EAL: Detected lcore 122 as core 2088 on socket 8
EAL: Detected lcore 123 as core 2088 on socket 8
EAL: Detected lcore 124 as core 2092 on socket 8
EAL: Detected lcore 125 as core 2092 on socket 8
EAL: Detected lcore 126 as core 2092 on socket 8
EAL: Detected lcore 127 as core 2092 on socket 8
EAL: Detected lcore 128 as core 2096 on socket 8
EAL: Detected lcore 129 as core 2096 on socket 8
EAL: Detected lcore 130 as core 2096 on socket 8
EAL: Detected lcore 131 as core 2096 on socket 8
EAL: Detected lcore 132 as core 2100 on socket 8
EAL: Detected lcore 133 as core 2100 on socket 8
EAL: Detected lcore 134 as core 2100 on socket 8
EAL: Detected lcore 135 as core 2100 on socket 8
EAL: Detected lcore 136 as core 2104 on socket 8
EAL: Detected lcore 137 as core 2104 on socket 8
EAL: Detected lcore 138 as core 2104 on socket 8
EAL: Detected lcore 139 as core 2104 on socket 8
EAL: Detected lcore 140 as core 2108 on socket 8
EAL: Detected lcore 141 as core 2108 on socket 8
EAL: Detected lcore 142 as core 2108 on socket 8
EAL: Detected lcore 143 as core 2108 on socket 8
EAL: Detected lcore 144 as core 2112 on socket 8
EAL: Detected lcore 145 as core 2112 on socket 8
EAL: Detected lcore 146 as core 2112 on socket 8
EAL: Detected lcore 147 as core 2112 on socket 8
EAL: Detected lcore 148 as core 2116 on socket 8
EAL: Detected lcore 149 as core 2116 on socket 8
EAL: Detected lcore 150 as core 2116 on socket 8
EAL: Detected lcore 151 as core 2116 on socket 8
EAL: Detected lcore 152 as core 2136 on socket 8
EAL: Detected lcore 153 as core 2136 on socket 8
EAL: Detected lcore 154 as core 2136 on socket 8
EAL: Detected lcore 155 as core 2136 on socket 8
EAL: Detected lcore 156 as core 2140 on socket 8
EAL: Detected lcore 157 as core 2140 on socket 8
EAL: Detected lcore 158 as core 2140 on socket 8
EAL: Detected lcore 159 as core 2140 on socket 8
EAL: Support maximum 1536 logical core(s) by configuration.
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No shared files mode enabled, IPC will be disabled
EAL: No shared files mode enabled, IPC is disabled
EAL: VFIO PCI modules not loaded
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL:   IOMMU type 1 (Type 1) is not supported
EAL:   IOMMU type 7 (sPAPR) is supported
EAL:   IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x30000 bytes
EAL: Virtual area found at 0x200000000000 (size = 0x30000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 1048576
EAL: Detected memory type: socket_id:0 hugepage_sz:2097152
EAL: Detected memory type: socket_id:8 hugepage_sz:2097152
EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200000030000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200000200000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200400200000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200400400000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200800400000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200800600000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200c00600000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200c00800000 (size = 0x400000000)
EAL: Creating 4 segment lists: n_segs:8192 socket_id:8 hugepage_sz:2097152
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201000800000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201000a00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201400a00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201400c00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201800c00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201800e00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201c00e00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201c01000000 (size = 0x400000000)
EAL: Hugepages will be freed exactly as allocated.
EAL: No shared files mode enabled, IPC is disabled
EAL: TSC frequency is ~510000 KHz
EAL: Master lcore 0 is ready (tid=7d49a88844a0;cpuset=[0])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: No shared files mode enabled, IPC is disabled
EAL: Heap on socket 0 was expanded by 2MB
EAL: No shared files mode enabled, IPC is disabled
EAL: Mem event callback 'spdk:(nil)' registered
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0003:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0030:01:00.0 on NUMA socket 8
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   set IOMMU type 1 (Type 1) failed, error 19 (No such device)
EAL:   using IOMMU type 7 (sPAPR)
EAL: Mem event callback 'vfio_mem_event_clb:(nil)' registered
EAL: Installed memory event callback for VFIO
EAL: VFIO reports MSI-X BAR as mappable
EAL:   PCI memory mapped at 0x202001000000
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: Calling mem event callback 'spdk:(nil)'
EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
EAL: request: mp_malloc_sync
EAL: No shared files mode enabled, IPC is disabled
EAL: Heap on socket 0 was expanded by 4MB
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: Calling mem event callback 'spdk:(nil)'
EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
EAL: request: mp_malloc_sync
EAL: No shared files mode enabled, IPC is disabled
EAL: Heap on socket 0 was expanded by 4MB
=====================================================
NVMe Controller at 0030:01:00.0 [144d:a822]
=====================================================
Controller Capabilities/Features
================================
Vendor ID:                             144d
Subsystem Vendor ID:                   1014
Serial Number:                         S3RXNA0K300102
Model Number:                          PCIe3 6.4TB NVMe Flash Adapter II x8
Firmware Version:                      MN12MN12
Recommended Arb Burst:                 8
IEEE OUI Identifier:                   38 25 00
Multi-path I/O
  May have multiple subsystem ports:   Yes
  May be connected to multiple hosts:  Yes
  Associated with SR-IOV VF:           No
Max Data Transfer Size:                1048576
Error Recovery Timeout:                Unlimited
NVMe Specification Version (VS):       1.2.1
NVMe Specification Version (Identify): 1.2.1
Maximum Queue Entries:                 1024
Contiguous Queues Required:            Yes
Arbitration Mechanisms Supported
  Weighted Round Robin:                Not Supported
  Vendor Specific:                     Not Supported
Reset Timeout:                         60000 ms
Doorbell Stride:                       4 bytes
NVM Subsystem Reset:                   Supported
Command Sets Supported
  NVM Command Set:                     Supported
Boot Partition:                        Not Supported
Memory Page Size Minimum:              4096 bytes
Memory Page Size Maximum:              4096 bytes
Optional Asynchronous Events Supported
  Namespace Attribute Notices:         Supported
  Firmware Activation Notices:         Supported
128-bit Host Identifier:               Not Supported

Controller Memory Buffer Support
================================
Supported:                             No

Admin Command Set Attributes
============================
Security Send/Receive:                 Not Supported
Format NVM:                            Supported
Firmware Activate/Download:            Supported
Namespace Management:                  Supported
Device Self-Test:                      Not Supported
Directives:                            Not Supported
NVMe-MI:                               Not Supported
Virtualization Management:             Not Supported
Doorbell Buffer Config:                Not Supported
Abort Command Limit:                   8
Async Event Request Limit:             16
Number of Firmware Slots:              3
Firmware Slot 1 Read-Only:             No
Firmware Update Granularity:           No Information Provided
Per-Namespace SMART Log:               No
Command Effects Log Page:              Supported
Get Log Page Extended Data:            Not Supported
Telemetry Log Pages:                   Not Supported
Error Log Page Entries Supported:      256
Keep Alive:                            Not Supported

NVM Command Set Attributes
==========================
Submission Queue Entry Size
  Max:                       64
  Min:                       64
Completion Queue Entry Size
  Max:                       16
  Min:                       16
Number of Namespaces:        32
Compare Command:             Not Supported
Write Uncorrectable Command: Supported
Dataset Management Command:  Supported
Write Zeroes Command:        Supported
Set Features Save Field:     Supported
Reservations:                Supported
Timestamp:                   Not Supported
Volatile Write Cache:        Not Present
Atomic Write Unit (Normal):  65536
Atomic Write Unit (PFail):   1
Atomic Compare & Write Unit: 1
Scatter-Gather List
  SGL Command Set:           Supported
  SGL Keyed:                 Not Supported
  SGL Bit Bucket Descriptor: Supported
  SGL Metadata Pointer:      Supported
  Oversized SGL:             Supported
  SGL Metadata Address:      Not Supported
  SGL Offset:                Not Supported
  Transport SGL Data Block:  Not Supported
Replay Protected Memory Block:  Not Supported

Firmware Slot Information
=========================
Active slot:                 1
Next controller reset slot:  1
Slot 1 Firmware Revision:    MN12MN12
Slot 2 Firmware Revision:    MN12MN12
Slot 3 Firmware Revision:    MN12MN12

Commands Supported and Effects
==============================
Admin Commands
--------------
   Delete I/O Submission Queue (00h): Supported 
   Create I/O Submission Queue (01h): Supported 
                  Get Log Page (02h): Supported 
   Delete I/O Completion Queue (04h): Supported 
   Create I/O Completion Queue (05h): Supported 
                      Identify (06h): Supported 
                         Abort (08h): Supported 
                  Set Features (09h): Supported 
                  Get Features (0Ah): Supported 
    Asynchronous Event Request (0Ch): Supported 
          Namespace Management (0Dh): Supported NS-Cap-Change NS-Inventory-Change All-NS-Exclusive
               Firmware Commit (10h): Supported 
       Firmware Image Download (11h): Supported 
          Namespace Attachment (15h): Supported All-NS-Exclusive
                    Format NVM (80h): Supported LBA-Change NS-Cap-Change Per-NS-Exclusive
I/O Commands
------------
                         Flush (00h): Supported 
                         Write (01h): Supported LBA-Change 
                          Read (02h): Supported 
           Write Uncorrectable (04h): Supported LBA-Change 
                  Write Zeroes (08h): Supported LBA-Change 
            Dataset Management (09h): Supported LBA-Change 
          Reservation Register (0Dh): Supported 
            Reservation Report (0Eh): Supported 
           Reservation Acquire (11h): Supported 
           Reservation Release (15h): Supported 

Error Log
=========
Entry: 0
Error Count:            0x2
Submission Queue Id:    0x0
Command Id:             0x1a
Phase Bit:              0
Status Code:            0x1c
Status Code Type:       0x1
Do Not Retry:           0
Error Location:         0x0
LBA:                    0x0
Namespace:              0x1
Vendor Log Page:        0x0
-----------
Entry: 1
Error Count:            0x1
Submission Queue Id:    0x0
Command Id:             0x5
Phase Bit:              0
Status Code:            0x1c
Status Code Type:       0x1
Do Not Retry:           0
Error Location:         0x0
LBA:                    0x0
Namespace:              0x1
Vendor Log Page:        0x0

Power Management
================
Number of Power States:      1
Current Power State:         Power State #0
Power State #0:  Max Power:  25.00 W
Non-Operational Permissive Mode: Not Supported

Health Information
==================
Critical Warnings:
  Available Spare Space:     OK
  Temperature:               OK
  Device Reliability:        OK
  Read Only:                 No
  Volatile Memory Backup:    OK
Current Temperature:         319 Kelvin (46 Celsius)
Temperature Threshold:       353 Kelvin (80 Celsius)
Available Spare:             100%
Available Spare Threshold:   10%
Life Percentage Used:        0%
Data Units Read:             44913828
Data Units Written:          67060524
Host Read Commands:          1363758490
Host Write Commands:         2804327733
Controller Busy Time:        715 minutes
Power Cycles:                104
Power On Hours:              6530 hours
Unsafe Shutdowns:            41
Unrecoverable Media Errors:  0
Lifetime Error Log Entries:  2
Warning Temperature Time:    0 minutes
Critical Temperature Time:   0 minutes
Temperature Sensor 1:        319 Kelvin (46 Celsius)
Temperature Sensor 2:        316 Kelvin (43 Celsius)
Temperature Sensor 3:        316 Kelvin (43 Celsius)

Number of Queues
================
Number of I/O Submission Queues:      128
Number of I/O Completion Queues:      128

Namespace ID:1
Deallocate:                            Supported
Deallocated/Unwritten Error:           Not Supported
Deallocated Read Value:                Unknown
Deallocate in Write Zeroes:            Not Supported
Deallocated Guard Field:               0xFFFF
Flush:                                 Not Supported
Reservation:                           Supported
Namespace Sharing Capabilities:        Private
Size (in LBAs):                        1562805846 (1490M)
Capacity (in LBAs):                    1562805846 (1490M)
Utilization (in LBAs):                 1562805846 (1490M)
NGUID:                                 335258304B3001020025384100000001
Thin Provisioning:                     Not Supported
Per-NS Atomic Units:                   No
NGUID/EUI64 Never Reused:              No
Number of LBA Formats:                 4
Current LBA Format:                    LBA Format #00
LBA Format #00: Data Size:  4096  Metadata Size:     0
LBA Format #01: Data Size:  4096  Metadata Size:     8
LBA Format #02: Data Size:   512  Metadata Size:     0
LBA Format #03: Data Size:   512  Metadata Size:     8

EAL: Calling mem event callback 'spdk:(nil)'
EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
EAL: request: mp_malloc_sync
EAL: No shared files mode enabled, IPC is disabled
EAL: Heap on socket 0 was shrunk by 4MB
EAL: Calling mem event callback 'spdk:(nil)'
EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
EAL: request: mp_malloc_sync
EAL: No shared files mode enabled, IPC is disabled
EAL: Heap on socket 0 was shrunk by 4MB
EAL: request: eal_dev_mp_request
EAL: No shared files mode enabled, IPC is disabled
EAL: PCI device 0030:01:00.0 on NUMA socket 8
EAL:   remove driver: 144d:a822 spdk_nvme
EAL: Mem event callback 'vfio_mem_event_clb:(nil)' unregistered
EAL: Releasing pci mapped resource for 0030:01:00.0
EAL: Calling pci_unmap_resource for 0030:01:00.0 at 0x202001000000
EAL:   PCI memory unmapped at 0x202001000000

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[David Marchand]

On Thu, Aug 1, 2019 at 10:46 AM Takeshi T Yoshimura <TYOS@jp.ibm.com> wrote:
>
> > - Are the (below) devices bound to vfio-pci or any other kmod at this point of the init?
>
> Yes, vfio-pci is bound to 0030:01:00.0 in the log.
>
> > - Is the spdk_nvme driver loaded at this point?
>
> No, spdk just attempts to initialize dpdk at this point.
> After the initialization finishes, spdk probes the nvme driver.

Do you mean that spdk registers its drivers after rte_eal_init() has returned?

Nothing should prevent from registering the drivers them before
calling rte_eal_init(), did you try this?
If the drivers are in the form of shared libraries (.so files), did
you try to load them using the -d eal option?


-- 
David Marchand

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[Takeshi T Yoshimura]

-----David Marchand <david.marchand@redhat.com> wrote: -----

>To: Takeshi T Yoshimura <TYOS@jp.ibm.com>
>From: David Marchand <david.marchand@redhat.com>
>Date: 08/01/2019 06:47PM
>Cc: dev <dev@dpdk.org>, David Christensen <drc@linux.vnet.ibm.com>
>Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in
>ppc
>
>On Thu, Aug 1, 2019 at 10:46 AM Takeshi T Yoshimura <TYOS@jp.ibm.com>
>wrote:
>>
>> > - Are the (below) devices bound to vfio-pci or any other kmod at
>this point of the init?
>>
>> Yes, vfio-pci is bound to 0030:01:00.0 in the log.
>>
>> > - Is the spdk_nvme driver loaded at this point?
>>
>> No, spdk just attempts to initialize dpdk at this point.
>> After the initialization finishes, spdk probes the nvme driver.
>
>Do you mean that spdk registers its drivers after rte_eal_init() has
>returned?
>
>Nothing should prevent from registering the drivers them before
>calling rte_eal_init(), did you try this?
>If the drivers are in the form of shared libraries (.so files), did
>you try to load them using the -d eal option?
>
>
>-- 
>David Marchand
>
>

SPDK apps have to call rte_eal_init() before registering drivers,
since the driver allocates DMA memory.
Do I have to change existing apps and spdk core to workaround this issue to use new versions of dpdk?

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[David Marchand]

On Fri, Aug 2, 2019 at 10:09 AM Takeshi T Yoshimura <TYOS@jp.ibm.com> wrote:
>
> -----David Marchand <david.marchand@redhat.com> wrote: -----
>
> >To: Takeshi T Yoshimura <TYOS@jp.ibm.com>
> >From: David Marchand <david.marchand@redhat.com>
> >Date: 08/01/2019 06:47PM
> >Cc: dev <dev@dpdk.org>, David Christensen <drc@linux.vnet.ibm.com>
> >Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in
> >ppc
> >
> >On Thu, Aug 1, 2019 at 10:46 AM Takeshi T Yoshimura <TYOS@jp.ibm.com>
> >wrote:
> >>
> >> > - Are the (below) devices bound to vfio-pci or any other kmod at
> >this point of the init?
> >>
> >> Yes, vfio-pci is bound to 0030:01:00.0 in the log.
> >>
> >> > - Is the spdk_nvme driver loaded at this point?
> >>
> >> No, spdk just attempts to initialize dpdk at this point.
> >> After the initialization finishes, spdk probes the nvme driver.
> >
> >Do you mean that spdk registers its drivers after rte_eal_init() has
> >returned?
> >
> >Nothing should prevent from registering the drivers them before
> >calling rte_eal_init(), did you try this?
> >If the drivers are in the form of shared libraries (.so files), did
> >you try to load them using the -d eal option?
> >
> >
> >--
> >David Marchand
> >
> >
>
> SPDK apps have to call rte_eal_init() before registering drivers,
> since the driver allocates DMA memory.

Registering a driver before calling rte_eal_init is something that
works most of the time.
Not sure I get your point here.


> Do I have to change existing apps and spdk core to workaround this issue to use new versions of dpdk?

I understand this is a regression and I am looking at it.
If we are too short for 19.08, then we can go with your workaround,
but it needs a better commitlog.


-- 
David Marchand

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[Takeshi T Yoshimura]

-----David Marchand <david.marchand@redhat.com> wrote: -----

>To: Takeshi T Yoshimura <TYOS@jp.ibm.com>
>From: David Marchand <david.marchand@redhat.com>
>Date: 08/01/2019 04:45AM
>Cc: dev <dev@dpdk.org>, David Christensen <drc@linux.vnet.ibm.com>
>Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in
>ppc
>
>On Wed, Jul 31, 2019 at 12:44 PM Takeshi T Yoshimura
><TYOS@jp.ibm.com> wrote:
>>
>> -----David Marchand <david.marchand@redhat.com> wrote: -----
>>
>> >To: Takeshi Yoshimura <tyos@jp.ibm.com>
>> >From: David Marchand <david.marchand@redhat.com>
>> >Date: 07/31/2019 06:29PM
>> >Cc: dev <dev@dpdk.org>
>> >Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA
>in
>> >ppc
>> >
>> >On Wed, Jul 31, 2019 at 5:36 AM Takeshi Yoshimura
><tyos@jp.ibm.com>
>> >wrote:
>> >>
>> >> Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI
>> >> drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC)
>> >> because of the inconsistency of user's request and the result of
>> >> device capability for IOVA mode. I updated the code to force
>IOVA
>> >as
>> >> PA in ppc as before because current ppc driver does not support
>VA
>> >> mode.
>> >
>> >What is the "ppc driver" you are referring to?
>> >Do you mean the vfio implementation for ppc?
>> >
>> >>
>> >> Theoretically, ppc can support VA mode, but I suspect that ppc
>with
>> >> VA mode may have performance issues to create a big DMA window
>> >> (VA often uses higher addresses than PA). So, I didn't change
>the
>> >> code to check device capability in ppc.
>> >
>> >Sorry, I am confused.
>> >Is VA mode not working at all? or is it a performance issue?
>> >
>> >
>> >--
>> >David Marchand
>> >
>> >
>>
>> Ah, I overlooked your first question. The "ppc driver" is in
>> drivers/bus/pci/linux/pci.c. In pci.c,
>pci_one_device_iommu_support_va()
>> always returns false in ppc. so, the driver does not allow VA mode
>in ppc.
>>
>> This return value is passed to rte_pci_probe_one_driver()
>> in drivers/bus/pci/pci_common.c. And the error log appeared like:
>>
>> EAL: PCI device 0000:01:00.0 on NUMA socket 0
>> EAL:   probe driver: 144d:a822 spdk_nvme
>> EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
>initializing
>
>Can you confirm on which kernel driver this device was bound ?
>I'd like to be sure this is the same issue than David.
>
>
>-- 
>David Marchand
>
>

Here's the full debug log I got. I ran upstream SPDK on NVMe on IBM Power9 AC922.
The operating system is Ubuntu 18.04 and Linux 4.17.

Starting SPDK v19.07-pre / DPDK 19.08.0-rc3 initialization...
[ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --log-level=lib.eal:8 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid139059 ]
EAL: Probing VFIO support...
EAL:   IOMMU type 1 (Type 1) is not supported
EAL:   IOMMU type 7 (sPAPR) is supported
EAL:   IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x30000 bytes
EAL: Virtual area found at 0x200000000000 (size = 0x30000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 1048576
EAL: Detected memory type: socket_id:0 hugepage_sz:2097152
EAL: Detected memory type: socket_id:8 hugepage_sz:2097152
EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200000030000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200000200000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200400200000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200400400000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200800400000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200800600000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200c00600000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200c00800000 (size = 0x400000000)
EAL: Creating 4 segment lists: n_segs:8192 socket_id:8 hugepage_sz:2097152
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201000800000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201000a00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201400a00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201400c00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201800c00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201800e00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201c00e00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201c01000000 (size = 0x400000000)
EAL: Hugepages will be freed exactly as allocated.
EAL: No shared files mode enabled, IPC is disabled
EAL: TSC frequency is ~510000 KHz
EAL: Master lcore 0 is ready (tid=77e7345a44a0;cpuset=[0])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: No shared files mode enabled, IPC is disabled
EAL: Heap on socket 0 was expanded by 2MB
EAL: No shared files mode enabled, IPC is disabled
EAL: Mem event callback 'spdk:(nil)' registered
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0000:01:00.0 cannot be used
EAL: PCI device 0003:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0003:01:00.0 cannot be used
EAL: PCI device 0030:01:00.0 on NUMA socket 8
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0030:01:00.0 cannot be used

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[David Marchand]

Hello Anatoly,

On Wed, Jul 31, 2019 at 5:36 AM Takeshi Yoshimura <tyos@jp.ibm.com> wrote:
>
> Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI
> drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC)
> because of the inconsistency of user's request and the result of
> device capability for IOVA mode. I updated the code to force IOVA as
> PA in ppc as before because current ppc driver does not support VA
> mode.
>
> Theoretically, ppc can support VA mode, but I suspect that ppc with
> VA mode may have performance issues to create a big DMA window
> (VA often uses higher addresses than PA). So, I didn't change the
> code to check device capability in ppc.
>
> Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>
> ---
>  lib/librte_eal/linux/eal/eal.c | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c
> index 946222ccd..db2dec922 100644
> --- a/lib/librte_eal/linux/eal/eal.c
> +++ b/lib/librte_eal/linux/eal/eal.c
> @@ -1121,6 +1121,12 @@ rte_eal_init(int argc, char **argv)
>                                 RTE_LOG(DEBUG, EAL, "KNI can not work since physical addresses are unavailable\n");
>                         }
>                 }
> +#endif
> +#ifdef RTE_ARCH_PPC_64
> +               if (iova_mode == RTE_IOVA_VA) {
> +                       iova_mode = RTE_IOVA_PA;
> +                       RTE_LOG(WARNING, EAL, "Forcing IOVA as 'PA' because PPC uses PA mode.\n");
> +               }
>  #endif
>                 rte_eal_get_configuration()->iova_mode = iova_mode;
>         } else {
> --
> 2.17.1
>

Afaiu, this problem is linked to a specific iommu implementation.
Does it make sense to introduce a check "iommu_support_va" in eal, a
bit like what we have in the pci bus ?


-- 
David Marchand

Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[Burakov, Anatoly]

On 01-Aug-19 2:01 PM, David Marchand wrote:
> Hello Anatoly,
> 
> On Wed, Jul 31, 2019 at 5:36 AM Takeshi Yoshimura <tyos@jp.ibm.com> wrote:
>>
>> Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI
>> drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC)
>> because of the inconsistency of user's request and the result of
>> device capability for IOVA mode. I updated the code to force IOVA as
>> PA in ppc as before because current ppc driver does not support VA
>> mode.
>>
>> Theoretically, ppc can support VA mode, but I suspect that ppc with
>> VA mode may have performance issues to create a big DMA window
>> (VA often uses higher addresses than PA). So, I didn't change the
>> code to check device capability in ppc.
>>
>> Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>
>> ---
>>   lib/librte_eal/linux/eal/eal.c | 6 ++++++
>>   1 file changed, 6 insertions(+)
>>
>> diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c
>> index 946222ccd..db2dec922 100644
>> --- a/lib/librte_eal/linux/eal/eal.c
>> +++ b/lib/librte_eal/linux/eal/eal.c
>> @@ -1121,6 +1121,12 @@ rte_eal_init(int argc, char **argv)
>>                                  RTE_LOG(DEBUG, EAL, "KNI can not work since physical addresses are unavailable\n");
>>                          }
>>                  }
>> +#endif
>> +#ifdef RTE_ARCH_PPC_64
>> +               if (iova_mode == RTE_IOVA_VA) {
>> +                       iova_mode = RTE_IOVA_PA;
>> +                       RTE_LOG(WARNING, EAL, "Forcing IOVA as 'PA' because PPC uses PA mode.\n");
>> +               }
>>   #endif
>>                  rte_eal_get_configuration()->iova_mode = iova_mode;
>>          } else {
>> --
>> 2.17.1
>>
> 
> Afaiu, this problem is linked to a specific iommu implementation.
> Does it make sense to introduce a check "iommu_support_va" in eal, a
> bit like what we have in the pci bus ?
> 

It probably does. Some platforms simply don't support VA mode (FreeBSD, 
for one), so that would be useful instead of hardcoding IOVA as PA/VA as 
defaults.

-- 
Thanks,
Anatoly