Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

["Takeshi T Yoshimura" <TYOS@jp.ibm.com>]

-----David Marchand <david.marchand@redhat.com> wrote: -----

>To: Takeshi T Yoshimura <TYOS@jp.ibm.com>
>From: David Marchand <david.marchand@redhat.com>
>Date: 08/01/2019 04:45AM
>Cc: dev <dev@dpdk.org>, David Christensen <drc@linux.vnet.ibm.com>
>Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in
>ppc
>
>On Wed, Jul 31, 2019 at 12:44 PM Takeshi T Yoshimura
><TYOS@jp.ibm.com> wrote:
>>
>> -----David Marchand <david.marchand@redhat.com> wrote: -----
>>
>> >To: Takeshi Yoshimura <tyos@jp.ibm.com>
>> >From: David Marchand <david.marchand@redhat.com>
>> >Date: 07/31/2019 06:29PM
>> >Cc: dev <dev@dpdk.org>
>> >Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA
>in
>> >ppc
>> >
>> >On Wed, Jul 31, 2019 at 5:36 AM Takeshi Yoshimura
><tyos@jp.ibm.com>
>> >wrote:
>> >>
>> >> Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI
>> >> drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC)
>> >> because of the inconsistency of user's request and the result of
>> >> device capability for IOVA mode. I updated the code to force
>IOVA
>> >as
>> >> PA in ppc as before because current ppc driver does not support
>VA
>> >> mode.
>> >
>> >What is the "ppc driver" you are referring to?
>> >Do you mean the vfio implementation for ppc?
>> >
>> >>
>> >> Theoretically, ppc can support VA mode, but I suspect that ppc
>with
>> >> VA mode may have performance issues to create a big DMA window
>> >> (VA often uses higher addresses than PA). So, I didn't change
>the
>> >> code to check device capability in ppc.
>> >
>> >Sorry, I am confused.
>> >Is VA mode not working at all? or is it a performance issue?
>> >
>> >
>> >--
>> >David Marchand
>> >
>> >
>>
>> Ah, I overlooked your first question. The "ppc driver" is in
>> drivers/bus/pci/linux/pci.c. In pci.c,
>pci_one_device_iommu_support_va()
>> always returns false in ppc. so, the driver does not allow VA mode
>in ppc.
>>
>> This return value is passed to rte_pci_probe_one_driver()
>> in drivers/bus/pci/pci_common.c. And the error log appeared like:
>>
>> EAL: PCI device 0000:01:00.0 on NUMA socket 0
>> EAL:   probe driver: 144d:a822 spdk_nvme
>> EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
>initializing
>
>Can you confirm on which kernel driver this device was bound ?
>I'd like to be sure this is the same issue than David.
>
>
>-- 
>David Marchand
>
>

Here's the full debug log I got. I ran upstream SPDK on NVMe on IBM Power9 AC922.
The operating system is Ubuntu 18.04 and Linux 4.17.

Starting SPDK v19.07-pre / DPDK 19.08.0-rc3 initialization...
[ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --log-level=lib.eal:8 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid139059 ]
EAL: Probing VFIO support...
EAL:   IOMMU type 1 (Type 1) is not supported
EAL:   IOMMU type 7 (sPAPR) is supported
EAL:   IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x30000 bytes
EAL: Virtual area found at 0x200000000000 (size = 0x30000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 1048576
EAL: Detected memory type: socket_id:0 hugepage_sz:2097152
EAL: Detected memory type: socket_id:8 hugepage_sz:2097152
EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200000030000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200000200000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200400200000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200400400000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200800400000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200800600000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200c00600000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200c00800000 (size = 0x400000000)
EAL: Creating 4 segment lists: n_segs:8192 socket_id:8 hugepage_sz:2097152
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201000800000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201000a00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201400a00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201400c00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201800c00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201800e00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201c00e00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201c01000000 (size = 0x400000000)
EAL: Hugepages will be freed exactly as allocated.
EAL: No shared files mode enabled, IPC is disabled
EAL: TSC frequency is ~510000 KHz
EAL: Master lcore 0 is ready (tid=77e7345a44a0;cpuset=[0])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: No shared files mode enabled, IPC is disabled
EAL: Heap on socket 0 was expanded by 2MB
EAL: No shared files mode enabled, IPC is disabled
EAL: Mem event callback 'spdk:(nil)' registered
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0000:01:00.0 cannot be used
EAL: PCI device 0003:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0003:01:00.0 cannot be used
EAL: PCI device 0030:01:00.0 on NUMA socket 8
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0030:01:00.0 cannot be used