Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

[David Christensen <drc@linux.vnet.ibm.com>]

>> Here's the output from my Power 9 system running 19.08-RC2 (I'm not
>> using VFIO since I have an MLX5 adapter).
>>
>> sudo -E LD_LIBRARY_PATH=/home/davec/src/dpdk/build/lib
>> /home/davec/src/dpdk/build/app/testpmd --log-level lib.\*:debug -w
>> 0000:01:00.0 -w 0000:01:00.1 -l 4,8-79 -n 4  -- --rxq=32 --rxd=1024
>> --txq=32 --txd=4096 --nb-cores=64 -i -a --numa --forward-mode=rxonly
>> EAL: Detected lcore 0 as core 0 on socket 0
>> <snip> ...
>> EAL: Detected lcore 159 as core 2140 on socket 8
>> EAL: Support maximum 1536 logical core(s) by configuration.
>> EAL: Detected 160 lcore(s)
>> EAL: Detected 2 NUMA nodes
>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or
>> directory)
>> EAL: VFIO PCI modules not loaded
>> EAL: DPAA Bus not present. Skipping.
>> EAL: Bus pci wants IOVA as 'DC'
> 
> Ok, I think I understand now.
> 
> Can you try this hunk ?
> 

IOVA mode is now reported as PA.

MLX5 PMD fails to load correctly, but that also occurs without the patch 
and when the EAL command-line option "--iova-mode=pa" is used.  This 
worked previously in RC1 and may be a separate issue.

sudo -E LD_LIBRARY_PATH=/home/davec/src/dpdk/build/lib 
/home/davec/src/dpdk/build/app/testpmd --log-level="lib.*:debug" -w 
0000:01:00.0 -w 0000:01:00.1 -l 4,8-79 -n 4  -- --rxq=32 --rxd=1024 
--txq=32 --txd=4096 --nb-cores=64 -i -a --numa --forward-mode=rxonly
EAL: Detected lcore 0 as core 0 on socket 0
<snip> ...
EAL: Detected lcore 159 as core 2140 on socket 8
EAL: Support maximum 1536 logical core(s) by configuration.
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or 
directory)
EAL: VFIO PCI modules not loaded
EAL: DPAA Bus not present. Skipping.
EAL: PCI driver net_mlx5 for device 0000:01:00.0 wants IOVA as 'PA'
EAL: PCI driver net_mlx5 for device 0000:01:00.1 wants IOVA as 'PA'
EAL: Bus pci wants IOVA as 'PA'
EAL: Bus dpaa_bus wants IOVA as 'DC'
EAL: Bus fslmc wants IOVA as 'DC'
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
EAL: VFIO modules not loaded, skipping VFIO support...
EAL: Ask a virtual area of 0x30000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x30000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 4096
EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
EAL: Detected memory type: socket_id:8 hugepage_sz:1073741824
EAL: Creating 4 segment lists: n_segs:32 socket_id:0 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x100030000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x140000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x940000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x980000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x1180000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x11c0000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x19c0000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x1a00000000 (size = 0x800000000)
EAL: Creating 4 segment lists: n_segs:32 socket_id:8 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x2200000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x2240000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x2a40000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x2a80000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x3280000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x32c0000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x3ac0000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x3b00000000 (size = 0x800000000)
EAL: TSC frequency is ~510000 KHz
EAL: Master lcore 4 is ready (tid=7fffabb46c90;cpuset=[4])
EAL: lcore 9 is ready (tid=7fffa9c5dc90;cpuset=[9])
<snip> ...
EAL: lcore 73 is ready (tid=7ffea77edc90;cpuset=[73])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 1024MB
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 15b3:1019 net_mlx5
EAL: Mem event callback 'MLX5_MEM_EVENT_CB:(nil)' registered
net_mlx5: mlx5.c:1683: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel 
offloading disabled due to old OFED/rdma-core version or firmware 
configuration
EAL: PCI device 0000:01:00.1 on NUMA socket 0
EAL:   probe driver: 15b3:1019 net_mlx5
net_mlx5: mlx5.c:1683: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel 
offloading disabled due to old OFED/rdma-core version or firmware 
configuration
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
testpmd: /home/davec/src/dpdk/drivers/net/mlx5/mlx5_ethdev.c:520: 
mlx5_set_txlimit_params: Assertion `config->txq_inline_min >= 0' failed.
Aborted


Dave