Re: [dpdk-dev] [PATCH v3 1/3] eal/linux: select iova-mode va with no-huge option

[dwilder <dwilder@us.ibm.com>]

On 2020-03-23 23:19, Jerin Jacob wrote:
> On Mon, Mar 23, 2020 at 11:11 PM dwilder <dwilder@us.ibm.com> wrote:
>> 
>> Thanks you for your review Jerin.  See my responses are inline.
>> 
>> On 2020-03-20 06:24, Jerin Jacob wrote:
>> > On Fri, Feb 21, 2020 at 4:22 AM David Wilder <dwilder@us.ibm.com>
>> > wrote:
>> >>
>> >> If --no-huge is set and iova-mode has not been specified force VA
>> >> mode.
>> >> If --no-huge and --iova-mode=PA is requested error out as this is
>> >> an impossible configuration.
>> >>
>> >> Signed-off-by: David Wilder <dwilder@us.ibm.com>
>> >> ---
>> >>  lib/librte_eal/linux/eal/eal.c | 14 ++++++++++++++
>> >>  1 file changed, 14 insertions(+)
>> >>
>> >> diff --git a/lib/librte_eal/linux/eal/eal.c
>> >> b/lib/librte_eal/linux/eal/eal.c
>> >> index 9530ee55f..d3a0a1731 100644
>> >> --- a/lib/librte_eal/linux/eal/eal.c
>> >> +++ b/lib/librte_eal/linux/eal/eal.c
>> >> @@ -1062,9 +1062,16 @@ rte_eal_init(int argc, char **argv)
>> >>
>> >>         /* if no EAL option "--iova-mode=<pa|va>", use bus IOVA scheme
>> >> */
>> >>         if (internal_config.iova_mode == RTE_IOVA_DC) {
>> >> +
>> >>                 /* autodetect the IOVA mapping mode */
>> >>                 enum rte_iova_mode iova_mode =
>> >> rte_bus_get_iommu_class();
>> >>
>> >> +               if (iova_mode == RTE_IOVA_PA &&
>> >> !rte_eal_has_hugepages()) {
>> >> +                       iova_mode = RTE_IOVA_VA;
>> 
>> >
>> > What if igb_uio or vfio_nommu has been loaded(i.e no iommu support
>> > enabled from the driver)? This would fail.
>> 
>> Yes they would fail.  If igb_uio or vfio_nommu (or any driver) cant be
>> forced to VA mode it cant be used with out hugepages.  Drivers can be
>> available but not used therefor we print a warning message.
> 
> I think, the warning will not be enough as the system will fail anyway.
> 
> iova_mode == RTE_IOVA_PA && rte_eal_has_hugepages() == 0 && no_iommu == 
> 1
> case, we need to return error.
> 
> iova_mode == RTE_IOVA_PA && rte_eal_has_hugepages() == 0 && no_iommu == 
> 0
> case warning is enough.

The current code will skip the bus if iova-mode is not supported,  this 
allow other devices to continu on.  The handing of an unsupported 
iova-mode is done in rte_pci_probe_one_driver(). See also 
rte_bus_get_iommu_class() if multiple busses cant agree on iova-mode a 
warning is given.

Here I have bound 0002:01:00.1 to igb_uio and forced iova-mode=pa,  much 
as my code did when --no-huge is used.

./dpdk-devbind.py -s
0002:01:00.1 'Ethernet Controller X710/X557-AT 10GBASE-T 1589' 
drv=igb_uio unused=i40e
<....>
dpdk-testpmd -c 3  --iova-mode=va -w 0002:01:00.1 -- -ia
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: PCI device 0002:01:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1589 net_i40e
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not 
initializing  <<<<<<<<<
EAL: Requested device 0002:01:00.1 cannot be used
<.....>


>> 
>> >
>> >> +                       RTE_LOG(WARNING, EAL, "Some buses want 'PA'
>> >> but forcing 'VA' because --no-huge is requested.\n");
>> >> +                       RTE_LOG(WARNING, EAL, "Not all buses may be
>> >> able to initialize.\n");
>> >> +               }
>> >> +
>> >>                 if (iova_mode == RTE_IOVA_DC) {
>> >>                         RTE_LOG(DEBUG, EAL, "Buses did not request a
>> >> specific IOVA mode.\n");
>> >>
>> >> @@ -1111,6 +1118,13 @@ rte_eal_init(int argc, char **argv)
>> >>                         internal_config.iova_mode;
>> >>         }
>> >>
>> >> +       if (rte_eal_iova_mode() == RTE_IOVA_PA &&
>> >> +           rte_eal_has_hugepages() == 0) {
>> >> +               rte_eal_init_alert("Cannot use IOVA as 'PA' with
>> >> --no-huge");
>> >
>> > Top of the tree already detecting this case. am I missing anything?
>> >
>> > [master]dell[dpdk.org] $ sudo ./build/app/test/dpdk-test  -c 0x3
>> > --no-huge --iova-mode=pa
>> > EAL: Detected 56 lcore(s)
>> > EAL: Detected 2 NUMA nodes
>> > EAL: Static memory layout is selected, amount of reserved memory can
>> > be adjusted with -m or --socket-mem
>> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> > EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not
>> > available
>> > EAL: Cannot use IOVA as 'PA' since physical addresses are not available
>> >
>> 
>> The check you reference is reporting that physical address are not
>> available, for example no permissions to read /proc/self/pagemap.  In
>> this case, if --no-huge is set then PA mode is not allowed. There is 
>> no
>> guarantee that physical address are persistent with out using 
>> hugepages.
> 
> Since this check is under the following, Yes, make sense for the check.
> The old command has explicit  --iova-mode=pa. So it is in the
> different code paths.
> 
> /* if no EAL option "--iova-mode=<pa|va>", use bus IOVA scheme */
>         if (internal_config.iova_mode == RTE_IOVA_DC) {
> 
>> 
>> 
>> >> +               rte_errno = EINVAL;
>> >> +               return -1;
>> >> +       }
>> >> +
>> >>         if (rte_eal_iova_mode() == RTE_IOVA_PA && !phys_addrs) {
>> >>                 rte_eal_init_alert("Cannot use IOVA as 'PA' since
>> >> physical addresses are not available");
>> >>                 rte_errno = EINVAL;
>> >> --
>> >> 2.25.0
>> >>