Re: [dpdk-dev] [PATCH v4 2/4] eal: fix IOVA mode selection as VA for PCI drivers

[Jerin Jacob <jerinjacobk@gmail.com>]

On Mon, Nov 25, 2019 at 6:33 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
>
> On 7/22/2019 1:56 PM, David Marchand wrote:
> > The incriminated commit broke the use of RTE_PCI_DRV_IOVA_AS_VA which
> > was intended to mean "driver only supports VA" but had been understood
> > as "driver supports both PA and VA" by most net drivers and used to let
> > dpdk processes to run as non root (which do not have access to physical
> > addresses on recent kernels).
> >
> > The check on physical addresses actually closed the gap for those
> > drivers. We don't need to mark them with RTE_PCI_DRV_IOVA_AS_VA and this
> > flag can retain its intended meaning.
> > Document explicitly its meaning.
> >
> > We can check that a driver requirement wrt to IOVA mode is fulfilled
> > before trying to probe a device.
> >
> > Finally, document the heuristic used to select the IOVA mode and hope
> > that we won't break it again.
> >
> > Fixes: 703458e19c16 ("bus/pci: consider only usable devices for IOVA mode")
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > Reviewed-by: Jerin Jacob <jerinj@marvell.com>
> > Tested-by: Jerin Jacob <jerinj@marvell.com>
> > Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> <...>
>
> > diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
> > index d2af472..9794552 100644
> > --- a/drivers/bus/pci/pci_common.c
> > +++ b/drivers/bus/pci/pci_common.c
> > @@ -169,8 +169,22 @@ static struct rte_devargs *pci_devargs_lookup(struct rte_pci_device *dev)
> >        * This needs to be before rte_pci_map_device(), as it enables to use
> >        * driver flags for adjusting configuration.
> >        */
> > -     if (!already_probed)
> > +     if (!already_probed) {
> > +             enum rte_iova_mode dev_iova_mode;
> > +             enum rte_iova_mode iova_mode;
> > +
> > +             dev_iova_mode = pci_device_iova_mode(dr, dev);
> > +             iova_mode = rte_eal_iova_mode();
> > +             if (dev_iova_mode != RTE_IOVA_DC &&
> > +                 dev_iova_mode != iova_mode) {
> > +                     RTE_LOG(ERR, EAL, "  Expecting '%s' IOVA mode but current mode is '%s', not initializing\n",
> > +                             dev_iova_mode == RTE_IOVA_PA ? "PA" : "VA",
> > +                             iova_mode == RTE_IOVA_PA ? "PA" : "VA");
> > +                     return -EINVAL;
> > +             }
> > +
>
> OvS reported an error while hotplugging a device.
>
> It looks like DPDK application initialized as IOVA=VA, and the new device is bound to 'igb_uio' which forces it to PA, fails on above check.

Why they are binding to igb_uio if there is NO need for it?

>
> I would like to get your comment on the issue.
>
> For the OvS mode, hopefully binding the device to 'vfio-pci' can be a solution, but for the cases we don't have that option, can/should we force the DPDK to PA mode after initialization?

On the other side, If we are forcing DPDK to PA then the same fate
will be for VFIO only devices.
There are two cases:
1) The system has a limitation on the specific mode
2) The devices have a limitation on the specific mod.e.

Case (1), It is not applicable for hotplug cases as the system can run
only one mode. We should able to detect in the first pass(Before the
hotplug devices runs)
Case (2), Is there any devices that can work ONLY in IOVA as PA mode?
If yes, Please enumerate. Maybe something in the storage domain.



>