RE: [PATCH v4] bus/pci: fix legacy device IO port map in secondary process

["Ma, WenwuX" <wenwux.ma@intel.com>]

Hi,

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: 2023年10月3日 18:21
> To: Ma, WenwuX <wenwux.ma@intel.com>; nipun.gupta@amd.com;
> chenbo.xia@outlook.com
> Cc: dev@dpdk.org; maxime.coquelin@redhat.com; Li, Miao
> <miao.li@intel.com>; Ling, WeiX <weix.ling@intel.com>; stable@dpdk.org
> Subject: Re: [PATCH v4] bus/pci: fix legacy device IO port map in secondary
> process
> 
> On Wed, Aug 30, 2023 at 7:07 AM Wenwu Ma <wenwux.ma@intel.com>
> wrote:
> >
> > When doing IO port mapping for legacy device in secondary process, the
> > region information is missing, so, we need to refill it.
> >
> > Fixes: 4b741542ecde ("bus/pci: avoid depending on private kernel
> > value")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Wenwu Ma <wenwux.ma@intel.com>
> >
> > ---
> >  drivers/bus/pci/linux/pci_vfio.c | 43
> > ++++++++++++++++++++++++++++++--
> >  1 file changed, 41 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/bus/pci/linux/pci_vfio.c
> > b/drivers/bus/pci/linux/pci_vfio.c
> > index e634de8322..5ef26c98d1 100644
> > --- a/drivers/bus/pci/linux/pci_vfio.c
> > +++ b/drivers/bus/pci/linux/pci_vfio.c
> > @@ -1314,6 +1314,27 @@ pci_vfio_ioport_map(struct rte_pci_device
> *dev, int bar,
> >                 return -1;
> >         }
> >
> > +       if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
> > +               struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
> > +               char pci_addr[PATH_MAX];
> > +               int vfio_dev_fd;
> > +               struct rte_pci_addr *loc = &dev->addr;
> > +               int ret;
> > +               /* store PCI address string */
> > +               snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
> > +                               loc->domain, loc->bus, loc->devid,
> > + loc->function);
> > +
> > +               ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
> > +                                               &vfio_dev_fd,
> > + &device_info);
> 
> From a pci bus API pov, nothing prevents a driver from mixing memory
> mapped with vfio and ioport resources (iow, calls to
> rte_pci_map_resource() and rte_pci_ioport_map()).
> IOW, it may not be the case with the net/virtio driver but, in theory,
> rte_pci_ioport_map()/pci_vfio_ioport_map() may be called after a
> rte_pci_map_resource() call.
> 
> In a similar manner, from the API pov,
> rte_pci_ioport_map()/pci_vfio_ioport_map() may be called for multiple bars.
> 
> In summary, nothing in this patch checks that vfio has been configured already
> and I think we need a refcount to handle those situations.
> 
We call rte_vfio_setup_device just to get device info, we can call rte_vfio_release_device as soon as pci_vfio_fill_regions is done.
This avoids reference counting operations, do you think it works?

> Nipun, Chenbo, WDYT?
> 
> 
> > +               if (ret)
> > +                       return -1;
> 
> ret value is not used, so there is no need for this variable.
> 
> if (rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
>         &vfio_dev_fd, &device_info) != 0)
>     return -1;
> 
> > +
> > +               ret = pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info);
> > +               if (ret)
> > +                       return -1;
> 
> Same here, ret is not needed.
> 
> 
> > +
> > +       }
> > +
> >         if (pci_vfio_get_region(dev, bar, &size, &offset) != 0) {
> >                 RTE_LOG(ERR, EAL, "Cannot get offset of region %d.\n", bar);
> >                 return -1;
> > @@ -1361,8 +1382,26 @@ pci_vfio_ioport_write(struct rte_pci_ioport *p,
> > int  pci_vfio_ioport_unmap(struct rte_pci_ioport *p)  {
> > -       RTE_SET_USED(p);
> > -       return -1;
> > +       char pci_addr[PATH_MAX] = {0};
> > +       struct rte_pci_addr *loc = &p->dev->addr;
> > +       int ret, vfio_dev_fd;
> > +
> > +       /* store PCI address string */
> > +       snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
> > +                       loc->domain, loc->bus, loc->devid,
> > + loc->function);
> > +
> > +       vfio_dev_fd = rte_intr_dev_fd_get(p->dev->intr_handle);
> > +       if (vfio_dev_fd < 0)
> > +               return -1;
> 
> This check is odd and does not seem related.
> 
> 
> > +
> > +       ret = rte_vfio_release_device(rte_pci_get_sysfs_path(), pci_addr,
> > +                                     vfio_dev_fd);
> > +       if (ret < 0) {
> > +               RTE_LOG(ERR, EAL, "Cannot release VFIO device\n");
> > +               return ret;
> > +       }
> 
> 
> --
> David Marchand