Re: [dpdk-dev] [PATCH] vfio: try physical address if virtual address fails

[Chas Williams <3chas3@gmail.com>]

On Mon, Nov 27, 2017 at 4:30 AM, Zhang, Qi Z <qi.z.zhang@intel.com> wrote:

> Hi William:
>
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Chas Williams
> > Sent: Saturday, November 25, 2017 7:57 AM
> > To: dev@dpdk.org
> > Cc: skhare@vmware.com; Chas Williams <3chas3@gmail.com>; Chas
> > Williams <chas3@att.com>
> > Subject: [dpdk-dev] [PATCH] vfio: try physical address if virtual
> address fails
> >
> > Some machines appear to have buggy DMAR mappings.  A typical mapping
> > error looks like:
> >
> >     DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
> > mapped address (7fc4fa800000)
> >     DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
> > mapped address (7fc4fa800000)
> >     DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
> > mapped address (7fc4fa800000)
> >     DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
> > mapped address (7fc4fa800000)
> >
> I met the same issue on some intel atom platform, the root cause is IOMMU
> only support 39 bit virtual address.
> Not sure retry with physical address will be the right fix. I saw
> rte_eal_iova_mode is called at other place, it still take the virtual
> address as the mapped result, does that break something?
> So far the workaround may works by using --virtbase-addr to assign a
> address in range explicitly (for example 0x70,0000,0000)
> Regards
> Qi
>

I don't think the IOVA usage elsewhere is an issue since I limited my
changes to reworking what was done in commit e85a919286d2 which appears to
be what broken things for me.

It's not clear that passing --base-virtaddr is a workable solution.  First,
it's only a hint to mmap().  The kernel really can do whatever it wants.
And then, what values do I pick?  If one value fails, do I just restart and
keep trying new values until it succeeds?  I just want to fall back to the
previous behavior.  I know the physical mapping will succeed since the DMAR
tables are limited to 36 bits in width.

And yes, I should update the commit to blame the IOMMU width.


>
> > To work around this, attempt to do a physical address mapping if the
> virtual
> > address mapping fails.
> >
> > Fixes: e85a919286d2 ("vfio: honor IOVA mode before mapping")
> >
> > Signed-off-by: Chas Williams <chas3@att.com>
> > ---
> >  lib/librte_eal/linuxapp/eal/eal_vfio.c | 8 ++++++++
> >  1 file changed, 8 insertions(+)
> >
> > diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c
> > b/lib/librte_eal/linuxapp/eal/eal_vfio.c
> > index 58f0123..6250676 100644
> > --- a/lib/librte_eal/linuxapp/eal/eal_vfio.c
> > +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c
> > @@ -35,6 +35,7 @@
> >  #include <fcntl.h>
> >  #include <unistd.h>
> >  #include <sys/ioctl.h>
> > +#include <stdbool.h>
> >
> >  #include <rte_log.h>
> >  #include <rte_memory.h>
> > @@ -702,6 +703,7 @@ vfio_type1_dma_map(int vfio_container_fd)
> >       /* map all DPDK segments for DMA. use 1:1 PA to IOVA mapping */
> >       for (i = 0; i < RTE_MAX_MEMSEG; i++) {
> >               struct vfio_iommu_type1_dma_map dma_map;
> > +             int retried = false;
> >
> >               if (ms[i].addr == NULL)
> >                       break;
> > @@ -716,9 +718,15 @@ vfio_type1_dma_map(int vfio_container_fd)
> >                       dma_map.iova = ms[i].iova;
> >               dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
> > VFIO_DMA_MAP_FLAG_WRITE;
> >
> > +retry:
> >               ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA,
> > &dma_map);
> >
> >               if (ret) {
> > +                     if (!retried && rte_eal_iova_mode() ==
> RTE_IOVA_VA) {
> > +                             dma_map.iova = ms[i].iova;
>
>
> > +                             retried = true;
> > +                             goto retry;
> > +                     }
> >                       RTE_LOG(ERR, EAL, "  cannot set up DMA remapping, "
> >                                         "error %i (%s)\n", errno,
> >                                         strerror(errno));
> > --
> > 2.9.5
> Regards
> Qi
>