* [dpdk-dev] [PATCH] vfio: try physical address if virtual address fails
@ 2017-11-24 23:57 Chas Williams
2017-11-27 9:30 ` Zhang, Qi Z
0 siblings, 1 reply; 4+ messages in thread
From: Chas Williams @ 2017-11-24 23:57 UTC (permalink / raw)
To: dev; +Cc: skhare, Chas Williams, Chas Williams
Some machines appear to have buggy DMAR mappings. A typical mapping
error looks like:
DMAR: intel_iommu_map: iommu width (39) is not sufficient for the mapped address (7fc4fa800000)
DMAR: intel_iommu_map: iommu width (39) is not sufficient for the mapped address (7fc4fa800000)
DMAR: intel_iommu_map: iommu width (39) is not sufficient for the mapped address (7fc4fa800000)
DMAR: intel_iommu_map: iommu width (39) is not sufficient for the mapped address (7fc4fa800000)
To work around this, attempt to do a physical address mapping if the
virtual address mapping fails.
Fixes: e85a919286d2 ("vfio: honor IOVA mode before mapping")
Signed-off-by: Chas Williams <chas3@att.com>
---
lib/librte_eal/linuxapp/eal/eal_vfio.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c b/lib/librte_eal/linuxapp/eal/eal_vfio.c
index 58f0123..6250676 100644
--- a/lib/librte_eal/linuxapp/eal/eal_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c
@@ -35,6 +35,7 @@
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
+#include <stdbool.h>
#include <rte_log.h>
#include <rte_memory.h>
@@ -702,6 +703,7 @@ vfio_type1_dma_map(int vfio_container_fd)
/* map all DPDK segments for DMA. use 1:1 PA to IOVA mapping */
for (i = 0; i < RTE_MAX_MEMSEG; i++) {
struct vfio_iommu_type1_dma_map dma_map;
+ int retried = false;
if (ms[i].addr == NULL)
break;
@@ -716,9 +718,15 @@ vfio_type1_dma_map(int vfio_container_fd)
dma_map.iova = ms[i].iova;
dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
+retry:
ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
if (ret) {
+ if (!retried && rte_eal_iova_mode() == RTE_IOVA_VA) {
+ dma_map.iova = ms[i].iova;
+ retried = true;
+ goto retry;
+ }
RTE_LOG(ERR, EAL, " cannot set up DMA remapping, "
"error %i (%s)\n", errno,
strerror(errno));
--
2.9.5
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [dpdk-dev] [PATCH] vfio: try physical address if virtual address fails
2017-11-24 23:57 [dpdk-dev] [PATCH] vfio: try physical address if virtual address fails Chas Williams
@ 2017-11-27 9:30 ` Zhang, Qi Z
2017-11-27 12:58 ` Chas Williams
0 siblings, 1 reply; 4+ messages in thread
From: Zhang, Qi Z @ 2017-11-27 9:30 UTC (permalink / raw)
To: Chas Williams, dev; +Cc: skhare, Chas Williams
Hi William:
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Chas Williams
> Sent: Saturday, November 25, 2017 7:57 AM
> To: dev@dpdk.org
> Cc: skhare@vmware.com; Chas Williams <3chas3@gmail.com>; Chas
> Williams <chas3@att.com>
> Subject: [dpdk-dev] [PATCH] vfio: try physical address if virtual address fails
>
> Some machines appear to have buggy DMAR mappings. A typical mapping
> error looks like:
>
> DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
> mapped address (7fc4fa800000)
> DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
> mapped address (7fc4fa800000)
> DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
> mapped address (7fc4fa800000)
> DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
> mapped address (7fc4fa800000)
>
I met the same issue on some intel atom platform, the root cause is IOMMU only support 39 bit virtual address.
Not sure retry with physical address will be the right fix. I saw rte_eal_iova_mode is called at other place, it still take the virtual address as the mapped result, does that break something?
So far the workaround may works by using --virtbase-addr to assign a address in range explicitly (for example 0x70,0000,0000)
Regards
Qi
> To work around this, attempt to do a physical address mapping if the virtual
> address mapping fails.
>
> Fixes: e85a919286d2 ("vfio: honor IOVA mode before mapping")
>
> Signed-off-by: Chas Williams <chas3@att.com>
> ---
> lib/librte_eal/linuxapp/eal/eal_vfio.c | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c
> b/lib/librte_eal/linuxapp/eal/eal_vfio.c
> index 58f0123..6250676 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_vfio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c
> @@ -35,6 +35,7 @@
> #include <fcntl.h>
> #include <unistd.h>
> #include <sys/ioctl.h>
> +#include <stdbool.h>
>
> #include <rte_log.h>
> #include <rte_memory.h>
> @@ -702,6 +703,7 @@ vfio_type1_dma_map(int vfio_container_fd)
> /* map all DPDK segments for DMA. use 1:1 PA to IOVA mapping */
> for (i = 0; i < RTE_MAX_MEMSEG; i++) {
> struct vfio_iommu_type1_dma_map dma_map;
> + int retried = false;
>
> if (ms[i].addr == NULL)
> break;
> @@ -716,9 +718,15 @@ vfio_type1_dma_map(int vfio_container_fd)
> dma_map.iova = ms[i].iova;
> dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
> VFIO_DMA_MAP_FLAG_WRITE;
>
> +retry:
> ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA,
> &dma_map);
>
> if (ret) {
> + if (!retried && rte_eal_iova_mode() == RTE_IOVA_VA) {
> + dma_map.iova = ms[i].iova;
> + retried = true;
> + goto retry;
> + }
> RTE_LOG(ERR, EAL, " cannot set up DMA remapping, "
> "error %i (%s)\n", errno,
> strerror(errno));
> --
> 2.9.5
Regards
Qi
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [dpdk-dev] [PATCH] vfio: try physical address if virtual address fails
2017-11-27 9:30 ` Zhang, Qi Z
@ 2017-11-27 12:58 ` Chas Williams
2017-11-30 2:03 ` Chas Williams
0 siblings, 1 reply; 4+ messages in thread
From: Chas Williams @ 2017-11-27 12:58 UTC (permalink / raw)
To: Zhang, Qi Z; +Cc: dev, skhare, Chas Williams
On Mon, Nov 27, 2017 at 4:30 AM, Zhang, Qi Z <qi.z.zhang@intel.com> wrote:
> Hi William:
>
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Chas Williams
> > Sent: Saturday, November 25, 2017 7:57 AM
> > To: dev@dpdk.org
> > Cc: skhare@vmware.com; Chas Williams <3chas3@gmail.com>; Chas
> > Williams <chas3@att.com>
> > Subject: [dpdk-dev] [PATCH] vfio: try physical address if virtual
> address fails
> >
> > Some machines appear to have buggy DMAR mappings. A typical mapping
> > error looks like:
> >
> > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
> > mapped address (7fc4fa800000)
> > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
> > mapped address (7fc4fa800000)
> > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
> > mapped address (7fc4fa800000)
> > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
> > mapped address (7fc4fa800000)
> >
> I met the same issue on some intel atom platform, the root cause is IOMMU
> only support 39 bit virtual address.
> Not sure retry with physical address will be the right fix. I saw
> rte_eal_iova_mode is called at other place, it still take the virtual
> address as the mapped result, does that break something?
> So far the workaround may works by using --virtbase-addr to assign a
> address in range explicitly (for example 0x70,0000,0000)
> Regards
> Qi
>
I don't think the IOVA usage elsewhere is an issue since I limited my
changes to reworking what was done in commit e85a919286d2 which appears to
be what broken things for me.
It's not clear that passing --base-virtaddr is a workable solution. First,
it's only a hint to mmap(). The kernel really can do whatever it wants.
And then, what values do I pick? If one value fails, do I just restart and
keep trying new values until it succeeds? I just want to fall back to the
previous behavior. I know the physical mapping will succeed since the DMAR
tables are limited to 36 bits in width.
And yes, I should update the commit to blame the IOMMU width.
>
> > To work around this, attempt to do a physical address mapping if the
> virtual
> > address mapping fails.
> >
> > Fixes: e85a919286d2 ("vfio: honor IOVA mode before mapping")
> >
> > Signed-off-by: Chas Williams <chas3@att.com>
> > ---
> > lib/librte_eal/linuxapp/eal/eal_vfio.c | 8 ++++++++
> > 1 file changed, 8 insertions(+)
> >
> > diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c
> > b/lib/librte_eal/linuxapp/eal/eal_vfio.c
> > index 58f0123..6250676 100644
> > --- a/lib/librte_eal/linuxapp/eal/eal_vfio.c
> > +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c
> > @@ -35,6 +35,7 @@
> > #include <fcntl.h>
> > #include <unistd.h>
> > #include <sys/ioctl.h>
> > +#include <stdbool.h>
> >
> > #include <rte_log.h>
> > #include <rte_memory.h>
> > @@ -702,6 +703,7 @@ vfio_type1_dma_map(int vfio_container_fd)
> > /* map all DPDK segments for DMA. use 1:1 PA to IOVA mapping */
> > for (i = 0; i < RTE_MAX_MEMSEG; i++) {
> > struct vfio_iommu_type1_dma_map dma_map;
> > + int retried = false;
> >
> > if (ms[i].addr == NULL)
> > break;
> > @@ -716,9 +718,15 @@ vfio_type1_dma_map(int vfio_container_fd)
> > dma_map.iova = ms[i].iova;
> > dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
> > VFIO_DMA_MAP_FLAG_WRITE;
> >
> > +retry:
> > ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA,
> > &dma_map);
> >
> > if (ret) {
> > + if (!retried && rte_eal_iova_mode() ==
> RTE_IOVA_VA) {
> > + dma_map.iova = ms[i].iova;
>
>
> > + retried = true;
> > + goto retry;
> > + }
> > RTE_LOG(ERR, EAL, " cannot set up DMA remapping, "
> > "error %i (%s)\n", errno,
> > strerror(errno));
> > --
> > 2.9.5
> Regards
> Qi
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [dpdk-dev] [PATCH] vfio: try physical address if virtual address fails
2017-11-27 12:58 ` Chas Williams
@ 2017-11-30 2:03 ` Chas Williams
0 siblings, 0 replies; 4+ messages in thread
From: Chas Williams @ 2017-11-30 2:03 UTC (permalink / raw)
To: Zhang, Qi Z; +Cc: dev, skhare, Chas Williams
OK, I found a machine in our local collection that couldn't get by with
this patch, so I will submitting a follow up that forces
rte_eval_iova_mode() into PA mode.
On Mon, Nov 27, 2017 at 7:58 AM, Chas Williams <3chas3@gmail.com> wrote:
>
>
> On Mon, Nov 27, 2017 at 4:30 AM, Zhang, Qi Z <qi.z.zhang@intel.com> wrote:
>
>> Hi William:
>>
>> > -----Original Message-----
>> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Chas Williams
>> > Sent: Saturday, November 25, 2017 7:57 AM
>> > To: dev@dpdk.org
>> > Cc: skhare@vmware.com; Chas Williams <3chas3@gmail.com>; Chas
>> > Williams <chas3@att.com>
>> > Subject: [dpdk-dev] [PATCH] vfio: try physical address if virtual
>> address fails
>> >
>> > Some machines appear to have buggy DMAR mappings. A typical mapping
>> > error looks like:
>> >
>> > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
>> > mapped address (7fc4fa800000)
>> > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
>> > mapped address (7fc4fa800000)
>> > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
>> > mapped address (7fc4fa800000)
>> > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
>> > mapped address (7fc4fa800000)
>> >
>> I met the same issue on some intel atom platform, the root cause is IOMMU
>> only support 39 bit virtual address.
>> Not sure retry with physical address will be the right fix. I saw
>> rte_eal_iova_mode is called at other place, it still take the virtual
>> address as the mapped result, does that break something?
>> So far the workaround may works by using --virtbase-addr to assign a
>> address in range explicitly (for example 0x70,0000,0000)
>> Regards
>> Qi
>>
>
> I don't think the IOVA usage elsewhere is an issue since I limited my
> changes to reworking what was done in commit e85a919286d2 which appears to
> be what broken things for me.
>
> It's not clear that passing --base-virtaddr is a workable solution.
> First, it's only a hint to mmap(). The kernel really can do whatever it
> wants. And then, what values do I pick? If one value fails, do I just
> restart and keep trying new values until it succeeds? I just want to fall
> back to the previous behavior. I know the physical mapping will succeed
> since the DMAR tables are limited to 36 bits in width.
>
> And yes, I should update the commit to blame the IOMMU width.
>
>
>>
>> > To work around this, attempt to do a physical address mapping if the
>> virtual
>> > address mapping fails.
>> >
>> > Fixes: e85a919286d2 ("vfio: honor IOVA mode before mapping")
>> >
>> > Signed-off-by: Chas Williams <chas3@att.com>
>> > ---
>> > lib/librte_eal/linuxapp/eal/eal_vfio.c | 8 ++++++++
>> > 1 file changed, 8 insertions(+)
>> >
>> > diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c
>> > b/lib/librte_eal/linuxapp/eal/eal_vfio.c
>> > index 58f0123..6250676 100644
>> > --- a/lib/librte_eal/linuxapp/eal/eal_vfio.c
>> > +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c
>> > @@ -35,6 +35,7 @@
>> > #include <fcntl.h>
>> > #include <unistd.h>
>> > #include <sys/ioctl.h>
>> > +#include <stdbool.h>
>> >
>> > #include <rte_log.h>
>> > #include <rte_memory.h>
>> > @@ -702,6 +703,7 @@ vfio_type1_dma_map(int vfio_container_fd)
>> > /* map all DPDK segments for DMA. use 1:1 PA to IOVA mapping */
>> > for (i = 0; i < RTE_MAX_MEMSEG; i++) {
>> > struct vfio_iommu_type1_dma_map dma_map;
>> > + int retried = false;
>> >
>> > if (ms[i].addr == NULL)
>> > break;
>> > @@ -716,9 +718,15 @@ vfio_type1_dma_map(int vfio_container_fd)
>> > dma_map.iova = ms[i].iova;
>> > dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
>> > VFIO_DMA_MAP_FLAG_WRITE;
>> >
>> > +retry:
>> > ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA,
>> > &dma_map);
>> >
>> > if (ret) {
>> > + if (!retried && rte_eal_iova_mode() ==
>> RTE_IOVA_VA) {
>> > + dma_map.iova = ms[i].iova;
>>
>>
>> > + retried = true;
>> > + goto retry;
>> > + }
>> > RTE_LOG(ERR, EAL, " cannot set up DMA remapping,
>> "
>> > "error %i (%s)\n", errno,
>> > strerror(errno));
>> > --
>> > 2.9.5
>> Regards
>> Qi
>>
>
>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2017-11-30 2:03 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-11-24 23:57 [dpdk-dev] [PATCH] vfio: try physical address if virtual address fails Chas Williams
2017-11-27 9:30 ` Zhang, Qi Z
2017-11-27 12:58 ` Chas Williams
2017-11-30 2:03 ` Chas Williams
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).