From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <3chas3@gmail.com> Received: from mail-it0-f66.google.com (mail-it0-f66.google.com [209.85.214.66]) by dpdk.org (Postfix) with ESMTP id 471752BE9 for ; Thu, 30 Nov 2017 03:03:12 +0100 (CET) Received: by mail-it0-f66.google.com with SMTP id u62so6486931ita.2 for ; Wed, 29 Nov 2017 18:03:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=gTqiVvujtBhItAMfyr5PsQzhjcPU0njBqV7Ir/2EMXw=; b=rZPsYjqao28cE2ZobvW6he/rfSqKWpQVgXI3QS8RMJsPVC0u5jS0CaM9ER3o5Kk5wL rw+oQenuDNRH1qLA98klq8XTgbPttWP+nWEv5fYUL/tAYRUHN1bz+4c8iPO0EPsK+DIY sMZKb5DpeUKTHArJcornC4H0xjj2XfXvoEy3j0rvKXIQBUOiLOGSnu6wVdIME+9MMG6H bz1pt6UDIu4krMSvToBXlasl4C8+ST8m8Mn0Ciysh4u0MurK+mt6RjSUL0pwZt6FjIRE yHn/Ij3i+WVqLLF4W3DD4nrZ+RZb8RfBqndd2h+1ExgoUH4U0YCBOKVhgBoXUkrp88kk Sv8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=gTqiVvujtBhItAMfyr5PsQzhjcPU0njBqV7Ir/2EMXw=; b=jBFZ9j5+8++pSOle13jhHc7ASDOQR/tyvHfPTBcG/8LJ8tFeAPzvXoJ7ZXxYxOpLsX +FwoE13TpxH6NTZDULyAsL2MbS0cnsneuzlAqHJ7QuoE5gRo2/vcUcN0G0Ft/9u2TrYX nz8OBkGgT+dO1E/J2e+1A64QIn/65XWpPEklOdB5zw9NXcbqWlaBbZ01YXYy7tqsG1wc JdeNoRpS3qtOaxxnJgb6wDQnpjnhmb/kvH3wdNPRwN9P8kgySLveKgGi+7kiWsjOerVC PIiu9n5PSF/2tpheVJiWrczyThjIHmCFJK13IhgKT7aOmM+t6uA2xm52lcOdl21erxmq WOMA== X-Gm-Message-State: AJaThX7iKcLgdtAo9jGRjQnHXA9YLTcvcwuELpOEGgJA3OYt7j6Z35EC bziWXCfG6H1bz5pPYTp8Q1hTdQV+Wa/suO5O6lQ= X-Google-Smtp-Source: AGs4zMa2kwxSQZanCyPCwYD8Jl8zu8MVp/uEBhr/cdWPdYPAbtfZNqSGeAC10H/nsb3sW8jHY/UNeGiiYEj8+4iLQXw= X-Received: by 10.36.108.83 with SMTP id w80mr1116988itb.3.1512007391314; Wed, 29 Nov 2017 18:03:11 -0800 (PST) MIME-Version: 1.0 Received: by 10.107.2.5 with HTTP; Wed, 29 Nov 2017 18:03:10 -0800 (PST) In-Reply-To: References: <20171124235718.6064-1-3chas3@gmail.com> <039ED4275CED7440929022BC67E706115311683A@SHSMSX103.ccr.corp.intel.com> From: Chas Williams <3chas3@gmail.com> Date: Wed, 29 Nov 2017 21:03:10 -0500 Message-ID: To: "Zhang, Qi Z" Cc: "dev@dpdk.org" , "skhare@vmware.com" , Chas Williams Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH] vfio: try physical address if virtual address fails X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Nov 2017 02:03:12 -0000 OK, I found a machine in our local collection that couldn't get by with this patch, so I will submitting a follow up that forces rte_eval_iova_mode() into PA mode. On Mon, Nov 27, 2017 at 7:58 AM, Chas Williams <3chas3@gmail.com> wrote: > > > On Mon, Nov 27, 2017 at 4:30 AM, Zhang, Qi Z wrote: > >> Hi William: >> >> > -----Original Message----- >> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Chas Williams >> > Sent: Saturday, November 25, 2017 7:57 AM >> > To: dev@dpdk.org >> > Cc: skhare@vmware.com; Chas Williams <3chas3@gmail.com>; Chas >> > Williams >> > Subject: [dpdk-dev] [PATCH] vfio: try physical address if virtual >> address fails >> > >> > Some machines appear to have buggy DMAR mappings. A typical mapping >> > error looks like: >> > >> > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the >> > mapped address (7fc4fa800000) >> > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the >> > mapped address (7fc4fa800000) >> > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the >> > mapped address (7fc4fa800000) >> > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the >> > mapped address (7fc4fa800000) >> > >> I met the same issue on some intel atom platform, the root cause is IOMMU >> only support 39 bit virtual address. >> Not sure retry with physical address will be the right fix. I saw >> rte_eal_iova_mode is called at other place, it still take the virtual >> address as the mapped result, does that break something? >> So far the workaround may works by using --virtbase-addr to assign a >> address in range explicitly (for example 0x70,0000,0000) >> Regards >> Qi >> > > I don't think the IOVA usage elsewhere is an issue since I limited my > changes to reworking what was done in commit e85a919286d2 which appears to > be what broken things for me. > > It's not clear that passing --base-virtaddr is a workable solution. > First, it's only a hint to mmap(). The kernel really can do whatever it > wants. And then, what values do I pick? If one value fails, do I just > restart and keep trying new values until it succeeds? I just want to fall > back to the previous behavior. I know the physical mapping will succeed > since the DMAR tables are limited to 36 bits in width. > > And yes, I should update the commit to blame the IOMMU width. > > >> >> > To work around this, attempt to do a physical address mapping if the >> virtual >> > address mapping fails. >> > >> > Fixes: e85a919286d2 ("vfio: honor IOVA mode before mapping") >> > >> > Signed-off-by: Chas Williams >> > --- >> > lib/librte_eal/linuxapp/eal/eal_vfio.c | 8 ++++++++ >> > 1 file changed, 8 insertions(+) >> > >> > diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c >> > b/lib/librte_eal/linuxapp/eal/eal_vfio.c >> > index 58f0123..6250676 100644 >> > --- a/lib/librte_eal/linuxapp/eal/eal_vfio.c >> > +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c >> > @@ -35,6 +35,7 @@ >> > #include >> > #include >> > #include >> > +#include >> > >> > #include >> > #include >> > @@ -702,6 +703,7 @@ vfio_type1_dma_map(int vfio_container_fd) >> > /* map all DPDK segments for DMA. use 1:1 PA to IOVA mapping */ >> > for (i = 0; i < RTE_MAX_MEMSEG; i++) { >> > struct vfio_iommu_type1_dma_map dma_map; >> > + int retried = false; >> > >> > if (ms[i].addr == NULL) >> > break; >> > @@ -716,9 +718,15 @@ vfio_type1_dma_map(int vfio_container_fd) >> > dma_map.iova = ms[i].iova; >> > dma_map.flags = VFIO_DMA_MAP_FLAG_READ | >> > VFIO_DMA_MAP_FLAG_WRITE; >> > >> > +retry: >> > ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, >> > &dma_map); >> > >> > if (ret) { >> > + if (!retried && rte_eal_iova_mode() == >> RTE_IOVA_VA) { >> > + dma_map.iova = ms[i].iova; >> >> >> > + retried = true; >> > + goto retry; >> > + } >> > RTE_LOG(ERR, EAL, " cannot set up DMA remapping, >> " >> > "error %i (%s)\n", errno, >> > strerror(errno)); >> > -- >> > 2.9.5 >> Regards >> Qi >> > >