From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <3chas3@gmail.com> Received: from mail-it0-f42.google.com (mail-it0-f42.google.com [209.85.214.42]) by dpdk.org (Postfix) with ESMTP id F062AE5D for ; Mon, 27 Nov 2017 13:58:08 +0100 (CET) Received: by mail-it0-f42.google.com with SMTP id n134so20886976itg.1 for ; Mon, 27 Nov 2017 04:58:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=qcSDUNL7fLK23cH3LxebchtqER/1C1xIhWqPx0OFaLM=; b=PhREnm84olFrdNWgp3a192KPzcHjIc/Y5N8xQO88CfSk1ytX/b5wahpXfTMgQKt5S7 FKfjrPfzMy6h5cYSsr6b2bf6g1xq/bXiLz3VnigMrMv7xib+FLHrbnjL8Cnw9swQ3GEx L08yolmdO9qhaiFLayxabyYJHSOPLEaTOZYlmahml85ih+jRUneEBoNBuJ91Mes+8+2R qu3l4J7cZha/mJcH5ropErI5Ksw/HSY7wsheV71zjHTONcU+l2ygHC0e9aKf8gwX2Tyd YLgZ59XA3E6BcYTgwQGPmzqQfk+VYQvjQPV/36ASFbduRvU4aNgelPOGfUd5a9sL/bIq 87EQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=qcSDUNL7fLK23cH3LxebchtqER/1C1xIhWqPx0OFaLM=; b=RKW8Sz5H8gkZFoeos9UxaSJs9cGY+Cig+to80ZotTBfvXcRXEGdgMMPgHG6qFsGqjW ntZ+lE/sFA4jqjNlQNAIGnEVTc3whCAy6J1GKyi9J8cQriuB8HWGOslE+8voDPVIxU7M PDeAempJdHBe1RN6FqSwpcWp6/jYhk2RyZR6nBrKoZ74LhcmHG09R3TyrZp+lucw/ESH YWKtC3QXKkBv1wIaBfZzT77ACclIWHZImiIyIrzGP3kZpFIS6Fv3HWEdRni6CWezegrt 8rAqNW/U1iwkCLIg+Vu9eu8XPDaI3uMHa4rWN5N90GJwt91WEFErHx+y2I5ACy6+LFB/ 7cWg== X-Gm-Message-State: AJaThX6sPZsj1j6ynfxqs5QRkHoafcbxs9h+aDiNhnDpJKyUktQ9KzNj pqHRaZjkU2S8QKbHpNKvrjXQ+bWpj2Y94Fhahfc= X-Google-Smtp-Source: AGs4zMYbTdY/ovscUEVuqpvz2RuhUR+e/QXfeaO0PeZGL1CFpx1BiEYkuSSnyIBLI+qK1XLb0HbJcxuTJzh+qv5xLRQ= X-Received: by 10.36.95.5 with SMTP id r5mr22396319itb.70.1511787488207; Mon, 27 Nov 2017 04:58:08 -0800 (PST) MIME-Version: 1.0 Received: by 10.107.2.5 with HTTP; Mon, 27 Nov 2017 04:58:07 -0800 (PST) In-Reply-To: <039ED4275CED7440929022BC67E706115311683A@SHSMSX103.ccr.corp.intel.com> References: <20171124235718.6064-1-3chas3@gmail.com> <039ED4275CED7440929022BC67E706115311683A@SHSMSX103.ccr.corp.intel.com> From: Chas Williams <3chas3@gmail.com> Date: Mon, 27 Nov 2017 07:58:07 -0500 Message-ID: To: "Zhang, Qi Z" Cc: "dev@dpdk.org" , "skhare@vmware.com" , Chas Williams Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH] vfio: try physical address if virtual address fails X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Nov 2017 12:58:09 -0000 On Mon, Nov 27, 2017 at 4:30 AM, Zhang, Qi Z wrote: > Hi William: > > > -----Original Message----- > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Chas Williams > > Sent: Saturday, November 25, 2017 7:57 AM > > To: dev@dpdk.org > > Cc: skhare@vmware.com; Chas Williams <3chas3@gmail.com>; Chas > > Williams > > Subject: [dpdk-dev] [PATCH] vfio: try physical address if virtual > address fails > > > > Some machines appear to have buggy DMAR mappings. A typical mapping > > error looks like: > > > > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the > > mapped address (7fc4fa800000) > > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the > > mapped address (7fc4fa800000) > > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the > > mapped address (7fc4fa800000) > > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the > > mapped address (7fc4fa800000) > > > I met the same issue on some intel atom platform, the root cause is IOMMU > only support 39 bit virtual address. > Not sure retry with physical address will be the right fix. I saw > rte_eal_iova_mode is called at other place, it still take the virtual > address as the mapped result, does that break something? > So far the workaround may works by using --virtbase-addr to assign a > address in range explicitly (for example 0x70,0000,0000) > Regards > Qi > I don't think the IOVA usage elsewhere is an issue since I limited my changes to reworking what was done in commit e85a919286d2 which appears to be what broken things for me. It's not clear that passing --base-virtaddr is a workable solution. First, it's only a hint to mmap(). The kernel really can do whatever it wants. And then, what values do I pick? If one value fails, do I just restart and keep trying new values until it succeeds? I just want to fall back to the previous behavior. I know the physical mapping will succeed since the DMAR tables are limited to 36 bits in width. And yes, I should update the commit to blame the IOMMU width. > > > To work around this, attempt to do a physical address mapping if the > virtual > > address mapping fails. > > > > Fixes: e85a919286d2 ("vfio: honor IOVA mode before mapping") > > > > Signed-off-by: Chas Williams > > --- > > lib/librte_eal/linuxapp/eal/eal_vfio.c | 8 ++++++++ > > 1 file changed, 8 insertions(+) > > > > diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c > > b/lib/librte_eal/linuxapp/eal/eal_vfio.c > > index 58f0123..6250676 100644 > > --- a/lib/librte_eal/linuxapp/eal/eal_vfio.c > > +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c > > @@ -35,6 +35,7 @@ > > #include > > #include > > #include > > +#include > > > > #include > > #include > > @@ -702,6 +703,7 @@ vfio_type1_dma_map(int vfio_container_fd) > > /* map all DPDK segments for DMA. use 1:1 PA to IOVA mapping */ > > for (i = 0; i < RTE_MAX_MEMSEG; i++) { > > struct vfio_iommu_type1_dma_map dma_map; > > + int retried = false; > > > > if (ms[i].addr == NULL) > > break; > > @@ -716,9 +718,15 @@ vfio_type1_dma_map(int vfio_container_fd) > > dma_map.iova = ms[i].iova; > > dma_map.flags = VFIO_DMA_MAP_FLAG_READ | > > VFIO_DMA_MAP_FLAG_WRITE; > > > > +retry: > > ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, > > &dma_map); > > > > if (ret) { > > + if (!retried && rte_eal_iova_mode() == > RTE_IOVA_VA) { > > + dma_map.iova = ms[i].iova; > > > > + retried = true; > > + goto retry; > > + } > > RTE_LOG(ERR, EAL, " cannot set up DMA remapping, " > > "error %i (%s)\n", errno, > > strerror(errno)); > > -- > > 2.9.5 > Regards > Qi >