From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io0-f196.google.com (mail-io0-f196.google.com [209.85.223.196]) by dpdk.org (Postfix) with ESMTP id 5D45D2B84 for ; Fri, 21 Apr 2017 00:01:19 +0200 (CEST) Received: by mail-io0-f196.google.com with SMTP id d203so21439316iof.2 for ; Thu, 20 Apr 2017 15:01:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ozlabs-ru.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=7NO4vDuyRswyZtSNubMQ/2m+0fTLqMMfR2PUs5cxM44=; b=elCTwc5C7joyK9UMHDx3PrKm43774WCIx6haTWrJwoUTktjQ2ZyEw9V+I/PYTGcV5J aeQRNPXxvddI2E3bnm6tUeyFYAJxunV+J7IWihr2i/cUfl8ruTxMAo602i0yzwUuiLpZ wdknUov+v9RJ4YFDNK8Cx4aBpzesxdI0hTieZkQL1n9/zBzJ2RQpqjSJrKqhOsO4ahS4 PREHg6WJgIkY39xllqJbag45hAq5hIcQ/j+E8xfLzsQ/2xEbOaLMxwF3HeNtDtlyhgi2 kVMv5oIFjPUKdRyVKFShxd6H20iDeIuVIg8H2fu4AihOr5DIY1NptRJOH5UIJ/VZletr nX3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=7NO4vDuyRswyZtSNubMQ/2m+0fTLqMMfR2PUs5cxM44=; b=sJnaAlsV7Qj/89nNw2Dvc3Dd8PVY6YaoPCxmRnA6bkBkwr4HHvnoEyv4tAEKT0GuLm bHQILt4CQ2cLzE/kbVyxJQak37rf9ioqGQtTpnX6yTumO229ERMdeAsF7mEO6BBtLZQM 34Duzu3XG+qtnOuleMoxBVKltthpkCCDId6VwH4i8oqXThuRMPMSEA1p6DC4oVMqNiCi j95UD5cOrgArQs4OGThg3jXles6KcGoIMdz0H3KZ1pdbofeaxIT9wWC0oc7Sfv3JIUvY 7SGKBlQgCqL1B5OCkjH3cwYrtnjXPFHdiulvRekYW0eLma/yrPq8o1w3JnFgygQJCo9u 9IAQ== X-Gm-Message-State: AN3rC/69+ECW7Dl5eTqbHh3wlVMNkbLfuqNUD1qQBJaF24LGzK2eweoq p92DF99A4ZqBNA== X-Received: by 10.99.209.5 with SMTP id k5mr9368531pgg.58.1492725678642; Thu, 20 Apr 2017 15:01:18 -0700 (PDT) Received: from [192.168.1.22] (124-171-134-5.dyn.iinet.net.au. [124.171.134.5]) by smtp.googlemail.com with ESMTPSA id r1sm12187671pfg.12.2017.04.20.15.01.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Apr 2017 15:01:17 -0700 (PDT) To: Jonas Pfefferle1 Cc: dev@dpdk.org, Adrian Schuepbach , Gowrishankar Muthukrishnan References: <20170420072402.38106-1-aik@ozlabs.ru> <20170420072402.38106-6-aik@ozlabs.ru> <12566b0a-8f9a-4040-a37d-2a106e49adcf@ozlabs.ru> <6e669e2b-2cfd-078d-b6b0-5c3819fad796@ozlabs.ru> From: Alexey Kardashevskiy Message-ID: <54d14e1f-56a5-5fa5-9939-dfd466c45052@ozlabs.ru> Date: Fri, 21 Apr 2017 08:01:13 +1000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=koi8-r Content-Language: en-AU Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH dpdk 5/5] RFC: vfio/ppc64/spapr: Use correct bus addresses for DMA map X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Apr 2017 22:01:19 -0000 On 21/04/17 01:15, Jonas Pfefferle1 wrote: > Alexey Kardashevskiy wrote on 20/04/2017 16:22:01: > >> From: Alexey Kardashevskiy >> To: Jonas Pfefferle1 >> Cc: dev@dpdk.org, Gowrishankar Muthukrishnan >> , Adrian Schuepbach >> Date: 20/04/2017 16:22 >> Subject: Re: [PATCH dpdk 5/5] RFC: vfio/ppc64/spapr: Use correct bus >> addresses for DMA map >> >> On 20/04/17 23:25, Alexey Kardashevskiy wrote: >> > On 20/04/17 19:04, Jonas Pfefferle1 wrote: >> >> Alexey Kardashevskiy wrote on 20/04/2017 09:24:02: >> >> >> >>> From: Alexey Kardashevskiy >> >>> To: dev@dpdk.org >> >>> Cc: Alexey Kardashevskiy , JPF@zurich.ibm.com, >> >>> Gowrishankar Muthukrishnan >> >>> Date: 20/04/2017 09:24 >> >>> Subject: [PATCH dpdk 5/5] RFC: vfio/ppc64/spapr: Use correct bus >> >>> addresses for DMA map >> >>> >> >>> VFIO_IOMMU_SPAPR_TCE_CREATE ioctl() returns the actual bus address for >> >>> just created DMA window. It happens to start from zero because the > default >> >>> window is removed (leaving no windows) and new window starts from zero. >> >>> However this is not guaranteed and the new window may start from another >> >>> address, this adds an error check. >> >>> >> >>> Another issue is that IOVA passed to VFIO_IOMMU_MAP_DMA should be a PCI >> >>> bus address while in this case a physical address of a user page is used. >> >>> This changes IOVA to start from zero in a hope that the rest of DPDK >> >>> expects this. >> >> >> >> This is not the case. DPDK expects a 1:1 mapping PA==IOVA. It will use the >> >> phys_addr of the memory segment it got from /proc/self/pagemap cf. >> >> librte_eal/linuxapp/eal/eal_memory.c. We could try setting it here to the >> >> actual iova which basically makes the whole virtual to phyiscal mapping >> >> with pagemap unnecessary which I believe should be the case for VFIO >> >> anyway. Pagemap should only be needed when using pci_uio. >> > >> > >> > Ah, ok, makes sense now. But it sure needs a big fat comment there as it is >> > not obvious why host RAM address is used there as DMA window start is not >> > guaranteed. >> >> Well, either way there is some bug - ms[i].phys_addr and ms[i].addr_64 both >> have exact same value, in my setup it is 3fffb33c0000 which is a userspace >> address - at least ms[i].phys_addr must be physical address. > > This might be the case if you are not using hugetlbfs i.e. passing > "--no-huge" cf. eal_memory.c:980 > > /* hugetlbfs can be disabled */ > if (internal_config.no_hugetlbfs) { > addr = mmap(NULL, internal_config.memory, PROT_READ | PROT_WRITE, > MAP_PRIVATE | MAP_ANONYMOUS, 0, 0); > if (addr == MAP_FAILED) { > RTE_LOG(ERR, EAL, "%s: mmap() failed: %s\n", __func__, > strerror(errno)); > return -1; > } > mcfg->memseg[0].phys_addr = (phys_addr_t)(uintptr_t)addr; > mcfg->memseg[0].addr = addr; > mcfg->memseg[0].hugepage_sz = RTE_PGSIZE_4K; > mcfg->memseg[0].len = internal_config.memory; > mcfg->memseg[0].socket_id = 0; > return 0; > } > > If it fails to get the virt2phys mapping it actually assigns iovas starting > from 0 to the memory segments, cf. set_physaddrs eal_memory.c:263 Right, this is the case here. > >> >> >> > >> > >> >> >> >>> >> >>> Signed-off-by: Alexey Kardashevskiy >> >>> --- >> >>> lib/librte_eal/linuxapp/eal/eal_vfio.c | 12 ++++++++++-- >> >>> 1 file changed, 10 insertions(+), 2 deletions(-) >> >>> >> >>> diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c b/lib/ >> >>> librte_eal/linuxapp/eal/eal_vfio.c >> >>> index 46f951f4d..8b8e75c4f 100644 >> >>> --- a/lib/librte_eal/linuxapp/eal/eal_vfio.c >> >>> +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c >> >>> @@ -658,7 +658,7 @@ vfio_spapr_dma_map(int vfio_container_fd) >> >>> { >> >>> const struct rte_memseg *ms = rte_eal_get_physmem_layout(); >> >>> int i, ret; >> >>> - >> >>> + phys_addr_t io_offset; >> >>> struct vfio_iommu_spapr_register_memory reg = { >> >>> .argsz = sizeof(reg), >> >>> .flags = 0 >> >>> @@ -702,6 +702,13 @@ vfio_spapr_dma_map(int vfio_container_fd) >> >>> return -1; >> >>> } >> >>> >> >>> + io_offset = create.start_addr; >> >>> + if (io_offset) { >> >>> + RTE_LOG(ERR, EAL, " DMA offsets other than zero is not >> supported, " >> >>> + "new window is created at %lx\n", io_offset); >> >>> + return -1; >> >>> + } >> >>> + >> >>> /* map all DPDK segments for DMA. use 1:1 PA to IOVA mapping */ >> >>> for (i = 0; i < RTE_MAX_MEMSEG; i++) { >> >>> struct vfio_iommu_type1_dma_map dma_map; >> >>> @@ -723,7 +730,7 @@ vfio_spapr_dma_map(int vfio_container_fd) >> >>> dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map); >> >>> dma_map.vaddr = ms[i].addr_64; >> >>> dma_map.size = ms[i].len; >> >>> - dma_map.iova = ms[i].phys_addr; >> >>> + dma_map.iova = io_offset; >> >>> dma_map.flags = VFIO_DMA_MAP_FLAG_READ | >> >>> VFIO_DMA_MAP_FLAG_WRITE; >> >>> >> >>> @@ -735,6 +742,7 @@ vfio_spapr_dma_map(int vfio_container_fd) >> >>> return -1; >> >>> } >> >>> >> >>> + io_offset += dma_map.size; >> >>> } >> >>> >> >>> return 0; >> >>> -- >> >>> 2.11.0 >> >>> >> >> >> > >> > >> >> >> -- >> Alexey >> > -- Alexey