From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-it0-f67.google.com (mail-it0-f67.google.com [209.85.214.67]) by dpdk.org (Postfix) with ESMTP id 56F282BE9 for ; Tue, 8 Aug 2017 09:38:07 +0200 (CEST) Received: by mail-it0-f67.google.com with SMTP id t78so2032635ita.1 for ; Tue, 08 Aug 2017 00:38:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ozlabs-ru.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=UIfQntpCX0TNzimul1oYCqa0X51HBt4ew+5WjMi3CDc=; b=J4ZrezEymsXimHczGVtOQh61CnPoFm2krG/DuHF1j5c7ujDT3yDQjhwx2NkTjBNT2E tO0n2utaLffPTb65FvXgqo8DAGLZ3rO+4c/bFYWo6fcdlSzk3/NOtLsXQ4Ut0IoNTENM 9irJ4pgMeHymnQpX51ahJ6tEXwrUQm2Ip16jqX0EH3oPJmg1J9uiqtjEA98NqDGqxXLV MFp79VKKhy6aX6/epSU53YNAMEf0UW4YAhrTucsJDt8Pv22KSGeKbV6CkSlThLuDmng9 +nkMpTxy5mrukwChNlrBNlOgnj4k274TnILjQiM1lOACYB+xcgAbJTAa91x+ZBtqe1b0 pLWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=UIfQntpCX0TNzimul1oYCqa0X51HBt4ew+5WjMi3CDc=; b=TUSIZ+/LnXENvNkmyoeXkUKDui94jA5ZCZySBMB+zCJaOZCWVn7idece8mBebCmNI5 G4kACQT/W8yYd+MYGbojUj2/z+ci8VMBXloeIBFYsoB+jmt+ol3HtH1b0SgA4zs19jQp NPUZ/Y8IoI4QqELS3XB9H7mD3p7H8B3grq9woufpWnt/0+DmX94ksTjsyFRr3f8aHXrm 789PRTgQIhTNabbSKY/jAmR08scs5apzSzfxh55bhrgp1qmUphLIsUqKT8RUrryv/YsL JcG4Y7vrnQpUARjbwaNQ8AXUXUaJ5QNkf2nSeon58YD2wU6/xUbnDAGAAL8fPT0BJ9kD FRkg== X-Gm-Message-State: AHYfb5jLf77uwSXUPFskJczyA474SZ/QOe4afJC5ocpv3JZv1/mnnESz v37bMx8NqJW/2rucPwM= X-Received: by 10.36.112.12 with SMTP id f12mr3218374itc.8.1502177886292; Tue, 08 Aug 2017 00:38:06 -0700 (PDT) Received: from [10.61.2.175] ([122.99.82.10]) by smtp.googlemail.com with ESMTPSA id k94sm363940ioi.17.2017.08.08.00.38.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 Aug 2017 00:38:05 -0700 (PDT) To: Jonas Pfefferle , anatoly.burakov@intel.com Cc: dev@dpdk.org References: <1502118665-27439-1-git-send-email-jpf@zurich.ibm.com> From: Alexey Kardashevskiy Message-ID: <33a0b391-0a97-0392-39c1-377af4196c69@ozlabs.ru> Date: Tue, 8 Aug 2017 17:38:00 +1000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <1502118665-27439-1-git-send-email-jpf@zurich.ibm.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-AU Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH] vfio: fix sPAPR IOMMU DMA window size X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Aug 2017 07:38:07 -0000 On 08/08/17 01:11, Jonas Pfefferle wrote: > DMA window size needs to be big enough to span all memory segment's > physical addresses. We do not need multiple levels of IOMMU tables > as we already span ~70TB of physical memory with 16MB hugepages. > > Signed-off-by: Jonas Pfefferle > --- > lib/librte_eal/linuxapp/eal/eal_vfio.c | 25 ++++++++++++++++++++++--- > 1 file changed, 22 insertions(+), 3 deletions(-) > > diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c b/lib/librte_eal/linuxapp/eal/eal_vfio.c > index 946df7e..8502216 100644 > --- a/lib/librte_eal/linuxapp/eal/eal_vfio.c > +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c > @@ -722,6 +722,18 @@ vfio_type1_dma_map(int vfio_container_fd) > return 0; > } > > +static uint64_t > +roundup_next_pow2(uint64_t n) > +{ > + uint32_t i; > + > + n--; > + for (i = 1; i < sizeof(n) * CHAR_BIT; i += i) > + n |= n >> i; > + > + return ++n; > +} > + wow :) QEMU does it using __builtin_ctzll() (used below for the page_shift) without a loop: https://git.qemu.org/gitweb.cgi?p=qemu.git;a=blob;f=include/qemu/host-utils.h;h=95cf4f4163e50457cdf808263065ca5ef3f935da;hb=f22ab6cb0c47bd2a2785b7d58130949bd7d8d9af#l382 Anyway, seems working. Reviewed-by: Alexey Kardashevskiy > static int > vfio_spapr_dma_map(int vfio_container_fd) > { > @@ -759,10 +771,12 @@ vfio_spapr_dma_map(int vfio_container_fd) > return -1; > } > > - /* calculate window size based on number of hugepages configured */ > - create.window_size = rte_eal_get_physmem_size(); > + /* physicaly pages are sorted descending i.e. ms[0].phys_addr is max */ > + /* create DMA window from 0 to max(phys_addr + len) */ > + /* sPAPR requires window size to be a power of 2 */ > + create.window_size = roundup_next_pow2(ms[0].phys_addr + ms[0].len); > create.page_shift = __builtin_ctzll(ms->hugepage_sz); > - create.levels = 2; > + create.levels = 1; > > ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create); > if (ret) { > @@ -771,6 +785,11 @@ vfio_spapr_dma_map(int vfio_container_fd) > return -1; > } > > + if (create.start_addr != 0) { > + RTE_LOG(ERR, EAL, " DMA window start address != 0\n"); > + return -1; > + } > + > /* map all DPDK segments for DMA. use 1:1 PA to IOVA mapping */ > for (i = 0; i < RTE_MAX_MEMSEG; i++) { > struct vfio_iommu_type1_dma_map dma_map; > -- Alexey