From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-f48.google.com (mail-ed1-f48.google.com [209.85.208.48]) by dpdk.org (Postfix) with ESMTP id 924261B006 for ; Thu, 28 Jun 2018 12:27:52 +0200 (CEST) Received: by mail-ed1-f48.google.com with SMTP id m21-v6so147303eds.10 for ; Thu, 28 Jun 2018 03:27:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=KDsSPCHIu6CM/t+9G1Cq/hDdJtEoj5mD4aJLI+F6oMs=; b=giD6KYpvs2j4iJmZFAWyBa/EkY8C3+QrFVPCVrO8SaonpoIA4G2Q/D8ALyf7Um7u5B DzG4GkCnwJBnEjNmeXsSO29z8H+mqhblKk2V1tVnHqzY8WpDMrSh0wiDGuHNdNTVzlfM P/eDZ+sfAgl5BSvbC5DhUOfp+aheR3d6yQVwqo2UBV0vLSE7lke4o7QPrpzFt6NBdDYw LQx69BIMo0xct06yzm3WwmTV+H80/ZoYQgbUVUOxuIn7dU18F53UhE3D0a/FY2tTRajr yCMB4mQqTqAqm4W4YaEmKQQ5W23GCSaK1Sl3vJLQIxZ3kiUxuoMaOZ7+eEyHp0Jcyday 6BrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=KDsSPCHIu6CM/t+9G1Cq/hDdJtEoj5mD4aJLI+F6oMs=; b=VzMZcfZEiK3m0TN4yA46x8bQgAt7vhWJhW4VfWSCobG7Wp1e/+VVWJwRvyPhxQrFnK +tZ+zZHnO1q8XJIUhJTWDOMMptxUCretL6P+d+x0P5QB21eBGqj38ZoLqlNUfwrDzgLx ph3v4iJqzfwC7zTisvy/5Xzo32ULUFCVGPNmm+eEk/nI7pQCz6uLUlhFYmN8OHOBu2go blDa7L6BEHpzSXovyEoC3PT82ckptG+J79Yr8yhY2+WDt2ZEnK8yeNtB+PZ4BRSlYqjE I0yRPINjnJzM9nJ6FduqcYDjbIbmHlYWyLr5kc4Gqyczygf8vBgRV2suntL5oeWtD7pW vQcA== X-Gm-Message-State: APt69E3js20twjSVjAxnv3ksv8oH3Bej+4P/S6I3JHG5o8sBxXFlFRww S0SwDc4RLWqdlG44sa8Zs0BZs73aOyBi9ONjL3ThIw== X-Google-Smtp-Source: AAOMgpc6Ar022ij36yCClJFvX1MxqwLP3MP4nzHteecikSMgwK1ipP0Txbfei14uqHeoTCjtcvJzrHvYhhRllOvsDO8= X-Received: by 2002:a50:adfd:: with SMTP id b58-v6mr8887952edd.168.1530181672319; Thu, 28 Jun 2018 03:27:52 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a50:b194:0:0:0:0:0 with HTTP; Thu, 28 Jun 2018 03:27:51 -0700 (PDT) In-Reply-To: <04086169-4102-2b23-c9c5-4be1784ef7c3@intel.com> References: <1530034653-28299-1-git-send-email-alejandro.lucero@netronome.com> <552f939e-f28e-0648-1796-8f42269887a2@intel.com> <03046f23-2466-cbb7-ae2b-f2770d5c6b0f@intel.com> <35c86511-7bf7-4840-d7ba-8362ddefc8ec@intel.com> <04086169-4102-2b23-c9c5-4be1784ef7c3@intel.com> From: Alejandro Lucero Date: Thu, 28 Jun 2018 11:27:51 +0100 Message-ID: To: "Burakov, Anatoly" Cc: dev , stable@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-stable] [RFC] Add support for device dma mask X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jun 2018 10:27:52 -0000 On Thu, Jun 28, 2018 at 11:03 AM, Burakov, Anatoly < anatoly.burakov@intel.com> wrote: > On 28-Jun-18 10:56 AM, Alejandro Lucero wrote: > >> >> >> On Thu, Jun 28, 2018 at 9:54 AM, Burakov, Anatoly < >> anatoly.burakov@intel.com > wrote: >> >> On 27-Jun-18 5:52 PM, Alejandro Lucero wrote: >> >> >> >> On Wed, Jun 27, 2018 at 2:24 PM, Burakov, Anatoly >> >> > >> wrote: >> >> On 27-Jun-18 11:13 AM, Alejandro Lucero wrote: >> >> >> >> On Wed, Jun 27, 2018 at 9:17 AM, Burakov, Anatoly >> > >> > > >> > >> >> > >>> wrote: >> >> On 26-Jun-18 6:37 PM, Alejandro Lucero wrote: >> >> This RFC tries to handle devices with addressing >> limitations. >> NFP devices >> 4000/6000 can just handle addresses with 40 >> bits implying >> problems for handling >> physical address when machines have more than >> 1TB of >> memory. But >> because how >> iovas are configured, which can be equivalent >> to physical >> addresses or based on >> virtual addresses, this can be a more likely >> problem. >> >> I tried to solve this some time ago: >> >> https://www.mail-archive.com/dev@dpdk.org/msg45214.html >> >> > dev@dpdk.org/msg45214.html >> > >> > dev@dpdk.org/msg45214.html >> >> > dev@dpdk.org/msg45214.html >> >> >> >> It was delayed because there was some changes in >> progress with >> EAL device >> handling, and, being honest, I completely >> forgot about this >> until now, when >> I have had to work on supporting NFP devices >> with DPDK and >> non-root users. >> >> I was working on a patch for being applied on >> main DPDK >> branch >> upstream, but >> because changes to memory initialization >> during the >> last months, >> this can not >> be backported to stable versions, at least the >> part >> where the >> hugepages iovas >> are checked. >> >> I realize stable versions only allow bug >> fixing, and this >> patchset could >> arguably not be considered as so. But without >> this, it >> could be, >> although >> unlikely, a DPDK used in a machine with more >> than 1TB, >> and then >> NFP using >> the wrong DMA host addresses. >> >> Although virtual addresses used as iovas are >> more >> dangerous, for >> DPDK versions >> before 18.05 this is not worse than with >> physical >> addresses, >> because iovas, >> when physical addresses are not available, are >> based on a >> starting address set >> to 0x0. >> >> >> You might want to look at the following patch: >> >> http://patches.dpdk.org/patch/37149/ >> >> > > >> > >> > >> >> >> Since this patch, IOVA as VA mode uses VA >> addresses, and >> that has >> been backported to earlier releases. I don't think >> there's >> any case >> where we used zero-based addresses any more. >> >> >> But memsegs get the iova based on hugepages physaddr, >> and for VA >> mode that is based on 0x0 as starting point. >> >> And as far as I know, memsegs iovas are what end up >> being used >> for IOMMU mappings and what devices will use. >> >> >> For when physaddrs are available, IOVA as PA mode assigns >> IOVA >> addresses to PA, while IOVA as VA mode assigns IOVA >> addresses to VA >> (both 18.05+ and pre-18.05 as per above patch, which was >> applied to >> pre-18.05 stable releases). >> >> When physaddrs aren't available, IOVA as VA mode assigns IOVA >> addresses to VA, both 18.05+ and pre-18.05, as per above >> patch. >> >> >> This is right. >> >> If physaddrs aren't available and IOVA as PA mode is used, >> then i as >> far as i can remember, even though technically memsegs get >> their >> addresses set to 0x0 onwards, the actual addresses we get in >> memzones etc. are RTE_BAD_IOVA. >> >> >> This is not right. Not sure if this was the intention, but if PA >> mode and physaddrs not available, this code inside >> vfio_type1_dma_map: >> >> if(rte_eal_iova_mode() == RTE_IOVA_VA) >> >> dma_map.iova = dma_map.vaddr; >> >> else >> >> dma_map.iova = ms[i].iova; >> >> >> does the IOMMU mapping using the iovas and not the vaddr, with >> the iovas starting at 0x0. >> >> >> Yep, you're right, apologies. I confused this with no-huge option. >> >> >> So, what do you think about the patchset? Could it be this applied to >> stable versions? >> >> I'll send a patch for current 18.05 code which will have the dma mask and >> the hugepage check, along with changes for doing the mmaps below the dma >> mask limit. >> > > I've looked through the code, it looks OK to me (bar some things like > missing .map file additions and a gratuitous rte_panic :) ). > > There was a patch/discussion not too long ago about DMA masks for some > IOMMU's - perhaps we can also extend this approach to that? > > https://patches.dpdk.org/patch/33192/ > > > I completely missed that patch. It seems it could also be applied for that case adding a dma mask set if it is an emulated VT-d with that 39 bits restriction. I'll take a look at that patch and submit a new patchset including changes for that case. I did also forget the hotplug case where the hugepage checking needs to be invoked. Thanks > >> >> >> -- Thanks, >> Anatoly >> >> >> > > -- > Thanks, > Anatoly >