From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-f68.google.com (mail-ed1-f68.google.com [209.85.208.68]) by dpdk.org (Postfix) with ESMTP id 5FB7F1C1DB for ; Wed, 27 Jun 2018 18:52:44 +0200 (CEST) Received: by mail-ed1-f68.google.com with SMTP id u9-v6so1516374eds.10 for ; Wed, 27 Jun 2018 09:52:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=oMFoHR5npW1sEA8frwx/vUDiy3Esno5jmwn7SEardDE=; b=tDi7AJG2d9Wp7TAX4U7rm+BtSwO1Pcu1xkABt0ZO1DWxetZAOGOCu6FLv5UBeVbiyU L9OkmEkhh7hRQN0joNKnwf45h/fh8cJm/T/pFUNvELRWkvgB/mRWdUDGAktcAq8vSXAT 3wQI8n/Inb8hdmaJfVnBlStx5FT4TfbIWp1eYLohcsoDf/wn4BXwMfZgDOP7rNRGxG9t 6E/6RIrMwcj4z+W76OUPrzhz/xxghlAPT+SC0Wfu5q2+KvOmeNYw3YPYUnccLfkiNJcs QGO+NWgkvQ8+u6fFuQtyAKrnK4SoLWMtxv+IadKwanMTWqu5TeoreArSCCdH8ZgSez1N SjEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=oMFoHR5npW1sEA8frwx/vUDiy3Esno5jmwn7SEardDE=; b=K0xQDhoxrgsvS6vHe6++3JJo7qAB0KEhloStZiFgFMolHsD5qxR89ro1uP3OW21v+h QCoLBabr+BIf2LXNJF+zPf7Ik8xTzdqwbxaLrVpqkxTWCYOYhYCrdI+mgNrZE6OTn8F9 M3j3pe7geAbsz1kKvCZ+uKMsCxAIt/R2U6ZAq7TO6V15V7hb3BbGqb7jjx3zqIn3nPta 16EuHYv1Iv2xdHih8XfkVeuS0QmLyNX7qKl62rdldGWQpEWBPtrBk48Dz8r5B7mhrLF9 tld8f/awitWG+wazQp/Mot3sHICMepDgo/dMkUmUlkHJTosRycyyorBb+pJxyTBUE0xo 9xpA== X-Gm-Message-State: APt69E2wTaCQK3aMIG7SNt0ddnT6w68rJtxxbRm4nMOPYE4PWCE3+5qM VYaiUZtTdH921UgZ5EXFhulDDdXC2uds2TcwNeicoQ== X-Google-Smtp-Source: AAOMgpeeVeiG28Uk8m0B80OMlP9sNZAAelXNHS/TxvDOj/meUhm1uVSd9KQf5ch06/Bs9lSUrm6+Mc3MqWkDtmTDysc= X-Received: by 2002:a50:ec0b:: with SMTP id g11-v6mr2719748edr.299.1530118364025; Wed, 27 Jun 2018 09:52:44 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a50:b194:0:0:0:0:0 with HTTP; Wed, 27 Jun 2018 09:52:43 -0700 (PDT) In-Reply-To: <03046f23-2466-cbb7-ae2b-f2770d5c6b0f@intel.com> References: <1530034653-28299-1-git-send-email-alejandro.lucero@netronome.com> <552f939e-f28e-0648-1796-8f42269887a2@intel.com> <03046f23-2466-cbb7-ae2b-f2770d5c6b0f@intel.com> From: Alejandro Lucero Date: Wed, 27 Jun 2018 17:52:43 +0100 Message-ID: To: "Burakov, Anatoly" Cc: dev , stable@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-stable] [RFC] Add support for device dma mask X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jun 2018 16:52:44 -0000 On Wed, Jun 27, 2018 at 2:24 PM, Burakov, Anatoly wrote: > On 27-Jun-18 11:13 AM, Alejandro Lucero wrote: > > >> >> On Wed, Jun 27, 2018 at 9:17 AM, Burakov, Anatoly < >> anatoly.burakov@intel.com > wrote: >> >> On 26-Jun-18 6:37 PM, Alejandro Lucero wrote: >> >> This RFC tries to handle devices with addressing limitations. >> NFP devices >> 4000/6000 can just handle addresses with 40 bits implying >> problems for handling >> physical address when machines have more than 1TB of memory. But >> because how >> iovas are configured, which can be equivalent to physical >> addresses or based on >> virtual addresses, this can be a more likely problem. >> >> I tried to solve this some time ago: >> >> https://www.mail-archive.com/dev@dpdk.org/msg45214.html >> >> >> It was delayed because there was some changes in progress with >> EAL device >> handling, and, being honest, I completely forgot about this >> until now, when >> I have had to work on supporting NFP devices with DPDK and >> non-root users. >> >> I was working on a patch for being applied on main DPDK branch >> upstream, but >> because changes to memory initialization during the last months, >> this can not >> be backported to stable versions, at least the part where the >> hugepages iovas >> are checked. >> >> I realize stable versions only allow bug fixing, and this >> patchset could >> arguably not be considered as so. But without this, it could be, >> although >> unlikely, a DPDK used in a machine with more than 1TB, and then >> NFP using >> the wrong DMA host addresses. >> >> Although virtual addresses used as iovas are more dangerous, for >> DPDK versions >> before 18.05 this is not worse than with physical addresses, >> because iovas, >> when physical addresses are not available, are based on a >> starting address set >> to 0x0. >> >> >> You might want to look at the following patch: >> >> http://patches.dpdk.org/patch/37149/ >> >> >> Since this patch, IOVA as VA mode uses VA addresses, and that has >> been backported to earlier releases. I don't think there's any case >> where we used zero-based addresses any more. >> >> >> But memsegs get the iova based on hugepages physaddr, and for VA mode >> that is based on 0x0 as starting point. >> >> And as far as I know, memsegs iovas are what end up being used for IOMMU >> mappings and what devices will use. >> > > For when physaddrs are available, IOVA as PA mode assigns IOVA addresses > to PA, while IOVA as VA mode assigns IOVA addresses to VA (both 18.05+ and > pre-18.05 as per above patch, which was applied to pre-18.05 stable > releases). > > When physaddrs aren't available, IOVA as VA mode assigns IOVA addresses to > VA, both 18.05+ and pre-18.05, as per above patch. > > This is right. > If physaddrs aren't available and IOVA as PA mode is used, then i as far > as i can remember, even though technically memsegs get their addresses set > to 0x0 onwards, the actual addresses we get in memzones etc. are > RTE_BAD_IOVA. > > This is not right. Not sure if this was the intention, but if PA mode and physaddrs not available, this code inside vfio_type1_dma_map: if (rte_eal_iova_mode() == RTE_IOVA_VA) dma_map.iova = dma_map.vaddr; else dma_map.iova = ms[i].iova; does the IOMMU mapping using the iovas and not the vaddr, with the iovas starting at 0x0. Note that NFP PMD has not the RTE_PCI_DRV_IOVA_AS_VA flag, so this is always the case when executing DPDK apps as non-root users. I would say, if there is no such a flag, and then IOVA mode is PA, the mapping should fail, as it occurs with 18.05. I could send a patch for having this behaviour, but in that case I would like to add that flag to NFP PMD and include the hugepage checking along with changes to how iovas are obtained when mmaping, keeping the iovas below the dma mask proposed. > > >> >> Since 18.05, those iovas can, and usually are, higher than 1TB, as >> they >> >> are based on 64 bits address space addresses, and by default the >> kernel uses a >> starting point far higher than 1TB. >> >> This patchset applies to stable 17.11.3 but I will be happy to >> submit patches, if >> required, for other DPDK stable versions. >> >> >> >> >> -- Thanks, >> Anatoly >> >> >> > > -- > Thanks, > Anatoly >