Re: [dpdk-stable] [RFC] Add support for device dma mask

[Alejandro Lucero <alejandro.lucero@netronome.com>]

On Thu, Jun 28, 2018 at 9:54 AM, Burakov, Anatoly <anatoly.burakov@intel.com
> wrote:

> On 27-Jun-18 5:52 PM, Alejandro Lucero wrote:
>
>>
>>
>> On Wed, Jun 27, 2018 at 2:24 PM, Burakov, Anatoly <
>> anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>>
>>     On 27-Jun-18 11:13 AM, Alejandro Lucero wrote:
>>
>>
>>
>>         On Wed, Jun 27, 2018 at 9:17 AM, Burakov, Anatoly
>>         <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>
>>         <mailto:anatoly.burakov@intel.com
>>
>>         <mailto:anatoly.burakov@intel.com>>> wrote:
>>
>>              On 26-Jun-18 6:37 PM, Alejandro Lucero wrote:
>>
>>                  This RFC tries to handle devices with addressing
>>         limitations.
>>                  NFP devices
>>                  4000/6000 can just handle addresses with 40 bits implying
>>                  problems for handling
>>                  physical address when machines have more than 1TB of
>>         memory. But
>>                  because how
>>                  iovas are configured, which can be equivalent to physical
>>                  addresses or based on
>>                  virtual addresses, this can be a more likely problem.
>>
>>                  I tried to solve this some time ago:
>>
>>         https://www.mail-archive.com/dev@dpdk.org/msg45214.html
>>         <https://www.mail-archive.com/dev@dpdk.org/msg45214.html>
>>                         <https://www.mail-archive.com/
>> dev@dpdk.org/msg45214.html
>>         <https://www.mail-archive.com/dev@dpdk.org/msg45214.html>>
>>
>>                  It was delayed because there was some changes in
>>         progress with
>>                  EAL device
>>                  handling, and, being honest, I completely forgot about
>> this
>>                  until now, when
>>                  I have had to work on supporting NFP devices with DPDK
>> and
>>                  non-root users.
>>
>>                  I was working on a patch for being applied on main DPDK
>>         branch
>>                  upstream, but
>>                  because changes to memory initialization during the
>>         last months,
>>                  this can not
>>                  be backported to stable versions, at least the part
>>         where the
>>                  hugepages iovas
>>                  are checked.
>>
>>                  I realize stable versions only allow bug fixing, and this
>>                  patchset could
>>                  arguably not be considered as so. But without this, it
>>         could be,
>>                  although
>>                  unlikely, a DPDK used in a machine with more than 1TB,
>>         and then
>>                  NFP using
>>                  the wrong DMA host addresses.
>>
>>                  Although virtual addresses used as iovas are more
>>         dangerous, for
>>                  DPDK versions
>>                  before 18.05 this is not worse than with physical
>>         addresses,
>>                  because iovas,
>>                  when physical addresses are not available, are based on a
>>                  starting address set
>>                  to 0x0.
>>
>>
>>              You might want to look at the following patch:
>>
>>         http://patches.dpdk.org/patch/37149/
>>         <http://patches.dpdk.org/patch/37149/>
>>              <http://patches.dpdk.org/patch/37149/
>>         <http://patches.dpdk.org/patch/37149/>>
>>
>>              Since this patch, IOVA as VA mode uses VA addresses, and
>>         that has
>>              been backported to earlier releases. I don't think there's
>>         any case
>>              where we used zero-based addresses any more.
>>
>>
>>         But memsegs get the iova based on hugepages physaddr, and for VA
>>         mode that is based on 0x0 as starting point.
>>
>>         And as far as I know, memsegs iovas are what end up being used
>>         for IOMMU mappings and what devices will use.
>>
>>
>>     For when physaddrs are available, IOVA as PA mode assigns IOVA
>>     addresses to PA, while IOVA as VA mode assigns IOVA addresses to VA
>>     (both 18.05+ and pre-18.05 as per above patch, which was applied to
>>     pre-18.05 stable releases).
>>
>>     When physaddrs aren't available, IOVA as VA mode assigns IOVA
>>     addresses to VA, both 18.05+ and pre-18.05, as per above patch.
>>
>>
>> This is right.
>>
>>     If physaddrs aren't available and IOVA as PA mode is used, then i as
>>     far as i can remember, even though technically memsegs get their
>>     addresses set to 0x0 onwards, the actual addresses we get in
>>     memzones etc. are RTE_BAD_IOVA.
>>
>>
>> This is not right. Not sure if this was the intention, but if PA mode and
>> physaddrs not available, this code inside vfio_type1_dma_map:
>>
>> if(rte_eal_iova_mode() == RTE_IOVA_VA)
>>
>> dma_map.iova = dma_map.vaddr;
>>
>> else
>>
>> dma_map.iova = ms[i].iova;
>>
>>
>> does the IOMMU mapping using the iovas and not the vaddr, with the iovas
>> starting at 0x0.
>>
>
> Yep, you're right, apologies. I confused this with no-huge option.


So, what do you think about the patchset? Could it be this applied to
stable versions?

I'll send a patch for current 18.05 code which will have the dma mask and
the hugepage check, along with changes for doing the mmaps below the dma
mask limit.


>
>
> --
> Thanks,
> Anatoly
>