From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id CE06E2BE2 for ; Tue, 6 Nov 2018 11:48:06 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Nov 2018 02:48:05 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,471,1534834800"; d="scan'208";a="278716960" Received: from aburakov-mobl1.ger.corp.intel.com (HELO [10.237.220.143]) ([10.237.220.143]) by fmsmga006.fm.intel.com with ESMTP; 06 Nov 2018 02:48:04 -0800 To: Alejandro Lucero Cc: wenjiex.a.li@intel.com, dev , Ferruh Yigit , xueqin.lin@intel.com References: <20181101195330.19464-1-alejandro.lucero@netronome.com> <20181101195330.19464-6-alejandro.lucero@netronome.com> <8688172CD5C0B74590FAE19D9579F94B535FDD90@SHSMSX103.ccr.corp.intel.com> <72c61eb1-4dca-e16c-54f7-b14d2ba1ae4c@intel.com> From: "Burakov, Anatoly" Message-ID: <9f54507c-1062-77e8-22c9-287660e1b66b@intel.com> Date: Tue, 6 Nov 2018 10:48:03 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] [PATCH v2 5/7] mem: modify error message for DMA mask check X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2018 10:48:07 -0000 On 06-Nov-18 10:37 AM, Alejandro Lucero wrote: > > > On Tue, Nov 6, 2018 at 10:31 AM Burakov, Anatoly > > wrote: > > On 06-Nov-18 9:32 AM, Alejandro Lucero wrote: > > > > > > On Mon, Nov 5, 2018 at 4:35 PM Burakov, Anatoly > > > >> wrote: > > > >     On 05-Nov-18 3:33 PM, Alejandro Lucero wrote: > >      > > >      > > >      > On Mon, Nov 5, 2018 at 3:12 PM Burakov, Anatoly > >      > > > >      > >      >>> wrote: > >      > > >      >     On 05-Nov-18 10:13 AM, Alejandro Lucero wrote: > >      >      > On Mon, Nov 5, 2018 at 10:01 AM Li, WenjieX A > >      >      > > >      >>> > >      >      > wrote: > >      >      > > >      >      >> 1. With GCC32, testpmd could not startup without > >     '--iova-mode pa'. > >      >      >> ./i686-native-linuxapp-gcc/app/testpmd -c f -n 4 -- -i > >      >      >> The output is: > >      >      >> EAL: Detected 16 lcore(s) > >      >      >> EAL: Detected 1 NUMA nodes > >      >      >> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > >      >      >> EAL: Some devices want iova as va but pa will be used > >     because.. > >      >     EAL: few > >      >      >> device bound to UIO > >      >      >> EAL: No free hugepages reported in hugepages-1048576kB > >      >      >> EAL: Probing VFIO support... > >      >      >> EAL: VFIO support initialized > >      >      >> EAL: wrong dma mask size 48 (Max: 31) > >      >      >> EAL: alloc_pages_on_heap(): couldn't allocate > memory due > >     to IOVA > >      >     exceeding > >      >      >> limits of current DMA mask > >      >      >> error allocating rte services array > >      >      >> EAL: FATAL: rte_service_init() failed > >      >      >> EAL: rte_service_init() failed > >      >      >> PANIC in main(): > >      >      >> Cannot init EAL > >      >      >> 5: [./i686-native-linuxapp-gcc/app/testpmd(+0x95fda) > >     [0x56606fda]] > >      >      >> 4: > [/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf6) > >      >     [0xf74d1276]] > >      >      >> 3: [./i686-native-linuxapp-gcc/app/testpmd(main+0xf21) > >     [0x565fcee1]] > >      >      >> 2: > [./i686-native-linuxapp-gcc/app/testpmd(__rte_panic+0x3d) > >      >     [0x565edc68]] > >      >      >> 1: > >     [./i686-native-linuxapp-gcc/app/testpmd(rte_dump_stack+0x33) > >      >      >> [0x5675f333]] > >      >      >> Aborted > >      >      >> > >      >      >> 2. With '--iova-mode pa', testpmd could startup. > >      >      >> 3. With GCC64, there is no such issue. > >      >      >> Thanks! > >      >      >> > >      >      >> > >      >      > Does 32 bits support require IOMMU? It would be a > surprise. If > >      >     there is no > >      >      > IOMMU hardware, no dma mask should be there at all. > >      > > >      >     IOMMU is supported on 32-bits, however limited the address > >     space might > >      >     be. Maybe limit IOMMU width to RTE_MIN(31, value) bits for > >      >     everything on > >      >     32-bit? > >      > > >      > > >      > If IOMMU is supported in 32 bits, then the DMA mask check > should > >     not be > >      > happening. AFAIK, the IOMMU hardware addressing > limitations is a > >     problem > >      > only in 64 bits systems. The worst situation I have head > of is 39 > >     bits > >      > for virtualized IOMMU with QEMU. > >      > > >      > I would prefer not to invoke rte_mem_set_dma_mask for 32 bits > >     system for > >      > the Intel IOMMU case. The only other dma mask client is > the NFP > >     PMD and > >      > we do not support 32 bits systems. > >      > > > > >     I don't think not invoking DMA mask check is the right choice > here. In > >     practice it may be, but i'd rather the behavior to be > "correct", if at > >     all possible :) It is theoretically possible to have an IOMMU > with an > >     addressing limitation of, say, 30 bits (even though they > don't exist in > >     reality), so therefore our code should handle it, should it > encounter > >     one, and it should also handle the "proper" ones correctly > (as in, > >     treat > >     them as 32-bit-limited instead of 39- or 48-bit-limited). > > > > > > Fine. > > > > The problem is the current sanity check about the dma mask width, > what > > is 31 for 32 bits systems. > > Should we just leave a single max dma width to 63? This covers the > > possibility of 32 bit systems integrating an IOMMU designed for  64 > > bits. I really doubt this is a real possibility in x86, although > I can > > see it more likely in embedded systems where this sort of hardware > > components integration happens. > > Actually (and after a quick chat with Ferruh), is this even needed? > IOVA > addresses are independent from VA width, IOVA can happily be bigger > than > 32-bits if i understand things correctly. All of our IOVA addresses are > always 64-bit throughout DPDK. I don't think this check is even valid. > > > Although iova_t is 64 bits, there should not be a IOVA higher than 32 > bits, although there could be exceptions like PAE extensions (I'm old > enough for remembering that option :-( ). > > Anyway, the original idea of dma mask sanity check is 32 bits systems > was assuming there should not be a dma mask above 32 bits, but I'm happy > with removing that sanity check for 32 bits systems. So, do you agree to > just leave the sanity check for a max width of 63 bits? > So, the issue with 32-bit here is that for this check to make sense, the *kernel* must be 32-bit - not just userspace. IOW, this check should *not* be present in a 32-bit application running on a 64-bit kernel. So IMO, unless you know of a way to easily check if 1) kernel is 32-bit, and 2) PAE is enabled/disabled (and by easily i mean using something other than reading sysfs etc.), i don't think this check should be in there :) > > > >      > > >      >     -- > >      >     Thanks, > >      >     Anatoly > >      > > > > > > >     -- > >     Thanks, > >     Anatoly > > > > > -- > Thanks, > Anatoly > -- Thanks, Anatoly