From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-f66.google.com (mail-ed1-f66.google.com [209.85.208.66]) by dpdk.org (Postfix) with ESMTP id 72D0B2862 for ; Tue, 6 Nov 2018 13:55:41 +0100 (CET) Received: by mail-ed1-f66.google.com with SMTP id w19-v6so10408145eds.1 for ; Tue, 06 Nov 2018 04:55:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=zou6vD+1QhBAh8TiLFziYQpLZpTUTDeXnk+wfBTQE30=; b=qpxnF9bIkOrJFYx3HSWl5gtR9hswlDs/19D7FaX29cb6SQa3TdlSSVF09Umih7DYhk uxEX6Ife1erk3J+85UlNENAtl2c4NZt1qNdrTrJL4U7RkLT79gX2Hag3pz2RBelltPUW 3NDAKYFJ/4Fx/tufzzmj/KYWuUuSKfvaKFwfNLjK/Z7y/ZBJXstslbqnw1tYZ9cVIP13 xXqFF25KIDCgPQ3pryWmfL568lh36TAo/G+eaDq+vCEmN53Q2xW05fXBOxdqtdG/bCRe 5hXIBjQoW6vV5Mo3MVk4MralpyC+usw9kxxiFGuOdff3hu36aJP3BrWIAr59S/oryuqE yV9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=zou6vD+1QhBAh8TiLFziYQpLZpTUTDeXnk+wfBTQE30=; b=cMsQGpRH/IYlBcE0m70jBy9EzB77R20UTNPdFyGVKqMBwz9LJ6o63hELF6kY41mRGA WZ2fLFokRTU3gTiTa4MAyAgvNfSwwAJkjsJFZF7aO/zuDxw0JO3rmuNwBzzgTuGKtbOt gukWVJg1sSiPSnj8jssTWb2+lETH+eW1RRL8eo+s2/RxhMzws2uVyy2ZbvgFr7Wn87pM vkIjdIjmW3+yEDJs0v6MiJ+khwJ+rqFk8D2AlUsxCm+by/yivkGi+AInuHlN0nE8SyX8 H7KxztUV7J9XoHspd+D0t2C8fiimlNi0826l5RWKfV63vyDibrial/JzhJJEo8+NQFf6 Q62w== X-Gm-Message-State: AGRZ1gKnVZENmE/VrKuDPDkdRXWrSHu/YS+OV6uzTn+2GxXQyABoa/9B EherLwY+hQCsimGqcJMJJ+ZhsBqJ77XyeQCaxy2vKg== X-Google-Smtp-Source: AJdET5cEc7j/XTncbbegUmobtJ2OpDGpPzy7YLmdDFhhkEiMwlrGWSF+QLgO7yuB8vxlWlV3fjS3EPhRpLL9rK7oulU= X-Received: by 2002:a50:ee91:: with SMTP id f17-v6mr21412731edr.136.1541508941097; Tue, 06 Nov 2018 04:55:41 -0800 (PST) MIME-Version: 1.0 References: <20181101195330.19464-1-alejandro.lucero@netronome.com> <20181101195330.19464-6-alejandro.lucero@netronome.com> <8688172CD5C0B74590FAE19D9579F94B535FDD90@SHSMSX103.ccr.corp.intel.com> <72c61eb1-4dca-e16c-54f7-b14d2ba1ae4c@intel.com> <9f54507c-1062-77e8-22c9-287660e1b66b@intel.com> In-Reply-To: <9f54507c-1062-77e8-22c9-287660e1b66b@intel.com> From: Alejandro Lucero Date: Tue, 6 Nov 2018 12:55:30 +0000 Message-ID: To: "Burakov, Anatoly" Cc: wenjiex.a.li@intel.com, dev , Ferruh Yigit , xueqin.lin@intel.com Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH v2 5/7] mem: modify error message for DMA mask check X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2018 12:55:41 -0000 On Tue, Nov 6, 2018 at 10:48 AM Burakov, Anatoly wrote: > On 06-Nov-18 10:37 AM, Alejandro Lucero wrote: > > > > > > On Tue, Nov 6, 2018 at 10:31 AM Burakov, Anatoly > > > wrote: > > > > On 06-Nov-18 9:32 AM, Alejandro Lucero wrote: > > > > > > > > > On Mon, Nov 5, 2018 at 4:35 PM Burakov, Anatoly > > > > > > >> wrote: > > > > > > On 05-Nov-18 3:33 PM, Alejandro Lucero wrote: > > > > > > > > > > > > On Mon, Nov 5, 2018 at 3:12 PM Burakov, Anatoly > > > > > > > > > > > > > > > >>> wrote: > > > > > > > > On 05-Nov-18 10:13 AM, Alejandro Lucero wrote: > > > > > On Mon, Nov 5, 2018 at 10:01 AM Li, WenjieX A > > > > > > > > > > > > >>> > > > > > wrote: > > > > > > > > > >> 1. With GCC32, testpmd could not startup without > > > '--iova-mode pa'. > > > > >> ./i686-native-linuxapp-gcc/app/testpmd -c f -n 4 > -- -i > > > > >> The output is: > > > > >> EAL: Detected 16 lcore(s) > > > > >> EAL: Detected 1 NUMA nodes > > > > >> EAL: Multi-process socket > /var/run/dpdk/rte/mp_socket > > > > >> EAL: Some devices want iova as va but pa will be > used > > > because.. > > > > EAL: few > > > > >> device bound to UIO > > > > >> EAL: No free hugepages reported in > hugepages-1048576kB > > > > >> EAL: Probing VFIO support... > > > > >> EAL: VFIO support initialized > > > > >> EAL: wrong dma mask size 48 (Max: 31) > > > > >> EAL: alloc_pages_on_heap(): couldn't allocate > > memory due > > > to IOVA > > > > exceeding > > > > >> limits of current DMA mask > > > > >> error allocating rte services array > > > > >> EAL: FATAL: rte_service_init() failed > > > > >> EAL: rte_service_init() failed > > > > >> PANIC in main(): > > > > >> Cannot init EAL > > > > >> 5: > [./i686-native-linuxapp-gcc/app/testpmd(+0x95fda) > > > [0x56606fda]] > > > > >> 4: > > [/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf6) > > > > [0xf74d1276]] > > > > >> 3: > [./i686-native-linuxapp-gcc/app/testpmd(main+0xf21) > > > [0x565fcee1]] > > > > >> 2: > > [./i686-native-linuxapp-gcc/app/testpmd(__rte_panic+0x3d) > > > > [0x565edc68]] > > > > >> 1: > > > [./i686-native-linuxapp-gcc/app/testpmd(rte_dump_stack+0x33) > > > > >> [0x5675f333]] > > > > >> Aborted > > > > >> > > > > >> 2. With '--iova-mode pa', testpmd could startup. > > > > >> 3. With GCC64, there is no such issue. > > > > >> Thanks! > > > > >> > > > > >> > > > > > Does 32 bits support require IOMMU? It would be a > > surprise. If > > > > there is no > > > > > IOMMU hardware, no dma mask should be there at all. > > > > > > > > IOMMU is supported on 32-bits, however limited the > address > > > space might > > > > be. Maybe limit IOMMU width to RTE_MIN(31, value) bits > for > > > > everything on > > > > 32-bit? > > > > > > > > > > > > If IOMMU is supported in 32 bits, then the DMA mask check > > should > > > not be > > > > happening. AFAIK, the IOMMU hardware addressing > > limitations is a > > > problem > > > > only in 64 bits systems. The worst situation I have head > > of is 39 > > > bits > > > > for virtualized IOMMU with QEMU. > > > > > > > > I would prefer not to invoke rte_mem_set_dma_mask for 32 > bits > > > system for > > > > the Intel IOMMU case. The only other dma mask client is > > the NFP > > > PMD and > > > > we do not support 32 bits systems. > > > > > > > > > > I don't think not invoking DMA mask check is the right choice > > here. In > > > practice it may be, but i'd rather the behavior to be > > "correct", if at > > > all possible :) It is theoretically possible to have an IOMMU > > with an > > > addressing limitation of, say, 30 bits (even though they > > don't exist in > > > reality), so therefore our code should handle it, should it > > encounter > > > one, and it should also handle the "proper" ones correctly > > (as in, > > > treat > > > them as 32-bit-limited instead of 39- or 48-bit-limited). > > > > > > > > > Fine. > > > > > > The problem is the current sanity check about the dma mask width, > > what > > > is 31 for 32 bits systems. > > > Should we just leave a single max dma width to 63? This covers the > > > possibility of 32 bit systems integrating an IOMMU designed for > 64 > > > bits. I really doubt this is a real possibility in x86, although > > I can > > > see it more likely in embedded systems where this sort of hardware > > > components integration happens. > > > > Actually (and after a quick chat with Ferruh), is this even needed? > > IOVA > > addresses are independent from VA width, IOVA can happily be bigger > > than > > 32-bits if i understand things correctly. All of our IOVA addresses > are > > always 64-bit throughout DPDK. I don't think this check is even > valid. > > > > > > Although iova_t is 64 bits, there should not be a IOVA higher than 32 > > bits, although there could be exceptions like PAE extensions (I'm old > > enough for remembering that option :-( ). > > > > Anyway, the original idea of dma mask sanity check is 32 bits systems > > was assuming there should not be a dma mask above 32 bits, but I'm happy > > with removing that sanity check for 32 bits systems. So, do you agree to > > just leave the sanity check for a max width of 63 bits? > > > > So, the issue with 32-bit here is that for this check to make sense, the > *kernel* must be 32-bit - not just userspace. IOW, this check should > *not* be present in a 32-bit application running on a 64-bit kernel. > > So IMO, unless you know of a way to easily check if 1) kernel is 32-bit, > and 2) PAE is enabled/disabled (and by easily i mean using something > other than reading sysfs etc.), i don't think this check should be in > there :) > Ok then. If there are no other opinions, I will remove the sanity check for 32 bits systems. Thanks > > > > > > > > > > > > -- > > > > Thanks, > > > > Anatoly > > > > > > > > > > > > > -- > > > Thanks, > > > Anatoly > > > > > > > > > -- > > Thanks, > > Anatoly > > > > > -- > Thanks, > Anatoly >