From: Alejandro Lucero <alejandro.lucero@netronome.com>
To: "Burakov, Anatoly" <anatoly.burakov@intel.com>
Cc: wenjiex.a.li@intel.com, dev <dev@dpdk.org>,
Ferruh Yigit <ferruh.yigit@intel.com>,
xueqin.lin@intel.com
Subject: Re: [dpdk-dev] [PATCH v2 5/7] mem: modify error message for DMA mask check
Date: Tue, 6 Nov 2018 12:55:30 +0000 [thread overview]
Message-ID: <CAD+H993zEhB9qjBDLmBqC7tBLDpankGTy5s=-QRaiqG5zUM8yA@mail.gmail.com> (raw)
In-Reply-To: <9f54507c-1062-77e8-22c9-287660e1b66b@intel.com>
On Tue, Nov 6, 2018 at 10:48 AM Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:
> On 06-Nov-18 10:37 AM, Alejandro Lucero wrote:
> >
> >
> > On Tue, Nov 6, 2018 at 10:31 AM Burakov, Anatoly
> > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> >
> > On 06-Nov-18 9:32 AM, Alejandro Lucero wrote:
> > >
> > >
> > > On Mon, Nov 5, 2018 at 4:35 PM Burakov, Anatoly
> > > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>
> > <mailto:anatoly.burakov@intel.com
> > <mailto:anatoly.burakov@intel.com>>> wrote:
> > >
> > > On 05-Nov-18 3:33 PM, Alejandro Lucero wrote:
> > > >
> > > >
> > > > On Mon, Nov 5, 2018 at 3:12 PM Burakov, Anatoly
> > > > <anatoly.burakov@intel.com
> > <mailto:anatoly.burakov@intel.com> <mailto:anatoly.burakov@intel.com
> > <mailto:anatoly.burakov@intel.com>>
> > > <mailto:anatoly.burakov@intel.com
> > <mailto:anatoly.burakov@intel.com>
> > > <mailto:anatoly.burakov@intel.com
> > <mailto:anatoly.burakov@intel.com>>>> wrote:
> > > >
> > > > On 05-Nov-18 10:13 AM, Alejandro Lucero wrote:
> > > > > On Mon, Nov 5, 2018 at 10:01 AM Li, WenjieX A
> > > > <wenjiex.a.li@intel.com
> > <mailto:wenjiex.a.li@intel.com> <mailto:wenjiex.a.li@intel.com
> > <mailto:wenjiex.a.li@intel.com>>
> > > <mailto:wenjiex.a.li@intel.com
> > <mailto:wenjiex.a.li@intel.com> <mailto:wenjiex.a.li@intel.com
> > <mailto:wenjiex.a.li@intel.com>>>>
> > > > > wrote:
> > > > >
> > > > >> 1. With GCC32, testpmd could not startup without
> > > '--iova-mode pa'.
> > > > >> ./i686-native-linuxapp-gcc/app/testpmd -c f -n 4
> -- -i
> > > > >> The output is:
> > > > >> EAL: Detected 16 lcore(s)
> > > > >> EAL: Detected 1 NUMA nodes
> > > > >> EAL: Multi-process socket
> /var/run/dpdk/rte/mp_socket
> > > > >> EAL: Some devices want iova as va but pa will be
> used
> > > because..
> > > > EAL: few
> > > > >> device bound to UIO
> > > > >> EAL: No free hugepages reported in
> hugepages-1048576kB
> > > > >> EAL: Probing VFIO support...
> > > > >> EAL: VFIO support initialized
> > > > >> EAL: wrong dma mask size 48 (Max: 31)
> > > > >> EAL: alloc_pages_on_heap(): couldn't allocate
> > memory due
> > > to IOVA
> > > > exceeding
> > > > >> limits of current DMA mask
> > > > >> error allocating rte services array
> > > > >> EAL: FATAL: rte_service_init() failed
> > > > >> EAL: rte_service_init() failed
> > > > >> PANIC in main():
> > > > >> Cannot init EAL
> > > > >> 5:
> [./i686-native-linuxapp-gcc/app/testpmd(+0x95fda)
> > > [0x56606fda]]
> > > > >> 4:
> > [/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf6)
> > > > [0xf74d1276]]
> > > > >> 3:
> [./i686-native-linuxapp-gcc/app/testpmd(main+0xf21)
> > > [0x565fcee1]]
> > > > >> 2:
> > [./i686-native-linuxapp-gcc/app/testpmd(__rte_panic+0x3d)
> > > > [0x565edc68]]
> > > > >> 1:
> > > [./i686-native-linuxapp-gcc/app/testpmd(rte_dump_stack+0x33)
> > > > >> [0x5675f333]]
> > > > >> Aborted
> > > > >>
> > > > >> 2. With '--iova-mode pa', testpmd could startup.
> > > > >> 3. With GCC64, there is no such issue.
> > > > >> Thanks!
> > > > >>
> > > > >>
> > > > > Does 32 bits support require IOMMU? It would be a
> > surprise. If
> > > > there is no
> > > > > IOMMU hardware, no dma mask should be there at all.
> > > >
> > > > IOMMU is supported on 32-bits, however limited the
> address
> > > space might
> > > > be. Maybe limit IOMMU width to RTE_MIN(31, value) bits
> for
> > > > everything on
> > > > 32-bit?
> > > >
> > > >
> > > > If IOMMU is supported in 32 bits, then the DMA mask check
> > should
> > > not be
> > > > happening. AFAIK, the IOMMU hardware addressing
> > limitations is a
> > > problem
> > > > only in 64 bits systems. The worst situation I have head
> > of is 39
> > > bits
> > > > for virtualized IOMMU with QEMU.
> > > >
> > > > I would prefer not to invoke rte_mem_set_dma_mask for 32
> bits
> > > system for
> > > > the Intel IOMMU case. The only other dma mask client is
> > the NFP
> > > PMD and
> > > > we do not support 32 bits systems.
> > > >
> > >
> > > I don't think not invoking DMA mask check is the right choice
> > here. In
> > > practice it may be, but i'd rather the behavior to be
> > "correct", if at
> > > all possible :) It is theoretically possible to have an IOMMU
> > with an
> > > addressing limitation of, say, 30 bits (even though they
> > don't exist in
> > > reality), so therefore our code should handle it, should it
> > encounter
> > > one, and it should also handle the "proper" ones correctly
> > (as in,
> > > treat
> > > them as 32-bit-limited instead of 39- or 48-bit-limited).
> > >
> > >
> > > Fine.
> > >
> > > The problem is the current sanity check about the dma mask width,
> > what
> > > is 31 for 32 bits systems.
> > > Should we just leave a single max dma width to 63? This covers the
> > > possibility of 32 bit systems integrating an IOMMU designed for
> 64
> > > bits. I really doubt this is a real possibility in x86, although
> > I can
> > > see it more likely in embedded systems where this sort of hardware
> > > components integration happens.
> >
> > Actually (and after a quick chat with Ferruh), is this even needed?
> > IOVA
> > addresses are independent from VA width, IOVA can happily be bigger
> > than
> > 32-bits if i understand things correctly. All of our IOVA addresses
> are
> > always 64-bit throughout DPDK. I don't think this check is even
> valid.
> >
> >
> > Although iova_t is 64 bits, there should not be a IOVA higher than 32
> > bits, although there could be exceptions like PAE extensions (I'm old
> > enough for remembering that option :-( ).
> >
> > Anyway, the original idea of dma mask sanity check is 32 bits systems
> > was assuming there should not be a dma mask above 32 bits, but I'm happy
> > with removing that sanity check for 32 bits systems. So, do you agree to
> > just leave the sanity check for a max width of 63 bits?
> >
>
> So, the issue with 32-bit here is that for this check to make sense, the
> *kernel* must be 32-bit - not just userspace. IOW, this check should
> *not* be present in a 32-bit application running on a 64-bit kernel.
>
> So IMO, unless you know of a way to easily check if 1) kernel is 32-bit,
> and 2) PAE is enabled/disabled (and by easily i mean using something
> other than reading sysfs etc.), i don't think this check should be in
> there :)
>
Ok then. If there are no other opinions, I will remove the sanity check for
32 bits systems.
Thanks
>
> > >
> > > >
> > > > --
> > > > Thanks,
> > > > Anatoly
> > > >
> > >
> > >
> > > --
> > > Thanks,
> > > Anatoly
> > >
> >
> >
> > --
> > Thanks,
> > Anatoly
> >
>
>
> --
> Thanks,
> Anatoly
>
next prev parent reply other threads:[~2018-11-06 12:55 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-11-01 19:53 [dpdk-dev] [PATCH v2 0/7] fix " Alejandro Lucero
2018-11-01 19:53 ` [dpdk-dev] [PATCH v2 1/7] mem: fix call to " Alejandro Lucero
2018-11-01 19:53 ` [dpdk-dev] [PATCH v2 2/7] mem: use proper prefix Alejandro Lucero
2018-11-01 19:53 ` [dpdk-dev] [PATCH v2 3/7] mem: add function for setting DMA mask Alejandro Lucero
2018-11-01 19:53 ` [dpdk-dev] [PATCH v2 4/7] bus/pci: avoid call to DMA mask check Alejandro Lucero
2018-11-01 19:53 ` [dpdk-dev] [PATCH v2 5/7] mem: modify error message for " Alejandro Lucero
2018-11-05 10:01 ` Li, WenjieX A
2018-11-05 10:13 ` Alejandro Lucero
2018-11-05 15:12 ` Burakov, Anatoly
2018-11-05 15:33 ` Alejandro Lucero
2018-11-05 16:34 ` Burakov, Anatoly
2018-11-06 9:32 ` Alejandro Lucero
2018-11-06 10:31 ` Burakov, Anatoly
2018-11-06 10:37 ` Alejandro Lucero
2018-11-06 10:48 ` Burakov, Anatoly
2018-11-06 12:55 ` Alejandro Lucero [this message]
2018-11-01 19:53 ` [dpdk-dev] [PATCH v2 6/7] eal/mem: use DMA mask check for legacy memory Alejandro Lucero
2018-11-01 19:53 ` [dpdk-dev] [PATCH v2 7/7] mem: add thread unsafe version for checking DMA mask Alejandro Lucero
2018-11-02 18:38 ` [dpdk-dev] [PATCH v2 0/7] fix DMA mask check Ferruh Yigit
2018-11-05 0:03 ` Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAD+H993zEhB9qjBDLmBqC7tBLDpankGTy5s=-QRaiqG5zUM8yA@mail.gmail.com' \
--to=alejandro.lucero@netronome.com \
--cc=anatoly.burakov@intel.com \
--cc=dev@dpdk.org \
--cc=ferruh.yigit@intel.com \
--cc=wenjiex.a.li@intel.com \
--cc=xueqin.lin@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).