Re: [dpdk-dev] [PATCH v3 0/6] use IOVAs check based on DMA mask

["Yao, Lei A" <lei.a.yao@intel.com>]

Hi, Lucero

My server info:
Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz
Hugepage: 1G
Kernel: 4.15.0
OS: Ubuntu

Steps are simple:

1.       Bind one i40e/ixgbe  NIC to igb_uio

2.       Launch testpmd:

./x86_64-native-linuxapp-gcc/app/testpmd -c 0x03 -n 4 --log-level=eal,8 -- -i

BRs
Lei


From: Alejandro Lucero [mailto:alejandro.lucero@netronome.com]
Sent: Monday, October 29, 2018 5:26 PM
To: Thomas Monjalon <thomas@monjalon.net>
Cc: Yao, Lei A <lei.a.yao@intel.com>; dev <dev@dpdk.org>; Xu, Qian Q <qian.q.xu@intel.com>; Lin, Xueqin <xueqin.lin@intel.com>; Burakov, Anatoly <anatoly.burakov@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>
Subject: Re: [dpdk-dev] [PATCH v3 0/6] use IOVAs check based on DMA mask

Can we have the configuration triggering this issue?

On Mon, Oct 29, 2018 at 9:07 AM Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>> wrote:
One more comment about this issue,

There was no reply to the question asked by Alejandro on October 11th:
        http://mails.dpdk.org/archives/dev/2018-October/115402.html
and there were no more reviews despite all my requests:
        http://mails.dpdk.org/archives/dev/2018-October/117475.html
Without any more comment, I had to apply the patchset.

Now we need to find a solution. Please suggest.


29/10/2018 09:42, Thomas Monjalon:
> 29/10/2018 09:23, Yao, Lei A:
> > Hi, Lucero, Thomas
> >
> > This patch set will cause deadlock during memory initialization.
> > rte_memseg_walk and try_expand_heap both will lock
> > the file &mcfg->memory_hotplug_lock. So dead lock will occur.
> >
> > #0       rte_memseg_walk
> > #1  <-rte_eal_check_dma_mask
> > #2  <-alloc_pages_on_heap
> > #3  <-try_expand_heap_primary
> > #4  <-try_expand_heap
> >
> > Log as following:
> > EAL: TSC frequency is ~2494156 KHz
> > EAL: Master lcore 0 is ready (tid=7ffff7fe3c00;cpuset=[0])
> > [New Thread 0x7ffff5e0d700 (LWP 330350)]
> > EAL: lcore 1 is ready (tid=7ffff5e0d700;cpuset=[1])
> > EAL: Trying to obtain current memory policy.
> > EAL: Setting policy MPOL_PREFERRED for socket 0
> > EAL: Restoring previous memory policy: 0
> >
> > Could you have a check on this? A lot of test cases in our validation
> > team fail because of this. Thanks a lot!
>
> Can we just call rte_memseg_walk_thread_unsafe()?
>
> +Cc Anatoly
>
>
> > From: dev [mailto:dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>] On Behalf Of Thomas Monjalon
> > > 05/10/2018 14:45, Alejandro Lucero:
> > > > I sent a patchset about this to be applied on 17.11 stable. The memory
> > > > code has had main changes since that version, so here it is the patchset
> > > > adjusted to current master repo.
> > > >
> > > > This patchset adds, mainly, a check for ensuring IOVAs are within a
> > > > restricted range due to addressing limitations with some devices. There
> > > > are two known cases: NFP and IOMMU VT-d emulation.
> > > >
> > > > With this check IOVAs out of range are detected and PMDs can abort
> > > > initialization. For the VT-d case, IOVA VA mode is allowed as long as
> > > > IOVAs are within the supported range, avoiding to forbid IOVA VA by
> > > > default.
> > > >
> > > > For the addressing limitations known cases, there are just 40(NFP) or
> > > > 39(VT-d) bits for handling IOVAs. When using IOVA PA, those limitations
> > > > imply 1TB(NFP) or 512M(VT-d) as upper limits, which is likely enough for
> > > > most systems. With machines using more memory, the added check will
> > > > ensure IOVAs within the range.
> > > >
> > > > With IOVA VA, and because the way the Linux kernel serves mmap calls
> > > > in 64 bits systems, 39 or 40 bits are not enough. It is possible to
> > > > give an address hint with a lower starting address than the default one
> > > > used by the kernel, and then ensuring the mmap uses that hint or hint plus
> > > > some offset. With 64 bits systems, the process virtual address space is
> > > > large enoguh for doing the hugepages mmaping within the supported
> > > range
> > > > when those addressing limitations exist. This patchset also adds a change
> > > > for using such a hint making the use of IOVA VA a more than likely
> > > > possibility when there are those addressing limitations.
> > > >
> > > > The check is not done by default but just when it is required. This
> > > > patchset adds the check for NFP initialization and for setting the IOVA
> > > > mode is an emulated VT-d is detected. Also, because the recent patchset
> > > > adding dynamic memory allocation, the check is also invoked for ensuring
> > > > the new memsegs are within the required range.
> > > >
> > > > This patchset could be applied to stable 18.05.
> > >
> > > Applied, thanks