Re: [dpdk-dev] [PATCH v2 00/41] Memory Hotplug for DPDK

[Pavan Nikhilesh <pbhagavatula@caviumnetworks.com>]

Hi Anatoly,

We are currently facing issues with running testpmd on thunderx platform.
The issue seems to be with vfio

EAL: Detected 24 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: No free hugepages reported in hugepages-2048kB
EAL: Multi-process socket /var/run/.rte_unix
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL:   VFIO support not initialized

<snip>

EAL:   probe driver: 177d:a053 octeontx_fpavf
EAL: PCI device 0001:01:00.1 on NUMA socket 0
EAL:   probe driver: 177d:a034 net_thunderx
EAL:   using IOMMU type 1 (Type 1)
EAL:   cannot set up DMA remapping, error 22 (Invalid argument)
EAL:   0001:01:00.1 DMA remapping failed, error 22 (Invalid argument)
EAL: Requested device 0001:01:00.1 cannot be used
EAL: PCI device 0001:01:00.2 on NUMA socket 0
<snip>
testpmd: No probed ethernet devices
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=251456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
EAL:   VFIO support not initialized
EAL:   VFIO support not initialized
EAL:   VFIO support not initialized
Done


This is because rte_service_init() calls rte_calloc() before
rte_bus_probe() and vfio_dma_mem_map fails because iommu type is not set.

Call stack:
gdb) bt
#0  vfio_dma_mem_map (vaddr=281439006359552, iova=11274289152, len=536870912, do_map=1) at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:967
#1  0x00000000004fd974 in rte_vfio_dma_map (vaddr=281439006359552, iova=11274289152, len=536870912) at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:988
#2  0x00000000004fbe78 in vfio_mem_event_callback (type=RTE_MEM_EVENT_ALLOC, addr=0xfff7a0000000, len=536870912) at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:240
#3  0x00000000005070ac in eal_memalloc_notify (event=RTE_MEM_EVENT_ALLOC, start=0xfff7a0000000, len=536870912) at /root/clean/dpdk/lib/librte_eal/common/eal_common_memalloc.c:177
#4  0x0000000000515c98 in try_expand_heap_primary (heap=0xffffb7fb167c, pg_sz=536870912, elt_size=8192, socket=0, flags=0, align=128, bound=0, contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:247
#5  0x0000000000515e94 in try_expand_heap (heap=0xffffb7fb167c, pg_sz=536870912, elt_size=8192, socket=0, flags=0, align=128, bound=0, contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:327
#6  0x00000000005163a0 in alloc_more_mem_on_socket (heap=0xffffb7fb167c, size=8192, socket=0, flags=0, align=128, bound=0, contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:455
#7  0x0000000000516514 in heap_alloc_on_socket (type=0x85bf90 "rte_services", size=8192, socket=0, flags=0, align=128, bound=0, contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:491
#8  0x0000000000516664 in malloc_heap_alloc (type=0x85bf90 "rte_services", size=8192, socket_arg=-1, flags=0, align=128, bound=0, contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:527
#9  0x0000000000513b54 in rte_malloc_socket (type=0x85bf90 "rte_services", size=8192, align=128, socket_arg=-1) at /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:54
#10 0x0000000000513bc8 in rte_zmalloc_socket (type=0x85bf90 "rte_services", size=8192, align=128, socket=-1) at /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:72
#11 0x0000000000513c00 in rte_zmalloc (type=0x85bf90 "rte_services", size=8192, align=128) at /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:81
#12 0x0000000000513c90 in rte_calloc (type=0x85bf90 "rte_services", num=64, size=128, align=128) at /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:99
#13 0x0000000000518cec in rte_service_init () at /root/clean/dpdk/lib/librte_eal/common/rte_service.c:81
#14 0x00000000004f55f4 in rte_eal_init (argc=3, argv=0xfffffffff488) at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal.c:959
#15 0x000000000045af5c in main (argc=3, argv=0xfffffffff488) at /root/clean/dpdk/app/test-pmd/testpmd.c:2483


Also, I have tried running with --legacy-mem but I'm stuck in
`pci_find_max_end_va` loop  because `rte_fbarray_find_next_used` always return
0.

HugePages_Total:      15
HugePages_Free:       11
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:     524288 kB

Call Stack:
(gdb) bt
#0  find_next (arr=0xffffb7fb009c, start=0, used=true) at /root/clean/dpdk/lib/librte_eal/common/eal_common_fbarray.c:248
#1  0x00000000005132a8 in rte_fbarray_find_next_used (arr=0xffffb7fb009c, start=0) at /root/clean/dpdk/lib/librte_eal/common/eal_common_fbarray.c:700
#2  0x000000000052d030 in pci_find_max_end_va () at /root/clean/dpdk/drivers/bus/pci/linux/pci.c:138
#3  0x0000000000530ab8 in pci_vfio_map_resource_primary (dev=0xeae700) at /root/clean/dpdk/drivers/bus/pci/linux/pci_vfio.c:499
#4  0x0000000000530ffc in pci_vfio_map_resource (dev=0xeae700) at /root/clean/dpdk/drivers/bus/pci/linux/pci_vfio.c:601
#5  0x000000000052ce90 in rte_pci_map_device (dev=0xeae700) at /root/clean/dpdk/drivers/bus/pci/linux/pci.c:75
#6  0x0000000000531a20 in rte_pci_probe_one_driver (dr=0x997e20 <rte_nicvf_pmd>, dev=0xeae700) at /root/clean/dpdk/drivers/bus/pci/pci_common.c:164
#7  0x0000000000531c68 in pci_probe_all_drivers (dev=0xeae700) at /root/clean/dpdk/drivers/bus/pci/pci_common.c:249
#8  0x0000000000531f68 in rte_pci_probe () at /root/clean/dpdk/drivers/bus/pci/pci_common.c:359
#9  0x000000000050a140 in rte_bus_probe () at /root/clean/dpdk/lib/librte_eal/common/eal_common_bus.c:98
#10 0x00000000004f55f4 in rte_eal_init (argc=1, argv=0xfffffffff498) at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal.c:967
#11 0x000000000045af5c in main (argc=1, argv=0xfffffffff498) at /root/clean/dpdk/app/test-pmd/testpmd.c:2483

Am I missing something here?


Thanks,
Pavan.

On Thu, Mar 08, 2018 at 04:43:38PM +0530, Pavan Nikhilesh wrote:
> On Thu, Mar 08, 2018 at 10:46:46AM +0000, Burakov, Anatoly wrote:
> > On 08-Mar-18 10:18 AM, Pavan Nikhilesh wrote:
> > > Hi Anatoly,
> > >
> > > I am trying to verify this patchset and have encountered few issues.
> > >
> > > Few -Werror=maybe-uninitialized errors in eal_memalloc.c/eal_memory.c/
> > > eal_common_memzone.c files.
> >
> > Thanks for the heads up, i'll fix those in the next revision. Out of
> > curiousity, which compiler version are you using?
>
> I'm using gcc 5.3.0.
>
> >
> > >
> > > diff --git a/lib/librte_eal/common/eal_common_memzone.c b/lib/librte_eal/common/eal_common_memzone.c
> > > index a7cfdaf03..ad4413507 100644
> > > --- a/lib/librte_eal/common/eal_common_memzone.c
> > > +++ b/lib/librte_eal/common/eal_common_memzone.c
> > > @@ -321,7 +321,7 @@ rte_memzone_free(const struct rte_memzone *mz)
> > >          struct rte_fbarray *arr;
> > >          struct rte_memzone *found_mz;
> > >          int ret = 0;
> > > -       void *addr;
> > > +       void *addr = NULL;
> > >          unsigned idx;
> > >
> > >          if (mz == NULL)
> > > diff --git a/lib/librte_eal/linuxapp/eal/eal_memalloc.c b/lib/librte_eal/linuxapp/eal/eal_memalloc.c
> > > index 1008faed6..32b0d5133 100644
> > > --- a/lib/librte_eal/linuxapp/eal/eal_memalloc.c
> > > +++ b/lib/librte_eal/linuxapp/eal/eal_memalloc.c
> > > @@ -570,7 +570,7 @@ eal_memalloc_alloc_page_bulk(struct rte_memseg **ms, int n,
> > >          unsigned int msl_idx;
> > >          int cur_idx, start_idx, end_idx, i, j, ret = -1;
> > >   #ifdef RTE_EAL_NUMA_AWARE_HUGEPAGES
> > > -       bool have_numa;
> > > +       bool have_numa = false;
> > >          int oldpolicy;
> > >          struct bitmask *oldmask = numa_allocate_nodemask();
> > >   #endif
> > > diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
> > > index f74291fb6..d37b4a59b 100644
> > > --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
> > > +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
> > > @@ -1386,9 +1386,9 @@ eal_legacy_hugepage_attach(void)
> > >          struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
> > >          struct hugepage_file *hp = NULL;
> > >          unsigned int num_hp = 0;
> > > -       unsigned int i;
> > > +       unsigned int i = 0;
> > >          int ms_idx, msl_idx;
> > > -       unsigned int cur_seg, max_seg;
> > > +       unsigned int cur_seg, max_seg = 0;
> > >          off_t size = 0;
> > >          int fd, fd_hugepage = -1;
> > >
> > >
> > >
> > > @Hemanth
> > > Also, this patchset breaks dpaa/dpaa2 bus drivers (they rely on
> > > `rte_eal_get_physmem_layout` that is depricated
> > > http://dpdk.org/dev/patchwork/patch/34002/)
> > > So, generic arm64 linuxapp build is broken.
> >
> > Should the deprecation notice have been accompanied with marking that
> > function as __rte_deprecated?
>
> Yup that's the general sequence.
>
> >
> > >
> > > Regards,
> > > Pavan.
> > >
> > > On Wed, Mar 07, 2018 at 04:56:28PM +0000, Anatoly Burakov wrote:
> > > > This patchset introduces dynamic memory allocation for DPDK (aka memory
> > > > hotplug). Based upon RFC submitted in December [1].
> > > >
> > > > Dependencies (to be applied in specified order):
> > > > - IPC bugfixes patchset [2]
> > > > - IPC improvements patchset [3]
> > > > - IPC asynchronous request API patch [4]
> > > > - Function to return number of sockets [5]
> > > >
> > > <snip>
> > >   --
> > > > 2.7.4
> > >
> >
> >
> > --
> > Thanks,
> > Anatoly