Re: [dpdk-dev] dpdk starting issue with descending virtual address allocation in new kernel

["Richardson, Bruce" <bruce.richardson@intel.com>]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Michael Hu (NSBU)
> Sent: Wednesday, September 10, 2014 11:41 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] dpdk starting issue with descending virtual address
> allocation in new kernel
> 
> Hi All,
> 
> We have a kernel config question to consult you.
> DPDK failed to start due to mbuf creation issue with new kernel 3.14.17 +
> grsecurity patches.
> We tries to trace down the issue, it seems that  the virtual address of huge page
> is allocated from high address to low address by kernel where dpdk expects it to
> be low to high to think it is as consecutive. See dumped virt address bellow. It is
> first 0x710421400000, then 0x710421200000. Where previously it would be
> 0x710421200000 first , then 0x710421400000. But they are still consecutive.
> ----
> Initialize Port 0 -- TxQ 1, RxQ 1,  Src MAC 00:0c:29:b3:30:db
>     Create: Default RX  0:0  - Memory used (MBUFs 4096 x (size 1984 + Hdr 64)) +
> 790720 =   8965 KB
> Zone 0: name:<RG_MP_log_history>, phys:0x6ac00000, len:0x2080,
> virt:0x710421400000, socket_id:0, flags:0
> Zone 1: name:<MP_log_history>, phys:0x6ac02080, len:0x1d10c0,
> virt:0x710421402080, socket_id:0, flags:0
> Zone 2: name:<MALLOC_S0_HEAP_0>, phys:0x6ae00000, len:0x160000,
> virt:0x710421200000, socket_id:0, flags:0
> Zone 3: name:<rte_eth_dev_data>, phys:0x6add3140, len:0x11a00,
> virt:0x7104215d3140, socket_id:0, flags:0
> Zone 4: name:<rte_vmxnet3_pmd_0_shared>, phys:0x6ade4b40, len:0x300,
> virt:0x7104215e4b40, socket_id:0, flags:0
> Zone 5: name:<rte_vmxnet3_pmd_0_queuedesc>, phys:0x6ade4e80, len:0x200,
> virt:0x7104215e4e80, socket_id:0, flags:0
> Zone 6: name:<RG_MP_Default RX  0:0>, phys:0x6ade5080, len:0x10080,
> virt:0x7104215e5080, socket_id:0, flags:0
> Segment 0: phys:0x6ac00000, len:2097152, virt:0x710421400000, socket_id:0,
> hugepage_sz:2097152, nchannel:0, nrank:0
> Segment 1: phys:0x6ae00000, len:2097152, virt:0x710421200000, socket_id:0,
> hugepage_sz:2097152, nchannel:0, nrank:0
> Segment 2: phys:0x6b000000, len:2097152, virt:0x710421000000, socket_id:0,
> hugepage_sz:2097152, nchannel:0, nrank:0
> Segment 3: phys:0x6b200000, len:2097152, virt:0x710420e00000, socket_id:0,
> hugepage_sz:2097152, nchannel:0, nrank:0
> Segment 4: phys:0x6b400000, len:2097152, virt:0x710420c00000, socket_id:0,
> hugepage_sz:2097152, nchannel:0, nrank:0
> Segment 5: phys:0x6b600000, len:2097152, virt:0x710420a00000, socket_id:0,
> hugepage_sz:2097152, nchannel:0, nrank:0
> Segment 6: phys:0x6b800000, len:2097152, virt:0x710420800000, socket_id:0,
> hugepage_sz:2097152, nchannel:0, nrank:0
> Segment 7: phys:0x6ba00000, len:2097152, virt:0x710420600000, socket_id:0,
> hugepage_sz:2097152, nchannel:0, nrank:0
> Segment 8: phys:0x6bc00000, len:2097152, virt:0x710420400000, socket_id:0,
> hugepage_sz:2097152, nchannel:0, nrank:0
> Segment 9: phys:0x6be00000, len:2097152, virt:0x710420200000, socket_id:0,
> hugepage_sz:2097152, nchannel:0, nrank:0
> ---
> 
> 
> 
> 
> 
> Related dpdk code is in
> dpdk/lib/librte_eal/linuxapp/eal/eal_memory.c  :: rte_eal_hugepage_init()
>     for (i = 0; i < nr_hugefiles; i++) {
>         new_memseg = 0;
> 
>         /* if this is a new section, create a new memseg */
>         if (i == 0)
>             new_memseg = 1;
>         else if (hugepage[i].socket_id != hugepage[i-1].socket_id)
>             new_memseg = 1;
>         else if (hugepage[i].size != hugepage[i-1].size)
>             new_memseg = 1;
>         else if ((hugepage[i].physaddr - hugepage[i-1].physaddr) !=
>             hugepage[i].size)
>             new_memseg = 1;
>         else if (((unsigned long)hugepage[i].final_va -
>             (unsigned long)hugepage[i-1].final_va) != hugepage[i].size) {
>             new_memseg = 1;
>         }
> 
> 
> 
> Is this a known issue? Is there any workaround? Or Could you advise which
> kernel config may relate this this kernel behavior change?
> 
> Thanks,
> Michael

This should not be a problem for Intel DPDK startup, as the EAL should take care of mmaps being done in this order at startup. 
By way of background, where I have seen this occur before is on 32-bit systems, while 64-bit systems tend to mmap in ascending order in every case I've looked at. [Q: is this 32-bit you are running, or 64-bit]. In either case, we modified the EAL memory mapping code some time back to try and take account of this - if you look in map_all_hugepages function in eal_memory.c, you will see that, when we go to do the second mapping of hugepages to line the pages up, we do so by explicitly specifying our preferred address. We get this address by allocating a large block of memory from /dev/zero, taking the address and then freeing it again. Then we map the pages one-at-a-time into that free address block, so that even when the kernel wants to map things from hi to lo address, our address hints still cause things to map from lo to hi. If this does not work for you, I'd be curious to find out why. Do any of the security patches you have applied prevent mmap address hinting from working, for instance?

Regards,
/Bruce