Re: [dpdk-dev] dpdk starting issue with descending virtual address allocation in new kernel

["Michael Hu (NSBU)" <humichael@vmware.com>]

Thanks for the info, Bruce.
We are using 64bit system.

The kernel config that affect this behavior is CONFIG_PAX_RANDMMAP where
it is defined as
---
config PAX_RANDMMAP
    bool "Randomize user stack and mmap() bases"
    default y if GRKERNSEC_CONFIG_AUTO
    depends on PAX_ASLR
    select PAX_RANDUSTACK
    help
      By saying Y here the kernel will randomize every task's userland
      stack and use a randomized base address for mmap() requests that
      do not specify one themselves.

      The stack randomization is done in two steps where the second
      one may apply a big amount of shift to the top of the stack and
      cause problems for programs that want to use lots of memory (more
      than 2.5 GB if SEGMEXEC is not active, or 1.25 GB when it is).

      As a result of mmap randomization all dynamically loaded libraries
      will appear at random addresses and therefore be harder to exploit
      by a technique where an attacker attempts to execute library code
      for his purposes (e.g. spawn a shell from an exploited program that
    
      is running at an elevated privilege level).

      Furthermore, if a program is relinked as a dynamic ELF file, its
      base address will be randomized as well, completing the full
      randomization of the address space layout.  Attacking such programs
    
      becomes a guess game.  You can find an example of doing this at
      http://pax.grsecurity.net/et_dyn.tar.gz and practical samples at
      http://www.grsecurity.net/grsec-gcc-specs.tar.gz .

      NOTE: you can use the 'chpax' or 'paxctl' utilities to control this
    
      feature on a per file basis.
---

With this config, DPDK seems not able to start due to change in mmap.
Please refer to mmap patches in (search CONFIG_PAX_RANDMMAP)
https://grsecurity.net/stable/grsecurity-3.0-3.14.18-201409082127.patch




Thanks,
Michael 






On 9/11/14 2:53 AM, "Richardson, Bruce" <bruce.richardson@intel.com> wrote:

>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Michael Hu (NSBU)
>> Sent: Wednesday, September 10, 2014 11:41 PM
>> To: dev@dpdk.org
>> Subject: [dpdk-dev] dpdk starting issue with descending virtual address
>> allocation in new kernel
>> 
>> Hi All,
>> 
>> We have a kernel config question to consult you.
>> DPDK failed to start due to mbuf creation issue with new kernel 3.14.17
>>+
>> grsecurity patches.
>> We tries to trace down the issue, it seems that  the virtual address of
>>huge page
>> is allocated from high address to low address by kernel where dpdk
>>expects it to
>> be low to high to think it is as consecutive. See dumped virt address
>>bellow. It is
>> first 0x710421400000, then 0x710421200000. Where previously it would be
>> 0x710421200000 first , then 0x710421400000. But they are still
>>consecutive.
>> ----
>> Initialize Port 0 -- TxQ 1, RxQ 1,  Src MAC 00:0c:29:b3:30:db
>>     Create: Default RX  0:0  - Memory used (MBUFs 4096 x (size 1984 +
>>Hdr 64)) +
>> 790720 =   8965 KB
>> Zone 0: name:<RG_MP_log_history>, phys:0x6ac00000, len:0x2080,
>> virt:0x710421400000, socket_id:0, flags:0
>> Zone 1: name:<MP_log_history>, phys:0x6ac02080, len:0x1d10c0,
>> virt:0x710421402080, socket_id:0, flags:0
>> Zone 2: name:<MALLOC_S0_HEAP_0>, phys:0x6ae00000, len:0x160000,
>> virt:0x710421200000, socket_id:0, flags:0
>> Zone 3: name:<rte_eth_dev_data>, phys:0x6add3140, len:0x11a00,
>> virt:0x7104215d3140, socket_id:0, flags:0
>> Zone 4: name:<rte_vmxnet3_pmd_0_shared>, phys:0x6ade4b40, len:0x300,
>> virt:0x7104215e4b40, socket_id:0, flags:0
>> Zone 5: name:<rte_vmxnet3_pmd_0_queuedesc>, phys:0x6ade4e80, len:0x200,
>> virt:0x7104215e4e80, socket_id:0, flags:0
>> Zone 6: name:<RG_MP_Default RX  0:0>, phys:0x6ade5080, len:0x10080,
>> virt:0x7104215e5080, socket_id:0, flags:0
>> Segment 0: phys:0x6ac00000, len:2097152, virt:0x710421400000,
>>socket_id:0,
>> hugepage_sz:2097152, nchannel:0, nrank:0
>> Segment 1: phys:0x6ae00000, len:2097152, virt:0x710421200000,
>>socket_id:0,
>> hugepage_sz:2097152, nchannel:0, nrank:0
>> Segment 2: phys:0x6b000000, len:2097152, virt:0x710421000000,
>>socket_id:0,
>> hugepage_sz:2097152, nchannel:0, nrank:0
>> Segment 3: phys:0x6b200000, len:2097152, virt:0x710420e00000,
>>socket_id:0,
>> hugepage_sz:2097152, nchannel:0, nrank:0
>> Segment 4: phys:0x6b400000, len:2097152, virt:0x710420c00000,
>>socket_id:0,
>> hugepage_sz:2097152, nchannel:0, nrank:0
>> Segment 5: phys:0x6b600000, len:2097152, virt:0x710420a00000,
>>socket_id:0,
>> hugepage_sz:2097152, nchannel:0, nrank:0
>> Segment 6: phys:0x6b800000, len:2097152, virt:0x710420800000,
>>socket_id:0,
>> hugepage_sz:2097152, nchannel:0, nrank:0
>> Segment 7: phys:0x6ba00000, len:2097152, virt:0x710420600000,
>>socket_id:0,
>> hugepage_sz:2097152, nchannel:0, nrank:0
>> Segment 8: phys:0x6bc00000, len:2097152, virt:0x710420400000,
>>socket_id:0,
>> hugepage_sz:2097152, nchannel:0, nrank:0
>> Segment 9: phys:0x6be00000, len:2097152, virt:0x710420200000,
>>socket_id:0,
>> hugepage_sz:2097152, nchannel:0, nrank:0
>> ---
>> 
>> 
>> 
>> 
>> 
>> Related dpdk code is in
>> dpdk/lib/librte_eal/linuxapp/eal/eal_memory.c  ::
>>rte_eal_hugepage_init()
>>     for (i = 0; i < nr_hugefiles; i++) {
>>         new_memseg = 0;
>> 
>>         /* if this is a new section, create a new memseg */
>>         if (i == 0)
>>             new_memseg = 1;
>>         else if (hugepage[i].socket_id != hugepage[i-1].socket_id)
>>             new_memseg = 1;
>>         else if (hugepage[i].size != hugepage[i-1].size)
>>             new_memseg = 1;
>>         else if ((hugepage[i].physaddr - hugepage[i-1].physaddr) !=
>>             hugepage[i].size)
>>             new_memseg = 1;
>>         else if (((unsigned long)hugepage[i].final_va -
>>             (unsigned long)hugepage[i-1].final_va) != hugepage[i].size)
>>{
>>             new_memseg = 1;
>>         }
>> 
>> 
>> 
>> Is this a known issue? Is there any workaround? Or Could you advise
>>which
>> kernel config may relate this this kernel behavior change?
>> 
>> Thanks,
>> Michael
>
>This should not be a problem for Intel DPDK startup, as the EAL should
>take care of mmaps being done in this order at startup.
>By way of background, where I have seen this occur before is on 32-bit
>systems, while 64-bit systems tend to mmap in ascending order in every
>case I've looked at. [Q: is this 32-bit you are running, or 64-bit]. In
>either case, we modified the EAL memory mapping code some time back to
>try and take account of this - if you look in map_all_hugepages function
>in eal_memory.c, you will see that, when we go to do the second mapping
>of hugepages to line the pages up, we do so by explicitly specifying our
>preferred address. We get this address by allocating a large block of
>memory from /dev/zero, taking the address and then freeing it again. Then
>we map the pages one-at-a-time into that free address block, so that even
>when the kernel wants to map things from hi to lo address, our address
>hints still cause things to map from lo to hi. If this does not work for
>you, I'd be curious to find out why. Do any of the security patches you
>have applied prevent mmap address hinting from working, for instance?
>
>Regards,
>/Bruce