[dpdk-users] Secondary process cannot call functions on hugepages

[dpdk-users] Secondary process cannot call functions on hugepages

[Fengkai Sun]

Hi list,

Recently I read about the multi-process support of dpdk and found the
feature of keeping the hugepage mappings between primary and secondary
processes very interesting.
Therefore, I made the following experiments:

1. I modify the glibc to make dlopen() map shared libraries on hugepages.
This is done by changing the underlying mapping strategy from mmap to pread.
2. I allocate a continuous address by rte_malloc(), and instruct dlopen()
to map a shared library there, let's call it lib1.so.
3. After dlopen() succeeds, I use dlsym() in the primary process to locate
the global variables and functions in lib1.so, and it turns out they are
correctly mapped and functions can be called without a problem.
4. Because hugepage seems to get away with the effect of ASLR, I record the
address of those global variables and functions in lib1.so, and verify them
in the secondary process.
    The secondary can access the global variables at the same address, but
when it tries to call the function, a segfault occurs.
5. I try to use dlopen() with the same arguments as the primary process in
the secondary process, but it just gives a segfault.

Unfortunately, gdb also gives a segfault when the program starts up, so I
cannot give some useful debug info.

My question is, does dpdk permit functions to be loaded on hugepages and be
called by multiple functions? Though the two processes see the exact same
content on the hugepage, the secondary just cannot call the function on it.

Any intuitive advice is appreciated.

--
Best,
Fengkai

Re: [dpdk-users] Secondary process cannot call functions on hugepages

[Stephen Hemminger]

On Tue, 20 Jul 2021 17:28:59 +0800
Fengkai Sun <qcloud1014@gmail.com> wrote:

> Hi list,
> 
> Recently I read about the multi-process support of dpdk and found the
> feature of keeping the hugepage mappings between primary and secondary
> processes very interesting.
> Therefore, I made the following experiments:
> 
> 1. I modify the glibc to make dlopen() map shared libraries on hugepages.
> This is done by changing the underlying mapping strategy from mmap to pread.
> 2. I allocate a continuous address by rte_malloc(), and instruct dlopen()
> to map a shared library there, let's call it lib1.so.
> 3. After dlopen() succeeds, I use dlsym() in the primary process to locate
> the global variables and functions in lib1.so, and it turns out they are
> correctly mapped and functions can be called without a problem.
> 4. Because hugepage seems to get away with the effect of ASLR, I record the
> address of those global variables and functions in lib1.so, and verify them
> in the secondary process.
>     The secondary can access the global variables at the same address, but
> when it tries to call the function, a segfault occurs.
> 5. I try to use dlopen() with the same arguments as the primary process in
> the secondary process, but it just gives a segfault.
> 
> Unfortunately, gdb also gives a segfault when the program starts up, so I
> cannot give some useful debug info.
> 
> My question is, does dpdk permit functions to be loaded on hugepages and be
> called by multiple functions? Though the two processes see the exact same
> content on the hugepage, the secondary just cannot call the function on it.


Sounds like no-exec bit.
You might want to look into other libraries that already handle huge
pages and libraries such as hugetlbfs

Re: [dpdk-users] Secondary process cannot call functions on hugepages

[Fengkai Sun]

Hi Stephen,

Thanks for your reply!

I'm sure the protection bit is correctly OR-ed with PROT_EXEC. I use
mprotect() on the entire hugepage on both primary and secondary processes,
and /proc/pid/maps file can verify that.

Thanks for your hint on libhugetlbfs. I skimmed through its HOWTO and found
that it uses a custom linker script to load the text/data/bss segment on
hugepages.
However, I didn't see how it can cooperate with the shared objects. And I
also found a SO question on how to do that:
https://stackoverflow.com/questions/40285971/how-to-load-text-segments-of-shared-libraries-into-huge-pages-on-linux

Also, as far as I know, the loading of shared objects is handled by
ld-linux.so, and changing the behavior of ld-linux.so needs some source
level modifications or symbol interpositioning. I compile libhugetlbfs but
find no obvious sign of symbol interpositioning.


--
Best,
Fengkai

On Tue, Jul 20, 2021 at 11:44 PM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Tue, 20 Jul 2021 17:28:59 +0800
> Fengkai Sun <qcloud1014@gmail.com> wrote:
>
> > Hi list,
> >
> > Recently I read about the multi-process support of dpdk and found the
> > feature of keeping the hugepage mappings between primary and secondary
> > processes very interesting.
> > Therefore, I made the following experiments:
> >
> > 1. I modify the glibc to make dlopen() map shared libraries on hugepages.
> > This is done by changing the underlying mapping strategy from mmap to
> pread.
> > 2. I allocate a continuous address by rte_malloc(), and instruct dlopen()
> > to map a shared library there, let's call it lib1.so.
> > 3. After dlopen() succeeds, I use dlsym() in the primary process to
> locate
> > the global variables and functions in lib1.so, and it turns out they are
> > correctly mapped and functions can be called without a problem.
> > 4. Because hugepage seems to get away with the effect of ASLR, I record
> the
> > address of those global variables and functions in lib1.so, and verify
> them
> > in the secondary process.
> >     The secondary can access the global variables at the same address,
> but
> > when it tries to call the function, a segfault occurs.
> > 5. I try to use dlopen() with the same arguments as the primary process
> in
> > the secondary process, but it just gives a segfault.
> >
> > Unfortunately, gdb also gives a segfault when the program starts up, so I
> > cannot give some useful debug info.
> >
> > My question is, does dpdk permit functions to be loaded on hugepages and
> be
> > called by multiple functions? Though the two processes see the exact same
> > content on the hugepage, the secondary just cannot call the function on
> it.
>
>
> Sounds like no-exec bit.
> You might want to look into other libraries that already handle huge
> pages and libraries such as hugetlbfs
>
>
>