Re: [dpdk-users] Secondary process cannot call functions on hugepages

[Fengkai Sun <qcloud1014@gmail.com>]

Hi Stephen,

Thanks for your reply!

I'm sure the protection bit is correctly OR-ed with PROT_EXEC. I use
mprotect() on the entire hugepage on both primary and secondary processes,
and /proc/pid/maps file can verify that.

Thanks for your hint on libhugetlbfs. I skimmed through its HOWTO and found
that it uses a custom linker script to load the text/data/bss segment on
hugepages.
However, I didn't see how it can cooperate with the shared objects. And I
also found a SO question on how to do that:
https://stackoverflow.com/questions/40285971/how-to-load-text-segments-of-shared-libraries-into-huge-pages-on-linux

Also, as far as I know, the loading of shared objects is handled by
ld-linux.so, and changing the behavior of ld-linux.so needs some source
level modifications or symbol interpositioning. I compile libhugetlbfs but
find no obvious sign of symbol interpositioning.


--
Best,
Fengkai

On Tue, Jul 20, 2021 at 11:44 PM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Tue, 20 Jul 2021 17:28:59 +0800
> Fengkai Sun <qcloud1014@gmail.com> wrote:
>
> > Hi list,
> >
> > Recently I read about the multi-process support of dpdk and found the
> > feature of keeping the hugepage mappings between primary and secondary
> > processes very interesting.
> > Therefore, I made the following experiments:
> >
> > 1. I modify the glibc to make dlopen() map shared libraries on hugepages.
> > This is done by changing the underlying mapping strategy from mmap to
> pread.
> > 2. I allocate a continuous address by rte_malloc(), and instruct dlopen()
> > to map a shared library there, let's call it lib1.so.
> > 3. After dlopen() succeeds, I use dlsym() in the primary process to
> locate
> > the global variables and functions in lib1.so, and it turns out they are
> > correctly mapped and functions can be called without a problem.
> > 4. Because hugepage seems to get away with the effect of ASLR, I record
> the
> > address of those global variables and functions in lib1.so, and verify
> them
> > in the secondary process.
> >     The secondary can access the global variables at the same address,
> but
> > when it tries to call the function, a segfault occurs.
> > 5. I try to use dlopen() with the same arguments as the primary process
> in
> > the secondary process, but it just gives a segfault.
> >
> > Unfortunately, gdb also gives a segfault when the program starts up, so I
> > cannot give some useful debug info.
> >
> > My question is, does dpdk permit functions to be loaded on hugepages and
> be
> > called by multiple functions? Though the two processes see the exact same
> > content on the hugepage, the secondary just cannot call the function on
> it.
>
>
> Sounds like no-exec bit.
> You might want to look into other libraries that already handle huge
> pages and libraries such as hugetlbfs
>
>
>