DPDK usage discussions
 help / color / mirror / Atom feed
* [dpdk-users] Secondary process cannot call functions on hugepages
@ 2021-07-20  9:28 Fengkai Sun
  2021-07-20 15:44 ` Stephen Hemminger
  0 siblings, 1 reply; 3+ messages in thread
From: Fengkai Sun @ 2021-07-20  9:28 UTC (permalink / raw)
  To: users

Hi list,

Recently I read about the multi-process support of dpdk and found the
feature of keeping the hugepage mappings between primary and secondary
processes very interesting.
Therefore, I made the following experiments:

1. I modify the glibc to make dlopen() map shared libraries on hugepages.
This is done by changing the underlying mapping strategy from mmap to pread.
2. I allocate a continuous address by rte_malloc(), and instruct dlopen()
to map a shared library there, let's call it lib1.so.
3. After dlopen() succeeds, I use dlsym() in the primary process to locate
the global variables and functions in lib1.so, and it turns out they are
correctly mapped and functions can be called without a problem.
4. Because hugepage seems to get away with the effect of ASLR, I record the
address of those global variables and functions in lib1.so, and verify them
in the secondary process.
    The secondary can access the global variables at the same address, but
when it tries to call the function, a segfault occurs.
5. I try to use dlopen() with the same arguments as the primary process in
the secondary process, but it just gives a segfault.

Unfortunately, gdb also gives a segfault when the program starts up, so I
cannot give some useful debug info.

My question is, does dpdk permit functions to be loaded on hugepages and be
called by multiple functions? Though the two processes see the exact same
content on the hugepage, the secondary just cannot call the function on it.

Any intuitive advice is appreciated.

--
Best,
Fengkai

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [dpdk-users] Secondary process cannot call functions on hugepages
  2021-07-20  9:28 [dpdk-users] Secondary process cannot call functions on hugepages Fengkai Sun
@ 2021-07-20 15:44 ` Stephen Hemminger
  2021-07-21  2:34   ` Fengkai Sun
  0 siblings, 1 reply; 3+ messages in thread
From: Stephen Hemminger @ 2021-07-20 15:44 UTC (permalink / raw)
  To: Fengkai Sun; +Cc: users

On Tue, 20 Jul 2021 17:28:59 +0800
Fengkai Sun <qcloud1014@gmail.com> wrote:

> Hi list,
> 
> Recently I read about the multi-process support of dpdk and found the
> feature of keeping the hugepage mappings between primary and secondary
> processes very interesting.
> Therefore, I made the following experiments:
> 
> 1. I modify the glibc to make dlopen() map shared libraries on hugepages.
> This is done by changing the underlying mapping strategy from mmap to pread.
> 2. I allocate a continuous address by rte_malloc(), and instruct dlopen()
> to map a shared library there, let's call it lib1.so.
> 3. After dlopen() succeeds, I use dlsym() in the primary process to locate
> the global variables and functions in lib1.so, and it turns out they are
> correctly mapped and functions can be called without a problem.
> 4. Because hugepage seems to get away with the effect of ASLR, I record the
> address of those global variables and functions in lib1.so, and verify them
> in the secondary process.
>     The secondary can access the global variables at the same address, but
> when it tries to call the function, a segfault occurs.
> 5. I try to use dlopen() with the same arguments as the primary process in
> the secondary process, but it just gives a segfault.
> 
> Unfortunately, gdb also gives a segfault when the program starts up, so I
> cannot give some useful debug info.
> 
> My question is, does dpdk permit functions to be loaded on hugepages and be
> called by multiple functions? Though the two processes see the exact same
> content on the hugepage, the secondary just cannot call the function on it.


Sounds like no-exec bit.
You might want to look into other libraries that already handle huge
pages and libraries such as hugetlbfs



^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [dpdk-users] Secondary process cannot call functions on hugepages
  2021-07-20 15:44 ` Stephen Hemminger
@ 2021-07-21  2:34   ` Fengkai Sun
  0 siblings, 0 replies; 3+ messages in thread
From: Fengkai Sun @ 2021-07-21  2:34 UTC (permalink / raw)
  To: stephen; +Cc: users

Hi Stephen,

Thanks for your reply!

I'm sure the protection bit is correctly OR-ed with PROT_EXEC. I use
mprotect() on the entire hugepage on both primary and secondary processes,
and /proc/pid/maps file can verify that.

Thanks for your hint on libhugetlbfs. I skimmed through its HOWTO and found
that it uses a custom linker script to load the text/data/bss segment on
hugepages.
However, I didn't see how it can cooperate with the shared objects. And I
also found a SO question on how to do that:
https://stackoverflow.com/questions/40285971/how-to-load-text-segments-of-shared-libraries-into-huge-pages-on-linux

Also, as far as I know, the loading of shared objects is handled by
ld-linux.so, and changing the behavior of ld-linux.so needs some source
level modifications or symbol interpositioning. I compile libhugetlbfs but
find no obvious sign of symbol interpositioning.


--
Best,
Fengkai

On Tue, Jul 20, 2021 at 11:44 PM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Tue, 20 Jul 2021 17:28:59 +0800
> Fengkai Sun <qcloud1014@gmail.com> wrote:
>
> > Hi list,
> >
> > Recently I read about the multi-process support of dpdk and found the
> > feature of keeping the hugepage mappings between primary and secondary
> > processes very interesting.
> > Therefore, I made the following experiments:
> >
> > 1. I modify the glibc to make dlopen() map shared libraries on hugepages.
> > This is done by changing the underlying mapping strategy from mmap to
> pread.
> > 2. I allocate a continuous address by rte_malloc(), and instruct dlopen()
> > to map a shared library there, let's call it lib1.so.
> > 3. After dlopen() succeeds, I use dlsym() in the primary process to
> locate
> > the global variables and functions in lib1.so, and it turns out they are
> > correctly mapped and functions can be called without a problem.
> > 4. Because hugepage seems to get away with the effect of ASLR, I record
> the
> > address of those global variables and functions in lib1.so, and verify
> them
> > in the secondary process.
> >     The secondary can access the global variables at the same address,
> but
> > when it tries to call the function, a segfault occurs.
> > 5. I try to use dlopen() with the same arguments as the primary process
> in
> > the secondary process, but it just gives a segfault.
> >
> > Unfortunately, gdb also gives a segfault when the program starts up, so I
> > cannot give some useful debug info.
> >
> > My question is, does dpdk permit functions to be loaded on hugepages and
> be
> > called by multiple functions? Though the two processes see the exact same
> > content on the hugepage, the secondary just cannot call the function on
> it.
>
>
> Sounds like no-exec bit.
> You might want to look into other libraries that already handle huge
> pages and libraries such as hugetlbfs
>
>
>

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2021-07-21 11:38 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-20  9:28 [dpdk-users] Secondary process cannot call functions on hugepages Fengkai Sun
2021-07-20 15:44 ` Stephen Hemminger
2021-07-21  2:34   ` Fengkai Sun

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).