From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B2CA3A0C50 for ; Wed, 21 Jul 2021 13:38:42 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9F8E540DF4; Wed, 21 Jul 2021 13:38:42 +0200 (CEST) Received: from mail-lj1-f180.google.com (mail-lj1-f180.google.com [209.85.208.180]) by mails.dpdk.org (Postfix) with ESMTP id F0E784014E for ; Wed, 21 Jul 2021 04:34:36 +0200 (CEST) Received: by mail-lj1-f180.google.com with SMTP id l11so1025121ljq.4 for ; Tue, 20 Jul 2021 19:34:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=TNY2uTShbukuMJh2kVgCF6jhHucyjhYuHf/8W/oz8mk=; b=WvuB6VyB+OWOUpJPot6QabejUWsLJeqTvNIMwtSxflBx81MCPD63dzGO8Juwvsipw7 kUf3wTyG9y+JmI6/ap7nCoNViBGx0zdKPNAG0dLSKSwqcvxYAa8pM0hvAavUakDZO06t lPQoSDDPY/yVG4Q0LdHqq28LAJA5FyNwJku5iXhRb9bZs7OKy1zkx4ARpd3X5iOnomqT 5SMLqSvgzovfO/Xue17refhqDNwah89tPhIYJJ9Wy6Ec1CsuBXgmzuMCNKFNPyU7m67t s3NYR322fGZxZoadgwAb3AcrY2rC3DPH//0fEGQvFr4aFocJwGP2p9bT+0LOklPOHJAy 9XNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=TNY2uTShbukuMJh2kVgCF6jhHucyjhYuHf/8W/oz8mk=; b=kDgim/SUHpUN9d/fcnANWZwApcGvx6Az4fin+gWr2TQruHaWllKAua4q9lErT2D+8g JWCZjdSIOtJ1JNu2qdUC/WL45zPdfdEFkHlNE4HlEY8rHlS5/Iyk7T4jhuihFdpE8KQc wr63IOpqExvJhq8QhRBoqGlSJMnhjFZP15mtIi4x6eN1FraoH2rpxOBVxzcMPF/COOlX gsQShWOa7521cFgWl/qLYn0lOC2ckYFYhkJFCblsMbhR9ZkS1NH10z26vV0zdW3OfErc HobuTqQF1P99OwB996dWaFLalw9Z2EqSxRCuV1jjZ0ebLLIQC1AyFQRtdz00HIOU/3ow 2row== X-Gm-Message-State: AOAM531gQ1mR9s4XiENADC4BE0ei95zn4NQeYbZ4cnVdjC2tUfKdB88x 9qOXzzae22RK6G9ULlTQT2At3DtoaNq3YP0ZFHkI8vg2wYyt5smPay8= X-Google-Smtp-Source: ABdhPJyCwrRuVJOpVy/W3N169WzVARF2bXy4QOqPbH4L+SpmQOH1Wa4sJg0auDPYTFfKmFh+4Nx5rgnPO2nsazWnP4Y= X-Received: by 2002:a2e:9d1a:: with SMTP id t26mr29180755lji.10.1626834876456; Tue, 20 Jul 2021 19:34:36 -0700 (PDT) MIME-Version: 1.0 References: <20210720084430.10dc648c@hermes.local> In-Reply-To: <20210720084430.10dc648c@hermes.local> From: Fengkai Sun Date: Wed, 21 Jul 2021 10:34:24 +0800 Message-ID: To: stephen@networkplumber.org Cc: users@dpdk.org X-Mailman-Approved-At: Wed, 21 Jul 2021 13:38:41 +0200 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [dpdk-users] Secondary process cannot call functions on hugepages X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi Stephen, Thanks for your reply! I'm sure the protection bit is correctly OR-ed with PROT_EXEC. I use mprotect() on the entire hugepage on both primary and secondary processes, and /proc/pid/maps file can verify that. Thanks for your hint on libhugetlbfs. I skimmed through its HOWTO and found that it uses a custom linker script to load the text/data/bss segment on hugepages. However, I didn't see how it can cooperate with the shared objects. And I also found a SO question on how to do that: https://stackoverflow.com/questions/40285971/how-to-load-text-segments-of-shared-libraries-into-huge-pages-on-linux Also, as far as I know, the loading of shared objects is handled by ld-linux.so, and changing the behavior of ld-linux.so needs some source level modifications or symbol interpositioning. I compile libhugetlbfs but find no obvious sign of symbol interpositioning. -- Best, Fengkai On Tue, Jul 20, 2021 at 11:44 PM Stephen Hemminger < stephen@networkplumber.org> wrote: > On Tue, 20 Jul 2021 17:28:59 +0800 > Fengkai Sun wrote: > > > Hi list, > > > > Recently I read about the multi-process support of dpdk and found the > > feature of keeping the hugepage mappings between primary and secondary > > processes very interesting. > > Therefore, I made the following experiments: > > > > 1. I modify the glibc to make dlopen() map shared libraries on hugepages. > > This is done by changing the underlying mapping strategy from mmap to > pread. > > 2. I allocate a continuous address by rte_malloc(), and instruct dlopen() > > to map a shared library there, let's call it lib1.so. > > 3. After dlopen() succeeds, I use dlsym() in the primary process to > locate > > the global variables and functions in lib1.so, and it turns out they are > > correctly mapped and functions can be called without a problem. > > 4. Because hugepage seems to get away with the effect of ASLR, I record > the > > address of those global variables and functions in lib1.so, and verify > them > > in the secondary process. > > The secondary can access the global variables at the same address, > but > > when it tries to call the function, a segfault occurs. > > 5. I try to use dlopen() with the same arguments as the primary process > in > > the secondary process, but it just gives a segfault. > > > > Unfortunately, gdb also gives a segfault when the program starts up, so I > > cannot give some useful debug info. > > > > My question is, does dpdk permit functions to be loaded on hugepages and > be > > called by multiple functions? Though the two processes see the exact same > > content on the hugepage, the secondary just cannot call the function on > it. > > > Sounds like no-exec bit. > You might want to look into other libraries that already handle huge > pages and libraries such as hugetlbfs > > >