Re: [dpdk-dev] [RFC PATCH] app/testpmd: only lock text pages

[Ferruh Yigit <ferruh.yigit@intel.com>]

On 3/10/2020 2:55 PM, David Marchand wrote:
> On Tue, Mar 10, 2020 at 3:49 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>>
>> On 3/6/2020 2:48 PM, David Marchand wrote:
>>> Since 18.05 and the memory subsystem rework, EAL reserves some big
>>> (unused) mappings.
>>>
>>> In testpmd, we have been locking all pages to avoid page faults during
>>> benchmark/performance regression tests [1].
>>> However, asking for locking all the pages triggers issues on FreeBSD [2]
>>> and becomes really heavy in some Linux configurations (see [3], [4]).
>>>
>>> This patch changes the behavior so that testpmd only lock pages
>>> containing .text by default.
>>>
>>> 1: https://git.dpdk.org/dpdk/commit/?id=1c036b16c284
>>> 2: https://git.dpdk.org/dpdk/commit/?id=fb7b8b32cd95
>>> 3: https://bugzilla.redhat.com/show_bug.cgi?id=1786923
>>> 4: http://mails.dpdk.org/archives/dev/2020-February/158477.html
>>>
>>> Signed-off-by: David Marchand <david.marchand@redhat.com>
>>
>> <...>
>>
>>> @@ -3455,6 +3456,42 @@ signal_handler(int signum)
>>>       }
>>>  }
>>>
>>> +static void
>>> +lock_pages(const void *_addr, size_t _len, const char *prefix)
>>> +{
>>> +     const void *addr;
>>> +     size_t pagesize;
>>> +     size_t len;
>>> +
>>> +     /* While Linux does not care, FreeBSD mlock expects page aligned
>>> +      * address (according to the man).
>>> +      */
>>> +     pagesize = sysconf(_SC_PAGESIZE);
>>> +     addr = RTE_PTR_ALIGN_FLOOR(_addr, pagesize);
>>> +     len = _len + ((uintptr_t)_addr & (pagesize - 1));
>>> +     if (mlock(addr, len)) {
>>> +             TESTPMD_LOG(NOTICE, "%s: mlock %p (0x%zx) aligned to %p (0x%zx) failed with error \"%s\"\n",
>>> +                     prefix, _addr, _len, addr, len, strerror(errno));
>>> +     }
>>> +}
>>> +
>>> +static int
>>> +lock_text_cb(struct dl_phdr_info *info, __rte_unused size_t size,
>>> +             __rte_unused void *data)
>>> +{
>>> +     int i;
>>> +
>>> +     for (i = 0; i < info->dlpi_phnum; i++) {
>>> +             void *addr;
>>> +
>>> +             if (info->dlpi_phdr[i].p_memsz == 0)
>>> +                     continue;
>>> +             addr = (void *)(info->dlpi_addr + info->dlpi_phdr[i].p_vaddr);
>>> +             lock_pages(addr, info->dlpi_phdr[i].p_memsz, info->dlpi_name);
>>> +     }
>>> +     return 0;
>>> +}
>>
>> +1 to the idea, testpmd initialization was taking too lock without
>> '--no-mlockall', but this code looks complex for the application level.
>>
>> We can do this for testpmd but does all applications need to do something
>> similar? If so can we have a solution on eal level instead?
> 
> I submitted a patch on the EAL side.
> This makes mlockall way lighter, since it skips pages marked with PROT_NONE.
> http://patchwork.dpdk.org/patch/66469/
> 

Cool,

With that patch timing improves a lot, in my system testpmd initialization
reduced from 38s to 9s. (it was 6s with --no-mlockall).

Do we still need this testpmd patch?