From: Antonio Di Bacco <a.dibacco.ks@gmail.com>
To: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
Cc: users@dpdk.org
Subject: Re: Failure while allocating 1GB hugepages
Date: Mon, 3 Jun 2024 14:39:56 +0200 [thread overview]
Message-ID: <CAO8pfFmiTQzS9VgUVhm3o_hDa2biz62QdGFKc5EWgjFd2TKxSA@mail.gmail.com> (raw)
In-Reply-To: <20240530180022.58cbc8a3@sovereign>
Hi,
I have the same behaviour with the code in this message.
The first rte_memzone_reserve_aligned() call requesting 1.5GB
contiguous memory always fails, while the second one is always
successful.
It seems in eal_memalloc_is_contig() the 'msl->memseg_arr' items are inverted:
when there is the sequence FC0000000, F80000000 the allocation fails,
while the segments sequence F80000000, FC0000000 is fine.
From my understaning 'msl->memseg_arr' comes from
'rte_eal_get_configuration()->mem_config;' which is rte_config
declared in eal_common_config.c
Is there an explanation for this swinging behaviour ?
Br,
Here the source code:
#include <stdio.h>
#include <rte_eal.h>
#include <rte_memzone.h>
#include <rte_errno.h>
#include <unistd.h>
int main(int argc, char **argv)
{
const struct rte_memzone *mz;
if (rte_eal_init(argc, argv) < 0)
return -1;
printf("Allocating : 1.5GB\n");
mz = rte_memzone_reserve_aligned("my_huge_mem", 0x60000000,
rte_socket_id(), RTE_MEMZONE_1GB | RTE_MEMZONE_IOVA_CONTIG,
RTE_CACHE_LINE_SIZE);
if (mz == NULL)
{
printf(" Fail(1): errno %s\n", rte_strerror(rte_errno));
mz = rte_memzone_reserve_aligned("my_huge_mem", 0x60000000,
rte_socket_id(), RTE_MEMZONE_1GB | RTE_MEMZONE_IOVA_CONTIG,
RTE_CACHE_LINE_SIZE);
if (mz == NULL)
{
printf(" Fail(2): errno %s\n", rte_strerror(rte_errno));
return -2;
}
}
printf(" Success: phy[%p] size[%zu]\n", mz->iova, mz->len);
rte_memzone_free(mz);
rte_eal_cleanup();
return 0;
}
I added two RTE_LOG notices in eal_memalloc_is_contig @
eal_common_memalloc.c around line 324, after rte_fbarray_get() to
print ms->iova
/* skip first iteration */
ms = rte_fbarray_get(&msl->memseg_arr, start_seg);
RTE_LOG(NOTICE, EAL, "memseg_arr[0] = %lX \n", ms->iova); // DEBUG
cur = ms->iova;
expected = cur + pgsz;
/* if we can't access IOVA addresses, assume non-contiguous */
if (cur == RTE_BAD_IOVA)
return false;
for (cur_seg = start_seg + 1; cur_seg < end_seg;
cur_seg++, expected += pgsz) {
ms = rte_fbarray_get(&msl->memseg_arr, cur_seg);
RTE_LOG(NOTICE, EAL, "memseg_arr[%d] = %lX \n", cur_seg, ms->iova); // DEBUG
if (ms->iova != expected)
return false;
}
The output is:
Allocating : 1.5GB
EAL: memseg_arr[0] = FC0000000
EAL: memseg_arr[1] = F80000000
Fail(1): errno Cannot allocate memory
EAL: memseg_arr[0] = F80000000
EAL: memseg_arr[1] = FC0000000
EAL: memseg_arr[0] = F80000000
EAL: memseg_arr[1] = FC0000000
EAL: memseg_arr[0] = F80000000
EAL: memseg_arr[1] = FC0000000
EAL: memseg_arr[0] = F80000000
EAL: memseg_arr[1] = FC0000000
Success: phy[0xfa0000000] size[1610612736]
On Thu, May 30, 2024 at 5:00 PM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
>
> 2024-05-30 12:28 (UTC+0200), Antonio Di Bacco:
> > Just in case I need, let us say, 1.5 GB CONTIGUOUS memory zone,
> > would it be fine to use something like this as GRUB config in Linux?
> >
> > default_hugepagesz=2G hugepagesz=2G hugepages=4"
>
> On x86, "hugepagesz" and "default_hugepagesz" may be either 2M or 1G.
> There is no way to *guarantee* that there will be
> two physically adjacent 1G hugepages forming 1.5GB contiguous space,
> but in practice these options, with the above correction, will do.
>
> Note that by default the kernel will spread hugepages between NUMA nodes.
> You can control this by a more elaborate form of "hugepages" option:
>
> https://docs.kernel.org/admin-guide/mm/hugetlbpage.html
next prev parent reply other threads:[~2024-06-03 12:40 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-10 9:33 Antonio Di Bacco
2024-05-10 15:07 ` Dmitry Kozlyuk
2024-05-22 10:22 ` Antonio Di Bacco
2024-05-30 10:28 ` Antonio Di Bacco
2024-05-30 15:00 ` Dmitry Kozlyuk
2024-06-03 12:39 ` Antonio Di Bacco [this message]
2024-06-04 22:50 ` Dmitry Kozlyuk
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAO8pfFmiTQzS9VgUVhm3o_hDa2biz62QdGFKc5EWgjFd2TKxSA@mail.gmail.com \
--to=a.dibacco.ks@gmail.com \
--cc=dmitry.kozliuk@gmail.com \
--cc=users@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).