You should not use that memory’s real physical addresses, because we cannot guarantee that the kernel won’t change them under our feet. Please do not do that.

 

However, if you use IOVA as VA mode (that is, if you have an IOMMU on your machine and you’re using VFIO to bind devices to DPDK), then no-huge memory can be used with IOMMU, because then the kernel/IOMMU takes care of all the VA to PA mappings. I honestly cannot answer the “cacheable” question, as it never comes up (at least on IA). What are you trying to do, and how is this relevant?

 

In general, no-huge is meant to be a debug option, and is neither intended nor adequately tested for production workloads.

 

Thanks,

Anatoly

 

From: Nick Tian <nick.tian@longsailingsemi.com>
Sent: Tuesday, August 9, 2022 10:42 AM
To: Burakov, Anatoly <anatoly.burakov@intel.com>; Kinsella, Ray <ray.kinsella@intel.com>; users@dpdk.org
Cc: Jason Liu <jason.liu@longsailingsemi.com>; Sunshine Qin <sunshine.qin@longsailingsemi.com>; Mediter Li <mediter.li@longsailingsemi.com>
Subject: 回复：RE: About memory coherency

 

Hi Burakov

Thanks for your reply.

BTW, about the memory reserved by  memfd_create-->ftruncate-->mmap, 

what on earth is the coherency between cache and DDR? In another word, is it cacheable?uncacheable?

Is it possible for application to pass this memory to a device with DMA controller(I mean pass the PHY addr coverted by  rte_mem_virt2phy to DMA controller)?

If yes, how can we ensure the coherency between cache and DDR?

 

 

static int eal_legacy_hugepage_init(void)

  memfd = memfd_create("nohuge", 0);
...

   fd = memfd;
    flags = MAP_SHARED; //MAP_SHARED means UNCACHEABLE?

 

 

 

------------------原始邮件 ------------------

发件人:Burakov, Anatoly <anatoly.burakov@intel.com>

发送时间:08/09/22 17:25:35

主题:RE: About memory coherency

There are two different issues at play here.

 

The purpose of “no-huge” flag is to run DPDK without requiring hugepage memory. Originally, this has been done using an anonymous mmap() call – so, this memory was not using any fd’s at all. This presents a problem with vhost-user, because it relies on fd’s for its shared memory implementation. This is what memfd (a relatively recent addition to the kernel) is addressing – it’s enabling usage of vhost-user with no-huge because memfd actually does create an fd to back our memory.

 

That said, while description says “malloc”, it istechnically incorrect because there’s no malloc involved in the process. The “malloc” term is simply shorthand for “use regular memory”, and should be understood in that context.

 

Thanks,

Anatoly

 

From: Kinsella, Ray <ray.kinsella@intel.com>
Sent: Tuesday, August 9, 2022 10:04 AM
To: Nick Tian <nick.tian@longsailingsemi.com>; users@dpdk.org
Cc: Burakov, Anatoly <anatoly.burakov@intel.com>
Subject: RE: About memory coherency

 

 

I may be incorrect, but is it not simply the case, that when using the no-huge parameter that MAP_HUGETLB is omitted from flags?

 

Ray K

 

From: Nick Tian <nick.tian@longsailingsemi.com>
Sent: Tuesday 9 August 2022 03:55
To: users@dpdk.org
Subject: About memory coherency

 

Hi

I am confusing about the "no-huge" option of DPDK 21.11.

The dpdk usage said: --no-huge:Use malloc instead of hugetlbfs.

But when I check the EAL source code, I found some code piece like this:

It's look like "no-huge" option will lead dpdk use memfd_create-->ftruncate-->mmap to reserve memory

and then provide to application with rte_malloc.

Am I right?

If so, what the "malloc" in "use malloc instead of hugelbfs" refer to?

 

EAL_memory.c

static int eal_legacy_hugepage_init(void){

....

 if (internal_conf->no_hugetlbfs) {
....
#ifdef MEMFD_SUPPORTED
  /* create a memfd and store it in the segment fd table */
  memfd = memfd_create("nohuge", 0);
......

  /* we got an fd - now resize it */
   if (ftruncate(memfd, internal_conf->memory) < 0) {
.....

   fd = memfd;
    flags = MAP_SHARED;   }
....
  prealloc_addr = msl->base_va;
  addr = mmap(prealloc_addr, mem_sz, PROT_READ | PROT_WRITE,
    flags | MAP_FIXED, fd, 0);

...