Re: Shared memory between two primary DPDK processes

[Antonio Di Bacco <a.dibacco.ks@gmail.com>]

You are right, process 1 is always allocating 1GB even if I request
only 10MB, and memseg->hugepage_sz is 1GB.
When I use rte_memseg_get_fd_offset I get an FD and the offset is 0
(correct) because that's the memseg offset not the memzone.
To access the memory allocated by process 1 I need to take into
account also the offset between memzone and memseg.
And then I need to add (memzone->iova -  memseg->iova) to the address
returned by mmap.






On Thu, Jul 7, 2022 at 2:26 AM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
>
> 2022-07-07 00:14 (UTC+0200), Antonio Di Bacco:
> > Dear Dmitry,
> >
> > I tried to follow this approach and if I allocate 1GB on primary
> > process number 1, then I can mmap that memory on the primary process
> > number 2.
> > I also tried to convert the virt addr of the allocation made in
> > primary 1 to phys and then I converted the virt addr returned by mmap
> > in primary 2 and I got the same phys addr.
> >
> > Unfortunately, if I try to allocated only 10 MB for example in primary
> > 1, then mmap in primary 2 succeeds but it seems that this virt addr
> > doesn't correspond to the same phys memory as in primary 1.
> >
> > In the primary 2, the mmap is used like this:
> >
> >     int flags = MAP_SHARED | MAP_HUGETLB ;
> >
> >     uint64_t* addr = (uint64_t*) mmap(NULL, sz, PROT_READ|PROT_WRITE,
> > flags, my_mem_fd, off);
>
> Hi Antonio,
>
> From `man 2 mmap`:
>
>    Huge page (Huge TLB) mappings
>        For  mappings that employ huge pages, the requirements for the
>        arguments of mmap() and munmap() differ somewhat from the requirements
>        for mappings that use the native system page size.
>
>        For mmap(), offset must be a multiple of the underlying huge page
>        size.  The system automatically aligns length to be a  multiple  of
>        the underlying huge page size.
>
>        For munmap(), addr, and length must both be a multiple of the
>        underlying huge page size.
>
> Probably process 1 maps a 1 GB hugepage:
> DPDK does so if 1 GB hugepages are used even if you only allocate 10 MB.
> You can examine memseg to see what size it is (not memzone!).
> Hugepage size is a property of each mounted HugeTBL filesystem.
> It determines which kernel pool to use.
> Pools are over different sets of physical pages.
> This means that the kernel doesn't allow to map given page frames
> as 1 GB and 2 MB hugepages at the same time via hugetlbfs.
> I'm surprised mmap() works at all in your case
> and suspect that it is mapping 2 MB hugepages in process 2.
>
> The solution may be, in process 2:
>
> base_offset = RTE_ALIGN_FLOOR(offset, hugepage_size)
> map_addr = mmap(fd, size=hugepage_size, offset=base_offset)
> addr = RTE_PTR_ADD(map_addr, offset - base_offset)
>
> Note that if [offset; offset+size) crosses a hugepage boundary,
> you have to map more than one page.