DPDK usage discussions
 help / color / mirror / Atom feed
* Shared memory between two primary DPDK processes
@ 2022-04-08 12:31 Antonio Di Bacco
  2022-04-08 13:26 ` Dmitry Kozlyuk
  0 siblings, 1 reply; 13+ messages in thread
From: Antonio Di Bacco @ 2022-04-08 12:31 UTC (permalink / raw)
  To: users

[-- Attachment #1: Type: text/plain, Size: 384 bytes --]

I know that it is possible to share memory between a primary and secondary
process using rte_memzone_reserve_aligned to allocate memory in primary
that is "seen" also by the secondary. If we have two primary processes
(started with different file-prefix) the same approach is not feasible. I
wonder how to share a chunk of memory hosted on a hugepage between two
primaries.

Regards.

[-- Attachment #2: Type: text/html, Size: 442 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Shared memory between two primary DPDK processes
  2022-04-08 12:31 Shared memory between two primary DPDK processes Antonio Di Bacco
@ 2022-04-08 13:26 ` Dmitry Kozlyuk
  2022-04-08 14:36   ` Ferruh Yigit
  2022-04-08 21:08   ` Antonio Di Bacco
  0 siblings, 2 replies; 13+ messages in thread
From: Dmitry Kozlyuk @ 2022-04-08 13:26 UTC (permalink / raw)
  To: Antonio Di Bacco; +Cc: users

2022-04-08 14:31 (UTC+0200), Antonio Di Bacco:
> I know that it is possible to share memory between a primary and secondary
> process using rte_memzone_reserve_aligned to allocate memory in primary
> that is "seen" also by the secondary. If we have two primary processes
> (started with different file-prefix) the same approach is not feasible. I
> wonder how to share a chunk of memory hosted on a hugepage between two
> primaries.
> 
> Regards.

Hi Antonio,

Correction: all hugepages allocated by DPDK are shared
between primary and secondary processes, not only memzones.

I assume we're talking about processes within one host,
because your previous similar question was about sharing memory between hosts
(as we have discussed offline), which is out of scope for DPDK.

As for the question directly, you need to map the same part of the same file
in the second primary as the hugepage is mapped from in the first primary.
I don't recommend to work with file paths, because their management
is not straightforward (--single-file-segments, for one) and is undocumented.

There is a way to share DPDK memory segment file descriptors.
Although public, this DPDK API is dangerous in the sense that you must
clearly understand what you're doing and how DPDK works.
Hence the question: what is the task you need this sharing for?
Maybe there is a simpler way.

1. In the first primary:

	mz = rte_memzone_reserve()
	ms = rte_mem_virt2memseg(mz->addr)
	fd = rte_memseg_get_fd(ms)
	offset = rte_memseg_get_fd_offset(ms)

2. Use Unix domain sockets with SCM_RIGHTS
   to send "fd" and "offset" to the second primary.

3. In the second primary, after receiving "fd" and "offset":

	flags = MAP_SHARED | MAP_HUGE | (30 << MAP_HUGE_SHIFT)
	addr = mmap(fd, offset, flags)

Note that "mz" may consist of multiple "ms" depending on the sizes
of the zone and hugepages, and on the zone alignment.
Also "addr" may (and probably will) differ from "mz->addr".
It is possible to pass "mz->addr" and try to force it,
like DPDK does for primary/secondary.


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Shared memory between two primary DPDK processes
  2022-04-08 13:26 ` Dmitry Kozlyuk
@ 2022-04-08 14:36   ` Ferruh Yigit
  2022-04-08 21:14     ` Antonio Di Bacco
  2022-04-08 21:08   ` Antonio Di Bacco
  1 sibling, 1 reply; 13+ messages in thread
From: Ferruh Yigit @ 2022-04-08 14:36 UTC (permalink / raw)
  To: Dmitry Kozlyuk, Antonio Di Bacco; +Cc: users

On 4/8/2022 2:26 PM, Dmitry Kozlyuk wrote:
> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
> 
> 
> 2022-04-08 14:31 (UTC+0200), Antonio Di Bacco:
>> I know that it is possible to share memory between a primary and secondary
>> process using rte_memzone_reserve_aligned to allocate memory in primary
>> that is "seen" also by the secondary. If we have two primary processes
>> (started with different file-prefix) the same approach is not feasible. I
>> wonder how to share a chunk of memory hosted on a hugepage between two
>> primaries.
>>
>> Regards.
> 
> Hi Antonio,
> 
> Correction: all hugepages allocated by DPDK are shared
> between primary and secondary processes, not only memzones.
> 
> I assume we're talking about processes within one host,
> because your previous similar question was about sharing memory between hosts
> (as we have discussed offline), which is out of scope for DPDK.
> 
> As for the question directly, you need to map the same part of the same file
> in the second primary as the hugepage is mapped from in the first primary.
> I don't recommend to work with file paths, because their management
> is not straightforward (--single-file-segments, for one) and is undocumented.
> 
> There is a way to share DPDK memory segment file descriptors.
> Although public, this DPDK API is dangerous in the sense that you must
> clearly understand what you're doing and how DPDK works.
> Hence the question: what is the task you need this sharing for?
> Maybe there is a simpler way.
> 
> 1. In the first primary:
> 
>          mz = rte_memzone_reserve()
>          ms = rte_mem_virt2memseg(mz->addr)
>          fd = rte_memseg_get_fd(ms)
>          offset = rte_memseg_get_fd_offset(ms)
> 
> 2. Use Unix domain sockets with SCM_RIGHTS
>     to send "fd" and "offset" to the second primary.
> 
> 3. In the second primary, after receiving "fd" and "offset":
> 
>          flags = MAP_SHARED | MAP_HUGE | (30 << MAP_HUGE_SHIFT)
>          addr = mmap(fd, offset, flags)
> 
> Note that "mz" may consist of multiple "ms" depending on the sizes
> of the zone and hugepages, and on the zone alignment.
> Also "addr" may (and probably will) differ from "mz->addr".
> It is possible to pass "mz->addr" and try to force it,
> like DPDK does for primary/secondary.
> 

Also 'net/memif' driver can be used:
https://doc.dpdk.org/guides/nics/memif.html

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Shared memory between two primary DPDK processes
  2022-04-08 13:26 ` Dmitry Kozlyuk
  2022-04-08 14:36   ` Ferruh Yigit
@ 2022-04-08 21:08   ` Antonio Di Bacco
  2022-04-11 13:03     ` Antonio Di Bacco
  1 sibling, 1 reply; 13+ messages in thread
From: Antonio Di Bacco @ 2022-04-08 21:08 UTC (permalink / raw)
  To: Dmitry Kozlyuk; +Cc: users

[-- Attachment #1: Type: text/plain, Size: 2369 bytes --]

Il giorno ven 8 apr 2022 alle ore 15:26 Dmitry Kozlyuk <
dmitry.kozliuk@gmail.com> ha scritto:

> 2022-04-08 14:31 (UTC+0200), Antonio Di Bacco:
> > I know that it is possible to share memory between a primary and
> secondary
> > process using rte_memzone_reserve_aligned to allocate memory in primary
> > that is "seen" also by the secondary. If we have two primary processes
> > (started with different file-prefix) the same approach is not feasible. I
> > wonder how to share a chunk of memory hosted on a hugepage between two
> > primaries.
> >
> > Regards.
>
> Hi Antonio,
>
> Correction: all hugepages allocated by DPDK are shared
> between primary and secondary processes, not only memzones.
>
> I assume we're talking about processes within one host,
> because your previous similar question was about sharing memory between
> hosts
> (as we have discussed offline), which is out of scope for DPDK.
>
> As for the question directly, you need to map the same part of the same
> file
> in the second primary as the hugepage is mapped from in the first primary.
> I don't recommend to work with file paths, because their management
> is not straightforward (--single-file-segments, for one) and is
> undocumented.
>
> There is a way to share DPDK memory segment file descriptors.
> Although public, this DPDK API is dangerous in the sense that you must
> clearly understand what you're doing and how DPDK works.
> Hence the question: what is the task you need this sharing for?
> Maybe there is a simpler way.
>
> 1. In the first primary:
>
>         mz = rte_memzone_reserve()
>         ms = rte_mem_virt2memseg(mz->addr)
>         fd = rte_memseg_get_fd(ms)
>         offset = rte_memseg_get_fd_offset(ms)
>
> 2. Use Unix domain sockets with SCM_RIGHTS
>    to send "fd" and "offset" to the second primary.
>
> 3. In the second primary, after receiving "fd" and "offset":
>
>         flags = MAP_SHARED | MAP_HUGE | (30 << MAP_HUGE_SHIFT)
>         addr = mmap(fd, offset, flags)
>
> Note that "mz" may consist of multiple "ms" depending on the sizes
> of the zone and hugepages, and on the zone alignment.
> Also "addr" may (and probably will) differ from "mz->addr".
> It is possible to pass "mz->addr" and try to force it,
> like DPDK does for primary/secondary.
>


Thank you Dmitry, it is really incredible how deep your knowledge is. I
will give it a try.

[-- Attachment #2: Type: text/html, Size: 3046 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Shared memory between two primary DPDK processes
  2022-04-08 14:36   ` Ferruh Yigit
@ 2022-04-08 21:14     ` Antonio Di Bacco
  0 siblings, 0 replies; 13+ messages in thread
From: Antonio Di Bacco @ 2022-04-08 21:14 UTC (permalink / raw)
  To: Ferruh Yigit; +Cc: Dmitry Kozlyuk, users

[-- Attachment #1: Type: text/plain, Size: 3142 bytes --]

Il giorno ven 8 apr 2022 alle ore 16:36 Ferruh Yigit <
ferruh.yigit@xilinx.com> ha scritto:

> On 4/8/2022 2:26 PM, Dmitry Kozlyuk wrote:
> > CAUTION: This message has originated from an External Source. Please use
> proper judgment and caution when opening attachments, clicking links, or
> responding to this email.
> >
> >
> > 2022-04-08 14:31 (UTC+0200), Antonio Di Bacco:
> >> I know that it is possible to share memory between a primary and
> secondary
> >> process using rte_memzone_reserve_aligned to allocate memory in primary
> >> that is "seen" also by the secondary. If we have two primary processes
> >> (started with different file-prefix) the same approach is not feasible.
> I
> >> wonder how to share a chunk of memory hosted on a hugepage between two
> >> primaries.
> >>
> >> Regards.
> >
> > Hi Antonio,
> >
> > Correction: all hugepages allocated by DPDK are shared
> > between primary and secondary processes, not only memzones.
> >
> > I assume we're talking about processes within one host,
> > because your previous similar question was about sharing memory between
> hosts
> > (as we have discussed offline), which is out of scope for DPDK.
> >
> > As for the question directly, you need to map the same part of the same
> file
> > in the second primary as the hugepage is mapped from in the first
> primary.
> > I don't recommend to work with file paths, because their management
> > is not straightforward (--single-file-segments, for one) and is
> undocumented.
> >
> > There is a way to share DPDK memory segment file descriptors.
> > Although public, this DPDK API is dangerous in the sense that you must
> > clearly understand what you're doing and how DPDK works.
> > Hence the question: what is the task you need this sharing for?
> > Maybe there is a simpler way.
> >
> > 1. In the first primary:
> >
> >          mz = rte_memzone_reserve()
> >          ms = rte_mem_virt2memseg(mz->addr)
> >          fd = rte_memseg_get_fd(ms)
> >          offset = rte_memseg_get_fd_offset(ms)
> >
> > 2. Use Unix domain sockets with SCM_RIGHTS
> >     to send "fd" and "offset" to the second primary.
> >
> > 3. In the second primary, after receiving "fd" and "offset":
> >
> >          flags = MAP_SHARED | MAP_HUGE | (30 << MAP_HUGE_SHIFT)
> >          addr = mmap(fd, offset, flags)
> >
> > Note that "mz" may consist of multiple "ms" depending on the sizes
> > of the zone and hugepages, and on the zone alignment.
> > Also "addr" may (and probably will) differ from "mz->addr".
> > It is possible to pass "mz->addr" and try to force it,
> > like DPDK does for primary/secondary.
> >
>
> Also 'net/memif' driver can be used:
> https://doc.dpdk.org/guides/nics/memif.html


Yes, I know about memif. Our application is currently using a chunk of
shared memory, a primary process writes on it and a secondary reads from
it.  Now the secondary will become a primary, sort of a promotion, and
MEMIF would be fine but the paradigm should change a little bit compared to
a shared memory approach.
MEMIF is an interface over a shared memory, we would need the opposite, a
shared memory over a network interface.

Thank you.

[-- Attachment #2: Type: text/html, Size: 4119 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Shared memory between two primary DPDK processes
  2022-04-08 21:08   ` Antonio Di Bacco
@ 2022-04-11 13:03     ` Antonio Di Bacco
  2022-04-11 17:30       ` Dmitry Kozlyuk
  0 siblings, 1 reply; 13+ messages in thread
From: Antonio Di Bacco @ 2022-04-11 13:03 UTC (permalink / raw)
  To: Dmitry Kozlyuk; +Cc: users

[-- Attachment #1: Type: text/plain, Size: 2923 bytes --]

I did a short program where a  primary (--file-prefix=p1) allocates a
memzone and generates a file descriptor that is passed to another primary
(--file-prefix=p2) .
The process P2 tries to mmap the memory but I get an error (Bad file
descriptor):

        uint64_t* addr = mmap(NULL, 1024*1024*1024, PROT_READ, flags,
mem_fd, 0);
        if (addr == -1)
            perror("mmap");

Il giorno ven 8 apr 2022 alle ore 23:08 Antonio Di Bacco <
a.dibacco.ks@gmail.com> ha scritto:

>
>
> Il giorno ven 8 apr 2022 alle ore 15:26 Dmitry Kozlyuk <
> dmitry.kozliuk@gmail.com> ha scritto:
>
>> 2022-04-08 14:31 (UTC+0200), Antonio Di Bacco:
>> > I know that it is possible to share memory between a primary and
>> secondary
>> > process using rte_memzone_reserve_aligned to allocate memory in primary
>> > that is "seen" also by the secondary. If we have two primary processes
>> > (started with different file-prefix) the same approach is not feasible.
>> I
>> > wonder how to share a chunk of memory hosted on a hugepage between two
>> > primaries.
>> >
>> > Regards.
>>
>> Hi Antonio,
>>
>> Correction: all hugepages allocated by DPDK are shared
>> between primary and secondary processes, not only memzones.
>>
>> I assume we're talking about processes within one host,
>> because your previous similar question was about sharing memory between
>> hosts
>> (as we have discussed offline), which is out of scope for DPDK.
>>
>> As for the question directly, you need to map the same part of the same
>> file
>> in the second primary as the hugepage is mapped from in the first primary.
>> I don't recommend to work with file paths, because their management
>> is not straightforward (--single-file-segments, for one) and is
>> undocumented.
>>
>> There is a way to share DPDK memory segment file descriptors.
>> Although public, this DPDK API is dangerous in the sense that you must
>> clearly understand what you're doing and how DPDK works.
>> Hence the question: what is the task you need this sharing for?
>> Maybe there is a simpler way.
>>
>> 1. In the first primary:
>>
>>         mz = rte_memzone_reserve()
>>         ms = rte_mem_virt2memseg(mz->addr)
>>         fd = rte_memseg_get_fd(ms)
>>         offset = rte_memseg_get_fd_offset(ms)
>>
>> 2. Use Unix domain sockets with SCM_RIGHTS
>>    to send "fd" and "offset" to the second primary.
>>
>> 3. In the second primary, after receiving "fd" and "offset":
>>
>>         flags = MAP_SHARED | MAP_HUGE | (30 << MAP_HUGE_SHIFT)
>>         addr = mmap(fd, offset, flags)
>>
>> Note that "mz" may consist of multiple "ms" depending on the sizes
>> of the zone and hugepages, and on the zone alignment.
>> Also "addr" may (and probably will) differ from "mz->addr".
>> It is possible to pass "mz->addr" and try to force it,
>> like DPDK does for primary/secondary.
>>
>
>
> Thank you Dmitry, it is really incredible how deep your knowledge is. I
> will give it a try.
>

[-- Attachment #2: Type: text/html, Size: 3867 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Shared memory between two primary DPDK processes
  2022-04-11 13:03     ` Antonio Di Bacco
@ 2022-04-11 17:30       ` Dmitry Kozlyuk
  2022-04-14  8:20         ` Antonio Di Bacco
  0 siblings, 1 reply; 13+ messages in thread
From: Dmitry Kozlyuk @ 2022-04-11 17:30 UTC (permalink / raw)
  To: Antonio Di Bacco; +Cc: users

2022-04-11 15:03 (UTC+0200), Antonio Di Bacco:
> I did a short program where a  primary (--file-prefix=p1) allocates a
> memzone and generates a file descriptor that is passed to another primary
> (--file-prefix=p2) .
> The process P2 tries to mmap the memory but I get an error (Bad file
> descriptor):
> 
>         uint64_t* addr = mmap(NULL, 1024*1024*1024, PROT_READ, flags,
> mem_fd, 0);
>         if (addr == -1)
>             perror("mmap");

How do you pass the FD?
Memif does the same thing under the hood, so you should be able too.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Shared memory between two primary DPDK processes
  2022-04-11 17:30       ` Dmitry Kozlyuk
@ 2022-04-14  8:20         ` Antonio Di Bacco
  2022-04-14 19:01           ` Dmitry Kozlyuk
  0 siblings, 1 reply; 13+ messages in thread
From: Antonio Di Bacco @ 2022-04-14  8:20 UTC (permalink / raw)
  To: Dmitry Kozlyuk; +Cc: users

[-- Attachment #1: Type: text/plain, Size: 1161 bytes --]

Il giorno lun 11 apr 2022 alle ore 19:30 Dmitry Kozlyuk <
dmitry.kozliuk@gmail.com> ha scritto:

> 2022-04-11 15:03 (UTC+0200), Antonio Di Bacco:
> > I did a short program where a  primary (--file-prefix=p1) allocates a
> > memzone and generates a file descriptor that is passed to another primary
> > (--file-prefix=p2) .
> > The process P2 tries to mmap the memory but I get an error (Bad file
> > descriptor):
> >
> >         uint64_t* addr = mmap(NULL, 1024*1024*1024, PROT_READ, flags,
> > mem_fd, 0);
> >         if (addr == -1)
> >             perror("mmap");
>
> How do you pass the FD?
> Memif does the same thing under the hood, so you should be able too.
>


Ok, after having a look to memif I managed to exchange the fd  between the
two processes and it works.
Anyway the procedure seems a little bit clunky and I think I'm going to use
the new SYSCALL pidfd_getfd
to achieve the same result.  In your opinion this method (getfd_pidfd)
could also work if the two DPDK processes
are inside different docker containers?
Or is there another mechanims like using handles to hugepages present in
the filesystem to share between two
different containers?

[-- Attachment #2: Type: text/html, Size: 1699 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Shared memory between two primary DPDK processes
  2022-04-14  8:20         ` Antonio Di Bacco
@ 2022-04-14 19:01           ` Dmitry Kozlyuk
  2022-04-14 19:51             ` Antonio Di Bacco
  0 siblings, 1 reply; 13+ messages in thread
From: Dmitry Kozlyuk @ 2022-04-14 19:01 UTC (permalink / raw)
  To: Antonio Di Bacco; +Cc: users

2022-04-14 10:20 (UTC+0200), Antonio Di Bacco:
[...]
> Ok, after having a look to memif I managed to exchange the fd  between the
> two processes and it works.
> Anyway the procedure seems a little bit clunky and I think I'm going to use
> the new SYSCALL pidfd_getfd
> to achieve the same result.  In your opinion this method (getfd_pidfd)
> could also work if the two DPDK processes
> are inside different docker containers?

Honestly, I've just learned about pidfd_getfd() from you.
But I know that containers use PID namespaces, so there's a question
how you will obtain the pidfd of a process in another container.

In general, any method of sharing FD will work.
Remember that you also need offset and size.
Given that some channel is required to share those,
I think Unix domain socket is still the preferred way.

> Or is there another mechanims like using handles to hugepages present in
> the filesystem to share between two
> different containers?

FD is needed for mmap().
You need to either pass the FD or open() the same hugepage file by path.
I advise against using paths because they are not a part of DPDK API contract.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Shared memory between two primary DPDK processes
  2022-04-14 19:01           ` Dmitry Kozlyuk
@ 2022-04-14 19:51             ` Antonio Di Bacco
  2022-04-18 17:34               ` Antonio Di Bacco
  0 siblings, 1 reply; 13+ messages in thread
From: Antonio Di Bacco @ 2022-04-14 19:51 UTC (permalink / raw)
  To: Dmitry Kozlyuk; +Cc: users

[-- Attachment #1: Type: text/plain, Size: 1511 bytes --]

Il giorno gio 14 apr 2022 alle ore 21:01 Dmitry Kozlyuk <
dmitry.kozliuk@gmail.com> ha scritto:

> 2022-04-14 10:20 (UTC+0200), Antonio Di Bacco:
> [...]
> > Ok, after having a look to memif I managed to exchange the fd  between
> the
> > two processes and it works.
> > Anyway the procedure seems a little bit clunky and I think I'm going to
> use
> > the new SYSCALL pidfd_getfd
> > to achieve the same result.  In your opinion this method (getfd_pidfd)
> > could also work if the two DPDK processes
> > are inside different docker containers?
>
> Honestly, I've just learned about pidfd_getfd() from you.
> But I know that containers use PID namespaces, so there's a question
> how you will obtain the pidfd of a process in another container.
>
> In general, any method of sharing FD will work.
> Remember that you also need offset and size.
> Given that some channel is required to share those,
> I think Unix domain socket is still the preferred way.
>
> > Or is there another mechanims like using handles to hugepages present in
> > the filesystem to share between two
> > different containers?
>
> FD is needed for mmap().
> You need to either pass the FD or open() the same hugepage file by path.
> I advise against using paths because they are not a part of DPDK API
> contract.
>

Thank you very much Dmitry, your answers are always enlightening.
I'm going to ask a different question on the dpdk.org about the best
practice to share memory between two dpdk processes running in different
containers.

[-- Attachment #2: Type: text/html, Size: 2007 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Shared memory between two primary DPDK processes
  2022-04-14 19:51             ` Antonio Di Bacco
@ 2022-04-18 17:34               ` Antonio Di Bacco
  2022-04-18 17:53                 ` Antonio Di Bacco
  0 siblings, 1 reply; 13+ messages in thread
From: Antonio Di Bacco @ 2022-04-18 17:34 UTC (permalink / raw)
  To: Dmitry Kozlyuk; +Cc: users

[-- Attachment #1: Type: text/plain, Size: 2779 bytes --]

At the end I tried the pidfd_getfd syscall that is working really fine and
giving me back a "clone" fd of an fd in that was opened from another
process. I tested it opening a text file in the first process  and after
cloning the fd , I could really read the file also in the second process.
Now the weird thing:
1) In the first process I allocate- a huge page, then get the fd
2) In the second process I get my "clone" fd and do an mmap, it works but
if I write on that memory, the first process cannot see what I wrote

int second_process(int remote_pid, int remote_mem_fd) {

        printf("remote_pid %d remote_mem_fd %d\n", remote_pid,
remote_mem_fd);
        int pidfd = syscall(__NR_pidfd_open, remote_pid, 0);

        int my_mem_fd = syscall(438, pidfd, remote_mem_fd, 0);
        printf("my_mem_fd %d\n", my_mem_fd);   // This is nice

        int flags = MAP_SHARED | MAP_HUGETLB | (30 << MAP_HUGE_SHIFT);
        uint64_t* addr = (uint64_t*) mmap(NULL, 1024 * 1024 * 1024,
PROT_READ|PROT_WRITE, flags, my_mem_fd, 0);
        if (addr == -1)
            perror("mmap");
        *addr = 0x0101010102020202;
}


Il giorno gio 14 apr 2022 alle ore 21:51 Antonio Di Bacco <
a.dibacco.ks@gmail.com> ha scritto:

>
>
> Il giorno gio 14 apr 2022 alle ore 21:01 Dmitry Kozlyuk <
> dmitry.kozliuk@gmail.com> ha scritto:
>
>> 2022-04-14 10:20 (UTC+0200), Antonio Di Bacco:
>> [...]
>> > Ok, after having a look to memif I managed to exchange the fd  between
>> the
>> > two processes and it works.
>> > Anyway the procedure seems a little bit clunky and I think I'm going to
>> use
>> > the new SYSCALL pidfd_getfd
>> > to achieve the same result.  In your opinion this method (getfd_pidfd)
>> > could also work if the two DPDK processes
>> > are inside different docker containers?
>>
>> Honestly, I've just learned about pidfd_getfd() from you.
>> But I know that containers use PID namespaces, so there's a question
>> how you will obtain the pidfd of a process in another container.
>>
>> In general, any method of sharing FD will work.
>> Remember that you also need offset and size.
>> Given that some channel is required to share those,
>> I think Unix domain socket is still the preferred way.
>>
>> > Or is there another mechanims like using handles to hugepages present in
>> > the filesystem to share between two
>> > different containers?
>>
>> FD is needed for mmap().
>> You need to either pass the FD or open() the same hugepage file by path.
>> I advise against using paths because they are not a part of DPDK API
>> contract.
>>
>
> Thank you very much Dmitry, your answers are always enlightening.
> I'm going to ask a different question on the dpdk.org about the best
> practice to share memory between two dpdk processes running in different
> containers.
>

[-- Attachment #2: Type: text/html, Size: 3773 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Shared memory between two primary DPDK processes
  2022-04-18 17:34               ` Antonio Di Bacco
@ 2022-04-18 17:53                 ` Antonio Di Bacco
  2022-04-18 19:08                   ` Dmitry Kozlyuk
  0 siblings, 1 reply; 13+ messages in thread
From: Antonio Di Bacco @ 2022-04-18 17:53 UTC (permalink / raw)
  To: Dmitry Kozlyuk; +Cc: users

[-- Attachment #1: Type: text/plain, Size: 3502 bytes --]

Another info to add:

The process that allocates the 1GB page has this map:
antodib@Ubuntu-20.04-5:: /proc> sudo cat /proc/27812/maps | grep huge
140000000-180000000 rw-s 00000000 00:46 97193
 /dev/huge1G/rtemap_0

while the process that maps the 1GB page (--file-prefix p2) has this maps,
is stealing a new page?
antodib@Ubuntu-20.04-5:: /proc> sudo cat /proc/27906/maps | grep huge
140000000-180000000 rw-s 00000000 00:46 113170
/dev/huge1G/p2map_0
7f7bc0000000-7f7c00000000 rw-s 00000000 00:46 97193
 /dev/huge1G/rtemap_0

Il giorno lun 18 apr 2022 alle ore 19:34 Antonio Di Bacco <
a.dibacco.ks@gmail.com> ha scritto:

> At the end I tried the pidfd_getfd syscall that is working really fine and
> giving me back a "clone" fd of an fd in that was opened from another
> process. I tested it opening a text file in the first process  and after
> cloning the fd , I could really read the file also in the second process.
> Now the weird thing:
> 1) In the first process I allocate- a huge page, then get the fd
> 2) In the second process I get my "clone" fd and do an mmap, it works but
> if I write on that memory, the first process cannot see what I wrote
>
> int second_process(int remote_pid, int remote_mem_fd) {
>
>         printf("remote_pid %d remote_mem_fd %d\n", remote_pid,
> remote_mem_fd);
>         int pidfd = syscall(__NR_pidfd_open, remote_pid, 0);
>
>         int my_mem_fd = syscall(438, pidfd, remote_mem_fd, 0);
>         printf("my_mem_fd %d\n", my_mem_fd);   // This is nice
>
>         int flags = MAP_SHARED | MAP_HUGETLB | (30 << MAP_HUGE_SHIFT);
>         uint64_t* addr = (uint64_t*) mmap(NULL, 1024 * 1024 * 1024,
> PROT_READ|PROT_WRITE, flags, my_mem_fd, 0);
>         if (addr == -1)
>             perror("mmap");
>         *addr = 0x0101010102020202;
> }
>
>
> Il giorno gio 14 apr 2022 alle ore 21:51 Antonio Di Bacco <
> a.dibacco.ks@gmail.com> ha scritto:
>
>>
>>
>> Il giorno gio 14 apr 2022 alle ore 21:01 Dmitry Kozlyuk <
>> dmitry.kozliuk@gmail.com> ha scritto:
>>
>>> 2022-04-14 10:20 (UTC+0200), Antonio Di Bacco:
>>> [...]
>>> > Ok, after having a look to memif I managed to exchange the fd  between
>>> the
>>> > two processes and it works.
>>> > Anyway the procedure seems a little bit clunky and I think I'm going
>>> to use
>>> > the new SYSCALL pidfd_getfd
>>> > to achieve the same result.  In your opinion this method (getfd_pidfd)
>>> > could also work if the two DPDK processes
>>> > are inside different docker containers?
>>>
>>> Honestly, I've just learned about pidfd_getfd() from you.
>>> But I know that containers use PID namespaces, so there's a question
>>> how you will obtain the pidfd of a process in another container.
>>>
>>> In general, any method of sharing FD will work.
>>> Remember that you also need offset and size.
>>> Given that some channel is required to share those,
>>> I think Unix domain socket is still the preferred way.
>>>
>>> > Or is there another mechanims like using handles to hugepages present
>>> in
>>> > the filesystem to share between two
>>> > different containers?
>>>
>>> FD is needed for mmap().
>>> You need to either pass the FD or open() the same hugepage file by path.
>>> I advise against using paths because they are not a part of DPDK API
>>> contract.
>>>
>>
>> Thank you very much Dmitry, your answers are always enlightening.
>> I'm going to ask a different question on the dpdk.org about the best
>> practice to share memory between two dpdk processes running in different
>> containers.
>>
>

[-- Attachment #2: Type: text/html, Size: 4887 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Shared memory between two primary DPDK processes
  2022-04-18 17:53                 ` Antonio Di Bacco
@ 2022-04-18 19:08                   ` Dmitry Kozlyuk
  0 siblings, 0 replies; 13+ messages in thread
From: Dmitry Kozlyuk @ 2022-04-18 19:08 UTC (permalink / raw)
  To: Antonio Di Bacco; +Cc: users

2022-04-18 19:53 (UTC+0200), Antonio Di Bacco:
> Another info to add:
> 
> The process that allocates the 1GB page has this map:
> antodib@Ubuntu-20.04-5:: /proc> sudo cat /proc/27812/maps | grep huge
> 140000000-180000000 rw-s 00000000 00:46 97193
>  /dev/huge1G/rtemap_0
> 
> while the process that maps the 1GB page (--file-prefix p2) has this maps,
> is stealing a new page?
> antodib@Ubuntu-20.04-5:: /proc> sudo cat /proc/27906/maps | grep huge
> 140000000-180000000 rw-s 00000000 00:46 113170
> /dev/huge1G/p2map_0
> 7f7bc0000000-7f7c00000000 rw-s 00000000 00:46 97193
>  /dev/huge1G/rtemap_0
> 
> Il giorno lun 18 apr 2022 alle ore 19:34 Antonio Di Bacco <
> a.dibacco.ks@gmail.com> ha scritto:  
> 
> > At the end I tried the pidfd_getfd syscall that is working really fine and
> > giving me back a "clone" fd of an fd in that was opened from another
> > process. I tested it opening a text file in the first process  and after
> > cloning the fd , I could really read the file also in the second process.
> > Now the weird thing:
> > 1) In the first process I allocate- a huge page, then get the fd
> > 2) In the second process I get my "clone" fd and do an mmap, it works but
> > if I write on that memory, the first process cannot see what I wrote
> >
> > int second_process(int remote_pid, int remote_mem_fd) {
> >
> >         printf("remote_pid %d remote_mem_fd %d\n", remote_pid,
> > remote_mem_fd);
> >         int pidfd = syscall(__NR_pidfd_open, remote_pid, 0);
> >
> >         int my_mem_fd = syscall(438, pidfd, remote_mem_fd, 0);
> >         printf("my_mem_fd %d\n", my_mem_fd);   // This is nice
> >
> >         int flags = MAP_SHARED | MAP_HUGETLB | (30 << MAP_HUGE_SHIFT);
> >         uint64_t* addr = (uint64_t*) mmap(NULL, 1024 * 1024 * 1024,
> > PROT_READ|PROT_WRITE, flags, my_mem_fd, 0);
> >         if (addr == -1)
> >             perror("mmap");
> >         *addr = 0x0101010102020202;
> > }

I don't quite understand what do you mean by "stealing a new page",
but it shows that the second process has mapped the same hugepage file
as the first process (just confirms that mmap indeed works).
rte_mem_virt2phy() can tell if the physical address
is really the same in both processes.
Are you sure that the 2nd process writes before the 1st one reads?
Are you sure read/write is not optimized out?

^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2022-04-18 19:08 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-04-08 12:31 Shared memory between two primary DPDK processes Antonio Di Bacco
2022-04-08 13:26 ` Dmitry Kozlyuk
2022-04-08 14:36   ` Ferruh Yigit
2022-04-08 21:14     ` Antonio Di Bacco
2022-04-08 21:08   ` Antonio Di Bacco
2022-04-11 13:03     ` Antonio Di Bacco
2022-04-11 17:30       ` Dmitry Kozlyuk
2022-04-14  8:20         ` Antonio Di Bacco
2022-04-14 19:01           ` Dmitry Kozlyuk
2022-04-14 19:51             ` Antonio Di Bacco
2022-04-18 17:34               ` Antonio Di Bacco
2022-04-18 17:53                 ` Antonio Di Bacco
2022-04-18 19:08                   ` Dmitry Kozlyuk

DPDK usage discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://inbox.dpdk.org/users/0 users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 users users/ http://inbox.dpdk.org/users \
		users@dpdk.org
	public-inbox-index users

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.users


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git