From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AE5F4A0093 for ; Mon, 11 Apr 2022 15:03:29 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9C55440698; Mon, 11 Apr 2022 15:03:29 +0200 (CEST) Received: from mail-lf1-f42.google.com (mail-lf1-f42.google.com [209.85.167.42]) by mails.dpdk.org (Postfix) with ESMTP id C2D9F4068C for ; Mon, 11 Apr 2022 15:03:28 +0200 (CEST) Received: by mail-lf1-f42.google.com with SMTP id u19so6402797lff.4 for ; Mon, 11 Apr 2022 06:03:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=+IrWBgKYPeqTMt7OsWaht38/i3dgIbYYMT9DLHobri8=; b=QS4PuJxvT983GHtX80VO4FYcngi7ohHqv4AnaFM6qBi+O2Mn+lhN6BBCiv4Bk1UG9g hNSuONQxUfIGZ3s4oIq9aRQGO9p2lWULXYgW+2/cPfUeOYIlvUzueNYVHx91nEFQ9QuU u30rriBJq9XuZsY5dHPIYlKf67qRZTelNmMV4DZcthbq2jBzOE99yj5ptde4JxLtYeXy HRYAR/GIuj5+kWkP1slZWjnpzwL3GUiajKI0ioN6oan45P8pGxam16TYLo1OXRJFKIKU 5E11Mz2EDLKL2xjSWIaV/oystZGJ9o20W0mHYROwOJOuv0Hfv4/j/iOnqKUIxtHjZzkq GQdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=+IrWBgKYPeqTMt7OsWaht38/i3dgIbYYMT9DLHobri8=; b=l6+pTMITp6FctM4uJJOcEDaf/tamZPoOG+APXhuhhM2boq2SuTJYyCOawjemlC+RGR xUEDEvikEEXetYMo59kp5xBmLpXH+FRrdySyzGP5pqjgmU5M76Ktt2TG5FFHxQUXyWor 3NKDWuaoJQ6LRsISUPvAClg4wter7Jr615kJYvEkwedJLCcvs4QinX57XonC4/1JElCz 4IWKfH3xrvxVDXAqAmMSumSjXDzyL+sqYIGe4+0Z2g3ipy+5QH+rcl1KfRY/ijLfMPNQ /d19WFKantjuA/hy2sjgRln7bY/bQjDSD0rADx6zIo+EY0T5QXQfoJ5I649spW74m2tR MMLQ== X-Gm-Message-State: AOAM530FFVFM+AO/VB5rFz51ryxFnfd513xMKwaKBv08wiQ8zKL2eYS1 ZVlIrOmQ9VqIsIW59Vhc6ou8XECl7XoAMCqSWY17dZlRKIGQrA== X-Google-Smtp-Source: ABdhPJyjOpIbW+8IGj2CjI6C/6+Kf/FSasIrGDNOFgjMNiRow7mLFoKG3zSdUiw3MkxEtzqi38qL7vPysq+d1r/wccQ= X-Received: by 2002:a05:6512:3408:b0:44f:5478:8e48 with SMTP id i8-20020a056512340800b0044f54788e48mr21673788lfr.173.1649682208191; Mon, 11 Apr 2022 06:03:28 -0700 (PDT) MIME-Version: 1.0 References: <20220408162629.372dfd0d@sovereign> In-Reply-To: From: Antonio Di Bacco Date: Mon, 11 Apr 2022 15:03:16 +0200 Message-ID: Subject: Re: Shared memory between two primary DPDK processes To: Dmitry Kozlyuk Cc: users@dpdk.org Content-Type: multipart/alternative; boundary="00000000000002f9a605dc60920a" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --00000000000002f9a605dc60920a Content-Type: text/plain; charset="UTF-8" I did a short program where a primary (--file-prefix=p1) allocates a memzone and generates a file descriptor that is passed to another primary (--file-prefix=p2) . The process P2 tries to mmap the memory but I get an error (Bad file descriptor): uint64_t* addr = mmap(NULL, 1024*1024*1024, PROT_READ, flags, mem_fd, 0); if (addr == -1) perror("mmap"); Il giorno ven 8 apr 2022 alle ore 23:08 Antonio Di Bacco < a.dibacco.ks@gmail.com> ha scritto: > > > Il giorno ven 8 apr 2022 alle ore 15:26 Dmitry Kozlyuk < > dmitry.kozliuk@gmail.com> ha scritto: > >> 2022-04-08 14:31 (UTC+0200), Antonio Di Bacco: >> > I know that it is possible to share memory between a primary and >> secondary >> > process using rte_memzone_reserve_aligned to allocate memory in primary >> > that is "seen" also by the secondary. If we have two primary processes >> > (started with different file-prefix) the same approach is not feasible. >> I >> > wonder how to share a chunk of memory hosted on a hugepage between two >> > primaries. >> > >> > Regards. >> >> Hi Antonio, >> >> Correction: all hugepages allocated by DPDK are shared >> between primary and secondary processes, not only memzones. >> >> I assume we're talking about processes within one host, >> because your previous similar question was about sharing memory between >> hosts >> (as we have discussed offline), which is out of scope for DPDK. >> >> As for the question directly, you need to map the same part of the same >> file >> in the second primary as the hugepage is mapped from in the first primary. >> I don't recommend to work with file paths, because their management >> is not straightforward (--single-file-segments, for one) and is >> undocumented. >> >> There is a way to share DPDK memory segment file descriptors. >> Although public, this DPDK API is dangerous in the sense that you must >> clearly understand what you're doing and how DPDK works. >> Hence the question: what is the task you need this sharing for? >> Maybe there is a simpler way. >> >> 1. In the first primary: >> >> mz = rte_memzone_reserve() >> ms = rte_mem_virt2memseg(mz->addr) >> fd = rte_memseg_get_fd(ms) >> offset = rte_memseg_get_fd_offset(ms) >> >> 2. Use Unix domain sockets with SCM_RIGHTS >> to send "fd" and "offset" to the second primary. >> >> 3. In the second primary, after receiving "fd" and "offset": >> >> flags = MAP_SHARED | MAP_HUGE | (30 << MAP_HUGE_SHIFT) >> addr = mmap(fd, offset, flags) >> >> Note that "mz" may consist of multiple "ms" depending on the sizes >> of the zone and hugepages, and on the zone alignment. >> Also "addr" may (and probably will) differ from "mz->addr". >> It is possible to pass "mz->addr" and try to force it, >> like DPDK does for primary/secondary. >> > > > Thank you Dmitry, it is really incredible how deep your knowledge is. I > will give it a try. > --00000000000002f9a605dc60920a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I did a short program where a =C2=A0primary (--file-prefix= =3Dp1) allocates a memzone and generates a file descriptor that is passed t= o another primary (--file-prefix=3Dp2) .
The process P2 tries to mmap th= e memory but I get an error (Bad file descriptor):=C2=A0 =C2=A0

=C2= =A0 =C2=A0 =C2=A0 =C2=A0 uint64_t* addr =3D mmap(NULL, 1024*1024*1024, PROT= _READ, flags, mem_fd, 0);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (addr =3D=3D -1= )
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 perror("mmap");
Il = giorno ven 8 apr 2022 alle ore 23:08 Antonio Di Bacco <a.dibacco.ks@gmail.com> ha scritto:


Il giorno ven 8 apr 2022 alle ore 15:26 Dmitry Kozlyuk <= dmitry.kozliu= k@gmail.com> ha scritto:
2022-04-08 14:31 (UTC+0200), Antonio Di Bacco:
> I know that it is possible to share memory between a primary and secon= dary
> process using rte_memzone_reserve_aligned to allocate memory in primar= y
> that is "seen" also by the secondary. If we have two primary= processes
> (started with different file-prefix) the same approach is not feasible= . I
> wonder how to share a chunk of memory hosted on a hugepage between two=
> primaries.
>
> Regards.

Hi Antonio,

Correction: all hugepages allocated by DPDK are shared
between primary and secondary processes, not only memzones.

I assume we're talking about processes within one host,
because your previous similar question was about sharing memory between hos= ts
(as we have discussed offline), which is out of scope for DPDK.

As for the question directly, you need to map the same part of the same fil= e
in the second primary as the hugepage is mapped from in the first primary.<= br> I don't recommend to work with file paths, because their management
is not straightforward (--single-file-segments, for one) and is undocumente= d.

There is a way to share DPDK memory segment file descriptors.
Although public, this DPDK API is dangerous in the sense that you must
clearly understand what you're doing and how DPDK works.
Hence the question: what is the task you need this sharing for?
Maybe there is a simpler way.

1. In the first primary:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 mz =3D rte_memzone_reserve()
=C2=A0 =C2=A0 =C2=A0 =C2=A0 ms =3D rte_mem_virt2memseg(mz->addr)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 fd =3D rte_memseg_get_fd(ms)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 offset =3D rte_memseg_get_fd_offset(ms)

2. Use Unix domain sockets with SCM_RIGHTS
=C2=A0 =C2=A0to send "fd" and "offset" to the second pr= imary.

3. In the second primary, after receiving "fd" and "offset&q= uot;:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 flags =3D MAP_SHARED | MAP_HUGE | (30 << = MAP_HUGE_SHIFT)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 addr =3D mmap(fd, offset, flags)

Note that "mz" may consist of multiple "ms" depending o= n the sizes
of the zone and hugepages, and on the zone alignment.
Also "addr" may (and probably will) differ from "mz->addr= ".
It is possible to pass "mz->addr" and try to force it,
like DPDK does for primary/secondary.

<= br>
Thank you Dmitry, it is really incredible how deep your knowl= edge is. I will give it a try.=C2=A0
--00000000000002f9a605dc60920a--