DPDK patches and discussions
 help / color / mirror / Atom feed
From: Slava Ovsiienko <viacheslavo@nvidia.com>
To: Stephen Hemminger <stephen@networkplumber.org>
Cc: Erez Ferber <erezf@nvidia.com>, "dev@dpdk.org" <dev@dpdk.org>,
	Matan Azrad <matan@nvidia.com>,
	Raslan Darawsheh <rasland@nvidia.com>,
	"stable@dpdk.org" <stable@dpdk.org>
Subject: RE: [PATCH] common/mlx5: adjust fork call with the new kernel API
Date: Fri, 26 May 2023 08:05:34 +0000	[thread overview]
Message-ID: <DM6PR12MB3753378500E5DFCBCD6657CBDF479@DM6PR12MB3753.namprd12.prod.outlook.com> (raw)
In-Reply-To: <20230525082756.630ab7f6@hermes.local>

> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Thursday, May 25, 2023 6:28 PM
> To: Slava Ovsiienko <viacheslavo@nvidia.com>
> Cc: Erez Ferber <erezf@nvidia.com>; dev@dpdk.org; Matan Azrad
> <matan@nvidia.com>; Raslan Darawsheh <rasland@nvidia.com>;
> stable@dpdk.org
> Subject: Re: [PATCH] common/mlx5: adjust fork call with the new kernel API
> 
> On Thu, 25 May 2023 08:10:03 +0000
> Slava Ovsiienko <viacheslavo@nvidia.com> wrote:
> 
> > > -----Original Message-----
> > > From: Stephen Hemminger <stephen@networkplumber.org>
> > > Sent: Wednesday, May 24, 2023 5:50 PM
> > > To: Erez Ferber <erezf@nvidia.com>
> > > Cc: dev@dpdk.org; Slava Ovsiienko <viacheslavo@nvidia.com>; Matan
> > > Azrad <matan@nvidia.com>; Raslan Darawsheh <rasland@nvidia.com>;
> > > stable@dpdk.org
> > > Subject: Re: [PATCH] common/mlx5: adjust fork call with the new
> > > kernel API
> > >
> > > On Wed, 24 May 2023 15:01:40 +0300
> > > <erezf@nvidia.com> wrote:
> > >
> > > > From: Erez Ferber <erezf@nvidia.com>
> > > >
> > > > While doing process fork() the operating system remaps all the
> > > > parent process's memory to the address space of the child process
> > > > and activates the Copy-on-Write mechanics - it duplicates physical
> > > > pages once memory writing happens in the child process. Sometimes
> > > > memory duplication is not allowed - for example, if the page
> > > > contains hardware queue descriptors. To handle similar issues the
> > > > rdma-core library should be prepared for forking.
> > > >
> > > > The ibv_fork_init() prepares the library to track all the related
> > > > memory and prevent it from forking using madvise() system API.
> > > > This approach allows fork, but not all the memory is forked to the
> > > > child process and, application should care not to touch pages
> > > > where the parent application allocated the rdma-core objects.
> > > >
> > > > The newer kernels propose an option of copy-on-fork for DMA pages
> > > > and tracking all the memory and disabling it for the forking is no
> > > > longer needed. The new API routine ibv_is_fork_initialized()
> > > > should be involved to decide if library initialization for forking is
> required.
> > > >
> > > > Fixes: 0e83b8e536 ("net/mlx5: move rdma-core calls to separate
> > > > file")
> > > > Cc: stable@dpdk.org
> > > > Signed-off-by: Erez Ferber <erezf@nvidia.com>
> > >
> > Hi,
> >
> > > I don't think DPDK applications should fork(), and lots other parts
> > > of the shared huge pages will break if an application does this.
> >
> > I agree - application should not, we have the secondary/primary processes
> approach.
> > Nonetheless, we have the real use case - DPDK application does fork() and
> works well.
> > Without mlx5 PMD 😊. With mlx5 it ran into some troubles. Now we have
> the solution.
> 
> The problem is you are allowing fork(). And many other libraries may break.
> Imagine a DPDK library which has some local mutex and hugepages.
> If two forked processes use it then the locks won't work and hugepage data
> will be corrupted.

IMO, this problem is not fork() specific - applications/libraries supporting 
secondary/primary process approach should be developed with the similar
precautions.

And, IIRC, fork() is neither disallowed nor discouraged in DPDK documentation.
Moreover, some library unit tests ensure fork() works. No wonder users
may develop applications using fork(). They do, and app works well, beside mlx5 PMD.
Also, the recommendations to re-design with pri/sec approach was given.

With best regards,
Slava
 



  reply	other threads:[~2023-05-26  8:05 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-05-24 12:01 erezf
2023-05-24 14:50 ` Stephen Hemminger
2023-05-25  8:10   ` Slava Ovsiienko
2023-05-25 15:27     ` Stephen Hemminger
2023-05-26  8:05       ` Slava Ovsiienko [this message]
2023-05-24 16:05 ` Slava Ovsiienko
2023-06-22 12:19 ` Raslan Darawsheh

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DM6PR12MB3753378500E5DFCBCD6657CBDF479@DM6PR12MB3753.namprd12.prod.outlook.com \
    --to=viacheslavo@nvidia.com \
    --cc=dev@dpdk.org \
    --cc=erezf@nvidia.com \
    --cc=matan@nvidia.com \
    --cc=rasland@nvidia.com \
    --cc=stable@dpdk.org \
    --cc=stephen@networkplumber.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).