[dpdk-dev] DPDK and forked processes

[dpdk-dev] DPDK and forked processes

[Eads, Gage]

Hi all,

Does DPDK support forking secondary processes after executing rte_eal_init()? The l2fwd_fork example and at least one application (OpenEM: https://sourceforge.net/projects/eventmachine/) use this model, and they do so by fixing up the EAL internals (e.g. manually changing process_type from primary to secondary) at the start of the child process. This feels like a hack, and I can't find any documentation describing this model.

Moreover, this approach doesn't appear to be compatible with recent EAL changes. For instance, the multi-process communication creates a couple handler threads ("rte_mp_handle" and "rte_mp_async") during EAL initialization. The child processes won't inherit these threads, and so won't be able to participate in multi-process comms. This means the reworked memory subsystem and upcoming device hotplug support (http://mails.dpdk.org/archives/dev/2018-July/107704.html) won't work with this fork-after-init model.

This is just one example - there may be other features/subsystems that won't work. As far as I can tell there is no official stance (though the l2fwd_fork example implies it's supported, IMO); I think either DPDK should either drop the example and not support this model, or support it and either document its limitations or resolve them. This model could be an interesting way to run multi-process DPDK on an ASLR-enabled system, but supporting this wouldn't be trivial.

Thanks,
Gage

Re: [dpdk-dev] DPDK and forked processes

[Burakov, Anatoly]

On 16-Jul-18 4:00 PM, Eads, Gage wrote:
> Hi all,
> 
> Does DPDK support forking secondary processes after executing 
> rte_eal_init()? The l2fwd_fork example and at least one application 
> (OpenEM: https://sourceforge.net/projects/eventmachine/) use this model, 
> and they do so by fixing up the EAL internals (e.g. manually changing 
> process_type from primary to secondary) at the start of the child 
> process. This feels like a hack, and I can’t find any documentation 
> describing this model.
> 
> Moreover, this approach doesn’t appear to be compatible with recent EAL 
> changes. For instance, the multi-process communication creates a couple 
> handler threads (“rte_mp_handle” and “rte_mp_async”) during EAL 
> initialization. The child processes won’t inherit these threads, and so 
> won’t be able to participate in multi-process comms. This means the 
> reworked memory subsystem and upcoming device hotplug support 
> (http://mails.dpdk.org/archives/dev/2018-July/107704.html) won’t work 
> with this fork-after-init model.
> 
> This is just one example – there may be other features/subsystems that 
> won’t work. As far as I can tell there is no official stance (though the 
> l2fwd_fork example implies it’s supported, IMO); I think either DPDK 
> should either drop the example and not support this model, or support it 
> and either document its limitations or resolve them. This model could be 
> an interesting way to run multi-process DPDK on an ASLR-enabled system, 
> but supporting this wouldn’t be trivial.
> 
> Thanks,
> 
> Gage
> 

I think it's a very bad idea to use such a model in recent versions of 
DPDK. As you have correctly pointed out, IPC will not work in such a 
scenario, and given how our memory subsystem relies on IPC, this is a 
recipe for memory corruption and divergent memory maps (since 
technically both initial and forked processes believe they are primary).

Even hacking rte_config to make DPDK think it's a secondary process will 
not work, because the initialization has already completed, but all of 
the threads (IPC, interrupt, etc.) are gone and correct IPC socket was 
not created, which means the process becomes invisible to the primary 
for all intents and purposes.

We _could_ introduce some kind of "official DPDK fork" function that 
would fork the process and then restart interrupt, IPC etc. stuff on an 
already running instance of DPDK, but that seems like a workaround for a 
problem that shouldn't exist in the first place, because such usage is 
fundamentally incompatible with DPDK as it stands now.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] DPDK and forked processes

[Eads, Gage]

As this discussion has broad implications for DPDK, is it a good candidate for a techboard meeting topic? 

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Monday, July 16, 2018 10:09 AM
> To: Eads, Gage <gage.eads@intel.com>; dev@dpdk.org
> Subject: Re: DPDK and forked processes
> 
> On 16-Jul-18 4:00 PM, Eads, Gage wrote:
> > Hi all,
> >
> > Does DPDK support forking secondary processes after executing
> > rte_eal_init()? The l2fwd_fork example and at least one application
> > (OpenEM: https://sourceforge.net/projects/eventmachine/) use this
> > model, and they do so by fixing up the EAL internals (e.g. manually
> > changing process_type from primary to secondary) at the start of the
> > child process. This feels like a hack, and I can’t find any
> > documentation describing this model.
> >
> > Moreover, this approach doesn’t appear to be compatible with recent
> > EAL changes. For instance, the multi-process communication creates a
> > couple handler threads (“rte_mp_handle” and “rte_mp_async”) during EAL
> > initialization. The child processes won’t inherit these threads, and
> > so won’t be able to participate in multi-process comms. This means the
> > reworked memory subsystem and upcoming device hotplug support
> > (http://mails.dpdk.org/archives/dev/2018-July/107704.html) won’t work
> > with this fork-after-init model.
> >
> > This is just one example – there may be other features/subsystems that
> > won’t work. As far as I can tell there is no official stance (though
> > the l2fwd_fork example implies it’s supported, IMO); I think either
> > DPDK should either drop the example and not support this model, or
> > support it and either document its limitations or resolve them. This
> > model could be an interesting way to run multi-process DPDK on an
> > ASLR-enabled system, but supporting this wouldn’t be trivial.
> >
> > Thanks,
> >
> > Gage
> >
> 
> I think it's a very bad idea to use such a model in recent versions of DPDK. As you
> have correctly pointed out, IPC will not work in such a scenario, and given how
> our memory subsystem relies on IPC, this is a recipe for memory corruption and
> divergent memory maps (since technically both initial and forked processes
> believe they are primary).
> 
> Even hacking rte_config to make DPDK think it's a secondary process will not
> work, because the initialization has already completed, but all of the threads
> (IPC, interrupt, etc.) are gone and correct IPC socket was not created, which
> means the process becomes invisible to the primary for all intents and purposes.
> 
> We _could_ introduce some kind of "official DPDK fork" function that would fork
> the process and then restart interrupt, IPC etc. stuff on an already running
> instance of DPDK, but that seems like a workaround for a problem that shouldn't
> exist in the first place, because such usage is fundamentally incompatible with
> DPDK as it stands now.
> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] DPDK and forked processes

[Thomas Monjalon]

27/07/2018 15:46, Eads, Gage:
> As this discussion has broad implications for DPDK, is it a good candidate for a techboard meeting topic? 

We can discuss it in techboard, but usually we prefer discussing topics
whose resolution is not clear.
In this case, I think everybody agree with Anatoly, isn't it?


> > -----Original Message-----
> > From: Burakov, Anatoly
> > Sent: Monday, July 16, 2018 10:09 AM
> > To: Eads, Gage <gage.eads@intel.com>; dev@dpdk.org
> > Subject: Re: DPDK and forked processes
> > 
> > On 16-Jul-18 4:00 PM, Eads, Gage wrote:
> > > Hi all,
> > >
> > > Does DPDK support forking secondary processes after executing
> > > rte_eal_init()? The l2fwd_fork example and at least one application
> > > (OpenEM: https://sourceforge.net/projects/eventmachine/) use this
> > > model, and they do so by fixing up the EAL internals (e.g. manually
> > > changing process_type from primary to secondary) at the start of the
> > > child process. This feels like a hack, and I can’t find any
> > > documentation describing this model.
> > >
> > > Moreover, this approach doesn’t appear to be compatible with recent
> > > EAL changes. For instance, the multi-process communication creates a
> > > couple handler threads (“rte_mp_handle” and “rte_mp_async”) during EAL
> > > initialization. The child processes won’t inherit these threads, and
> > > so won’t be able to participate in multi-process comms. This means the
> > > reworked memory subsystem and upcoming device hotplug support
> > > (http://mails.dpdk.org/archives/dev/2018-July/107704.html) won’t work
> > > with this fork-after-init model.
> > >
> > > This is just one example – there may be other features/subsystems that
> > > won’t work. As far as I can tell there is no official stance (though
> > > the l2fwd_fork example implies it’s supported, IMO); I think either
> > > DPDK should either drop the example and not support this model, or
> > > support it and either document its limitations or resolve them. This
> > > model could be an interesting way to run multi-process DPDK on an
> > > ASLR-enabled system, but supporting this wouldn’t be trivial.
> > >
> > > Thanks,
> > >
> > > Gage
> > >
> > 
> > I think it's a very bad idea to use such a model in recent versions of DPDK. As you
> > have correctly pointed out, IPC will not work in such a scenario, and given how
> > our memory subsystem relies on IPC, this is a recipe for memory corruption and
> > divergent memory maps (since technically both initial and forked processes
> > believe they are primary).
> > 
> > Even hacking rte_config to make DPDK think it's a secondary process will not
> > work, because the initialization has already completed, but all of the threads
> > (IPC, interrupt, etc.) are gone and correct IPC socket was not created, which
> > means the process becomes invisible to the primary for all intents and purposes.
> > 
> > We _could_ introduce some kind of "official DPDK fork" function that would fork
> > the process and then restart interrupt, IPC etc. stuff on an already running
> > instance of DPDK, but that seems like a workaround for a problem that shouldn't
> > exist in the first place, because such usage is fundamentally incompatible with
> > DPDK as it stands now.
> > 
> > --
> > Thanks,
> > Anatoly
> 

Re: [dpdk-dev] DPDK and forked processes

[Stephen Hemminger]

On Fri, 27 Jul 2018 17:03:48 +0200
Thomas Monjalon <thomas@monjalon.net> wrote:

> 27/07/2018 15:46, Eads, Gage:
> > As this discussion has broad implications for DPDK, is it a good candidate for a techboard meeting topic?   
> 
> We can discuss it in techboard, but usually we prefer discussing topics
> whose resolution is not clear.
> In this case, I think everybody agree with Anatoly, isn't it?

I would prefer that decisions like this be done by rough consensus on the mailing list.

As far as applications messing with internals, in reality any application can change
anything. Just don't come crying to DPDK community for help.
 

Re: [dpdk-dev] DPDK and forked processes

[Eads, Gage]

Agreed on both points. I'll submit a patchset to remove the l2fwd_fork example and its user-guide, so it doesn't appear that DPDK supports this model. If anyone on the ML disagrees, they can respond here or on the patch thread.

> -----Original Message-----
> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> Sent: Friday, July 27, 2018 11:00 AM
> To: Thomas Monjalon <thomas@monjalon.net>
> Cc: Eads, Gage <gage.eads@intel.com>; Burakov, Anatoly
> <anatoly.burakov@intel.com>; dev@dpdk.org; Richardson, Bruce
> <bruce.richardson@intel.com>; jerin.jacob@caviumnetworks.com; Yigit, Ferruh
> <ferruh.yigit@intel.com>; hemant.agrawal@nxp.com; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; Olivier Matz <olivier.matz@6wind.com>
> Subject: Re: DPDK and forked processes
> 
> On Fri, 27 Jul 2018 17:03:48 +0200
> Thomas Monjalon <thomas@monjalon.net> wrote:
> 
> > 27/07/2018 15:46, Eads, Gage:
> > > As this discussion has broad implications for DPDK, is it a good candidate for
> a techboard meeting topic?
> >
> > We can discuss it in techboard, but usually we prefer discussing
> > topics whose resolution is not clear.
> > In this case, I think everybody agree with Anatoly, isn't it?
> 
> I would prefer that decisions like this be done by rough consensus on the mailing
> list.
> 
> As far as applications messing with internals, in reality any application can
> change anything. Just don't come crying to DPDK community for help.
>