From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
To: David Marchand <david.marchand@redhat.com>
Cc: Ray Kinsella <mdr@ashroe.eu>,
Thomas Monjalon <thomas@monjalon.net>,
techboard@dpdk.org, Bruce Richardson <bruce.richardson@intel.com>,
dev <dev@dpdk.org>, Kevin Traynor <ktraynor@redhat.com>
Subject: Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
Date: Mon, 8 Apr 2019 16:50:46 +0100 [thread overview]
Message-ID: <ba964efb-c995-cd7f-c77a-794ba5934662@intel.com> (raw)
Message-ID: <20190408155046.giculyP9aiX-tNZaQuTTNLD_aUhjDcqOwtKIG3Hyjhg@z> (raw)
In-Reply-To: <CAJFAV8z1HGjCbsQJsZQfVTufy3fd4Mqg6W4VfoZe1xaPwUOB1Q@mail.gmail.com>
On 08-Apr-19 3:38 PM, David Marchand wrote:
>
>
> On Mon, Apr 8, 2019 at 4:03 PM Burakov, Anatoly
> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>
> On 08-Apr-19 2:58 PM, David Marchand wrote:
> > On Mon, Apr 8, 2019 at 3:39 PM Burakov, Anatoly
> > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>
> <mailto:anatoly.burakov@intel.com
> <mailto:anatoly.burakov@intel.com>>> wrote:
> >
> > As a concrete proposal, my number one dream would be to see
> > multiprocess
> > gone. I also recall desire for "DPDK to be more lightweight",
> and i
> > maintain that DPDK *cannot* be lightweight if we are to support
> > multiprocess - we can have one or the other, but not both.
> However,
> > realistically, i don't think dropping multiprocess is ever
> going to
> > happen - not only it is too entrenched in DPDK use cases, it is
> > actually
> > quite useful despite its flaws.
> >
> >
> > Well, honestly, I'd like to hear about this.
> > What are the real usecases for multi process support?
> > Do we have even a single opensource project that uses it?
> >
>
> I'm aware of a few closed source usages of multiprocess. I also think
> current versions of collectd rely on secondary process (there's been a
> Telemetry API added to avoid that, but AFAIK the support for Telemetry
> is not upstream in collectd yet), and so do/would any dump-style
> applications - in fact, we ourselves include one such application in
> our
> codebase (pdump, proc-info, etc.).
>
>
> Sorry, I don't want to highjack this thread, I can start a separate
> thread if people feel like it.
> If we go with stabilisation, we must be careful that we want to support
> the features.
>
> So about multiprocess, again, in those closed source projects you know
> of, what are the usecases?
>
> For what we provide in dpdk pdump, proc-info, referring to oneself is
> not that convincing to me as I don't use those tools.
>
> I don't see what we could not achieve the same with a control thread
> running in the dpdk process and handling commands.
> It would be open to the outside via a more standard channel, like a UNIX
> socket or something like this.
> If we need to declare a dynamic channel, it can be constructed as an
> extension of the existing standard channel: we can open something like a
> POSIX shm and push things in it.
> Was this explored ?
There are certainly things that we can do that can make some aspects of
multiprocess redundant. For example, for any kind of collectd-like
scenario, the Telemetry API (or Keith's DFS, or...) could conceivably
provide a better and more maintainable way of doing things.
Our multiprocess also makes it easier to write pipeline/load-balancing
type applications. To see an example, look at our
multiprocess/client-server example. This is demonstrating how, instead
of writing one big monolithic application, one could instead write a
number of smaller applications each doing their thing. It is of course
possible to do the same without multiprocess, as evidenced by our sample
applications such as load-balancer, distributor, ip-pipeline etc., but
it is arguably easier to implement *real* applications that way due to
separation of concerns and more focused codebase.
However, there are two use cases that i can think of that are either
hard or outright not possible without our multiprocess API's. The first
one is dumping functionality. For example, dpdk_proc_info can display
info from a currently-running or defunct process - list its
memzones/mempools/etc. - basically, everything there is to know about
the shared memory can be known that way. While this isn't a "real" use
case, it is useful for debugging.
More importantly, our multiprocess model provides resilience. In an
event of a crash, the entire application is not brought down - instead,
only the crashed process goes down. It's not /perfect/ resilience, of
course, and there are caveats (memory leaking, locks, etc.), but you do
get /some/ resilience that way - your process went down, you spin
another secondary and you're back up and running again.
The above described scenario is how most people (that i know of) appear
to be using multiprocess - some kind of "crash-resilient"
load-balancing/pipelining app.
>
>
> --
> David Marchand
--
Thanks,
Anatoly
next prev parent reply other threads:[~2019-04-08 15:50 UTC|newest]
Thread overview: 78+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-04-03 15:42 [dpdk-dev] " Ray Kinsella
2019-04-03 15:42 ` Ray Kinsella
2019-04-03 19:53 ` Luca Boccassi
2019-04-03 19:53 ` Luca Boccassi
2019-04-04 9:29 ` Burakov, Anatoly
2019-04-04 9:29 ` Burakov, Anatoly
2019-04-04 10:54 ` [dpdk-dev] [dpdk-techboard] " Bruce Richardson
2019-04-04 10:54 ` Bruce Richardson
2019-04-04 12:02 ` Luca Boccassi
2019-04-04 12:02 ` Luca Boccassi
2019-04-04 13:05 ` Ray Kinsella
2019-04-04 13:05 ` Ray Kinsella
2019-04-04 13:10 ` Bruce Richardson
2019-04-04 13:10 ` Bruce Richardson
2019-04-05 13:25 ` Ray Kinsella
2019-04-05 13:25 ` Ray Kinsella
2019-04-07 9:37 ` Thomas Monjalon
2019-04-07 9:37 ` Thomas Monjalon
2019-04-04 13:21 ` Luca Boccassi
2019-04-04 13:21 ` Luca Boccassi
2019-04-04 12:52 ` Ray Kinsella
2019-04-04 12:52 ` Ray Kinsella
2019-04-04 14:07 ` Burakov, Anatoly
2019-04-04 14:07 ` Burakov, Anatoly
2019-04-07 9:48 ` Thomas Monjalon
2019-04-07 9:48 ` Thomas Monjalon
2019-04-08 9:04 ` Ray Kinsella
2019-04-08 9:04 ` Ray Kinsella
2019-04-08 10:15 ` Burakov, Anatoly
2019-04-08 10:15 ` Burakov, Anatoly
2019-04-08 13:00 ` Ray Kinsella
2019-04-08 13:00 ` Ray Kinsella
2019-04-08 13:38 ` Burakov, Anatoly
2019-04-08 13:38 ` Burakov, Anatoly
2019-04-08 13:58 ` David Marchand
2019-04-08 13:58 ` David Marchand
2019-04-08 14:02 ` Burakov, Anatoly
2019-04-08 14:02 ` Burakov, Anatoly
2019-04-08 14:38 ` David Marchand
2019-04-08 14:38 ` David Marchand
2019-04-08 15:13 ` Stephen Hemminger
2019-04-08 15:13 ` Stephen Hemminger
2019-04-08 15:49 ` Burakov, Anatoly
2019-04-08 15:49 ` Burakov, Anatoly
2019-04-10 8:35 ` David Marchand
2019-04-10 8:35 ` David Marchand
2019-04-08 15:50 ` Burakov, Anatoly [this message]
2019-04-08 15:50 ` Burakov, Anatoly
2019-04-09 9:42 ` Ray Kinsella
2019-04-09 9:42 ` Ray Kinsella
2019-04-14 0:42 ` Neil Horman
2019-04-14 0:42 ` Neil Horman
2019-04-15 9:10 ` Bruce Richardson
2019-04-15 9:10 ` Bruce Richardson
2019-04-04 15:51 ` Stephen Hemminger
2019-04-04 15:51 ` Stephen Hemminger
2019-04-04 16:37 ` Burakov, Anatoly
2019-04-04 16:37 ` Burakov, Anatoly
2019-04-04 16:56 ` Kevin Traynor
2019-04-04 16:56 ` Kevin Traynor
2019-04-04 19:08 ` Wiles, Keith
2019-04-04 19:08 ` Wiles, Keith
2019-04-04 20:13 ` Kevin Traynor
2019-04-04 20:13 ` Kevin Traynor
2019-04-05 13:30 ` Ray Kinsella
2019-04-05 13:30 ` Ray Kinsella
2019-04-05 13:29 ` Ray Kinsella
2019-04-05 13:29 ` Ray Kinsella
2019-04-04 9:47 ` [dpdk-dev] " Kevin Traynor
2019-04-04 9:47 ` Kevin Traynor
2019-04-04 13:16 ` Ray Kinsella
2019-04-04 13:16 ` Ray Kinsella
2019-04-10 5:14 ` Honnappa Nagarahalli
2019-04-10 5:14 ` Honnappa Nagarahalli
2019-04-10 9:03 ` [dpdk-dev] [dpdk-techboard] " Bruce Richardson
2019-04-10 9:03 ` Bruce Richardson
2019-04-10 9:43 ` [dpdk-dev] " Luca Boccassi
2019-04-10 9:43 ` Luca Boccassi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ba964efb-c995-cd7f-c77a-794ba5934662@intel.com \
--to=anatoly.burakov@intel.com \
--cc=bruce.richardson@intel.com \
--cc=david.marchand@redhat.com \
--cc=dev@dpdk.org \
--cc=ktraynor@redhat.com \
--cc=mdr@ashroe.eu \
--cc=techboard@dpdk.org \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).