DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
To: David Marchand <david.marchand@redhat.com>
Cc: Ray Kinsella <mdr@ashroe.eu>,
	Thomas Monjalon <thomas@monjalon.net>,
	techboard@dpdk.org, Bruce Richardson <bruce.richardson@intel.com>,
	dev <dev@dpdk.org>, Kevin Traynor <ktraynor@redhat.com>
Subject: Re: [dpdk-dev] [dpdk-techboard] DPDK ABI/API Stability
Date: Mon, 8 Apr 2019 16:49:43 +0100	[thread overview]
Message-ID: <c19663cb-3c33-0934-4032-1ecc2114e497@intel.com> (raw)
Message-ID: <20190408154943.GRlN0XaS7nc5GO0qIbqup7WtYL8c4br317AXKfmzR6g@z> (raw)
In-Reply-To: <CAJFAV8z1HGjCbsQJsZQfVTufy3fd4Mqg6W4VfoZe1xaPwUOB1Q@mail.gmail.com>

On 08-Apr-19 3:38 PM, David Marchand wrote:
> 
> 
> On Mon, Apr 8, 2019 at 4:03 PM Burakov, Anatoly 
> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> 
>     On 08-Apr-19 2:58 PM, David Marchand wrote:
>      > On Mon, Apr 8, 2019 at 3:39 PM Burakov, Anatoly
>      > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>
>     <mailto:anatoly.burakov@intel.com
>     <mailto:anatoly.burakov@intel.com>>> wrote:
>      >
>      >     As a concrete proposal, my number one dream would be to see
>      >     multiprocess
>      >     gone. I also recall desire for "DPDK to be more lightweight",
>     and i
>      >     maintain that DPDK *cannot* be lightweight if we are to support
>      >     multiprocess - we can have one or the other, but not both.
>     However,
>      >     realistically, i don't think dropping multiprocess is ever
>     going to
>      >     happen - not only it is too entrenched in DPDK use cases, it is
>      >     actually
>      >     quite useful despite its flaws.
>      >
>      >
>      > Well, honestly, I'd like to hear about this.
>      > What are the real usecases for multi process support?
>      > Do we have even a single opensource project that uses it?
>      >
> 
>     I'm aware of a few closed source usages of multiprocess. I also think
>     current versions of collectd rely on secondary process (there's been a
>     Telemetry API added to avoid that, but AFAIK the support for Telemetry
>     is not upstream in collectd yet), and so do/would any dump-style
>     applications - in fact, we ourselves include one such application in
>     our
>     codebase (pdump, proc-info, etc.).
> 
> 
> Sorry, I don't want to highjack this thread, I can start a separate 
> thread if people feel like it.
> If we go with stabilisation, we must be careful that we want to support 
> the features.
> 
> So about multiprocess, again, in those closed source projects you know 
> of, what are the usecases?
> 
> For what we provide in dpdk pdump, proc-info, referring to oneself is 
> not that convincing to me as I don't use those tools.
> 
> I don't see what we could not achieve the same with a control thread 
> running in the dpdk process and handling commands.
> It would be open to the outside via a more standard channel, like a UNIX 
> socket or something like this.
> If we need to declare a dynamic channel, it can be constructed as an 
> extension of the existing standard channel: we can open something like a 
> POSIX shm and push things in it.
> Was this explored ?

There are certainly things that we can do that can make some aspects of 
multiprocess redundant. For example, for any kind of collectd-like 
scenario, the Telemetry API (or Keith's DFS, or...) could conceivably 
provide a better and more maintainable way of doing things.

Our multiprocess also makes it easier to write pipeline/load-balancing 
type applications. To see an example, look at our 
multiprocess/client-server example. This is demonstrating how, instead 
of writing one big monolithic application, one could instead write a 
number of smaller applications each doing their thing. It is of course 
possible to do the same without multiprocess, as evidenced by our sample 
applications such as load-balancer, distributor, ip-pipeline etc., but 
it is arguably easier to implement *real* applications that way due to 
separation of concerns and more focused codebase.

However, there are two use cases that i can think of that are either 
hard or outright not possible without our multiprocess API's. The first 
one is dumping functionality. For example, dpdk_proc_info can display 
info from a currently-running or defunct process - list its 
memzones/mempools/etc. - basically, everything there is to know about 
the shared memory can be known that way. While this isn't a "real" use 
case, it is useful for debugging.

More importantly, our multiprocess model provides resilience. In an 
event of a crash, the entire application is not brought down - instead, 
only the crashed process goes down. It's not /perfect/ resilience, of 
course, and there are caveats (memory leaking, locks, etc.), but you do 
get /some/ resilience that way - your process went down, you spin 
another secondary and you're back up and running again.

The above described scenario is how most people (that i know of) appear 
to be using multiprocess - some kind of "crash-resilient" 
load-balancing/pipelining app.

> 
> 
> -- 
> David Marchand


-- 
Thanks,
Anatoly

  parent reply	other threads:[~2019-04-08 15:51 UTC|newest]

Thread overview: 78+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-04-03 15:42 [dpdk-dev] " Ray Kinsella
2019-04-03 15:42 ` Ray Kinsella
2019-04-03 19:53 ` Luca Boccassi
2019-04-03 19:53   ` Luca Boccassi
2019-04-04  9:29 ` Burakov, Anatoly
2019-04-04  9:29   ` Burakov, Anatoly
2019-04-04 10:54   ` [dpdk-dev] [dpdk-techboard] " Bruce Richardson
2019-04-04 10:54     ` Bruce Richardson
2019-04-04 12:02     ` Luca Boccassi
2019-04-04 12:02       ` Luca Boccassi
2019-04-04 13:05       ` Ray Kinsella
2019-04-04 13:05         ` Ray Kinsella
2019-04-04 13:10         ` Bruce Richardson
2019-04-04 13:10           ` Bruce Richardson
2019-04-05 13:25           ` Ray Kinsella
2019-04-05 13:25             ` Ray Kinsella
2019-04-07  9:37             ` Thomas Monjalon
2019-04-07  9:37               ` Thomas Monjalon
2019-04-04 13:21         ` Luca Boccassi
2019-04-04 13:21           ` Luca Boccassi
2019-04-04 12:52     ` Ray Kinsella
2019-04-04 12:52       ` Ray Kinsella
2019-04-04 14:07       ` Burakov, Anatoly
2019-04-04 14:07         ` Burakov, Anatoly
2019-04-07  9:48         ` Thomas Monjalon
2019-04-07  9:48           ` Thomas Monjalon
2019-04-08  9:04           ` Ray Kinsella
2019-04-08  9:04             ` Ray Kinsella
2019-04-08 10:15             ` Burakov, Anatoly
2019-04-08 10:15               ` Burakov, Anatoly
2019-04-08 13:00               ` Ray Kinsella
2019-04-08 13:00                 ` Ray Kinsella
2019-04-08 13:38                 ` Burakov, Anatoly
2019-04-08 13:38                   ` Burakov, Anatoly
2019-04-08 13:58                   ` David Marchand
2019-04-08 13:58                     ` David Marchand
2019-04-08 14:02                     ` Burakov, Anatoly
2019-04-08 14:02                       ` Burakov, Anatoly
2019-04-08 14:38                       ` David Marchand
2019-04-08 14:38                         ` David Marchand
2019-04-08 15:13                         ` Stephen Hemminger
2019-04-08 15:13                           ` Stephen Hemminger
2019-04-08 15:49                         ` Burakov, Anatoly [this message]
2019-04-08 15:49                           ` Burakov, Anatoly
2019-04-10  8:35                           ` David Marchand
2019-04-10  8:35                             ` David Marchand
2019-04-08 15:50                         ` Burakov, Anatoly
2019-04-08 15:50                           ` Burakov, Anatoly
2019-04-09  9:42                   ` Ray Kinsella
2019-04-09  9:42                     ` Ray Kinsella
2019-04-14  0:42             ` Neil Horman
2019-04-14  0:42               ` Neil Horman
2019-04-15  9:10               ` Bruce Richardson
2019-04-15  9:10                 ` Bruce Richardson
2019-04-04 15:51     ` Stephen Hemminger
2019-04-04 15:51       ` Stephen Hemminger
2019-04-04 16:37       ` Burakov, Anatoly
2019-04-04 16:37         ` Burakov, Anatoly
2019-04-04 16:56     ` Kevin Traynor
2019-04-04 16:56       ` Kevin Traynor
2019-04-04 19:08       ` Wiles, Keith
2019-04-04 19:08         ` Wiles, Keith
2019-04-04 20:13         ` Kevin Traynor
2019-04-04 20:13           ` Kevin Traynor
2019-04-05 13:30           ` Ray Kinsella
2019-04-05 13:30             ` Ray Kinsella
2019-04-05 13:29         ` Ray Kinsella
2019-04-05 13:29           ` Ray Kinsella
2019-04-04  9:47 ` [dpdk-dev] " Kevin Traynor
2019-04-04  9:47   ` Kevin Traynor
2019-04-04 13:16   ` Ray Kinsella
2019-04-04 13:16     ` Ray Kinsella
2019-04-10  5:14 ` Honnappa Nagarahalli
2019-04-10  5:14   ` Honnappa Nagarahalli
2019-04-10  9:03   ` [dpdk-dev] [dpdk-techboard] " Bruce Richardson
2019-04-10  9:03     ` Bruce Richardson
2019-04-10  9:43   ` [dpdk-dev] " Luca Boccassi
2019-04-10  9:43     ` Luca Boccassi

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=c19663cb-3c33-0934-4032-1ecc2114e497@intel.com \
    --to=anatoly.burakov@intel.com \
    --cc=bruce.richardson@intel.com \
    --cc=david.marchand@redhat.com \
    --cc=dev@dpdk.org \
    --cc=ktraynor@redhat.com \
    --cc=mdr@ashroe.eu \
    --cc=techboard@dpdk.org \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).