Re: [dpdk-dev] [PATCH v4 06/24] ethdev: enable hotplug on multi-process

["Zhang, Qi Z" <qi.z.zhang@intel.com>]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Tuesday, June 26, 2018 11:12 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v4 06/24] ethdev: enable hotplug on multi-process
> 
> On 26-Jun-18 3:24 PM, Zhang, Qi Z wrote:
> >
> >
> >> -----Original Message-----
> >> From: Burakov, Anatoly
> >> Sent: Tuesday, June 26, 2018 9:46 PM
> >> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> >> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> >> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> >> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> >> <benjamin.h.shelton@intel.com>; Vangati, Narender
> >> <narender.vangati@intel.com>
> >> Subject: Re: [PATCH v4 06/24] ethdev: enable hotplug on multi-process
> >>
> >> On 26-Jun-18 2:25 PM, Zhang, Qi Z wrote:
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Burakov, Anatoly
> >>>> Sent: Tuesday, June 26, 2018 9:21 PM
> >>>> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> >>>> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> >>>> dev@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>;
> >>>> Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton, Benjamin H
> >>>> <benjamin.h.shelton@intel.com>; Vangati, Narender
> >>>> <narender.vangati@intel.com>
> >>>> Subject: Re: [PATCH v4 06/24] ethdev: enable hotplug on
> >>>> multi-process
> >>>>
> >>>> On 26-Jun-18 1:58 PM, Zhang, Qi Z wrote:
> >>>>>
> >>>>> my understand is peer is identified by a string (or filename) what
> >>>>> I mean is clone the content of the buffer that peer point to , So
> >>>>> I don't need to worry if the original peer be used to point to
> >>>>> some other data
> >>>>>
> >>>>
> >>>> As far as the application is concerned, peer is an opaque pointer,
> >>>> and should be treated as such. Peeking behind a void pointer that
> >>>> is not designed for this purpose is not a good idea, even if
> >>>> technically you
> >> know what's in there.
> >>>
> >>> We can expose a clone interface, like MP_PEER_CLONE, so we don't
> >>> need to
> >> know what's inside, just need to know that it can be used on another
> thread?
> >>>
> >>
> >> Well, that can probably work. Feels like a hacky workaround though.
> >>
> >> Another way to do the same thing would be to store peer information
> >> right in the message, as opposed to providing it separately. Still a
> >> hack though, and will require far more changes, but it could be a
> >> better solution as (if done right) it would allow identifying which reply came
> from which peer.
> >>
> >> Of course, an even better approach would be to devise some kind of
> >> addressing scheme (uuid?), so that peer addresses are no longer
> >> opaque pointers but rather are valid data types.
> >>
> >> Thoughts?
> >
> > I may not give insight comment from the IPC implementation, but from
> user's view, what required is a unique token,
> >   it can be used for reply at anywhere and at any time, to me, currently
> implementation looks like missing some mapping management between an
> abstract token to its real data.
> >
> 
> Well, the idea was to not provide that :) We wanted to keep it simple and
> actively discourage any attempts to know which peer you're communicating
> with, as allowing that would imply some kind of addressing scheme, peer
> discovery and things like that, which we don't want to bother with. If you want
> replies, you use callbacks, which provide you with a peer address.
> 
> On the one hand, i understand the frustration of being forced to deal with
> deliberate simplicity of IPC API and to create workarounds like doing sendmsg()
> and storing requests that you're waiting on inside the application and
> launching interrupt threads, all for things that should Just Work in an IPC API.
> Believe me, i know about its deficiencies more than most people :)
> 
> On the other hand, we don't want to introduce hacks to solve a specific
> problem. If we are to solve this in IPC, we should do it properly. I don't think
> adding a workaround with "cloning" peer address is the solution. A proper
> solution would've been to implement addressing scheme, so that any
> response could be submitted at any time.
> 
> The hard route would be to add peer discovery, etc. The easy route would be
> to just change the peer address to something typed, something that we can
> use to reconstruct the original peer id. A great candidate for this is the uuid,
> but it's an external dependency. A workaround for this would be to just use a
> uint64 value obtained through some magic that is part of socket path. For
> example, replace "/var/run/dpdk/rte/mp_socket_pid_rdtsc" with something
> just as (likely) unique, but something that can be put into a uint64.
> 
> Without significantly changing the API and the internals, i think this is the best
> we will be able to do.

Ok, let me summarize this.

Current sync IPC does not support reply to a sync request outside the callback function, 
Of cause it will not support reply on separate thread which required by mp hotplug implementation.

So, the question is do we agree that IPC should support a more relax sync reply? 
For my option, I think the requirement is reasonable, because it greatly simplified the implementation of the case when the receiver need to do some task on a separate thread and reply on that thread.(note, this is typical case since the IPC deadlock limitation)

If the answer is yes:

	The question is will we able to enable this in this release?
	For me I think I may not able to solve this quickly, I need more time to review the implementation detail of IPC , not sure if you can cover this on your new patchset for IPC. 
	if yes, I assume I don't need to change any code (or a little bit) of current mp implementation, 	
	because "peer"(or whatever) as an identity of reply target can be parsed to other thread

	If not, we can take workaround for mp hotplug, 
		since it is workaround I prefer the simplest way with less impact and enough comments, since we will fix this anyway.
		Yes, clone is the simplest way :), I can just implement it inside ethdev_mp.c and add comment say "Fix me, this is hack"., so it will not impact other IPC users.

If the answer is NO
	I have to give up all sync IPC, and use sendmsg to replace sync request and to match request / reply in ethdev_mp.c , though it looks like should be a part of IPC and this looks like we are going to enable another set of sync IPC which support reply on another thread.

Thanks
Qi
> 
> > Thanks
> > Qi.
> >
> >>
> >> --
> >> Thanks,
> >> Anatoly
> 
> 
> --
> Thanks,
> Anatoly