From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <thomas@monjalon.net>
Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com
[66.111.4.29]) by dpdk.org (Postfix) with ESMTP id 5C777FFA
for <dev@dpdk.org>; Thu, 25 Jan 2018 12:28:05 +0100 (CET)
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
by mailout.nyi.internal (Postfix) with ESMTP id 9544720CA7;
Thu, 25 Jan 2018 06:28:03 -0500 (EST)
Received: from frontend2 ([10.202.2.161])
by compute1.internal (MEProxy); Thu, 25 Jan 2018 06:28:03 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h=
cc:content-transfer-encoding:content-type:date:from:in-reply-to
:message-id:mime-version:references:subject:to:x-me-sender
:x-me-sender:x-sasl-enc; s=mesmtp; bh=ov1bNEXYJCDW70wOaZ3u70dQ8b
ouVxYS6ai2EVMnVUg=; b=lVV4P6q7Ydx3GqYNLOe8ALOJeF9lPa1+xJs/XoOLaZ
g1BnG6TFrQS/3TA3DM25gsUOUkSPr/kTSNNdQWOWjGKqD4FUYptDz4foeAfbZHim
7Ank85ng7OM+I6xObHT+2GQvGsrF9PQpzgEfIyw4rdm+E2VbWpYy5YX/Jp77jONf
g=
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
messagingengine.com; h=cc:content-transfer-encoding:content-type
:date:from:in-reply-to:message-id:mime-version:references
:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=ov1bNE
XYJCDW70wOaZ3u70dQ8bouVxYS6ai2EVMnVUg=; b=l0PEt/07r5T3MuIHmtz8wu
f5dC1ryJHRMXFlrCgyPVcn2hx7RZTAVplKMd1od7rtdcehHBJv9N/x/Mi7ho2gg7
XoyyA4fpFkHsGkl8B0H/0ssptk/faHief0AN/HLxvENky7+m7GtN7BJSY43O0FRD
zLpfj5Q1OFmwwJW41mBcm/TaQPYETYUdUOGr0SbnCqMhv94RLsC/G/YJvjRmHgBh
8ZTg1Mnsctg7uk/ockLVjHpL5SiCXna4AO9UDMrreoN/FtJa/LtgrKqWLlKYvart
gl3J+h05DyVBCxo/bflhdtYgW+ULgssFaSMfEbnNhZFwpJC4b0SoCzQBO7dGpaqQ
==
X-ME-Sender: <xms:Q79pWlQh18jEILgWV6x_G5SJydtJUwwUnFVqMiLoGamht3_rz5wakA>
Received: from xps.localnet (184.203.134.77.rev.sfr.net [77.134.203.184])
by mail.messagingengine.com (Postfix) with ESMTPA id 46AC524636;
Thu, 25 Jan 2018 06:28:03 -0500 (EST)
From: Thomas Monjalon <thomas@monjalon.net>
To: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>
Cc: Matan Azrad <matan@mellanox.com>,
=?ISO-8859-1?Q?Ga=EBtan?= Rivet <gaetan.rivet@6wind.com>, "Wu,
Jingjing" <jingjing.wu@intel.com>, dev@dpdk.org,
Neil Horman <nhorman@tuxdriver.com>, "Richardson,
Bruce" <bruce.richardson@intel.com>
Date: Thu, 25 Jan 2018 12:27:21 +0100
Message-ID: <5896276.RKtlGRRNmM@xps>
In-Reply-To: <2601191342CEEE43887BDE71AB977258862835EB@irsmsx105.ger.corp.intel.com>
References: <1515318351-4756-1-git-send-email-matan@mellanox.com>
<42152390.b8nyHhbJZJ@xps>
<2601191342CEEE43887BDE71AB977258862835EB@irsmsx105.ger.corp.intel.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 7Bit
Content-Type: text/plain; charset="us-ascii"
Subject: Re: [dpdk-dev] [PATCH v3 7/7] app/testpmd: adjust ethdev port
ownership
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://dpdk.org/ml/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://dpdk.org/ml/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Thu, 25 Jan 2018 11:28:05 -0000
25/01/2018 12:09, Ananyev, Konstantin:
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > 24/01/2018 19:30, Ananyev, Konstantin:
> > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > 23/01/2018 22:18, Ananyev, Konstantin:
> > > > > >
> > > > > > 23/01/2018 16:18, Ananyev, Konstantin:
> > > > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ananyev, Konstantin
> > > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > > > 23/01/2018 14:34, Ananyev, Konstantin:
> > > > > > > > > > If that' s the use case, then I think you need to set device ownership at creation time -
> > > > > > > > > > inside dev_allocate().
> > > > > > > > > > Again that would avoid such racing conditions inside testpmd.
> > > > > > > > >
> > > > > > > > > The devices must be allocated at a low level layer.
> > > > > > > >
> > > > > > > > No one arguing about that.
> > > > > > > > But we can provide owner id information to the low level.
> > > > > >
> > > > > > Sorry, you did not get it.
> > > > >
> > > > > Might be.
> > > > >
> > > > > > We cannot provide owner id at the low level
> > > > > > because it is not yet decided who will be the owner
> > > > > > before the port is allocated.
> > > > >
> > > > > Why is that?
> > > > > What prevents us decide who will manage that device *before* allocating port of it?
> > > > > IMO we do have all needed information at that stage.
> > > >
> > > > We don't have the information.
> > >
> > > At that point we do have dev name and all parameters, right?
> >
> > We just have the PCI id.
> >
> > > Plus we do have blacklist/whitelist, etc.
> > > So what else are we missing to make the decision at that point?
> >
> > It depends on the ownership policy.
> > Example: we can decide to take ownership based on a MAC address.
>
> That's sounds a bit articificial (mac address can be changed on the fly), but ok -
> for such devices user can decide to use default id first and change
> it later after port is allocated and dev_init() is passed.
> Though as I understand there situations (like in failsafe PMD) when we do
> know for sure owner_id before calling dev_allocate().
In the general case, when hotplug will be managed by EAL in an
asynchronous way, the port allocation will be done without any knowledge
about the port owner.
> > Another example: it can be decided to take ownership of a given driver.
> > We don't have these informations with the PCI id.
> >
> > > > It is a new device, it is matched by a driver which allocates a port.
> > > > I don't see where the higher level can interact here.
> > > > And even if you manage a trick, the higher level needs to read the port
> > > > informations to decide the ownership.
> > >
> > > Could you specify what particular port information it needs?
> >
> > Replied to the same question above :)
> >
> >
> > > > > > > > > When a new device appears (hotplug), an ethdev port should be allocated
> > > > > > > > > automatically if it passes the whitelist/blacklist policy test.
> > > > > > > > > Then we must decide who will manage this device.
> > > > > > > > > I suggest notifying the DPDK libs first.
> > > > > > > > > So a DPDK lib or PMD like failsafe can have the priority to take the
> > > > > > > > > ownership in its notification callback.
> > > > > > > >
> > > > > > > > Possible, but seems a bit overcomplicated.
> > > > > > > > Why not just:
> > > > > > > >
> > > > > > > > Have a global variable process_default_owner_id, that would be init once at startup.
> > > > > > > > Have an LTS variable default_owner_id.
> > > > > > > > It will be used by rte_eth_dev_allocate() caller can set dev->owner_id at creation time,
> > > > > > > > so port allocation and setting ownership - will be an atomic operation.
> > > > > > > > At the exit rte_eth_dev_allocate() will always reset default_owner_id=0:
> > > > > > > >
> > > > > > > > rte_eth_dev_allocate(...)
> > > > > > > > {
> > > > > > > > lock(owner_lock);
> > > > > > > > <allocate_port>
> > > > > > > > owner = RTE_PER_LCORE(default_owner_id);
> > > > > > > > if (owner == 0)
> > > > > > > > owner = process_default_owner_id;
> > > > > > > > set_owner(port, ..., owner);
> > > > > > > > unlock(owner_lock);
> > > > > > > > RTE_PER_LCORE(default_owner_id) = 0;
> > > > > > >
> > > > > > > Or probably better to leave default_owner_id reset to the caller.
> > > > > > > Another thing - we can use same LTS variable in all control ops to
> > > > > > > allow/disallow changing of port configuration based on ownership.
> > > > > > > Konstantin
> > > > > > >
> > > > > > > > }
> > > > > > > >
> > > > > > > > So callers who don't need any special ownership - don't need to do anything.
> > > > > > > > Special callers (like failsafe) can set default_owenr_id just before calling hotplug
> > > > > > > > handling routine.
> > > > > >
> > > > > > No, hotplug will not be a routine.
> > > > > > I am talking about real hotplug, like a device which appears in the VM.
> > > > > > This new device must be handled by EAL and probed automatically if
> > > > > > comply with whitelist/blacklist policy given by the application or user.
> > > > > > Real hotplug is asynchronous.
> > > > >
> > > > > By 'asynchronous' here you mean it would be handled in the EAL interrupt thread
> > > > > or something different?
> > > >
> > > > Yes, we receive an hotplug event which is processed in the event thread.
> > > >
> > > > > Anyway, I suppose you do need a function inside DPDK that will do the actual work in response
> > > > > on hotplug event, right?
> > > >
> > > > Yes
> > >
> > > Ok, btw why that function has to be always called from interrupt thread?
> > > Why not to allow user to decide?
> >
> > Absolutely, the user must decide.
> > In the example of failsafe, the user instructs a policy to decide
> > which devices will be owned, so failsafe takes the decision based
> > on user inputs.
> >
> > > Some apps have their own thread dedicated for control ops (like testpmd)
> > > For them it might be plausible to call that function from their own control thread context.
> > > Konstantin
> > >
> > > >
> > > > > That's what I refer to as 'hotplug routine' above.
> > > > >
> > > > > > We will just receive notifications that it appeared.
> > > > > >
> > > > > > Note: there is some temporary code in failsafe to manage some hotplug.
> > > > > > This code must be removed when it will be properly handled in EAL.
> > > > >
> > > > > Ok, if it is just a temporary code, that would be removed soon -
> > > > > then it definitely seems wrong to modify tespmd (or any other user app)
> > > > > to comply with that temporary solution.
> > > >
> > > > It will be modified when EAL hotplug will be implemented.
> > > >
> > > > However, the ownership issue will be the same:
> > > > we don't know the owner when allocating a port.
>