From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <matan@mellanox.com>
Received: from EUR01-DB5-obe.outbound.protection.outlook.com
 (mail-db5eur01on0062.outbound.protection.outlook.com [104.47.2.62])
 by dpdk.org (Postfix) with ESMTP id BB0BF1B01C
 for <dev@dpdk.org>; Thu, 25 Jan 2018 10:36:41 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com;
 s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version;
 bh=E2PV2wz16/G9HMh4FZbfJXePYJG9lU/t21WUSmivznM=;
 b=BWfvPjnaWTUbphwd4rijjcRJNnH6oEObGlGyqXMJ6O8NX+UFXimoJT1z5UCFpwsVb9EFffXBplXeqAtWVmig6cs0PYA5qlvTb4bRFdvJ6vz9Cjca2IqXaMflrYuPjggLcRO2gcYx1v8obTDht+cDxGO2WVcIsAL/hz0/XuNw4sQ=
Received: from AM6PR0502MB3797.eurprd05.prod.outlook.com (52.133.21.26) by
 AM6PR0502MB4085.eurprd05.prod.outlook.com (52.133.30.160) with Microsoft SMTP
 Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id
 15.20.444.14; Thu, 25 Jan 2018 09:36:39 +0000
Received: from AM6PR0502MB3797.eurprd05.prod.outlook.com
 ([fe80::6c28:c6b3:de94:a733]) by AM6PR0502MB3797.eurprd05.prod.outlook.com
 ([fe80::6c28:c6b3:de94:a733%13]) with mapi id 15.20.0444.015; Thu, 25 Jan
 2018 09:36:39 +0000
From: Matan Azrad <matan@mellanox.com>
To: Matan Azrad <matan@mellanox.com>, =?iso-8859-1?Q?Ga=EBtan_Rivet?=
 <gaetan.rivet@6wind.com>, "Ananyev, Konstantin"
 <konstantin.ananyev@intel.com>, Thomas Monjalon <thomas@monjalon.net>
CC: "Wu, Jingjing" <jingjing.wu@intel.com>, "dev@dpdk.org" <dev@dpdk.org>,
 Neil Horman <nhorman@tuxdriver.com>, "Richardson, Bruce"
 <bruce.richardson@intel.com>
Thread-Topic: [dpdk-dev] [PATCH v3 7/7] app/testpmd: adjust ethdev port
 ownership
Thread-Index: AQHTlcAAsrC3L8dqnUaEGw/B6oZpaA==
Date: Thu, 25 Jan 2018 09:36:39 +0000
Message-ID: <AM6PR0502MB37974A3305A09994C1CAB06DD2E10@AM6PR0502MB3797.eurprd05.prod.outlook.com>
References: <1516293317-30748-8-git-send-email-matan@mellanox.com>
 <2601191342CEEE43887BDE71AB97725886280A68@irsmsx105.ger.corp.intel.com>
 <AM6PR0502MB37970EE52B78BA0B3E691A69D2EF0@AM6PR0502MB3797.eurprd05.prod.outlook.com>
 <2601191342CEEE43887BDE71AB97725886280AE8@irsmsx105.ger.corp.intel.com>
 <AM6PR0502MB3797B1AE571D56DBEB8C05CAD2EF0@AM6PR0502MB3797.eurprd05.prod.outlook.com>
 <20180119150017.mljpcdmldqx32mkq@bidouze.vm.6wind.com>
 <2601191342CEEE43887BDE71AB97725886281B1D@irsmsx105.ger.corp.intel.com>
 <AM6PR0502MB3797FB780B588499A71189DCD2EC0@AM6PR0502MB3797.eurprd05.prod.outlook.com>
 <2601191342CEEE43887BDE71AB97725886281E73@irsmsx105.ger.corp.intel.com>
 <AM6PR0502MB37976C3DFD1ABAF6439356BED2E30@AM6PR0502MB3797.eurprd05.prod.outlook.com>
 <20180123125637.p2kufd6n2erpiar5@bidouze.vm.6wind.com>
 <AM6PR0502MB37979FF971BEAE971C4714D6D2E30@AM6PR0502MB3797.eurprd05.prod.outlook.com>
In-Reply-To: <AM6PR0502MB37979FF971BEAE971C4714D6D2E30@AM6PR0502MB3797.eurprd05.prod.outlook.com>
Accept-Language: en-US, he-IL
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
authentication-results: spf=none (sender IP is )
 smtp.mailfrom=matan@mellanox.com; 
x-originating-ip: [193.47.165.251]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1; AM6PR0502MB4085;
 7:eqd/5l3zppCR6FetNQsT/0zKcXkv1OL43isgc4AnDsdGNEdl7o9qA/2OiA/P6DFA5u0B/ePoa9rGI43oZWRxIWVjeX40NN8kuFGXyWAgfHPUTmmyt6KBknPORsgkvnjV4TyuZ35noagFnMf6R6O0DtPV2JZfmih59GDkZrpG/Nn5k3twCMOsvQkrHjEaLi/QAmW9mxh39jsMbIPEWEe1N2ed4M6V85BNO4TTPHcNygDwnij+cOdShXdIYfboyaJc
x-ms-exchange-antispam-srfa-diagnostics: SSOS;
x-ms-office365-filtering-ht: Tenant
x-ms-office365-filtering-correlation-id: ed70341a-25bf-4e90-b92c-08d563d722c3
x-microsoft-antispam: UriScan:; BCL:0; PCL:0;
 RULEID:(7020095)(4652020)(48565401081)(5600026)(4604075)(3008032)(2017052603307)(7153060)(7193020);
 SRVR:AM6PR0502MB4085; 
x-ms-traffictypediagnostic: AM6PR0502MB4085:
x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr
x-microsoft-antispam-prvs: <AM6PR0502MB4085E5C94BACDE566C313B08D2E10@AM6PR0502MB4085.eurprd05.prod.outlook.com>
x-exchange-antispam-report-test: UriScan:(60795455431006)(278428928389397)(17755550239193); 
x-exchange-antispam-report-cfa-test: BCL:0; PCL:0;
 RULEID:(6040501)(2401047)(8121501046)(5005006)(93006095)(93001095)(3231023)(2400081)(944501161)(3002001)(10201501046)(6055026)(6041288)(20161123560045)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(20161123558120)(6072148)(201708071742011);
 SRVR:AM6PR0502MB4085; BCL:0; PCL:0; RULEID:; SRVR:AM6PR0502MB4085; 
x-forefront-prvs: 0563F2E8B7
x-forefront-antispam-report: SFV:NSPM;
 SFS:(10009020)(366004)(39380400002)(39860400002)(396003)(346002)(376002)(189003)(199004)(76104003)(55674003)(8936002)(99286004)(97736004)(33656002)(81156014)(81166006)(66066001)(54906003)(478600001)(3846002)(305945005)(7736002)(74316002)(68736007)(106356001)(229853002)(93886005)(316002)(110136005)(2906002)(2950100002)(55016002)(5250100002)(2900100001)(7696005)(6436002)(3660700001)(6506007)(76176011)(59450400001)(3280700002)(26005)(102836004)(105586002)(53936002)(4326008)(5660300001)(9686003)(6116002)(25786009)(86362001)(14454004)(6246003);
 DIR:OUT; SFP:1101; SCL:1; SRVR:AM6PR0502MB4085;
 H:AM6PR0502MB3797.eurprd05.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords;
 MX:1; A:1; LANG:en; 
received-spf: None (protection.outlook.com: mellanox.com does not designate
 permitted sender hosts)
x-microsoft-antispam-message-info: uAkA9QJ4OpLcXZv8erlvUH6XFlNwQ3N7TsfxEG0YWlMIkt0OM2YfmEB89OX30GPxrBUnSsyBBoUf8kk2R7OsmQ==
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: Mellanox.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ed70341a-25bf-4e90-b92c-08d563d722c3
X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Jan 2018 09:36:39.3757 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR0502MB4085
Subject: Re: [dpdk-dev] [PATCH v3 7/7] app/testpmd: adjust ethdev
	port	ownership
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Thu, 25 Jan 2018 09:36:41 -0000

Gaetan, Konstantin, Thomas

Any response to my suggestion below?

From: Matan Azrad
> Hi
>=20
> From: Ga=EBtan Rivet [mailto:gaetan.rivet@6wind.com] <snip>
> > > > > > > Look,
> > > > > > > > Testpmd initiates some of its internal databases depends
> > > > > > > > on specific port iteration, In some time someone may take
> > > > > > > > ownership of Testpmd ports and testpmd will continue to
> > > > > > > > touch
> > them.
> > > > > >
> > > > > > But if someone will take the ownership (assign new owner_id)
> > > > > > that port will not appear in RTE_ETH_FOREACH_DEV() any more.
> > > > > >
> > > > >
> > > > > Yes, but testpmd sometimes depends on previous iteration using
> > internal database.
> > > > > So it uses internal database that was updated by old iteration.
> > > >
> > > > That sounds like just a bug in testpmd that need to be fixed, no?
> > >
> > > If Testpmd already took ownership for these ports(like I did), it is =
ok.
> > >
> >
> > Have you tested using the default iterator (NO_OWNER)?
> > It worked until now with the bare minimal device tagging using
> > DEV_DEFERRED. Testpmd did not seem to mind having to skip this port.
> >
> > I'm sure there were places where this was overlooked, but overall, I'd
> > think everything should be fixable using only the NO_OWNER iteration.
>=20
> I don't think so.
>=20
> > Can you point to a specific scenario (command line, chain of event)
> > that would lead to a problem?
> >
>=20
> I didn't construct a race test to catch testpmd issue, but I think withou=
t this
> patch, there is a lot of issues.
> Go to the testpmd code (before ownership) and find usage of the old
> iterator(after the first iteration in main), Ask yourself what should hap=
pen if
> exactly in this time, a new port is created by fail-safe(plug in event).
>=20
> > > > Any particular places where outdated device info is used?
> > >
> > > For example, look for the stream management in testpmd(I think I saw
> > > it
> > there).
> > >
> >
> > The stream management is certainly shaky, but it happens after the EAL
> > initial port creation, and is not able to update itself for new
> > hotplugged ports (unless something changed).
> >
>=20
> Yes, but conceptually someone in the future may take the port(because it
> ownerless).
>=20
> > > > > > > If I look back on the fail-safe, its sole purpose is to have
> > > > > > > seamless hotplug with existing applications.
> > > > > > >
> > > > > > > Port ownership is a genericization of some functions
> > > > > > > introduced by the fail-safe, that could structure DPDK
> > > > > > > further. It should allow applications to have a seamless
> > > > > > > integration with subsystems using port ownership. Without
> > > > > > > this,
> > port ownership cannot be used.
> > > > > > >
> > > > > > > Testpmd should be fixed, but follow the most common design
> > > > > > > patterns of DPDK applications. Going with port ownership
> > > > > > > seems like a paradigm shift.
> > > > > > >
> > > > > > > > In addition
> > > > > > > > Using the old iterator in some places in testpmd will
> > > > > > > > cause a race for run-
> > > > > > time new ports(can be created by failsafe or any hotplug code):
> > > > > > > > - testpmd finds an ownerless port(just now created) by the
> > > > > > > > old iterator and start traffic there,
> >
> > How does testpmd start traffic there? Testpmd has only a callback for
> > displaying that it received an event for a new port. It has no concept
> > of hotplugging beyond that.
> >
>=20
> Yes, so no traffic just some control command.
>=20
> > Testpmd will not start using any new port probed using the hotplug API
> > on its own, again, unless something has drastically changed.
> >
>=20
> Every iterator using in testpmd is exposed to race.
>=20
> > > > > > > > - failsafe takes ownership of this new port and start traff=
ic there.
> > > > > > > > Problem!
> > > > > >
> > > > > > Could you shed a bit more light here - it would be race
> > > > > > condition between whom and whom?
> > > > >
> > > > > Sure.
> > > > >
> > > > > > As I remember in testpmd all control ops are done within one
> > > > > > thread (main lcore).
> > > > >
> > > > > But other dpdk entity can use another thread, for example:
> > > > > Failsafe uses the host thread(using alarm callback) to create a
> > > > > new port and
> > > > to take ownership of a port.
> > > >
> > > > Hm, and you create new ports inside failsafe PMD, right and then
> > > > set new owner_id for it?
> > >
> > > Yes.
> > >
> > > > And all this in alarm in interrupt thread?
> > >
> > > Yes.
> > >
> > > > If so I wonder how you can guarantee that no-one else will set
> > > > different owner_id between
> > > > rte_eth_dev_allocate() and rte_eth_dev_owner_set()?
> > >
> > > I check it (see failsafe patch to this series - V5).
> > > Function: fs_bus_init.
> > >
> > > > Could you point me to that place (I am not really familiar with
> > > > familiar with failsafe code)?
> > > >
> > > > >
> > > > > The race:
> > > > > Testpmd iterates over all ports by the master thread.
> > > > > Failsafe takes ownership of a port by the host thread and start u=
sing
> it.
> > > > > =3D> The two dpdk entities may use the device at same time!
> > > >
> >
> > When can this happen? Fail-safe creates its initial pool of ports
> > during EAL init, before testpmd scans eth_dev ports and configure its
> streams.
> > At that point, it has taken ownership, from the master lcore context.
> >
> > After this point, new ports could be detected and hotplugged by fail-sa=
fe.
> > However, even if testpmd had a callback to capture those new ports and
> > reconfigure its streams, it would be executed from within the
> > intr-thread, same as failsafe. If the thread was interrupted, by a
> > dataplane-lcore for example, streams would not have been reconfigured.
> > The fail-safe would execute its callback and set the owner-id before
> > the callback chains goes to the application.
> >
>=20
> Some iterator may be invoked in plug out process by other thread in testp=
md
> and causes to control command
>=20
> > And that would only be if testpmd had any callback for hotplugging
> > ports and reconfiguring its streams, which it hasn't, as far as I know.
> >
>=20
> We don't need to implement it in testpmd.
>=20
> > > > Ok, if failsafe really assigns its owner_id(s) to ports that are
> > > > already in use by the app, then how such scheme supposed to work
> > > > at
> > all?
> > >
> > > If the app works well (with the new rules) it already took ownership
> > > and
> > failsafe will see it and will wait until the application release it.
> > > Every dpdk entity should know which port it wants to manage, If 2
> > > entities want to manage the same device -  it can be ok and port
> > > ownership
> > can synchronize the usage.
> > >
> > > Probably, application which will run fail-safe wants to manage only
> > > the fail-
> > safe port and therefor to take ownership only for it.
> > >
> > > > I.E. application has a port - it assigns some owner_id !=3D 0 to it=
,
> > > > then PMD tries to set its owner_id tot the same port.
> > > > Obviously failsafe's set_owner() will always fail in such case.
> > > >
> > > Yes, and will try again after some time.
> > >
> > > > From what I hear we need to introduce a concept of 'default owner i=
d'.
> > > > I.E. when failsafe PMD is created - user assigns some owner_id to
> > > > it
> > (default).
> > > > Then failsafe PMD generates it's own owner_id and assigns it only
> > > > to the ports whose current owner_id is equal either 0 or 'default'
> owner_id.
> > > >
> > >
> > > It is a suggestion and we need to think about it more (I'm talking
> > > about it
> > with Gaetan in another thread).
> > > Actually I think, if we want a generic solution to the generic
> > > problem the
> > current solution is ok.
> > >
> >
> > We could as well conclude this other thread there.
> >
> > The only solution would be to have a default relationship between
> > owners, something that goes beyond the scope assigned by Thomas to
> > your evolution, but would be necessary for this API to be properly
> > used by existing applications.
> >
> > I think it's the only way to have a sane default behavior with your
> > API, but I also think this goes beyong the scope of the DPDK altogether=
.
> >
> > But even with those considerations that could be ironed out later (API
> > is still experimental anyway), in the meantime, I think we should
> > strive not to break "userland" as much as possible. Meaning that
> > unless you have a specific situation creating a bug, you shouldn't
> > have to modify testpmd, and if an issues arises, you need to try to
> > improve your API before resorting to changing the resource management
> model of all existing applications.
> >
>=20
> I understand it.
> Suggestion:
>=20
> 2 system owners.
> APP_OWNER - 1.
> NO_OWNER - 0.
>=20
> And allowing for more owners as now.
>=20
> 1. Every port creation will set the owner for NO_OWNER (as now).
> 2. There is option for all dpdk entities to take owner of  NO_OWNER ports=
 all
> the time(as now).
> 3. In some point in the end of EAL init: set all the NO_OWNER to
> APP_OWNER(for V6).
> 4. Change the old iterator to iterate over APP_OWNER ports(for V6).
>=20
> What do you think?
>=20
> <snip>