From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <matan@mellanox.com>
Received: from EUR01-VE1-obe.outbound.protection.outlook.com
(mail-ve1eur01on0040.outbound.protection.outlook.com [104.47.1.40])
by dpdk.org (Postfix) with ESMTP id BC1047CEF
for <dev@dpdk.org>; Tue, 23 Jan 2018 09:54:29 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com;
s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version;
bh=gluqaUi2JSifVf5N6ky0XrgyDcI6GGsb+2IqNmJsIiQ=;
b=F8+mNz3LDD9g5LSn6g0+IPbmqgq8TrwrP9oXYwy0jOKPLul8ZsnIl1YNff5d7nxL7n6qCpDu6kIM6Brixi/1qSN3/U5cXqEfoIUE/Ml00H2eKYRIR5qQHccuUoKU5RySuLauQflbHu7wVpoFdR57JYZPwTeCzOserzX1blBiK9M=
Received: from AM6PR0502MB3797.eurprd05.prod.outlook.com (52.133.21.26) by
AM6PR0502MB4054.eurprd05.prod.outlook.com (52.133.30.153) with Microsoft SMTP
Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id
15.20.428.17; Tue, 23 Jan 2018 08:54:28 +0000
Received: from AM6PR0502MB3797.eurprd05.prod.outlook.com
([fe80::6c28:c6b3:de94:a733]) by AM6PR0502MB3797.eurprd05.prod.outlook.com
([fe80::6c28:c6b3:de94:a733%13]) with mapi id 15.20.0428.023; Tue, 23 Jan
2018 08:54:28 +0000
From: Matan Azrad <matan@mellanox.com>
To: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>,
=?iso-8859-1?Q?Ga=EBtan_Rivet?= <gaetan.rivet@6wind.com>
CC: Thomas Monjalon <thomas@monjalon.net>, "Wu, Jingjing"
<jingjing.wu@intel.com>, "dev@dpdk.org" <dev@dpdk.org>, Neil Horman
<nhorman@tuxdriver.com>, "Richardson, Bruce" <bruce.richardson@intel.com>
Thread-Topic: [PATCH v3 7/7] app/testpmd: adjust ethdev port ownership
Thread-Index: AQHTkSJSs9wv4iQCB0Cv+2LP7vySOKN7IsfQgAAIKACAAAHXQIAAHWSAgASMgQCAAAupAIAAgCeAgAC+c1A=
Date: Tue, 23 Jan 2018 08:54:27 +0000
Message-ID: <AM6PR0502MB37976C3DFD1ABAF6439356BED2E30@AM6PR0502MB3797.eurprd05.prod.outlook.com>
References: <1515318351-4756-1-git-send-email-matan@mellanox.com>
<1516293317-30748-1-git-send-email-matan@mellanox.com>
<1516293317-30748-8-git-send-email-matan@mellanox.com>
<2601191342CEEE43887BDE71AB97725886280A68@irsmsx105.ger.corp.intel.com>
<AM6PR0502MB37970EE52B78BA0B3E691A69D2EF0@AM6PR0502MB3797.eurprd05.prod.outlook.com>
<2601191342CEEE43887BDE71AB97725886280AE8@irsmsx105.ger.corp.intel.com>
<AM6PR0502MB3797B1AE571D56DBEB8C05CAD2EF0@AM6PR0502MB3797.eurprd05.prod.outlook.com>
<20180119150017.mljpcdmldqx32mkq@bidouze.vm.6wind.com>
<2601191342CEEE43887BDE71AB97725886281B1D@irsmsx105.ger.corp.intel.com>
<AM6PR0502MB3797FB780B588499A71189DCD2EC0@AM6PR0502MB3797.eurprd05.prod.outlook.com>
<2601191342CEEE43887BDE71AB97725886281E73@irsmsx105.ger.corp.intel.com>
In-Reply-To: <2601191342CEEE43887BDE71AB97725886281E73@irsmsx105.ger.corp.intel.com>
Accept-Language: en-US, he-IL
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
authentication-results: spf=none (sender IP is )
smtp.mailfrom=matan@mellanox.com;
x-originating-ip: [193.47.165.251]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1; AM6PR0502MB4054;
7:FNMM0eOD5U3Vd42CFyvFVbN5QijzGb/bKLZnK8/eCZCFWxRn1t8LAyoz6L41CedzCICbqWaACxa6G+/rDfX4Ke7iARXghJsvTqBY4Jpq0dBDwrgjeDlWfpjpZwpZSmJQQnwoHx77XKqI8YF/fKi7D/5UOsKf159q1j1d8E3FLXUn63Ni6Dj1iFsf3vC5ZrCb3CEhEC8Ac4PiO67xKeMBKTCRY8AiwWyaG9TFz8xQ7bFwBCcCwIn/3sngqXJJ6AZL
x-ms-exchange-antispam-srfa-diagnostics: SSOS;
x-ms-office365-filtering-ht: Tenant
x-ms-office365-filtering-correlation-id: b3bd3634-e4fd-4511-8d5c-08d5623ee915
x-microsoft-antispam: UriScan:; BCL:0; PCL:0;
RULEID:(7020095)(4652020)(4534145)(4627221)(201703031133081)(201702281549075)(48565401081)(5600026)(4604075)(3008032)(2017052603307)(7153060)(7193020);
SRVR:AM6PR0502MB4054;
x-ms-traffictypediagnostic: AM6PR0502MB4054:
x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr
x-microsoft-antispam-prvs: <AM6PR0502MB4054FAC107499DE89CFB4F60D2E30@AM6PR0502MB4054.eurprd05.prod.outlook.com>
x-exchange-antispam-report-test: UriScan:(60795455431006)(278428928389397)(228905959029699)(17755550239193);
x-exchange-antispam-report-cfa-test: BCL:0; PCL:0;
RULEID:(6040501)(2401047)(8121501046)(5005006)(10201501046)(3002001)(93006095)(93001095)(3231023)(2400081)(944501161)(6055026)(6041288)(20161123560045)(20161123562045)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(6072148)(201708071742011);
SRVR:AM6PR0502MB4054; BCL:0; PCL:0; RULEID:(100000803101)(100110400095);
SRVR:AM6PR0502MB4054;
x-forefront-prvs: 05610E64EE
x-forefront-antispam-report: SFV:NSPM;
SFS:(10009020)(396003)(39380400002)(366004)(376002)(39860400002)(346002)(51444003)(189003)(199004)(13464003)(57704003)(5890100001)(26005)(229853002)(25786009)(6116002)(3846002)(4326008)(5660300001)(2950100002)(74316002)(2900100001)(93886005)(14454004)(8936002)(53936002)(55016002)(316002)(97736004)(81166006)(54906003)(110136005)(5250100002)(8676002)(9686003)(81156014)(6506007)(3280700002)(53546011)(59450400001)(7736002)(7696005)(6246003)(105586002)(478600001)(99286004)(106356001)(102836004)(2906002)(6436002)(68736007)(66066001)(86362001)(3660700001)(76176011)(305945005)(33656002);
DIR:OUT; SFP:1101; SCL:1; SRVR:AM6PR0502MB4054;
H:AM6PR0502MB3797.eurprd05.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords;
MX:1; A:1; LANG:en;
received-spf: None (protection.outlook.com: mellanox.com does not designate
permitted sender hosts)
x-microsoft-antispam-message-info: mYYlWE+DafteI0ecxo3v1tzBi9NZMgyKQlnSD5OkNZKBRg+9hBBW5b0xBt8EuWYAPm9P2voYpESDkF3XHVqsvw==
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: Mellanox.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b3bd3634-e4fd-4511-8d5c-08d5623ee915
X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Jan 2018 08:54:27.9697 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR0502MB4054
Subject: Re: [dpdk-dev] [PATCH v3 7/7] app/testpmd: adjust ethdev port
ownership
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://dpdk.org/ml/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://dpdk.org/ml/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Tue, 23 Jan 2018 08:54:30 -0000
Hi Konstantin
From: Ananyev, Konstantin, Monday, January 22, 2018 10:49 PM
> Hi Matan,
>=20
> > -----Original Message-----
> > From: Matan Azrad [mailto:matan@mellanox.com]
> > Sent: Monday, January 22, 2018 1:23 PM
> > To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; Ga=EBtan Rivet
> > <gaetan.rivet@6wind.com>
> > Cc: Thomas Monjalon <thomas@monjalon.net>; Wu, Jingjing
> > <jingjing.wu@intel.com>; dev@dpdk.org; Neil Horman
> > <nhorman@tuxdriver.com>; Richardson, Bruce
> > <bruce.richardson@intel.com>
> > Subject: RE: [PATCH v3 7/7] app/testpmd: adjust ethdev port ownership
> >
> >
> > Hi
> > From: Ananyev, Konstantin [mailto:konstantin.ananyev@intel.com]
> > > Hi lads,
> > >
> > > >
> > > > Hi Matan,
> > > >
> > > > On Fri, Jan 19, 2018 at 01:35:10PM +0000, Matan Azrad wrote:
> > > > > Hi Konstantin
> > > > >
> > > > > From: Ananyev, Konstantin, Friday, January 19, 2018 3:09 PM
> > > > > > > -----Original Message-----
> > > > > > > From: Matan Azrad [mailto:matan@mellanox.com]
> > > > > > > Sent: Friday, January 19, 2018 12:52 PM
> > > > > > > To: Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> > > > > > > Thomas Monjalon <thomas@monjalon.net>; Gaetan Rivet
> > > > > > <gaetan.rivet@6wind.com>;
> > > > > > > Wu, Jingjing <jingjing.wu@intel.com>
> > > > > > > Cc: dev@dpdk.org; Neil Horman <nhorman@tuxdriver.com>;
> > > > > > > Richardson, Bruce <bruce.richardson@intel.com>
> > > > > > > Subject: RE: [PATCH v3 7/7] app/testpmd: adjust ethdev port
> > > > > > > ownership
> > > > > > >
> > > > > > > Hi Konstantin
> > > > > > >
> > > > > > > From: Ananyev, Konstantin, Friday, January 19, 2018 2:38 PM
> > > > > > > > To: Matan Azrad <matan@mellanox.com>; Thomas Monjalon
> > > > > > > > <thomas@monjalon.net>; Gaetan Rivet
> > > <gaetan.rivet@6wind.com>;
> > > > > > Wu,
> > > > > > > > Jingjing <jingjing.wu@intel.com>
> > > > > > > > Cc: dev@dpdk.org; Neil Horman <nhorman@tuxdriver.com>;
> > > > > > > > Richardson, Bruce <bruce.richardson@intel.com>
> > > > > > > > Subject: RE: [PATCH v3 7/7] app/testpmd: adjust ethdev
> > > > > > > > port ownership
> > > > > > > >
> > > > > > > > Hi Matan,
> > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Matan Azrad [mailto:matan@mellanox.com]
> > > > > > > > > Sent: Thursday, January 18, 2018 4:35 PM
> > > > > > > > > To: Thomas Monjalon <thomas@monjalon.net>; Gaetan Rivet
> > > > > > > > > <gaetan.rivet@6wind.com>; Wu, Jingjing
> > > > > > > > > <jingjing.wu@intel.com>
> > > > > > > > > Cc: dev@dpdk.org; Neil Horman <nhorman@tuxdriver.com>;
> > > > > > Richardson,
> > > > > > > > > Bruce <bruce.richardson@intel.com>; Ananyev, Konstantin
> > > > > > > > > <konstantin.ananyev@intel.com>
> > > > > > > > > Subject: [PATCH v3 7/7] app/testpmd: adjust ethdev port
> > > > > > > > > ownership
> > > > > > > > >
> > > > > > > > > Testpmd should not use ethdev ports which are managed by
> > > > > > > > > other DPDK entities.
> > > > > > > > >
> > > > > > > > > Set Testpmd ownership to each port which is not used by
> > > > > > > > > other entity and prevent any usage of ethdev ports which
> > > > > > > > > are not owned by
> > > > > > Testpmd.
> > > > > > > > >
> > > > > > > > > Signed-off-by: Matan Azrad <matan@mellanox.com>
> > > > > > > > > ---
> > > > > > > > > app/test-pmd/cmdline.c | 89 +++++++++++++++++++----=
--
> ----
> > > -------
> > > > > > ----
> > > > > > > > -----
> > > > > > > > > app/test-pmd/cmdline_flow.c | 2 +-
> > > > > > > > > app/test-pmd/config.c | 37 ++++++++++---------
> > > > > > > > > app/test-pmd/parameters.c | 4 +-
> > > > > > > > > app/test-pmd/testpmd.c | 63 ++++++++++++++++++++---=
-
> ----
> > > ----
> > > > > > > > > app/test-pmd/testpmd.h | 3 ++
> > > > > > > > > 6 files changed, 103 insertions(+), 95 deletions(-)
> > > > > > > > >
> > > > > > > > > diff --git a/app/test-pmd/cmdline.c
> > > > > > > > > b/app/test-pmd/cmdline.c index
> > > > > > > > > 31919ba..6199c64 100644
> > > > > > > > > --- a/app/test-pmd/cmdline.c
> > > > > > > > > +++ b/app/test-pmd/cmdline.c
> > > > > > > > > @@ -1394,7 +1394,7 @@ struct cmd_config_speed_all {
> > > > > > > > > &link_speed) < 0)
> > > > > > > > > return;
> > > > > > > > >
> > > > > > > > > - RTE_ETH_FOREACH_DEV(pid) {
> > > > > > > > > + RTE_ETH_FOREACH_DEV_OWNED_BY(pid,
> my_owner.id) {
> > > > > > > >
> > > > > > > > Why do we need all these changes?
> > > > > > > > As I understand you changed definition of
> > > > > > > > RTE_ETH_FOREACH_DEV(), so no testpmd should work ok
> > > > > > > > default
> > > (no_owner case).
> > > > > > > > Am I missing something here?
> > > > > > >
> > > > > > > Now, After Gaetan suggestion RTE_ETH_FOREACH_DEV(pid) will
> > > > > > > iterate
> > > > > > over all valid and ownerless ports.
> > > > > >
> > > > > > Yes.
> > > > > >
> > > > > > > Here Testpmd wants to iterate over its owned ports.
> > > > > >
> > > > > > Why? Why it can't just iterate over all valid and ownerless por=
ts?
> > > > > > As I understand it would be enough to fix current problems and
> > > > > > would allow us to avoid any changes in testmpd (which I think
> > > > > > is a good
> > > thing).
> > > > >
> > > > > Yes, I understand that this big change is very daunted, But I
> > > > > think the current a lot of bugs in testpmd(regarding port
> > > > > ownership) even more
> > > > daunted.
> > > > >
> > > > > Look,
> > > > > Testpmd initiates some of its internal databases depends on
> > > > > specific port iteration, In some time someone may take ownership
> > > > > of Testpmd
> > > ports and testpmd will continue to touch them.
> > >
> > > But if someone will take the ownership (assign new owner_id) that
> > > port will not appear in RTE_ETH_FOREACH_DEV() any more.
> > >
> >
> > Yes, but testpmd sometimes depends on previous iteration using internal
> database.
> > So it uses internal database that was updated by old iteration.
>=20
> That sounds like just a bug in testpmd that need to be fixed, no?
If Testpmd already took ownership for these ports(like I did), it is ok.
> Any particular places where outdated device info is used?
For example, look for the stream management in testpmd(I think I saw it the=
re).
> > > > If I look back on the fail-safe, its sole purpose is to have
> > > > seamless hotplug with existing applications.
> > > >
> > > > Port ownership is a genericization of some functions introduced by
> > > > the fail-safe, that could structure DPDK further. It should allow
> > > > applications to have a seamless integration with subsystems using
> > > > port ownership. Without this, port ownership cannot be used.
> > > >
> > > > Testpmd should be fixed, but follow the most common design
> > > > patterns of DPDK applications. Going with port ownership seems
> > > > like a paradigm shift.
> > > >
> > > > > In addition
> > > > > Using the old iterator in some places in testpmd will cause a
> > > > > race for run-
> > > time new ports(can be created by failsafe or any hotplug code):
> > > > > - testpmd finds an ownerless port(just now created) by the old
> > > > > iterator and start traffic there,
> > > > > - failsafe takes ownership of this new port and start traffic the=
re.
> > > > > Problem!
> > >
> > > Could you shed a bit more light here - it would be race condition
> > > between whom and whom?
> >
> > Sure.
> >
> > > As I remember in testpmd all control ops are done within one thread
> > > (main lcore).
> >
> > But other dpdk entity can use another thread, for example:
> > Failsafe uses the host thread(using alarm callback) to create a new por=
t and
> to take ownership of a port.
>=20
> Hm, and you create new ports inside failsafe PMD, right and then set new
> owner_id for it?
Yes.
> And all this in alarm in interrupt thread?
Yes.
> If so I wonder how you can guarantee that no-one else will set different
> owner_id between
> rte_eth_dev_allocate() and rte_eth_dev_owner_set()?
I check it (see failsafe patch to this series - V5).
Function: fs_bus_init.
> Could you point me to that place (I am not really familiar with familiar =
with
> failsafe code)?
>=20
> >
> > The race:
> > Testpmd iterates over all ports by the master thread.
> > Failsafe takes ownership of a port by the host thread and start using i=
t.
> > =3D> The two dpdk entities may use the device at same time!
>=20
> Ok, if failsafe really assigns its owner_id(s) to ports that are already =
in use by
> the app, then how such scheme supposed to work at all?
If the app works well (with the new rules) it already took ownership and fa=
ilsafe will see it and will wait until the application release it.
Every dpdk entity should know which port it wants to manage,
If 2 entities want to manage the same device - it can be ok and port owner=
ship can synchronize the usage.
Probably, application which will run fail-safe wants to manage only the fai=
l-safe port and therefor to take ownership only for it.
> I.E. application has a port - it assigns some owner_id !=3D 0 to it, then=
PMD tries
> to set its owner_id tot the same port.
> Obviously failsafe's set_owner() will always fail in such case.
>
Yes, and will try again after some time.
=20
> From what I hear we need to introduce a concept of 'default owner id'.
> I.E. when failsafe PMD is created - user assigns some owner_id to it (def=
ault).
> Then failsafe PMD generates it's own owner_id and assigns it only to the
> ports whose current owner_id is equal either 0 or 'default' owner_id.
>=20
It is a suggestion and we need to think about it more (I'm talking about it=
with Gaetan in another thread).
Actually I think, if we want a generic solution to the generic problem the =
current solution is ok.=20
> >
> > Obeying the new ownership rules can prevent all these races.
> >
>=20
> When we discussed RFC of owner_id patch, I thought we all agreed that
> owner_id API shouldn't be mandatory - i.e. existing apps not required to
> change to work normally with that.
Yes, it is not mandatory if app doesn't use hotplug.
I think with hotplug it is mandatory in the most cases.
And it can ease the secondary process model too.
Again, in the generic ownership problem as discussed in RFC:
Every entity, include app, should know which ports it wants to manage and t=
o take ownership only for them.
> Though right now it seems that application changes seems necessary, at le=
ast
> to work ok with failsafe PMD.
And for solving the generic problem of ownership.(will defend from future i=
ssues by sure).
> Which makes we wonder was it some sort of misunderstanding or we did we
> do something wrong here?
Mistakes can be done all the time, but I think we are all understand the bi=
g issue of ownership and how the current solution solves it.
fail-safe it is just a current example for the problems which are possible =
because of the generic ownership issue.
Thanks,
Matan
> Konstantin
>=20
> > > The only way to attach/detach port with it - invoke testpmd CLI
> > > "attach/detach" port.
> > >
> > > Konstantin
> > >
> > > >
> > > > Testpmd does not handle detection of new port. If it did, testing
> > > > fail-safe with it would be wrong.
> > > >
> > > > At startup, RTE_ETH_FOREACH_DEV already fixed the issue of
> > > > registering DEFERRED ports. There are still remaining issues
> > > > regarding this, but I think they should be fixed. The architecture
> > > > does not need to be completely moved to port ownership.
> > > >
> > > > If anything, this should serve as a test for your API with common
> > > > applications. I think you'd prefer to know and debug with testpmd
> > > > instead of firing up VPP or something like that to determine what
> > > > went wrong with using the fail-safe.
> > > >
> > > > >
> > > > > In addition
> > > > > As a good example for well-done application (free from ownership
> > > > > bugs) I tried here to adjust Tespmd to the new rules and BTW to
> > > > > fix a
> > > > lot of bugs.
> > > >
> > > > Testpmd has too much cruft, it won't ever be a good example of a
> > > > well-done application.
> > > >
> > > > If you want to demonstrate ownership, I think you should start an
> > > > example application demonstrating race conditions and their mitigat=
ion.
> > > >
> > > > I think that would be interesting for many DPDK users.
> > > >
> > > > >
> > > > >
> > > > > So actually applications which are not aware to the port
> > > > > ownership still are exposed to races, but if there use the old
> > > > > iterator(with the new
> > > > change) the amount of races decreases.
> > > > >
> > > > > Thanks, Matan.
> > > > > > Konstantin
> > > > > >
> > > > > > >
> > > > > > > I added to Testpmd ability to take an ownership of ports as
> > > > > > > the new ownership and synchronization rules suggested, Since
> > > > > > > Tespmd is a DPDK entity which wants that no one will touch
> > > > > > > its owned ports, It must allocate
> > > > > > an unique ID, set owner for its ports (see in main function)
> > > > > > and recognizes them by its owner ID.
> > > > > > >
> > > > > > > > Konstantin
> > > > > > > >
> > > >
> > > > Regards,
> > > > --
> > > > Ga=EBtan Rivet
> > > > 6WIND