From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <matan@mellanox.com> Received: from EUR02-VE1-obe.outbound.protection.outlook.com (mail-eopbgr20081.outbound.protection.outlook.com [40.107.2.81]) by dpdk.org (Postfix) with ESMTP id E00073238 for <dev@dpdk.org>; Mon, 4 Dec 2017 19:10:57 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=f7HntU1bSH2gmGFYpAL3u9do3wKAKs0pqYPs8WHmOt8=; b=A4mLH1Vjh+uxjZu0nePKhwcJrUec7Gaim0WreN+tJhD7T5BxmwjWA3/obLX/vBXGy5abBmM4C2BN5oQGMxLAPxpvPZHcS9QFgSKlEsvbnUPCt4cgSsEX+HnJqAViWskaFDV2mBdEGHsGsFO6GqKw/eADnkApm8lG0bSBKDlZgVQ= Received: from HE1PR0502MB3659.eurprd05.prod.outlook.com (10.167.127.17) by HE1PR0502MB3659.eurprd05.prod.outlook.com (10.167.127.17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.282.5; Mon, 4 Dec 2017 18:10:56 +0000 Received: from HE1PR0502MB3659.eurprd05.prod.outlook.com ([fe80::982e:2dce:9449:6891]) by HE1PR0502MB3659.eurprd05.prod.outlook.com ([fe80::982e:2dce:9449:6891%13]) with mapi id 15.20.0282.007; Mon, 4 Dec 2017 18:10:56 +0000 From: Matan Azrad <matan@mellanox.com> To: Neil Horman <nhorman@tuxdriver.com> CC: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>, =?iso-8859-1?Q?Ga=EBtan_Rivet?= <gaetan.rivet@6wind.com>, Thomas Monjalon <thomas@monjalon.net>, "Wu, Jingjing" <jingjing.wu@intel.com>, "dev@dpdk.org" <dev@dpdk.org> Thread-Topic: [dpdk-dev] [PATCH 2/5] ethdev: add port ownership Thread-Index: AQHTaEBDeXaioloFskiHncvIK1YNTKMs3xqAgAANj4CAAX1kAIAC1eJggAA+JgCAABoCUIAByaUAgAAcouA= Date: Mon, 4 Dec 2017 18:10:56 +0000 Message-ID: <HE1PR0502MB36599F82DA6268F5387239E8D23C0@HE1PR0502MB3659.eurprd05.prod.outlook.com> References: <1511870281-15282-1-git-send-email-matan@mellanox.com> <1511870281-15282-3-git-send-email-matan@mellanox.com> <20171130123611.GA20914@hmswarspite.think-freely.org> <20171130132443.4htutb5gpktcshgh@bidouze.vm.6wind.com> <20171201120946.GA23598@hmswarspite.think-freely.org> <HE1PR0502MB36599441AFBE3AA0488AFBE0D23F0@HE1PR0502MB3659.eurprd05.prod.outlook.com> <2601191342CEEE43887BDE71AB9772585FAC3E77@irsmsx105.ger.corp.intel.com> <HE1PR0502MB3659E7B8BBD3C5D9364AC909D23F0@HE1PR0502MB3659.eurprd05.prod.outlook.com> <20171204160117.GB25048@hmswarspite.think-freely.org> In-Reply-To: <20171204160117.GB25048@hmswarspite.think-freely.org> Accept-Language: en-US, he-IL Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=matan@mellanox.com; x-originating-ip: [85.64.136.190] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; HE1PR0502MB3659; 6:MjFVpb40r7kQYIoC/s2VRukN83zQoQALYsmke9mn8izdTh602bFcZQIpHzFHY7kb+4htdh4YvzghkpgjFZL0s4kKTuUDpL3jeRBiEc43fr9XZO/GAPVZYPFwV88wE5e55N/QS3ZPvYqcvXTXe5fN/PLN5aoGDuPGznQsUGjSDlns1J888eMiceMOBOaZeBn1o0w+oesNVFKeHNSACtOTHQWSoXloQJwk3GU8O06bhB3caP+2oN195qTDM6fDufOtAEQPxHZBK4ta03wrrTsEEGbHJ9DGvQv39RIAZWSLQSGB4SVf8+2+2SpLdT+bbKdCAk08vwJhTg18yD2EQSDVvfaNvMKXEfKsrEI0fUkoRHM=; 5:vJkZa1AFObFWasXQXKh2JKrUGmfQvjaRymRXiUsgSBeeoPsdJI7iBQok2gZ7PUw2EwiKZ6AXVcafo9ypMASMpQAzTmlJEiT/X1pkBkoC/i4fmEnEO2K4+cXex23f+bm37E8RLLx84XTafz+02yAQLTEhrvA+WJ7EPJY6edtXQYs=; 24:jfY6Iy2uxW39py/lI9EqkJ7C1bONQzw/3khAKb+52LkJULBdOdgg0PPP0ofr82tQkn7HhUsdqgJseMWSf+JXuI7+bthrEwa7g9kUgKtkEJk=; 7:a6nmMajeGbrLwTIjXQ6m+YrLdSxV5QLr4D1heO20B/fHcarMejIzMh8a2GUEU1U/vOtnbni0TZSXmYzVp23llg4LpWyw62V3ufJ6hcVfbFor16g0iH0ODpYZIiT8lC0tqnzqXwpfssB/fx3i8GgGhgKTcwEDdMOTJSn/suJEPrz8dctKq43xuxCx2D+1NDdg6AvPE81GW6hziuY21TIkvwGpozlTy7OE7zjGZeRJALWgA9qEAm93AbpLuwBAUlyN x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 32df990f-b612-478c-e1dd-08d53b425d63 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(5600026)(4604075)(4534020)(4602075)(4627115)(201703031133081)(201702281549075)(48565401081)(2017052603286); SRVR:HE1PR0502MB3659; x-ms-traffictypediagnostic: HE1PR0502MB3659: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: <HE1PR0502MB36599E1C7927BD2894AB5F79D23C0@HE1PR0502MB3659.eurprd05.prod.outlook.com> x-exchange-antispam-report-test: UriScan:(60795455431006)(278428928389397)(228905959029699); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040450)(2401047)(8121501046)(5005006)(10201501046)(3002001)(3231022)(93006095)(93001095)(6055026)(6041248)(20161123560025)(20161123562025)(20161123555025)(20161123558100)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:HE1PR0502MB3659; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:HE1PR0502MB3659; x-forefront-prvs: 051158ECBB x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(366004)(346002)(376002)(39860400002)(189002)(13464003)(199003)(24454002)(76176011)(6436002)(2906002)(189998001)(9686003)(105586002)(229853002)(6246003)(68736007)(99286004)(53936002)(7696005)(6506006)(53546010)(81166006)(101416001)(33656002)(25786009)(54356011)(4326008)(7736002)(55016002)(305945005)(97736004)(316002)(54906003)(86362001)(2900100001)(2950100002)(6116002)(3280700002)(66066001)(478600001)(5890100001)(93886005)(74316002)(5250100002)(102836003)(6916009)(81156014)(3846002)(8936002)(106356001)(3660700001)(14454004)(5660300001)(8676002); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR0502MB3659; H:HE1PR0502MB3659.eurprd05.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 32df990f-b612-478c-e1dd-08d53b425d63 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Dec 2017 18:10:56.2094 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0502MB3659 Subject: Re: [dpdk-dev] [PATCH 2/5] ethdev: add port ownership X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <http://dpdk.org/ml/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://dpdk.org/ml/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <http://dpdk.org/ml/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> X-List-Received-Date: Mon, 04 Dec 2017 18:10:58 -0000 Hi Neil > -----Original Message----- > From: Neil Horman [mailto:nhorman@tuxdriver.com] > Sent: Monday, December 4, 2017 6:01 PM > To: Matan Azrad <matan@mellanox.com> > Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; Ga=EBtan Rivet > <gaetan.rivet@6wind.com>; Thomas Monjalon <thomas@monjalon.net>; > Wu, Jingjing <jingjing.wu@intel.com>; dev@dpdk.org > Subject: Re: [dpdk-dev] [PATCH 2/5] ethdev: add port ownership >=20 > On Sun, Dec 03, 2017 at 01:46:49PM +0000, Matan Azrad wrote: > > Hi Konstantine > > > > > -----Original Message----- > > > From: Ananyev, Konstantin [mailto:konstantin.ananyev@intel.com] > > > Sent: Sunday, December 3, 2017 1:10 PM > > > To: Matan Azrad <matan@mellanox.com>; Neil Horman > > > <nhorman@tuxdriver.com>; Ga=EBtan Rivet <gaetan.rivet@6wind.com> > > > Cc: Thomas Monjalon <thomas@monjalon.net>; Wu, Jingjing > > > <jingjing.wu@intel.com>; dev@dpdk.org > > > Subject: RE: [dpdk-dev] [PATCH 2/5] ethdev: add port ownership > > > > > > > > > > > > Hi Matan, > > > > > > > -----Original Message----- > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Matan Azrad > > > > Sent: Sunday, December 3, 2017 8:05 AM > > > > To: Neil Horman <nhorman@tuxdriver.com>; Ga=EBtan Rivet > > > > <gaetan.rivet@6wind.com> > > > > Cc: Thomas Monjalon <thomas@monjalon.net>; Wu, Jingjing > > > > <jingjing.wu@intel.com>; dev@dpdk.org > > > > Subject: Re: [dpdk-dev] [PATCH 2/5] ethdev: add port ownership > > > > > > > > Hi > > > > > > > > > -----Original Message----- > > > > > From: Neil Horman [mailto:nhorman@tuxdriver.com] > > > > > Sent: Friday, December 1, 2017 2:10 PM > > > > > To: Ga=EBtan Rivet <gaetan.rivet@6wind.com> > > > > > Cc: Matan Azrad <matan@mellanox.com>; Thomas Monjalon > > > > > <thomas@monjalon.net>; Jingjing Wu <jingjing.wu@intel.com>; > > > > > dev@dpdk.org > > > > > Subject: Re: [dpdk-dev] [PATCH 2/5] ethdev: add port ownership > > > > > > > > > > On Thu, Nov 30, 2017 at 02:24:43PM +0100, Ga=EBtan Rivet wrote: > > > > > > Hello Matan, Neil, > > > > > > > > > > > > I like the port ownership concept. I think it is needed to > > > > > > clarify some operations and should be useful to several subsyst= ems. > > > > > > > > > > > > This patch could certainly be sub-divided however, and your > > > > > > current > > > > > > 1/5 should probably come after this one. > > > > > > > > > > > > Some comments inline. > > > > > > > > > > > > On Thu, Nov 30, 2017 at 07:36:11AM -0500, Neil Horman wrote: > > > > > > > On Tue, Nov 28, 2017 at 11:57:58AM +0000, Matan Azrad wrote: > > > > > > > > The ownership of a port is implicit in DPDK. > > > > > > > > Making it explicit is better from the next reasons: > > > > > > > > 1. It may be convenient for multi-process applications to > > > > > > > > know > > > which > > > > > > > > process is in charge of a port. > > > > > > > > 2. A library could work on top of a port. > > > > > > > > 3. A port can work on top of another port. > > > > > > > > > > > > > > > > Also in the fail-safe case, an issue has been met in testpm= d. > > > > > > > > We need to check that the user is not trying to use a port > > > > > > > > which is already managed by fail-safe. > > > > > > > > > > > > > > > > Add ownership mechanism to DPDK Ethernet devices to avoid > > > > > > > > multiple management of a device by different DPDK entities. > > > > > > > > > > > > > > > > A port owner is built from owner id(number) and owner > > > > > > > > name(string) while the owner id must be unique to > > > > > > > > distinguish between two identical entity instances and the > > > > > > > > owner name can be > > > any name. > > > > > > > > The name helps to logically recognize the owner by > > > > > > > > different DPDK entities and allows easy debug. > > > > > > > > Each DPDK entity can allocate an owner unique identifier > > > > > > > > and can use it and its preferred name to owns valid ethdev > ports. > > > > > > > > Each DPDK entity can get any port owner status to decide > > > > > > > > if it can manage the port or not. > > > > > > > > > > > > > > > > The current ethdev internal port management is not > > > > > > > > affected by this feature. > > > > > > > > > > > > > > > > > > > > The internal port management is not affected, but the external > > > > > > interface is, however. In order to respect port ownership, > > > > > > applications are forced to modify their port iterator, as > > > > > > shown by your > > > > > testpmd patch. > > > > > > > > > > > > I think it would be better to modify the current > > > > > > RTE_ETH_FOREACH_DEV to call RTE_FOREACH_DEV_OWNED_BY, > and > > > > > > introduce a default owner that would represent the application > > > > > > itself (probably with the ID 0 and an owner string ""). Only > > > > > > with specific additional configuration should this default > > > > > > subset of ethdev be > > > divided. > > > > > > > > > > > > This would make this evolution seamless for applications, at > > > > > > no cost to the complexity of the design. > > > > > > > > > > > > > > Signed-off-by: Matan Azrad <matan@mellanox.com> > > > > > > > > > > > > > > > > > > > > > This seems fairly racy. What if one thread attempts to set > > > > > > > ownership on a port, while another is checking it on another > > > > > > > cpu in parallel. It doesn't seem like it will protect agains= t that at all. > > > > > > > It also doesn't protect against the possibility of multiple > > > > > > > threads attempting to poll for rx in parallel, which I think > > > > > > > was part of Thomas's origional statement regarding port > > > > > > > ownership (he noted that the lockless design implied only a > > > > > > > single thread should be allowed to poll > > > > > for receive or make configuration changes at a time. > > > > > > > > > > > > > > Neil > > > > > > > > > > > > > > > > > > > Isn't this race already there for any configuration operation > > > > > > / polling function? The DPDK arch is expecting applications to = solve > it. > > > > > > Why should port ownership be designed differently from other > > > > > > DPDK > > > > > components? > > > > > > > > > > > Yes, but that doesn't mean it should exist in purpituity, nor > > > > > does it mean that your new api should contain it as well. > > > > > > > > > > > Embedding checks for port ownership within operations will > > > > > > force everyone to bear their costs, even those not interested > > > > > > in using this API. These checks should be kept outside, within > > > > > > the entity claiming ownership of the port, in the form of > > > > > > using the proper port iterator IMO. > > > > > > > > > > > No. At the very least, you need to make the API itself exclusive= . > > > > > That is to say, you should at least ensure that a port ownership > > > > > get call doesn't race with a port ownership set call. It seems > > > > > rediculous to just leave that sort of locking as an exercize to t= he user. > > > > > > > > > > Neil > > > > > > > > > Neil, > > > > As Thomas mentioned, a DPDK port is designed to be managed by only > > > > one thread (or synchronized DPDK entity). > > > > So all the port management includes port ownership shouldn't be > > > > synchronized, i.e. locks are not needed. > > > > If some application want to do dual thread port management, the > > > > responsibility to synchronize the port ownership or any other port > > > > management is on this application. > > > > Port ownership doesn't come to allow synchronized management of > > > > the port by two DPDK entities in parallel, it is just nice way to > > > > answer next > > > questions: > > > > 1. Is the port already owned by some DPDK entity(in early control > > > path)? > > > > 2. If yes, Who is the owner? > > > > If the answer to the first question is no, the current entity can > > > > take the ownership without any lock(1 thread). > > > > If the answer to the first question is yes, you can recognize the > > > > owner and take decisions accordingly, sometimes you can decide to > > > > use the port because you logically know what the current owner > > > > does and you can be logically synchronized with it, sometimes you > > > > can just leave this port because you have not any deal with this o= wner. > > > > > > If the goal is just to have an ability to recognize is that device > > > is managed by another device (failsafe, bonding, etc.), then I > > > think all we need is a pointer to rte_eth_dev_data of the owner (NULL > would mean no owner). > > > > I think string is better than a pointer from the next reasons: > > 1. It is more human friendly than pointers for debug and printing. > > 2. it is flexible and allows to forward logical owner message to other = DPDK > entities. > > > > > Also I think if we'd like to introduce that mechanism, then it needs > > > to be > > > - mandatory (control API just don't allow changes to the device > > > configuration if caller is not an owner). > > > > But what if 2 DPDK entities should manage the same port \ using it and = they > are synchronized? > > > > > - transparent to the user (no API changes). > > > > For now, there is not API change but new suggested API to use for port > iteration. > > > > > - set/get owner ops need to be atomic if we want this mechanism to > > > be usable for MP. > > > > But also without atomic this mechanism is usable in MP. > > For example: > > PRIMARY application can set its owner with string "primary A". > > SECONDARY process (which attach to the ports only after the primary > created them )is not allowed to set owner(As you can see in the code) but= it > can read the owner string and see that the port owner is the primary > application. > > The "A" can just sign specific port type to the SECONDARY that this por= t > works with logic A which means, for example, primary should send the > packets and secondary should receive the packets. > > > But thats just the point, the operations that you are describing are not = atomic > at all. If the primary process is interrupted during its setting of a po= rts owner > ship after its read the current owner field, its entirely possible for th= e > secondary proces to set the owner as itself, and then have the primary > process set it back. Without locking, its just broken. I know that the = dpdk > likes to say its lockless, because it has no locks, but here we're just k= icking the > can down the road, by making the application add the locks for the librar= y. >=20 > Neil >=20 As I wrote before and as is in the code you can understand that secondary p= rocess should not take ownership of ports. Any port configuration (for example port creation and release) is not inter= nally synchronized between the primary to secondary processes so I don't se= e any reason to synchronize port ownership. If the primary-secondary process want to manage(configure) same port in sam= e time, they must be synchronized by the applications, so this is the case = in port ownership too (actually I don't think this synchronization is reali= stic because many configurations of the port are not in shared memory). So, actually secondary process should start its activity on ports only afte= r the primary process done with all configurations includes port ownership,= this part must already be synchronized. =20 > > > Konstantin > >