From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-ve1eur01on0076.outbound.protection.outlook.com [104.47.1.76]) by dpdk.org (Postfix) with ESMTP id 5E65C1D8E for ; Tue, 5 Dec 2017 07:08:37 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=IgQq5nmqnt06cx/tckq4oiibPe501TRShStJsskh+UM=; b=vS/tF1536kOPVj+gpg9WUEkisaieCyRzfNXFxwTH5KFk0xvJYvJNj36DgHOOv3VXtZTYUfsZ8mUQBblktX6BakEnku/fTu5Ni3bOtDcerijzRO33RmMSZw/DlJtBYOTdTF6hR0yWE1GAL0QOW1u7ov9yu4GJvX+BZ4Zo32fZfrc= Received: from HE1PR0502MB3659.eurprd05.prod.outlook.com (10.167.127.17) by HE1PR0502MB3657.eurprd05.prod.outlook.com (10.167.127.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.282.5; Tue, 5 Dec 2017 06:08:35 +0000 Received: from HE1PR0502MB3659.eurprd05.prod.outlook.com ([fe80::982e:2dce:9449:6891]) by HE1PR0502MB3659.eurprd05.prod.outlook.com ([fe80::982e:2dce:9449:6891%13]) with mapi id 15.20.0282.007; Tue, 5 Dec 2017 06:08:35 +0000 From: Matan Azrad To: Neil Horman CC: "Ananyev, Konstantin" , =?iso-8859-1?Q?Ga=EBtan_Rivet?= , Thomas Monjalon , "Wu, Jingjing" , "dev@dpdk.org" Thread-Topic: [dpdk-dev] [PATCH 2/5] ethdev: add port ownership Thread-Index: AQHTaEBDeXaioloFskiHncvIK1YNTKMs3xqAgAANj4CAAX1kAIAC1eJggAA+JgCAABoCUIAByaUAgAAcouCAAFA1gIAAeA4Q Date: Tue, 5 Dec 2017 06:08:35 +0000 Message-ID: References: <1511870281-15282-1-git-send-email-matan@mellanox.com> <1511870281-15282-3-git-send-email-matan@mellanox.com> <20171130123611.GA20914@hmswarspite.think-freely.org> <20171130132443.4htutb5gpktcshgh@bidouze.vm.6wind.com> <20171201120946.GA23598@hmswarspite.think-freely.org> <2601191342CEEE43887BDE71AB9772585FAC3E77@irsmsx105.ger.corp.intel.com> <20171204160117.GB25048@hmswarspite.think-freely.org> <20171204223051.GA11618@hmswarspite.think-freely.org> In-Reply-To: <20171204223051.GA11618@hmswarspite.think-freely.org> Accept-Language: en-US, he-IL Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=matan@mellanox.com; x-originating-ip: [193.47.165.251] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; HE1PR0502MB3657; 6:ZlBat8fqYOAd5uuSJnOznlrDH5mMRVvBbOP4qmjQ6dwsddEjY2UlCTl+VovI9scTBTND0nnwicUOFfXCc4Dy6U7NsX94OwDwcNa2WCefzNGbgNGrOmTfOxItikRKwei7eS+OUG+f68waGOSmcA6A0tHiAekLU5Fkn3iIFl0kqI8+Jo6xNXsxI7m8QDtL6jeN5fU4fO2bT6Fw5EQMksRUupcOdlb/8MWmtzRm5x1ECeKbjS1vT3ZWl/eH+O7MuKktBzJ5j7cLWFP7kCO6RAKz4XnjA5s6uSBdAUm47uNxXN2oQ0TMP7wIjW10/VGH/8Q7OSdFJJ26c7AmQplSNEqLKv9fQbASrzfxB93NUG/6/nU=; 5:EhT90ZAAUTXTS6OwfhNkcP0ZuVc6VX+sRyGP0iIWBJELdZefCU8J1bl+w5h2UFA3vTAI4/wGr/0/8a+CGa5xKlXPinchKvGbs7gvgjVSrxfTxJFehektapHWplOze3r/hldhLnGBwNv18t0sO5hqF9UZGwMK+gdYywe2T0dc52g=; 24:npMF9RRKuZAZAyFrHSGbBm2Rbei7ewNkIxpxejRyM6DpcKEZOJV93DRqq9F2zmtQ+hzNHGzcKrQ4plNXA2htTCKacwXyZn3IRbQjjuG+tg8=; 7:77gEWr9iVEz2OZoX2KJf7nMQhiv/fnwHGxzDotwK9wib0iy67rQrQdo+hZEYBSr0Z6BveER4oeRdZnYU34i+oAyqK82kWwv046vrIsJqa8/ouFbQrol2sjFWdGLCl0xLDYzNYMWlpv+mvGQBtzrlVRkpyU7UV0XYdEUUyFYvF09PLoZ/smv4q2D8lSu+GzolEWN3VxvLw8en54eTYKYytAoN2TI3hjcplQG3dgSetlef6lIsC22O8bvBW9k2F7cx x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 59679417-156d-4b3e-3c6b-08d53ba69edb x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(5600026)(4604075)(4534020)(4602075)(4627115)(201703031133081)(201702281549075)(48565401081)(2017052603286); SRVR:HE1PR0502MB3657; x-ms-traffictypediagnostic: HE1PR0502MB3657: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(60795455431006)(278428928389397)(228905959029699); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040450)(2401047)(5005006)(8121501046)(93006095)(93001095)(3231022)(10201501046)(3002001)(6055026)(6041248)(20161123562025)(20161123558100)(20161123564025)(20161123555025)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:HE1PR0502MB3657; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:HE1PR0502MB3657; x-forefront-prvs: 0512CC5201 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(346002)(376002)(366004)(39860400002)(189002)(13464003)(24454002)(199003)(5660300001)(14454004)(102836003)(101416001)(6246003)(53936002)(5890100001)(53946003)(5250100002)(6916009)(2950100002)(99286004)(478600001)(2906002)(6116002)(3846002)(7736002)(81166006)(54356011)(7696005)(2900100001)(8676002)(3280700002)(3660700001)(229853002)(105586002)(74316002)(6436002)(6506006)(106356001)(4326008)(97736004)(81156014)(305945005)(76176011)(33656002)(53546010)(189998001)(54906003)(66066001)(316002)(9686003)(93886005)(86362001)(55016002)(8936002)(25786009)(68736007); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR0502MB3657; H:HE1PR0502MB3659.eurprd05.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 59679417-156d-4b3e-3c6b-08d53ba69edb X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Dec 2017 06:08:35.6696 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0502MB3657 Subject: Re: [dpdk-dev] [PATCH 2/5] ethdev: add port ownership X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 06:08:37 -0000 Hi Neil > -----Original Message----- > From: Neil Horman [mailto:nhorman@tuxdriver.com] > Sent: Tuesday, December 5, 2017 12:31 AM > To: Matan Azrad > Cc: Ananyev, Konstantin ; Ga=EBtan Rivet > ; Thomas Monjalon ; > Wu, Jingjing ; dev@dpdk.org > Subject: Re: [dpdk-dev] [PATCH 2/5] ethdev: add port ownership >=20 > On Mon, Dec 04, 2017 at 06:10:56PM +0000, Matan Azrad wrote: > > Hi Neil > > > > > -----Original Message----- > > > From: Neil Horman [mailto:nhorman@tuxdriver.com] > > > Sent: Monday, December 4, 2017 6:01 PM > > > To: Matan Azrad > > > Cc: Ananyev, Konstantin ; Ga=EBtan Rive= t > > > ; Thomas Monjalon > ; Wu, > > > Jingjing ; dev@dpdk.org > > > Subject: Re: [dpdk-dev] [PATCH 2/5] ethdev: add port ownership > > > > > > On Sun, Dec 03, 2017 at 01:46:49PM +0000, Matan Azrad wrote: > > > > Hi Konstantine > > > > > > > > > -----Original Message----- > > > > > From: Ananyev, Konstantin [mailto:konstantin.ananyev@intel.com] > > > > > Sent: Sunday, December 3, 2017 1:10 PM > > > > > To: Matan Azrad ; Neil Horman > > > > > ; Ga=EBtan Rivet > > > > > > Cc: Thomas Monjalon ; Wu, Jingjing > > > > > ; dev@dpdk.org > > > > > Subject: RE: [dpdk-dev] [PATCH 2/5] ethdev: add port ownership > > > > > > > > > > > > > > > > > > > > Hi Matan, > > > > > > > > > > > -----Original Message----- > > > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Matan > > > > > > Azrad > > > > > > Sent: Sunday, December 3, 2017 8:05 AM > > > > > > To: Neil Horman ; Ga=EBtan Rivet > > > > > > > > > > > > Cc: Thomas Monjalon ; Wu, Jingjing > > > > > > ; dev@dpdk.org > > > > > > Subject: Re: [dpdk-dev] [PATCH 2/5] ethdev: add port ownership > > > > > > > > > > > > Hi > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Neil Horman [mailto:nhorman@tuxdriver.com] > > > > > > > Sent: Friday, December 1, 2017 2:10 PM > > > > > > > To: Ga=EBtan Rivet > > > > > > > Cc: Matan Azrad ; Thomas Monjalon > > > > > > > ; Jingjing Wu ; > > > > > > > dev@dpdk.org > > > > > > > Subject: Re: [dpdk-dev] [PATCH 2/5] ethdev: add port > > > > > > > ownership > > > > > > > > > > > > > > On Thu, Nov 30, 2017 at 02:24:43PM +0100, Ga=EBtan Rivet wrot= e: > > > > > > > > Hello Matan, Neil, > > > > > > > > > > > > > > > > I like the port ownership concept. I think it is needed to > > > > > > > > clarify some operations and should be useful to several > subsystems. > > > > > > > > > > > > > > > > This patch could certainly be sub-divided however, and > > > > > > > > your current > > > > > > > > 1/5 should probably come after this one. > > > > > > > > > > > > > > > > Some comments inline. > > > > > > > > > > > > > > > > On Thu, Nov 30, 2017 at 07:36:11AM -0500, Neil Horman wrote= : > > > > > > > > > On Tue, Nov 28, 2017 at 11:57:58AM +0000, Matan Azrad > wrote: > > > > > > > > > > The ownership of a port is implicit in DPDK. > > > > > > > > > > Making it explicit is better from the next reasons: > > > > > > > > > > 1. It may be convenient for multi-process applications > > > > > > > > > > to know > > > > > which > > > > > > > > > > process is in charge of a port. > > > > > > > > > > 2. A library could work on top of a port. > > > > > > > > > > 3. A port can work on top of another port. > > > > > > > > > > > > > > > > > > > > Also in the fail-safe case, an issue has been met in te= stpmd. > > > > > > > > > > We need to check that the user is not trying to use a > > > > > > > > > > port which is already managed by fail-safe. > > > > > > > > > > > > > > > > > > > > Add ownership mechanism to DPDK Ethernet devices to > > > > > > > > > > avoid multiple management of a device by different DPDK > entities. > > > > > > > > > > > > > > > > > > > > A port owner is built from owner id(number) and owner > > > > > > > > > > name(string) while the owner id must be unique to > > > > > > > > > > distinguish between two identical entity instances and > > > > > > > > > > the owner name can be > > > > > any name. > > > > > > > > > > The name helps to logically recognize the owner by > > > > > > > > > > different DPDK entities and allows easy debug. > > > > > > > > > > Each DPDK entity can allocate an owner unique > > > > > > > > > > identifier and can use it and its preferred name to > > > > > > > > > > owns valid ethdev > > > ports. > > > > > > > > > > Each DPDK entity can get any port owner status to > > > > > > > > > > decide if it can manage the port or not. > > > > > > > > > > > > > > > > > > > > The current ethdev internal port management is not > > > > > > > > > > affected by this feature. > > > > > > > > > > > > > > > > > > > > > > > > > > The internal port management is not affected, but the > > > > > > > > external interface is, however. In order to respect port > > > > > > > > ownership, applications are forced to modify their port > > > > > > > > iterator, as shown by your > > > > > > > testpmd patch. > > > > > > > > > > > > > > > > I think it would be better to modify the current > > > > > > > > RTE_ETH_FOREACH_DEV to call > RTE_FOREACH_DEV_OWNED_BY, > > > and > > > > > > > > introduce a default owner that would represent the > > > > > > > > application itself (probably with the ID 0 and an owner > > > > > > > > string ""). Only with specific additional configuration > > > > > > > > should this default subset of ethdev be > > > > > divided. > > > > > > > > > > > > > > > > This would make this evolution seamless for applications, > > > > > > > > at no cost to the complexity of the design. > > > > > > > > > > > > > > > > > > Signed-off-by: Matan Azrad > > > > > > > > > > > > > > > > > > > > > > > > > > > This seems fairly racy. What if one thread attempts to > > > > > > > > > set ownership on a port, while another is checking it on > > > > > > > > > another cpu in parallel. It doesn't seem like it will pr= otect > against that at all. > > > > > > > > > It also doesn't protect against the possibility of > > > > > > > > > multiple threads attempting to poll for rx in parallel, > > > > > > > > > which I think was part of Thomas's origional statement > > > > > > > > > regarding port ownership (he noted that the lockless > > > > > > > > > design implied only a single thread should be allowed to > > > > > > > > > poll > > > > > > > for receive or make configuration changes at a time. > > > > > > > > > > > > > > > > > > Neil > > > > > > > > > > > > > > > > > > > > > > > > > Isn't this race already there for any configuration > > > > > > > > operation / polling function? The DPDK arch is expecting > > > > > > > > applications to solve > > > it. > > > > > > > > Why should port ownership be designed differently from > > > > > > > > other DPDK > > > > > > > components? > > > > > > > > > > > > > > > Yes, but that doesn't mean it should exist in purpituity, > > > > > > > nor does it mean that your new api should contain it as well. > > > > > > > > > > > > > > > Embedding checks for port ownership within operations will > > > > > > > > force everyone to bear their costs, even those not > > > > > > > > interested in using this API. These checks should be kept > > > > > > > > outside, within the entity claiming ownership of the port, > > > > > > > > in the form of using the proper port iterator IMO. > > > > > > > > > > > > > > > No. At the very least, you need to make the API itself exclu= sive. > > > > > > > That is to say, you should at least ensure that a port > > > > > > > ownership get call doesn't race with a port ownership set > > > > > > > call. It seems rediculous to just leave that sort of locking= as an > exercize to the user. > > > > > > > > > > > > > > Neil > > > > > > > > > > > > > Neil, > > > > > > As Thomas mentioned, a DPDK port is designed to be managed by > > > > > > only one thread (or synchronized DPDK entity). > > > > > > So all the port management includes port ownership shouldn't > > > > > > be synchronized, i.e. locks are not needed. > > > > > > If some application want to do dual thread port management, > > > > > > the responsibility to synchronize the port ownership or any > > > > > > other port management is on this application. > > > > > > Port ownership doesn't come to allow synchronized management > > > > > > of the port by two DPDK entities in parallel, it is just nice > > > > > > way to answer next > > > > > questions: > > > > > > 1. Is the port already owned by some DPDK entity(in early > > > > > > control > > > > > path)? > > > > > > 2. If yes, Who is the owner? > > > > > > If the answer to the first question is no, the current entity > > > > > > can take the ownership without any lock(1 thread). > > > > > > If the answer to the first question is yes, you can recognize > > > > > > the owner and take decisions accordingly, sometimes you can > > > > > > decide to use the port because you logically know what the > > > > > > current owner does and you can be logically synchronized with > > > > > > it, sometimes you can just leave this port because you have not= any > deal with this owner. > > > > > > > > > > If the goal is just to have an ability to recognize is that > > > > > device is managed by another device (failsafe, bonding, etc.), > > > > > then I think all we need is a pointer to rte_eth_dev_data of the > > > > > owner (NULL > > > would mean no owner). > > > > > > > > I think string is better than a pointer from the next reasons: > > > > 1. It is more human friendly than pointers for debug and printing. > > > > 2. it is flexible and allows to forward logical owner message to > > > > other DPDK > > > entities. > > > > > > > > > Also I think if we'd like to introduce that mechanism, then it > > > > > needs to be > > > > > - mandatory (control API just don't allow changes to the device > > > > > configuration if caller is not an owner). > > > > > > > > But what if 2 DPDK entities should manage the same port \ using it > > > > and they > > > are synchronized? > > > > > > > > > - transparent to the user (no API changes). > > > > > > > > For now, there is not API change but new suggested API to use for > > > > port > > > iteration. > > > > > > > > > - set/get owner ops need to be atomic if we want this mechanism > > > > > to be usable for MP. > > > > > > > > But also without atomic this mechanism is usable in MP. > > > > For example: > > > > PRIMARY application can set its owner with string "primary A". > > > > SECONDARY process (which attach to the ports only after the > > > > primary > > > created them )is not allowed to set owner(As you can see in the > > > code) but it can read the owner string and see that the port owner > > > is the primary application. > > > > The "A" can just sign specific port type to the SECONDARY that > > > > this port > > > works with logic A which means, for example, primary should send the > > > packets and secondary should receive the packets. > > > > > > > But thats just the point, the operations that you are describing are > > > not atomic at all. If the primary process is interrupted during its > > > setting of a ports owner ship after its read the current owner > > > field, its entirely possible for the secondary proces to set the > > > owner as itself, and then have the primary process set it back. > > > Without locking, its just broken. I know that the dpdk likes to say > > > its lockless, because it has no locks, but here we're just kicking th= e can > down the road, by making the application add the locks for the library. > > > > > > Neil > > > > > As I wrote before and as is in the code you can understand that seconda= ry > process should not take ownership of ports. > But you allow for it, and if you do, you should write your api to be safe= for > such opperations. Please look at the code again, secondary process cannot take ownership, I d= on't allow it! Actually, I think it must not be like that as no limitation for that in any= other ethdev configurations. > > Any port configuration (for example port creation and release) is not > internally synchronized between the primary to secondary processes so I > don't see any reason to synchronize port ownership. > Yes, and I've never agreed with that design point either, because the fac= t of > the matter is that one of two things must be true in relation to port > configuration: >=20 > 1) Either multiple processes will attempt to read/change port > configuration/ownership >=20 > or >=20 > 2) port ownership is implicitly given to a single task (be it a primary o= r > secondary task), and said ownership is therefore implicitly known by the > application >=20 > Either situation may be true depending on the details of the application = being > built, but regardless, if (2) is true, then this api isn't really needed = at all, > because the application implicitly has been designed to have a port be > owned by a given task.=20 Here I think you miss something, the port ownership is not mainly for proce= ss DPDK entities, The entity can be also PMD, library, application in same process. For MP it is nice to have, the secondary just can see the primary owners an= d take decision accordingly (please read my answer to Konstatin above).=20 If (1) is true, then all the locking required to access > the data of port ownership needs to be adhered to. >=20 And all the port configurations! I think it is behind to this thread. > I understand that you are taking Thomas' words to mean that all paths are > lockless in the DPDK, and that is true as a statement of fact (in that th= e DPDK > doesn't synchronize access to internal data). What it does do, is leave = that > locking as an exercize for the downstream consumer of the library. While= I > don't agree with it, I can see that such an argument is valid for hot pat= hs such > as transmission and reception where you may perhaps want to minimize > your locking by assigning a single task to do that work, but port configu= ration > and ownership isn't a hot path. If you mean to allow potentially multipl= e > tasks to access configuration (including port ownership), be it frequentl= y or > just occasionaly, why are you making a downstream developer recognize the > need for locking here outside the library? It just seems like very bad g= eneral > practice to me. >=20 > > If the primary-secondary process want to manage(configure) same port in > same time, they must be synchronized by the applications, so this is the = case > in port ownership too (actually I don't think this synchronization is rea= listic > because many configurations of the port are not in shared memory). > Yes, it is the case, my question is, why? We're not talking about a time > sensitive execution path here. And by avoiding it you're just making wor= k > that has to be repeated by every downstream consumer. I think you suggest to make all the ethdev configuration race safe, it is b= ehind to this thread. Current ethdev implementation leave the race management to applications, so= port ownership as any other port configurations should be designed in the = same method. >=20 > Neil