From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR03-AM5-obe.outbound.protection.outlook.com (mail-eopbgr30047.outbound.protection.outlook.com [40.107.3.47]) by dpdk.org (Postfix) with ESMTP id B21721B2E7 for ; Thu, 21 Dec 2017 22:57:45 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=r7QdQcXA2SY855KVOsr6yvb0toUH9uZVTpdlAviM/q0=; b=wVEV6UEnonCGpdvTBuVWbrGCiPrp8Y6ATFmOg4BxbGmJmDbqaJOieoPOEmWDLhkqw9csO+hVU7s96yDVy71OuOJbjwwk0VJFgU5EQOiKrw6jSMsdn5pAxKR8nh6eLd0kWzEAlo0B9e5sowayrRnQGZmfgSv5hVUFEST2wZIAiKI= Received: from HE1PR0502MB3659.eurprd05.prod.outlook.com (10.167.127.17) by HE1PR0502MB3660.eurprd05.prod.outlook.com (10.167.127.18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.323.15; Thu, 21 Dec 2017 21:57:43 +0000 Received: from HE1PR0502MB3659.eurprd05.prod.outlook.com ([fe80::adf6:87de:2761:8e4]) by HE1PR0502MB3659.eurprd05.prod.outlook.com ([fe80::adf6:87de:2761:8e4%13]) with mapi id 15.20.0345.013; Thu, 21 Dec 2017 21:57:43 +0000 From: Matan Azrad To: Neil Horman CC: Thomas Monjalon , "dev@dpdk.org" , Bruce Richardson , "Ananyev, Konstantin" , =?iso-8859-1?Q?Ga=EBtan_Rivet?= , "Wu, Jingjing" Thread-Topic: [dpdk-dev] [PATCH 2/5] ethdev: add port ownership Thread-Index: AQHTaEBDeXaioloFskiHncvIK1YNTKMs3xqAgAANj4CAAX1kAIAC1eJggAA+JgCAABoCUIAByaUAgAAcouCAAFA1gIAAeA4QgABKFgCABNAHAIAAD8IAgBS7KwCAAAoiAIAADZrAgAAcqgCAAAb+4A== Date: Thu, 21 Dec 2017 21:57:43 +0000 Message-ID: References: <20171130123611.GA20914@hmswarspite.think-freely.org> <5212147.QN8ImyqEg2@xps> <20171208123142.GA6955@hmswarspite.think-freely.org> <1567916.dnd6Z652YM@xps> <20171221174304.GB23958@hmswarspite.think-freely.org> <20171221201420.GC23958@hmswarspite.think-freely.org> In-Reply-To: <20171221201420.GC23958@hmswarspite.think-freely.org> Accept-Language: en-US, he-IL Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [85.64.136.190] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; HE1PR0502MB3660; 6:oVhpQ2PSszUp4D+LmGb9p6phsQKHSSugXO8ChJxWuXqJkZB7uudkD178VmgYx9hGQwdg+V4YQCPDcGKKo5UBmMFHHgPFh2UYBsLG9q2Oftp8ABPtiDV90nSCHtY9gPPrjQxNYbrSWA552Pzhq/3SDB5lEo3gptdW+kZsjBz9cV5NyUDetpZz8Xl+q5/S8o6q3HhCxH2oezAGfcUb2vhamQ/p/YAvTeACO2VDh85cZ8dSf7fRohjGfsz5+44hnW7/6xvFySV5xiIyLSPN47n3ukO3I51Ouf1flVlh7pDTZ8UU3Dv5gyMNmpoB5++hs8ix5vvBCBGFTbO2N7m9HGO0utmgtEOp/ZOxvwSqJcgbKUg=; 5:cC1ShcRsG0ggunBBfngk2RnRtMxP8Uxc2lSnZXAMzhsE/K8WZ2pZkinxrUg5MD5+iBU9zjja2Q+7iFTKSQHwgiakphu90Y7278tp0nm8PX3x454dF3ZkFXf01jSWkZcs0pDxFOs/QKc2hQnw3Ar14rdhLZ3gBg4eNBj9SvbQNQw=; 24:Vx3hT+4rVRylBgCAMZ4X2HP662o7mK1x5LS/Fh+hqbAoe3k+Rujs5jmG4Do/IpIfpPpPZQYs84QgZ9I+kkYCOrcGPxmpMJhO/OhxbhiztDs=; 7:4P/3Ry3SfJx2mvkxyA5c27eAR/CVnS16jttlZijdBp0APL+Qn0JyFyJBXxtL74D0XmdrRqF1nVmI/ef9yOn8OqMhMEslIaooZx2CyLaetKCT0CFBdb4lNKwY49j29iGn+FQh7zlsloctNIcwZkht9b2tJfL99xXsbMdpvCSYYCo566IamJPhoKJGzUVpce6zh91Qp2puPH67UHi/tLFP5SNKueP97JQ9VRsGKzWrqyH4mKgOSAFkahmxF4wmo5+q x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: e35ed2aa-8639-4504-049b-08d548bddcd3 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(48565401081)(5600026)(4604075)(3008031)(4534020)(4602075)(4627115)(201703031133081)(201702281549075)(2017052603307)(7153060); SRVR:HE1PR0502MB3660; x-ms-traffictypediagnostic: HE1PR0502MB3660: authentication-results: spf=none (sender IP is ) smtp.mailfrom=matan@mellanox.com; x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(60795455431006)(100405760836317)(228905959029699); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040470)(2401047)(8121501046)(5005006)(3002001)(10201501046)(93006095)(93001095)(3231023)(944501032)(6055026)(6041268)(20161123562045)(20161123564045)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(6072148)(201708071742011); SRVR:HE1PR0502MB3660; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:HE1PR0502MB3660; x-forefront-prvs: 0528942FD8 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(376002)(39380400002)(39860400002)(396003)(346002)(24454002)(199004)(189003)(13464003)(51444003)(76104003)(53936002)(7696005)(316002)(8676002)(2900100001)(106356001)(74316002)(3280700002)(6916009)(6246003)(5660300001)(2950100002)(3660700001)(2906002)(68736007)(99286004)(305945005)(7736002)(97736004)(102836004)(86362001)(76176011)(54906003)(3846002)(105586002)(8936002)(59450400001)(53546011)(33656002)(81156014)(229853002)(14454004)(6116002)(55016002)(478600001)(9686003)(25786009)(66066001)(4326008)(81166006)(6436002)(5250100002)(93886005)(6506007); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR0502MB3660; H:HE1PR0502MB3659.eurprd05.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: e35ed2aa-8639-4504-049b-08d548bddcd3 X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Dec 2017 21:57:43.2222 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0502MB3660 Subject: Re: [dpdk-dev] [PATCH 2/5] ethdev: add port ownership X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 21:57:45 -0000 > -----Original Message----- > From: Neil Horman [mailto:nhorman@tuxdriver.com] > Sent: Thursday, December 21, 2017 10:14 PM > To: Matan Azrad > Cc: Thomas Monjalon ; dev@dpdk.org; Bruce > Richardson ; Ananyev, Konstantin > ; Ga=EBtan Rivet ; > Wu, Jingjing > Subject: Re: [dpdk-dev] [PATCH 2/5] ethdev: add port ownership >=20 > On Thu, Dec 21, 2017 at 07:37:06PM +0000, Matan Azrad wrote: > > Hi > > > > > > > > I think we need to clearly describe what is the tread-safety > > > > > > policy in DPDK (especially in ethdev as a first example). > > > > > > Let's start with obvious things: > > > > > > > > > > > > 1/ A queue is not protected for races with multiple Rx or Tx > > > > > > - no planned change because of performance > > > purpose > > > > > > 2/ The list of devices is racy > > > > > > - to be fixed with atomics > > > > > > 3/ The configuration of different devices is thread-safe > > > > > > - the configurations are different per-device > > > > > > 4/ The configuration of a given device is racy > > > > > > - can be managed by the owner of the device > > > > > > 5/ The device ownership is racy > > > > > > - to be fixed with atomics > > > > > > > > > > > > What am I missing? > > > > > > > > > > Thank you Thomas for this order. > > Actually the port ownership is a good opportunity to redefine the > > synchronization rules in ethdev :) > > > > > > > There is fan out to consider here: > > > > > > > > > > 1) Is device configuration racy with ownership? That is to say, > > > > > can I change ownership of a device safely while another thread > > > > > that currently owns it modifies its configuration? > > > > > > > > If an entity steals ownership to another one, either it is agreed > > > > earlier, or it is done by a central authority. > > > > When it is acked that ownership can be moved, there should not be > > > > any configuration in progress. > > > > So it is more a communication issue than a race. > > > > > > > But if thats the case (specifically that mutual exclusion between > > > port ownership and configuration is an exercize left to an > > > application developer), then port ownership itself is largely > > > meaningless within the dpdk, because the notion of who owns the port > > > needs to be codified within the application anyway. > > > > > > > Bruce, As I understand it, only the dpdk entity who took ownership of a > port successfully can configure the device by default, if other dpdk enti= ties > want to configure it too they must to be synchronized with the port owner > while it is not recommended after the port ownership integration. > > > Can you clarify what you mean by "it is not recommended after the port > ownership integration"? Sure, The new defining of ethdev synchronization doesn't recommend to manage a po= rt by 2 different dpdk entities, it can be done but not recommended. =20 > I think there is consensus that the port owner must > be the only entitiy to operate on a port (be that configuration/frame rx/= tx, or > some other operation). Your question above caused me to think that you don't understand it, How ca= n someone who is not the port owner to change the port owner? Changing the port owner, like port configuration and port release must be d= one by the owner itself except the case that there is no owner to the port. See the API rte_eth_dev_owner_remove. > Multithreaded operation on a port always means > some level of synchronization between application threads and the dpdk > library, Yes. >but I'm not sure why that would be different if we introduced a more > concrete notion of port ownership via a new library. > What do you mean by "new library"?, port is an ethdev instance and should b= e managed by ethdev. > > So, for example, if the dpdk entity is an application, the applicatio= n should >> take ownership of the port and manage the synchronization of this port >> configuration between the application threads and its EAL host thread >> callbacks, no other dpdk entity should configure the same port because t= hey >> should fail when they try to take ownership of the same port too. > Well, failing is one good approach, yes, blocking on port relenquishment > could be another. I'd recommend an API with the following interface: >=20 > rte_port_ownership_claim(int port_id) - blocks execution of the calling > thread until the previous owner releases ownership, then claims it and > returns >=20 > rte_port_ownership_release(int port_id) - releases ownership of port, or > returns error if the port was not owned by this execution context > > rte_port_owernship_try_claim(int port_id) - same as > rte_port_ownership_claim, but fails if the port is already owned. >=20 > That would give the option for both semantics. I think the current APIs are better because of the next reasons: - It defines well who is the owner. - The owner structure includes string to allow better debug and printing fo= r humans.=20 Did you read it? I can add there an API that wait until the port ownership is released as yo= u suggested in V2. =20 > > Each dpdk entity which wants to take ownership must to be able to > >synchronize the port configuration in its level. > Can you elaborate on what you mean by level here? Are you envisioning a > scheme in which multiple execution contexts might own a port for various > non-conflicting purposes? =20 Sure, 1) Application with 2 threads wanting to configure the same port: level =3D application code. =09 a. The main thread should create owner identifier(rte_eth_dev_owner_new). b. The main thread should take the port ownership(rte_eth_dev_owner_set). c. Synchronization between the two threads should be done for the conflict= ed configurations by the application. d. when the application finishes the port usage it should release the owne= r(rte_eth_dev_owner_remove). 2) Fail-safe PMD manages 2 sub-devices (2 ports) and uses alarm for hotplug= detections which can configure the 2 ports(by the host thread). Level =3D fail-safe code. a. Application starts the eal and the fail-safe driver probing function is= called. b. Fail-safe probes the 2 sub-devices(2 ports are created) and takes owner= ship for them. c. Failsafe creates itself port and leaves it ownerless.=20 d. Failsafe starts the hotplug alarm mechanism. e. Application tries to take ownership for all ports and success only for = failsafe port. f. Application start to configure the failsafe port asynchronously to fail= safe hotplug alarm. g. Failsafe must use synchronization between failsafe alarm callback code = and failsafe configuration APIs called by the application because they both= try to configure the same sub-devices ports. h. When fail-safe finishes with the two sub devices it should release the = ports owner. > > > > > > > > > > 2) Is device configuration racy with device addition/removal? > > > > > That is to say, can one thread remove a device while another > configures it. > > > > > > > > I think it is the same as two threads configuring the same device > > > > (item 4/ above). It can be managed with port ownership. > > > > > > > Only if you assert that application is required to have the owning > > > port be responsible for the ports deletion, which we can say, but > > > that leads to the issue above again. > > > > > > > > As Thomas said in item 2 the port creation must be synchronized by ethd= ev > and we need to add it there. > > I think it is obvious that port removal must to be done only by the por= t > owner. > > > You say that, but its obvious to you as a developer who has looked > extensively at the code. It may well be less so to a consumer who is not= an > active member of the community, for instance one who obtains the dpdk via > pre-built package. > Yes I can understand, but new rules should be documented and be adjusted ea= sy easy by the customers, no? The old way to sync configuration still exists. =20 > > > > I think we need to add synchronization for port ownership management in > this patch V2 and add port creation synchronization in ethdev in separate > patch to fill the new rules Thomas suggested. > I think that makes sense, yes. >=20 > Neil >=20 > > > > What do you think? > > > > > > > There are probably many subsystem interactions that need to be > > > addressed here. > > > > > > > > > > Neil > > > > > > > > > > > I am also wondering whether the device ownership can be a > > > > > > separate library used in several device class interfaces? > > > > > > > > > > > > > >