From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-AM5-obe.outbound.protection.outlook.com (mail-eopbgr00070.outbound.protection.outlook.com [40.107.0.70]) by dpdk.org (Postfix) with ESMTP id 5D7ED1B329 for ; Thu, 18 Jan 2018 18:20:33 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=ACL5BZVSLn8tNoSHiMpN8kZAOlrixXJxtgVZryKGELo=; b=TL2SdzCETNYKkN05TYOnE5qlNiVmyJJWaUGYYP+im4z63/KspBVOdLH/Rhe5zmw5cvxiUrZpa8FFFrHGyMY8F9F4NTQEEMdiAJa29t7cbiLJkYDUQf1Mh8Hs4VpZrOcS2a8iwsxDMrFQk2VPOTq9GJfWpmBy41Nfykc98GmxrBY= Received: from AM6PR0502MB3797.eurprd05.prod.outlook.com (52.133.21.26) by AM6PR0502MB3992.eurprd05.prod.outlook.com (52.133.30.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.428.17; Thu, 18 Jan 2018 17:20:31 +0000 Received: from AM6PR0502MB3797.eurprd05.prod.outlook.com ([fe80::6c28:c6b3:de94:a733]) by AM6PR0502MB3797.eurprd05.prod.outlook.com ([fe80::6c28:c6b3:de94:a733%13]) with mapi id 15.20.0428.014; Thu, 18 Jan 2018 17:20:31 +0000 From: Matan Azrad To: Neil Horman CC: "Ananyev, Konstantin" , Thomas Monjalon , Gaetan Rivet , "Wu, Jingjing" , "dev@dpdk.org" , "Richardson, Bruce" Thread-Topic: [PATCH v2 2/6] ethdev: add port ownership Thread-Index: AQHTihf/M9xg8LYorUSRFqZtTc27hqNtNdVQgAFomACAAAOCwIAAuwQAgABvY+CABQwJgIAAEk3ggABinoCAANUIAIAAxQGAgAAGCnCAAQnKAIAAAmNwgAApSwCAADKGAIABUdOAgAAMD/CAADKfgIAAAbBA Date: Thu, 18 Jan 2018 17:20:31 +0000 Message-ID: References: <2601191342CEEE43887BDE71AB9772588627DE30@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772588627E954@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772588627EE60@irsmsx105.ger.corp.intel.com> <20180117140020.GA5432@hmswarspite.think-freely.org> <2601191342CEEE43887BDE71AB9772588627F0E9@irsmsx105.ger.corp.intel.com> <20180118131017.GA1622@hmswarspite.think-freely.org> <20180118165436.GD1622@hmswarspite.think-freely.org> In-Reply-To: <20180118165436.GD1622@hmswarspite.think-freely.org> Accept-Language: en-US, he-IL Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=matan@mellanox.com; x-originating-ip: [193.47.165.251] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; AM6PR0502MB3992; 7:273afnEXQsY/Sof3sgY/u/On5mPihK9EBOfA4dq1RU8GOWan7l5bQbF6DQE4aN3wgveojV3mCadtO3dpHzygLURFSy7UwUlBIDvjFsGOKopCRjy1vluay/+tAybTTaqKetevQRRj5vv5Le2VkbiWuvvStvbHYJD71eOUBonuIgZcQAJ27wkxSM9n3p7473ypTrhOJmnFGFC/7FbDWP8tvH+/6zgC/LQ6H+I0u5bcx6YgnKNpdATumpHgGqNBQL7i x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 0a2eec8c-4890-49fc-8d10-08d55e97c705 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4604075)(3008032)(48565401081)(2017052603307)(7153060)(7193020); SRVR:AM6PR0502MB3992; x-ms-traffictypediagnostic: AM6PR0502MB3992: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(60795455431006)(228905959029699); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040470)(2401047)(5005006)(8121501046)(10201501046)(3231023)(2400064)(944501161)(3002001)(93006095)(93001095)(6055026)(6041268)(20161123564045)(20161123560045)(20161123558120)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:AM6PR0502MB3992; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:AM6PR0502MB3992; x-forefront-prvs: 05568D1FF7 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(979002)(366004)(396003)(39380400002)(39860400002)(376002)(346002)(189003)(199004)(13464003)(51444003)(59450400001)(102836004)(6506007)(53546011)(105586002)(5660300001)(6436002)(99286004)(68736007)(2906002)(2900100001)(93886005)(7696005)(106356001)(33656002)(305945005)(5250100002)(8936002)(54906003)(76176011)(6116002)(7736002)(53936002)(3846002)(25786009)(9686003)(4326008)(55016002)(14454004)(3280700002)(3660700001)(74316002)(229853002)(66066001)(26005)(97736004)(2950100002)(316002)(81156014)(478600001)(81166006)(86362001)(6246003)(8676002)(6916009)(969003)(989001)(999001)(1009001)(1019001); DIR:OUT; SFP:1101; SCL:1; SRVR:AM6PR0502MB3992; H:AM6PR0502MB3797.eurprd05.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: rLJzUnujcXl0FIrtxJmzFU2F1cEqjcqw7oC9YEyrSeTsROKBjSMWQ7bhDzKIXOaTewh1GUdSNhGSrwfxmcML+Q== spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0a2eec8c-4890-49fc-8d10-08d55e97c705 X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Jan 2018 17:20:31.3801 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR0502MB3992 Subject: Re: [dpdk-dev] [PATCH v2 2/6] ethdev: add port ownership X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Jan 2018 17:20:33 -0000 Hi Neil From: Neil Horman, Thursday, January 18, 2018 6:55 PM > On Thu, Jan 18, 2018 at 02:00:23PM +0000, Matan Azrad wrote: > > Hi Neil > > > > From: Neil Horman, Thursday, January 18, 2018 3:10 PM > > > On Wed, Jan 17, 2018 at 05:01:10PM +0000, Ananyev, Konstantin wrote: > > > > > > > > > > > > > -----Original Message----- > > > > > From: Neil Horman [mailto:nhorman@tuxdriver.com] > > > > > Sent: Wednesday, January 17, 2018 2:00 PM > > > > > To: Matan Azrad > > > > > Cc: Ananyev, Konstantin ; Thomas > > > > > Monjalon ; Gaetan Rivet > > > > > ; Wu, Jingjing ; > > > > > dev@dpdk.org; Richardson, Bruce > > > > > Subject: Re: [PATCH v2 2/6] ethdev: add port ownership > > > > > > > > > > On Wed, Jan 17, 2018 at 12:05:42PM +0000, Matan Azrad wrote: > > > > > > > > > > > > Hi Konstantin > > > > > > From: Ananyev, Konstantin, Sent: Wednesday, January 17, 2018 > > > > > > 1:24 PM > > > > > > > Hi Matan, > > > > > > > > > > > > > > > Hi Konstantin > > > > > > > > > > > > > > > > From: Ananyev, Konstantin, Tuesday, January 16, 2018 9:11 > > > > > > > > PM > > > > > > > > > Hi Matan, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Konstantin > > > > > > > > > > From: Ananyev, Konstantin, Monday, January 15, 2018 > > > > > > > > > > 8:44 PM > > > > > > > > > > > Hi Matan, > > > > > > > > > > > > Hi Konstantin > > > > > > > > > > > > From: Ananyev, Konstantin, Monday, January 15, > > > > > > > > > > > > 2018 > > > > > > > > > > > > 1:45 PM > > > > > > > > > > > > > Hi Matan, > > > > > > > > > > > > > > Hi Konstantin > > > > > > > > > > > > > > From: Ananyev, Konstantin, Friday, January 12, > > > > > > > > > > > > > > 2018 2:02 AM > > > > > > > > > > > > > > > Hi Matan, > > > > > > > > > > > > > > > > Hi Konstantin > > > > > > > > > > > > > > > > From: Ananyev, Konstantin, Thursday, > > > > > > > > > > > > > > > > January 11, 2018 > > > > > > > > > > > > > > > > 2:40 PM > > > > > > > > > > > > > > > > > Hi Matan, > > > > > > > > > > > > > > > > > > Hi Konstantin > > > > > > > > > > > > > > > > > > From: Ananyev, Konstantin, Wednesday, > > > > > > > > > > > > > > > > > > January 10, > > > > > > > > > > > > > > > > > > 2018 > > > > > > > > > > > > > > > > > > 3:36 PM > > > > > > > > > > > > > > > > > > > Hi Matan, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It is good to see that now > > > > > > > > > > > > > > > > > > > scanning/updating rte_eth_dev_data[] > > > > > > > > > > > > > > > > > > > is lock protected, but it might be > > > > > > > > > > > > > > > > > > > not very plausible to protect both > > > > > > > > > > > > > > > > > > > data[] and next_owner_id using the > > > > > > > > > > > > > same lock. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I guess you mean to the owner > > > > > > > > > > > > > > > > > > structure in > > > > > > > > > > > > > rte_eth_dev_data[port_id]. > > > > > > > > > > > > > > > > > > The next_owner_id is read by ownership > > > > > > > > > > > > > > > > > > APIs(for owner validation), so it > > > > > > > > > > > > > > > > > makes sense to use the same lock. > > > > > > > > > > > > > > > > > > Actually, why not? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well to me next_owner_id and > > > > > > > > > > > > > > > > > rte_eth_dev_data[] are not directly > > > > > > > > > > > > > > > related. > > > > > > > > > > > > > > > > > You may create new owner_id but it > > > > > > > > > > > > > > > > > doesn't mean you would update > > > > > > > > > > > > > > > > > rte_eth_dev_data[] > > > immediately. > > > > > > > > > > > > > > > > > And visa-versa - you might just want to > > > > > > > > > > > > > > > > > update rte_eth_dev_data[].name or > .owner_id. > > > > > > > > > > > > > > > > > It is not very good coding practice to > > > > > > > > > > > > > > > > > use same lock for non-related data struct= ures. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I see the relation like next: > > > > > > > > > > > > > > > > Since the ownership mechanism > > > > > > > > > > > > > > > > synchronization is in ethdev > > > > > > > > > > > > > > > > responsibility, we must protect against > > > > > > > > > > > > > > > > user mistakes as much as we can by > > > > > > > > > > > > > > > using the same lock. > > > > > > > > > > > > > > > > So, if user try to set by invalid owner > > > > > > > > > > > > > > > > (exactly the ID which currently is > > > > > > > > > > > > > > > allocated) we can protect on it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hmm, not sure why you can't do same checking > > > > > > > > > > > > > > > with different lock or atomic variable? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The set ownership API is protected by > > > > > > > > > > > > > > ownership lock and checks the owner ID > > > > > > > > > > > > > > validity By reading the next > > > owner ID. > > > > > > > > > > > > > > So, the owner ID allocation and set API should > > > > > > > > > > > > > > use the same atomic > > > > > > > > > > > > > mechanism. > > > > > > > > > > > > > > > > > > > > > > > > > > Sure but all you are doing for checking > > > > > > > > > > > > > validity, is check that owner_id > 0 &&& > > > > > > > > > > > > > owner_id < next_ownwe_id, > > > right? > > > > > > > > > > > > > As you don't allow owner_id overlap (16/3248 > > > > > > > > > > > > > bits) you can safely do same check with just > > > atomic_get(&next_owner_id). > > > > > > > > > > > > > > > > > > > > > > > > > It will not protect it, scenario: > > > > > > > > > > > > - current next_id is X. > > > > > > > > > > > > - call set ownership of port A with owner id X by > > > > > > > > > > > > thread 0(by user > > > > > > > > > mistake). > > > > > > > > > > > > - context switch > > > > > > > > > > > > - allocate new id by thread 1 and get X and change > > > > > > > > > > > > next_id to > > > > > > > > > > > > X+1 > > > > > > > > > > > atomically. > > > > > > > > > > > > - context switch > > > > > > > > > > > > - Thread 0 validate X by atomic_read and succeed > > > > > > > > > > > > to take > > > > > > > ownership. > > > > > > > > > > > > - The system loosed the port(or will be managed by > > > > > > > > > > > > two > > > > > > > > > > > > entities) - > > > > > > > > > crash. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ok, and how using lock will protect you with such sce= nario? > > > > > > > > > > > > > > > > > > > > The owner set API validation by thread 0 should fail > > > > > > > > > > because the owner > > > > > > > > > validation is included in the protected section. > > > > > > > > > > > > > > > > > > Then your validation function would fail even if you'll > > > > > > > > > use atomic ops instead of lock. > > > > > > > > No. > > > > > > > > With atomic this specific scenario will cause the validatio= n to > pass. > > > > > > > > > > > > > > Can you explain to me how? > > > > > > > > > > > > > > rte_eth_is_valid_owner_id(uint16_t owner_id) { > > > > > > > int32_t cur_owner_id =3D > > > > > > > RTE_MIN(rte_atomic32_get(next_owner_id), > > > > > > > UINT16_MAX); > > > > > > > > > > > > > > if (owner_id =3D=3D RTE_ETH_DEV_NO_OWNER || owner > > > > > > > > cur_owner_id) { > > > > > > > RTE_LOG(ERR, EAL, "Invalid owner_id=3D%d.\n", > owner_id); > > > > > > > return 0; > > > > > > > } > > > > > > > return 1; > > > > > > > } > > > > > > > > > > > > > > Let say your next_owne_id=3D=3DX, and you invoke > > > > > > > rte_eth_is_valid_owner_id(owner_id=3DX+1) - it would fail. > > > > > > > > > > > > Explanation: > > > > > > The scenario with locks: > > > > > > next_owner_id =3D X. > > > > > > Thread 0 call to set API(with invalid owner Y=3DX) and take loc= k. > > > > > > Context switch. > > > > > > Thread 1 call to owner_new and stuck in the lock. > > > > > > Context switch. > > > > > > Thread 0 does owner id validation and failed(Y>=3DX) - unlock > > > > > > the lock and > > > return failure to the user. > > > > > > Context switch. > > > > > > Thread 1 take the lock and update X to X+1, then, unlock the lo= ck. > > > > > > Everything is OK! > > > > > > > > > > > > The same scenario with atomics: > > > > > > next_owner_id =3D X. > > > > > > Thread 0 call to set API(with invalid owner Y=3DX) and take loc= k. > > > > > > Context switch. > > > > > > Thread 1 call to owner_new and change X to X+1(atomically). > > > > > > Context switch. > > > > > > Thread 0 does owner id validation and success(Y<(atomic)X+1) - > > > > > > unlock > > > the lock and return success to the user. > > > > > > Problem! > > > > > > > > > > > > > > > > > > > > > Matan is correct here, there is no way to preform parallel set > > > > > operations using just and atomic variable here, because multiple > > > > > reads of next_owner_id need to be preformed while it is stable. > > > > > That is to say rte_eth_next_owner_id must be compared to > > > > > RTE_ETH_DEV_NO_OWNER and owner_id in > rte_eth_is_valid_owner_id. > > > If > > > > > you were to only use an atomic_read on such a variable, it could > > > > > be incremented by the owner_new function between the checks and > > > > > an invalid owner value could become valid because a third > > > > > thread incremented the next value. The state of next_owner_id > > > > > must be kept stable during any validity checks > > > > > > > > It could still be incremented between the checks - if let say > > > > different thread will invoke new_onwer_id, grab the lock update > > > > counter, release the lock - all that before the check. > > > I don't see how all of the contents of rte_eth_dev_owner_set is > > > protected under rte_eth_dev_ownership_lock, as is > rte_eth_dev_owner_new. > > > Next_owner might increment between another threads calls to > > > owner_new and owner_set, but that will just cause a transition from > > > an ownership id being valid to invalid, and thats ok, as long as > > > there is consistency in the model that enforces a single valid owner > > > at a time (in that case the subsequent caller to owner_new). > > > > > > > I'm not sure I fully understand you, but see: > > we can't protect all of the user mistakes(using the wrong owner id). > > But we are doing the maximum for it. > > > Yeah, my writing was atrocious, apologies. All I meant to say was that t= he > locking you have is ok, in that it maintains a steady state for the data = being > read during the period its being read. The fact that a given set operati= on may > fail because someone else created an ownership record is an artifact of t= he > api, not a bug in its implementation. I think we're basically in agreeme= nt on > the semantics here, but this goes to my argument about complexity (more > below). >=20 > > > > > Though this confusion does underscore my assertion I think that this > > > API is overly complicated > > > > > > > I really don't think it is complicated. - just take ownership of a port= (by > owner id allocation and set APIs) and manage the port as you want. > > > But thats not all. The determination of success or failure in claiming > ownership is largely dependent on the behavior of other threads actions, = not > a function of the state of the system at the moment ownership is requeste= d. > That is to say, if you have N threads, and they all create ownership obje= cts > identified as X, x+1, X+2...X+N, only the thread with id X+N will be able= to > claim ownership of any port, because they all will have incremented the > shared nex_id variable. Why? Each one will get its owner id according to some order(The critical se= ction is protected by spinlock). > Determination of ownership by the programmer will > have to be done via debugging, and errors will likely be transient depend= ent > on the order in which threads execute (subject to scheduling jitter). >=20 Yes. > Rather than making ownership success dependent on any data contained > within the ownership record, ownership should be entirely dependent on > the state of port ownership at the time that it was requested. That is t= o say, > port ownership should succede if and only if the port is unowned at the t= ime > that a given thread requets ownership. Yes. > Any ancilliary data regarding which > context owns the port should be exactly that, ancilliary, and have no imp= act > on weather or not the port ownership request succedes. >=20 Yes, I understand what you say - there is no deterministic state for owners= hip set success. Actually I think it will be very hard to arrive to determination in DPDK re= garding port ownership when multi-thread is in the game, Especially it depend in a lot of DPDK entities implementation.. But the current non-deterministic approach makes good order in the game.=20 > Regards > Neil >=20 > > > Neil > > > >