From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id 2ACA5199B0 for ; Wed, 17 Jan 2018 17:53:00 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Jan 2018 08:53:00 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,372,1511856000"; d="scan'208";a="10479063" Received: from irsmsx154.ger.corp.intel.com ([163.33.192.96]) by orsmga007.jf.intel.com with ESMTP; 17 Jan 2018 08:52:58 -0800 Received: from irsmsx156.ger.corp.intel.com (10.108.20.68) by IRSMSX154.ger.corp.intel.com (163.33.192.96) with Microsoft SMTP Server (TLS) id 14.3.319.2; Wed, 17 Jan 2018 16:52:57 +0000 Received: from irsmsx105.ger.corp.intel.com ([169.254.7.236]) by IRSMSX156.ger.corp.intel.com ([169.254.3.33]) with mapi id 14.03.0319.002; Wed, 17 Jan 2018 16:52:57 +0000 From: "Ananyev, Konstantin" To: Matan Azrad , Thomas Monjalon , Gaetan Rivet , "Wu, Jingjing" CC: "dev@dpdk.org" , Neil Horman , "Richardson, Bruce" Thread-Topic: [PATCH v2 2/6] ethdev: add port ownership Thread-Index: AQHTh5xreoY7ZC2VXUKPLFrmK4AmzKNtEwCwgABGVwCAATRvEIAAOkGAgACPrvCAAIW5gIAE/GJggAAbIoCAAD5+YIAA/qMAgACx27CAAB8nAIAA79jQgAAU2ACAAAwxIIAABgqAgAATZLA= Date: Wed, 17 Jan 2018 16:52:56 +0000 Message-ID: <2601191342CEEE43887BDE71AB9772588627F076@irsmsx105.ger.corp.intel.com> References: <1511870281-15282-1-git-send-email-matan@mellanox.com> <1515318351-4756-1-git-send-email-matan@mellanox.com> <1515318351-4756-3-git-send-email-matan@mellanox.com> <2601191342CEEE43887BDE71AB97725880E3B9D6@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772588627B12A@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772588627CCB0@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772588627DC25@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772588627DE30@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772588627E954@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772588627EE60@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772588627EEDA@irsmsx105.ger.corp.intel.com> In-Reply-To: Accept-Language: en-IE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiZWZlODU2Y2MtYjNjOS00MWIzLWE5NmItOTEzZGRhZTIzYWFlIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE2LjUuOS4zIiwiVHJ1c3RlZExhYmVsSGFzaCI6Ik9uS2pKNjRqcFwvQ1VLZDRYaUxBMVhuRGpkN1lNdnp5Vm1TTGNoSjVjVEw4PSJ9 x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.0.0.116 dlp-reaction: no-action x-originating-ip: [163.33.239.181] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v2 2/6] ethdev: add port ownership X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Jan 2018 16:53:01 -0000 Hi Matan, >=20 > Hi Konstantin > From: Ananyev, Konstantin, Wednesday, January 17, 2018 2:55 PM > > > > > > > > > Hi Konstantin > > > From: Ananyev, Konstantin, Sent: Wednesday, January 17, 2018 1:24 PM > > > > Hi Matan, > > > > > > > > > Hi Konstantin > > > > > > > > > > From: Ananyev, Konstantin, Tuesday, January 16, 2018 9:11 PM > > > > > > Hi Matan, > > > > > > > > > > > > > > > > > > > > Hi Konstantin > > > > > > > From: Ananyev, Konstantin, Monday, January 15, 2018 8:44 PM > > > > > > > > Hi Matan, > > > > > > > > > Hi Konstantin > > > > > > > > > From: Ananyev, Konstantin, Monday, January 15, 2018 1:45 > > > > > > > > > PM > > > > > > > > > > Hi Matan, > > > > > > > > > > > Hi Konstantin > > > > > > > > > > > From: Ananyev, Konstantin, Friday, January 12, 2018 > > > > > > > > > > > 2:02 AM > > > > > > > > > > > > Hi Matan, > > > > > > > > > > > > > Hi Konstantin > > > > > > > > > > > > > From: Ananyev, Konstantin, Thursday, January 11, > > > > > > > > > > > > > 2018 > > > > > > > > > > > > > 2:40 PM > > > > > > > > > > > > > > Hi Matan, > > > > > > > > > > > > > > > Hi Konstantin > > > > > > > > > > > > > > > From: Ananyev, Konstantin, Wednesday, January > > > > > > > > > > > > > > > 10, > > > > > > > > > > > > > > > 2018 > > > > > > > > > > > > > > > 3:36 PM > > > > > > > > > > > > > > > > Hi Matan, > > > > > > > > > > > > > > > > > > > > > > > It is good to see that now scanning/updatin= g > > > > > > > > > > > > > > > > rte_eth_dev_data[] is lock protected, but i= t > > > > > > > > > > > > > > > > might be not very plausible to protect both > > > > > > > > > > > > > > > > data[] and next_owner_id using the > > > > > > > > > > same lock. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I guess you mean to the owner structure in > > > > > > > > > > rte_eth_dev_data[port_id]. > > > > > > > > > > > > > > > The next_owner_id is read by ownership > > > > > > > > > > > > > > > APIs(for owner validation), so it > > > > > > > > > > > > > > makes sense to use the same lock. > > > > > > > > > > > > > > > Actually, why not? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well to me next_owner_id and rte_eth_dev_data[] > > > > > > > > > > > > > > are not directly > > > > > > > > > > > > related. > > > > > > > > > > > > > > You may create new owner_id but it doesn't mean > > > > > > > > > > > > > > you would update rte_eth_dev_data[] immediately= . > > > > > > > > > > > > > > And visa-versa - you might just want to update > > > > > > > > > > > > > > rte_eth_dev_data[].name or .owner_id. > > > > > > > > > > > > > > It is not very good coding practice to use same > > > > > > > > > > > > > > lock for non-related data structures. > > > > > > > > > > > > > > > > > > > > > > > > > > > I see the relation like next: > > > > > > > > > > > > > Since the ownership mechanism synchronization is > > > > > > > > > > > > > in ethdev responsibility, we must protect against > > > > > > > > > > > > > user mistakes as much as we can by > > > > > > > > > > > > using the same lock. > > > > > > > > > > > > > So, if user try to set by invalid owner (exactly > > > > > > > > > > > > > the ID which currently is > > > > > > > > > > > > allocated) we can protect on it. > > > > > > > > > > > > > > > > > > > > > > > > Hmm, not sure why you can't do same checking with > > > > > > > > > > > > different lock or atomic variable? > > > > > > > > > > > > > > > > > > > > > > > The set ownership API is protected by ownership lock > > > > > > > > > > > and checks the owner ID validity By reading the next = owner > > ID. > > > > > > > > > > > So, the owner ID allocation and set API should use th= e > > > > > > > > > > > same atomic > > > > > > > > > > mechanism. > > > > > > > > > > > > > > > > > > > > Sure but all you are doing for checking validity, is > > > > > > > > > > check that owner_id > 0 &&& owner_id < next_ownwe_id, > > right? > > > > > > > > > > As you don't allow owner_id overlap (16/3248 bits) you > > > > > > > > > > can safely do same check with just > > atomic_get(&next_owner_id). > > > > > > > > > > > > > > > > > > > It will not protect it, scenario: > > > > > > > > > - current next_id is X. > > > > > > > > > - call set ownership of port A with owner id X by thread > > > > > > > > > 0(by user > > > > > > mistake). > > > > > > > > > - context switch > > > > > > > > > - allocate new id by thread 1 and get X and change next_i= d > > > > > > > > > to > > > > > > > > > X+1 > > > > > > > > atomically. > > > > > > > > > - context switch > > > > > > > > > - Thread 0 validate X by atomic_read and succeed to take > > > > ownership. > > > > > > > > > - The system loosed the port(or will be managed by two > > > > > > > > > entities) - > > > > > > crash. > > > > > > > > > > > > > > > > > > > > > > > > Ok, and how using lock will protect you with such scenario? > > > > > > > > > > > > > > The owner set API validation by thread 0 should fail because > > > > > > > the owner > > > > > > validation is included in the protected section. > > > > > > > > > > > > Then your validation function would fail even if you'll use > > > > > > atomic ops instead of lock. > > > > > No. > > > > > With atomic this specific scenario will cause the validation to p= ass. > > > > > > > > Can you explain to me how? > > > > > > > > rte_eth_is_valid_owner_id(uint16_t owner_id) { > > > > int32_t cur_owner_id =3D > > > > RTE_MIN(rte_atomic32_get(next_owner_id), > > > > UINT16_MAX); > > > > > > > > if (owner_id =3D=3D RTE_ETH_DEV_NO_OWNER || owner > > > > > cur_owner_id) { > > > > RTE_LOG(ERR, EAL, "Invalid owner_id=3D%d.\n", owner_id); > > > > return 0; > > > > } > > > > return 1; > > > > } > > > > > > > > Let say your next_owne_id=3D=3DX, and you invoke > > > > rte_eth_is_valid_owner_id(owner_id=3DX+1) - it would fail. > > > > > > Explanation: > > > The scenario with locks: > > > next_owner_id =3D X. > > > Thread 0 call to set API(with invalid owner Y=3DX) and take lock. > > > > Ok I see what you mean. > > But, as I said before, if thread 0 will grab the lock first - you'll ex= perience the > > same failure. > > I understand now that by some reason you treat these two scenarios as > > something different, but for me it is pretty much the same case. > > And to me it means that neither lock, neither atomic can fully protect = you > > here. > > >=20 > I agree that we are not fully protected even when using locks but one loc= k are more protected than ether atomics or 2 different locks. > So, I think keeping it as is (with one lock) makes sense. Ok if that your preference - let's keep your current approach here. >=20 > > > Context switch. > > > Thread 1 call to owner_new and stuck in the lock. > > > Context switch. > > > Thread 0 does owner id validation and failed(Y>=3DX) - unlock the loc= k and > > return failure to the user. > > > Context switch. > > > Thread 1 take the lock and update X to X+1, then, unlock the lock. > > > Everything is OK! > > > > > > The same scenario with atomics: > > > next_owner_id =3D X. > > > Thread 0 call to set API(with invalid owner Y=3DX) and take lock. > > > Context switch. > > > Thread 1 call to owner_new and change X to X+1(atomically). > > > Context switch. > > > Thread 0 does owner id validation and success(Y<(atomic)X+1) - unlock= the > > lock and return success to the user. > > > Problem! > > > > > > > > With lock no next_id changes can be done while the thread is in > > > > > the set > > > > API. > > > > > > > > > > > But in fact your code is not protected for that scenario - > > > > > > doesn't matter will you'll use lock or atomic ops. > > > > > > Let's considerer your current code with the following scenario: > > > > > > > > > > > > next_owner_id =3D=3D 1 > > > > > > 1) Process 0: > > > > > > rte_eth_dev_owner_new(&owner_id); > > > > > > now owner_id =3D=3D 1 and next_owner_id =3D=3D 2 > > > > > > 2) Process 1 (by mistake): > > > > > > rte_eth_dev_owner_set(port_id=3D1, owner->id=3D1); It will > > > > > > complete successfully, as owner_id =3D=3D1 is considered as val= id. > > > > > > 3) Process 0: > > > > > > rte_eth_dev_owner_set(port_id=3D1, owner->id=3D1); It wil= l > > > > > > also complete with success, as owner->id is valid is equal to > > > > > > current port > > > > owner_id. > > > > > > So you finished with 2 processes assuming that they do own > > > > > > exclusively then same port. > > > > > > > > > > > > Honestly in that situation locking around nest_owner_id > > > > > > wouldn't give you any advantages over atomic ops. > > > > > > > > > > > > > > > > This is a different scenario that we can't protect on it with ato= mic or > > locks. > > > > > But for the first scenario I described I think we can. > > > > > Please read it again, I described it step by step. > > > > > > > > > > > > > > > > > > > > I don't think you can protect yourself against such scenari= o > > > > > > > > with or without locking. > > > > > > > > Unless you'll make it harder for the mis-behaving thread to > > > > > > > > guess valid owner_id, or add some extra logic here. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The set(and others) ownership APIs already uses the > > > > > > > > > > > ownership lock so I > > > > > > > > > > think it makes sense to use the same lock also in ID al= location. > > > > > > > > > > > > > > > > > > > > > > > > > > > In fact, for next_owner_id, you don't need = a > > > > > > > > > > > > > > > > lock - just rte_atomic_t should be enough. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I don't think so, it is problematic in > > > > > > > > > > > > > > > next_owner_id wraparound and may > > > > > > > > > > > > > > complicate the code in other places which read = it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > IMO it is not that complicated, something like > > > > > > > > > > > > > > that should work I > > > > > > > > think. > > > > > > > > > > > > > > > > > > > > > > > > > > > > /* init to 0 at startup*/ rte_atomic32_t > > > > > > > > > > > > > > *owner_id; > > > > > > > > > > > > > > > > > > > > > > > > > > > > int new_owner_id(void) { > > > > > > > > > > > > > > int32_t x; > > > > > > > > > > > > > > x =3D rte_atomic32_add_return(&owner_id, 1)= ; > > > > > > > > > > > > > > if (x > UINT16_MAX) { > > > > > > > > > > > > > > rte_atomic32_dec(&owner_id); > > > > > > > > > > > > > > return -EOVERWLOW; > > > > > > > > > > > > > > } else > > > > > > > > > > > > > > return x; } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Why not just to keep it simple and using the = same > > lock? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Lock is also fine, I just think it better be a = separate > > > > > > > > > > > > > > one > > > > > > > > > > > > > > - that would protext just next_owner_id. > > > > > > > > > > > > > > Though if you are going to use uuid here - all = that > > > > > > > > > > > > > > probably not relevant any more. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I agree about the uuid but still think the same l= ock > > > > > > > > > > > > > should be used for > > > > > > > > > > both. > > > > > > > > > > > > > > > > > > > > > > > > But with uuid you don't need next_owner_id at all, = right? > > > > > > > > > > > > So lock will only be used for rte_eth_dev_data[] fi= elds > > > > anyway. > > > > > > > > > > > > > > > > > > > > > > > Sorry, I meant uint64_t, not uuid. > > > > > > > > > > > > > > > > > > > > Ah ok, my thought uuid_t is better as with it you don't= need to > > > > > > > > > > support your own code to allocate new owner_id, but rel= y on > > > > > > > > > > system libs > > > > > > > > instead. > > > > > > > > > > But wouldn't insist here. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Another alternative would be to use 2 locks= - one > > > > > > > > > > > > > > > > for next_owner_id second for actual data[] > > protection. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Another thing - you'll probably need to > > grab/release > > > > > > > > > > > > > > > > a lock inside > > > > > > > > > > > > > > > > rte_eth_dev_allocated() too. > > > > > > > > > > > > > > > > It is a public function used by drivers, so= need to > > > > > > > > > > > > > > > > be protected > > > > > > > > too. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, I thought about it, but decided not to u= se lock in > > > > next: > > > > > > > > > > > > > > > rte_eth_dev_allocated > > > > > > > > > > > > > > > rte_eth_dev_count > > > > > > > > > > > > > > > rte_eth_dev_get_name_by_port > > > > > > > > rte_eth_dev_get_port_by_name > > > > > > > > > > > > > > > maybe more... > > > > > > > > > > > > > > > > > > > > > > > > > > > > As I can see in patch #3 you protect by lock ac= cess to > > > > > > > > > > > > > > rte_eth_dev_data[].name (which seems like a goo= d > > > > thing). > > > > > > > > > > > > > > So I think any other public function that acces= s > > > > > > > > > > > > > > rte_eth_dev_data[].name should be protected by = the > > > > same > > > > > > lock. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I don't think so, I can understand to use the own= ership > > > > > > > > > > > > > lock here(as in port > > > > > > > > > > > > creation) but I don't think it is necessary too. > > > > > > > > > > > > > What are we exactly protecting here? > > > > > > > > > > > > > Don't you think it is just timing?(ask in the nex= t moment > > > > > > > > > > > > > and you may get another answer) I don't see optio= nal > > crash. > > > > > > > > > > > > > > > > > > > > > > > > Not sure what you mean here by timing... > > > > > > > > > > > > As I understand rte_eth_dev_data[].name unique > > identifies > > > > > > > > > > > > device and is used by port allocation/release/find > > functions. > > > > > > > > > > > > As you stated above: > > > > > > > > > > > > "1. The port allocation and port release synchroniz= ation > > > > > > > > > > > > will be managed by ethdev." > > > > > > > > > > > > To me it means that ethdev layer has to make sure t= hat all > > > > > > > > > > > > accesses to rte_eth_dev_data[].name are atomic. > > > > > > > > > > > > Otherwise what would prevent the situation when one > > > > process > > > > > > > > > > > > does > > > > > > > > > > > > rte_eth_dev_allocate()- > > >snprintf(rte_eth_dev_data[x].name, > > > > > > > > > > > > ...) while second one does > > > > > > > > > > rte_eth_dev_allocated(rte_eth_dev_data[x].name, ...) ? > > > > > > > > > > > > > > > > > > > > > > > The second will get True or False and that is it. > > > > > > > > > > > > > > > > > > > > Under race condition - in the worst case it might crash= , though > > > > > > > > > > for that you'll have to be really unlucky. > > > > > > > > > > Though in most cases as you said it would just not oper= ate > > > > correctly. > > > > > > > > > > I think if we start to protect dev->name by lock we nee= d to do > > > > > > > > > > it for all instances (both read and write). > > > > > > > > > > > > > > > > > > > Since under the ownership rules, the user must take owner= ship > > of a > > > > > > > > > port > > > > > > > > before using it, I still don't see a problem here. > > > > > > > > > > > > > > > > I am not talking about owner id or name here. > > > > > > > > I am talking about dev->name. > > > > > > > > > > > > > > > So? The user still should take ownership of a device before u= sing it > > (by > > > > > > name or by port id). > > > > > > > It can just read it without owning it, but no managing it. > > > > > > > > > > > > > > > > Please, Can you describe specific crash scenario and expl= ain how > > > > > > > > > could the > > > > > > > > locking fix it? > > > > > > > > > > > > > > > > Let say thread 0 doing rte_eth_dev_allocate()- > > > > > > > > >snprintf(rte_eth_dev_data[x].name, ...), thread 1 doing > > > > > > > > rte_pmd_ring_remove()->rte_eth_dev_allocated()->strcmp(). > > > > > > > > And because of race condition - rte_eth_dev_allocated() wil= l > > return > > > > > > > > rte_eth_dev * for the wrong device. > > > > > > > Which wrong device do you mean? I guess it is the device whic= h > > > > currently is > > > > > > being created by thread 0. > > > > > > > > Then rte_pmd_ring_remove() will call rte_free() for related > > > > > > > > resources, while It can still be in use by someone else. > > > > > > > The rte_pmd_ring_remove caller(some DPDK entity) must take > > > > ownership > > > > > > > (or validate that he is the owner) of a port before doing it(= free, > > > > release), so > > > > > > no issue here. > > > > > > > > > > > > Forget about ownership for a second. > > > > > > Suppose we have a process it created ring port for itself (with= out > > setting > > > > any > > > > > > ownership) and used it for some time. > > > > > > Then it decided to remove it, so it calls rte_pmd_ring_remove()= for it. > > > > > > At the same time second process decides to call > > rte_eth_dev_allocate() > > > > (let > > > > > > say for anither ring port). > > > > > > They could collide trying to read (process 0) and modify (proce= ss 1) > > same > > > > > > string rte_eth_dev_data[].name. > > > > > > > > > > > Do you mean that process 0 will compare successfully the process = 1 > > new > > > > port name? > > > > > > > > Yes. > > > > > > > > > The state are in local process memory - so process 0 will not com= pare > > the > > > > process 1 port, from its point of view this port is in UNUSED > > > > > state. > > > > > > > > > > > > > Ok, and why it can't be in attached state in process 0 too? > > > > > > Someone in process 0 should attach it using protected attach_secondar= y > > somewhere in your scenario. > > > > Yes, process 0 can have this port attached too, why not? > See the function with inline comments: >=20 > struct rte_eth_dev * > rte_eth_dev_allocated(const char *name) > { > unsigned i; >=20 > for (i =3D 0; i < RTE_MAX_ETHPORTS; i++) { >=20 > The below state are in local process memory, > So, if here process 1 will allocate a new port (the current i), update = its local state to ATTACHED and write the name, > the state is not visible by process 0 until someone in process 0 will a= ttach it by rte_eth_dev_attach_secondary. > So, to use rte_eth_dev_attach_secondary process 0 must take the lock an= d it can't, because it is currently locked by > process 1. Ok I see. Thanks for your patience. BTW, that means that if let say process 0 will call rte_eth_dev_allocate("x= xx") and process 1 will call rte_eth_dev_allocate("yyy") we can endup with same port_id be used for different devices and 2 processes will overwrite t= he same rte_eth_dev_data[port_id]? Konstantin >=20 > if ((rte_eth_devices[i].state =3D=3D RTE_ETH_DEV_ATTACHED) && > strcmp(rte_eth_devices[i].data->name, name) =3D=3D 0) > return &rte_eth_devices[i]; > } > return NULL; >=20 >=20