From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 29F191B30F for ; Thu, 18 Jan 2018 15:41:51 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 18 Jan 2018 06:41:50 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,378,1511856000"; d="scan'208";a="196636046" Received: from irsmsx107.ger.corp.intel.com ([163.33.3.99]) by fmsmga005.fm.intel.com with ESMTP; 18 Jan 2018 06:41:49 -0800 Received: from irsmsx105.ger.corp.intel.com ([169.254.7.236]) by IRSMSX107.ger.corp.intel.com ([169.254.10.239]) with mapi id 14.03.0319.002; Thu, 18 Jan 2018 14:41:48 +0000 From: "Ananyev, Konstantin" To: Matan Azrad , Thomas Monjalon , Gaetan Rivet , "Wu, Jingjing" CC: "dev@dpdk.org" , Neil Horman , "Richardson, Bruce" Thread-Topic: [PATCH v2 2/6] ethdev: add port ownership Thread-Index: AQHTh5xreoY7ZC2VXUKPLFrmK4AmzKNtNdVQgAFomACAAAOCwIAAuwQAgABvY+CABQwJgIAAEk3ggABinoCAANUIAIAAxQGAgAAGCnCAAQnKAIAAAmNwgAAXCYCAAABqkIAAQhEAgAASagCAACqV0IABEBGggAAhRACAAAK94A== Date: Thu, 18 Jan 2018 14:41:47 +0000 Message-ID: <2601191342CEEE43887BDE71AB9772588628032A@irsmsx105.ger.corp.intel.com> References: <1511870281-15282-1-git-send-email-matan@mellanox.com> <1515318351-4756-1-git-send-email-matan@mellanox.com> <1515318351-4756-3-git-send-email-matan@mellanox.com> <2601191342CEEE43887BDE71AB97725880E3B9D6@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772588627B12A@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772588627CCB0@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772588627DC25@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772588627DE30@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772588627E954@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772588627EE60@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772588627EEDA@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772588627F076@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772588628029A@irsmsx105.ger.corp.intel.com> In-Reply-To: Accept-Language: en-IE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiMzNiMjNiNjQtOTEyNC00ZWZkLWIyZjAtYmZmNzJlYmIzZDNlIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE2LjUuOS4zIiwiVHJ1c3RlZExhYmVsSGFzaCI6IkZTWXZ2TEdzZnBEbG5CYzNtY0xVVVlpTlNZOERBMXJcL0JCanJncFBPajZjPSJ9 x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.0.0.116 dlp-reaction: no-action x-originating-ip: [163.33.239.181] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v2 2/6] ethdev: add port ownership X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Jan 2018 14:41:52 -0000 >=20 > Hi Konstantine >=20 > > Hi Matan, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Another thing - you'll probably > > > > > > > > > > > > > > > > > > > > > need to > > > > > > > grab/release > > > > > > > > > > > > > > > > > > > > > a lock inside > > > > > > > > > > > > > > > > > > > > > rte_eth_dev_allocated() too. > > > > > > > > > > > > > > > > > > > > > It is a public function used by > > > > > > > > > > > > > > > > > > > > > drivers, so need to be protected > > > > > > > > > > > > > too. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, I thought about it, but decide= d > > > > > > > > > > > > > > > > > > > > not to use lock in > > > > > > > > > next: > > > > > > > > > > > > > > > > > > > > rte_eth_dev_allocated > > > > > > > > > > > > > > > > > > > > rte_eth_dev_count > > > > > > > > > > > > > > > > > > > > rte_eth_dev_get_name_by_port > > > > > > > > > > > > > rte_eth_dev_get_port_by_name > > > > > > > > > > > > > > > > > > > > maybe more... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As I can see in patch #3 you protect > > > > > > > > > > > > > > > > > > > by lock access to > > > > > > > > > > > > > > > > > > > rte_eth_dev_data[].name (which seems > > > > > > > > > > > > > > > > > > > like a good > > > > > > > > > thing). > > > > > > > > > > > > > > > > > > > So I think any other public function > > > > > > > > > > > > > > > > > > > that access rte_eth_dev_data[].name > > > > > > > > > > > > > > > > > > > should be protected by the > > > > > > > > > same > > > > > > > > > > > lock. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I don't think so, I can understand to > > > > > > > > > > > > > > > > > > use the ownership lock here(as in port > > > > > > > > > > > > > > > > > creation) but I don't think it is necessa= ry too. > > > > > > > > > > > > > > > > > > What are we exactly protecting here? > > > > > > > > > > > > > > > > > > Don't you think it is just timing?(ask > > > > > > > > > > > > > > > > > > in the next moment and you may get > > > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > > answer) I don't see optional > > > > > > > crash. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Not sure what you mean here by timing... > > > > > > > > > > > > > > > > > As I understand rte_eth_dev_data[].name > > > > > > > > > > > > > > > > > unique > > > > > > > identifies > > > > > > > > > > > > > > > > > device and is used by port > > > > > > > > > > > > > > > > > allocation/release/find > > > > > > > functions. > > > > > > > > > > > > > > > > > As you stated above: > > > > > > > > > > > > > > > > > "1. The port allocation and port release > > > > > > > > > > > > > > > > > synchronization will be managed by ethdev= ." > > > > > > > > > > > > > > > > > To me it means that ethdev layer has to > > > > > > > > > > > > > > > > > make sure that all accesses to > > > > > > > > > > > > > > > > > rte_eth_dev_data[].name are > > > > > atomic. > > > > > > > > > > > > > > > > > Otherwise what would prevent the situatio= n > > > > > > > > > > > > > > > > > when one > > > > > > > > > process > > > > > > > > > > > > > > > > > does > > > > > > > > > > > > > > > > > rte_eth_dev_allocate()- > > > > > > > >snprintf(rte_eth_dev_data[x].name, > > > > > > > > > > > > > > > > > ...) while second one does > > > > > > > > > > > > > > > rte_eth_dev_allocated(rte_eth_dev_data[x].nam= e, > > ...) ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The second will get True or False and that = is it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Under race condition - in the worst case it > > > > > > > > > > > > > > > might crash, though for that you'll have to b= e really > > unlucky. > > > > > > > > > > > > > > > Though in most cases as you said it would jus= t > > > > > > > > > > > > > > > not operate > > > > > > > > > correctly. > > > > > > > > > > > > > > > I think if we start to protect dev->name by > > > > > > > > > > > > > > > lock we need to do it for all instances (both= read and > > write). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Since under the ownership rules, the user must > > > > > > > > > > > > > > take ownership > > > > > > > of a > > > > > > > > > > > > > > port > > > > > > > > > > > > > before using it, I still don't see a problem here= . > > > > > > > > > > > > > > > > > > > > > > > > > > I am not talking about owner id or name here. > > > > > > > > > > > > > I am talking about dev->name. > > > > > > > > > > > > > > > > > > > > > > > > > So? The user still should take ownership of a devic= e > > > > > > > > > > > > before using it > > > > > > > (by > > > > > > > > > > > name or by port id). > > > > > > > > > > > > It can just read it without owning it, but no manag= ing it. > > > > > > > > > > > > > > > > > > > > > > > > > > Please, Can you describe specific crash scenari= o > > > > > > > > > > > > > > and explain how could the > > > > > > > > > > > > > locking fix it? > > > > > > > > > > > > > > > > > > > > > > > > > > Let say thread 0 doing rte_eth_dev_allocate()- > > > > > > > > > > > > > >snprintf(rte_eth_dev_data[x].name, ...), thread = 1 > > > > > > > > > > > > > >doing > > > > > > > > > > > > > rte_pmd_ring_remove()->rte_eth_dev_allocated()- > > > > > >strcmp(). > > > > > > > > > > > > > And because of race condition - > > > > > > > > > > > > > rte_eth_dev_allocated() will > > > > > > > return > > > > > > > > > > > > > rte_eth_dev * for the wrong device. > > > > > > > > > > > > Which wrong device do you mean? I guess it is the > > > > > > > > > > > > device which > > > > > > > > > currently is > > > > > > > > > > > being created by thread 0. > > > > > > > > > > > > > Then rte_pmd_ring_remove() will call rte_free() > > > > > > > > > > > > > for related resources, while It can still be in > > > > > > > > > > > > > use by someone > > > > else. > > > > > > > > > > > > The rte_pmd_ring_remove caller(some DPDK entity) > > > > > > > > > > > > must take > > > > > > > > > ownership > > > > > > > > > > > > (or validate that he is the owner) of a port before > > > > > > > > > > > > doing it(free, > > > > > > > > > release), so > > > > > > > > > > > no issue here. > > > > > > > > > > > > > > > > > > > > > > Forget about ownership for a second. > > > > > > > > > > > Suppose we have a process it created ring port for > > > > > > > > > > > itself (without > > > > > > > setting > > > > > > > > > any > > > > > > > > > > > ownership) and used it for some time. > > > > > > > > > > > Then it decided to remove it, so it calls > > > > > > > > > > > rte_pmd_ring_remove() > > > > > for it. > > > > > > > > > > > At the same time second process decides to call > > > > > > > rte_eth_dev_allocate() > > > > > > > > > (let > > > > > > > > > > > say for anither ring port). > > > > > > > > > > > They could collide trying to read (process 0) and > > > > > > > > > > > modify (process 1) > > > > > > > same > > > > > > > > > > > string rte_eth_dev_data[].name. > > > > > > > > > > > > > > > > > > > > > Do you mean that process 0 will compare successfully th= e > > > > > > > > > > process 1 > > > > > > > new > > > > > > > > > port name? > > > > > > > > > > > > > > > > > > Yes. > > > > > > > > > > > > > > > > > > > The state are in local process memory - so process 0 > > > > > > > > > > will not compare > > > > > > > the > > > > > > > > > process 1 port, from its point of view this port is in > > > > > > > > > UNUSED > > > > > > > > > > state. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ok, and why it can't be in attached state in process 0 to= o? > > > > > > > > > > > > > > > > Someone in process 0 should attach it using protected > > > > > > > > attach_secondary > > > > > > > somewhere in your scenario. > > > > > > > > > > > > > > Yes, process 0 can have this port attached too, why not? > > > > > > See the function with inline comments: > > > > > > > > > > > > struct rte_eth_dev * > > > > > > rte_eth_dev_allocated(const char *name) { > > > > > > unsigned i; > > > > > > > > > > > > for (i =3D 0; i < RTE_MAX_ETHPORTS; i++) { > > > > > > > > > > > > The below state are in local process memory, > > > > > > So, if here process 1 will allocate a new port (the current > > > > > > i), > > > > > update its local state to ATTACHED and write the name, > > > > > > the state is not visible by process 0 until someone in proces= s > > > > > 0 will attach it by rte_eth_dev_attach_secondary. > > > > > > So, to use rte_eth_dev_attach_secondary process 0 must > > > > > take the lock > > > > > > and it can't, because it is currently locked by process 1. > > > > > > > > > > Ok I see. > > > > > Thanks for your patience. > > > > > BTW, that means that if let say process 0 will call > > > > > rte_eth_dev_allocate("xxx") and process 1 will call > > > > > rte_eth_dev_allocate("yyy") we can endup with same port_id be use= d > > > > > for different devices and 2 processes will overwrite the same > > > > rte_eth_dev_data[port_id]? > > > > > > > > No, contrary to the state, the lock itself is in shared memory, so = 2 > > > > processes cannot allocate port in the same time.(you can see it in > > > > the next patch of this series). > > > > I am not talking about racing here. > > Let say process 0 calls rte_pmd_ring_probe()->....- > > >rte_eth_dev_allocate("xxx") > > rte_eth_dev_allocate() finds that port N is 'free', i.e. > > local rte_eth_devices[N].state =3D=3D RTE_ETH_DEV_UNUSED so it assigns = new > > dev ("xxx") to port N. > > Then after some time process 1 calls rte_pmd_ring_probe()->....- > > >rte_eth_dev_allocate("yyy"). > > From its perspective port N is still free: rte_eth_devices[N].state = =3D=3D > > RTE_ETH_DEV_UNUSED, so it will assign new dev ("yyy") to the same port. > > >=20 > Yes you right, this is a problem(not related actually to port ownership) Yep that's true - it was there before your patches. > but look: > As I understand the secondary processes are not allowed to create a ports= and they must to use attach_secondary API, but there is not > hardcoded check which prevent them to do it. Secondary processes ae the ability to allocate their own vdevs and probably= it should stay like that. I just thought it is a good opportunity to fix it while you are on these ch= anges anyway, but ok we can leave it for now. =20 Konstantin