From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by dpdk.org (Postfix) with ESMTP id 923542C2A for ; Thu, 29 Jun 2017 18:35:09 +0200 (CEST) Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga105.jf.intel.com with ESMTP; 29 Jun 2017 09:35:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.40,281,1496127600"; d="scan'208";a="120864113" Received: from irsmsx106.ger.corp.intel.com ([163.33.3.31]) by fmsmga006.fm.intel.com with ESMTP; 29 Jun 2017 09:35:07 -0700 Received: from irsmsx102.ger.corp.intel.com ([169.254.2.211]) by IRSMSX106.ger.corp.intel.com ([169.254.8.236]) with mapi id 14.03.0319.002; Thu, 29 Jun 2017 17:35:06 +0100 From: "Van Haaren, Harry" To: Thomas Monjalon CC: "dev@dpdk.org" , 'Jerin Jacob' , "Wiles, Keith" , "Richardson, Bruce" Thread-Topic: Service lcores and Application lcores Thread-Index: AdLw4cih5pwlzbSuRKKmF+/821WngAAAHVUAAAPYppA= Date: Thu, 29 Jun 2017 16:35:05 +0000 Message-ID: References: <7268949.8nYIVvgy1g@xps> In-Reply-To: <7268949.8nYIVvgy1g@xps> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiYTc4YTVlYTktOWFhNy00YzIwLTg2NDktMDIwY2RhYThjNjE0IiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX0lDIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE2LjUuOS4zIiwiVHJ1c3RlZExhYmVsSGFzaCI6IlwvditjV085Y0V6ckJIKzZJRWpFUUZLSDdKUGJYVU1FS2plUlJvRE5wcWQ0PSJ9 x-ctpclassification: CTP_IC dlp-product: dlpe-windows dlp-version: 10.0.102.7 dlp-reaction: no-action x-originating-ip: [163.33.239.181] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] Service lcores and Application lcores X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jun 2017 16:35:10 -0000 > From: Thomas Monjalon [mailto:thomas@monjalon.net] > Sent: Thursday, June 29, 2017 4:16 PM > To: Van Haaren, Harry > Cc: dev@dpdk.org; 'Jerin Jacob' ; Wiles, = Keith > ; Richardson, Bruce > Subject: Re: Service lcores and Application lcores >=20 > 29/06/2017 16:36, Van Haaren, Harry: > > The topic of discussion in this thread is how we can ensure > > that application lcores do not interfere with service cores. >=20 > Please could you give more details on the issue? Sure, hope I didn't write too much! > I think you need to clearly explain the 3 types of cores: > - DPDK mainloop > - DPDK services > - not used by DPDK DPDK cores continue to function as they currently do, with the execption th= at service-cores are removed from the coremask. Details in 0) below. DPDK service cores run services - they are not visible to the application. = (AKA; the application does not perform any remote_launch() on these cores, = it is handled internally in EAL). Service lcores are just normal lcores, on= ly the lcore_config[lcore_id].core_role =3D=3D ROLE_SERVICE instead of ROLE_RTE. Non DPDK cores are not changed. I'll run through the following scenarios to detail the problem; 0) Explain where service cores come from in relation to non-DPDK cores 1) Describe the current usage of DPDK cores, and how the eventdev scheduler= is used 2) Introduce the a service core only usage of eventdev 3) Introduce the problem: service cores and application cores concurrently = run a multi-thread unsafe service 4) The proposed solution 0) At application startup, the EAL coremask detects DPDK cores, and "brings= them up". Service cores are "stolen" from the previous core-mask, so the service-c= ore mask is a subset of the EAL coremask. Service cores are marked as ROLE_SERVICE, and the application "FOR_EACH_= LCORE" will not use them. Non-DPDK cores are not affected - they remain as they were. 1) Currently, a DPDK application directly invokes rte_eventdev_schedule() u= sing an ordinary app lcore. The application is responsible for multiplexing work on cores (assuming = multiple PMDs are running on one core). The app logic *must* be updated if we wish to add more SW workloads (aka= , using a SW PMD instead of HW acceleration). This change in app logic is the workaround to DPDK failing to abstract H= W / SW PMD requirements. Service cores provides the abstraction of environment (SW/HW PMD) differ= ence to the application. 2) In a service core "only" situation, the SW PMD registers a service. This= service is run on a service core. The application logic does not have to change, as the service-core runni= ng the service is not under app control. Note that the application does NOT call rte_eventdev_schedule() as in 1)= above, since the service core now performs this. 3) The problem; If a service core runs the SW PMD schedule() function (option 2) *AND* the application lcore runs schedule() func (option 1), the result is tha= t two threads are concurrently running a multi-thread unsafe function. The issue is not that the application is wrong: it correctly called rte_= schedule() It is also not that service core infra is wrong: it correctly ran the se= rvice The combination of both (and the un-awareness of eachother) that causes = the issue. 4) The proposed solution; In order to ensure that multiple threads do not concurrently run a multi= -thread unsafe service function, the threads must be aware of runtime of the other threads. The service c= ore code handles this using an atomic operation per service; multiple service cores operate co= rrectly, no problem. The root cause of the issue is that the application cores are not using = the service atomic. The rte_service_run() function, allows the application to be aware of se= rvice-core runtime habits due to calling into the service library, and running the service from th= ere. With this additional rule, all cores (service and application owned) will be aware of eachother, an= d can run multi-thread unsafe services safely, in a co-operative manner. In order to allow the application core still run the eventdev PMD "manua= lly" if it insists, I am proposing to allow it to invoke a specific function, which is aware= of the service atomic. This will result in all cores "playing nice", regardless of if i= t is app or service owned. The rte_service_run() function (which allows an application-lcore to run= a service) allows much easier porting of applications to the service-core infrastructure. = It is easier because the threading model of the application does not have to change, it looks= up the service it would like to run, and can repeatedly call the rte_service_run() functio= n to have the application behave in the same way as before the service core addition. Ok, this got longer than intended - but hopefully clearly articulates the m= otivation for the rte_service_run() concept. Regards, -Harry