From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9CF74A0566; Tue, 10 Mar 2020 17:06:17 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id ECA291BFF5; Tue, 10 Mar 2020 17:06:16 +0100 (CET) Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-eopbgr60045.outbound.protection.outlook.com [40.107.6.45]) by dpdk.org (Postfix) with ESMTP id B31F72BE6 for ; Tue, 10 Mar 2020 17:06:15 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9rTTsoVxbwXNvOzNhaGM4qU74BdcvfYswl7KufJcPTc=; b=4o2po4EN8QGsmypBHUocR+Pi4WoURF6qI4rPkm83zr2B3RqycpibfkuBYnu8TyqYNfT4Mt4pRUEZRizlHJE3lGEh1jVlLppkjVvRmAYzdnJTO43i3ju6+GIXC+IdkPCLwMJG/7yIUuWYThg5hNDIAhFJUFrpUEYGfgsggtg83bA= Received: from AM0PR05CA0093.eurprd05.prod.outlook.com (2603:10a6:208:136::33) by VI1PR0801MB2080.eurprd08.prod.outlook.com (2603:10a6:800:88::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2793.17; Tue, 10 Mar 2020 16:06:14 +0000 Received: from VE1EUR03FT007.eop-EUR03.prod.protection.outlook.com (2603:10a6:208:136:cafe::e3) by AM0PR05CA0093.outlook.office365.com (2603:10a6:208:136::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2793.16 via Frontend Transport; Tue, 10 Mar 2020 16:06:14 +0000 Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=bestguesspass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by VE1EUR03FT007.mail.protection.outlook.com (10.152.18.114) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2793.11 via Frontend Transport; Tue, 10 Mar 2020 16:06:13 +0000 Received: ("Tessian outbound 3a0cbd311638:v42"); Tue, 10 Mar 2020 16:06:13 +0000 X-CR-MTA-TID: 64aa7808 Received: from 6141be58b20c.2 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 20B42B10-0F01-489D-AE61-6A7298075B24.1; Tue, 10 Mar 2020 16:06:08 +0000 Received: from EUR01-HE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 6141be58b20c.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Tue, 10 Mar 2020 16:06:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RivaeHD+sNO3AVNdjQxAUebnMzbJaC9Rb1sq39wte6rHNBJ6NJM3/oOWmO6cDDSytGMywSQu7RJbtQ/ACM2ttjme3S9xuRxxzx5ff+n6xKyF4m+vMPCpLvCNqXsRyYl2RlA0/5/YGcWVBBIfb+lJOpkifHTfxbTDjQ0/GYxRi34FIjeZpJ7y2nEtRIhzw6OIwqc/tiGu/iaNveSgwxrdpxjIJMeKegJ88aTb4c4qMOee6R7EN6jE7qx94fAESvYwvH5sLIuCpSAmdge/axeL4n/w45jdGTnQrxfQmOdWq6g/RM+aXqFRg0xyJMA3cqI9Zz5qflampzX8HfC1kXmfew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9rTTsoVxbwXNvOzNhaGM4qU74BdcvfYswl7KufJcPTc=; b=Jwl0rNv0lPrahSipzUi+xTTndCSZfWsMS82AYYX0SqaEv3l6PfuBrOSaCwFzCc6XUbgpz0rxlXkHAG4HEw4EWRCnkg8g+0TksF/h5Ly3bo02Xhb+TL32FTjY6mgTLCd93057MFYgsoq35SKZaIasGN2tYiTisqws5QngTAZQHBbgZl5QcnZcUtQtJFr/AMQqttAXP7MH1MOWMCqxUoeDI8wawK/bcj78GQYC5QX5Z3v+dI0UvjtLXaXnKNzQoGZbJDH31I++0wBnhBe8ufxTIzBQjdquNOHgTtJFHEFJmValKRAquZz+4fBeBBdD3irsKL0NH+2KJgd7hUujixO/rw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9rTTsoVxbwXNvOzNhaGM4qU74BdcvfYswl7KufJcPTc=; b=4o2po4EN8QGsmypBHUocR+Pi4WoURF6qI4rPkm83zr2B3RqycpibfkuBYnu8TyqYNfT4Mt4pRUEZRizlHJE3lGEh1jVlLppkjVvRmAYzdnJTO43i3ju6+GIXC+IdkPCLwMJG/7yIUuWYThg5hNDIAhFJUFrpUEYGfgsggtg83bA= Received: from VE1PR08MB5149.eurprd08.prod.outlook.com (20.179.30.27) by VE1SPR01MB0014.eurprd08.prod.outlook.com (20.179.31.219) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2793.16; Tue, 10 Mar 2020 16:06:04 +0000 Received: from VE1PR08MB5149.eurprd08.prod.outlook.com ([fe80::712c:ce3b:a827:4f7c]) by VE1PR08MB5149.eurprd08.prod.outlook.com ([fe80::712c:ce3b:a827:4f7c%6]) with mapi id 15.20.2793.013; Tue, 10 Mar 2020 16:06:04 +0000 From: Honnappa Nagarahalli To: "Van Haaren, Harry" CC: "dev@dpdk.org" , Phil Yang , Gavin Hu , nd , Honnappa Nagarahalli , nd Thread-Topic: Questions on service core feature implementation Thread-Index: AdX0FYNEgdKQmrv+TaKE0Ss7kzbiUwCxCW4AAATz6IA= Date: Tue, 10 Mar 2020 16:06:04 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: e0efc65a-7283-4b4b-9732-9d6eb79b4877.0 x-checkrecipientchecked: true Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; x-originating-ip: [217.140.111.135] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: ebccc987-0ace-4be5-3e10-08d7c50cf536 X-MS-TrafficTypeDiagnostic: VE1SPR01MB0014:|VE1SPR01MB0014:|VI1PR0801MB2080: x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:6108;OLM:6108; x-forefront-prvs: 033857D0BD X-Forefront-Antispam-Report-Untrusted: SFV:NSPM; SFS:(10009020)(4636009)(346002)(39860400002)(376002)(396003)(366004)(136003)(199004)(189003)(5660300002)(478600001)(7696005)(55016002)(33656002)(86362001)(9686003)(2906002)(6506007)(6916009)(8936002)(76116006)(71200400001)(186003)(54906003)(26005)(8676002)(81166006)(81156014)(52536014)(66946007)(66446008)(64756008)(66556008)(66476007)(4326008)(316002); DIR:OUT; SFP:1101; SCL:1; SRVR:VE1SPR01MB0014; H:VE1PR08MB5149.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: SxqewI/dI+9BF14NrpupAXgAvuLqnWDzFoSAMYdUyd1a7yJJgTAavqglcguNHxfU9LQpruHIhL5/wKNzshdz8t66w5nTgKWKJtsUgh0OaIBJIn0+9BaJNdVC03C6N7wLsCRr9SYvhJB2CIdtHEmWBW5l9YBXVqPKT0P6LWmoPuW2YLkm89lOkoX4XaqPipovd8YBHeNlBZjKqc+uTvxoqKRq+spI5x/K3n6t9ITC62LVeA2Uog56EvhHaQenAXa0Tgy+mT942Io1LNTiDR4A9p1UifG3XKClfWMATDzjatCmlfAsYaFdP07CccTwhx+QF0Qh4t6TJZf9U9eqPIbyt0/ciFjIZuatms4JIIITlYKBpTzZ1Y6qCqp9v8TP4YCzCt1P9SMwVVEHluebfEh1BIGMSHCHbLSLvUeDCMsVaSMbm/UPXT0SnjtEK7DaRz6I x-ms-exchange-antispam-messagedata: HkTH7mZ6zzsnSt+7YvOax3sCuAvmlcqrQC7hNmUQsXzLXs0DwFfT3Yl421oHPUf4L3UUmrVqMgAPNxMFDp8tOn0vIut6HWR65kVDow1APL1s6Z602p5qwF1gF0uc6MKdzjVPG2beD6eN7AO4ZmUexw== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1SPR01MB0014 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: VE1EUR03FT007.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; IPV:CAL; SCL:-1; CTRY:IE; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(346002)(396003)(39860400002)(136003)(376002)(199004)(189003)(478600001)(4326008)(5660300002)(9686003)(33656002)(6506007)(54906003)(186003)(316002)(55016002)(36906005)(356004)(2906002)(52536014)(70586007)(81156014)(26826003)(8936002)(7696005)(86362001)(70206006)(81166006)(6862004)(26005)(336012)(8676002); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR0801MB2080; H:64aa7808-outbound-1.mta.getcheckrecipient.com; FPR:; SPF:Pass; LANG:en; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; A:1; MX:1; X-MS-Office365-Filtering-Correlation-Id-Prvs: 50700966-457f-4deb-9e56-08d7c50cefc8 X-Forefront-PRVS: 033857D0BD X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 67luGO8xxMlIZspjY11T+TeSa3moYrFIJNN3qfYDzIScJPurYWn3xyJEGn3brOCyAMygEuYEW3Bq5k2szUTO917dVJF0vfZ4dMeiNOkpRhEDU/ceNZPxpvezDYXZBAcKD5XvRNoNpzJNUS4a6SEH4iEdi35fyOvM6EJ+/j25l/6Kp81G81mPtAfVCkjzeauFzE/8OHEtBjQuIAuD600KgNHjOsXdv0ByTz9seI8YHsl4YS+dI0aBmhifpT6hlK5R/kh+bnnjrbu7NnaURO0vkGPBIIYe3ACueYfOqA03octzW64zm6zbMQ7c5jhM6c6eS/pSeI4ZhZ9GAFISbhXid7vU6OQXP9BDXqqsSDs4L2zHovY/FNte5h/5Elsr59ux5ROgqbeDHBRP291oMZa2TarCTHy87RaCaeKZdDInzmjIFGoYmHLgxhAMYCGdWOlo X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Mar 2020 16:06:13.9375 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ebccc987-0ace-4be5-3e10-08d7c50cf536 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0801MB2080 Subject: Re: [dpdk-dev] Questions on service core feature implementation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > > Hi Harry, >=20 > Hey Honnappa, Thanks for your answers. >=20 > > I have few observations on service core feature implementation. > > > > I believe it is allowed to map/unmap a lcore to service while the > > service is running (i.e. not just during initialization stage). Please > > clarify otherwise. >=20 > Service-cores wasn't designed for this use case, but this "popped up" > as the implementation was being finalized. Service cores' solves the prob= lem > of HW vs SW PMDs lcore requirements being different - and service cores i= s > the abstraction to hide that lcore requirement difference. >=20 > So I don't think we made a very conscious decision on if this is allowed = or not. >=20 >=20 > > Assuming yes - > > > > In the 'service_run' function, the code to detect use of atomics seems > > to result in allowing the mt-unsafe service to run on multiple cores. > > Looking at the following code: > > > > 1 const int use_atomics =3D (service_mt_safe(s) =3D=3D 0) && > > 2 (rte_atomic32_read(&s->num_mapped_cores) > 1); > > 3 if (use_atomics) { > > 4 if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock,= 0, > > 1)) > > 5 return -EBUSY; > > 6 > > 7 rte_service_runner_do_callback(s, cs, i); > > 8 rte_atomic32_clear(&s->execute_lock); > > 9 } else > > 10 rte_service_runner_do_callback(s, cs, i); > > > > Let us say, on core1, after line 4, 'use_atomics' is set to 0. core1 > > is running the service function. > > On the control plane let us say 'rte_service_map_lcore_set' is called > > to map the service to an additional core, core2. > > Now, on core2, after line 4, 'use_atomics' is set to 1. But since > > core1 did not take execute_lock, core2 will take the lock and starts > > running the service function. This should not be allowed since the > > service is a mt- unsafe service. > > > > Please let me know if my assessment is correct. > > A possible solution is to take the execute_lock all the time. >=20 > Yes agreed that your scenario seems to show an inconsistency. > As per above, I'm not sure this use case was really the target of the ser= vice- > cores design - but we as a community should identify if we want to suppor= t > this use case. >=20 > We could either update docs explicitly dis-allowing the above situation -= this > puts the burden on applications to understand what services could be runn= ing, > and if they're being scheduled already. Agree with you, it will be difficult for the application to determine how t= he service is scheduled. >=20 > The other solution being to change the implementation to allow this use-c= ase, > which will likely have a performance impact for those already using servi= ce > cores for its original function, by (as you suggested) taking execute_loc= k all the > time (and paying the atomic cost for that..) 'num_mapped_cores' indicates the number of cores the service can run on. Bu= t, to be able to effectively avoid the use of atomic instructions, we need = to know if a service is 'currently' running on another core. This informati= on is not maintained by the library I think maintaining that will also intr= oduce additional performance impacts. So, currently, a) the atomics are avoided only when the user configures MT-Unsafe service = to run on a single core. b) if the user really wants to allow a MT-Unsafe service to run on multiple= cores non-concurrently, atomics are used. I suggest the following changes: a) library will use atomics if the service is MT-Unsafe irrespective of the= value of 'num_mapped_cores' b) if the user wants to run a MT-Unsafe service on a single core and avoid = using atomics, then he should set the service capability to RTE_SERVICE_CAP= _MT_SAFE (This might result in some performance improvements since 'num_map= ped_cores' will not be checked anymore). Impact on the current users a) if they have configured MT-Unsafe service to run on multiple cores, then= they are already taking the hit of atomic instructions (and some additiona= l cost of reading/checking the 'num_mapped_cores', they should see some per= f improvements). No impact here. b) if they have configured MT-Unsafe service to run on single core, then th= ey might experience additional performance penalty. It might not be full pe= nalty of using atomics as we will be removing some instructions. This needs= to be measured. They can regain the performance loss by configuring their = service RTE_SERVICE_CAP_MT_SAFE. >=20 >=20 > > Thank you, > > Honnappa >=20 > Regards, -HvH