From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM05-DM3-obe.outbound.protection.outlook.com (mail-eopbgr730062.outbound.protection.outlook.com [40.107.73.62]) by dpdk.org (Postfix) with ESMTP id B3D9CDE3 for ; Mon, 8 Oct 2018 10:42:04 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=L4Vl4KhurH9LZn89K+IDcCxa4v1D5Zr1UfrI2zDzGgY=; b=FdAAYJCbVErTqgiOt06TobFHZn2BEbICIlUm8ccGnWksQVYzF8TpobAVtYDCc7+CgTbg9+Knr0rEbi8PchvBgQz78N/wTbPLW/Zg23IQLOzDz5rMt/QSLhMzloOHy8ZQLK5xMncLc1RbnplR6NWjoPrkbPi/cvFbpcsazgSQWKw= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jerin.JacobKollanukkaran@cavium.com; Received: from jerin (115.113.156.3) by BYAPR07MB4999.namprd07.prod.outlook.com (2603:10b6:a03:5b::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1207.28; Mon, 8 Oct 2018 08:41:59 +0000 Date: Mon, 8 Oct 2018 14:11:44 +0530 From: Jerin Jacob To: Ola Liljedahl Cc: Jan Viktorin , "Gavin Hu (Arm Technology China)" , "dev@dpdk.org" , "thomas@monjalon.net" Message-ID: <20181008084142.GA5363@jerin> References: <20181007063127.27960-1-jerin.jacob@caviumnetworks.com> <20181008062728.GA12197@jerin> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Originating-IP: [115.113.156.3] X-ClientProxiedBy: BM1PR0101CA0035.INDPRD01.PROD.OUTLOOK.COM (2603:1096:b00:1a::21) To BYAPR07MB4999.namprd07.prod.outlook.com (2603:10b6:a03:5b::24) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 7f65aaf3-162f-4df7-754c-08d62cf9eaf5 X-Microsoft-Antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(7168020)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:BYAPR07MB4999; X-Microsoft-Exchange-Diagnostics: 1; BYAPR07MB4999; 3:r8akFgo9UOP5EjvQo0tjPUDCBrj10bp2Oy4s/2FwWKx7JReYVzCk28xUa/IWl2NE7udZum1T7e/3IfqbFco0ACHVUXDStq2D7jRtRt2w331fyC7T6Armcm1NhQDtZIhIXtwJDubDccncn8T3hdAjP7bcWu2AkxfikuWLHdi9zjm6w1Og96vctiDdmEO5ZOGyPkz9bbi+TzeGG5jgYFrk9gE3a/9WjxTq9k2KL3QBsFZNYt2aKOdzkrA7uTye9rwh; 25:8wvyyJoFzldWfVAFSQ8FNLoEUs+DsJVC4sSg0I+BX1WgQ+29JQse7UoRdv+nCnHVAnjluGMHbPfsRTvtHkEOgQH8U6ef1zrwwwLByVAHbAjPQnCRnYsyJR5ZVA9uISXWTcR++zqv3Z2WTuWVE29aLHhEbL+sRr9Lwe1E18nSSqFY1UrHyIVY7OnEV0MPc0UIVBOBvmwla7AZtTawYvjgBpWyHs95fmJ3zOWSzSG8ncnNI/HBfBLMIZqYImFuPbJXbsEaq1XEweQWC9FmHGSYWhaNCkdxrgIwjUqcgVqGXfl9HV5TdrGEo6zbvc8eKqN0KKnfHEbiPIu55BybkKVFyQ==; 31:GMNZXm0sccAq3GdEcCVQhQ4m4EUnSgEEIHoKTXQiecJfUTi/0jEmd3Y8Lb3H7MH0sebYpiWUb5XdxCxtYcbczj6NL9lUPM3/xxKeiNE/UrEXrPhs1dKexDRBsT7psu3jEgW3Tzv6tg4qAOeukIeaA6CD6dxXrhAshjYCWzRf0e5WDlHbHVeRWqJ0eriq5yjF5sCwK4RjFSh88SAQiwPPd/6yG2zewOxiXNEw+vAAFL8= X-MS-TrafficTypeDiagnostic: BYAPR07MB4999: X-Microsoft-Exchange-Diagnostics: 1; BYAPR07MB4999; 20:tMozZ95vxLxKEu/q10Lkbz1Twk4iQ1TOxzVJLLVNbMaEDPNPkMQ7kaou27ofcYaVRY2/qKe2MAimmpv4kcDACUz7GeTzSFuqzRWChNg5252kBtjxHwIAs7E9GCDPX1+fOnTu+Y/ZHaoqR7tpI1XsLlpdZkerARtSGrVcfrzoomrRvNhrJrvXffwNlv8a7K1ErdmWpGEIFcJy5y+KVGAGphS27O1N7pT1CVR0hfHY49G/CdY+ZqTVJI7+jKtKqqUIKy3IxDUIFvIckdziLc+79Y+IvEMdvjzlb5Or0b4pTutVPvrFhJpiupuSp8Y7AXZKBi6HtmHGclKD2eyoQhCueiPtpTtiI/EAB+2nokuYeWJHhIqmdszxZec/7dlFGZ57Bj3pphgGqwbYYlsp8cg/ug5iSUCkiDWImqp/DWaU7DdG3BFgoWvZ1qkpp0Qd672VHJkFq4NnRc5NgS1m2riD4fg9nxWvnnMMQgT0omwz3KKVQ6iRqnbdmxoTyhsI9RP9crPyNaQZVqIMXUF3YfdvRNEvHQ1030kDmXCgKwoja4h8HfcvoFQNCRujmT4Et9deKNJiRODyZ89YVFD1BwoXFIDkp9z3VuB0Kn9ZQqw0wsU= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(180628864354917)(183786458502308)(22074186197030); X-MS-Exchange-SenderADCheck: 1 X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(3231355)(944501410)(4982022)(52105095)(3002001)(93006095)(10201501046)(149066)(150057)(6041310)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123562045)(20161123558120)(201708071742011)(7699051); SRVR:BYAPR07MB4999; BCL:0; PCL:0; RULEID:; SRVR:BYAPR07MB4999; X-Microsoft-Exchange-Diagnostics: 1; BYAPR07MB4999; 4:+daC6FzHTyhuHLsmhQN4oZUzGa1sfS02asUTCMqk41v7XR20aLUCMeJZ4dN8eIjE1uUN8MDQAZYJyfD2Ap1l8ezJZFDQj21jLxsjsdNCri1fPwuJgq4Nsm72WVY9PTsbizNvG5vTuD9v5r2OyJH1pwEhY80sEJheVK7+06n5dEE5w7RQA8q2uaSoimF8lpd/A04lRcRC2Uxj2oHcqK1z5tVfsFN4yyBP87W+KdUL60QfiENXE6Ef2k6u/qs+Ov/ZzyI5CALITqJrF14xiA0AzSrSiPIcViSaIOvnlKvYV2HEAYghwIcvq1poj/Mharhf8p7NAjhjOwA7bZ1Wydx+k7QnT3wWGCCFJmqskvgJNW6epkGs9pB9ODfalL3NjM85 X-Forefront-PRVS: 081904387B X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(366004)(376002)(136003)(346002)(39860400002)(396003)(13464003)(40434004)(199004)(189003)(42882007)(186003)(44832011)(16526019)(2870700001)(8676002)(25786009)(93886005)(6246003)(26005)(66066001)(316002)(81156014)(6116002)(1076002)(486006)(446003)(5024004)(81166006)(3846002)(14444005)(58126008)(11346002)(575784001)(19627235002)(47776003)(476003)(4326008)(956004)(478600001)(6496006)(45080400002)(76176011)(52146003)(33896004)(105586002)(53936002)(33656002)(7736002)(305945005)(23676004)(52116002)(2486003)(97736004)(8936002)(53546011)(966005)(386003)(55236004)(50466002)(54906003)(106356001)(2906002)(229853002)(33716001)(5660300001)(9686003)(68736007)(72206003)(6306002)(6666003)(6916009)(55016002)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:BYAPR07MB4999; H:jerin; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtCWUFQUjA3TUI0OTk5OzIzOnhzWkhKZXV1NjRjRGtDdldiOFhZNUgvWmFy?= =?utf-8?B?bDQvZ0NXd2NWWnZiSXYxY256Zmp4Nlg1U2ZCeHNEYlpIQjlGV1FYdytoVEF3?= =?utf-8?B?OThiQ3R1SG15ZU9YbGFJSkU2enc1UE9XWFdpMm1xUjBxTys1T2hkN1RPU3lp?= =?utf-8?B?Y0tnMG5EYjRnNXpPVHNoTitGM0ZTRDFoVGJFdkxnVjZGaWw2QTY5MjVXU2Fz?= =?utf-8?B?eXUzbFVKcGpXSG85UzJTdkZFRDlTdnhIc2tZL2duTG9EamZ1SEJVTitqdnRs?= =?utf-8?B?T3QwaTBLS0ZKTzVSaEs0QmE3OXZ0Uy9TMWRveVdCRVBnTjlTTitpRTF1Vy9y?= =?utf-8?B?Ly8vOTZXanZ3RTc0R0xvTnN4dVNreWZweHFOZDliSm1HOHp6MmpwdVZlbENW?= =?utf-8?B?cW5Nc3FLZHQ2bXZLNE0wdG55eFhNdlJXeDhyUEVtQjViVHg1cG9yVStYVi91?= =?utf-8?B?OXFTdnpLQU5uUmVCODRMUnhJVDl5SW1Dem02eVdMNVRLRlNNZE1vNllFWVVD?= =?utf-8?B?STkvc0JMT0szaHFDMUlqYllCdFlBNVJPNGF2TVJHRnhITlhESUR0VHNwYzQr?= =?utf-8?B?WkdhZ0pUdUdWbnEvS0VsWG4wM25VczZEUTJOOERhYk0rVnl1SFJDTjJIWmZ3?= =?utf-8?B?S3lxUXEzTE9nZVU4WnRybXpGYURGRlNLalVLZjgzNHltT05ZNDZRTGFBa1N1?= =?utf-8?B?ZEFReng0bk0wV3FJVVpEQTl2dEFhMUZYb2J5VWF1RHFLc2c2SXcydGx2MGpt?= =?utf-8?B?WWM2bG9xbzdFaUgrSTNZUnFvblM1ejd6SzI0VWhITXBDZ0I2aEhQWmUzeDYr?= =?utf-8?B?SWFVS0wzMDVpMlZXZGhmRWttelRaV0RuaGhSVUJydmIvellpbHRtbUVtSzdH?= =?utf-8?B?VHVsRGE4OFFvZVNHSzEwNi93cytzUUFKcUhKQTBpb0NiQ29sOFp2cWFpUHRq?= =?utf-8?B?UHcxdFlUTXdCenRiYlBlalFsMkZUZVlCbERLa3V3SWhZOWJXODhiVzVocGhh?= =?utf-8?B?d1Q3MW1NL2Q5UkVvcUY5SXZvS0tic0VOV2gxWEhibDl1NmZSSzl6OExDMCta?= =?utf-8?B?OFpIRTRPczRxZVl3R09TOW5XcCtycnd0SjZkelQxRVJFMklXL241UU9FbXMv?= =?utf-8?B?QUdxbGpiRWNyZUxpWkRWUm5UbU5aQjFTbEw4aHRDdUJCdjZsRElXWVNGSHpF?= =?utf-8?B?aE02MVU1T2NJS0o5YTdFdVJRaksrMkNrd1dWdjJEWnVsZVJydzl4K0dmNGdw?= =?utf-8?B?RStoa3crMmFYZU9LbnVncTdNK0FnMHB5Y1FEQmVEK2xDTzJHZ0xDc0JCMmk1?= =?utf-8?B?WXhVSm8yb0RRN0t5M1lzZ1pubWlKclduek9yL2liUE5PTU5LcDBITEtRV04x?= =?utf-8?B?VjV4RHp4R1pOR1FCOGZVOFpBVXJkZllRL2FsVnk2amM5T0lIWGRObUFSR2Q0?= =?utf-8?B?eEZNWVVNNDVGS3pySTNFUlpUUmVjM2o3WEMxTTgvcUNKbFRNU0gyU2FHbEww?= =?utf-8?B?czVDZXF2WmgxbzBWTGFIWjhsQWd1eDFtSDFDRDNhNkd0NGNKcC82WjZJS1F2?= =?utf-8?B?RktHWVp2c09MdWQ5RHpDNFhRZEhxd2RwZkRmS29jdm0zTTZTTVhDNy9nQ1dQ?= =?utf-8?B?bE1FUzFmOWt0UUVZZWg4ejJIL1JGc1RrQ2s0N2FBa0xqS2ZnM3hyYmtpS2hS?= =?utf-8?B?OTZNR0pYUVU1bWJqTWhLaUpoNGJFWm9zbC9BYTdwcnlGOVY1MFpLWCtxZjF4?= =?utf-8?B?aXNINmxLY0RYeTJ1WkxteWI1TU41UHVqcjc0U1U4NERHYVVVWWFjYkpIamds?= =?utf-8?B?Tkd0a2VUNE1mcTlCc0FaeFgyQkQ5SnhlbmtnclZyRGl2VzJXcEZZYmhuMGpS?= =?utf-8?B?SDkvakVUdlRndEUrSDRxYkMxb1ZrTGk2aHErMFVCM2ZzQVlOWTExWmJORis3?= =?utf-8?B?aE42ZjdPSXhhVks3T2ZLUmVQMS9GRUlqR0laaVNpNnNRU0NXM1ZQWURkZmlE?= =?utf-8?B?NUg0OXBvU1BJcUhxdUkwUW9aWWZTbTUxQ21FUzZDdnRBKzBSMnpNdVhRa1hh?= =?utf-8?B?NU50K2RubXpacThQeURBVWtKVTM1T0pFU25sYUtTbFcvcU1CKzJkNTc0aXVW?= =?utf-8?B?WEFPSTlaUFFUd0JVdUlTTzBlaXZwU0Z4dVlKZ1diRnYwNm9BNmYxMWVXUGVU?= =?utf-8?B?c3dIQzRHM3orVjR4NTJUWUQxT0VnPT0=?= X-Microsoft-Antispam-Message-Info: GE+Ofuqg/P9MCaYG3BBYt/Uv1meePNc6r3wZpzx4Ncj5td7Vlrnyzp4Vos1z7E6SNFNc2g9a+h5gneg2ZcT25TdcpBGncQs8rqKbqS2104rB74d/4P/z1qd6Cl2lbsD5/xciwW+f/SEEsa0g72eB1mCTaC8NLV5GhoItVMJ+qP48PavMQYN+c4KpumpotkwusaXUCk+sWpjTeRrYmOA7PZpUCEv30MxxdA3fQKaK1YzZeU8XTuKeQRyfV3uDDwL6mY1pyExI8OY7a6Bg1acm8+MK+Av6lb/KqHqQIQ33oZ+ah+7EHig8lL1xiu7eZfLUBD8NlxVyj3PRp/+wZybmXMxScQGIwVrooWnCpVQXsdg= X-Microsoft-Exchange-Diagnostics: 1; BYAPR07MB4999; 6:vscuJZjUKh9wG//NBNHOJbH0v/mM7mBwrqXwOsfP0cSlpCQ5H7pK9R3XhNOEhajg95zJxN8vAnlnW1lKZnwI2WIFvYfC+LrKiuQxsC+fsjVBqNNM5koPiyzhqK45FhTWq1FVOg5I0GnbADmG1zMQcQY781U9pn4CFaQ+/PirGGjc8SKheNIjcLzINO5GnDOV23YBkNwWE8srgkQ3JWZ2rYNx5XRX4ISz4mVYqOHM4FSiLB+QaJw4QWliWVJeQ/eWkNncrWkcu2GSSNaGyMJ9rwUWDO6vjV8tZkTQv6A/BnWSIneYZG42INvwaVgD9CfBrOeRB5N4Crn3biIyyl8OaMMqt0qJO+/ugc4QLzqH3IsB8w2nK8d5H2KX+3P2m4eUTZvZjN2zZI3ZA7yD/xywMkRV0rQXT7j3c375HRE9rWmht0wTcmi0dX0zLXAirlPwI4NCvJRNKvWM1sb4gilWyA==; 5:KUT8Ip33lfx8shHdOTH3CoKmXuuiEPLNqm52sOR618LRQVi+kicby5yh5ZOpPGsj94Ftal5BIg72l5YXfkmSIUCtpEH3IOQBh+NVUIMaL+QAYupELVr8dxNFkxnKh17R9INWV1myTp9g09tX+GaC8L7Hl9qvHBvGH8xi6e8rzWo=; 7:gsq5/v8HosrAVh0tRf7iroTh79bi/YIZLESuIJ2EOpUBewyWVHAz88BeAwr0hSVO7lANrwvPRq10rAAgUmmqOMe3VRzNEzDfM2f+BrAsH01fTz4YMkU1bSdKx3C+xM/WJOyQEupD3tU/LlqkDv/2RCtG0JMw0HBuRDbQVlwh1nF1FvIsd0tPk/4aam7kwt43mYi1TKcilfEOQmZuEX1EyWpkC8AmBDJTwWagnR0krREzfGJPlQv7arRUdZQ1mZg9 SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Oct 2018 08:41:59.3108 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7f65aaf3-162f-4df7-754c-08d62cf9eaf5 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR07MB4999 Subject: Re: [dpdk-dev] [PATCH] eal/armv7: add support for rte pause X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Oct 2018 08:42:05 -0000 -----Original Message----- > Date: Mon, 8 Oct 2018 08:25:28 +0000 > From: Ola Liljedahl > To: Jerin Jacob > CC: Jan Viktorin , "Gavin Hu (Arm Technology > China)" , "dev@dpdk.org" , > "thomas@monjalon.net" > Subject: Re: [dpdk-dev] [PATCH] eal/armv7: add support for rte pause > user-agent: Microsoft-MacOutlook/10.11.0.180909 > > > On 08/10/2018, 08:27, "Jerin Jacob" wrote: > > -----Original Message----- > > Date: Sun, 7 Oct 2018 21:09:25 +0000 > > From: Ola Liljedahl > > To: Jerin Jacob , Jan Viktorin > > , "Gavin Hu (Arm Technology China)" > > > > CC: "dev@dpdk.org" , "thomas@monjalon.net" > > > > Subject: Re: [dpdk-dev] [PATCH] eal/armv7: add support for rte pause > > user-agent: Microsoft-MacOutlook/10.11.0.180909 > > > > External Email > > > > On 07/10/2018, 08:32, "Jerin Jacob" wrote: > > > > Add support for rte_pause() implementation for armv7. > > > > Signed-off-by: Jerin Jacob > > --- > > > > The reference implementation for Linux's cpu_relax() for armv7 is at > > https://elixir.bootlin.com/linux/latest/source/arch/arm/include/asm/processor.h#L100 > > > > --- > > lib/librte_eal/common/include/arch/arm/rte_pause_32.h | 4 +++- > > 1 file changed, 3 insertions(+), 1 deletion(-) > > > > diff --git a/lib/librte_eal/common/include/arch/arm/rte_pause_32.h b/lib/librte_eal/common/include/arch/arm/rte_pause_32.h > > index d4768c7a9..9b856e0cf 100644 > > --- a/lib/librte_eal/common/include/arch/arm/rte_pause_32.h > > +++ b/lib/librte_eal/common/include/arch/arm/rte_pause_32.h > > @@ -9,11 +9,13 @@ > > extern "C" { > > #endif > > > > -#include > > +#include > > + > > #include "generic/rte_pause.h" > > > > static inline void rte_pause(void) > > { > > +rte_compiler_barrier(); > > The compiler barrier is not mandated by the DPDK documentation for rte_pause(): > > http://doc.dpdk.org/api/rte__pause_8h.html > > We can add that explicitly if required to inline with other arch. Just like > Linux kernel's cpu_relax() > I think the documentation should specify this compiler barrier if it is needed for correct behaviour. Yes. > > > > > > You have to go all the way to the source and GCC documentation to discover that for GCC, rte_pause calls _mm_pause() which in turn is implemented using __builtin_ia32_pause(). > > https://gcc.gnu.org/onlinedocs/gcc-4.9.2/gcc/X86-Built-in-Functions.html > > void __builtin_ia32_pause (void) > > Generates the pause machine instruction with a compiler memory barrier. > > Yes. IMO, it makes sense to have compiler memory barrier to make sure it > waits semantically at least WRT current rte_pause() usage. > Current *non-C11* usage. But more and more code in DPDK uses the C11 memory model. Probably we need a different API to CPU wait,n otherwise exiting rte_pause() will break. For example, lib/librte_ring/rte_ring_generic.h also uses rte_pause() > > > > > > If you are using C11 atomic operations e.g. for polling a location, the atomic operations will be able to provide the required semantics (e.g. don't merge atomic loads from different iterations of a loop, optionally provide acquire and/or release (or stronger) ordering. A compiler barrier here interferes with the (possibly weaker) barriers from the atomic operations. We could use a C11-version of rte_pause() that doesn't have the compiler barrier. But actually, we want support for WFE, x86 also has something similar now, MONITOR/MWAIT > > If it is WFE then who will wake up from the power saving state. SEV from the > other thread? > SEV/WFE is the ARMv7 way of waiting for event but the waking up is very crude (SEV broadcasts an event to *all* cores). ARMv8 introduces a new way where the waiting thread uses SEVL/WFE/LDXR/WFE to wait for a specific location (in practice cache line) to be updated and whichever thread writes the location will automatically notify any waiters (no SEV needed). See code example in other email thread. Yes. The context was ARMv7 patch so I said about SEV. > > > What would be a C11 version of rte_pause()? > A function that stalls the CPU for some ten(s) of cycles. No implicit or explicit (compiler) barriers. E.g. ISB on ARM which - unlink NOP - actually stalls the pipeline for 10-20 cycles (but ISB will also have HW barrier semantics). But as I wrote above, using WFE would be better (at least has been better in the internal benchmarks I have done/seen). Much better to focus our efforts on how to make use of WFE for C11 code. Is there any API in C11 which maps WFE and LDXR pair.? If not, We need introduce new API to use conjunction with LDXR. I would say more than C11, Address use case through load acquire and store release semantics. > > > > > -- Ola > > > > > > } > > > > #ifdef __cplusplus > > -- > > 2.19.0 > > > > > > > > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. > > > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.