From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM01-BY2-obe.outbound.protection.outlook.com (mail-by2nam01on0046.outbound.protection.outlook.com [104.47.34.46]) by dpdk.org (Postfix) with ESMTP id A1440376C for ; Tue, 11 Jul 2017 10:30:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=+GOR8PDVcJrE9yRPAsd0nafRtvOLFTwePBjOFvRR/r4=; b=fE6AdOtGT/vG/Y/fBzZcmLrgiqSVHfb1BmJHngYuRzumtkprHYm9z6jRY6frSko/QPW89RUKp1pxnv8Y2sVwb45h8hGa+5WAPm1+Z25fb07R4pzwsiokG1HHaPTBKE/Gyf4Xa0/zTdkw6NfXifR3TN/+M30VbZpK7ShcnQwMV3I= Authentication-Results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=caviumnetworks.com; Received: from jerin (106.201.60.201) by BN3PR0701MB1717.namprd07.prod.outlook.com (10.163.39.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1240.13; Tue, 11 Jul 2017 08:30:10 +0000 Date: Tue, 11 Jul 2017 13:59:52 +0530 From: Jerin Jacob To: Harry van Haaren Cc: dev@dpdk.org, thomas@monjalon.net, keith.wiles@intel.com, bruce.richardson@intel.com Message-ID: <20170711082950.GB29792@jerin> References: <1498735421-100164-1-git-send-email-harry.van.haaren@intel.com> <1499445667-32588-1-git-send-email-harry.van.haaren@intel.com> <1499445667-32588-2-git-send-email-harry.van.haaren@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1499445667-32588-2-git-send-email-harry.van.haaren@intel.com> User-Agent: Mutt/1.8.3 (2017-05-23) X-Originating-IP: [106.201.60.201] X-ClientProxiedBy: MA1PR01CA0112.INDPRD01.PROD.OUTLOOK.COM (10.174.56.156) To BN3PR0701MB1717.namprd07.prod.outlook.com (10.163.39.16) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f83cbfd8-a24b-4c9f-fea5-08d4c8370c5d X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(300000503095)(300135400095)(201703131423075)(201703031133081)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:BN3PR0701MB1717; X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1717; 3:GY/BskZ2G66NQgZlq8hrIuewfnEmYGdk6JBt4GJfN1iAaVR9qm53rzQRGat7l+8DuVL6KToGTirxlnazrZoQK+icE2w0JFEyzhXphoZLLKNxgXNZhchmhovCYFwVFmCjhqXlScqCkYSJ0so+JIJ60WsjcnCi4N7Wrj8SIVLH+D3pkzRPj+3wRRBEJRZk39hJjvE+cmkjyGphZPELEHkdYss4cEWU80SwvZMTZAx6NlZoIkI0mEpAquLLjpNH4FoC94k0d0wjuoOB9jYcb7Mr9fq+BjM5mO2VLD0xAZLMQpJ7RDCoU/vA7wxCtCPkyyqPw1mezNpjaKEu/nZSyg1wZLt0nJ1zFT4ColpUKHK8YmAb38+Eowm3rzCQc9GOSeh9mvvMCNaD5Rhy42AMPoBzMczt34WoUmEK0kkLK5Y/x9QF8+zftJUax/mu0tQC99PF9liQYkXgWXAaCyMpRQjdmaK6fbK4xWBvOiZ/uMyz+XcE1wsowccH3PLS8XDxBLp0whl/G0f0Mj/JR8CEzIYC8Xb4EgCzTCaMc9u7X1jRwjlCfiJPCkzg/WabFNFcc55z+y0FbTOABQcOQ0LJMxoR0N5G55UgaOEhZGoqTdJOwbDMmo+iCd9tu4xiITL0pau/M4+UJFi8iSWUb2GEK4WjYaKK1UMKyCopT1gaw69CEHIlUltaVpR6qUwQHjOC5PTJl2kItuBNonNjDw2ZByIyChpsD7eOrvX2e21K5sSlVTI= X-MS-TrafficTypeDiagnostic: BN3PR0701MB1717: X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1717; 25:i+yK8gXNjClG/hE/8bX6w+PfrIpgZaswKIhCrTrf2BWRoz+Ojd5ZcjqJacbZwYLYRTEe1Vh9HFbLKVv8ovEhmZsic0f3z137SHCkNp4ZmJR7c9clzGrVnY2YKACdy71PtMys2gRDdfJIxWp4f3fA2cSPddj+lhzLEVresLFqnD4QIiFESjl5moyxXUpaB4wdrKRMtmQBYqFSRbi3a0h6tWem1Yo7WcCDNH02EAlXyGXv8PuULYf9fKr1boN70B9UBpYAX6/zuc+1u/62ZmXYreW9QKIE+W/aF/zDt05mNJ510icQ6k5Mb7X+dv5bxv0qWtFhFZergOaml849TalqvLjzCZ4Wsbt3v5PNe1YWKg9SWVSgq3B80ODKtaIM/KuqbbWOuoNOK5cv8eU9WcHa0a72XKFimpwsV/N/+R1EbgmhIqoYJyC758ZALPVoALPo/YOgg2KTxn0zZyLFr2VQj7k4j74iwnxtcDJ5b2pFkFcQ3nd0Ryds+HQeCqL3UL2mkDwo2rIjIw/zK7Kqdoj4Be5FazMBa4PYBByxQxE6JbRYIZwQ7Ce9smVZLaL68ZF2ht7bX+T4C5g5SgFeAoW2/Mtu56/Gqay1er8wnpTuMscwFFWkBK3UsQTNPEiVOSBG7rWnVkdqfYWmc8sjHFNQaz8J9Aol2qDQHkvrsaQFGSP82TDMgpnnerOMShqa1XOM1tr0FBcn0u3lQbc0wnoL8zHehEcmtIF4UawZeLFGVugmDqDyqbhYYQ5mlIEyceBe2T4gl3OZlMJ/OktukUTe6KycJL6V95wrXGVKBoNKarrzVFjm1MYzSK1wdPSaqfP9YCl6XunQ8krBGPedZr94so+OY/FEEky+iEI76df/h/T6SExB1UIUxmcENl+o9Y7SCryv2QXTpMtELFjsFduzzv2nlkGeffzjxGtYCAhdAiw= X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1717; 31:QbohVsNw4SYPK2FhAaUs3rB6H7dwWzMGxNopNZPVzj6JFUxd30gRIMKA8yD1nIpcmrchDQO/PcuFAdJfxRGp0mDD3tc0B+zR7q9aI5eKzF/8Hop83T0lzsA3f27hCsznGEDQKrWNAMkDhu542KKDx96357tuZwGR0rWPmrH+w3lWLCaKy3T3g7Guz9mokOPkhQiz7xlWZyUNLucCCzB/7ORDHafd8Z+QUHtXiETNlHJbBDBWaKBruvVsgX0Gz6cf1Be2pItv32kcqmrrqg2x66PJXDGgb8zYJMjKX8p6L1zdjDNpOv8XeiPwD1v8F9MZRyKuHwfW5+w7m0nQS4KdA/Fj9P80LlIWKaDKoYTO5DenlhdWoJ2F2bTGoc1x8bCAk+Aup8c3VlTD3vmMi4TFXOmvsKH142uA83QWGHnPb9Vo7aQXqR33+5/tZjrCxFQc91ulKWKzux7ZrHwunSAyjAlgOIvHsJApWUnigi7fqdBxmt6rwqcn9kX7U7W5n4zojeUwoyzg0w83+1rbhXsBEIBiLjfRqvDdFFHFcd/UYKwbIxYqR2UQ4uD2U0BEOSx5qvRkCKMNuOYKW0NLMNoiiT4GjqhIDEGrQHqOoidbf03+n0FH8y0d7SNWg70nj4dkzUc3jSUgWKr9DS/g9MSf1ZbqyWD4xP48tlXfL1lJ7e3jErZOtLKbmxpX7DRHV7wh X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1717; 20:kpkGhRl0EHsEAEQq+iJvBaF4HpwzBS3AS7IXUt03dkaJz59yPvjkmEtVHEbXdsMSq7DAsWiKjRRz2eQK6p17fSoMF95TGAMOeN1njFawikAEq+ApGckdR00Jmmrr8NSVhz/r3u2eelYIfO/WKdrEkqGbOXDKpP7IZXEQvetct8FrW7AGVweUTuOf6ytYAnvuZ3TzMjLLKmmqCyasPyTz0ILRh22qiaDdw1vA9/oMi4m/L7RYb5zJ903Yfu8AZjULN3wcVx5Ck5hxZYQP+2+JNc1goIG8gS4CxhinrXCLKzEpOpkaZLdNJvkl0kn1fA1csIjfUAsG1p3nLLgv3WcOlZFcsOF0JxTa7MjnlJjhdFPY0NMbIS3fZ+8B7Tj4ZZF6nlVRTH/RLJCngW+0FPjFLs8GWc40uZe01n3M0JufXaZk5GkkxPUWRN2Wlz1F98OaK9LWUEcfFx2n8/Tty2dbDIluD05hSRh8HqeclgM/t03wrzWGSh2pPt6piqizmoWDHeojD9T/z/U4IjQmZHEjZ28MZUwTlYG21Nq8CfO7mcEjt7XEnUyqYUO8wMZJvQjrvldXrqgnQSfNKTHHzlIBCwah0P6fgRnE0QMHQCNK5lc= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(236129657087228)(148574349560750)(167848164394848)(158140799945019)(228905959029699); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(5005006)(8121501046)(2017060910075)(93006095)(10201501046)(3002001)(100000703101)(100105400095)(6041248)(20161123564025)(20161123562025)(20161123558100)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123555025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:BN3PR0701MB1717; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:BN3PR0701MB1717; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN3PR0701MB1717; 4:p4w+d6EXtzf0+oJGSvfzKW+HZ2s0SqGziiVCQT4q?= =?us-ascii?Q?8s6J+PJL+sGfb0YN7AhrNaUkJwLt+KH3egU+ItEl8zKC93FMFmX9qAm1XeOp?= =?us-ascii?Q?32gMwYbW0lovbg24vZHHOW3pvmIcZ1G0vVYlYV0a6vzA72/6JrWz550+eJOQ?= =?us-ascii?Q?Ccz/157MFw6+bhMsgWqPavWFAdiqOumS6cuX0fikz80w+Bz+T52BXq0bLiQ8?= =?us-ascii?Q?1Q3w1PvkckXITEtruKhnJpOQBn8mkryoi2eUXgxi59xUr+PH6ocZcQw9shUR?= =?us-ascii?Q?71/ziL12PpUEKYdlxcBGmnUM1/Cmt2SDttDfPH3fsE/ZfelY+HlTz8wh/zI7?= =?us-ascii?Q?QWOww/Zk87QJS/NWCkoEMk+vTH9761TodJ7kqLkgv2pZ1PN8B21EuBzBWDws?= =?us-ascii?Q?dPb73nSQDUM8X23y1NM+NF/BB+AaET4MUXMsuT4RQCxmQNE/ZzL3q1WPr539?= =?us-ascii?Q?smwX7zZptM31vqtJUKwIvgY+5EWFevkAK6pET0et0qrFaZyxi9r3V0EcBNo/?= =?us-ascii?Q?8Xy/1/95I1zFO0hGK8VecGxMnNR4LUamLVmJZX/bPDGU2+Vf+A1Lu0EW9ka7?= =?us-ascii?Q?4yjaCQKvOUk+02VVvQ3iJk9kOUl2fqukEg39QX4WsZysdFDw5elzGvfgjN8F?= =?us-ascii?Q?Onv32i5WzMlcvhNTrOhCeIJ+pr0liEfnh/pejAfFZ4s9Pbf9lNRHMo6S9RJn?= =?us-ascii?Q?LOADdcvh07ybXMnynwirdSFRFcWxYfV7WtnUDUlpuUJ0DSyusRQNA6i8Z6YA?= =?us-ascii?Q?OUGdQZQSQtjYwQshBdHrFH6FpXPr0QM6PCTzZT3NGFSU6SoE9+Q5vpVBARgc?= =?us-ascii?Q?71dAHjNDbJ+9EfnXrnpROVTFkTcmHmijYjniYuMi8+LShf7SBonnBdZBQ8MY?= =?us-ascii?Q?6RCejONW9N67Wq9Wy81oXHwaLzBiOUtr3kfqyWSz66fmjVfnWaxpVC/W7xTW?= =?us-ascii?Q?Yq8l4LdA44K0CNwj05oEchWgv5LUi6ThWGM6XilZywLwlqhMrK9C4QH8gBn5?= =?us-ascii?Q?3260GLMwGWbI82H6I/fSr48U4eG1DvTySDLOCZiW8ZuAHAHPDI5kJoqQ3Ixu?= =?us-ascii?Q?ibN9S2rKvv7glXI4qpvMFIHL4OftW49M0r/AJpKqxXhW4hYokoOC1nk34ng1?= =?us-ascii?Q?HB+osMdP/cCfaDvj71CUZWq47ssszOyYnF2XxVaNjTza8CB007izvcJpX4Sf?= =?us-ascii?Q?lNn398O9RLvGEedUReq0z6NM4p+aSgDHJYss+YiuWYlieZFTWDH5xP3kvtC9?= =?us-ascii?Q?9WJ46SWpxx7ZAXmCj3fPf7axBYNiXUqk0N63XADbnN7wcvMvxDxNF1qywoe6?= =?us-ascii?Q?bWSDR7UWS5aL/rzltLux939c0aTYYV6EC5Upl1cfpxP/?= X-Forefront-PRVS: 0365C0E14B X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(7370300001)(6009001)(39410400002)(39850400002)(39400400002)(39840400002)(39450400003)(13464003)(33656002)(76176999)(9686003)(54356999)(53936002)(50986999)(2906002)(66066001)(6246003)(50466002)(38730400002)(55016002)(47776003)(42186005)(6666003)(110136004)(53376002)(6496005)(6916009)(42882006)(6306002)(2950100002)(33716001)(575784001)(229853002)(5009440100003)(189998001)(6116002)(3846002)(5660300001)(23726003)(1076002)(81166006)(72206003)(478600001)(4001350100001)(83506001)(7736002)(305945005)(4326008)(7350300001)(8676002)(966005)(25786009)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:BN3PR0701MB1717; H:jerin; FPR:; SPF:None; MLV:sfv; LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN3PR0701MB1717; 23:BBfrNJy6ZyRoZkBn+SBCuqPif4DtsGWnSkmyJx9?= =?us-ascii?Q?mS9Fj+rWNbPDb+d/fRUc0Q2dGKJCYS6wxuiXnQMygyWHTeVwh2osdFfGPSAp?= =?us-ascii?Q?+PWMszNlAIIqqU3DoMbukCQar7yOmiDWKuuCdh/QWNLTQt4uL5Q7IqWNB5Dd?= =?us-ascii?Q?T5tv+BiYUtJpNnZVySsT1BTzuKcivhJkB+wPZgyXkWv4oVUEjxghZZNKkpxY?= =?us-ascii?Q?MhPu88BMvXtaXuIjnjJXgCuXa3vE6s2KRDARVgqr8fYRAPMsSDZcJ+ixRlCc?= =?us-ascii?Q?6RUgTBj9VJCVrIjuJPJdS8s0Knv1i8gmk37m99qPgbmAj9q6PjFA892oyzSH?= =?us-ascii?Q?gINerQSBP85oAhXvuKNw07ZFeWsPsonbuWPfvVvgLDpJdUCHHS95NL5SYetu?= =?us-ascii?Q?1wvi/UHc2cWRW71iy2UD/oydJX2XT+0f8pJxcUg2naIyNZAIjfgx035rE8Kh?= =?us-ascii?Q?z5tAAdr2j/Xzv9J6u8t9OHfp7vKvBOQwbdbBd9+DGzjNy94vyorR+P2XsJS9?= =?us-ascii?Q?iTgT7k7SPPRL/wEhG6II3ZeZnboj40pXXgx1kgwjQIhVdWLtD9rKi75F5aIo?= =?us-ascii?Q?Mdf8r32tJ008chtyHULbDkABYz0iFcefJvIrXVpUUISI7q5Uvn0zE+S/Cmhi?= =?us-ascii?Q?sKYEByVYJXczGa1DzuTS3fpHTiQtdFB+IfadMT4+1fJODWcrZGelcDBNQkoj?= =?us-ascii?Q?cWHlwoiQLmgW3FjzQRQHYurK/GEaToNaZTLOwoL1bT0y+tOjKfLvgKKWkB2n?= =?us-ascii?Q?245XYwiyO8mtTppS12jZdA+DZIfipo/rOL6dSZ3KBQyIaY7Mw0eAkL81/+Nz?= =?us-ascii?Q?D6UZjrp283GkpIShPmZtyOpWPtUTvLxkPbmT6zONC9LrDelJg+vt+8Vvr55D?= =?us-ascii?Q?fXWHwAGIzYsmh9HEq0zpwqQuC6Q13/ODNQxiSw6ygfpD6gtQDHm2LtZbNudJ?= =?us-ascii?Q?OHV2DMjPyQ+cQeHL3Db0PZLPTjBTcOlMs2klA/hux6IYJsMgQtROo8d5J6RA?= =?us-ascii?Q?EEaDL3WB9upGM+H1Lg4SrE+upDVGiCO+khUPsOixjKslU0xyR1q6Zca4u6GQ?= =?us-ascii?Q?LsINTbM/V2wdkpWHVyi5rWRzoe9tFCb1hmuXEcBr4qLVgN6/+aG+ho4IiWA2?= =?us-ascii?Q?Ti7dSERcMv8Hlf0mzgLNQ1N5N9CRrLXnoZucK3pCw8vz2TpVP8d0Rk+f6Ovb?= =?us-ascii?Q?1HsNju/evHVDXMQiITkLrWO8N4aXt/exqpRPa8MkP4qxbaZs4I9tuoBPwbw/?= =?us-ascii?Q?kXsHUB8QiV7RhxbqGBf3h7qNv44s0tvHRi4UODLokBtYw1vNq39FxHrFifve?= =?us-ascii?Q?g9KRPg2ld1l8DJ8ERZZqZcFpadFX5gr/7F5JxiI1HBSBz?= X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN3PR0701MB1717; 6:IrptzGXW3VbeuAX7YcpBQrzHfHl5gMTRBSY4rSdz?= =?us-ascii?Q?i2oVR8XkG4NGtvUEd4FrKv7+XDezs0awLCSth58/4sMn16h0t0mtHPCqIl4N?= =?us-ascii?Q?sDeN8i08/vsM8onhGX4qS5PLggogRuQwYqgG5HhQj1TbWcYxijmKSybAWknn?= =?us-ascii?Q?5DKva6KA6afgazOZ+1OIm/Kxn8i2YFm6XfalwlgFajzrtLdEI02GQ6d/QJkT?= =?us-ascii?Q?+k0nG4gFNuBwxb5Ozril7usHhcrXiEFG1FT41t+0XoRfFF2ec3bmNJyIBlwS?= =?us-ascii?Q?Vw1sLgA5ogrPFVCiH/tAFx5ZPY9DD9E0t+NrIa7q7tLU+VdOyK4YpgyXMo6r?= =?us-ascii?Q?4pyhcAWdYB0UJFBguxnWH6YqaKh8eAv0INZTMypvBvr/7oXNfQdh/LPaSvlS?= =?us-ascii?Q?YIpiVuJw2poS4gFmTBE/oQ7lpNVFiVqD7zDh+13vCVyiWYlNakBYlyleonCh?= =?us-ascii?Q?1ORm1ehmUYJ5qXOxI9QRl+zEfa4DGoIwWWeYTRZrJtVNIpVOUErl+/rgas6x?= =?us-ascii?Q?sG0vJEuD+UrMRALVqrFSZzn1W1Bkkmm2GiQ+gU6DMYDiWtCG1s3PsqI8pKLZ?= =?us-ascii?Q?NNaZfAvbWKqczMFZig2UBa1Op4OKZpvxf8ukcH2IHdek19jVi/kVlvPdbeLo?= =?us-ascii?Q?n8aYPN6/85hdcLP2b4G9U/voIswB2JKaHMZMMwRwTQX8tHZ///bpeCdcJWAs?= =?us-ascii?Q?VBefiH9/HKf5iaIMMyC1ubSDtAh14M/UzcX2UJRyfn1aRClQ4OEGtnLsJIhZ?= =?us-ascii?Q?0eQGc39e9u0U5ZL7BWxjXz9fCjyOxhtso3f2mgpeXSyEJLarncwX34v6Gr+0?= =?us-ascii?Q?rBh14hSf6AnfMcLKIetvEuKt5qdeYkskY5eotX+65kRTnUUrVLCiL/WKtvTr?= =?us-ascii?Q?aZ3AwZVPivLFheywsilUb2an9lhBSjAFKBQk9+sGLdBZhj0RyqG8Ymb/0K3l?= =?us-ascii?Q?Czo4suP/7oGjZrik9jotrlLAMw3EY+lNPUhAPl216XqvxFzoTGZodK3NnHB4?= =?us-ascii?Q?Z8o=3D?= X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1717; 5:8xPN/vun8Y+IZ5JazuNV7U8wkmmtwL4sJGHz8QnjbR66sZuwU3tYbpdPjtd2vlTnVl3rR+6+3uFn+joYsqg84hfdfjiMZHJ/rLZJPSgrHKQdmsM46zdK46GdMvCgGHXIkdiupjMgNVWunzgoAlw5+aXpXHo24FY5ENyUjgIbMx97NJJvxVDs0EcuUFP2djFOUJJgqwGCyae9WBwHnE1Bth6fKQUY3UV3+ILLjzxjvlRnPvu+JcLrx5i21oHnFmJqpCHitsvmBM98Mz7Bal7KSGGGH06g0QeSOKd32nazx46tT/eKy11RdaDI38sMSG7q8HENyxHw6MTB/4pu+Nr8b5RjLWyBVMLPf8usqyWjbWJYTMJ+cUODIwCee6iVSxMmfTAIV7Ph3fEb/CreLkJbJUrGQqLLOWZf4lX+bbhgqOD3gVOhXQSphNOQvO4UjdZEHa7z5vLJtvDD2j1AQIxcA2KdvzjuDA7XdPERgyaJFqo0t47BEBNIMb4OvBu0/8bw; 24:ObQ9yTInQ0akRxiCSrq45CAfOBavaOAelEjbLw0pjz/OfN01SA5Xzjeso844InDBKjOyyos8859X1mAET+ptYFeo7d2ya5ONf9QlrDt21Mc= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1717; 7:bG1lwe7Gzjpk3VTopnMDZn2Hwh2R8AEsbWOo0LON5ormL/i0pCFN7jby9H5clrhvi8I14RpqGbhnlr3kUyfmyz3osGDJlpgSUr0a/4AUZrdeN8tcUWMNkSuNH2NRJZ5l0pwEJB490ANoHE5Bf2/f24DRxlLUVJih+2Gmmi64O+uj81Rogv/hEo8XyC//QMXhxx5vnXbBkvyHU/KpX7AGjFWv90yubI5iEb9wMgWMMU5G/KapZ2218rMv9RU8ywMqC+3rGrFIKnEJZ+fhGvkk0LE+hk43sw/TC8y5gPx6QZUEACOp6ixLSCV0+PG/NqB57CSn5C6A1ifEYH892W8+znsxeAWszDiwB7B3Ooca/yvNN6lPr068Zj/aFxsLUUvJcRWcgLGgFg53u/xYCJl9P9EID+XdDAzzX8V1U9T23pXpMJgGz7HblgRAzOpRLRk3l7SyyHKq8Cws/YtjAbWEs+JZX9f2yysB7568sVJAQd3DI5aHVEeOUjXBqiQrjKuIjympo4OrCWuIjl1C2l6vRxIeD7l+WJAeVJ55xU3eEmFnLj48bnmiA8dUkgRTInFapS2wEBtPJGVSUSGJyT4DeZIGcKh1R6tN2/WtY1GS/KbdvH3jSvy7f8AaaT5LyAskjl1pg5TyEnz1hxgGhFOBAWTen32BdKnwVHI0Fq++TmpeHIFDF74OtBDr5Ka37lwFLOuk9TtyVns0Gv3O/39hTDWZIdX4Iy0kAI8HrUtHqDXM0uxepO2b2ZqWJYO8VDH1BOSoYe5NpoMzhtWEIFezAcdqYjHIk6F10Q5bcNeNYIY= X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jul 2017 08:30:10.0293 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR0701MB1717 Subject: Re: [dpdk-dev] [PATCH v4 1/7] service cores: header and implementation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Jul 2017 08:30:16 -0000 -----Original Message----- > Date: Fri, 7 Jul 2017 17:41:01 +0100 > From: Harry van Haaren > To: dev@dpdk.org > CC: thomas@monjalon.net, jerin.jacob@caviumnetworks.com, > keith.wiles@intel.com, bruce.richardson@intel.com, Harry van Haaren > > Subject: [PATCH v4 1/7] service cores: header and implementation > X-Mailer: git-send-email 2.7.4 > > Add header files, update .map files with new service > functions, and add the service header to the doxygen > for building. > > This service header API allows DPDK to use services as > a concept of something that requires CPU cycles. An example > is a PMD that runs in software to schedule events, where a > hardware version exists that does not require a CPU. > > The code presented here is based on an initial RFC: > http://dpdk.org/ml/archives/dev/2017-May/065207.html > This was then reworked, and RFC v2 with the changes posted: > http://dpdk.org/ml/archives/dev/2017-June/067194.html > > This is the fourth iteration of the service core concept, > with 2 RFCs and this being v2 of the implementation. Remove above info from the git commit. > > Signed-off-by: Harry van Haaren > > --- > doc/api/doxy-api-index.md | 1 + > lib/librte_eal/bsdapp/eal/Makefile | 1 + > lib/librte_eal/bsdapp/eal/rte_eal_version.map | 22 + > lib/librte_eal/common/Makefile | 1 + > lib/librte_eal/common/eal_common_lcore.c | 1 + > lib/librte_eal/common/include/rte_eal.h | 4 + > lib/librte_eal/common/include/rte_lcore.h | 3 +- > lib/librte_eal/common/include/rte_service.h | 383 ++++++++++++ > .../common/include/rte_service_private.h | 140 +++++ > lib/librte_eal/common/rte_service.c | 687 +++++++++++++++++++++ > lib/librte_eal/linuxapp/eal/Makefile | 1 + > lib/librte_eal/linuxapp/eal/eal_thread.c | 9 +- > lib/librte_eal/linuxapp/eal/rte_eal_version.map | 22 + > 13 files changed, 1273 insertions(+), 2 deletions(-) > create mode 100644 lib/librte_eal/common/include/rte_service.h > create mode 100644 lib/librte_eal/common/include/rte_service_private.h > create mode 100644 lib/librte_eal/common/rte_service.c > > diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md > index 3b83288..e2abdf4 100644 > --- a/doc/api/doxy-api-index.md > +++ b/doc/api/doxy-api-index.md > @@ -159,6 +159,7 @@ There are many libraries, so their headers may be grouped by topics: > [common] (@ref rte_common.h), > [ABI compat] (@ref rte_compat.h), > [keepalive] (@ref rte_keepalive.h), > + [service cores] (@ref rte_service.h), > [device metrics] (@ref rte_metrics.h), > [bitrate statistics] (@ref rte_bitrate.h), > [latency statistics] (@ref rte_latencystats.h), Fix the below mentioned documentation warning. +/export/dpdk.org/lib/librte_eal/common/include/rte_service.h:338: warning: argument 'enabled' of command @param is not found in the argument list of rte_service_set_stats_enable(int enable) +/export/dpdk.org/lib/librte_eal/common/include/rte_service.h:346: warning: The following parameters of rte_service_set_stats_enable(int enable) are not documented: + parameter 'enable' +/export/dpdk.org/lib/librte_eal/common/include/rte_service.h:349: warning: argument 'The' of command @param is not found in the argument list of rte_service_lcore_list(uint32_t array[], uint32_t n) +/export/dpdk.org/lib/librte_eal/common/include/rte_service.h:367: warning: The following parameters of rte_service_lcore_list(uint32_t array[], uint32_t n) are not documented: + parameter 'n' command to reproduce: ./devtools/test-build.sh -j8 x86_64-native-linuxapp-gcc+shared x86_64-native-linuxapp-gcc+debug > } DPDK_17.08; > diff --git a/lib/librte_eal/common/Makefile b/lib/librte_eal/common/Makefile > index f2fe052..942f03c 100644 > --- a/lib/librte_eal/common/Makefile > +++ b/lib/librte_eal/common/Makefile > @@ -41,6 +41,7 @@ INC += rte_eal_memconfig.h rte_malloc_heap.h > INC += rte_hexdump.h rte_devargs.h rte_bus.h rte_dev.h rte_vdev.h > INC += rte_pci_dev_feature_defs.h rte_pci_dev_features.h > INC += rte_malloc.h rte_keepalive.h rte_time.h > +INC += rte_service.h rte_service_private.h IMO, We don't need to expose rte_service_private.h to application. If you agree, add the following or similar change diff --git a/drivers/event/sw/Makefile b/drivers/event/sw/Makefile index 857a87cc5..442652e93 100644 --- a/drivers/event/sw/Makefile +++ b/drivers/event/sw/Makefile @@ -36,6 +36,8 @@ LIB = librte_pmd_sw_event.a # build flags CFLAGS += -O3 CFLAGS += $(WERROR_FLAGS) +CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common/include/ + # for older GCC versions, allow us to initialize an event using # designated initializers. ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y) diff --git a/lib/librte_eal/common/Makefile b/lib/librte_eal/common/Makefile index 942f03cdd..ea6c1f8f1 100644 --- a/lib/librte_eal/common/Makefile +++ b/lib/librte_eal/common/Makefile @@ -41,7 +41,7 @@ INC += rte_eal_memconfig.h rte_malloc_heap.h INC += rte_hexdump.h rte_devargs.h rte_bus.h rte_dev.h rte_vdev.h INC += rte_pci_dev_feature_defs.h rte_pci_dev_features.h INC += rte_malloc.h rte_keepalive.h rte_time.h -INC += rte_service.h rte_service_private.h +INC += rte_service.h diff --git a/test/test/Makefile b/test/test/Makefile index 42d9a49e2..b603b6563 100644 --- a/test/test/Makefile +++ b/test/test/Makefile @@ -152,6 +152,7 @@ SRCS-y += test_version.c SRCS-y += test_func_reentrancy.c SRCS-y += test_service_cores.c +CFLAGS_test_service_cores.o += -I$(RTE_SDK)/lib/librte_eal/common/include/ SRCS-$(CONFIG_RTE_LIBRTE_CMDLINE) += test_cmdline.c SRCS-$(CONFIG_RTE_LIBRTE_CMDLINE) += test_cmdline_num. > > GENERIC_INC := rte_atomic.h rte_byteorder.h rte_cycles.h rte_prefetch.h > GENERIC_INC += rte_spinlock.h rte_memcpy.h rte_cpuflags.h rte_rwlock.h > diff --git a/lib/librte_eal/common/eal_common_lcore.c b/lib/librte_eal/common/eal_common_lcore.c > index 84fa0cb..0db1555 100644 > --- a/lib/librte_eal/common/eal_common_lcore.c > +++ b/lib/librte_eal/common/eal_common_lcore.c > @@ -81,6 +81,7 @@ rte_eal_cpu_init(void) > > /* By default, each detected core is enabled */ > config->lcore_role[lcore_id] = ROLE_RTE; > + lcore_config[lcore_id].core_role = ROLE_RTE; > lcore_config[lcore_id].core_id = eal_cpu_core_id(lcore_id); > lcore_config[lcore_id].socket_id = eal_cpu_socket_id(lcore_id); > if (lcore_config[lcore_id].socket_id >= RTE_MAX_NUMA_NODES) { > diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h > index abf020b..4dd0518 100644 > --- a/lib/librte_eal/common/include/rte_eal.h > +++ b/lib/librte_eal/common/include/rte_eal.h > @@ -61,6 +61,7 @@ extern "C" { > enum rte_lcore_role_t { > ROLE_RTE, > ROLE_OFF, > + ROLE_SERVICE, > }; > > + > +#define SERVICE_F_REGISTERED 0 > + > +/* runstates for services and lcores, denoting if they are active or not */ > +#define RUNSTATE_STOPPED 0 > +#define RUNSTATE_RUNNING 1 > + > +/* internal representation of a service */ > +struct rte_service_spec_impl { > + /* public part of the struct */ > + struct rte_service_spec spec; > + > + /* atomic lock that when set indicates a service core is currently > + * running this service callback. When not set, a core may take the > + * lock and then run the service callback. > + */ > + rte_atomic32_t execute_lock; > + > + /* API set/get-able variables */ > + int32_t runstate; > + uint8_t internal_flags; > + > + /* per service statistics */ > + uint32_t num_mapped_cores; > + uint64_t calls; > + uint64_t cycles_spent; > +} __rte_cache_aligned; > + > +/* the internal values of a service core */ > +struct core_state { Change to lcore_state. > + /* map of services IDs are run on this core */ > + uint64_t service_mask; > + uint8_t runstate; /* running or stopped */ > + uint8_t is_service_core; /* set if core is currently a service core */ > + uint8_t collect_statistics; /* if set, measure cycle counts */ > + > + /* extreme statistics */ > + uint64_t calls_per_service[RTE_SERVICE_NUM_MAX]; > +} __rte_cache_aligned; > + > +static uint32_t rte_service_count; > +static struct rte_service_spec_impl *rte_services; > +static struct core_state *cores_state; > +static uint32_t rte_service_library_initialized; > + > +int32_t rte_service_init(void) > +{ > + if (rte_service_library_initialized) { > + printf("service library init() called, init flag %d\n", > + rte_service_library_initialized); > + return -EALREADY; > + } > + > + rte_services = rte_calloc("rte_services", RTE_SERVICE_NUM_MAX, > + sizeof(struct rte_service_spec_impl), > + RTE_CACHE_LINE_SIZE); > + if (!rte_services) { > + printf("error allocating rte services array\n"); > + return -ENOMEM; > + } > + > + cores_state = rte_calloc("rte_service_core_states", RTE_MAX_LCORE, > + sizeof(struct core_state), RTE_CACHE_LINE_SIZE); > + if (!cores_state) { > + printf("error allocating core states array\n"); > + return -ENOMEM; > + } > + > + int i; > + int count = 0; > + struct rte_config *cfg = rte_eal_get_configuration(); > + for (i = 0; i < RTE_MAX_LCORE; i++) { > + if (lcore_config[i].core_role == ROLE_SERVICE) { > + if ((unsigned int)i == cfg->master_lcore) > + continue; > + rte_service_lcore_add(i); > + count++; > + } > + } > + > + rte_service_library_initialized = 1; > + return 0; > +} > + > +void rte_service_set_stats_enable(int enabled) IMO, It should be per service i.e rte_service_set_stats_enable(const struct rte_service_spec *spec, int enable) > +{ > + uint32_t i; > + for (i = 0; i < RTE_MAX_LCORE; i++) > + cores_state[i].collect_statistics = enabled; > +} > + > +/* returns 1 if service is registered and has not been unregistered > + * Returns 0 if service never registered, or has been unregistered > + */ > +static inline int > +service_valid(uint32_t id) { > + return !!(rte_services[id].internal_flags & > + (1 << SERVICE_F_REGISTERED)); > +} > + > +uint32_t > +rte_service_get_count(void) > +{ > + return rte_service_count; > +} > + > +struct rte_service_spec * > +rte_service_get_by_id(uint32_t id) > +{ > + struct rte_service_spec *service = NULL; > + if (id < rte_service_count) > + service = (struct rte_service_spec *)&rte_services[id]; > + > + return service; > +} > + > +struct rte_service_spec *rte_service_get_by_name(const char *name) > +{ > + struct rte_service_spec *service = NULL; > + int i; > + for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) { > + if (service_valid(i) && > + strcmp(name, rte_services[i].spec.name) == 0) { > + service = (struct rte_service_spec *)&rte_services[i]; > + break; > + } > + } > + > + return service; > +} > + > +const char * > +rte_service_get_name(const struct rte_service_spec *service) > +{ > + return service->name; > +} > + > +int32_t > +rte_service_probe_capability(const struct rte_service_spec *service, > + uint32_t capability) > +{ > + return service->capabilities & capability; > +} > + > +int32_t > +rte_service_is_running(const struct rte_service_spec *spec) > +{ > + const struct rte_service_spec_impl *impl = > + (const struct rte_service_spec_impl *)spec; > + if (!impl) > + return -EINVAL; > + > + return (impl->runstate == RUNSTATE_RUNNING) && > + (impl->num_mapped_cores > 0); > +} > + > +int32_t > +rte_service_register(const struct rte_service_spec *spec) > +{ > + uint32_t i; > + int32_t free_slot = -1; > + > + if (spec->callback == NULL || strlen(spec->name) == 0) > + return -EINVAL; > + > + for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) { > + if (!service_valid(i)) { > + free_slot = i; > + break; > + } > + } > + > + if ((free_slot < 0) || (i == RTE_SERVICE_NUM_MAX)) > + return -ENOSPC; > + > + struct rte_service_spec_impl *s = &rte_services[free_slot]; > + s->spec = *spec; > + s->internal_flags |= (1 << SERVICE_F_REGISTERED); > + > + rte_smp_wmb(); > + rte_service_count++; > + > + return 0; > +} > + > +int32_t > +rte_service_unregister(struct rte_service_spec *spec) > +{ > + struct rte_service_spec_impl *s = NULL; > + struct rte_service_spec_impl *spec_impl = > + (struct rte_service_spec_impl *)spec; > + > + uint32_t i; > + uint32_t service_id; > + for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) { > + if (&rte_services[i] == spec_impl) { > + s = spec_impl; > + service_id = i; > + break; > + } > + } > + > + if (!s) > + return -EINVAL; > + > + rte_service_count--; > + rte_smp_wmb(); > + > + s->internal_flags &= ~(1 << SERVICE_F_REGISTERED); > + > + for (i = 0; i < RTE_MAX_LCORE; i++) > + cores_state[i].service_mask &= ~(1 << service_id); > + > + memset(&rte_services[service_id], 0, > + sizeof(struct rte_service_spec_impl)); > + > + return 0; > +} > + > +int32_t > +rte_service_start(struct rte_service_spec *service) > +{ > + struct rte_service_spec_impl *s = > + (struct rte_service_spec_impl *)service; > + s->runstate = RUNSTATE_RUNNING; > + rte_smp_wmb(); > + return 0; > +} > + > +int32_t > +rte_service_stop(struct rte_service_spec *service) > +{ > + struct rte_service_spec_impl *s = > + (struct rte_service_spec_impl *)service; > + s->runstate = RUNSTATE_STOPPED; > + rte_smp_wmb(); > + return 0; > +} > + > +static int32_t > +rte_service_runner_func(void *arg) > +{ > + RTE_SET_USED(arg); > + uint32_t i; > + const int lcore = rte_lcore_id(); > + struct core_state *cs = &cores_state[lcore]; > + > + while (cores_state[lcore].runstate == RUNSTATE_RUNNING) { > + const uint64_t service_mask = cs->service_mask; > + for (i = 0; i < rte_service_count; i++) { > + struct rte_service_spec_impl *s = &rte_services[i]; > + if (s->runstate != RUNSTATE_RUNNING || > + !(service_mask & (1 << i))) > + continue; > + > + /* check if this is the only core mapped, else use > + * atomic to serialize cores mapped to this service > + */ > + uint32_t *lock = (uint32_t *)&s->execute_lock; > + if ((s->spec.capabilities & RTE_SERVICE_CAP_MT_SAFE) || > + (s->num_mapped_cores == 1 || > + rte_atomic32_cmpset(lock, 0, 1))) { > + void *userdata = s->spec.callback_userdata; > + > + if (cs->collect_statistics) { > + uint64_t start = rte_rdtsc(); > + s->spec.callback(userdata); > + uint64_t end = rte_rdtsc(); > + s->cycles_spent += end - start; > + cs->calls_per_service[i]++; > + s->calls++; > + } else > + s->spec.callback(userdata); > + > + if ((s->spec.capabilities & > + RTE_SERVICE_CAP_MT_SAFE) == 0 && > + s->num_mapped_cores > 1) How about computing the non rte_atomic32_cmpset() mode value first and using in both place i.e here and in the top "if" loop const int need_cmpset = (s->spec.capabilities & RTE_SERVICE_CAP_MT_SAFE)... if (need_cmpset || rte_atomic32_cmpset(lock, 0, 1)) .. if (need_cmpset) rte_atomic32_clear().. > + rte_atomic32_clear(&s->execute_lock); > + } > + } > + > + rte_smp_rmb(); > + } > + > + lcore_config[lcore].state = WAIT; > + > + return 0; > +} > + > +int32_t > +rte_service_lcore_count(void) > +{ > + int32_t count = 0; > + uint32_t i; > + for (i = 0; i < RTE_MAX_LCORE; i++) > + count += cores_state[i].is_service_core; > + return count; > +} > + > +int32_t > +rte_service_lcore_list(uint32_t array[], uint32_t n) > +{ > + uint32_t count = rte_service_lcore_count(); > + if (count > n) > + return -ENOMEM; > + > + if (!array) > + return -EINVAL; > + > + uint32_t i; > + uint32_t idx = 0; > + for (i = 0; i < RTE_MAX_LCORE; i++) { > + struct core_state *cs = &cores_state[i]; > + if (cs->is_service_core) { > + array[idx] = i; > + idx++; > + } > + } > + > + return count; > +} > + > +int32_t > +rte_service_set_default_mapping(void) > +{ > + /* create a default mapping from cores to services, then start the > + * services to make them transparent to unaware applications. > + */ > + uint32_t i; > + int ret; > + uint32_t count = rte_service_get_count(); > + > + int32_t lcore_iter = 0; > + uint32_t ids[RTE_MAX_LCORE]; > + int32_t lcore_count = rte_service_lcore_list(ids, RTE_MAX_LCORE); > + > + for (i = 0; i < count; i++) { > + struct rte_service_spec *s = rte_service_get_by_id(i); > + if (!s) > + return -EINVAL; > + > + /* if no lcores available as services cores, don't setup map. > + * This means app logic must add cores, and setup mappings > + */ > + if (lcore_count > 0) { > + /* do 1:1 core mapping here, with each service getting > + * assigned a single core by default. Adding multiple > + * services should multiplex to a single core, or 1:1 > + * if services == cores > + */ > + ret = rte_service_enable_on_lcore(s, ids[lcore_iter]); > + if (ret) > + return -ENODEV; > + } > + > + lcore_iter++; > + if (lcore_iter >= lcore_count) > + lcore_iter = 0; > + > + ret = rte_service_start(s); IMO, we don't need to start the service if lcore_count == 0. How about moving the "if (lcore_count > 0)" check on top of for the loop and exist from the function if lcore_count == 0. > + if (ret) > + return -ENOEXEC; > + } > + > + return 0; > +} > + > +static int32_t > +service_update(struct rte_service_spec *service, uint32_t lcore, > + uint32_t *set, uint32_t *enabled) > +{ > + uint32_t i; > + int32_t sid = -1; > + > + for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) { > + if ((struct rte_service_spec *)&rte_services[i] == service && > + service_valid(i)) { > + sid = i; > + break; > + } > + } > + > + if (sid == -1 || lcore >= RTE_MAX_LCORE) > + return -EINVAL; > + > + if (!cores_state[lcore].is_service_core) > + return -EINVAL; > + > + if (set) { > + if (*set) { > + cores_state[lcore].service_mask |= (1 << sid); > + rte_services[sid].num_mapped_cores++; > + } else { > + cores_state[lcore].service_mask &= ~(1 << sid); > + rte_services[sid].num_mapped_cores--; > + } > + } > + > + if (enabled) > + *enabled = (cores_state[lcore].service_mask & (1 << sid)); > + > + rte_smp_wmb(); > + > + return 0; > +} > + > +int32_t rte_service_get_enabled_on_lcore(struct rte_service_spec *service, > + uint32_t lcore) > +{ > + uint32_t enabled; > + int ret = service_update(service, lcore, 0, &enabled); > + if (ret == 0) > + return enabled; > + return -EINVAL; > +} > + > +int32_t > +rte_service_enable_on_lcore(struct rte_service_spec *service, uint32_t lcore) > +{ > + uint32_t on = 1; > + return service_update(service, lcore, &on, 0); > +} > + > +int32_t > +rte_service_disable_on_lcore(struct rte_service_spec *service, uint32_t lcore) > +{ > + uint32_t off = 0; > + return service_update(service, lcore, &off, 0); > +} > + > +int32_t rte_service_lcore_reset_all(void) > +{ > + /* loop over cores, reset all to mask 0 */ > + uint32_t i; > + for (i = 0; i < RTE_MAX_LCORE; i++) { > + cores_state[i].service_mask = 0; > + cores_state[i].is_service_core = 0; > + cores_state[i].runstate = RUNSTATE_STOPPED; > + } > + for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) > + rte_services[i].num_mapped_cores = 0; > + > + rte_smp_wmb(); > + > + return 0; > +} > + > +static void > +set_lcore_state(uint32_t lcore, int32_t state) > +{ > + /* mark core state in hugepage backed config */ > + struct rte_config *cfg = rte_eal_get_configuration(); > + cfg->lcore_role[lcore] = state; > + > + /* mark state in process local lcore_config */ > + lcore_config[lcore].core_role = state; > + > + /* update per-lcore optimized state tracking */ > + cores_state[lcore].is_service_core = (state == ROLE_SERVICE); > +} > + > +int32_t > +rte_service_lcore_add(uint32_t lcore) > +{ > + if (lcore >= RTE_MAX_LCORE) > + return -EINVAL; > + if (cores_state[lcore].is_service_core) > + return -EALREADY; > + > + set_lcore_state(lcore, ROLE_SERVICE); > + > + /* ensure that after adding a core the mask and state are defaults */ > + cores_state[lcore].service_mask = 0; > + cores_state[lcore].runstate = RUNSTATE_STOPPED; If worker core can call rte_service_lcore_add() then add rte_smp_wmb() here. Applies to rte_service_lcore_del() as well.