From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM01-SN1-obe.outbound.protection.outlook.com (mail-sn1nam01on0053.outbound.protection.outlook.com [104.47.32.53]) by dpdk.org (Postfix) with ESMTP id 0166E2030 for ; Thu, 14 Dec 2017 05:54:28 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=fex1MiyY3J/gRiR1j2y7mTFkYOoZsAGXJHDdr9/SHio=; b=JWWeeDGP8LyLbOxUDuQ4RdEFHAA9n506YL9+/u6TgfvuxXMCKS6zr8ABuFEI71aHRY6kN+pZ4ahK2zNAzsIyPBSCgGn3Wm1rqG1sNFBPkL1jq2ycpqNgyZj9KbTU02gUj9ewRjKPcgZc/rxPSYun1JvRqC84daXT8luLOyInlIk= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jerin.JacobKollanukkaran@cavium.com; Received: from jerin (122.167.65.15) by BN3PR07MB2513.namprd07.prod.outlook.com (10.167.4.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.302.9; Thu, 14 Dec 2017 04:54:26 +0000 Date: Thu, 14 Dec 2017 10:24:09 +0530 From: Jerin Jacob To: Konstantin Ananyev Cc: dev@dpdk.org, shahafs@mellanox.com Message-ID: <20171214045408.GA843@jerin> References: <1512140112-13067-1-git-send-email-konstantin.ananyev@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1512140112-13067-1-git-send-email-konstantin.ananyev@intel.com> User-Agent: Mutt/1.9.1 (2017-09-22) X-Originating-IP: [122.167.65.15] X-ClientProxiedBy: PN1PR0101CA0066.INDPRD01.PROD.OUTLOOK.COM (10.174.150.156) To BN3PR07MB2513.namprd07.prod.outlook.com (10.167.4.138) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: aa96c897-e894-46ba-4c4c-08d542aec12d X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(5600026)(4604075)(4534020)(4602075)(4627115)(201703031133081)(201702281549075)(2017052603307); SRVR:BN3PR07MB2513; X-Microsoft-Exchange-Diagnostics: 1; BN3PR07MB2513; 3:ptBcYEd4vJvnn4laKyVzdId5q70l+AkPi6GCTolF9MRl4z/CHalAd2gPQXR47plJbRCt/k1J5hBnERW2E3sDZyojZeDzfGV52XsGMwz5PH0+RgVeewf4sHDv+7ajFxRsdtIT8Dgx0vu+XqFOjCPoEtzCFgIj9+uxGCzr6y+pN/a99TRIMM81eNn2UKSzF/lNMQZSkynYuGJ13i/1pNGaFtJCD6fsQ8Ci/gNwnZxWgROpprVvYNrsutXGogq2outz; 25:zLxpME384MAtbAJ3cb6X4ZaNU4IM8XJTXtxbeeDOuahvx8rpS1n8Vj8RHVJyR5zm2qbPZoly5kyrwJanl2B90aBNmEPTIE/Hqsp6Q8IUHyFHedVId7bIAWSdZMLUycuMkURQwFhE+cIznN9AQOjJLlaDOeKtOwpK19RGG3Pq6mfucLSCNqy5AzjKVxQv8Ax907ig4xLVv5m7jVK2bE2xU3lmPDhkke6er4wIlB/kBwujPzvyJspyQrEm6ZUR5MJ/BqNmxWSpiIQzU7kAdNTqIqthKD9G4a5NdX5XIlRlZJipJqMj4Qk97ry1GJNd6hjhYH2n5fZTSw5KeKsc70Nzog==; 31:e1u1xJUEthKSNlGsEPeRWjMDR4AJD52qulxSqPVCoczXGL9qgn+FSGwaEL3p8gVaMKAR+SZ6Vc3XlmFpIxqHJjJXg2/pIQ4FtgKmAhOqOEvgpG1g9G7WgmrisRHJuvOzelgq5pEzjrnbR78bzGmdszURI8RTEmpQO/Bh9q2Zycy+Yp2D0KJu0xya5sFcDg1pZcFatQT1DGLWgr0wu42v2VuTIJQWsPNzAsrB4Thcsvo= X-MS-TrafficTypeDiagnostic: BN3PR07MB2513: X-Microsoft-Exchange-Diagnostics: 1; BN3PR07MB2513; 20:aCb0/SdelGGrSgr+b1COc0L4UFrOcGqpmeB2cLwBDNBuihDHedDaWcV2Aq2BbGbTiIt+WSKLzCOIrVOxJsZYpsIvJEFft2szCDAi/g1QeYO5opROaTvECOagd4ElAKRK7QZP8ZsjLit+c4yDfC2yMsrrhn3cxIbOpXqt3AMakiFkIPKXFRacwpXBkto6u50OZZROy2+Q+7KOV00iWSUHQLRZ2lj4Ek1j0NO5e3BBWf/MX3uP7UD9L0VGQ1DdsezRLm+u49nMMoOae2tDAUNiRF0BocDKErAy1dLc9n8YGih2SJdh3NWWywx0mguaJ5HLnhoKY9meNobVv/T6+txGGcC4XRtlnsEE+dilborufRjgF1hKm2e/OufIMuIDwCniIL2nvhNEbXTJMOQpCGJXURb/rbN8jr5pfuYq3YUyFhhFwkd0sFMMguIdGJR6pP3MjFWrVPhJV7GIUIVXx2NraXi51Ga1nnO76AVRqepOew8fRPONaLNQjh/HvT3ldA9Z68bnqQ9c9kUikflWMZHRHw8qE62dzHsq7Bq6Q41oyOZbA9DcX1Vf+8kGV/vGZ8OX76x9dIQJ1XDQGpP9A5ksf6slYkj6ZA9DP9WjqxTmyVk= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(228905959029699)(17755550239193); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040450)(2401047)(5005006)(8121501046)(10201501046)(3002001)(93006095)(3231023)(6041248)(20161123562025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123555025)(20161123564025)(20161123560025)(6072148)(201708071742011); SRVR:BN3PR07MB2513; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:BN3PR07MB2513; X-Microsoft-Exchange-Diagnostics: 1; BN3PR07MB2513; 4:gyCSR1dFBPKQwf0kvNZ+7gWURVjyum9V+MBgjYmTLR5tW7z3wpEsUXGEOz7CDhkLVLxZamG+z2NnKs9B25Z/ceLQeKSM3U8/8GFY7Fatb3OZQSUsf5ew0na/oTeEaCCmZ/r5s1xxJK+GrRr3SRXVNCo9OqJRCuRXRaXNM2kuzL7M60BheWXcuatLyr7PHrirYThIT5QR91WQFF/QG5alrYzTvAQ1PVCJ7tI0NiX1rhcazMrwaq217Vgd6Xbtlk0ABeRE9AONjb2xKw7HgmsSMM8HWdndCr1KldI93bUf01tJU1nGztV6rWEUwWS6UoT5kJYEvY6OaxwAAGm3S7B1MEH1Dvx4ApPsUFF9wDRMFzA= X-Forefront-PRVS: 05214FD68E X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(346002)(396003)(39860400002)(366004)(376002)(13464003)(199004)(189003)(6666003)(2950100002)(42882006)(33716001)(52116002)(50466002)(6916009)(33656002)(66066001)(47776003)(9686003)(6306002)(6496006)(33896004)(76176011)(386003)(97736004)(58126008)(16526018)(2906002)(55016002)(59450400001)(8676002)(7736002)(5660300001)(305945005)(81166006)(8936002)(81156014)(16586007)(316002)(229853002)(68736007)(6116002)(72206003)(3846002)(23726003)(478600001)(53376002)(1076002)(6246003)(53936002)(25786009)(4326008)(83506002)(105586002)(106356001)(966005)(18370500001)(217873001); DIR:OUT; SFP:1101; SCL:1; SRVR:BN3PR07MB2513; H:jerin; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN3PR07MB2513; 23:HQm0k/DMuGLKvN1FZIIr+nejVpuzipSjs8HJu1Up2?= =?us-ascii?Q?zEqTT+386A9vpuFafnDYLRYee9l5uSsOtIeqZNt3XTqvL21uMrfWrL757+4k?= =?us-ascii?Q?mqJE8sxLvqCg3jM59RoR+ujkZ9OwL07edIIxNoveYo5Hh3eR2CeVug2vPTdG?= =?us-ascii?Q?B8Ph8vQuuyjaDlk5jXR3fPsbJO8BzZMEFNqvivsLoT2WZwY/ijV+ci+SeEbc?= =?us-ascii?Q?HfzCBZxfHPNDxOj2FJo3mawdoTo6v9mSmfCCsMntO6ZcYduVKlQr4/CskGId?= =?us-ascii?Q?Pi8/1P4KHu6h4O/z3vc0g9LHD/SBHD3m4WBgL+f4F3QloJnQUBiV6BvwJm6u?= =?us-ascii?Q?xs+IUIRqz9Tmybd6vU0iaPgxQsg2fs2IJHbpm5scrq5XULLU89C9wzTDQt3V?= =?us-ascii?Q?n9Q5cjS8UAQwXUxUsDSdm4dd3ONUXCiOMqhmR4IpHyM5mRWkJ9Uw6wBaMZXA?= =?us-ascii?Q?Pb3Nx7N44nkO3fYjD9iBdHBhfUEn1f3qLhrCb2Vvbe3twWSz5slHe8Oyzdwa?= =?us-ascii?Q?NDFKDEgNu8z1GVLlBX/ncO+d+KzRumf+id/xvCY2+kBhxciYsb+4N3UAbW76?= =?us-ascii?Q?hZik0Nt2eGHgdtkvBSsFsVKrFYXW9lZotnkYcSoFsYtM8NFdu/OibfBYpwm+?= =?us-ascii?Q?q800VW04D3D4XQQl09OHungC4r8uW8jlAKvG2EDtXTaPRhCsxSUR0Udc3Apt?= =?us-ascii?Q?QKLYB4o+lnsjBO0IvFgFDsfHuMDOhzsjSgJj+7az0Rwcw9wxzmVdcxd5umtU?= =?us-ascii?Q?PZJGr0ELOWyPayTkATaArGE5VuoGXq98JcXHM3D0G83npfCtirSh6TxWxHrk?= =?us-ascii?Q?ploS5XUmnsZV2l+5gcpeimKS3TsaPcxpgyRCcbTWT7Kl3gSLyowdWJnf1/r9?= =?us-ascii?Q?olmkU0102XNfGnJTfexkZdYgLbMwxjUZ6pabNJhffL+lo1as01J9MpKSv68E?= =?us-ascii?Q?hyFL1HO63wTvWCIAj7SJ8aWZ2/O2EzvwZjeMziOWycB+HJYhT0gALtnCgOzj?= =?us-ascii?Q?HUmZ9/Fg9+eej3L9nYl/bfBq1jAkvsV/PEo4taqLJYiJRuCC0iiHaL4ajLUj?= =?us-ascii?Q?EkWgJ7HY3HgUMOltFqz/fMxfJKtQK6wTCkM7k1tJTjQh0w3eo9GHlvInshto?= =?us-ascii?Q?PMrSdvRt1KPlFFKkfR+rERSG+QLHCZsDB5olC9mnUaUyLHsT43d+O9Jc80yu?= =?us-ascii?Q?EVRKSfk0+idafW2RQv5NS7oBNQNLMBLp79QepIfiF2FRhBe7O4NCj5qlLA12?= =?us-ascii?Q?WN6olBxPFNOqeN1HJ//y+hKWcRLPBlaCqKXvsbtoFaldTY2TDEaUcWetDTdX?= =?us-ascii?Q?rB6gDBR1LEWJkt7m2I8mcA=3D?= X-Microsoft-Exchange-Diagnostics: 1; BN3PR07MB2513; 6:EBs02PIdb2XSFqgwIPE41YE76P7ISkW85m/ilUxtNjVXq5zYfPhvmhainhr9S73giXMQTlQAMGFE4ShRBupEYM8cx4ZAxLQlfvxuDxzSIxmSctUgJn88IVNWItR6xrlzw38g6YJIoMtDPUSLj1f1msbQYcH2Bi5w99hvTqmWcC4bCjIKLs4H/n31zNBLxyNQi+noZwke3ZwKYgsPxbkNtmoQqHjejUZ+hQIPeY8PwgPuvDLIlKsHZoOATtCSiXvmHlUJS5u0zi+H02hDLMCjYcKDWWEwtKM7x8ZLGBdwXnRYjSncyEWKGbWR2BVQ0L/pl98zLk9g/TrCATr9kd8TKCzXkxE6nXw/FE0GNbUfLbw=; 5:f5RqLNOXWL6yaPoN53YB3LHsTASD9qla6pYRS1kBBw2t1Tnvc3fGpEFx5792HXAbH89C31PLUHSr9S1plhzGbeZ8hdfDc/n55afEravx0rihwpiocj5mI3TGoLp1LFLO2tycSmNzFL+m05dzOuCemCPN8xnq6yuftHOzOV7VQZQ=; 24:5ts0pGmxf8YVZffq3kJnMY9s9BsDyoJw9lm96YJ+FGHoY4FrloY6z0D8Cq2zSoPy2BFos5Yu6HA5P7fmxqreZTZWvD+/XQWURGbjlBrZRVo=; 7:u8jNGx1Z/AqLl5GNazMK0PzK76LkFV5LDtUpXJBUtpKZuWaSA6FCzKPra/WTkeJBCV419ma3A53LcMS4yJX54F57bSk4Qm097fJsACFqeFnNbFTmEB6fp1CwH5gluLEkQ+0wKbLKiVQYVz61wC5LqhDhLXkWtTglnrm42qkRkumM4+D8P3ixXUuhXsrDke021dqdKTdsWJiMR/YmW9zP1Ttft3TIR+lIh+/uoG+voCE86dJx0s8tExPFuXmHndUT SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Dec 2017 04:54:26.2225 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: aa96c897-e894-46ba-4c4c-08d542aec12d X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR07MB2513 Subject: Re: [dpdk-dev] [RFC PATCH 0/3] ethdev: few changes in rte_ethdev layer X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Dec 2017 04:54:29 -0000 -----Original Message----- > Date: Fri, 1 Dec 2017 14:55:12 +0000 > From: Konstantin Ananyev > To: dev@dpdk.org, dev@dpdk.org > CC: Konstantin Ananyev > Subject: [dpdk-dev] [RFC PATCH 0/3] ethdev: few changes in rte_ethdev layer > X-Mailer: git-send-email 1.7.0.7 > Hi Konstantin, > The series introduces 2 main changes: > > 1.Introduce a separate data structure (rte_eth_queue_local) > to store local to given process (i.e. no-shareable) information > for each configured rx/tx queue. > Memory for that structure is allocated/freed dynamically during > rte_eth_dev_configure(). > Reserve a space for queue specific (rx|tx)_pkt_burst(), > tx_pkt_prepare() function pointers inside that structure. > Move rx/tx callback related information inside that structure. > That introduces a change in current behavior: all callbacks for > un-configured queues will be automatically removed. > Also as size of struct rte_eth_dev changes that patch is an ABI breakage, > so deprecation notice for 18.05 is filled. > Further suggestions how to introduce the same functionality > without ABI breakage are welcome. Are we doing this to enable, Tx/Rx queue specific burst functions to invoke based on new TX/RX queue specific offloads framework. Meaning, if a offload configured for the given queue, driver can select a different function pointer for Rx/Tx burst functions. > > 2. Make it safe to remove rx/tx callback at runtime. > Right now it is not possible for the application to figure out > when it is safe to free removed callback handle and > associated with it resources(unless the queue is stopped). > That's probably not a big problem if all callbacks are static > hange through whole application lifetime) > and/or application doesn't allocate any resources for the callback handler. > Though if callbacks have to be added/removed dynamically and > callback handler would require more resources to operate properly - > then it might become an issue. > So patch #2 fixes that problem - now as soon as > rte_eth_remove_(rx|tx)_callback() completes successfully, application > can safely free all associated with the removed callback resources. > > Performance impact: > If application doesn't use RX/TX callbacks, then the tests I run didn't > reveal any performance degradation. > Though if application do use RX/TX callbacks - patch #2 does introduce > some slowdown. > > To be more specific here, on BDW (E5-2699 v4) 2.2GHz, 4x10Gb (X520-4) > with http://dpdk.org/dev/patchwork/patch/31864/ patch installed I got: > 1) testpmd ... --latencystats=1 - slowdown < 1% > 2) examples//l3fwd ... --parse-ptype - - slowdown < 1% > 3) examples/rxtx_callbacks - slowdown ~8% > All that in terms of packet throughput (Mpps). We tried on an arm64 machine; We got following result on host and guest using l3fwd. Note: +ve number means "Performance regression with patches" -ve number means "Performance improvement with patches" Relative performance difference in percentage: % difference on host (2 x 40G) pkt_sz 1c 2c 4c 8c 64 1.41 0.18 0.51 0.29 72 0.50 0.53 0.09 0.19 128 0.31 0.31 0.42 0.00 256 -0.44 -0.44 0.00 0.00 512 1.94 1.94 0.01 0.01 1024 0.00 0.01 0.00 0.01 1518 0.02 0.01 0.02 0.02 % difference on guest (2 x 40G) pkt_sz 1c 2c 4c 8c 64 5.78 2.30 -2.45 -0.01 72 1.13 1.13 1.31 -4.29 128 1.36 1.36 1.54 -0.82 256 2.02 2.03 -0.01 -0.35 512 4.05 4.04 0.00 -0.46 1024 0.39 0.38 0.00 -0.74 1518 0.00 0.00 -0.05 -4.20 I think, it is because we added more code under the RTE_ETHDEV_RXTX_CALLBACKS and it is enabled by default. I think, the impact will vary based on architecture and underneath i-cache, d-cache and IPC. I am just thinking, How we can avoid such impact? How about, # Remove RTE_ETHDEV_RXTX_CALLBACKS option # Hook Rx/TX callbacks under TX/RX offload flags # ethdev layer gives helper function to invoke callbacks to driver. But, don't invoke from common code # When application needs the callback support, it configured the given RX/TX queue offloads # If the Rx/TX callback configure by the application then driver calls the helper functions exposed by the common code to invoke RX callbacks. Jerin > > Ability to safely remove callbacks at runtime implies > some sort of synchronization. > Even I tried to make it as light as possible, > probably some slowdown is unavoidable. > Of course instead of introducing these changes at rte_ethdev layer > similar technique could be applied on individual callback basis. > In that case it would be up to callback writer/installer to decide > does he/she need a removable at runtime callback or not. > Though in that case, each installed callback might introduce extra > synchronization overhead and slowdown. > > Konstantin Ananyev (3): > ethdev: introduce eth_queue_local > ethdev: make it safe to remove rx/tx callback at runtime > doc: ethdev ABI change deprecation notice > > doc/guides/rel_notes/deprecation.rst | 5 + > lib/librte_ether/rte_ethdev.c | 390 ++++++++++++++++++++++------------- > lib/librte_ether/rte_ethdev.h | 174 ++++++++++++---- > 3 files changed, 387 insertions(+), 182 deletions(-) > > -- > 2.13.5 >