From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM01-BN3-obe.outbound.protection.outlook.com (mail-bn3nam01on0084.outbound.protection.outlook.com [104.47.33.84]) by dpdk.org (Postfix) with ESMTP id 8CC685921 for ; Fri, 23 Sep 2016 12:29:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=oxcxLqqTtghRX5xbqZVT1qtqzefUDglc0zQZbJaBivU=; b=hckYt06We0HSUGxAmaLgEZcRGIW10prvP/YNbC11ksJKgK+QYPMtKr9m56WZLstqbFg0KXqGONR7vhVaMJovZeCIiW2rI8OK1AL69q4VRm8Colk80A4Cfzx6I90mCeO0j2A6+xloZz2EVh/qlRxcLzEDM5PkG0VLsm23gGfXQfM= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jerin.Jacob@cavium.com; Received: from localhost.localdomain (122.171.42.11) by BN3PR0701MB1720.namprd07.prod.outlook.com (10.163.39.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.639.5; Fri, 23 Sep 2016 10:29:26 +0000 Date: Fri, 23 Sep 2016 15:59:10 +0530 From: Jerin Jacob To: "Ananyev, Konstantin" CC: "Kulasek, TomaszX" , "dev@dpdk.org" Message-ID: <20160923102908.GA9141@localhost.localdomain> References: <1473691487-10032-1-git-send-email-tomaszx.kulasek@intel.com> <1473691487-10032-2-git-send-email-tomaszx.kulasek@intel.com> <2601191342CEEE43887BDE71AB9772583F0B583F@irsmsx105.ger.corp.intel.com> <3042915272161B4EB253DA4D77EB373A14F1A294@IRSMSX102.ger.corp.intel.com> <20160919160630.GA18610@localhost.localdomain> <2601191342CEEE43887BDE71AB9772583F0B8F33@irsmsx105.ger.corp.intel.com> <20160921082956.GA5358@localhost.localdomain> <2601191342CEEE43887BDE71AB9772583F0BA159@irsmsx105.ger.corp.intel.com> <20160922095947.GA6420@localhost.localdomain> <2601191342CEEE43887BDE71AB9772583F0BA7E0@irsmsx105.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <2601191342CEEE43887BDE71AB9772583F0BA7E0@irsmsx105.ger.corp.intel.com> User-Agent: Mutt/1.6.2 (2016-07-01) X-Originating-IP: [122.171.42.11] X-ClientProxiedBy: MA1PR01CA0041.INDPRD01.PROD.OUTLOOK.COM (10.164.116.141) To BN3PR0701MB1720.namprd07.prod.outlook.com (10.163.39.19) X-MS-Office365-Filtering-Correlation-Id: aa5b41e3-3664-4015-46fe-08d3e39c7f3a X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1720; 2:OzNDjyLttxDO0yqkMFqv5xega7YmbOabO8FwQWWuDGmUw2XFERvPNLrtkYW6Uj4U4V+kJad/Yk/mV+eOCA6rpGHzv4zXsRpiY8mz5R2O0iuxH3jTkSNvS/9LnndTw8vpSFg/5p4yLvE5UmsBx2JUmnG5XqKMbGxuqXyB56DMjX/kznYcZbJUQnPEkhyHYJdn; 3:H2AqcL0CLKIsbKUh1chjvFX4/dpehbPF3W8G7lAqmwnV05tRjriYNpjoBh9CKtzG/fIbRNfF+X6WyDGmswFSFzZ3d2J1D0iSpbkrV1wO3XafJQZ16z9bkyUDBNueBdBv; 25:Xx7tp6eZ7HVleak7T1PQgYnROjtJ3QIeDQlMW+2gWoT4Oy2vywFRPrIG1PJ7Na1eDEvCbtl2vMaBEmagZ+0ZLKuoEkzraYcqCY4ikyFbhebW5U5IborNISoMYR5GbStech/D02vtAauemrfV4ooAZpNSvO2jJBV1X8x1zRoSyOjqROQ+Fy6hJhpIFpphbi38/2R+5jMpLNpZEJuAm6b0uDDRMhtHDKfvlFlEq8EsxvSbhapcLxH3fj33/FWRs5hjJyPXcNo4sQF+zM2UEJgGCZstHB2da78aPCdOXmbRhSz83BZ8ULQnwVY4iCOJHBSMZcWzCaVK0l5AATi7UZGC71GmgnWNRPXKstilvCwrbgG3bwweybu4aaG4tOzjgV9yl+oAjzYM6YPiDVpW15S2NAkhh4s3c4wBbCXkxyYUHQw= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BN3PR0701MB1720; X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1720; 31:LQNqkH0EOzulwzHwC8OJnxH+koL1r+T2ip9ZWbFhUBV0B1RPkq0Cw/GTgJIUCMhhoVOwWU2AulOS3eb6JNnAJZmW4bRIy0zJeXfnv47BhwTea7uCWinTIlIBOAqELm0eMPM9KpvzMhb+Dt04ybe82nHqejZDUvije9rC9UA4wWAhRQOoQTwrTzDwsCfixDGkVt/MrV2oMeYOG5w8oudj7Kw4eBlXwiGZy2wEA9kmgHM=; 20:sUib7qIW8EEmawHoaejIb45r4vofHOKEcIBndn4+Rbo/Z9TyhftgecljwWQW5fk9/EibxWPd+eW2DkWeCHDksr/vLljuhDOlc2VTyjo5sO40z7HNJz+CVXDlgV5CHrT4+kRPRsW699cKyEEBhHdN7AUPTlWsmjjcPNP3cOm4nIKYF3WFiycOjBzaB3FHZoj4mT3Gwss/CNRK3X97SBquEKRkffzAHyX+j28bgID3wDrng3jELIx9Fp57B1sk6k8TcR6PT5F20WFm8qsYhC0vf6fdmgWfI8BcJp+GToSpuMRZ/g0oEDCiQgBa7vtEqPqYBr6NqZwLoy6+UHtAr7hfazbudYs7HBPLN0GzKbWL2Nz5VnDHHaQ+k3Adz9QUNWdWRX9HIMufA/hQbixgF8J5/K26JjTxb+tMFn41AUipKV2/5FBFUhGVY8Lm/yQnq+pifqPadbplX4lJ1DE3anBiBcS7UbmrAXUdOq0bsdAqZdl5oquvtJhCgw11r7SQgTFTetCXoQYtsBzjB0pLXgqT7oSrCXmPQkTDn7KHz5wiul/03aHeP4KvHIvT7kuBK1YuDz75reNBHVqLFyquFvQWd9KrNTaw4CAlLTgsJlcMhV0= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(158342451672863); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046); SRVR:BN3PR0701MB1720; BCL:0; PCL:0; RULEID:; SRVR:BN3PR0701MB1720; X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1720; 4:sVppCn66s2GKu7a+fqkAkCvuCp0mk++x7lVf0EMs8SJFtviMzfprZPmL2+WR6Mgjn6gQgLxvjQboXkyGMc61LnMGI0VVHUUgK6MwYBNGZYPX/aep/Q5ph3nusceTAeYUru8wh43lMsAc8YjJTsvOFoHkRpgqpQSimEHoHzozQRx3oSxUjL3+ugnw5l+yEVJ2HHnHX/5pNV4pIZEIy37LlVSz1uRhopnPQ6bLdZeNnez9FyV5I+uMP3a3TYxy0JpOKn+YoYboUdCyoQzEM1kZvsOtGqwYfoX8DBKouDgtvBguwu3cgOSjK3Nf2a5Zq21nH07y8uIZwKiF4Qgcg+JzPrrX/3rrXbzTukawr5N7ogedOi3bSwlX2yUQR2MylBImGSrCExbIraU+MbGo/O9H9f1dpz5UtTMkU2gqoMTUwH86Ie7TpY4mOm8KEQwLUE9R X-Forefront-PRVS: 0074BBE012 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(6069001)(7916002)(189002)(199003)(24454002)(33656002)(2950100001)(19580395003)(97736004)(92566002)(15975445007)(77096005)(1720100001)(4001350100001)(4326007)(110136003)(23726003)(189998001)(50466002)(76176999)(54356999)(5660300001)(66066001)(1076002)(50986999)(101416001)(83506001)(9686002)(68736007)(61506002)(47776003)(2906002)(81156014)(81166006)(42186005)(8676002)(97756001)(7846002)(305945005)(93886004)(7736002)(46406003)(106356001)(6116002)(15395725005)(105586002)(586003)(3846002)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:BN3PR0701MB1720; H:localhost.localdomain; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN3PR0701MB1720; 23:ipN+xfgomGZ3G3e4kgw5YyUySVBJaSHgyA19RrN?= =?us-ascii?Q?iwa+hDm9R8Bk2sUKav+cwYSSvRZkA/h/3LLZbRNvHpeUjJZEQZTPtsqg9V4S?= =?us-ascii?Q?Fm8UZIEf+zVxRT+bwaufAhuXqpRlguiVuygCB6ysJm+muhkMzHheHrfYjr0H?= =?us-ascii?Q?36S+/1mTaBNiwd+9ayyfXv1o1HfiI6bLaO9PQmaymcOFIhYJJFfnTKCEFQz/?= =?us-ascii?Q?WENbput+VKSasvvOLL/WYa/Reyo3eig1x99FvznnD1Qd4hzJBsHkhcef/xa2?= =?us-ascii?Q?36I7jiLLJeOgT5PkNQfiK7LezrenfyMvsV+vg4hDm9u7o327xoAXs/Bp9093?= =?us-ascii?Q?tk+QebYsQVWkjg/PzrXZ85UuC5Cxmf1OgaUAYDH7DZ6KR4eJ1dNiOOc49DtS?= =?us-ascii?Q?uotVX9CAW138nPLRszbkuceTlkANqhwLWacx6AK34bkBzu3CIKCqm8oLFwQ0?= =?us-ascii?Q?uP4BnVzJkHOC8p+0oWhZ5NcOL1SD75e6IAi585nuPU5kXj2zo52mQpf0FuUa?= =?us-ascii?Q?VXFckoLM3Clb29Vn4qZpJwx7A9RFmDOsqE7Ot6xSkBnRLgI60u7Sj4zpkB38?= =?us-ascii?Q?7OlMua0yjmi5E2K1dqAGF+prIf63xcLX6USj8vQ6DKY1qVnaRGyOjAqP3zMw?= =?us-ascii?Q?c6tEyHwHZg+sJaLVVgplpcAq6TO0LvSYsE98KmEUwitK+uu/e97zUr+t3Adw?= =?us-ascii?Q?Qe+T7E/gc3XwEB4ZDGYJfcTh7J42jZBX6htFCc4XknwNqX0fhSPBx4qOjO3T?= =?us-ascii?Q?/kJu+8Qb8HdMhN2YxMbMu50yM3/5JJBVwMNNzJTcuIyOkjSciRTQvTz+NETs?= =?us-ascii?Q?szJA5ivikbpzsjCAezZpdFYOT6FPILm59oTk7dOtnSVWraGobJ+IHC+thrjE?= =?us-ascii?Q?06yVrJXKBMX4dqNuSBFBJsXRq/YuUMJ3yLOszsByN/K5zbXDlg+XzuN55P3h?= =?us-ascii?Q?xJlHDRRqnA3mnZMUIezFwnT0gefMIMKuDPbL6yEAyq+uWJLZzbEPILOeFyNa?= =?us-ascii?Q?LHshfCxexqogbhlxGluxuw5xfNXe86Is9QBwoL5pQ7wnMcBT2gJXPSKCnWKB?= =?us-ascii?Q?pxs2OQEk/hh6+A8tZiMMrIH2Rfb5rNtQ0iWm4FtYvFPV1vqgX7eNt4f0SKJ5?= =?us-ascii?Q?D3qqKa4p/RWuB6SjPT9jMgrZCMIei5B5qGHpp9zs8ZZA3sc7tPOrE5kEWHTn?= =?us-ascii?Q?VWQJ4PMTdPPTZetbke6QR3FPN69dXGNVOS2ribTNWnUl8Zk2JftRgF2Q/Q0F?= =?us-ascii?Q?RQWyvaNpkihVkXMNGIvo4m/Jpu9VZHKREIjT1C+yS?= X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1720; 6:Ybc2Y47rbx2Mlqz3DX4z2n+bSLxzEBcH6xYMsAXEAvPM5sLq3OCjzTHUVWv7IHYUPSr5MbSj/xU4cIV6gMSir9igmQjPo5SH3Daxe83DpW1QnUlFAsN6sZx/FwXm4zXptaGaxg7gVKvL3/i4zYRLXJBHi4VbPaaRV6w3U7F9FZH+1+ilczxekZFchVOtrpNoCut9lw7rW2CpaxX01ZRc2I7Pe46HInP5z4wNhtuCH7xnEWhkcKqKSJuACMsOS1ha0svT0+yHNMQOz2yfnIbby09xROUDiIoIeHlxxLplp30=; 5:0DNbjPt6TuDR5nzxDq+yvDgvTsnUzijBEVWV8x8xEThASWz1onYC9ZJ+rmeHHVQVLVfYeJNon1ZEIRcYXYowhcUYUMMr4gTbrmkFT96LqIpm6EHSQP5fkKWiiIHPzLpY3ayi6889nllwYRHdNbdafw==; 24:KUAyOoLTwY+IpEZUhKS6iWx+aAgXTVHnIMGRKg7DeXFqY7dbdFP524iT/RL9SyzbzWTjV9O56/9Di7kemUSgPvMK0zaFtzJ8orgC/VkxSQo=; 7:eaNVQQLEjKi+zzRQBAhJ5YzWq1ZBwJkJzW9TswjaGXmoh4TgEOx+xssUXPzHA0QJ/1yuTJ/zsiUDJHQroPQd3T3haKq8318SeUripi9apBVv+wRCEKVXN6d+fZbaghNnCP4hTUjtP6xZ9/YzOSJpUOsEJq9Scqh5amKYPM0tkB3Aup8oemyf5M3mxz5RDoWOokhuD/mkrC+jW/YOdfBZiR+wmIZVmLh9YpTHwN9QXWcOoyha8QbyaqXZRlUs5zto6vh4vaCY3XmgCxrwqOOfVxzoO/+Qm2nWaHXNsyk3jnVUnNufTHxXYhA2vBjvEa1v SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Sep 2016 10:29:26.1947 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR0701MB1720 Subject: Re: [dpdk-dev] [PATCH v2 1/6] ethdev: add Tx preparation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Sep 2016 10:29:31 -0000 On Fri, Sep 23, 2016 at 09:41:52AM +0000, Ananyev, Konstantin wrote: Hi Konstantin, > Hi Jerin, > > > > > Hi Konstantin, > > > > > > > > Hi Jerin, > > > > > > > > > > > > > Hi Jerin, > > > > > > > > Hi Konstantin, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [...] > > > > > > > > > > > > > > > > + > > > > > > > > > +#ifdef RTE_ETHDEV_TX_PREP > > > > > > > > > > > > > > > > Sorry for being a bit late on that discussion, but what the > > > > > > > > point of having that config macro (RTE_ETHDEV_TX_PREP ) at all? > > > > > > > > As I can see right now, if driver doesn't setup tx_pkt_prep, > > > > > > > > then nb_pkts would be return anyway... > > > > > > > > > > > > > > > > BTW, there is my another question - should it be that way? > > > > > > > > Shouldn't we return 0 (and set rte_errno=ENOTSUP) here if > > > > > > > > dev->tx_pk_prep == NULL? > > > > > > > > > > > > > > > > > > > > > > It's an answer to the Jerin's request discussed here: > > > > > > > http://dpdk.org/ml/archives/dev/2016-September/046437.html > > > > > > > > > > > > > > When driver doesn't support tx_prep, default behavior is "we > > > > > > > don't know requirements, so we have nothing to do here". It > > > > > > > will simplify > > > > > > application logic and improve performance for these drivers, I think. Catching this error with every burst may be problematic. > > > > > > > > > > > > > > As for RTE_ETHDEV_TX_PREP macro, suggested by Jerin in the > > > > > > > same thread, I still don't think It's the best solution of the > > > > > > > problem > > > > > > described by him. I have added it here for further discussion. > > > > > > > > > > > > > > Jerin, have you something to add? > > > > > > > > > > > > Nothing very specific to add here. I think, I have tried to > > > > > > share the rational in, http://dpdk.org/ml/archives/dev/2016- > > > > > > September/046437.html > > > > > > > > > > > > > > > > Ok, not sure I am fully understand your intention here. > > > > > I think I understand why you propose rte_eth_tx_prep() to do: > > > > > if (!dev->tx_pkt_prep) > > > > > return nb_pkts; > > > > > > > > > > That allows drivers to NOOP the tx_prep functionality without > > > > > paying the price for callback invocation. > > > > > > > > In true sense, returning the nb_pkts makes it functional NOP as > > > > well(i.e The PMD does not have any limitation on Tx side, so all > > > > packets are _good_(no preparation is required)) > > > > > > > > > > > > > What I don't understand, why with that in place we also need a > > > > > NOOP for the whole rte_eth_tx_prep(): > > > > > +static inline uint16_t > > > > > +rte_eth_tx_prep(uint8_t port_id __rte_unused, uint16_t queue_id __rte_unused, > > > > > + struct rte_mbuf **tx_pkts __rte_unused, uint16_t nb_pkts) { > > > > > + return nb_pkts; > > > > > +} > > > > > + > > > > > +#endif > > > > > > > > > > What are you trying to save here: just reading ' dev->tx_pkt_prep'? > > > > > If so, then it seems not that performance critical for me. > > > > > Something else? > > > > > > > > The proposed scheme can make it as true NOOP from compiler > > > > perspective too if a target decided to do that, I have checked the instruction generation with arm Assembly, a non true compiler > > NOOP has following instructions overhead at minimum. > > > > > > > > # 1 load > > > > # 1 mov > > > > if (!dev->tx_pkt_prep) > > > > return nb_pkts; > > > > > > Yep. > > > > > > > > > > > # compile can't predict this function needs be executed or not so > > > > # pressure on register allocation and mostly likely it call for > > > > # some stack push and pop based load on outer function(as it is an > > > > # inline function) > > > > > > > > > Well, I suppose compiler wouldn't try to fill function argument registers before the branch above. > > > > Not the case with arm gcc compiler(may be based outer function load). > > Ok, so for my own curiosity (I am not very familiar with the ARM arch): > gcc generates several conditional execution instructions in a row to spill/fill > function arguments registers, and that comes at a price at execution time if condition is not met? Yes. That's what I see(at least for gcc 5.3 + arm64 back-end case) when I was debugging external mempool function pointer performance regression issue. The sad part is, I couldn't see any gcc option to override it. > > > The recent, external pool manager function pointer conversion reduced around 700kpps/core in local cache mode(even though the > > function pointers are not executed) > > > > > > > > > > > > > return (*dev->tx_pkt_prep)(dev->data->tx_queues[queue_id], tx_pkts, > > > > nb_pkts); > > > > > > > > # 1 branch > > > > if (unlikely(nb_prep < nb_rx)) { > > > > # bunch of code not executed, but pressure on i cache > > > > int i; > > > > for (i = nb_prep; i < nb_rx; i++) > > > > rte_pktmbuf_free(pkts_burst[i]); > > > > } > > > > > > > > From a server target(IA or high-end armv8) with external PCIe based > > > > system makes sense to have RTE_ETHDEV_TX_PREP option enabled(which > > > > is the case in proposed patch) but the low end arm platforms with > > > > - limited cores > > > > - less i cache > > > > - IPC == 1 > > > > - running around 1GHz > > > > - most importantly, _integrated_ nic controller with no external PCIE > > > > support > > > > does not make much sense to waste the cycles/time for it. > > > > cycle saved is cycle earned. > > > > > > Ok, so it is all to save one memory de-refrence and a comparison plus branch. > > > Do you have aby estimation how badly it would hit low-end cpu performance? > > > > around 400kpps/core. On four core systems, around 2 mpps.(4 core with > > 10G*2 ports) > > So it is about ~7% for 2x10G, correct? > I agree that seems big enough to keep the config option, > even though I am not quite happy with introducing new config option. > So no more objections from my side here. Thanks. That's was for very low end cpus. So even if it is for high-end cpu case, event if it calls for 100kpps drop/core, The Cavium configuration like 96 cores + >200G case will at least 9.6mpps worth of cycles drop. > Thanks > Konstantin > > > > > > The reason I am asking: obviously I would prefer to avoid to introduce > > > new build config option (that's the common dpdk coding practice these days) unless it is really important. > > Practice is something we need to revisit based on the new use case/usage. > > I think, the scheme of non external pcie based NW cards is new to DPDK. > > > > > > > > > > > > > Since DPDK compilation is based on _target_, I really don't see any > > > > issue with this approach nor it does not hurt anything on server target. > > > > So, IMO, It should be upto the target to decide what works better for the target. > > > > > > > > Jerin > > > > > > > > > From my point of view NOOP on the driver level is more than enough. > > > > > Again I would prefer to introduce new config option, if possible. > > > > > > > > > > Konstantin > > > > >