From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-VE1-obe.outbound.protection.outlook.com (mail-eopbgr20040.outbound.protection.outlook.com [40.107.2.40]) by dpdk.org (Postfix) with ESMTP id B507B1B645; Mon, 23 Oct 2017 19:24:27 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=lHtKjg2c4J1gIgcBP0102yjsOAtmwjvODkHnik/Hxa8=; b=nFBeTI0KvPY2aLCJkbM886dpd/pmnSQLiFyrgy9VsSHZnUCiXggZQW27OEUFZku0lVDTf3XaaLV6VEHRxHynI9eKJ/WFmebE5vAlTnOBpzw02rTH7JEZsJsXCM5UMbMaLaHYNRpl0aZiXzQ7CbvZZkMyPZC4tPbguxISBPjEdVY= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; Received: from yongseok-MBP.local (209.116.155.178) by AM5PR0501MB2034.eurprd05.prod.outlook.com (2603:10a6:203:1a::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.156.4; Mon, 23 Oct 2017 17:24:23 +0000 Date: Mon, 23 Oct 2017 10:24:09 -0700 From: Yongseok Koh To: =?iso-8859-1?Q?N=E9lio?= Laranjeiro Cc: Sagi Grimberg , adrien.mazarguil@6wind.com, dev@dpdk.org, stable@dpdk.org, Alexander Solganik Message-ID: <20171023172408.GA19208@yongseok-MBP.local> References: <20171022080022.13528-1-yskoh@mellanox.com> <20171022220103.GA18571@minint-98vp2qg> <20171023075014.bzubdh2y37s3dirk@laranjeiro-vm> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20171023075014.bzubdh2y37s3dirk@laranjeiro-vm> User-Agent: Mutt/1.7.2 (2016-11-26) X-Originating-IP: [209.116.155.178] X-ClientProxiedBy: MWHPR1301CA0027.namprd13.prod.outlook.com (2603:10b6:301:29::40) To AM5PR0501MB2034.eurprd05.prod.outlook.com (2603:10a6:203:1a::20) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2eafd56d-202e-4db8-4038-08d51a3ae876 X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(48565401081)(4534020)(4602075)(4627075)(201703031133081)(201702281549075)(2017052603199); SRVR:AM5PR0501MB2034; X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2034; 3:02KwATZ4SqLCy0wY87ERKBlx/GoCWZWtPrcy782kfEqZ/XbcWgdL0okpaP0Guj6TXLISxkV1QnNW9WQttkinNN+0Weq8XLIiyugb6KmlQKAa1d8G2/3Snu7l8lsusAqwGwyVHE7C8+1wpKpm4So3njK3BdCsgKjQd230/GQodyUuP1H9IQkbibzCie3ZWf18HAop+S4A7HHDRxCVUZTxVb6tu+j2ElSk4ep1GGc6yOtZHuvWT3mCt8G1vE5bV/96; 25:GhzAfM4jd0uFlmvMsZWC3OkmI2pIcAemNR/oGjpBpR0GDgm7S/snPKLSufyjVgwATq+uS+3yF3SRLoDrZ8F9XXmpntkhNJSmmnOA2odK6FLlbXP6g/CVsDtyolvMdoj0w4ey0uk30m6XcM6O4ftMCVH2onATjX4UFP/dA8BD2k9+KJxPPaCerHcBz990VOAiI94paLH9ZFWh/eqSZQ3kSbO9+A0UjP/m4NZyHgUy8ZLCpumkI8TcT6itXqLz+oAYVVfPxaGgTFn+hXfVmMXhFnE/bb2SqIDFeDLfzlHzpHfSpz99tPH20VOmVg9XwyxG+R8CEp88Cqrmi87H4L3QCg==; 31:XF3AMeHOkyyK0BgziHbOdfVdyafS1hbFe5PEw2yR5rt92WCMGzg75j+EaFrHu+MYFfoGGww44NCai+wRY2kINWKHn51ds56oxIThpyDI2Do8tIWZjBdfL06QgpBwvq5VWchcVAbr4EN17S+wyQH5QsDJL9UQm6G3b9YIioWYd8RZ+6p1aConhvycinUC943Qw9pxS9dPbqXAB/QDm1owcP/CRWTuXMfjh5VArtflTRo= X-MS-TrafficTypeDiagnostic: AM5PR0501MB2034: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2034; 20:YT2bh/vATtHl70ZYn5RtKC1hYnpak0hiHxGbZtsWnd06WO1YeAcYfjINum3FE/woQlm+8Q5CR7D26x+FAtDvRzdzq0ce8dIkvQOzD3YtdvIfZmnN+PD+dd/eOBDDiqT+Ry42QXEJLxIKoJuUoZH+WtL0X6gqx1kNjG9PWH+mZz85RneyrIWTQ+GtxDheigWnhodhb+1R+JAAr8WSp5YslpBNRoEbfmWiYnuv3R4hzvLoql+/wi3DDa7HBPStRFSYHIRp6I/JKsERHscmgF9VhCdMg3oLIXDLI5S3FcJP1T36k8MwGAFkfkBpNOr0M4wmVxJeN+gbQEcaA9FeMMD0H4kHE9ARklfP74nLJqtp6giCrC89HTQV8NOmDDbIakda2/vR21/FiiGpg8qU44gnLddj40rvDHvhscghOyqTFHCp1oyVSNkORzV7dpdWGTbBYnwTRCukt4zTsghwPkEmObV5VAdl79sYLL9zFFr/wEbPcG4DZX845eTicOE79H1L; 4:8aGsYNGYa9cFAiZugLYb7ukBnOGkfHz8RKnyN+zjBz1sPm69P7bwKO8ibqZNeG3wGwPjjPiURnP9zdMP2TJ3LoHviow1h1ZStgR9ntds68NARFGhZ6NoB1pve/cxe7Wy05zj6cJxJSRwGu5aMH7U/lObnxQiVYPZsJL9dhWUak1pKVa1+AZznp6unG5rIl3JNViWMddafc8BypIM6JuXBuyOGwDrAEnwPywacf6fkWhCi+tZ6XSzZqxjoCeEBNYP X-Exchange-Antispam-Report-Test: UriScan:; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(5005006)(8121501046)(93006095)(93001095)(100000703101)(100105400095)(3231020)(3002001)(10201501046)(6055026)(6041248)(20161123562025)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123560025)(20161123555025)(20161123558100)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:AM5PR0501MB2034; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:AM5PR0501MB2034; X-Forefront-PRVS: 046985391D X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(346002)(376002)(39860400002)(51694002)(24454002)(199003)(189002)(189998001)(101416001)(2950100002)(2870700001)(6916009)(106356001)(105586002)(50466002)(83506002)(33656002)(54356999)(8936002)(98436002)(8676002)(16526018)(81166006)(50986999)(76176999)(81156014)(229853002)(23756003)(58126008)(39060400002)(25786009)(54906003)(4326008)(93886005)(53936002)(47776003)(6246003)(86362001)(68736007)(6666003)(6116002)(2906002)(97736004)(6506006)(7736002)(3846002)(1076002)(5660300001)(55016002)(478600001)(305945005)(9686003)(316002)(66066001)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:AM5PR0501MB2034; H:yongseok-MBP.local; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?iso-8859-1?Q?1; AM5PR0501MB2034; 23:CEl/QtIf/z40jLM0rb9zouM79Kmd34Bq3UgPS?= =?iso-8859-1?Q?25COjKter3R2Ypa2XpN/pCeK291IFBu4AQU3JUq5KyV95UvxuapWtLIbQg?= =?iso-8859-1?Q?bU4LCKqCpRkqrxyAoUfT9X5BbgnFpfbUs4WpPb6cOJd2JcMWEvSl9mFa+1?= =?iso-8859-1?Q?43zFLU4s1+F2Fpcq/gaUSvpNgEte/xlxp4Y/RHzGu2G6dO/9x/Ep/Tfsfd?= =?iso-8859-1?Q?BpuHsp+ttnRK0dv9rrEMSijWLOEnu84MF0O5dPOMN4kxj07bsIFkpvB4b+?= =?iso-8859-1?Q?pwIqq1PGAQKqcl5RlcDma0sUWGFCvByj1SRo8iUhcyY//umg/9ETg119ue?= =?iso-8859-1?Q?6XbL7rZGs9uIzD8KREF60N3l9OMMnGqYok5xxE6wwd3q0Ygv1Tb8rregAc?= =?iso-8859-1?Q?Zws5GADSOuUrK0NnpAZHVZcxYtuQxcR8/5wcw1nF7eTdpP2Ww9VpZ2BZZi?= =?iso-8859-1?Q?7tbT/Wm1MdFH5XR1bfl3dgToTHSm9/3TztTvKbCZ9W9jzbicuoBoTybsfK?= =?iso-8859-1?Q?8u6/wiaDj46jpBhjdcqq5iZhjA3MS5qFdkV1B58YjFgHAuePnF0JsLciVn?= =?iso-8859-1?Q?sLFq1em4jgiD/vuh8cyYjbNCxEwuS1p/mRBqO04D9oYY0I8zYThcF0X+dj?= =?iso-8859-1?Q?mKpHwX9tKISapBZbcy8z/FKcsAreobcKTDfZ9PzVK7OeEhzRp08vD7NSPx?= =?iso-8859-1?Q?RFVTa1wzhG/xyWAonipKOvTRvk+16vEB85+PvG60anGxBSsko2efpl5Jx2?= =?iso-8859-1?Q?iYlXa+K39icDi6y8gaKtOmocVDVUfaEev5/lz6D2ukL3l2NTLNrV7cUAci?= =?iso-8859-1?Q?sDuktkZd7k6tB9BF0HfJgUK7Mqj+lo8jc8O0ohcaSRCsHsWnN6NFiagBL2?= =?iso-8859-1?Q?QqS3Mh2V1Y8F+YltggILq4GHcMhQAnHO4JtAfT8FaSODN0LydHujQZJ/3G?= =?iso-8859-1?Q?SFcgeQSWZuQQRv+HRmZfWeNs3x8hGAH9e9uAJC7CPjdicSW36dCeb0LAXj?= =?iso-8859-1?Q?5bOxF+4i8SkrkS9xqOaj99lzHzTpStG/6Mqo6Wwk5wzsYf/w0K7zEJs59l?= =?iso-8859-1?Q?sOL5nmLQP1msihlIJBFmarlAWbJfUnuGYuUcPJs0xQzMQaPGDXfajbE8Pg?= =?iso-8859-1?Q?CU76rQBsR2Pgr9UTC2tKP+cMvHbZMYwGtCoX8/wX29nZO+1FLmqctuWpud?= =?iso-8859-1?Q?nyEnk59R+VW75nXNFAwomW9utvSEj3GN+VtWO61m/K89v3Hbm+A96gUOoD?= =?iso-8859-1?Q?RPe+zRUhf0Smi+24gkYyaCHN4+x8tFnPpjCPq9UgUSmZXyKTzyN86jusUg?= =?iso-8859-1?Q?5JlSvfvsx7ilSAxXCwXledX2svWRsAXWSEXZnonfDiS+4gkGzodhWnRJdW?= =?iso-8859-1?Q?fB0wp/Djdg=3D?= X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2034; 6:n1SIekOqfXTqo0/QGVq37Fvaj1Jy0EidscMlhZPHpZZrHffytSVMwSPEoFalqlFFQAr5VpHqE33AmRNThbp75TbRvgtlAhRz9eKPBknJzA7D8ku4stNpoE8bgTQ3/jkplRe8FgA8su5ztA9BMfj9els91yUpYBXFpZsHpcz5C6suSPqn0nWQIX8Gam1JSpdu/3ymMHKncCPjnvPeOBQpX1rflxe8j77N0R/7bWlXKsDdQURDq/7UaE4IYddmJ1ifl+LZdJCDdOEskiTooXoXCDWqmY6xIyGQakBt0M7+Ynv/DOt2bc7b5WKD0bR9pqX28HgYeyt9imdqoHOW82/MYg==; 5:G6qfuA8ES0GokX60c3WD98T7onBNhKNTi9vA5JrUsEcl1PM1aJlNOExy6vQ7US/aT7EK5Z/8+ssjHxYjKHT6mZc+6orCwXorx3VkjqZRpO3Ly92qtjxqZFbh9BbD/4U5HlpgU50ZDnp0MlT6sTmrmA==; 24:Bv3LDbiqU4p67axlijgksGaGl5wQqCyxlvvQLpsbdXUronbUc5VHG6bA7TVerJ2miBLIio+z4p7jRzfUCZMhxm5h1LfhEt8zWMVLFQi4UZo=; 7:3E/2Xxp3e0aRINcb5bOwczAtqImPrEIR2PWtjortCXkShLiBwxHO+NjEvLocUqRxkh7nighIRq+BswTYMLRjQIUrNtba7c6LYfX0Aw3xVB7qUhm7BQgQTcGd/y4Wc6HWTiuAvbbpxICKJScQTDrqP0CGZRAgBC3VJ9Nbn3QMaZF4c2pBX4KfVLRbS0dbZrcYtuBU43000PFzuTZu8D0HkU469Zp6iyQO5WhQxjmfKMk= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Oct 2017 17:24:23.9063 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2eafd56d-202e-4db8-4038-08d51a3ae876 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM5PR0501MB2034 Subject: Re: [dpdk-stable] [dpdk-dev] [PATCH] net/mlx5: fix Tx doorbell memory barrier X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Oct 2017 17:24:28 -0000 On Mon, Oct 23, 2017 at 09:50:14AM +0200, Nélio Laranjeiro wrote: > Yongseok, Sagi, my small contribution to this discussion, > > On Sun, Oct 22, 2017 at 03:01:04PM -0700, Yongseok Koh wrote: > > On Sun, Oct 22, 2017 at 12:46:53PM +0300, Sagi Grimberg wrote: > > > > > > > Configuring UAR as IO-mapped makes maximum throughput decline by noticeable > > > > amount. If UAR is configured as write-combining register, a write memory > > > > barrier is needed on ringing a doorbell. rte_wmb() is mostly effective when > > > > the size of a burst is comparatively small. > > > > > > Personally I don't think that the flag is really a good interface > > > choice. But also I'm not convinced that its dependent on the burst size. > > > > > > What guarantees that even for larger bursts the mmio write was flushed? > > > it comes after a set of writes that were flushed prior to the db update > > > and its not guaranteed that the application will immediately have more > > > data to trigger this writes to flush. > > > > Yes, I already knew the concern. I don't know you were aware but that can only > > happen when burst size is exactly multiple of 32 in the vectorized Tx. If you > > look at the mlx5_tx_burst_raw_vec(), every Tx bursts having more than 32 packets > > will be calling txq_burst_v() more than one time. For example, if pkts_n is 45, > > then it will firstly call txq_burst_v(32) and txq_burst_v(13) will follow with > > setting barrier at the end. The only pitfall is when pkts_n is exactly multiple > > of 32, e.g. 32, 64, 96 and so on. This shall not be likely when an app is > > forwarding packets and the rate is low (if packet rate is high, we are good). > > > > So, the only possible case of it is when an app generates traffic at > > comparatively low rate in bursty way with burst size being multiple of 32. > > A routing application will consume more than the 50 cycles the PMD needs > to process such burst. It is not a rare case, it is the real one, > routing lookup table, parsing the packet to find the layers, modifying > them (decreasing the TTL, changing addresses and updating the checksum) > is not something fast. > The probability to have a full 32 packets burst entering in the PMD is > something "normal". Right. There could be more common cases of skipping the barrier. And that's a good example. But my point here is to give more options to users, not to defend all the possible singular cases. If an app is processing packets in bursty way, it is already compromise between latency and throughput. Like you mentioned below, that's one of design goals of DPDK. If latency is so critical, it would be better to use interrupt driven processing in kernel context. Back to original issue, MLX5 PMD had unreasonably high max latency, especially with low rate of traffic while min/avg was okay and this could be considered as a critical bug. This sort of patches are to resolve that issue, not to improve its overall latency results. [...] > > Before sending out this patch, I've done RFC2544 latency tests with Ixia and the > > result was as good as before (actually same). That's why we think it is a good > > compromise. > > You cannot do that with testpmd, it does not match the real application > behavior as it receives a burst of packets and send them back without > touching them. An application will at least process/route all received > packets to some other destination or port. The send action will only be > triggered when the whole routing process is finished to maximize the > burst sizes. According to the traffic, the latency will change. > From what I know, we don't have such kind of example/tool in DPDK. I wasn't saying that the test with testpmd represented real time use-cases but I just wanted to verify this patch is effective to resolve the original issue. Again, this isn't a patch for latency enhancement but to resolve the issue. And I think I should also change documentation to address the option (MLX5_SHUT_UP_BF=1) for v2, unless there's objection. Thanks, Yongseok