From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR03-DB5-obe.outbound.protection.outlook.com (mail-eopbgr40052.outbound.protection.outlook.com [40.107.4.52]) by dpdk.org (Postfix) with ESMTP id 9480C1B357; Mon, 23 Oct 2017 00:01:22 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=k1bTvZ7Bj9muqRxiU9c6d1TsTCG8KNgBh5eqbz7Sx1o=; b=yJ6Ip5QEpCi3ihtn5DBZo+tfuZwPanzxbl9Ii6MDqP7WA5b+/9teXcDuW5F7wYOF7jmRlY1avQYQaDBC00FSXq8izT3FBtbIx0WEHOwzWPuZTgcxVzvL3aBtKZWFAOVYR/xB6FbWIaFxPBAkurku1tqs+GKDU/WmlhyB4eYU2j0= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; Received: from minint-98vp2qg (73.222.116.174) by HE1PR0501MB2042.eurprd05.prod.outlook.com (2603:10a6:3:35::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.77.7; Sun, 22 Oct 2017 22:01:15 +0000 Date: Sun, 22 Oct 2017 15:01:04 -0700 From: Yongseok Koh To: Sagi Grimberg Cc: adrien.mazarguil@6wind.com, nelio.laranjeiro@6wind.com, dev@dpdk.org, stable@dpdk.org, Alexander Solganik Message-ID: <20171022220103.GA18571@minint-98vp2qg> References: <20171022080022.13528-1-yskoh@mellanox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.7.2 (2016-11-26) X-Originating-IP: [73.222.116.174] X-ClientProxiedBy: BN6PR14CA0010.namprd14.prod.outlook.com (2603:10b6:404:79::20) To HE1PR0501MB2042.eurprd05.prod.outlook.com (2603:10a6:3:35::20) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8f4922ed-e8df-41e6-ab8e-08d519986b8f X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(4534020)(4602075)(4627075)(201703031133081)(201702281549075)(48565401081)(2017052603199); SRVR:HE1PR0501MB2042; X-Microsoft-Exchange-Diagnostics: 1; HE1PR0501MB2042; 3:dLbQ/Qdk5JlfywD9c99hz0c4kUnjZ1IasiXtRb14dCYXKV2o6x1c0pOfoTeWUiXQCxh1RidCnPMAT8oGpwW5/38sdN0w5sIE0D1lcpb2cOFTScEdVO76Vw7NlsmKjhP/O8J0OxwlFXakQz7MSYkVf66YD9rx3Eyp8adHNUaD1HXisStaZ1TsvbUpP8RTUHTtsf0MCFAhyUUMx09B0fkP7+uYIf6dgM2Usu7qwUejDRVuT/Vn2R1Ri8TVheMrKyD0; 25:Py2e5MrEX4nLKH61mcvhZZGw4C14SXqPZV8pITMJ1zioN4tzJIMjOelP7LwgAZC7NPLGLRxvlE23m+/wSe3xQhrrKg3e86EgeJ73RkWFzucq4x+e+uc+HoSFSXC2tvKD8YEUVZvDvuyF9nVsBx4QYhwBMx+cwkBPNIJcps2wXJ24Da5Z4qeRCu4kzGe0KMpyudQS0Vr2WvbXrt9EIbS4DtMxLgBHNUsqeKbjXnH22FTQbgT2iYhYh9QNictnW3Zt2U4MHsLZY80RvFeh+WKEZ1fsS0lRvpP5O4WFVebe75I9tA157r0q6KKUPWLD0kwwD18H4SWPpMmSJZ/M0Lpa8aMc5HM6Nn6xtrTuHGvB1Lg=; 31:Z/sZXwNS/E2/tyL+ReB1hkqK2yfLr+KDVncqGIZO09sksfONVYPgpMT9ztFmSj4F7KblOhyCOtFFRHTwGRu7MvZ9cwS+1bZyqq/fdae+a/TPob0rggi2yuDmHRn8lTlRQ7XwliqlvLcujLNmsVqQ7xvj21QrBQMBNdMas1DjugLdbS0jx9nz/altHFrD9WtUUjTlGQs67nEF4cN7/6wvC14f0gVI2t+/EsP8Hzr4YGM= X-MS-TrafficTypeDiagnostic: HE1PR0501MB2042: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; HE1PR0501MB2042; 20:QQ25JzonvoLIumCnQW0Gjkk/Sxubsw9opkHfGuWHTCw7Dt7y0pCDdSnkphBMPIc/R4HnIcmWHNBjxYsc5wci5GVRzl/fHbwpaXKNXLiCmc3i/mmeOACm61u+ftYrbLXelJw7Y9ytHy51a8SDRHQ1/nRzI4D1EPY7BhQgfwKrwXk9+aVFciSCtv8MPbxvSgcAiOEoEq9Byf3i5k7RxbG8z33SywZ7ByCSnPAYCFL9tBy1QITOyduRWmHQlNTd3wmZ7MZQOToG7fiEc6M9/UQle/KHjlN3FpNukt5xhaEGWD4kKQf+Ypw7Ql3d2bzaKRv/WOdK7m0nB/asYaMtLrwxImcrLtpl/u7y9RZonNvZW13VNffTPlhgvXnqCrzVwJzltvOleQKG/iVaXy+NfKmEWqibuT7Jv7I49dSSgtT2C6Ngt9AYCKUC4XLrBxozsPF707uDizfa32OVpR3dccJnc/nl1h07+QLjMr312uxtSUMi46zgT5D/iM/bJyJv/xE8; 4:toinsJqOangTRKlpAo88GMwhcjQOBN8ZIJuAV434tZF8SVqhc4cLajEc1xBcalA+hRFMH0TnseSnCEaq3At/CgFTQyb1pxT36NMC1b4Hhjq6lnbzoEA8LF36d+PnQHobKCdwUmsJPL8gwavQ0Pk+SHo6Bsb0XW37FN3xdmr+W4UNc8zdhY+Y/cgKJSXy6Vo5wrQwtxgzsERcnYn+846UVfWMpXz6o9opknprqEJzehcRxc1F5TrWadwQLfzSUJih X-Exchange-Antispam-Report-Test: UriScan:; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(93006095)(93001095)(10201501046)(3002001)(100000703101)(100105400095)(3231020)(6055026)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(20161123564025)(20161123555025)(20161123558100)(20161123560025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:HE1PR0501MB2042; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:HE1PR0501MB2042; X-Forefront-PRVS: 0468FE4A2B X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(346002)(376002)(24454002)(199003)(189002)(8676002)(5660300001)(68736007)(101416001)(76176999)(106356001)(105586002)(8936002)(54356999)(6116002)(33656002)(3846002)(1076002)(23726003)(50986999)(81156014)(81166006)(189998001)(33716001)(2950100002)(6916009)(25786009)(97736004)(478600001)(86362001)(50466002)(16586007)(9686003)(6666003)(83506002)(55016002)(6246003)(39060400002)(16526018)(316002)(229853002)(2906002)(66066001)(53936002)(6496005)(47776003)(4326008)(305945005)(7736002)(58126008)(18370500001)(107986001); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR0501MB2042; H:minint-98vp2qg; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; HE1PR0501MB2042; 23:Q36SEHbeG24FoXiTjxINbs2f6J1c2Fu5k9REgNf?= =?us-ascii?Q?tqvFlYZMIvPfe/fSg2tWW2m94j5a1JkrDm5efp5b6SdTMgpn0iAIbrXt4wOG?= =?us-ascii?Q?fcfo+Ajg9bfOEERqjnkFHHad1O7GFXrLoA3Qw3gzCUlPTg08u0dgKli36J0G?= =?us-ascii?Q?v8WQzcKlkeahMjlwHj22iE47VZYz36XJTnBtOXwJWknlmLvfvDSNaWnLjqyt?= =?us-ascii?Q?knhTwb3Vtf1mkjfEHPznaqul3QHLtzmposn/19wcOwKvkgicsi7C19X0U+c5?= =?us-ascii?Q?mpun35CWeu7gNTPbFe/HWA0HtlZpkWPS1cHS5WpVEwMAFTxwibYrAKCClh+1?= =?us-ascii?Q?EM41zLuJvggjZY8zQI+o5cFxY1K9BP1c9nmrFTcN26ZlU2b2NNFCzWWGWji7?= =?us-ascii?Q?ac0p5iQz4KxtCLeSDSlXHpWvY3bNMQ6G4rBUK+tM9AFzfvl6xRT4wk+NOGsY?= =?us-ascii?Q?97+I2UWM6QALc4hAZM/9mXOhhiOlaNV2UiOA6BmJldNvp2woQroURcPJFQRg?= =?us-ascii?Q?7bK8vVxPgrlnl2Zxh3OEz6qfg4mEsPHi9doHvMecPvXiLtUUWzP6oIe426V8?= =?us-ascii?Q?sq7Qrsyfjfw77oehZ93D6EBRmvBgVWu+n1CdcQrLH0E/2/DdGWOkwO0awOtA?= =?us-ascii?Q?VwIxU4Hvr1B6k4vXlspOg0PI9DLpHBHfxjZk0aoTgUIXKohgUzqyYBabmEK3?= =?us-ascii?Q?iHG2xNFRgnhSvOZ7yl2ZtGm3YYB2Xv1/g/y7fmOj03gJp+Jbxl3ckv79/Ii/?= =?us-ascii?Q?PchPXkV3MzP1ERvqmfd7tswTIzai+XO7zrZJxESQ1WWnDECYFnPetjCiZi1C?= =?us-ascii?Q?Ep41osEAis8w1eQ4v2xrJBjtSlFfeBavZeA4RCIyTcLqzldqK8Z7b3nYhol6?= =?us-ascii?Q?BRvCDc/4OWtD/iubHKQnQO/5/HZlb5Feqbr9I2EPAgGyUGHigwl2mUeDmLJH?= =?us-ascii?Q?GQEQosLZxQq0N/RNN9GeJrXt8wy2JT8ObH+qMWf6kDHJ5vvWFsWwEMcwtK5p?= =?us-ascii?Q?ciwK7pyjcZkiryxKLZ1Flobw92+JuJhSAAFUr8AUHx+kSlpkTFIE2IrjYusN?= =?us-ascii?Q?cX7yK4MLb/e2uzY88THqmWSNDpQDdZ+nfH5StQi+i93VeZBIH0gfevxjftZq?= =?us-ascii?Q?ojsp8YZtcCyYRd/kiKngnu97TvSk/Lpt9moRQIGmKTFY7umdiyCHLVePt1Lj?= =?us-ascii?Q?d+oTVsPWOdCKB56SIYzjlP9V8lKGeO9qAO7gWLi3Goq2wkbvFL+7wSGLptw?= =?us-ascii?Q?=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; HE1PR0501MB2042; 6:dRXn8GGOo6Yq46XlKR4gPz5UZ/CgLzDWM4gtGTrEr0eLtE50vSolEfRA1gfBgLJNEpcouiGiI/bKCbqOiDIE1UIdmO+5paWwPKRml02RPF651FZ+9O9OIrN8AQYeci87h5xtFPEJVBSY4ojdM3Y1/inEndgRbHHQ8rizznJbr/Z00PYv6cjAWUufrWr9LXbDD/5W/sy8VHQFkup4kXwUCbqW2R5vRieuvVxfPuuxK7jauFjvxnpVU7qAxIQpdCzQGp/sLTr+XFK21zHEv+ucpY/Y0oXto9zRVW+8sOSFCYyFZsb/INxi1V1+ya2qiDOftLmmVnVbrZ9NtneE6WlFyQ==; 5:GKNeGWj2P0lFLORGSWitUUca/k2ef5jCb+YTORW4OAhT0ofRi/lledxKdt5/rpNYaps9SFmY2LAkpBRDAXQ5whXevC9ughYjrXdOsI+L262B5wPuWLZsjiLxPGy2Cu5nNIgNQ1UAeuLQO01RahcUMw==; 24:8PQMtnj98iaxZM1e7aBbKh0H6ECGDHHArWqx5Q+pQvuQU1Pk00olbYafDUSAV6na44K2nTYEJuKaKI3tBgLz+Fd9sm/OVybMxuAQEF/rp4U=; 7:degfCIqypqrZQBXCXu2vuHt/jYDXRhdLSG7WRFGb8L2k/mJY+LigUAU0ttd1bBev2qtdl7hdxfRhRu8ck+utUV38uZaZxDCGQkIWg/oqyBGSCE5zbtQDZupzrIBTeMWN/AMXcV/+tR6WjXtFceiUoziVc5jLTdSPYu7TsXUfuL3WfduZGvorrfkjVinIixKtAuszZJzS/O71fkBwhIhAW9fp9Q8Ja5O2G9T9XWQQovQ= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Oct 2017 22:01:15.6999 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0501MB2042 Subject: Re: [dpdk-dev] [PATCH] net/mlx5: fix Tx doorbell memory barrier X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Oct 2017 22:01:23 -0000 On Sun, Oct 22, 2017 at 12:46:53PM +0300, Sagi Grimberg wrote: > > > Configuring UAR as IO-mapped makes maximum throughput decline by noticeable > > amount. If UAR is configured as write-combining register, a write memory > > barrier is needed on ringing a doorbell. rte_wmb() is mostly effective when > > the size of a burst is comparatively small. > > Personally I don't think that the flag is really a good interface > choice. But also I'm not convinced that its dependent on the burst size. > > What guarantees that even for larger bursts the mmio write was flushed? > it comes after a set of writes that were flushed prior to the db update > and its not guaranteed that the application will immediately have more > data to trigger this writes to flush. Yes, I already knew the concern. I don't know you were aware but that can only happen when burst size is exactly multiple of 32 in the vectorized Tx. If you look at the mlx5_tx_burst_raw_vec(), every Tx bursts having more than 32 packets will be calling txq_burst_v() more than one time. For example, if pkts_n is 45, then it will firstly call txq_burst_v(32) and txq_burst_v(13) will follow with setting barrier at the end. The only pitfall is when pkts_n is exactly multiple of 32, e.g. 32, 64, 96 and so on. This shall not be likely when an app is forwarding packets and the rate is low (if packet rate is high, we are good). So, the only possible case of it is when an app generates traffic at comparatively low rate in bursty way with burst size being multiple of 32. If a user encounters such a rare case and latency is critical in their app, we will recommend to set MLX5_SHUT_UP_BF=1 either by exporting in a shell or by embedding it in their app's initialization. Or, they can use other non-vectorized tx_burst() functions because the barrier is still enforced in such functions like you firstly suggested. It is always true that we can't make everyone satisfied. Some apps prefers better performance to better latency. As vectorized Tx outperforms all the other tx_burst() functions, I want to leave it as only one exceptional case. Actually, we already received a complaint that 1-core performance of vPMD declined by 10% (53Mpps -> 48Mpps) due to the patch (MLX5_SHUT_UP_BF=1). So, I wanted to give users/apps more versatile options/knobs. Before sending out this patch, I've done RFC2544 latency tests with Ixia and the result was as good as before (actually same). That's why we think it is a good compromise. Thanks for your comment, Yongseok