From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-ve1eur01on0053.outbound.protection.outlook.com [104.47.1.53]) by dpdk.org (Postfix) with ESMTP id 2B89A2629 for ; Tue, 25 Jul 2017 09:44:13 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=5GZDzRIr7cwGy44kdEHmGVLuZfdyNs/6Eomf+OfBPVg=; b=ZGIkhJV+x6j/r6A/eCfJN7c62lrU9ygGQJaxUk4U43Z71UyIJHUav4cl5k+qXEREWFiUVNi1MBsNuAxxQ2nIJfDPhmTsmsYxPus6flHi4XY+k3E6Ul7P1KMntfPMuXo3OQRA4jm9aDwEit2GXaj4QjEvsw+T+RRYGjYT5iwfcbE= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; Received: from minint-98vp2qg (73.222.116.174) by AM5PR0501MB2036.eurprd05.prod.outlook.com (2603:10a6:203:1a::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1282.10; Tue, 25 Jul 2017 07:44:09 +0000 Date: Tue, 25 Jul 2017 00:43:57 -0700 From: Yongseok Koh To: Sagi Grimberg Cc: adrien.mazarguil@6wind.com, nelio.laranjeiro@6wind.com, dev@dpdk.org Message-ID: <20170725074356.GA4034@minint-98vp2qg> References: <20170720154835.13571-1-yskoh@mellanox.com> <20170721151006.GA38779@yongseok-MBP.local> <918eb17f-269c-7102-41ab-69ceb95fdacf@grimberg.me> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <918eb17f-269c-7102-41ab-69ceb95fdacf@grimberg.me> User-Agent: Mutt/1.7.2 (2016-11-26) X-Originating-IP: [73.222.116.174] X-ClientProxiedBy: MWHPR18CA0028.namprd18.prod.outlook.com (2603:10b6:320:31::14) To AM5PR0501MB2036.eurprd05.prod.outlook.com (2603:10a6:203:1a::22) X-MS-Office365-Filtering-Correlation-Id: 7848b778-e838-493f-1e10-08d4d330f04c X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(300000503095)(300135400095)(48565401081)(201703131423075)(201703031133081)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:AM5PR0501MB2036; X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2036; 3:qZo8B9cxNRGA/pB1P9SkqY5GFbhklTCA272RDdqoC8YLvGzc1nFPFanMS1p2h2zNseb6sJ/bauCtmX/Ak6QIYksoDNW3BEv1iip+8Lqa/6E1v4Bko5g8h1ULlQgMt8GjWPZBrifYfDqb2mttgScnLMTlVXYZ9pmoGaaclqnna0Za4E31YaLYxRT//UoKYYgr/I/HYtS9Vfk/0nTIOxvYMtYwggd41atXt/Q5a3UpLGxhuSi0BME6QHDzKjVZ2FtdUSyVcaz3QI5ABSSuo6EPhVLxB4XzCAz+guknlairINSbHjT99pr3OZdpq8IczNyArqWo5gCgjZfQikQ15hhAR3GTTvlv1LLymsY+VsPBfiKe5/7SXsa3OzJiPS7IDlG/nfGqah4M1beDirJcRlG9GYkmtrKCUi2O/IvL7z/tXcvqbUNeRWCs41fZTFyPxjCBht67LbhrCm5mst5/82Vz5ssFiaivHPnJB9DW7mduezzyOXg5KfY+YZ7Tr4ITDrlZrUDVX3/nb/y2z4rLmZjQNsG2FurZFYXwgMgwmNw6plzIot8VNVTgYJvtay8g1cPTk6Mg5kzJ2Yk/GBzAsTHTo68NH9qfd/rJy7sXtodpfjFJ5tJdVI43FFPT5xphNhzF6ctXasnxOlPn3psEunb9IKY+YBgO/X7aJ1ozbq5MbsZ2JrZJTTiHUA0mJyYC1NZIdaMfbIMKHm5e5GvFCbVv/BLp2dd7UldeZh71Z9FGiTXoEzthH6xsVB3X4RpTFmNwf8CKdVD++E5qzk2yS12D0A== X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM5PR0501MB2036: X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2036; 25:8BPdFmauqw17ai6+f6p0B/Yy8/cji4YNeG/21qtCUHTlxJI7fdtNYYB/MijulV8yE0ThLC3j6FhkbKedh+WYVC6iTjCPhMOB73x5UEmsWy0Otip1/9CwHNpFnjd7mOEJvBZQVXfwWxM7jKwck/1fzpcfYyQArvvkrHgsWzALGrMn9eAUE5IazRJuX4zW8yYGLEDliNdmcJKwuwC7ULNPNlKiduZQCMm2YZqSyIU6nkMTVa0tFf4knbZWoscK450QT884BBTsMcPvcIvcvllKzQI0YK1Ywwu0U7M/pGMQdeGMugZWVY83misySrRY4qm7j+icqjb1YRKIuISFI3/LYRY5U2JaRnB8z/kwaxDcGdWIsTpIW3zAamctvuk8QqqQGfKrJiYv6m9apXyIPYibSo+7YhhmMd9hKaAMYGMMpDGJYgOSchEecxhWnRl9Ma8CVOXmA6g32U4x9aymFNRoz6UsVzrsDSgRuwJ7oGV610d0k0eI7J3E7THki9VFgRrN0DlRe38clJIMkDEIzwPCw/wmT2pY2H/SE7LL+aa8XXJQ6RKW3RTHpdSQnxFusZSgvg3OgqX4tTg7lKNQgHsef/6NibGdOsUy6NXMmhNxP6fCPt+bxgY9RH6aHx2KBjd2MPLJ+zKrmrGozDTPYYW0vgyYBTmkCW5xhIXfEHZm2fi4vc46CZoFh64Wty7p8A3uX6vJhoFUsSmJTpQ+3cAKbaOK7Fu9xU9ikz5J0fo1j7eudLbB1KgbtkrggCmOZlq2spHikEnH1LFZtQz0g8/LvU/wPEukWsw8lN1nCgiK4Fn6Wxs5iPgMgFIsotyT6iPCjqNuQKn0wFeCiGZch3PHXrXhrjZC1/KPxl6L6Ck74k29sia75zIqDdG8awKgmsertTlIoPY+LD1JN6ujdjIA+S7AkTVr2v5DaSnQEu+l/SE= X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2036; 31:wlw8CSzfJuDI6ZkncgUhIee+aGu8eayuriVjnWiiVvzm3UQFZwV3DrLzReY4vHlGX2S2nzLQnGfcs7vnKZPpS2rT20A9RRPhk3ES4yC68VzU5tDWZZF9sCHsAZexi18UwR67PpbukftghFdH37GJoPIikyn3oKj+oLkqV3jrNUZOSVXLC7wUEKjVXRSIeSflGDJR/LRSs9HhLwzDlQYm+1IsUhTdgXVFEY6siROA9JCUfJ+F+N5qwlUnMZIfcdIgbeIbcMop/p9LZUvQr9QyWDEvzzy62DT0xDL5onayswUEmgWnT3qtB+gURApQwW1u3Pm7F9VwOO2lfdlXdFmshmVBZAMpdvxp/a4XU937T3Lpm8ydfNXIOFeVtuN1wVlKFIrxviT6x0Ogcc76wfvTOof/mheEL5YdVAUXcypm3CAZYu66q0xzi6ZVMmGzMHzWQsy/bivuCWqEOlWbwEzzNWE0lkxiliTL0Bo9BSXc1dhwyxz5x7DTMkEZdZ1IyW5Mj0UhaMKGIK6lgmRdyO5p+vo7dcTCjOg5Nw0CrmwSaZWkQuoVAUUnozn1xvcaKo228ViFMsljWJ6rjFKAjkhyJ//zJlaExpKusLigGqfHGS/zwWyPTlkArj/zpY7uGJ5NLGrWY+yIvqemjvJNA08L0BJDscca4sFG8mmwNHV/HOroUpcFIM9BsFnNSvSBU3puR8esz0aphuKx72U0cyWbgg== X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2036; 20:3D+D2PsljerK8hBkUXBc1MnyV5417vuaAbidApmOaG+04BPiW2KJH5xBkUSA8Cmx/0qQqsGd5kTuss0DM3FTMbjLAVF8pfT2gSW7NAc8nr9WU76lgeVh3R5+zxEP+Ub1xD28xLXPgiJjvB6NOl+wSxDe1Z7ix1azRVwL8C5KwDuYUktw/Wq863yl7qvpLZUrt0U8NwDoUhqkmy8ScLtUgQ1lM/y/eYJu4ZhiaAxaxI88dnLZE9oezv+fbV5Ko2LNxd+U+TGnzJd6cJyDq1ZfruY5JWJENQhQkPOIa7RtuyYT3W5ZFp3ijJRUH/2CdiFqX6z8OR3nEzHk05AXJabaiqhnaW+XUy5yBuOccOe9lNSsbM3NTm7gizihNtGBI91jc5sdIQWS5tBd2VaYqEvgz8Byq3m/uCUe8AmQbs8htV8Mop0EFK7qXtzDqbDfoF5Gg3HPbS8yCw10GUpW9hqYTzFIbWoLIGIvVshjm4RwJ3Tkx16NXwZ2uQ+zlcYwpAL2 X-Exchange-Antispam-Report-Test: UriScan:(209352067349851); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001)(100000703101)(100105400095)(93006095)(93001095)(6055026)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123562025)(20161123560025)(20161123564025)(20161123555025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:AM5PR0501MB2036; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:AM5PR0501MB2036; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; AM5PR0501MB2036; 4:SoRcODDrmmtJrJ7xj27TJev13xkgGcSUYxeVS6J+?= =?us-ascii?Q?Uds+C5+CfmmXkk4291uIwxsqLIRhtlgheLad3IeBTQ141AEz/V1CBycvYApn?= =?us-ascii?Q?5f45WHJIQu25Al+6or6GVziuLDBkGRnifM2NJLHRyAlS1bfkU6StD1nMgNhI?= =?us-ascii?Q?AVssa3eNx/3dS+Bn3HwUd1UxibS6G9w0h+oAjJl8T8Q1z6WlqBK6DKZdQR5C?= =?us-ascii?Q?QlJsyOwqgN2TYkMPiurhZgqAgJxrPoi68fYtHtNV+WZ6/UNkSPhwz4lwzjwY?= =?us-ascii?Q?G+Ib4KV3Ll2cVntHY9anWkkcOJHxCxd0o5O2oPkSb4soCP+GiO5nSy33rK9C?= =?us-ascii?Q?Je7ZChMlMSp0ksXHC0fm0fJ3WrQo2SwLx17mIoCsc2YqwQFk3UXt7j2N3/Em?= =?us-ascii?Q?2LfNHVep9DVe6l6JKXDhOq0Ze405go2SukcmvMORrcYM5jrG5ZM2dAvEavA5?= =?us-ascii?Q?iKQ69vhJCXmbfFrJ9ZB/hRLIlnxFcT6lkU8Yo1hAgxLnb2ahOzgcrkfTL63w?= =?us-ascii?Q?2ctIU9ujGtZxOgp1BlcZ+rTGx0rpUkK5XPjkeWXMuPZuKGNeWK8UqhSwwfww?= =?us-ascii?Q?9QW/SSvqme8QlIIwoGlEJeBjJiqEkipFGnFqV44GAaMM9fS//qycPE1eWvwc?= =?us-ascii?Q?vh9wd9LZsMBhpCfL5ft++ZAxX5H37AqZhYIhqSq9KETaoFFxIELTyf6O50vs?= =?us-ascii?Q?CAhK31sgXIFwMuCajsR7D0wPiN7Tkeb0HIii8WnvT2h2BTRSt2eZLmnj0D64?= =?us-ascii?Q?qSgmb9bBrN8qKxt72pg2MZSV3DZvBjZvnDF7uS4YaxfhPgh15ftkqbFw9xhY?= =?us-ascii?Q?wAKnUOJRiOCclf1mt2rlbzrrA6TslDsaK+VdsRhuoPhX1R5GLT83/h/9wkV0?= =?us-ascii?Q?eOlYWwbMmyXYgolh7sYrhpUJhW8pUO/D+fanoHuCQRS7wTYUxf3aU7XpbmtZ?= =?us-ascii?Q?hQXlPcuceqlBx41of76W2Anq1tHDVcH757rRruGd8vqCsZoHY7zGgZUm9erg?= =?us-ascii?Q?nCM1jlkEYg5VaOe3v+A/jH917XIvYVZidpebzqA28rWwXGloZ2+yA0LDBJam?= =?us-ascii?Q?xaSRUCW8yZHiCRvtznt2IObSatXH/xc8OeRnqtg/BUsOpv0Ac0apzdRfzJis?= =?us-ascii?Q?FjU8a3vZVc6lWLb8KWEJWh5XO0qQozQqWSk0KR/uxXDCU+R+fkgB0LM3LyAn?= =?us-ascii?Q?2OAQq/hYuYmK+dAi1sebD+k0z51Ww1WV9X0o?= X-Forefront-PRVS: 03793408BA X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(7370300001)(4630300001)(6009001)(39860400002)(39400400002)(39840400002)(39850400002)(39410400002)(39450400003)(52314003)(199003)(24454002)(189002)(5660300001)(33656002)(86362001)(9686003)(2906002)(229853002)(55016002)(106356001)(42186005)(93886004)(105586002)(68736007)(50466002)(4001350100001)(7736002)(76176999)(50986999)(81156014)(81166006)(54356999)(101416001)(8676002)(97736004)(83506001)(305945005)(110136004)(53936002)(38730400002)(23726003)(3846002)(6246003)(6116002)(6496005)(47776003)(2950100002)(6916009)(6666003)(189998001)(66066001)(1076002)(4326008)(25786009)(478600001)(33716001)(7350300001)(18370500001)(107986001); DIR:OUT; SFP:1101; SCL:1; SRVR:AM5PR0501MB2036; H:minint-98vp2qg; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; AM5PR0501MB2036; 23:kyoqdXjvB0R8NLGvRA5sb6tsWhP8EwPxM4Wa37l?= =?us-ascii?Q?J0p0dt1Xgf9rOTcMjzu26sNrqpL5CzCkz9+wE9wWrEeVt/w/utdnhdTAzyt/?= =?us-ascii?Q?SomKPIHHN0lSTcrk/lrtF5+46kEbDp4FnJMG1ARggdDagibo7mqsYnuJ18Oi?= =?us-ascii?Q?uoI1ydgUT45P8AUlMyyyaYLBkqoz/UAVhxFQ2apFMdMnjE+2E0qSltNb199p?= =?us-ascii?Q?+NYaGxAX8kJzMa5Wmbcg/DcgM4H//4mU9s435pjh+LVs6XXObE5ulIpRk1C6?= =?us-ascii?Q?PHbGVSuMVG6ykhL/+A0equKUe81XfChmxov+qCSXQ2fiMgxpqOnNBsAmJYUy?= =?us-ascii?Q?YG3DNLENSck0cLUr9kzf1JhUR+VJunMszbhWCDL0ui56d4ssbbSAyWNN13xV?= =?us-ascii?Q?2K+lW/ADZyPI1KhObYb3v/9OhuTKBtyVqRjr/je7YAOGB0hABvXH8QrdwGag?= =?us-ascii?Q?tVGxnr16ZogXYmwwho4afZYAOPazxXWPsEl2e25MxuKRU7UOWCK1wBaVKi83?= =?us-ascii?Q?/zjyrmNGQaMN2kPJ9/ynri2NEzwOKQ5YKEFDFSBjZ+1ctnffiwOm5BNXRTte?= =?us-ascii?Q?lwKOnbDI1m6dUxHgBoPQnY3L4fCzQjVssvt/78XhxK29k7sbXGkTj4BnJoB8?= =?us-ascii?Q?UE4zenCU/tH74Lb7tKnUGuzx0tqyc4zyvI/eLCwp5KSBG4wenGq9B9csOhG2?= =?us-ascii?Q?WXI5ocF2y52bCnHcMrtXoFkQeRmork6rorv3rDuQZB3MFHGQ/Lvc0Z4fpMjB?= =?us-ascii?Q?uuSlvpXsTVW6D8RC4uezowMDCiy0yqtTHEmXkpWAB3CQGBuIrptj/4DEAmfd?= =?us-ascii?Q?NjH30BQX8wZMcg6sMmDkiUw0aGoS2e+Jc+g+Xzolv3f1XpdGtpezWhLnYe+t?= =?us-ascii?Q?wluxk6tkGWwB5d+ZGBWyQl3s7/wLsPRjnAGi9w5bJLO9BgO6u1DEC1G7Jhqp?= =?us-ascii?Q?/fa1x6igHo9P2pZfRp5j8pGLuOGLKtvZsGuVnqKxIrPAJusOa2I8yf8eL4Lj?= =?us-ascii?Q?iTV4u2/WvZp8OS8KyWXOTPrNarzw4DgK7kFL7bF747ozZkYOJKRXdnBfcxGr?= =?us-ascii?Q?S05Tagxx5Kd89vCKK96CrNZn0SYZJJVmuXyT6q1nYEDY4/9vHS9VDSnSi9Zi?= =?us-ascii?Q?3dTtZ6+rnc6zDq30kokE/MmXLzy3Kg590bVpqdi9xf9fEPKn2K+7jiL+QD2b?= =?us-ascii?Q?UkEorbV1oAZmpglIlF4uZ5dVpPrDzMjfXKEprBey5ieuGNN30v990LO13r6U?= =?us-ascii?Q?NF75qdcpIX+1RaBP5ldj7OVa3CEyx7lbA4WrxTo7wd+z+SGckF44H/5xC4QM?= =?us-ascii?Q?ZXYeKC72PJfhG18OKnjIC6hEF2PLNCnVQyv+N8BPUGbzD1X1B1eTFTlsi6fi?= =?us-ascii?Q?ibA9Vgv1r7GqrsvgGCLSrzBXJPE2z9hIYjVu9qzW1+0Ld3zvhsrW6Z9awUGW?= =?us-ascii?Q?g05lTBCyqnSM1uLeUxkHc845TGBQtDbrC196bT+9ft5lrtc90Y2id?= X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; AM5PR0501MB2036; 6:mO0yBR6EQ8Wi1FmOKmjJ2YG+F5ko2d7clofTD01A?= =?us-ascii?Q?I6IRancx7WypIRghibQQOjQVr6GYPuyKGKs0RRZSRlC/SFghz1oVbpSsmBSv?= =?us-ascii?Q?wiUiB6D0jdF8CSgNo9JemBDPS4epDfEB0NI1pgAezM75m1vmohwBJT7lI3OK?= =?us-ascii?Q?DGS8wsxHAxAgj0LvByLcGQcWLHUUJuQRctiTmFqCJOXrm3tA+4ZGFRZ4VaFG?= =?us-ascii?Q?YJoPkbHE46ZRHSU3dyWrbnUYMQpsY6b8eqAJwN6Ivc6w+hX+z8Eo87pYLP5v?= =?us-ascii?Q?g+L5pLtfCS+TJmkXR34NrUaAt/hEv28NYhFDSMJis2BOe/GU8o+PHpAZVXbZ?= =?us-ascii?Q?t+O6cgabYBzImH4Puy1p3b9g2AYElBDGHmvqFB/zA9k72l5YNSjp8+H0++3q?= =?us-ascii?Q?YQ0C76Lw8HGlOh5F9LjUEOfA9jytby7OoxzpV6C1sLzkzy7DFiucyBB/lQF5?= =?us-ascii?Q?MSXCszz4bdqgWtWA/OU4URbTSxSQCCQbdiBkA1byDQhmOTiwAu75KQ5STVaS?= =?us-ascii?Q?GAPNrrbhxeU0oNeY5OyxR7vmW6/ZY4ibS5l0mrpcP3Q9A25IuiMWaotE9UgA?= =?us-ascii?Q?5VOMRsZdI43su8M3qUbcMnjQi+BxwSIuQSC98fqDAqC2sqI3lXdblHFkPqZ4?= =?us-ascii?Q?5scKCgbA54M40A+7GE8RS6jhvyzHJbr/UsQZEBJwTEp+hgJc0m4la1/EFwa7?= =?us-ascii?Q?SnyrpPvl9GcaK8MPV/FYZl07qoa1le4RB4Z4Wf7P++9wPVyagAF2y6cUAQV7?= =?us-ascii?Q?627aebIWhqTsJrem5c6RleOVSbvRQd5Qp1N6Gk8ATKGOiVGq9Bpsrviytypa?= =?us-ascii?Q?ziLzXVEQY/qrDeVrlv/D0ZLOqmPlpnsABcIf/eVvs+rp6FoL+uq56CQLg4j3?= =?us-ascii?Q?1+HzTXQEUKH6fG2MBf82fAD638UtXsgcZtvlOYP8ckukeZHiCO1Ep3wfbeoU?= =?us-ascii?Q?NZr2FXCxQZjVy000RmEB5srjnl6FPgPkTTkUzdKZlzg5uBOmd0sKGNkBw9Ib?= =?us-ascii?Q?jhmlPEpteU1N6Z1oADfxMycx?= X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2036; 5:iR55E9L4K192E2l7bjYmGbbf93uJOSan8qHX2fFLidicPKQqBQkvAOgyK9SX68oBXYYVzae0Ly7+a1xoE7kTj6YkCYmvhRoV4Y4vAlvK8+JfHvG2q595M07Oy/RpaActyt+ImwWG9UmWUuQmjoYwFC9AUdPBnCt1hrrU/7B1qWMWBYGdyJFIOlQdmfZQMrdKi9fDYVtVwCXxapW10j9WYOw5M3h+7qkXTMzqbZbyNu9O7w6KYzQFGgXXnQRcueSV9Br8PlVxwXzoALMny9jcRJWjSpj5w9gjOrO1EWt/YrqO2Wngiki6wX5vM/FrgvmENEF6ZnROdbexT0xK6hHbQRk+nqCeBNWzezRbo6WJfQbkqcWx7jKNLtTdhJ1gP1AOiZdowkwQQzde8uoSj5GHQbVfh+7rKVT6jWZCxGTTe+yfDYbSx0O73wTfOxU8FRVkRPKtXbQzq+lnG7bxAW/i4ECAXr5BuaN9YYC++2nKQIuDE3FBth+JQI6QhDExP3z3; 24:ZHQS2/y6GQDYbhdRSJzFmSlxgth6psYUgeMT8jgwzM7tyyR2fc7VitHj7FcRHW0CNnFlmN3xBq7hRpDsnvAgkPGmodEYW8a5AARqJKFBhJo= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2036; 7:vWwHFzdGk2DnBsNwDdMuzjXCyHcvJqwvColXDopLLklI5Rl1l3WfG5LsvZEuCts/1hNqNYFfCvzAasjIG3x+zUOc+l6+t4ogYHU4vxG5ngIYfUtQ8cAw2Hq06Z9BsM02dpcpl3KZwgkgRIxCfjPUzbMVViHUGA/GwI6HNKjhJ9nm5K4gpZr1hF0pZVkRIdsObUwLrQChbhpNe5qFTdnQsbz7n1am+i3xsGjxuP7aZspL7c5MBrLNqBGGSQ9AljeYVyR5zLyO6MQXXes+84CTkvaJHTv0wCzo44JYHm4xPZDvXTakJyzbLN4n81jzDMlNv/sjR5mDGgRoq7icENRAvt3nbo3BOChshRYjvZnQkd0c0xQ8OvjW9NCRRx5wro0Z7K45lM9FemIB9npkWSzTvuNdqewoXRzLK9R0W+IsrvFfdtgU+8sqPqqRaY6XZcW4XDM8I7qdDWuKtRtwPbfZoHh8pGCm7NP+Xe/YOFbIMBM2D1HTmt0V+tIuMUwyQyNnfhwlY3bezekZ57+sqiEV/0IgVZHnBF2bqdjK3THB+cl1WWqxatyaTxp/TVEejlmTxbpwHfIgPdauf4uvWcUK9VmqYapI2N2eNQWbBdhYn2Cgr1ikZbiBGgC6Eax05GUUOBpH1PwRPfEspC2DQYQNn7BOEDZNsNKQHb3rpXFP9mHAaY3dXlgSz2Ewyr6i0koHwixv2kW5SO5Ke049mW+DnhbaWMJ0kaxMwUPyxlyn739lWxBYlrF0/7ltZRI+tqBD/J4Jq44lpZQXxyjeXDCeyPKkMJkXkXA1WpvGRwLyO1E= X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jul 2017 07:44:09.8354 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM5PR0501MB2036 Subject: Re: [dpdk-dev] [PATCH] net/mlx5: poll completion queue once per a call X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jul 2017 07:44:13 -0000 On Sun, Jul 23, 2017 at 12:49:36PM +0300, Sagi Grimberg wrote: > > > > mlx5_tx_complete() polls completion queue multiple times until it > > > > encounters an invalid entry. As Tx completions are suppressed by > > > > MLX5_TX_COMP_THRESH, it is waste of cycles to expect multiple completions > > > > in a poll. And freeing too many buffers in a call can cause high jitter. > > > > This patch improves throughput a little. > > > > > > What if the device generates burst of completions? > > mlx5 PMD suppresses completions anyway. It requests a completion per every > > MLX5_TX_COMP_THRESH Tx mbufs, not every single mbuf. So, the size of completion > > queue is even much small. > > Yes I realize that, but can't the device still complete in a burst (of > unsuppressed completions)? I mean its not guaranteed that for every > txq_complete a signaled completion is pending right? What happens if > the device has inconsistent completion pacing? Can't the sw grow a > batch of completions if txq_complete will process a single completion > unconditionally? Speculation. First of all, device doesn't delay completion notifications for no reason. ASIC is not a SW running on top of a OS. If a completion comes up late, this means device really can't keep up the rate of posting descriptors. If so, tx_burst() should generate back-pressure by returning partial Tx, then app can make a decision between drop or retry. Retry on Tx means back-pressuring Rx side if app is forwarding packets. More serious problem I expected was a case that the THRESH is smaller than burst size. In that case, txq->elts[] will be short of slots all the time. But fortunately, in MLX PMD, we request one completion per a burst at most, not every THRESH of packets. If there's some SW jitter on Tx processiong, the Tx CQ can grow for sure. Question to myself was "when does it shrink?". It shrinks when Tx burst is light (burst size is smaller than THRESH) because mlx5_tx_complete() is always called every time tx_burst() is called. What if it keeps growing? Then, drop is necessary and natural like I mentioned above. It doesn't make sense for SW to absorb any possible SW jitters. Cost is high. It is usually done by increasing queue depth. Keeping steady state is more important. Rather, this patch is helpful for reducing jitters. When I run a profiler, the most cycle-consuming part on Tx is still freeing buffers. If we allow loops on checking valid CQE, many buffers could be freed in a single call of mlx5_tx_complete() at some moment, then it would cause a long delay. This would aggravate jitter. > > > Holding these completions un-reaped can theoretically cause resource stress on > > > the corresponding mempool(s). > > Can you make your point clearer? Do you think the "stress" can impact > > performance? I think stress doesn't matter unless it is depleted. And app is > > responsible for supplying enough mbufs considering the depth of all queues (max > > # of outstanding mbufs). > > I might be missing something, but # of outstanding mbufs should be > relatively small as the pmd reaps every MLX5_TX_COMP_THRESH mbufs right? > Why should the pool account for the entire TX queue depth (which can > be very large)? Reason is simple for Rx queue. If the number of mbufs in the provisioned mempool is less then rxq depth, PMD can't even successfully initialize device. PMD doesn't keep a private mempool. So, it is nonsensical to provision less mbufs than queue depth even if it isn't documented. It is obvious. No mempool is assigned for Tx. And in this case, app isn't forced to prepare enough mbufs to cover all the Tx queues. But the downside of it is significant performance degradation. From PMD perspective, it just needs to avoid any deadlock condition due to depletion. Even if freeing mbufs in bulk causes some resource depletion in app side, it is a fair trade-off to get higher performance unless there's no deadlock. And as far as I can tell, most of PMDs would free mbufs in bulk, not one by one. Also good for cache locality. Anyway, there are many examples according to packet processing mode - fwd/rxonly/txonly. But I won't explain all of them one by one. > Is there a hard requirement documented somewhere that the application > needs to account for the entire TX queue depths for sizing its mbuf > pool? If needed, we should document it and this will be a good start for you to contribute to DPDK community. But, think about the definition of Tx queue depth, doesn't it mean that a queue can hold that amount of descriptors? Then app should prepare more mbufs than the queue depth which is configured by the app. In my understanding, there's no point of having less mbufs than the total amount of queue entries. If resource is scarce, what's the point of having larger queue depth? It should have smaller queue. > My question is with the proposed change, doesn't this mean that the > application might need to allocate a bigger TX mbuf pool? Because the > pmd can theoretically consume completions slower (as in multiple TX > burst calls)? No. Explained above. [...] > > > Perhaps an adaptive budget (based on online stats) perform better? > > Please bring up any suggestion or submit a patch if any. > > I was simply providing a review for the patch. I don't have the time > to come up with a better patch unfortunately, but I still think its > fair to raise a point. Of course. I appreciate your time for the review. And keep in mind that nothing is impossible in an open source community. I always like to discuss about ideas with anyone. But I was just asking to hear more details about your suggestion if you wanted me to implement it, rather than giving me one-sentence question :-) > > Does "budget" mean the > > threshold? If so, calculation of stats for adaptive threshold can impact single > > core performance. With multiple cores, adjusting threshold doesn't affect much. > > If you look at mlx5e driver in the kernel, it maintains online stats on > its RX and TX queues. It maintain these stats mostly for adaptive > interrupt moderation control (but not only). > > I was suggesting maintaining per TX queue stats on average completions > consumed for each TX burst call, and adjust the stopping condition > according to a calculated stat. In case of interrupt mitigation, it could be beneficial because interrupt handling cost is too costly. But, the beauty of DPDK is polling, isn't it? And please remember to ack at the end of this discussion if you are okay so that this patch can gets merged. One data point is, single core performance (fwd) of vectorized PMD gets improved by more than 6% with this patch. 6% is never small. Thanks for your review again. Yongseok