From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM01-BY2-obe.outbound.protection.outlook.com (mail-by2nam01on0054.outbound.protection.outlook.com [104.47.34.54]) by dpdk.org (Postfix) with ESMTP id 24A937CCD for ; Mon, 5 Jun 2017 06:54:21 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=2zBIjjr8wF956EQ1N0LZqehlF83mJ0N816u3/Pml+yU=; b=Pp2XRONfdaMCPsV0g6mkcP2VT/2RlwF6RrxnSRkBN5F8jhwnGVChAOra1b1qz5f5OxULBrc+EltJpDFb/kaFFS0bjM5gUVD/P7oO65l6hnS2yTfvWsXctZjGrCzfiUjZRM+N6tGjyRlPTbVCZ5msgEoxQ86c+n1FQYwwHqpd/ho= Authentication-Results: nxp.com; dkim=none (message not signed) header.d=none; nxp.com; dmarc=none action=none header.from=caviumnetworks.com; Received: from [10.89.1.2] (14.140.2.178) by CY4PR07MB3093.namprd07.prod.outlook.com (10.172.115.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1124.9; Mon, 5 Jun 2017 04:54:17 +0000 To: Bruce Richardson References: <20170524161101.22863-1-santosh.shukla@caviumnetworks.com> <79f1e8ae-d2cc-e49e-a17d-73c7185b26f8@caviumnetworks.com> <20170602092735.GA51388@bricha3-MOBL3.ger.corp.intel.com> Cc: thomas@monjalon.net, dev@dpdk.org, jerin.jacob@caviumnetworks.com, hemant.agrawal@nxp.com From: santosh Message-ID: Date: Mon, 5 Jun 2017 10:24:11 +0530 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <20170602092735.GA51388@bricha3-MOBL3.ger.corp.intel.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Originating-IP: [14.140.2.178] X-ClientProxiedBy: BN6PR13CA0048.namprd13.prod.outlook.com (10.171.172.34) To CY4PR07MB3093.namprd07.prod.outlook.com (10.172.115.7) X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PR07MB3093: X-MS-Office365-Filtering-Correlation-Id: 7f0f5bb9-9664-445a-eb5c-08d4abceed0c X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(201703131423075)(201703031133081); SRVR:CY4PR07MB3093; X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3093; 3:pNLnXbLEtRk26LREprgJG59zQkiJKgltB8QQEY9OVgw6Lw2bd+BUv59V9MMhHeioBwiioi7jxFucRjEpOr8v4x7/QLVCu2ZB4jUYk0HWufV1o98j7CwthFQ22FZ6hlbzX1N+HUBw8QllCfeS6DibnXh5MChL0wg2NO3g9C6lb1+uv5qfLcdRtRFfwImjbTU9/T7/FDg8OshDCuJLdiM6Wl3z2v6K2MNSJNLdHSgWv6J1Xxuj5Nl3rtC8tsw9FjFE1l3xlJMqRfMUZJ0S5eGclO8kXRZM+xf44Tj9L5evld8tQur8orZOYbzjWzhVHuCX0lbX1uCb9eCfmfhRkvOh2A==; 25:L5rU7KICuYjd+zRBnAivmDbSZVCqJxPdP278dTA7e1aY4zYN4Kt+A6vHHmEu9XUgGpU16sDjofUVyqKzQDJV/SXC4uz7aKe0222VLyD1agKJMn7ba/FgETk5gtaZjsgvx/tnnKUmLHzPiWRWI+Xc31C3u0/5ld0NNDLGfGHFuDkn2x9osUQ4hSrpWuQWPsjxEplCH2kpzqIVZCTpuiHJjl/43uytUwi7ndvki7/399MpjdEZbtZ8TL1lylRub7zDUdx/U4pv0VuIVFYo2HhKjyY4S3AZB2Jxaoo5Hw915RSrJN3jip73GACblxIwt3SMd19qUwZISNEEOhPxMFAfVFFrEJGB0mw5fMrsPYutflS+8iBWR6hFhINNCNov1NIne37HWfJICAQm03yiJzeV68kbcyeBTZiPC8X31i1w0iAwbxVU5lWGvOKCZZYqB8fg9KBqHWlH5veDXGnTk/ISPWo9vr4yWpZLcT0cftxGCGY= X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3093; 31:GzRGIm4VvkHUkc5DRc8yj6pGXHV4ixJh53vkzCjpz3Yp1kq0wu/gwJbD271wrzo/tXxf+AVLVrrE2t3nXQf2hW1CcST/MVM7m9Jl6y6I3i1AIolLOlRZDorXyMXHg6it+hMnJvl49alX6bak0NK9nOmCUF6SzwNFiSEb7DUb2NbTc7iHc+hYGPYTIemxRQnH6rqsJMmTFWB9FwdkhdHRHUaeCvJDZvjc+4b9owOwo88=; 20:ceVUKPmrBuVq2Q30QgOqcFF2l0/5MN7qK7isIdrJjHq2PJQvPo3RHsXNdmIoRv1xXWUTTKXTQzkg7I5r9hWrQtXfQZQzSt7qF95Ka9BKAzrOpJ1JTFYmjPouLUxwkbiVBPYuuxQyb/Yf5S62tMPx0uucLqat6S3ldZfpxhj6TFbGbGhp4tL1vu2iHUtA+d5DYQSAQz5nGvVRx4RlfokPHXk2HVLcm1N2OeKC+Lroqt3SAvWJUnA3IjpSKSVFyOyr6P8GtLAU2VgH+Hg+WzPnElX4Ro6960uriPPDTK0KO6A1MPQ4EmzOM42HCYa84k0DbWIfXBbgtKYgD5Ux6/8d6Kkq7ygGvDpf0MdqST1wS0vKLoxD1rt3tUu3Wt08WfNXyZam0a7KgV0wK9H/+vbpYFR0q0K3y5Oe5KdalKxkcysrAjkhBTcC3il0f4Nj8ivrVQa1CAodERD9yCSkp8KdenB+FQOv/COWGVNAaEsstHCJQ/TVUXGj4aael4c000rDQ2GMwOtfdYQhb4hS3lIYqG0Cq/4fhmESOtci5e4+xaoVnOKseHr2/PDE9Plet6eV/T8i1x9eY+UFibE39mk9ny3j6xsSkCzhKWn240S9M9I= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001)(100000703101)(100105400095)(93006095)(6041248)(20161123555025)(20161123558100)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123564025)(20161123562025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:CY4PR07MB3093; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:CY4PR07MB3093; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1; CY4PR07MB3093; 4:6T3NJ2dep2efVuXL/6dV2kyNRsefD91uAuI2ct?= =?Windows-1252?Q?hNFTB3K1USAd0Y+wh52k/NjJY75ILoXCKKPVjnQke57VvY4TyZ14vXtD?= =?Windows-1252?Q?o17Ndna0RfWmg3sTxinCUxSU6JHiueezC9dW9SAcQuheQLzDBi/CDz48?= =?Windows-1252?Q?978l3uBYJjK13k1QnmciDTxeIsdDMOOwJsBgMwAATtIAINC1FF3HNmv5?= =?Windows-1252?Q?xdB43P8WmtEdKAtJmquGUFCbwKrOkMiOKq02sRCddiqJp32MjJIf4dZp?= =?Windows-1252?Q?BQ3kRLKl4dHrTf42r0fQOVN4ll6q/w5sUxuBjZXhwGjrAugr0CtXSNct?= =?Windows-1252?Q?w/YcDDzGGVDAEZ6182D6Aan+snLXpC0kB+uS7PFYCD/j2lkzOTXUP21I?= =?Windows-1252?Q?MGR91LbacuWLQBWJKdgI4S3oyUguy51t1g0zZ5h4jWtANHTedk2bYkOm?= =?Windows-1252?Q?lwkx/ECpv7roECKOWNH8yt60OdAVyzyiM25ejQqsbxWddji3y39U6hRJ?= =?Windows-1252?Q?E5eZaXwDGl5xO136mhjKrJn4rFqRqLPdojUNgY2ELkwSTXcqZWqCtMCj?= =?Windows-1252?Q?WHlUUxd/KSFMkThvh+1YVWfhwl8PkMmrBPceNGevXDNq3ZxARbenjZtk?= =?Windows-1252?Q?PhfoS66ATgaCbFGBRDcrIp2O1NEGkuK/n1Q6uCX7FHi29BDJZmgXd/W5?= =?Windows-1252?Q?i9mYT+XiwG0t3ptKmTbkuuWIWjMhILJLkae9ezLSBb6oiAOBmcX5o6MM?= =?Windows-1252?Q?iXBpZ+zJVVkI2j8cV+yDTCCLaTeDPu6rJ73dC9pkxy5iiJ/EMdpUSCWT?= =?Windows-1252?Q?GbPcHj/ExWJVK2yhLrE1etdec+BToj3c9W2dzclidBjzkC6zb+zDRF/J?= =?Windows-1252?Q?vPw3zdfdMFigiE62wJ5K1gFkFJHERQhE96LFdKpT+q8SVLLrLZgrxL9I?= =?Windows-1252?Q?JrLWAcTWdDZHXDjNkanPopf5MX1qZYgeXSlIFMHXfeewePfI5Fycqyxy?= =?Windows-1252?Q?BkTTlhxkEQqWbnkPDbXjKunyDAF+sIAvWOCdGQgvYYowLQcb4nyn98Ar?= =?Windows-1252?Q?lIVQB/4iRddAUTUVc1Hu3NzREXLIveKblOXNjx0LCegVgSQL4xri63iY?= =?Windows-1252?Q?Sve+Rg3y30jVUShVnsAfMvTF9ABuXtj5RR7gfJRGmoopDn+Q5Zz1AtyE?= =?Windows-1252?Q?qZRkMNbfoOeN6rjmxZmhjoDJr6qcU=3D?= X-Forefront-PRVS: 0329B15C8A X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6049001)(6009001)(39450400003)(39840400002)(39400400002)(39410400002)(24454002)(377454003)(2906002)(90366009)(47776003)(50986999)(54356999)(76176999)(5660300001)(65826007)(7736002)(305945005)(31696002)(72206003)(42186005)(50466002)(64126003)(65956001)(66066001)(65806001)(4001350100001)(230700001)(5009440100003)(31686004)(6246003)(189998001)(53936002)(6486002)(77096006)(229853002)(110136004)(38730400002)(561944003)(478600001)(83506001)(8656002)(23746002)(4326008)(81166006)(86152003)(6666003)(6116002)(2950100002)(25786009)(36756003)(33646002)(8676002)(42882006)(6916009)(3846002)(110426004); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR07MB3093; H:[10.89.1.2]; FPR:; SPF:None; MLV:nov; PTR:InfoNoRecords; LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1; CY4PR07MB3093; 23:mj0yABwkw8hmZHV83YYCRYfmahUAC2dJC3/w7?= =?Windows-1252?Q?92esFH50vcn6hTKWlNQS09wqssk0+UP8DjlGPpQrRKAVplGFVtXFWAJm?= =?Windows-1252?Q?qbKDly6SsyQEVy/Zi+QIP89fuwb2EAkIYN60+xfSUqSiGW9GSLwgdqdc?= =?Windows-1252?Q?v2UC6ss+5ylb+5jy2k6qCtwaY5vuBOAh0Y4uK7NUaT4BtuEZTdQwNHxG?= =?Windows-1252?Q?Whz8q8xuz6iS9NeF3LT4y0ntHzd5JXH3BrgZdYAq7Sg5Ql0k3SEzcL9F?= =?Windows-1252?Q?RKDfI+MHhPY2rNU59hg99FnSXs5DgJ1D9umDAPYvJl15HAdco9kUM5Jb?= =?Windows-1252?Q?dX/JTU4FDhTYC7twPkc3V+ae6UQzJYBKvaKja5vodmdnjfoNLwJPY9U0?= =?Windows-1252?Q?APmwJTBLSzbDnp6V6qSBgi6s9JH96rPhM87VsGWfJzyJIDtZXlyvsK0N?= =?Windows-1252?Q?dzmzQ0LngwJ4JXJhyCgtlrl4fam0i9SGl+396xLoEdExALnXMaiIBZaY?= =?Windows-1252?Q?enjr8VrjVGfC2g6QKS0HK40T6O86bt1qj36xDWP9wpRu/s3FKcJ03Og5?= =?Windows-1252?Q?UAKX8f3QKDZoYO770d3t+BBSo3zWAHp63ssDwWDGQmanaBLKQaZHNSEj?= =?Windows-1252?Q?74cgbHpoYTZFZ98q1gmr4pUPF0Dfo2lu0NI9Imh7WXw+GY0bcVNJcy7E?= =?Windows-1252?Q?1siSkJa3jPvTnpAODLVlVN+tZxGcAeiUllGq1c/v424kUiDQH0Y90QRd?= =?Windows-1252?Q?AJsfihIgGIBXg09CABIc2omRuA5LFFAytrTDQeh7F5s9l81AsLfbyy19?= =?Windows-1252?Q?2tF2903LOS6FpJZh6kYWCbgYNUufj7nBNXL8Lj68xy8enfe+e6EDK847?= =?Windows-1252?Q?AX/pJ2AEG0X5Hd/+OHYrkhC0qJjspCAeePUpk1+VAw3jmWyNbfKxTAMl?= =?Windows-1252?Q?VCtDxkSuwMl5RqTAxRWqKSudUHov4DE1k0VAdkmOZoEXVYQwxrDRo6B+?= =?Windows-1252?Q?ENyDiHnEi8N3wBvIyUwIPjyE6jp/pbPCv91h1emMn5PWX+U35r3BzdRT?= =?Windows-1252?Q?kP7pNE5HyPEeB7UNZSdu2UcLCjcRDvKDNagR1ElHQ3rgu83dvz+OTGIU?= =?Windows-1252?Q?RamxTzQq/Cewtt0RJdx5NA7DbPhS129Qaw8bves0fXTYWeOzURuVI48w?= =?Windows-1252?Q?53mN1g1PsAFskdXeQ9IQXcHwH4dqD/9NQkJvyRK+UIwhR9pY81X00ioN?= =?Windows-1252?Q?pTQdtGUdfc/GN/LDmBaM5YzEAMQJryRkPnf9xaw1R70la8v/ftcpdmJ5?= =?Windows-1252?Q?+2GkgMsc3em68sWAYjI0Zo5ru99H8V8dbkm9R6ritQ5YbMTJ+6Oflm+J?= =?Windows-1252?Q?Z0qx4MvivPcBG4M+jbzoWuwc5590TbANLyV8jm80zBvIwWilSncnbfFW?= =?Windows-1252?Q?DSSgo3j3Q/fw3K8sKtnVJlRzFlN/cc67iNsDzUqSamP6IKkL6wpPGw2g?= =?Windows-1252?Q?zZGLrcqpE/3G7Zc4+b2lLKgSRlgm23IyvIzaFnS0bXXDpA3Eg=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3093; 6:87x5MKgf1GoesQmzDPmbrFSQyp7hFMmbFaIj16uMdzkD39m0JQuL+hlcyus8Zm2+vsMNRdcviNqQBo+rpLhTRwLrRvhNv7pCeT6CNj5y8l9EPf5EiKA7rjtixKYQT+D1kp5tbmqU0ttONCI0hk1UZ0niEVs+LLdwHaqU8zXvGMbAyb6LYCQTTXVmq8RMm7Atkm7RKWwv3Wum+0qGYbE78spx5x3IKLcUMroa5sVUk4YFVjUVS+x+dzVhQehQwzTrL0CATydEZDStZRUCLAJckLwJyh0bAA3td8ZC+Rv+oNaOuQDm96GqTX7x4pkw/KPVFZ1N09ySYVjLXwWwc51eX+NXK//LM2LTcvLZ0VTABNg9EYMqf9+IhWirog8kENcfhx+9Alxi8HjeUDbNLlBKKsTIGXZ8G3WbWGtAKrB9EGrys6x/xZXPczroLeXJHsTFr07p6IzOqa9W79/J7iy0/yDhCd1l9UPRmCjpS17/b/bv2Jh/uhGGLdNJe0RU+IHDFaKRkBIpNhebg2CsZwezvg== X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3093; 5:d6kLNMHi1+f/eNOR8AvQC/iNFt5xwd+A6fPaCDDtU+Rd+Ho9UqatME7IvRvY634iV9romhD9+5Cr5tmX2jrayUe77eJBEk0krf7GK6y5oM8JAR/zrYGwT+SZnpIqna5bfQN+IWdnJihOgvpe4u7uM5s/UJQILpRLv4T/iDcKT8uk/zgDDFXU8D3ur0QVjjgMVVLMkhzF3Fs2aXbMKrDuP552ah+JcBcTNt7hgZTAKGWiafSn4J32PAPrJvfzzOUiZV/ds49VxsQR+MEe9XoVMXCYOOfPedPwzCRacYuvvRNLxVm1oGCTrVO3cgTXvOhvZym8TEymvpflzNIEQW17gy/xCUvpS7MsBgG0a0ymmTB4CFrxEAIyku1CIjRv81GstdDgfjB79ZH0qq72hI+JLi4wY0ooRM5T+UwZcSSpmFzutgUDx/4qhluWQtxULduvS0yHsqG7V4h108P6Xh7UaA3aXm94IN/Q5SDqQMEx5wDdaf8WTluFGr6CGcYuW3aH; 24:X1pB8hUERGz5RhJtQzlKuGGdYfVMghGv95E2VBPhfZSXsevBpqif6PTyBQx37RuyVFEbX5fj3f8DzO3qVu/mVcVXvu65x+Vy7k5MN71EyW0= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3093; 7:o/DXoeQUCsFQvh6gRpOyH82vqmAJLd44BXSgwVkUtwy6mpcn7B0E5mrwyPdUVSvhONvBHslyf8VrOxec7hyZGTVdJ/EXwY63nSuvP9FZ50lOMdZPXYjnYohIjdusNKw4c88xmLzPq0Ec+JebUJQZB0eNwJqSpv1A/camjTTkwOs9pBATh40BBHsQu93SPElg+aeUu/p3ASpZasHUzFIjLwi8Zdc3bdPLnmqZK0tBtaR64pQSFg6gtBQkpkDtX8m4+bOSpN0YtHuUJrtCie2iNtzipNgqlXCDUEcl7ENfYvegJQS3kFjGqp3Ygt7Zb2QMSy2FCjbK1dkqlmHVnp7UaQ== X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jun 2017 04:54:17.2241 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR07MB3093 Subject: Re: [dpdk-dev] [RFC] eal/memory: introducing an option to set iova as va X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jun 2017 04:54:22 -0000 Hi Bruce, On Friday 02 June 2017 02:57 PM, Bruce Richardson wrote: > On Fri, Jun 02, 2017 at 09:54:46AM +0530, santosh wrote: >> Ping? >> >> On Wednesday 24 May 2017 09:41 PM, Santosh Shukla wrote: >> >>> Some NPU hardware like OCTEONTX follows push model to get >>> the packet from the pktio device. Where packet allocation >>> and freeing done by the HW. Since HW can operate only on >>> IOVA with help of SMMU/IOMMU, When packet receives from the >>> Ethernet device, It is the IOVA address(which is PA in existing scheme). >>> >>> Mapping IOVA as PA is expensive on those HW, where every >>> packet needs to be converted to VA from PA/IOVA. >>> >>> This patch proposes the scheme where the user can set IOVA >>> as VA by using an eal command line argument. That helps to >>> avoid costly lookup for VA in SW by leveraging the SMMU >>> translation feature. >>> >>> Signed-off-by: Santosh Shukla >>> --- > Hi, > > I agree this is a problem that needs to be solved, but this doesn't look > like a particularly future-proofed solution. Given that we should > use the IOMMU on as many platforms as possible for protection, we > probably need to find an automatic way for DPDK to use IO addresses > correctly. Is this therefore better done as part of the VFIO and > UIO-specific code in EAL - as that is the part that knows how the memory > mapping is done, and in the VFIO case, what address ranges were > programmed in. The mempool driver was something else I considered but it > is probably too high a level to implement this. The other approach which we evaluated, Its detail: 0) Introduce a new bus api whose job is to detect iommu capable devices on that bus {/ are those devices bind to iommu capable driver or not?}. Let's call that api rte_bus_chk_iommu_dev(); 1) The scheme is like If _all_ the devices bind to iommu kdrv then return iova=va 2) Otherwise switch to default mode i.e.. iova=pa. 3) Based on rte_bus_chk_iommu_dev() return value, accordingly program iova=va Or iova=pa in vfio_type1/spapr_map(). 4) User from the command line can always override iova=va, in case if he wants to default scheme( iova=pa mode). For that purpose - Introduce eal option something like --iova-pa Or --override-iova Or --iova-default or some better name. Proposed API snap: enum iova_mode { iova_va; iova_pa; iova_unknown; }; /** * Look for iommu devices on that Bus. * And find out that those devices bind to iommu * capable driver example vfio. * * * @return * On success return valid iova mode (iova_va or iova_pa) * On failure return iova_unkown. */ typedef int (*rte_bus_chk_iommu_dev_t)(void); By this approach, - We can automatically detect iova is va or pa and then program accordingly. - Also, the user can always switch to default iova mode. - Drivers like dpaa2 can use this API to detect iova mode then program dma_map accordingly. Currently they are doing in ifdef-way. Comments? thoughts? Or if anyone has better proposal then, please suggest. > So, in short, I don't particularly like this solution, but I could live > with it as a short-term option. Longer term though, I think we need a > better way to support using IO addresses rather than physical addresses > - I just don't know what that would look like or where it would sit/live. > > /Bruce Thanks,.