From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id D7900A320B for ; Mon, 21 Oct 2019 13:45:35 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 939DA1BED0; Mon, 21 Oct 2019 13:45:34 +0200 (CEST) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by dpdk.org (Postfix) with ESMTP id 569AA1BECA for ; Mon, 21 Oct 2019 13:45:32 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Oct 2019 04:45:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.67,323,1566889200"; d="scan'208";a="209319604" Received: from fyigit-mobl.ger.corp.intel.com (HELO [10.237.221.10]) ([10.237.221.10]) by orsmga002.jf.intel.com with ESMTP; 21 Oct 2019 04:45:28 -0700 To: Vamsi Krishna Attunuru , Stephen Hemminger , "Yigit, Ferruh" Cc: "dev@dpdk.org" , "thomas@monjalon.net" , Jerin Jacob Kollanukkaran , "olivier.matz@6wind.com" , "anatoly.burakov@intel.com" , "arybchenko@solarflare.com" , Kiran Kumar Kokkilagadda References: <20190729121313.30639-2-vattunuru@marvell.com> <20190816061252.17214-1-vattunuru@marvell.com> <20190816061252.17214-5-vattunuru@marvell.com> <20191015084603.3b0f141f@hermes.lan> <5cd2398c-684c-44c5-1a0b-6cde4b7e961d@intel.com> From: Ferruh Yigit Openpgp: preference=signencrypt Autocrypt: addr=ferruh.yigit@intel.com; prefer-encrypt=mutual; keydata= mQINBFXZCFABEADCujshBOAaqPZpwShdkzkyGpJ15lmxiSr3jVMqOtQS/sB3FYLT0/d3+bvy qbL9YnlbPyRvZfnP3pXiKwkRoR1RJwEo2BOf6hxdzTmLRtGtwWzI9MwrUPj6n/ldiD58VAGQ +iR1I/z9UBUN/ZMksElA2D7Jgg7vZ78iKwNnd+vLBD6I61kVrZ45Vjo3r+pPOByUBXOUlxp9 GWEKKIrJ4eogqkVNSixN16VYK7xR+5OUkBYUO+sE6etSxCr7BahMPKxH+XPlZZjKrxciaWQb +dElz3Ab4Opl+ZT/bK2huX+W+NJBEBVzjTkhjSTjcyRdxvS1gwWRuXqAml/sh+KQjPV1PPHF YK5LcqLkle+OKTCa82OvUb7cr+ALxATIZXQkgmn+zFT8UzSS3aiBBohg3BtbTIWy51jNlYdy ezUZ4UxKSsFuUTPt+JjHQBvF7WKbmNGS3fCid5Iag4tWOfZoqiCNzxApkVugltxoc6rG2TyX CmI2rP0mQ0GOsGXA3+3c1MCdQFzdIn/5tLBZyKy4F54UFo35eOX8/g7OaE+xrgY/4bZjpxC1 1pd66AAtKb3aNXpHvIfkVV6NYloo52H+FUE5ZDPNCGD0/btFGPWmWRmkPybzColTy7fmPaGz cBcEEqHK4T0aY4UJmE7Ylvg255Kz7s6wGZe6IR3N0cKNv++O7QARAQABtCVGZXJydWggWWln aXQgPGZlcnJ1aC55aWdpdEBpbnRlbC5jb20+iQJUBBMBCgA+AhsDAh4BAheABQsJCAcDBRUK CQgLBRYCAwEAFiEE0jZTh0IuwoTjmYHH+TPrQ98TYR8FAl1meboFCQlupOoACgkQ+TPrQ98T YR9ACBAAv2tomhyxY0Tp9Up7mNGLfEdBu/7joB/vIdqMRv63ojkwr9orQq5V16V/25+JEAD0 60cKodBDM6HdUvqLHatS8fooWRueSXHKYwJ3vxyB2tWDyZrLzLI1jxEvunGodoIzUOtum0Ce gPynnfQCelXBja0BwLXJMplM6TY1wXX22ap0ZViC0m714U5U4LQpzjabtFtjT8qOUR6L7hfy YQ72PBuktGb00UR/N5UrR6GqB0x4W41aZBHXfUQnvWIMmmCrRUJX36hOTYBzh+x86ULgg7H2 1499tA4o6rvE13FiGccplBNWCAIroAe/G11rdoN5NBgYVXu++38gTa/MBmIt6zRi6ch15oLA Ln2vHOdqhrgDuxjhMpG2bpNE36DG/V9WWyWdIRlz3NYPCDM/S3anbHlhjStXHOz1uHOnerXM 1jEjcsvmj1vSyYoQMyRcRJmBZLrekvgZeh7nJzbPHxtth8M7AoqiZ/o/BpYU+0xZ+J5/szWZ aYxxmIRu5ejFf+Wn9s5eXNHmyqxBidpCWvcbKYDBnkw2+Y9E5YTpL0mS0dCCOlrO7gca27ux ybtbj84aaW1g0CfIlUnOtHgMCmz6zPXThb+A8H8j3O6qmPoVqT3qnq3Uhy6GOoH8Fdu2Vchh TWiF5yo+pvUagQP6LpslffufSnu+RKAagkj7/RSuZV25Ag0EV9ZMvgEQAKc0Db17xNqtSwEv mfp4tkddwW9XA0tWWKtY4KUdd/jijYqc3fDD54ESYpV8QWj0xK4YM0dLxnDU2IYxjEshSB1T qAatVWz9WtBYvzalsyTqMKP3w34FciuL7orXP4AibPtrHuIXWQOBECcVZTTOdZYGAzaYzxiA ONzF9eTiwIqe9/oaOjTwTLnOarHt16QApTYQSnxDUQljeNvKYt1lZE/gAUUxNLWsYyTT+22/ vU0GDUahsJxs1+f1yEr+OGrFiEAmqrzpF0lCS3f/3HVTU6rS9cK3glVUeaTF4+1SK5ZNO35p iVQCwphmxa+dwTG/DvvHYCtgOZorTJ+OHfvCnSVjsM4kcXGjJPy3JZmUtyL9UxEbYlrffGPQ I3gLXIGD5AN5XdAXFCjjaID/KR1c9RHd7Oaw0Pdcq9UtMLgM1vdX8RlDuMGPrj5sQrRVbgYH fVU/TQCk1C9KhzOwg4Ap2T3tE1umY/DqrXQgsgH71PXFucVjOyHMYXXugLT8YQ0gcBPHy9mZ qw5mgOI5lCl6d4uCcUT0l/OEtPG/rA1lxz8ctdFBVOQOxCvwRG2QCgcJ/UTn5vlivul+cThi 6ERPvjqjblLncQtRg8izj2qgmwQkvfj+h7Ex88bI8iWtu5+I3K3LmNz/UxHBSWEmUnkg4fJl Rr7oItHsZ0ia6wWQ8lQnABEBAAGJAjwEGAEKACYCGwwWIQTSNlOHQi7ChOOZgcf5M+tD3xNh HwUCXWZ5wAUJB3FgggAKCRD5M+tD3xNhH2O+D/9OEz62YuJQLuIuOfL67eFTIB5/1+0j8Tsu o2psca1PUQ61SZJZOMl6VwNxpdvEaolVdrpnSxUF31kPEvR0Igy8HysQ11pj8AcgH0a9FrvU /8k2Roccd2ZIdpNLkirGFZR7LtRw41Kt1Jg+lafI0efkiHKMT/6D/P1EUp1RxOBNtWGV2hrd 0Yg9ds+VMphHHU69fDH02SwgpvXwG8Qm14Zi5WQ66R4CtTkHuYtA63sS17vMl8fDuTCtvfPF HzvdJLIhDYN3Mm1oMjKLlq4PUdYh68Fiwm+boJoBUFGuregJFlO3hM7uHBDhSEnXQr5mqpPM 6R/7Q5BjAxrwVBisH0yQGjsWlnysRWNfExAE2sRePSl0or9q19ddkRYltl6X4FDUXy2DTXa9 a+Fw4e1EvmcF3PjmTYs9IE3Vc64CRQXkhujcN4ZZh5lvOpU8WgyDxFq7bavFnSS6kx7Tk29/ wNJBp+cf9qsQxLbqhW5kfORuZGecus0TLcmpZEFKKjTJBK9gELRBB/zoN3j41hlEl7uTUXTI JQFLhpsFlEdKLujyvT/aCwP3XWT+B2uZDKrMAElF6ltpTxI53JYi22WO7NH7MR16Fhi4R6vh FHNBOkiAhUpoXRZXaCR6+X4qwA8CwHGqHRBfYFSU/Ulq1ZLR+S3hNj2mbnSx0lBs1eEqe2vh cA== Message-ID: Date: Mon, 21 Oct 2019 12:45:27 +0100 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] [EXT] Re: [PATCH v10 4/5] kni: add IOVA=VA support in KNI module X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 10/18/2019 6:15 PM, Vamsi Krishna Attunuru wrote: > > >> -----Original Message----- >> From: dev On Behalf Of Ferruh Yigit >> Sent: Wednesday, October 16, 2019 9:44 PM >> To: Vamsi Krishna Attunuru ; Stephen Hemminger >> ; Yigit, Ferruh >> >> Cc: dev@dpdk.org; thomas@monjalon.net; Jerin Jacob Kollanukkaran >> ; olivier.matz@6wind.com; >> anatoly.burakov@intel.com; arybchenko@solarflare.com; Kiran Kumar >> Kokkilagadda >> Subject: Re: [dpdk-dev] [EXT] Re: [PATCH v10 4/5] kni: add IOVA=VA support >> in KNI module >> >> On 10/16/2019 12:26 PM, Vamsi Krishna Attunuru wrote: >>> >>> >>>> -----Original Message----- >>>> From: Stephen Hemminger >>>> Sent: Tuesday, October 15, 2019 9:16 PM >>>> To: Yigit, Ferruh >>>> Cc: Vamsi Krishna Attunuru ; dev@dpdk.org; >>>> thomas@monjalon.net; Jerin Jacob Kollanukkaran ; >>>> olivier.matz@6wind.com; ferruh.yigit@intel.com; >>>> anatoly.burakov@intel.com; arybchenko@solarflare.com; Kiran Kumar >>>> Kokkilagadda >>>> Subject: [EXT] Re: [dpdk-dev] [PATCH v10 4/5] kni: add IOVA=VA >>>> support in KNI module >>>> >>>> External Email >>>> >>>> --------------------------------------------------------------------- >>>> - >>>> On Tue, 15 Oct 2019 16:43:08 +0100 >>>> "Yigit, Ferruh" wrote: >>>> >>>>> On 8/16/2019 7:12 AM, vattunuru@marvell.com wrote: >>>>>> From: Kiran Kumar K >>>>>> >>>>>> Patch adds support for kernel module to work in IOVA = VA mode, the >>>>>> idea is to get physical address from IOVA address using >>>>>> iommu_iova_to_phys API and later use phys_to_virt API to convert >>>>>> the physical address to kernel virtual address. >>>>>> >>>>>> When compared with IOVA = PA mode, there is no performance drop >>>>>> with this approach. >>>>>> >>>>>> This approach does not work with the kernel versions less than >>>>>> 4.4.0 because of API compatibility issues. >>>>>> >>>>>> Patch also updates these support details in KNI documentation. >>>>>> >>>>>> Signed-off-by: Kiran Kumar K >>>>>> Signed-off-by: Vamsi Attunuru >>>>> >>>>> <...> >>>>> >>>>>> @@ -348,15 +351,65 @@ kni_ioctl_create(struct net *net, uint32_t >>>> ioctl_num, >>>>>> strncpy(kni->name, dev_info.name, RTE_KNI_NAMESIZE); >>>>>> >>>>>> /* Translate user space info into kernel space info */ >>>>>> - kni->tx_q = phys_to_virt(dev_info.tx_phys); >>>>>> - kni->rx_q = phys_to_virt(dev_info.rx_phys); >>>>>> - kni->alloc_q = phys_to_virt(dev_info.alloc_phys); >>>>>> - kni->free_q = phys_to_virt(dev_info.free_phys); >>>>>> - >>>>>> - kni->req_q = phys_to_virt(dev_info.req_phys); >>>>>> - kni->resp_q = phys_to_virt(dev_info.resp_phys); >>>>>> - kni->sync_va = dev_info.sync_va; >>>>>> - kni->sync_kva = phys_to_virt(dev_info.sync_phys); >>>>>> + if (dev_info.iova_mode) { >>>>>> +#ifdef HAVE_IOVA_AS_VA_SUPPORT >>>>>> + pci = pci_get_device(dev_info.vendor_id, >>>>>> + dev_info.device_id, NULL); >>>>>> + if (pci == NULL) { >>>>>> + pr_err("pci dev does not exist\n"); >>>>>> + return -ENODEV; >>>>>> + } >>>>> >>>>> If there is no PCI device KNI should still work. >>>> >>>> Right now it is possible to use KNI with netvsc PMD on Hyper-V/Azure. >>>> With this patch that won't be possible. >>> >>> Hi Ferruh, Stephen, >>> >>> These can be fixed by forcing iommu_mode as PA when vdevs are used for >>> KNI usecase. >>> >>> rte_bus_get_iommu_class(void) >>> { >>> enum rte_iova_mode mode = RTE_IOVA_DC; >>> + struct rte_devargs *devargs = NULL; >>> bool buses_want_va = false; >>> bool buses_want_pa = false; >>> struct rte_bus *bus; >>> >>> + if (rte_eal_check_module("rte_kni") == 1) { >>> + RTE_EAL_DEVARGS_FOREACH("vdev", devargs) { >>> + return RTE_IOVA_PA; >>> + } >>> + } >>> + >>> TAILQ_FOREACH(bus, &rte_bus_list, next) { >>> enum rte_iova_mode bus_iova_mode; >>> >>> I think this will solve various use cases/combinations like PA or VA mode, >> pdev or vdev used for KNI. >>> Existing use cases would not be affected by these patch series with above >> fix. >>> >> >> Hi Vamsi, >> >> I think this is not a problem of using vdev so I think we can't solve this via >> vdev check only. >> >> The sample I give (KNI PMD) is using vdev, but application can use KNI library >> APIs directly to create kni interface and have kernel/user space >> communication without any device involved. KNI PMD (vdev) is just a >> wrapper to make this easy. >> >> Just thinking aloud, >> KNI is sharing in userspace buffer with kernel, so basically it needs to do >> virtual address to kernel virtual address translation, in a reasonably fast >> manner. >> >> iova=va breaks KNI because: >> 1) physical memory is not continuous anymore which break our address >> translation logic >> 2) we were using physical address of the buffer for the address translation, >> but we have no more have it, we now have iova address. >> >> I assume 1) is resolved with 'rte_kni_pktmbuf_pool_create()' it would be >> helpful though if you can explain how this works? > > KNI kernel module uses pa2va, va2pa calls to translate buffer pointers, this will only work when both pa & va are 1-to-1 contiguous. In such cases, dpdk mbuf memory should not cross physical page boundary, If it crosses, it's iova/va will be contiguous, but it's pa may or may not be contiguous. Kni pktmbuf pool api ensures that mempool is not populated with those type of mbufs(that cross page boundary). Once the pool is created, all mbufs in that pool resides with in the page limits. ack. >> >> For second, a simple question, do we need to get a PCIe device information >> to be able to convert iova to kernel virtual address? Can't I get this >> information from iommu somehow? >> Think about a case "--iova-mode=va" provided but there is no physical device >> bind to vfio-pci, can I still allocated memor? And how can I use KNI in that >> case? > > We found a way to translate iova/uva to kva using kernel mm subsystem calls, using get_user_pages_remote() call KNI module is able to get mapping for iova to kva. This solution worked for pdevs(without sharing any dev info) and also for wrapper vdev pmds. Good to hear this. > > I will push next version of patches with these solution and other command line arg changes. Since the major architectural issue is fixed in enabling iova=va for KNI, we are planning to merge these patch series in 19.11 release. >