From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM02-CY1-obe.outbound.protection.outlook.com (mail-cys01nam02on0042.outbound.protection.outlook.com [104.47.37.42]) by dpdk.org (Postfix) with ESMTP id 8AC509FE for ; Wed, 20 Sep 2017 13:25:02 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=X8Bs8C7ko1nD/IJkHegD3LMlIuXA6QfKCmZv6FJ/Ixg=; b=M/MUKx75JLgRWVQIH6QmnxpvSrI+gL8bYV9seP2KlHS5QrOMRCG+U2CNZFP/6I5EwVAMqRp5XdDs820QH5UAHkmjC1uyT3IUhKjsaj+vip7PtLNM8LfzQM0vJzaQyAuYyKjOOuGej/yeqAlVIuMF6dvsA2E4f0QzGnouTJ40J4c= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Santosh.Shukla@cavium.com; Received: from localhost.localdomain (111.93.218.67) by DM5PR07MB3098.namprd07.prod.outlook.com (10.172.85.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.77.7; Wed, 20 Sep 2017 11:24:54 +0000 From: Santosh Shukla To: dev@dpdk.org Cc: olivier.matz@6wind.com, thomas@monjalon.net, jerin.jacob@caviumnetworks.com, hemant.agrawal@nxp.com, aconole@redhat.com, stephen@networkplumber.org, anatoly.burakov@intel.com, gaetan.rivet@6wind.com, shreyansh.jain@nxp.com, bruce.richardson@intel.com, sergio.gonzalez.monroy@intel.com, maxime.coquelin@redhat.com, Santosh Shukla Date: Wed, 20 Sep 2017 16:53:50 +0530 Message-Id: <20170920112356.17629-4-santosh.shukla@caviumnetworks.com> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20170920112356.17629-1-santosh.shukla@caviumnetworks.com> References: <20170918104234.9149-1-santosh.shukla@caviumnetworks.com> <20170920112356.17629-1-santosh.shukla@caviumnetworks.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [111.93.218.67] X-ClientProxiedBy: BM1PR01CA0076.INDPRD01.PROD.OUTLOOK.COM (10.174.208.144) To DM5PR07MB3098.namprd07.prod.outlook.com (10.172.85.8) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8cca87bb-813a-4ee4-7fc9-08d5001a3b79 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(300000503095)(300135400095)(2017052603199)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:DM5PR07MB3098; X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3098; 3:naRJnllnKr+kMD8TrUUC12CMZBtV1bQtWg7dBJAy5OWvnwnN78CjnRPENtquDFQG3B9bbFPZQ+GA0rlF67r6iARj0/gFdzKfFT2eCurFxFSi+AvSz1+Av1/GSQgHQ+9XYeiL3cY8BSytDu+5uwdklhcX3kVkYPhqC8kuxquzPpz6Oi2EwylIWX6mMqxaJhbTqkhck/rP6vSY47hvf70CzFbdy1vz1G1waCpUL4g/lGAN7Ac6GE8BHKuMprflykJU; 25:topBJShgw5odU908jQOnRATfrW10DCZrD5pjCfn+zxYY+QSbIkrksSf5i+tuQ8U5G+3mLN5pNNZndUJnzIaEu7Wrx5Gj58IHx9V8Z5xqtf0JRnyk9i8igoB/diNlCDHUEcf8axlR3HDmazx3RaMgCoJU7vx9VdaIZarmpTPYfTv0uVqk52SLXxVg8m1KL/5wgF2Hxth2DwzzrIBT1euGRmaGTkCIlZjnbpvEbP1XOC28UvEK/3qPoJsnZ+D09jqfy96EhL+POatmkkW8hSpTQHDSBgg9K8yGobR5grdR0sl2hzQLFWgS0X9yTgwWLeGYSS/wmRIE/8Jc8XYsj8s4YQ==; 31:qUH05TgnO73s5rgaJmqO76ImDdxnL97nW4WVLjjgwWtRB8/wHHKiY4FIE4kImd3Po0FI17XuUmLocOTzMR/x4l95Fjy+Y0LCFuIShd/Uwylt8qdfCV57W49ePkAm5r3gAcC1QVmYwgSzBRkxkqpBpNAfoXaGudtJRSjjsBlXEWcuWtuzaSsaXUJHAQtVzxiEAmbl8VEzjK9FRPxalFLRLsCzJoUHmUccee4kc5P8N90= X-MS-TrafficTypeDiagnostic: DM5PR07MB3098: X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3098; 20:SDo0lTOuCYqkS9KELZqQCr1ghMSo/WIeJ0XCAv3iuPmZoaENBavUbhTti+tc7AIW4VdXJCLahUNUkcoNLzRanI4hKyEs1itPThJjLF/okacu82BNoKNIVevO98Nkgj0zZ0LbNq5X0BbX3QedW8GOIgfJzeaTI/W+vAaDh2bT7Uiry93AGXL2935nw+xTCNv+lcXMhaT1VPJJ7XiJOvyyNRY7WJ6u1EJUs7YfH8sNqL/mzTGh3mj5WuZaCFUxloc1TgIstTWqWvQxwdjdyvYMB4E4iRa7eMZ9+MwfKhWWiXd1pLrH7/Ch7DOP+m9E/A+/qw4MuViIVi/lJWnxlBVHRg+scABTI9cVeRc0JWdWFhHowXFcbyLhMirhpeUHRDvlH7TcV5N7DTC22RDk3XwV0+VOAAtbKMp7OGnqnWGdufAPALnSHw1MdOeqDdA0pWR9adig1t5YR4xkXeSc7RR9rdhvzhspImf8besxOMMosg7BGSlcgdSDkgYqgNkvTZlYz4lCRH6kZjQMJOYw2VEfDm1Dn1VFAmLP2lMaiBfIL8wMMKpASoKlvXE8pgAPnI5fY1SJ6C88tmP9zGfxXrefkFKB7rClw/JnuIXzq875b2o= X-Exchange-Antispam-Report-Test: UriScan:(185117386973197)(228905959029699); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(10201501046)(100000703101)(100105400095)(93006095)(3002001)(6041248)(20161123564025)(20161123555025)(20161123562025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123560025)(20161123558100)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:DM5PR07MB3098; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:DM5PR07MB3098; X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3098; 4:mO71PsgH563utokFoVhLtVKFD0hKbt736NNSIQTFmWj6xrpcRp84YJajpftSA4cQT2x5KgODOyZjfuJwMHAwz2d7Z2KPP7lzCaBrRZTWJZg6eksp7l7VNDqhghD4PUU+FCZh1VWL9ADypA1AGOfUx2eVSbp0qic1GFCHZUGSL2ZjGzsZy3Q5nqAjpDnzyI07he8vgXT6D/7hqvZm/il5+ZkN/hjpceBflflnalAupJ0/wv2DGFk0yRDqnPy3wYMDpP7z9rrHeagCiz0OMpYzQxK6Fb3aV2osjFkcfmuwt7tk7mUpjBvQokRPadrD+VpI5ToijwuntgfUuPb770U5GA== X-Forefront-PRVS: 04362AC73B X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(6069001)(346002)(376002)(189002)(199003)(66066001)(3846002)(106356001)(8936002)(7736002)(2361001)(33646002)(6506006)(2351001)(47776003)(25786009)(105586002)(53936002)(6116002)(316002)(36756003)(6486002)(107886003)(68736007)(5009440100003)(6666003)(101416001)(6512007)(81156014)(81166006)(97736004)(4326008)(50986999)(76176999)(16526017)(8676002)(2906002)(16586007)(5890100001)(189998001)(305945005)(478600001)(72206003)(2950100002)(5003940100001)(1076002)(48376002)(50226002)(6916009)(42882006)(50466002)(5660300001)(8656003)(7416002); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR07MB3098; H:localhost.localdomain; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DM5PR07MB3098; 23:0iWFYNKbFaF+XgA3SHexjKo2W+uS0hkCexDe9piDG?= =?us-ascii?Q?tU+U/FROXdxOrXGNPsR3mVsJOF2syJuQl7/OKv+F/s6ACz0qmiK3vZBvg1ab?= =?us-ascii?Q?gxEe0XlioZHkra+xOLgLTZxY1qXTffF7pcq1r/xT94J0VgQY+rbzLrC0Z1tI?= =?us-ascii?Q?fWf6naeB+9yBBBEcqcqSgy/d8o5drkl0gA/dIOIa0671aBESkGyHJg5S+nzQ?= =?us-ascii?Q?OThMTaa/jAQ1p0Q3162K/yR3OyN3813GmhNh/kpdyZpdPzJ87MrJm+5py8tB?= =?us-ascii?Q?EID74Xnwlf98d+HMPnFhtKPAokSacvfZ+a4aUBKm+QTY29VZC9Er4oFyKn59?= =?us-ascii?Q?DmePPwCZuJZMmKItqTwHrD3Awgf9prBDKLbYtzAn25nZnfHko98jcxXY3Lq2?= =?us-ascii?Q?+6KLIu9Xf7d6vF0hX/QnpXGUgG8UNU/KRAKpoAfXd55WmkIqRfnJAGFJxLXs?= =?us-ascii?Q?iNFe/cU4me6S3q5reh64lDjm4Ittgwm/lENsSNs2CDJAldud7wu2QZ4rGqwK?= =?us-ascii?Q?hiYmySe6T4Ij6zPakVYnyUvOhj8z5NPPq4GGmLFNlgmGicqM7yXQJn7NWl8O?= =?us-ascii?Q?HnClSgSZ1i9QXP6sXlK5wWJ8OFRjQWVU5uSVM9ZlHPrc117ja2C4DThh0+dR?= =?us-ascii?Q?bITQY06DGEzeM4WIiXqPiV0JX7V5pLbFZw5Cwqr3+53qmZsTGRJDDx2UpwY8?= =?us-ascii?Q?MMQWbZ0S2SahNUboBffC8ZfOovRybSKggMpMOFh84rWpsEDwUgkGo+dXIzue?= =?us-ascii?Q?vp+RFlYM9hua2OmkPC6QbmxH0zLtH1F9GXGzoz1Lwp+pa9qFBjFcWqQdrYfS?= =?us-ascii?Q?hlUKgxPrHuutQHhSZ3N5nLggYFzIsEyBskwAEnlGm/QkNUIjbB/31AbWQIJR?= =?us-ascii?Q?veq8Bsl0fOT0oAtLov/GJeVxg/SNn/nxWMHoLVFCUbN2ChSYSsaQP6nY49Sb?= =?us-ascii?Q?KUkCYgiIUYleuZFwG1q5dJZvIgMo2e3w2CXMpiSSq4Au0wiHcRwWqG8iDGzP?= =?us-ascii?Q?z/QElhgUZshFIk1yxmKcwv17YNm0ZIQenzZ/Uh3va52L/CN1cMfznFTB20jj?= =?us-ascii?Q?Qeea6DYT/FFTeB5sw5J7UfT8KQnPZGlt0do8yrpKr79Fv9m4pD6qlt6ylcd2?= =?us-ascii?Q?7oYsaTsDxjUO+UhfMDY7ULSD65YdP5hWzsDEzvpej2VwKXvyxeh75q7dMCO9?= =?us-ascii?Q?oXxvrJ8wIDukFOuUP7cO7LqQztTLxzbDWy56+1nYZrTPyw2UOAjToFeZqEpj?= =?us-ascii?Q?ySktA1jFSEu56j3S76SU56lMonVpgMZKSyxwAVT?= X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3098; 6:vAb8ZWmJf2wMInkaf/mfoVD42MRpMaiLiPuEUsbvkmqvnIoz1t4XG12TOKJ4Zr+eSAmTbrU/LxwAkJ4ziJzJYhw9DwgIVrFj3c8lg7mNc++iDK+uSJxtru2qvYPzfO5ceDEoWigbklss3XAjsAPURkeemThAHbhGSGPofpbFq5lpMkKEYiuW8FTe3ppK0UaVig23Qww3+nfZ7MqPxFaU24DG0njJLXenQvK9qicsrsHMIqhH7obrVF9U0yetHIt1yEMmwHImPFtQ4qqI5V6Y8KpL9CTW9teN8i0xAYMLiZI7SH5Hsqb7x0F+zwZB1mvA2Q1S5C5PGPr1TETxo26wrQ==; 5:dV5bMeDhGcPX3/dvSF4K0M0tHUl8l3tpl2vV9Pg1wXxsMYvgju2iCsP5kV4cKkcZdj6gqXcBKrMmU1E0uDaUCg5WAAzyB4zRUoMqZwWfC7nEGmVqE6v4A9IDg7RWdTzWo+LFTowjQ4uzrQ0vxcaTQA==; 24:Fo/yXvgjbFhz/Ik1Lej2h1qXVCLQuqUjzAMtt3ndrkSre66OOnD+YUD/CNj8F22jP5OeLTjSj3Xto1qQsqxg0fKdtElIL3n2wDxDEQ0hwKw=; 7:6BCYqKzSdwKUUtJHd3ShkzC0N+s1vaVIjKor9bAJcRj4dMuEejzEhPZg9siLJOCxi1eYhe9H6K4FvlFgab/3Ui7cDClGTlyli9RREf8DcVwmXdSTdOgT7S2PUhaYxVs6xOSx2KWKoSHAifoHBq8N+QJYwywv91t81PUWnyxue7GwrEbzyj89My/xeFvhRkJRfZHsXsY7f9Shb6S6nu1Z/agToCgr4z4fePQIrLBzlTs= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2017 11:24:54.7885 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR07MB3098 Subject: [dpdk-dev] [PATCH v9 3/9] linuxapp/eal_pci: get iommu class X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Sep 2017 11:25:03 -0000 Get iommu class of PCI device on the bus and returns preferred iova mapping mode for that bus. Patch also introduces RTE_PCI_DRV_IOVA_AS_VA drv flag. Flag used when driver needs to operate in iova=va mode. Algorithm for iova scheme selection for PCI bus: 0. If no device bound then return with RTE_IOVA_DC mapping mode, else goto 1). 1. Look for device attached to vfio kdrv and has .drv_flag set to RTE_PCI_DRV_IOVA_AS_VA. 2. Look for any device attached to UIO class of driver. 3. Check for vfio-noiommu mode enabled. If 2) & 3) is false and 1) is true then select mapping scheme as RTE_IOVA_VA. Otherwise use default mapping scheme (RTE_IOVA_PA). Signed-off-by: Santosh Shukla Signed-off-by: Jerin Jacob Reviewed-by: Maxime Coquelin Acked-by: Hemant Agrawal Reviewed-by: Anatoly Burakov Tested-by: Hemant Agrawal --- v7 --> v8: - Replaced 0/1 with false/true boolean value (Suggested by Anatoly) v6 --> v7: - squashed v6 series patch no [01/12] & [05/12].. i.e.. moved RTE_PCI_DRV_IOVA_AS_VA flag into this patch. (Aaron comment). lib/librte_eal/common/include/rte_pci.h | 2 + lib/librte_eal/linuxapp/eal/eal_pci.c | 89 ++++++++++++++++++++++++++++++++- lib/librte_eal/linuxapp/eal/eal_vfio.c | 19 +++++++ lib/librte_eal/linuxapp/eal/eal_vfio.h | 4 ++ 4 files changed, 113 insertions(+), 1 deletion(-) diff --git a/lib/librte_eal/common/include/rte_pci.h b/lib/librte_eal/common/include/rte_pci.h index 0e36de093..a67d77f22 100644 --- a/lib/librte_eal/common/include/rte_pci.h +++ b/lib/librte_eal/common/include/rte_pci.h @@ -202,6 +202,8 @@ struct rte_pci_bus { #define RTE_PCI_DRV_INTR_RMV 0x0010 /** Device driver needs to keep mapped resources if unsupported dev detected */ #define RTE_PCI_DRV_KEEP_MAPPED_RES 0x0020 +/** Device driver supports iova as va */ +#define RTE_PCI_DRV_IOVA_AS_VA 0X0040 /** * A structure describing a PCI mapping. diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c index 26f2be822..2971f1d4f 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci.c @@ -45,6 +45,7 @@ #include "eal_filesystem.h" #include "eal_private.h" #include "eal_pci_init.h" +#include "eal_vfio.h" /** * @file @@ -488,11 +489,97 @@ rte_pci_scan(void) } /* - * Get iommu class of pci devices on the bus. + * Is pci device bound to any kdrv + */ +static inline int +pci_device_is_bound(void) +{ + struct rte_pci_device *dev = NULL; + int ret = 0; + + FOREACH_DEVICE_ON_PCIBUS(dev) { + if (dev->kdrv == RTE_KDRV_UNKNOWN || + dev->kdrv == RTE_KDRV_NONE) { + continue; + } else { + ret = 1; + break; + } + } + return ret; +} + +/* + * Any one of the device bound to uio + */ +static inline int +pci_device_bound_uio(void) +{ + struct rte_pci_device *dev = NULL; + + FOREACH_DEVICE_ON_PCIBUS(dev) { + if (dev->kdrv == RTE_KDRV_IGB_UIO || + dev->kdrv == RTE_KDRV_UIO_GENERIC) { + return 1; + } + } + return 0; +} + +/* + * Any one of the device has iova as va + */ +static inline int +pci_device_has_iova_va(void) +{ + struct rte_pci_device *dev = NULL; + struct rte_pci_driver *drv = NULL; + + FOREACH_DRIVER_ON_PCIBUS(drv) { + if (drv && drv->drv_flags & RTE_PCI_DRV_IOVA_AS_VA) { + FOREACH_DEVICE_ON_PCIBUS(dev) { + if (dev->kdrv == RTE_KDRV_VFIO && + rte_pci_match(drv, dev)) + return 1; + } + } + } + return 0; +} + +/* + * Get iommu class of PCI devices on the bus. */ enum rte_iova_mode rte_pci_get_iommu_class(void) { + bool is_bound; + bool is_vfio_noiommu_enabled = true; + bool has_iova_va; + bool is_bound_uio; + + is_bound = pci_device_is_bound(); + if (!is_bound) + return RTE_IOVA_DC; + + has_iova_va = pci_device_has_iova_va(); + is_bound_uio = pci_device_bound_uio(); +#ifdef VFIO_PRESENT + is_vfio_noiommu_enabled = vfio_noiommu_is_enabled() == true ? + true : false; +#endif + + if (has_iova_va && !is_bound_uio && !is_vfio_noiommu_enabled) + return RTE_IOVA_VA; + + if (has_iova_va) { + RTE_LOG(WARNING, EAL, "Some devices want iova as va but pa will be used because.. "); + if (is_vfio_noiommu_enabled) + RTE_LOG(WARNING, EAL, "vfio-noiommu mode configured\n"); + if (is_bound_uio) + RTE_LOG(WARNING, EAL, "few device bound to UIO\n"); + } + return RTE_IOVA_PA; } diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c b/lib/librte_eal/linuxapp/eal/eal_vfio.c index 946df7e31..c8a97b7e7 100644 --- a/lib/librte_eal/linuxapp/eal/eal_vfio.c +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c @@ -816,4 +816,23 @@ vfio_noiommu_dma_map(int __rte_unused vfio_container_fd) return 0; } +int +vfio_noiommu_is_enabled(void) +{ + int fd, ret, cnt __rte_unused; + char c; + + ret = -1; + fd = open(VFIO_NOIOMMU_MODE, O_RDONLY); + if (fd < 0) + return -1; + + cnt = read(fd, &c, 1); + if (c == 'Y') + ret = 1; + + close(fd); + return ret; +} + #endif diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.h b/lib/librte_eal/linuxapp/eal/eal_vfio.h index 5ff63e5d7..26ea8e119 100644 --- a/lib/librte_eal/linuxapp/eal/eal_vfio.h +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.h @@ -150,6 +150,8 @@ struct vfio_config { #define VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u" #define VFIO_GET_REGION_ADDR(x) ((uint64_t) x << 40ULL) #define VFIO_GET_REGION_IDX(x) (x >> 40) +#define VFIO_NOIOMMU_MODE \ + "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode" /* DMA mapping function prototype. * Takes VFIO container fd as a parameter. @@ -210,6 +212,8 @@ int pci_vfio_is_enabled(void); int vfio_mp_sync_setup(void); +int vfio_noiommu_is_enabled(void); + #define SOCKET_REQ_CONTAINER 0x100 #define SOCKET_REQ_GROUP 0x200 #define SOCKET_CLR_GROUP 0x300 -- 2.14.1