From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) by dpdk.org (Postfix) with ESMTP id E6D6CC344 for ; Tue, 4 Aug 2015 13:32:23 +0200 (CEST) Received: from epcpsbgr5.samsung.com (u145.gpu120.samsung.co.kr [203.254.230.145]) by mailout2.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTP id <0NSK00S002PWA2D0@mailout2.samsung.com> for dev@dpdk.org; Tue, 04 Aug 2015 20:32:20 +0900 (KST) Received: from epcpsbgm2new.samsung.com ( [172.20.52.122]) by epcpsbgr5.samsung.com (EPCPMTA) with SMTP id 35.38.17770.4C2A0C55; Tue, 4 Aug 2015 20:32:20 +0900 (KST) X-AuditID: cbfee691-f79ca6d00000456a-08-55c0a2c4041d Received: from epmmp2 ( [203.254.227.17]) by epcpsbgm2new.samsung.com (EPCPMTA) with SMTP id B4.BC.07062.4C2A0C55; Tue, 4 Aug 2015 20:32:20 +0900 (KST) Received: from ubuntu.sisodomain.com ([107.108.207.86]) by mmp2.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTPA id <0NSK00HV52NGTF00@mmp2.samsung.com>; Tue, 04 Aug 2015 20:32:19 +0900 (KST) From: Harpal Singh To: dev@dpdk.org Date: Tue, 04 Aug 2015 17:08:50 +0530 Message-id: <1438688332-3392-2-git-send-email-harpal.sd@samsung.com> X-Mailer: git-send-email 1.8.3.2 In-reply-to: <1438688332-3392-1-git-send-email-harpal.sd@samsung.com> References: <1438688332-3392-1-git-send-email-harpal.sd@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrCLMWRmVeSWpSXmKPExsWyRsSkSvfIogOhBseuKlu8+7SdyWLrQSkH Jo9fC5ayevRtWcUYwBTFZZOSmpNZllqkb5fAlfF7xlXWgq0qFe0v5rM3MN6W62Lk5JAQMJGY Of0+K4QtJnHh3nq2LkYuDiGBFYwSf/5MZoEpWtvQwQKRmMUosbXtClTVT0aJrta7bCBVbAJa EodPbAfrEBEQkPjcuRgsziygIdG7cRUjiC0sYCvxZ9V5dhCbRUBVouXfGWYQm1fARWLXxCls ENsUJJZ9WQsW5xRwlWhoewl2nhBQzbf7p8AWSwg8ZpPYsbqbGWKQgMS3yYeAFnMAJWQlNh1g hpgjKXFwxQ2WCYzCCxgZVjGKphYkFxQnpReZ6hUn5haX5qXrJefnbmIEBubpf88m7mC8f8D6 EKMAB6MSD6/Ay/2hQqyJZcWVuYcYTYE2TGSWEk3OB4Z/Xkm8obGZkYWpiamxkbmlmZI4r470 z2AhgfTEktTs1NSC1KL4otKc1OJDjEwcnFINjPMFp01Y9my+ywu+L3+Fvqrdunxq/T2fZUFr Tjx/vquh2HVZcGHRxSe2L85OsjN7ue/L6rtKqid/bPkWtPx725PLfGclsgUPrVlle+NmKf+e T6o/Wd38WzcLCD7enrroyANJ/oLfYYmpzit/p6/f8LAsZuu6vs/77/Ds5RTmjOHyfP4mePX/ 9CBHJZbijERDLeai4kQAzyKjFkcCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpkkeLIzCtJLcpLzFFi42I5/e+xoO6RRQdCDfYtkrB492k7k8XWg1IO TB6/Fixl9ejbsooxgCmqgdEmIzUxJbVIITUvOT8lMy/dVsk7ON453tTMwFDX0NLCXEkhLzE3 1VbJxSdA1y0zB2i+kkJZYk4pUCggsbhYSd8O04TQEDddC5jGCF3fkCC4HiMDNJCwhjHj94yr rAVbVSraX8xnb2C8LdfFyMkhIWAisbahgwXCFpO4cG89WxcjF4eQwCxGia1tV6Ccn4wSXa13 2UCq2AS0JA6f2A7WISIgIPG5czFYnFlAQ6J34ypGEFtYwFbiz6rz7CA2i4CqRMu/M8wgNq+A i8SuiVPYILYpSCz7shYszingKtHQ9pIVxBYCqvl2/xTbBEbeBYwMqxglUguSC4qT0nON8lLL 9YoTc4tL89L1kvNzNzGCg/+Z9A7Gw7vcDzEKcDAq8fDueLY/VIg1say4MvcQowQHs5IIr0HV gVAh3pTEyqrUovz4otKc1OJDjKZAh01klhJNzgdGZl5JvKGxibmpsamliYWJmaWSOK++yaZQ IYH0xJLU7NTUgtQimD4mDk6pBsY5e9Jc32ZKFIrvS91Zd9DWU3yVf9PdJU+m346tvum24pG3 4i7OLcJW226f9gvvdA184++jtUPj8Va1ljeCNznXPePLZ5B/rcmuVXmcN/HJtEnbv/Zu4MrQ YpS1ODlf6/7+tXPnfZ+k4Fbt7Dhzvf4ltriKjp0rOwtY/m+9ocxvYLji69MC/yVKLMUZiYZa zEXFiQAw8pw6lAIAAA== DLP-Filter: Pass X-MTR: 20000000000000000@CPGS X-CFilter-Loop: Reflected X-Mailman-Approved-At: Tue, 04 Aug 2015 14:10:06 +0200 Cc: Harpal Singh Subject: [dpdk-dev] [PATCH 1/3] vfio: Added hot removal feature for vfio X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Aug 2015 11:32:24 -0000 From: Harpal Singh This patch will add a new API i.e. pci_vfio_unmap_resource. It will basically cleanup all the vfio resources allocated for a device. cleanup includes :- 1) removing vfio_res from vfio_res_list 2) unmap mapped bars 3) close device fd 4) close group fd 5) free vfio_res Signed-off-by: Harpal Singh --- lib/librte_eal/common/include/rte_pci.h | 3 ++ lib/librte_eal/linuxapp/eal/eal_pci.c | 4 +- lib/librte_eal/linuxapp/eal/eal_pci_init.h | 1 + lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 60 ++++++++++++++++++++++++++++++ 4 files changed, 67 insertions(+), 1 deletion(-) diff --git a/lib/librte_eal/common/include/rte_pci.h b/lib/librte_eal/common/include/rte_pci.h index 3fb2d3a..fa8c1a8 100644 --- a/lib/librte_eal/common/include/rte_pci.h +++ b/lib/librte_eal/common/include/rte_pci.h @@ -243,6 +243,9 @@ struct mapped_pci_resource { struct rte_pci_addr pci_addr; char path[PATH_MAX]; int nb_maps; + int vfio_group_fd; + int vfio_dev_fd; + int iommu_group_no; struct pci_map maps[PCI_MAX_RESOURCE]; }; diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c index 0e62f65..edabde5 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci.c @@ -158,7 +158,9 @@ pci_unmap_device(struct rte_pci_device *dev) /* try unmapping the NIC resources using VFIO if it exists */ switch (dev->kdrv) { case RTE_KDRV_VFIO: - RTE_LOG(ERR, EAL, "Hotplug doesn't support vfio yet\n"); +#ifdef VFIO_PRESENT + pci_vfio_unmap_resource(dev); +#endif break; case RTE_KDRV_IGB_UIO: case RTE_KDRV_UIO_GENERIC: diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/eal_pci_init.h index a17c708..9a42ea2 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci_init.h +++ b/lib/librte_eal/linuxapp/eal/eal_pci_init.h @@ -70,6 +70,7 @@ int pci_vfio_write_config(const struct rte_intr_handle *intr_handle, /* map VFIO resource prototype */ int pci_vfio_map_resource(struct rte_pci_device *dev); +int pci_vfio_unmap_resource(struct rte_pci_device *dev); int pci_vfio_get_group_fd(int iommu_group_fd); int pci_vfio_get_container_fd(void); diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c index 0e6c48a..8631653 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c @@ -550,6 +550,18 @@ clear_current_group(void) } +static void +clear_group_by_number(int iommu_group_no) +{ + int i; + for (i = 0; i < vfio_cfg.vfio_group_idx; i++) + if (vfio_cfg.vfio_groups[i].group_no == iommu_group_no) + { + vfio_cfg.vfio_groups[i].group_no = 0; + vfio_cfg.vfio_groups[i].fd = -1; + } +} + /* * map the PCI resources of a PCI device in virtual memory (VFIO version). * primary and secondary processes follow almost exactly the same path @@ -876,6 +888,9 @@ pci_vfio_map_resource(struct rte_pci_device *dev) ioctl(vfio_dev_fd, VFIO_DEVICE_RESET); } + vfio_res->vfio_dev_fd = vfio_dev_fd; + vfio_res->vfio_group_fd = vfio_group_fd; + vfio_res->iommu_group_no = iommu_group_no; if (internal_config.process_type == RTE_PROC_PRIMARY) TAILQ_INSERT_TAIL(vfio_res_list, vfio_res, next); @@ -883,6 +898,51 @@ pci_vfio_map_resource(struct rte_pci_device *dev) } int +pci_vfio_unmap_resource(struct rte_pci_device *dev) +{ + int i; + struct mapped_pci_resource *vfio_res = NULL; + struct mapped_pci_res_list *vfio_res_list = + RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list); + + /// Remove TAILQ entry + if (internal_config.process_type == RTE_PROC_PRIMARY) + { + TAILQ_FOREACH(vfio_res, vfio_res_list, next) { + if (memcmp(&vfio_res->pci_addr, &dev->addr, + sizeof(dev->addr))) + continue; + TAILQ_REMOVE(vfio_res_list, vfio_res, next); + break; + } + } + + if(!vfio_res) + return -1; + + // /* unmap BARs */ + + for (i = 0; i < (int) vfio_res->nb_maps; i++) + { + /* skip non-mmapable BARs */ + if (!vfio_res->maps[i].addr) + continue; + munmap(vfio_res->maps[i].addr, vfio_res->maps[i].size); + } + + + if(TAILQ_EMPTY(vfio_res_list)) + vfio_cfg.vfio_container_has_dma = 0; + close(vfio_res->vfio_dev_fd); + close(vfio_res->vfio_group_fd); + clear_group_by_number(vfio_res->iommu_group_no); + if (internal_config.process_type == RTE_PROC_PRIMARY) + rte_free(vfio_res); + vfio_res = NULL; + return 0; +} + +int pci_vfio_enable(void) { /* initialize group list */ -- 1.9.1