From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f54.google.com (mail-pa0-f54.google.com [209.85.220.54]) by dpdk.org (Postfix) with ESMTP id 669515A9B for ; Tue, 19 Jan 2016 19:57:48 +0100 (CET) Received: by mail-pa0-f54.google.com with SMTP id yy13so363091076pab.3 for ; Tue, 19 Jan 2016 10:57:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mvista-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id; bh=K3YOqpAPvpyxKaZPMvpW7CHADX+3NwRgh94RFHcLjyI=; b=IErw4hBTjS0JWvQDNCAT9LIrSaL8N/fpzT30hA99qmvRv45XBvijDw+Y9GvxiRK5QY kD3C/qPfGMAlRkXy+QxxXNdv6+QEFhd6pnh8k4pjiPVCtax5GAaX9qQ5+bsXnpxcmj2q uQtJCGRlWgwPRxGQ5RXyNU8mipigrcNxdo7K+L0HayK6iIw/IlGzpG+NeK5eoP8pQWR+ H3bWwzi1RQAXX+HEs12y6+CognYoTsDPv0hINP14OzjX27sN+2RtFMSGg3SLD14rJwIP 2p2BnNfWnl0vHbm0b6kzEL6BtpD8siaDKNGU+HBBAj4xE894Zhjxj2z5++D1qUqZu9Tw ZGUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=K3YOqpAPvpyxKaZPMvpW7CHADX+3NwRgh94RFHcLjyI=; b=fAivWv5nxKRtfehIjZj18hj3nSDdAD8qS9bW/bpZn8KOctWCIvKVl6xFFrsA7gMU9T 3n34wB6eFuyVtFSowF9aRmIS4oas/m8XiHwNQvDgPvBfEjPymhQF6+bPkm8EiUpM6lFG r5HhZ0qKRDfIOmPmjKGoszZWEFKIR9tonM2uZKL/PambQpbi8g/ByjLvfbX/ogaLSur1 Nvsd6Udm7oLoPG2g7vFRJoEsL4hZ7uhpUMd1pCrP9a5Hp4cWCjhX8igkTRVGiCIY0HxJ BFd5SeB+QMsK9AOhnZV/Cjc9bly4UqtZbVXE2JIURP2ywfYxOWG9cPOS4opYvdk2/yTo hqVg== X-Gm-Message-State: ALoCoQnpEklVO7qMn8Rt/HeeTNFw4rd2DYbBqXhO1P5lqp7BDT5L9E04NLlCupZMw9NlotRPW1rj1jreYuwG7ulQGc6jfDd2QA== X-Received: by 10.66.237.102 with SMTP id vb6mr46638228pac.133.1453229867663; Tue, 19 Jan 2016 10:57:47 -0800 (PST) Received: from localhost.localdomain ([106.51.24.223]) by smtp.gmail.com with ESMTPSA id fc8sm43530755pab.21.2016.01.19.10.57.44 (version=TLS1_1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 19 Jan 2016 10:57:46 -0800 (PST) From: Santosh Shukla To: dev@dpdk.org Date: Wed, 20 Jan 2016 00:27:22 +0530 Message-Id: <1453229842-15310-1-git-send-email-sshukla@mvista.com> X-Mailer: git-send-email 1.7.9.5 Subject: [dpdk-dev] [PATCH v6 08/11] eal: pci: introduce RTE_KDRV_VFIO_NOIOMMUi driver mode X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jan 2016 18:57:48 -0000 Adding RTE_KDRV_VFIO_NOIOMMU mode in kernel driver. Also including rte_vfio_is_noiommu() helper function. This function will parse /sys/bus/pci/device// and make sure that - vfio noiommu mode set in kernel driver - pci device attached to vfio-noiommu driver only If both condition satisfies then set drv->kdrv = RTE_KDRV_VFIO_NOIOMMU Also did similar changes in virtio_rd/wr, Changes applicable for virtio spec 0.95 only. Signed-off-by: Santosh Shukla --- v5--> v6: - Include pci_dev == NULL check in pci_vfio_is_noiommu(), suggested by Anatoly. v4--> v5: - Removed virtio_xx_init_by_vfio and added new driver mode. - Now no need to parse vfio interface in virtio. As pci_eal module will take of vfio-noiommu driver parsing for virtio or any other future device willing to use vfio-noiommu driver. drivers/net/virtio/virtio_pci.c | 12 ++--- lib/librte_eal/common/include/rte_pci.h | 1 + lib/librte_eal/linuxapp/eal/eal_pci.c | 13 +++-- lib/librte_eal/linuxapp/eal/eal_pci_init.h | 1 + lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 72 ++++++++++++++++++++++++++++ 5 files changed, 90 insertions(+), 9 deletions(-) diff --git a/drivers/net/virtio/virtio_pci.c b/drivers/net/virtio/virtio_pci.c index 0c29f1d..537c552 100644 --- a/drivers/net/virtio/virtio_pci.c +++ b/drivers/net/virtio/virtio_pci.c @@ -60,7 +60,7 @@ virtio_read_reg_1(struct virtio_hw *hw, uint64_t reg_offset) struct rte_pci_device *dev; dev = hw->dev; - if (dev->kdrv == RTE_KDRV_VFIO) + if (dev->kdrv == RTE_KDRV_VFIO_NOIOMMU) ioport_inb(dev, reg_offset, &ret); else ret = inb(VIRTIO_PCI_REG_ADDR(hw, reg_offset)); @@ -75,7 +75,7 @@ virtio_read_reg_2(struct virtio_hw *hw, uint64_t reg_offset) struct rte_pci_device *dev; dev = hw->dev; - if (dev->kdrv == RTE_KDRV_VFIO) + if (dev->kdrv == RTE_KDRV_VFIO_NOIOMMU) ioport_inw(dev, reg_offset, &ret); else ret = inw(VIRTIO_PCI_REG_ADDR(hw, reg_offset)); @@ -90,7 +90,7 @@ virtio_read_reg_4(struct virtio_hw *hw, uint64_t reg_offset) struct rte_pci_device *dev; dev = hw->dev; - if (dev->kdrv == RTE_KDRV_VFIO) + if (dev->kdrv == RTE_KDRV_VFIO_NOIOMMU) ioport_inl(dev, reg_offset, &ret); else ret = inl(VIRTIO_PCI_REG_ADDR(hw, reg_offset)); @@ -104,7 +104,7 @@ virtio_write_reg_1(struct virtio_hw *hw, uint64_t reg_offset, uint8_t value) struct rte_pci_device *dev; dev = hw->dev; - if (dev->kdrv == RTE_KDRV_VFIO) + if (dev->kdrv == RTE_KDRV_VFIO_NOIOMMU) ioport_outb_p(dev, reg_offset, value); else outb_p((unsigned char)value, @@ -117,7 +117,7 @@ virtio_write_reg_2(struct virtio_hw *hw, uint64_t reg_offset, uint16_t value) struct rte_pci_device *dev; dev = hw->dev; - if (dev->kdrv == RTE_KDRV_VFIO) + if (dev->kdrv == RTE_KDRV_VFIO_NOIOMMU) ioport_outw_p(dev, reg_offset, value); else outw_p((unsigned short)value, @@ -130,7 +130,7 @@ virtio_write_reg_4(struct virtio_hw *hw, uint64_t reg_offset, uint32_t value) struct rte_pci_device *dev; dev = hw->dev; - if (dev->kdrv == RTE_KDRV_VFIO) + if (dev->kdrv == RTE_KDRV_VFIO_NOIOMMU) ioport_outl_p(dev, reg_offset, value); else outl_p((unsigned int)value, diff --git a/lib/librte_eal/common/include/rte_pci.h b/lib/librte_eal/common/include/rte_pci.h index 0c667ff..2dbc658 100644 --- a/lib/librte_eal/common/include/rte_pci.h +++ b/lib/librte_eal/common/include/rte_pci.h @@ -149,6 +149,7 @@ enum rte_kernel_driver { RTE_KDRV_VFIO, RTE_KDRV_UIO_GENERIC, RTE_KDRV_NIC_UIO, + RTE_KDRV_VFIO_NOIOMMU, RTE_KDRV_NONE, }; diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c index eb503f0..2936497 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci.c @@ -131,6 +131,7 @@ rte_eal_pci_map_device(struct rte_pci_device *dev) /* try mapping the NIC resources using VFIO if it exists */ switch (dev->kdrv) { case RTE_KDRV_VFIO: + case RTE_KDRV_VFIO_NOIOMMU: #ifdef VFIO_PRESENT if (pci_vfio_is_enabled()) ret = pci_vfio_map_resource(dev); @@ -158,6 +159,7 @@ rte_eal_pci_unmap_device(struct rte_pci_device *dev) /* try unmapping the NIC resources using VFIO if it exists */ switch (dev->kdrv) { case RTE_KDRV_VFIO: + case RTE_KDRV_VFIO_NOIOMMU: RTE_LOG(ERR, EAL, "Hotplug doesn't support vfio yet\n"); break; case RTE_KDRV_IGB_UIO: @@ -353,9 +355,12 @@ pci_scan_one(const char *dirname, uint16_t domain, uint8_t bus, } if (!ret) { - if (!strcmp(driver, "vfio-pci")) - dev->kdrv = RTE_KDRV_VFIO; - else if (!strcmp(driver, "igb_uio")) + if (!strcmp(driver, "vfio-pci")) { + if (pci_vfio_is_noiommu(dev) == 0) + dev->kdrv = RTE_KDRV_VFIO_NOIOMMU; + else + dev->kdrv = RTE_KDRV_VFIO; + } else if (!strcmp(driver, "igb_uio")) dev->kdrv = RTE_KDRV_IGB_UIO; else if (!strcmp(driver, "uio_pci_generic")) dev->kdrv = RTE_KDRV_UIO_GENERIC; @@ -630,6 +635,7 @@ int rte_eal_pci_read_bar(const struct rte_pci_device *device, switch (device->kdrv) { case RTE_KDRV_VFIO: + case RTE_KDRV_VFIO_NOIOMMU: return pci_vfio_read_bar(intr_handle, buf, len, offset, bar_idx); default: @@ -647,6 +653,7 @@ int rte_eal_pci_write_bar(const struct rte_pci_device *device, switch (device->kdrv) { case RTE_KDRV_VFIO: + case RTE_KDRV_VFIO_NOIOMMU: return pci_vfio_write_bar(intr_handle, buf, len, offset, bar_idx); default: diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/eal_pci_init.h index 3bc592b..60b95d7 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci_init.h +++ b/lib/librte_eal/linuxapp/eal/eal_pci_init.h @@ -60,6 +60,7 @@ int pci_uio_write_config(const struct rte_intr_handle *intr_handle, int pci_vfio_enable(void); int pci_vfio_is_enabled(void); +int pci_vfio_is_noiommu(struct rte_pci_device *pci_dev); int pci_vfio_mp_sync_setup(void); /* access config space */ diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c index df407ef..33f8808 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c @@ -973,4 +973,76 @@ pci_vfio_is_enabled(void) { return vfio_cfg.vfio_enabled; } + +int +pci_vfio_is_noiommu(struct rte_pci_device *pci_dev) +{ + FILE *fp; + struct rte_pci_addr *loc; + const char *path = "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"; + char filename[PATH_MAX] = {0}; + char buf[PATH_MAX] = {0}; + + /* + * 1. chk vfio-noiommu mode set in kernel driver + * 2. verify pci device attached to vfio-noiommu driver + * example: + * cd /sys/bus/pci/drivers/vfio-pci//iommu_group + * > cat name + * > vfio-noiommu --> means virtio_dev attached to vfio-noiommu driver + */ + + if (pci_dev == NULL) + return -1; + + fp = fopen(path, "r"); + if (fp == NULL) { + RTE_LOG(ERR, EAL, "can't open %s\n", path); + return -1; + } + + if (fread(buf, sizeof(char), 1, fp) != 1) { + RTE_LOG(ERR, EAL, "can't read from file %s\n", path); + fclose(fp); + return -1; + } + + if (strncmp(buf, "Y", 1) != 0) { + RTE_LOG(ERR, EAL, "[%s]: vfio: noiommu mode not set\n", path); + fclose(fp); + return -1; + } + + fclose(fp); + + /* 2. chk whether attached driver is vfio-noiommu or not */ + loc = &pci_dev->addr; + snprintf(filename, sizeof(filename), + SYSFS_PCI_DEVICES "/" PCI_PRI_FMT "/iommu_group/name", + loc->domain, loc->bus, loc->devid, loc->function); + + /* check for vfio-noiommu */ + fp = fopen(filename, "r"); + if (fp == NULL) { + RTE_LOG(ERR, EAL, "can't open %s\n", filename); + return -1; + } + + if (fread(buf, sizeof(char), sizeof("vfio-noiommu"), fp) != + sizeof("vfio-noiommu")) { + RTE_LOG(ERR, EAL, "can't read from file %s\n", filename); + fclose(fp); + return -1; + } + + if (strncmp(buf, "vfio-noiommu", strlen("vfio-noiommu")) != 0) { + RTE_LOG(ERR, EAL, "not a vfio-noiommu driver\n"); + fclose(fp); + return -1; + } + + fclose(fp); + + return 0; +} #endif -- 1.7.9.5